Subject: Re: [idn] Re: 7 bits forever!

From: "D. J. Bernstein" <[email-masked]>
Date: 2 Apr 2002 05:14:22 -0000

Dan Kohn writes:
> a UTF8HEADERS ESMTP extension

No! 8-bit support is not an option. 8-bit support is not something to be
negotiated. 8-bit support is _required_. Do you want implementors in
twenty years to be dealing with 7-bit MTAs?

> The challenge is that any mail relay implementing this
> extension MUST be able to downconvert headers to 7-bit RFC 2047 [3]
> compliant encoded words if the next hop does not support UTF8HEADERS.

Sorry to bother you with the facts, but the 8BITMIME requirement that
you're mindlessly imitating is flagrantly violated in practice by many
MTAs, including most versions of sendmail. (All versions from 6.57 in
1993 until 8.7 in 1995; at least a few years after that if the popular 8
flag is set; I haven't checked the most recent versions.)

Furthermore, converting an 8-bit email address to 7 bits would destroy
interoperability. This isn't just a Q-P display failure; this is mail
delivered to the wrong address. Reliably undoing the conversion means
upgrading every mail program on the Internet, rather than just the 7-bit
programs that are actually at fault.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Keith Moore <[email-masked]>
Date: Tue, 02 Apr 2002 02:30:05 -0500

> I agree.  8BITMIME has largely failed (strictly speaking, announcing
> 8BITMIME requires body rewrite features in mail relay software).  We
> should not introduce similar functionality, as an RFC for that will
> be ignored, implemented incorrectly, or not implemented due to the
> involved complexity.

It's a different world now.  At the time 8BITMIME was defined few MUAs 
supported MIME, but many could display 8bit text in message bodies
as long as it wasn't MIME encoded, and sender and receiver happened 
to use the same character set.  And given the demographics of the 
Internet at that time, there was at least a fair chance that both
sender and receiver used iso-8859-1 (though many others were in use).

At the present time most MUAs support MIME encoding.  Essentially all
MUAs that support utf-8 also support MIME; thus they are able to decode 
2047 headers and use the proper character encoding for display. OTOH 
unlabeled 8bit text in message headers is not treated consistently
across MUAs, and there are many more character sets in wide use than when
8BITMIME was defined.  Though it is generally possible to distinguish
between several of the popular encodings even without tagging, it's 
probably the case that header text encoded in 2047 is more likely to 
work with existing MUAs than unencoded utf-8.

The real problem with 8BITMIME was that it essentially forced the
MTA to make a guess about whether the recipient's MUA supported MIME.
I agree that we don't want to do that again.  MTAs should be encouraged 
to handle relayed messages as transparently as possible.

However, the benefit in doing so is mostly for the long term.  It still 
won't permit us to use unencoded utf-8 in message headers in the near term. 

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 2 Apr 2002 08:08:40 -0000

By the way, in case anyone thinks that my sarcastic ``7 bits forever!''
is an exaggeration, here's an old quote from Mark Crispin: ``7-bit MTA's
will never go away. Ever. Mail goes on many paths besides IP, some of
which are forever 7-bits. You will always have to solve the problem of
getting your 8-bit mail into 7-bits for some 7-bit MTA.''

It is reasonable to demand time to upgrade, so that there are no
interoperability problems. It is not reasonable to demand that this
7-bit garbage be allowed to remain on the network forever, at the
expense of all future implementors and users.

> > MTAs, including most versions of sendmail.
> qmail announces 8BITMIME support without actually implementing it, too.

Of course. Announcing 8BITMIME in the SMTP server benefits the users.
Deliberately ignoring the 8BITMIME requirements in the SMTP client also
benefits the users.

I thought it was clear that I was criticizing 8BITMIME, not sendmail's
violation of it. Essentially everything that sendmail has done wrong
with 8-bit support has complied with the relevant IETF specifications,
and essentially everything that sendmail has done right with 8-bit
support has violated the relevant IETF specifications.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 02 Apr 2002 00:18:42 -0800

>which are forever 7-bits. You will always have to solve the problem of

the word sarcasm is about tone, not truth.

in fact, there is quite a bit of experience that shows just how long 
established systems and services do stay around.

the difference between 20 years and forever is not worth measuring.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: Keith Moore <[email-masked]>
Date: Tue, 02 Apr 2002 08:53:56 -0500

> expense of all future implementors and users.

forever is a long time.  and yet, even if all use of quoted-printable
were to cease today, there would still be a need for new MUAs to be
able to read old messages stored on servers for many more years, perhaps
decades.

using raw utf-8 in message headers wouldn't replace 2047. instead,
it would add yet another format that MUAs are expected to parse
and display. it would not reduce complexity, it would add to it.

Keith

---

From: Florian Weimer <[email-masked]>
Date: Tue, 02 Apr 2002 08:48:14 +0200

"D. J. Bernstein" <[email-masked]> writes:

> negotiated. 8-bit support is _required_.

> MTAs, including most versions of sendmail.

qmail announces 8BITMIME support without actually implementing it,
too.  (This defect only affects mail relays, though.)

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Tue, 2 Apr 2002 10:19:45 +0200

> expense of all future implementors and users.

Why change? It works as designed. If we change this, we risk
fragmenting the net. Yes, it costs some more bits when 7-bit
downgrading is used, but that is not a general cost for
normal email use. And it costs some maintenance and some
space in MTAs, but that is not much in total.

keld

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Tue, 2 Apr 2002 13:42:00 -0500

> and display. it would not reduce complexity, it would add to it.

If we add new MIME headers and ESMTP commands, MUA and MTA should be able to
have a smooth transition from 7bit encoded 8bits to UTF8. If there is a
requirement of change anyway, why not change it to something that is more
simple and well support by the Unicode Consortium.