Subject: Re: [idn] Re: 7 bits forever!

From: Keith Moore <[email-masked]>
Date: Tue, 02 Apr 2002 00:20:17 -0500

> I believe that Einar could be most easily satisfied with something along
> the lines of a UTF8HEADERS ESMTP extension, which would specify both
> that 8 bit character are permitted in the header and that those
> characters MUST be interpreted as UTF-8.  

there are lots of problems with this idea.  for instance, there's no 
way for SMTP to know whether the recipient's MUA can deal with 
utf-8 even in 2047 encoded form, to say nothing about unencoded form.   
and it's not as simple as saying just use utf-8 in the message header -
there's also the question of which characters are legal and which aren't,
which ones are separator characters, which ones are white space, etc.

then consider that, except for the abliity to have non-ascii email
addresses, there's no functional advantage to having raw utf-8 in headers.
there might be a slight win to having a more efficient encoder, but 
MUAs would still have to implement both utf-8 and 2047 for many more years.

let's get the details of non-ascii email addresses worked out before
we start spewing utf-8 in random places in the message header.

Keith

---

From: Einar Stefferud <[email-masked]>
Date: Mon, 1 Apr 2002 22:40:49 -0800

I am definitely not the party to be satisfied here;-)...

I only jumped in to remind that the issues of IETF Message Bodies and 
Message Headers are separate and very different, so that the history 
of the "just send 8 bits" appeared to perhaps need some refreshment.

I think that Keith is talking along the same lines, but beyond what I 
have said, I am not able to contribute more.

I have stopped doing IETF work since the Chicago IETF where I 
finished my work with MHTML, and no longer had travel support for any 
more IETF work.

And then I also retired, more or less, mostly.

So, having had my say for whatever it might have been worth,
I would just as soon be omitted from the rest of this discussion.

Cheers;-)...\Stef

---

From: "Dan Kohn" <[email-masked]>
Date: Mon, 1 Apr 2002 17:31:28 -0800

I believe that Einar could be most easily satisfied with something along
the lines of a UTF8HEADERS ESMTP extension, which would specify both
that 8 bit character are permitted in the header and that those
characters MUST be interpreted as UTF-8.  It would be expected that this
extension would normally be used in conjunction with 8BITMIME [1] or
BINARYMIME [2].  The challenge is that any mail relay implementing this
extension MUST be able to downconvert headers to 7-bit RFC 2047 [3]
compliant encoded words if the next hop does not support UTF8HEADERS.

This would seem to be a middle ground between Dan's desire for
just-send-8 and other's demand that existing RFC 2821 [4] compliant
relays not be broken by adding support for UTF-8 headers.

This should be an easy RFC to write up.  If there is interest from
implementers, I would be happy to do so.  In any event, I would suggest
taking follow-up to ietf-smtp [5].

[1] http://www.ietf.org/rfc/rfc1652.txt
[2] http://www.ietf.org/rfc/rfc3030.txt
[3] http://www.ietf.org/rfc/rfc2047.txt
[4] http://www.ietf.org/rfc/rfc2821.txt
[5] http://www.imc.org/ietf-smtp/

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>
Essays announced on <mailto:[email-masked]>

-----Original Message-----
From: Einar Stefferud [mailto:[email-masked]]
Sent: Monday, April 01, 2002 11:47
To: [email-masked]; [email-masked]
Subject: Re: [idn] Re: 7 bits forever!

Hi John -- My thesis is that the problem lies now in the SMTP
standards, where the current "Send 8-bits" problem has always
resided.  The MIME working group could not solve that problem, and so
did not, but it made mail work since its deployment, which was the
objective in the MIME WG.  inside the RFC822 Body there is only one
way to handle 8-bit data, and that is to tag it, and bag it.  Even if
you can transmit 8-bit data without encoding it in the body of an
SMTP message, it needs to be tagged and delimited in the body.

MIME Quoted-Printable was designed to be a work-around-damage
solution at the time, and was not design to remove the need for
handling 8-bit data in SMTP.

So, my only point is that it appears that some work needs to be done
in the SMTP standards, and beating up on the MIME WG does not compute
given this observation.

That is all...  Just trying to shift the focus of this thread to
where someone might think of a way to solve the current problem.

In the meantime, Thankfully, Mail still works, which was the MIME
objective for Quoted-Printable.

Cheers;-)....\Stef

At 12:41 PM -0500 4/1/02, John C Klensin wrote:
>Stef,
>
>Either I misunderstand your message, or we have a bit of a
>disconnect.
>
>The SMTP extension "8BITMIME" was specified fairly early in the
>game.  My impression is that it is fairly widely deployed and
>used, possibly to the extent that most non-ASCII traffic is
>moving that way; someone on this list (or the SMTP extensions
>one) may have some numbers.  Where it is deployed, MIME is still
>required to label ("tag") the data -- one can still not tell one
>sort of 8bit character from another, and there is, I think,
>still much more traffic on the network in ISO 8859-N variants,
>or IS 2022-based systems, or local character sets than there is
>UTF-8 encoding of Unicode -- but quoted-printable (or base64)
>are not, unless the mail store or mail retrieval implementation
>cannot deal with 8bit materials.
>
>       john
>
>
>--On Monday, 01 April, 2002 08:59 -0800 Einar Stefferud
><[email-masked]> wrote:
>
>  > Not to pick on Dan's particular message.
>  > It was just the last one to provide me with a handy response
>  > point.
>  >
>  > Looking back, the original 1990-1991 argument was between
>  > "Just send 8 bits" with no tagging of the content, vs "tagging
>  > and bagging" to deal with the fact that 8-bit at that time
>  > would break too many systems, and the problem of needing to
>  > tag the types of text characters that were contained in RFC822
>  > message bodies.
>  >
>  > But, as there were two problems to solve (8 bit SMTP ) and
>  > (tagging RFC822 content), the decision was made (after a very
>  > long and very heated argument) to split that problem according
>  > to the boundaries of SMTP and RFC822.
>  >
>  > In fact, the two problems were and still are totally distinct.
>  >
>  > The RFC822 solution group produced MIME with Quoted-Printable
>  > as its charter specified, to assure interworkability as soon
>  > as possible.
>  >
>  > The 8-bit folks did not do anything then or since to require
>  > 8-bit carriage.
>  >
>  > So, it is time for a working group to solve this problem.
>  >
>  > It is not going to help anyone to argue about the past, as in
>  > fact, it was very rational at the time, and one side of the
>  > problem was ignored as soon as quoted printable relieved the
>  > immediate pressure for an SMTP solution.
>  >
>  > So, now is a good time for some constructive discussion of
>  > ways to solve the charset problems, hopefully with some useful
>  > charset tagging tools for the non-RFC-822 parts of the SMTP
>  > envelope.  (But I do not mean to specify either the work or
>  > the results at this point.)
>  >
>  > It is indeed unfortunate the the DRUMS WG did not find a way
>  > to solve the 8-bit SMTP problem, but now is the time to get to
>  > it.  There should not be too much 7-bit SMTP code to fix by
>  > now.
>  >
>  > Cheers...\Stef
>  >
>  >
>  >
>  >
>  > At 8:35 AM +0000 4/1/02, D. J. Bernstein wrote:
>  >> Claus Faerber writes:
>  >>  > D. J. Bernstein <[email-masked]> schrieb/wrote:
>  >>  > > I'm not saying that Quoted-Printable had no short-term
>  >>  > > benefits for
>  > its
>  >>  > > short-term costs. I'm saying that, viewed from our
>  >>  > > long-term
>  > perspective
>  >>  > > eleven years later, the failure to require 8-bit
>  >>  > > transparency was an amazingly stupid decision.
>  >>  > From our present perspective, that's true. Back then, it
>  >>  > might have been the best solution.
>  >>
>  >> No. The failure to require 8-bit transparency was
>  >> inexcusable. Every approach that failed to require 8-bit
>  >> transparency could have been dramatically improved. Consider,
>  >> for example, these three approaches:
>  >>
>  >>    (1) Quoted-Printable;
>  >>    (2) Quoted-Printable plus ``you must handle unencoded
>  >>    8-bit data too''; (3) pure 8-bit without this 7-bit
>  >>    garbage.
>  >>
>  >> Whether or not you understand that #3 would have been better
>  >> than #2, surely you understand that #2 would have been vastly
>  >> better than #1.
>  >>
>  >>  > Further, remember that the first MIME standards date back
>  >>  > to 1992. Back then, Unicode was brand-new and UTF-8 only
>  >>  > came with the 2.0 version in 1996. Without UTF-8, you just
>  >>  > could not even think about using Unicode in message
>  >>  > headers; and without Unicode, you could not solve the
>  >>  > charset-labelling problem.
>  >>
>  >> Get your facts straight. First, UTF-8 was introduced years
>  >> before 1996, although under another name. Second, even
>  >> without knowing about UTF-8, people _were_ thinking about
>  >> Unicode in headers, and proposed several workable approaches.
>  >> Third, even without Unicode, there were several solutions to
>  >> the character-set labelling problem.
>  >>
>  >> Anyway, none of this is relevant to IETF's acceptance of
>  >> 7-bit garbage in 1991, and none of it is relevant to IETF's
>  >> acceptance of 7-bit garbage today.
>  >>
>  >>  > The movement towards UTF-8 everywhere is quite new.
>  >>
>  >> Again, get your facts straight. The ``movement'' started with
>  >> X-Open, Rob Pike, and Ken Thompson a decade ago. RFC 2277,
>  >> requiring UTF-8 support for all text on the Internet, is four
>  >> years old.
>  >>
>  >> ---D. J. Bernstein, Associate Professor, Department of
>  >> Mathematics, Statistics, and Computer Science, University of
>  >> Illinois at Chicago
>  >
>  >
>  >
>  >
>  >
>  >

-
This message was passed through [email-masked],
which
is a sublist of [email-masked]. Not all messages are passed.
Decisions on what to pass are made solely by Raffaele D'Albenzio.

---

From: [email-masked] (Claus Färber)
Date: 11 Apr 2002 02:15:00 +0200

> compliant encoded words if the next hop does not support UTF8HEADERS.

To encode headers using the methods described in RFC 2047, you
need to know the exact structure of the header. This is impossible  
for future headers.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0