Subject: Re: Language Tagging within Unicode
From: "Charles Lindsey" <[email-masked]>
To: [email-masked]
Date: Fri, 31 Jan 2003 15:38:01 GMT
Newsgroups: local.usefor,local.mime
List-Id: <ietf-822.imc.org>

In <4.2.0.58.J.20030127183119.054ec5d8@localhost> Martin Duerst <[email-masked]> writes:

>Hello Charles,

>At 22:30 03/01/27 +0000, Charles Lindsey wrote:
>>Sorry, your mail address at <[email-masked]> seems not to work
>>any longer, so I am trying w3.org. The original was Cced to the Usefor
>>and rfc822-list, and I have set Reply-To there.

>Sorry, and thanks. Please keep me cc'ed; I'm not currently on
>either list.

>One way to check whether the two communities are actually pulling in
>opposite directions would be to check to what extent things such as
>Tag Characters (U+E0000-U+E007F) and/or RFC 2231 are actually implemented
>and used. I don't know of actual use, but I may just not be aware of it.

I doubt anyone actually uses the Unicode Tag characters because, in
bodies, there is the Content-Language header. RFC 2231 introduces language
codes into headers (e.g. by way of updating RFC 2047), but I wouldn't know
how widely used they are if, indeed, anyone actually implements them. I
suspect that it is the Chinese and the Japanese who will be the main
users, if any.  There is, of course, no significant current usage
(experiments apart) of raw UTF-8 in headers.

>>My immediate concern is with the possibility that raw UTF-8 will become
>>the charset for headers within Netnews (and maybe even for Email one day).

>That in and by itself would not be a concern at all for me.
>I think that for Netnews, and also for Email, using raw UTF-8
>is the right way to go.

>>It is in the nature of headers that the texts within them are short - they
>>are not intended for long essays (that is what message bodies are for). So
>>it could be argued that language tagging within them is hardly necessary;
>>in which case the question arises as to whether leaving it to Unicode
>>tagging for people who really want to do it would be enough.
>>
>>Effectively, one would say "It will usually be unnecessary to use language
>>tagging within headers but, if it is considered necessary, then the
>>language tagging defined for Unicode MAY be used" (note that the
>>contexts where this would be applicable are all for human consumption).

>>So is that an allowable usage according to Unicode 3.2? Observe that
>>headers using raw UTF-8 will not be using any MIME protocol.
>>

>My proposal for how to proceed, based on my understanding of
>Internet protocols and their usage, and of Unicode,
>would be to not use language tagging for headers (i.e. prefer
>a tiny degree of 'suboptimality' that in most cases won't matter
>at all over a complex design with low chance of getting
>implemented (correctly, or at all)).

The only real reason we are having to consider them at all is because RFC
2277 says you MUST make some provision for language tagging in any IETF
standard. I am sure they had bodies rather than headers in mind when they
wore that, but there it is (though one could argue that RFC 2277 is only
a BCP document).

>As an alternative, a wording like the one you propose, which
>leaves this to the implementers to 'vote with their feet'
>(as my understanding is they have done up to now) will probably
>also not cause too much damage.

Yes, that was the idea of my wording. "You MAY do it this way if you really
really must, but we really recommend you not to bother". Japanese browsers
might go so far as to recognise them in that context, though I doubt many
Japanese would bother to include them in their headers (except perhaps for
their names which would likely be configured with great care into their
browsers).

>However, in that case, I think it would be important to make a
>clear exception for identifiers (e.g. newsgroup names, email addresses,
>as opposed to simply descriptive text suc as subjects,...),
>where adding language tag characters would lead to a breakdown
>in interoperability and where they therefore clearly should
>be prohibited.

Absolutely so. They will be totally forbidden in newsgroup-names.

Anyway, any more opinions from either of these lists?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 30 Jan 2003 12:17:35 -0500
To: [email-masked], [email-masked]
From: Martin Duerst <[email-masked]>
Subject: Re: Language Tagging within Unicode
Sender: [email-masked]

Hello Charles,

At 22:30 03/01/27 +0000, Charles Lindsey wrote:
>Sorry, your mail address at <[email-masked]> seems not to work
>any longer, so I am trying w3.org. The original was Cced to the Usefor
>and rfc822-list, and I have set Reply-To there.

Sorry, and thanks. Please keep me cc'ed; I'm not currently on
either list.

See below for comments.

>--- Below this line is a copy of the message.
>
>Date: Mon, 27 Jan 2003 14:37:34 +0000 (GMT)
>From: Charles Lindsey <[email-masked]>
>Reply-To: Charles Lindsey <[email-masked]>
>Subject: Language Tagging within Unicode
>To: "Martin J. Duerst" <[email-masked]>
>Cc: [email-masked], [email-masked]
>
>This message is addressed primarily to Martin Duerst, but is Cced to
>the Usefor and ietf-822 lists. Will people replying please ensure that
>Martin is copied, since I am not aware that he regularly reads either of
>those lists.
>
>
>There currently appears to be some confusion between the Unicode community and
>the Email community as to the correct usage of Language Tagging.
>
>Within Unicode (since version 3.1) there have existed the Tag Characters
>U+E0000-U+E007F which can, in principle, change the language over and over
>again within a single text.
>
>However, there are discouraging remarks about the use of this facility:
>
>     ... However, the use of these characters is strongly discouraged. The
>     characters in this block are reserved for use with special protocols.
>     They are not to be used in the absence of such protocols, or with any
>     protocols that provide alternate means for language tagging, such as
>     HTML or XML.  The requirement for language information embedded in
>     plain text data is often overstated. ...
>
>(However, there is no defined list of those "special protocols".)

[aside: I think it would be inappropriate for Unicode to say which
  protocols should use them or not. I think one thing the above is
  saying is: don't assume you can use them unless the protocol you
  are using explicitly says you can.]

>     Because of the extra implementation burden, language tags should be
>     avoided in plain text unless language information is required and it is
>     known that the receivers of the text will properly recognize and maintain
>     the tags. ...
>
>     Language tags should also be avoided wherever higher-level protocols,
>     such as a rich-text format, HTML or MIME, provide language attributes.
>     This practice prevents cases where the higher-level protocol and the
>     language tags disagree. See Unicode Technical Report #20, "Unicode in
>     XML and other Markup Languages".
>
>All of which seems to be saying "Don't use Unicode language tagging if the
>protocol provides some alternative, and even if it doesn't, think twice
>about whether you really, really need to do it".

I would say that this is a fair summary.

>Indeed, it is rumoured
>that the next revision of Unicode will deprecate them even more.

I seem to remember having heard such rumors, although I couldn't
confirm off the top of my head.

>OTOH, within the Email community, I find the following within RFC 2231,
>when introducing a language-tag extension to RFC 2047:
>
>     In the future it is likely that some character sets will provide
>     facilities for inline language labeling. Such facilities are
>     inherently more flexible than those defined here as they allow for
>     language switching in the middle of a string.
>
>     If and when such facilities are developed they SHOULD be used in
>     preference to the language labeling facilities specified here. Note
>     that all the mechanisms defined here allow for the omission of
>     language labels so as to be able to accommodate this possible future
>     usage.
>
>And also within RFC 3066 I find:
>
>     The issue of deciding upon the rendering of a character set based on
>     the language tag is not addressed in this memo; however, it is
>     thought impossible to make such a decision correctly for all cases
>     unless means of switching language in the middle of a text are
>     defined (for example, a rendering engine that decides font based on
>     Japanese or Chinese language may produce suboptimal output when a
>     mixed Japanese-Chinese text is encountered).
>
>Which at least suggests that the ability to change languages in mid-text
>is a useful feature to have.
>
>Now whilst that may or may not represent the current view of the Email
>community, it is certainly on the record as such, and so the two
>communities would seem to be pulling in opposite directions. This really
>needs to be clarified.

One way to check whether the two communities are actually pulling in
opposite directions would be to check to what extent things such as
Tag Characters (U+E0000-U+E007F) and/or RFC 2231 are actually implemented
and used. I don't know of actual use, but I may just not be aware of it.

>My immediate concern is with the possibility that raw UTF-8 will become
>the charset for headers within Netnews (and maybe even for Email one day).

That in and by itself would not be a concern at all for me.
I think that for Netnews, and also for Email, using raw UTF-8
is the right way to go.

>It is in the nature of headers that the texts within them are short - they
>are not intended for long essays (that is what message bodies are for). So
>it could be argued that language tagging within them is hardly necessary;
>in which case the question arises as to whether leaving it to Unicode
>tagging for people who really want to do it would be enough.
>
>Effectively, one would say "It will usually be unnecessary to use language
>tagging within headers but, if it is considered necessary, then the
>language tagging defined for Unicode MAY be used" (note that the
>contexts where this would be applicable are all for human consumption).

I think that RFC 3066 is correct in saying 'may produce suboptimal
output' (when a mixed Japanese-Chinese text is encountered).
I think this is just an observation, without any kind of analysis
as to how suboptimal that output is.

I think we can agree that the rendering of email headers
(and also of email bodies) is in many ways 'suboptimal',
e.g. not of outstanding typographic quality, and so on.
In almost all cases, this suboptimality is widely accepted.
Of course, if you have a Chinese or Japanese name and want
it shown in a particular form, then this suboptimality is
less acceptable to you. But then again, there are people
who insist on particular shapes for a character in their
name, for example within Japan, in a way which cannot be
expressed in the current national standard (and therefore
not in current email practice) and also cannot be distinguished
by using language tags.
[Please note that there are a lot of heuristics (e.g. based
on analysis of the characters used; using user preference;
using the 'Content-Language' header on the body;...) and
other tricks (e.g. using a font that de-emphasizes the
differences, which will be the case anyway in many of the
small-size fonts used for headers) to easily get a very good
display in almost all cases.

>So is that an allowable usage according to Unicode 3.2? Observe that
>headers using raw UTF-8 will not be using any MIME protocol.
>
>It this is not allowable, then the only way to tag these short texts would
>be through the use of RFC 2047 or RFC 2231 in place of raw UTF-8, which is
>OK so far as it goes, but might not be so if those usages are phased out
>in the future (we are talking many many years down the line here, but it
>could happen one day).

My understanding of Unicode 3.2 is that if you really, really
think that you need to use these tags, then that's what they
are for. But before using them, you should really, really
consider the issue carefully.

My proposal for how to proceed, based on my understanding of
Internet protocols and their usage, and of Unicode,
would be to not use language tagging for headers (i.e. prefer
a tiny degree of 'suboptimality' that in most cases won't matter
at all over a complex design with low chance of getting
implemented (correctly, or at all)).

As an alternative, a wording like the one you propose, which
leaves this to the implementers to 'vote with their feet'
(as my understanding is they have done up to now) will probably
also not cause too much damage.

However, in that case, I think it would be important to make a
clear exception for identifiers (e.g. newsgroup names, email addresses,
as opposed to simply descriptive text suc as subjects,...),
where adding language tag characters would lead to a breakdown
in interoperability and where they therefore clearly should
be prohibited.

Regards,    Martin.

Date: Wed, 29 Jan 2003 13:11:02 +0000
To: [email-masked]
From: Graham Klyne <[email-masked]>
Subject: XML format for email and other messages
Sender: [email-masked]

For information:

Re:  http://www.ietf.org/internet-drafts/draft-klyne-message-xml-00.txt

I'm reviving an old project of mine to define an XML format to carry 
information from email and related message formats.  This work started life 
as a proposal for an instant messaging format, but since attracted interest 
from some XML application developers who wanted to process email-related 
information (e.g. weblog interfaces, RSS information feeds;  my employer at 
the time used in an email archive development).  This is NOT an attempt to 
do email-in-XML.

My plan for this is to update the document with feedback I'm getting from 
interested folks and request publication as an informational RFC.

Paul Hoffman has kindly provided a mailing list and web archive page at the 
IMC site:
     http://www.imc.org/ietf-message-xml/

#g

-------------------
Graham Klyne
<[email-masked]>

To: [email-masked]
Xref: clerew local.mime:2356 local.usefor:15634
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Language Tagging within Unicode
Date: Tue, 28 Jan 2003 16:51:06 GMT
Lines: 20
Sender: [email-masked]

In <[email-masked]> Charles Lindsey <[email-masked]> writes:

>This message is addressed primarily to Martin Duerst, but is Cced to
>the Usefor and ietf-822 lists. Will people replying please ensure that
>Martin is copied, since I am not aware that he regularly reads either of
>those lists.

It seems the email address I used for Martin is out of date. If you are
replying, please use <[email-masked]>.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 27 Jan 2003 14:37:34 +0000 (GMT)
From: Charles Lindsey <[email-masked]>
Reply-To: Charles Lindsey <[email-masked]>
Subject: Language Tagging within Unicode
To: "Martin J. Duerst" <[email-masked]>
Cc: [email-masked], [email-masked]
Content-MD5: FqRwvi1KcKqvT6ZVsza8XA==
Sender: [email-masked]

This message is addressed primarily to Martin Duerst, but is Cced to
the Usefor and ietf-822 lists. Will people replying please ensure that
Martin is copied, since I am not aware that he regularly reads either of
those lists.

There currently appears to be some confusion between the Unicode community and
the Email community as to the correct usage of Language Tagging.

Within Unicode (since version 3.1) there have existed the Tag Characters
U+E0000-U+E007F which can, in principle, change the language over and over
again within a single text.

However, there are discouraging remarks about the use of this facility:

    ... However, the use of these characters is strongly discouraged. The
    characters in this block are reserved for use with special protocols.
    They are not to be used in the absence of such protocols, or with any
    protocols that provide alternate means for language tagging, such as
    HTML or XML.  The requirement for language information embedded in
    plain text data is often overstated. ...

(However, there is no defined list of those "special protocols".)

    Because of the extra implementation burden, language tags should be
    avoided in plain text unless language information is required and it is
    known that the receivers of the text will properly recognize and maintain
    the tags. ...

    Language tags should also be avoided wherever higher-level protocols,
    such as a rich-text format, HTML or MIME, provide language attributes.
    This practice prevents cases where the higher-level protocol and the
    language tags disagree. See Unicode Technical Report #20, "Unicode in
    XML and other Markup Languages".

All of which seems to be saying "Don't use Unicode language tagging if the
protocol provides some alternative, and even if it doesn't, think twice
about whether you really, really need to do it". Indeed, it is rumoured
that the next revision of Unicode will deprecate them even more.

OTOH, within the Email community, I find the following within RFC 2231,
when introducing a language-tag extension to RFC 2047:

    In the future it is likely that some character sets will provide
    facilities for inline language labeling. Such facilities are
    inherently more flexible than those defined here as they allow for
    language switching in the middle of a string.

    If and when such facilities are developed they SHOULD be used in
    preference to the language labeling facilities specified here. Note
    that all the mechanisms defined here allow for the omission of
    language labels so as to be able to accommodate this possible future
    usage.

And also within RFC 3066 I find:

    The issue of deciding upon the rendering of a character set based on
    the language tag is not addressed in this memo; however, it is
    thought impossible to make such a decision correctly for all cases
    unless means of switching language in the middle of a text are
    defined (for example, a rendering engine that decides font based on
    Japanese or Chinese language may produce suboptimal output when a
    mixed Japanese-Chinese text is encountered).

Which at least suggests that the ability to change languages in mid-text
is a useful feature to have.

Now whilst that may or may not represent the current view of the Email
community, it is certainly on the record as such, and so the two
communities would seem to be pulling in opposite directions. This really
needs to be clarified.

My immediate concern is with the possibility that raw UTF-8 will become
the charset for headers within Netnews (and maybe even for Email one day).
It is in the nature of headers that the texts within them are short - they
are not intended for long essays (that is what message bodies are for). So
it could be argued that language tagging within them is hardly necessary;
in which case the question arises as to whether leaving it to Unicode
tagging for people who really want to do it would be enough.

Effectively, one would say "It will usually be unnecessary to use language
tagging within headers but, if it is considered necessary, then the
language tagging defined for Unicode MAY be used" (note that the
contexts where this would be applicable are all for human consumption).

So is that an allowable usage according to Unicode 3.2? Observe that
headers using raw UTF-8 will not be using any MIME protocol.

It this is not allowable, then the only way to tag these short texts would
be through the use of RFC 2047 or RFC 2231 in place of raw UTF-8, which is
OK so far as it goes, but might not be so if those usages are phased out
in the future (we are talking many many years down the line here, but it
could happen one day).

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2355
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Date: Mon, 27 Jan 2003 11:14:44 GMT
Lines: 53
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>>However, on reading the syntax in RFC 2231, I now find a few further
>>oddities.
>>
>>1. Broken backwards compatibility
>>
>>Under RFC 2046, I could write
>>
>>	x-foo'bar = baz
>>
>>But that is not allowed now.
>>
>>OTOH, I can still write
>>
>>	charset = x-foo'bar
>>
>>but not
>>
>>	charset* = ''x-foo'bar
>>
>Of course you can't; the asterisk means encoding is used, so that has to be:
>     charset* = ''x-foo%27bar

Yes, that is exactly what I said. My whole point in showing these examples
was to check that my understanding of RFC 2231 was correct, and to point
out that, although it make these small incompatible changes, it does not
admit to them in the text. You have to examine the syntax very carefully
in order to spot them.

>That is clearly specified in section 4.1 note (5).

>>But I thought the IETF "didn't do" backwards incompatibility (Cue howls of
>>protest from Bruce Lilley) :-( .
>>  
>>
>If you intend to use my name, please at least extend me the common courtesy
>of not misspelling it.  Perhaps, you meant somebody else....

My apologies for that. I too have a name which is commonly misspelt.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2354
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Date: Mon, 27 Jan 2003 10:59:16 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> [email-masked] (Claus Färber) writes:

>It seems you simply don't understand the concept of backwards  
>compatibility.
>RFC 2231 uses a subset of the format of RFC 2046, which means it *is*  
>compatible. It does not change the format for existing parameters, it  
>just defines new parameters with a ``special'' format.

No, it outlaws things that were legal beforehand. For example the
    x-foo'bar = baz
that I mentioned. That is a backwards incompatibility, as Ned acknowledged
in his reply, but which was a perfectly reasonable change in this case for
the reasons he gave.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2353
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Date: Mon, 27 Jan 2003 10:54:28 GMT
Lines: 73
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> >On Thursday 23 January 2003 11:58, Charles Lindsey wrote:

>> But I thought the IETF "didn't do" backwards incompatibility (Cue howls of
>> protest from Bruce Lilley) :-( .

>You're engaging in sophistry again. First of all, the one exception the IETF
>makes to the backwards compatibility rule is that of additional restrictions
>and feature elimination: It is common for capabilities to be removed, syntactic
>flexibility to be eliminated, unused options deprecated, and even for features
>to be excised. (This last is typically only done in the case of a burgeoning
>security problem of some sort.) So, to the extent RFC 2231 made some previous
>legal syntax illegal, this is a very different thing from making previously
>illegal syntax legal. Indeed, the entire RFC 2821/2822 exercise would not have
>been possible had it been necessary to preserve backwards compatibility with
>every prior jot and tittle.

Sure, I agree that is all quite reasonable. I was just pointing out that
the IETF postion on compatibilty was not as rigid as some here were
seeming to imply.

>Furthermore, in this particular case, the authors checked and made sure no
>existing registered media type parameters were affected by the change. Indeed,
>if memory serves, the original syntax we picked (I no longer recall what it
>was) fell foul of some obsure existing usage, and we abandoned it for that
>reason.

And, again, if some breech of compatibility is proposed, looking at real
observed existing usage (as opposed to a theoretical possibility which
some obscure site might _just_ be using, but noone has ever seen, like my
"x-foo'bar = baz" example) is the proper procedure.

>RFC 2978 section 2.3 is where charset name syntax is specified. Currently
>single quotes are allowed. And AFAIK no existing charset name uses a single
>quote. So there is no existing usage of a charset name which when coupled
>with the nonexistant charset* parameter would cause things to break.

Thanks for the pointer to RFC 2978. I see that '%' is also allowed, but
not '*'. So writing a foolproof parser for extended-initial-value would be
tricky (you have to scan from R to L until you have identified the two
rightmost "'"s), but I doubt implementors will bother to go so far.

>Now, it would be possible to remove this from the allowed syntax (which is
>already more restrictive than that of media type parameters, and has in  fact
>been made increasingly restrictive over time). But doing so really isn't
>necessary, so why bother?

Sounds like a sensible thing to fix next time that document comes up for
revision.

>> 3. IMAP

>> I see that RFC 2231 makes a minor change to IMAP ("SHOULD decode parameter
>> value continuations"). To be very pedantic, shouldn't RFC 2231 have been
>> described as "Updates RFC 2060"?

>Sure, whatever.

Yes, I was merely checking that, where a protocol changed, it was in order
to make consequential changes to IMAP, or other documents affected.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Sat, 25 Jan 2003 10:55:54 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Charles Lindsey wrote:

>However, on reading the syntax in RFC 2231, I now find a few further
>oddities.
>
>1. Broken backwards compatibility
>
>Under RFC 2046, I could write
>
>	x-foo'bar = baz
>
>But that is not allowed now.
>
>OTOH, I can still write
>
>	charset = x-foo'bar
>
>but not
>
>	charset* = ''x-foo'bar
>
Of course you can't; the asterisk means encoding is used, so that has to be:
     charset* = ''x-foo%27bar

That is clearly specified in section 4.1 note (5).

>But I thought the IETF "didn't do" backwards incompatibility (Cue howls of
>protest from Bruce Lilley) :-( .
>  
>
If you intend to use my name, please at least extend me the common courtesy
of not misspelling it.  Perhaps, you meant somebody else....

You might also save yourself some public humiliation by reading and thinking
about the topic; clearly an *extended* parameter (that is what the asterisk
indicates) is an extension.  Charles, based on this and a similar 
postting to the
usefor list, you clearly do not undersand backward compatibility -- the 
first
part (the line after OTOH) is precisely an example of working backwards
compatibility.

Regarding the change to the MIME definition of a regular parameter attribute
name, please note:
1.  2231 is a MIME extension RFC which updates 2045's (another MIME RFC)
    definition.  I.e. the auhors knew what they were doing -- quite a 
different
    situation from an attempt to produce an incompatible redefinition 
 of something
    defined in an RFC managed by another area.
2.  2231 was the result of discussions with MIME users; the 
compatibility issues
    were well understood.
3.  attribute-char is shared by the parameter attribute and the extended 
value. At
    first glance it might appear that there's no reason to exclude ' 
and/or % from
    an attribute name, but bear in mind the fact that an attribute name 
may appear
    in a different parameter's value.

>2. IANA registered charsets
>
>Under RFC 2046, it was allowed to register "foo'bar" as a charset (it was
>just required to be a token). Presumably, this is not allowed now, but
>where is it actually written down what may be registered as a charset?
>AFAICS, that IANA registry was created by RFC 2046.
>  
>
See RFC 2278.  A single quote is still valid in a charset name, and it 
is possible
to uambiguously parse an extended parameter containing a charset name 
with one
or more single quotes.  So it's not an issue (except, of course, insofar 
as implementors
need to take care to parse correctly).

Date: 24 Jan 2003 21:50:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Charles Lindsey <[email-masked]> schrieb/wrote:
> But I thought the IETF "didn't do" backwards incompatibility (Cue howls of
> protest from Bruce Lilley) :-( .

It seems you simply don't understand the concept of backwards  
compatibility.
RFC 2231 uses a subset of the format of RFC 2046, which means it *is*  
compatible. It does not change the format for existing parameters, it  
just defines new parameters with a ``special'' format.

> And it might have been simpler just to add "'", "*" and "%" to tspecials,
> for the removal of all confusion.

That would not be backwards compatible; it would change the format of  
(unencoded) parameters.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: Fri, 24 Jan 2003 09:48:00 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2231 and CFWS
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> Marc Mutz <[email-masked]> writes:

> >On Thursday 23 January 2003 11:58, Charles Lindsey wrote:
> >>
> >>     ()token*0()=3D()token()
> >>        ditto   =3D()"quoted-string"()
> >>        ditto   =3D()ascii'en_US'foo%20bar()

> >All extended-*-{name,value}s? productions are designed to be recognized
> >as "token" by pre-2231 parsers. Following that thought through will
> >give you quite naturally the idea of where CFWS can occur.

> OK, thanks to all concerned. I can now see how to write a syntax with CFWS
> in the right places.

> However, on reading the syntax in RFC 2231, I now find a few further
> oddities.

> 1. Broken backwards compatibility

> Under RFC 2046, I could write

> 	x-foo'bar = baz

> But that is not allowed now.

> OTOH, I can still write

> 	charset = x-foo'bar

> but not

> 	charset* = ''x-foo'bar

> But I thought the IETF "didn't do" backwards incompatibility (Cue howls of
> protest from Bruce Lilley) :-( .

You're engaging in sophistry again. First of all, the one exception the IETF
makes to the backwards compatibility rule is that of additional restrictions
and feature elimination: It is common for capabilities to be removed, syntactic
flexibility to be eliminated, unused options deprecated, and even for features
to be excised. (This last is typically only done in the case of a burgeoning
security problem of some sort.) So, to the extent RFC 2231 made some previous
legal syntax illegal, this is a very different thing from making previously
illegal syntax legal. Indeed, the entire RFC 2821/2822 exercise would not have
been possible had it been necessary to preserve backwards compatibility with
every prior jot and tittle.

Furthermore, in this particular case, the authors checked and made sure no
existing registered media type parameters were affected by the change. Indeed,
if memory serves, the original syntax we picked (I no longer recall what it
was) fell foul of some obsure existing usage, and we abandoned it for that
reason.

> Under RFC 2046, it was allowed to register "foo'bar" as a charset (it was
> just required to be a token). Presumably, this is not allowed now, but
> where is it actually written down what may be registered as a charset?
> AFAICS, that IANA registry was created by RFC 2046.

First of all, the restricitions on charset names are such that it is never
necessary to use RFC 2231 facilities to write them. Nor is charset* a
registered media type parameter. So this is an entirely academic exercise.

RFC 2978 section 2.3 is where charset name syntax is specified. Currently
single quotes are allowed. And AFAIK no existing charset name uses a single
quote. So there is no existing usage of a charset name which when coupled
with the nonexistant charset* parameter would cause things to break.

Now, it would be possible to remove this from the allowed syntax (which is
already more restrictive than that of media type parameters, and has in  fact
been made increasingly restrictive over time). But doing so really isn't
necessary, so why bother?

> 3. IMAP

> I see that RFC 2231 makes a minor change to IMAP ("SHOULD decode parameter
> value continuations"). To be very pedantic, shouldn't RFC 2231 have been
> described as "Updates RFC 2060"?

Sure, whatever.

				Ned

To: [email-masked]
Xref: clerew local.mime:2349
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Date: Fri, 24 Jan 2003 13:52:30 GMT
Lines: 64
Sender: [email-masked]

In <[email-masked]> Marc Mutz <[email-masked]> writes:

>On Thursday 23 January 2003 11:58, Charles Lindsey wrote:
>>
>>     ()token*0()=3D()token()
>>        ditto   =3D()"quoted-string"()
>>        ditto   =3D()ascii'en_US'foo%20bar()

>All extended-*-{name,value}s? productions are designed to be recognized
>as "token" by pre-2231 parsers. Following that thought through will
>give you quite naturally the idea of where CFWS can occur.

OK, thanks to all concerned. I can now see how to write a syntax with CFWS
in the right places.

However, on reading the syntax in RFC 2231, I now find a few further
oddities.

1. Broken backwards compatibility

Under RFC 2046, I could write

	x-foo'bar = baz

But that is not allowed now.

OTOH, I can still write

	charset = x-foo'bar

but not

	charset* = ''x-foo'bar

But I thought the IETF "didn't do" backwards incompatibility (Cue howls of
protest from Bruce Lilley) :-( .

And it might have been simpler just to add "'", "*" and "%" to tspecials,
for the removal of all confusion.

2. IANA registered charsets

Under RFC 2046, it was allowed to register "foo'bar" as a charset (it was
just required to be a token). Presumably, this is not allowed now, but
where is it actually written down what may be registered as a charset?
AFAICS, that IANA registry was created by RFC 2046.

3. IMAP

I see that RFC 2231 makes a minor change to IMAP ("SHOULD decode parameter
value continuations"). To be very pedantic, shouldn't RFC 2231 have been
described as "Updates RFC 2060"?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 23 Jan 2003 09:47:14 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2231 and CFWS
To: Keith Moore <[email-masked]>
Cc: Marc Mutz <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> > All extended-*-{name,value}s? productions are designed to be recognized
> > as "token" by pre-2231 parsers. Following that thought through will
> > give you quite naturally the idea of where CFWS can occur.

> furthermore the way I'd recommend implementing this is not by teaching the
> parser to recognize 2231 parameters differently than ordinary MIME parameters,
> but by parsing everything as MIME parameters.  then the implementation can
> look at the parameter names to see if there are any 2231 parameters there.

That's certainly how I implemented it.

				Ned

Date: Thu, 23 Jan 2003 12:17:53 -0500
From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

> All extended-*-{name,value}s? productions are designed to be recognized 
> as "token" by pre-2231 parsers. Following that thought through will 
> give you quite naturally the idea of where CFWS can occur.

furthermore the way I'd recommend implementing this is not by teaching the
parser to recognize 2231 parameters differently than ordinary MIME parameters,
but by parsing everything as MIME parameters.  then the implementation can
look at the parameter names to see if there are any 2231 parameters there.

Date: Thu, 23 Jan 2003 11:40:54 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Charles Lindsey wrote:

>So you are saying that the most that is allowed is
>
>    ()token*0()=()token()
>       ditto   =()"quoted-string"()
>       ditto   =()ascii'en_US'foo%20bar()
>  
>
The first two may appear as normal parameters (i.e. w/o *0) or
as any continuation section of an extended parameter.  The last
is incorrect; it requires an asterisk before the = to indicate that
it is an extended parameter (i.e. that it uses encoding), and the
language tag is botched:
      () token*0* () = () ascii'en-us'foo%20bar ()
Charset and language may only appear in the initial section, and
at minimum the two single quote delimiters must appear in the
initial section if any continuation section uses encoding.

Date: Thu, 23 Jan 2003 08:53:37 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

> >they are part of the token that is the parameter name.  no, you may not
> >put comments in the middle of the parameter name.
> 
> So you are saying that the most that is allowed is
> 
>     ()token*0()=()token()
>        ditto   =()"quoted-string"()
>        ditto   =()ascii'en_US'foo%20bar()

yes.

> >> Also, I presume that the charset and language form can have () before
> >> and after, but nowhere in the middle, as in
> 
> >nope, you cannot have white space in the middle of the parameter.
> 
> But it can still go either side of the '=' surely?

yes.

Date: Thu, 23 Jan 2003 13:52:55 +0100
From: Marc Mutz <[email-masked]>
Subject: Re: RFC 2231 and CFWS
To: [email-masked]
Organization: KDE
Sender: [email-masked]

--Boundary-02=_nW+L+8SwUC2HX7K
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Thursday 23 January 2003 11:58, Charles Lindsey wrote:
<snip>
> >they are part of the token that is the parameter name.  no, you may
> > not put comments in the middle of the parameter name.
>
> So you are saying that the most that is allowed is
>
>     ()token*0()=3D()token()
>        ditto   =3D()"quoted-string"()
>        ditto   =3D()ascii'en_US'foo%20bar()

After each "ditto", too.

All extended-*-{name,value}s? productions are designed to be recognized=20
as "token" by pre-2231 parsers. Following that thought through will=20
give you quite naturally the idea of where CFWS can occur.

Marc

=2D-=20
'When you see the ping of death, duck and cover.'
                        -- Bruce Schneier, Crypto-Gram Oct 2002

--Boundary-02=_nW+L+8SwUC2HX7K
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_nW+L+8SwUC2HX7K--

To: [email-masked]
Xref: clerew local.mime:2343
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Date: Thu, 23 Jan 2003 10:58:31 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> So, prior to 2231, a parameter can be
>> 
>>     ()token()=()token()
>> or  ()token()=()"quoted-string"()
>> 
>> Under 2231, we can also have an asterisk, possibly followed by DIGITS,
>> between the token and the '='. So are they to be considerd as an
>> appendage to the token, or as a prependage to the "=" or, IOW, where
>> may I put () amongst them? 

>they are part of the token that is the parameter name.  no, you may not
>put comments in the middle of the parameter name.

So you are saying that the most that is allowed is

    ()token*0()=()token()
       ditto   =()"quoted-string"()
       ditto   =()ascii'en_US'foo%20bar()

>> Also, I presume that the charset and language form can have () before
>> and after, but nowhere in the middle, as in

>nope, you cannot have white space in the middle of the parameter.

But it can still go either side of the '=' surely?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 22 Jan 2003 17:26:49 -0500
From: Keith Moore <[email-masked]>
To: Charles Lindsey <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

> So, prior to 2231, a parameter can be
> 
>     ()token()=()token()
> or  ()token()=()"quoted-string"()
> 
> Under 2231, we can also have an asterisk, possibly followed by DIGITS,
> between the token and the '='. So are they to be considerd as an
> appendage to the token, or as a prependage to the "=" or, IOW, where
> may I put () amongst them? 

they are part of the token that is the parameter name.  no, you may not
put comments in the middle of the parameter name.

> Also, I presume that the charset and language form can have () before
> and after, but nowhere in the middle, as in

nope, you cannot have white space in the middle of the parameter.

good questions though.

-- 
I tried enlightenment but it kept crashing.

Date: Wed, 22 Jan 2003 21:33:27 +0000 (GMT)
From: Charles Lindsey <[email-masked]>
Reply-To: Charles Lindsey <[email-masked]>
Subject: RFC 2231 and CFWS
To: [email-masked]
Content-MD5: wgkHiob/8LPpaZZgrnD3Dg==
Sender: [email-masked]

I want to understand exactly where FWS and comments may occur under RFC
2231 parameters, so that I may describe it correctly in Usefor.

In the following, '()' represents a position where whitespace, folding
or comments may occur (i.e. CFWS in RFC 2822 terminology).

So, prior to 2231, a parameter can be

    ()token()=()token()
or  ()token()=()"quoted-string"()

Under 2231, we can also have an asterisk, possibly followed by DIGITS,
between the token and the '='. So are they to be considerd as an
appendage to the token, or as a prependage to the "=" or, IOW, where may
I put () amongst them? RFC 2231 is not at all clear on this. The "worst
case" scenario would allow:

    ()token()*()0()=()........

Is that correct, and if not what is?

Also, I presume that the charset and language form can have () before
and after, but nowhere in the middle, as in

    ....=()us-ascii'en_US'foo%20bar()

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "Shmuel (Seymour J.) Metz" <[email-masked]>
Date: Sun, 19 Jan 2003 13:34:25 -0500
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Organization: Atid/2
Sender: [email-masked]

In <[email-masked]>, on 01/14/2003
   at 10:16 PM, Keith Moore <[email-masked]> said:

>Well, we need the critics too.

Only if they are rational. It is the responsibility of each of us to
decide independently which criticism has merit.
 
-- 
-----------------------------------------------------------
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2
     Team OS/2
     Team PL/I
-----------------------------------------------------------

To: [email-masked]
Xref: clerew local.mime:2340
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Date: Mon, 20 Jan 2003 11:43:26 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>For better or worse, usenet decided long ago to have a more restrictive
>syntax than that of email.  Gateways have been dealing with this for a
>long time.  It's occasionally annoying because a message that is gatewayed
>from email to news and back suffers information loss from the gateway.
>But that decision was made long ago.  Even if you relax the format of news
>articles existing news readers will still not know how to parse them.  So
>for practical purposes you need to continue adhering to those restrictions
>for a while.

Indeed so. In Usefor, most of those things are MUST accept, SHOULD NOT
generate yet.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
Date: Sat, 18 Jan 2003 18:25:19 +0100
Sender: [email-masked]

D. J. Bernstein writes:
> A byte-by-byte regexp matcher that doesn't know anything about UTF-8, 
> such as an ancient version of the UNIX grep program, nevertheless 
> does a perfect job of matching a UTF-8 regexp against a UTF-8 string.

Oh? Here's a grep that will find all four-letter sequences whose first 
two characters are 'ar' and last is 't'

    grep -i ar.t

...except those encoded in UTF-8 in which the third letter isn't also a 
US-ASCII letter.

--Arnt

Date: 18 Jan 2003 13:17:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

D. J. Bernstein <[email-masked]> schrieb/wrote:
> As for untagged encodings: There can be only one winner in the end,
> and it's obviously going to be UTF-8. Even if the Chinese government
> demands untagged GB18030,...

As GB18030 is just another transformation format for Unicode, the very
fact that they do demand support of GB18030 will also push other Unicode
transformation formats. For example, I'm quite certain that Microsoft
Windows will continue to use UTF-16 internally (actually, they switched
to UTF-16 from UCS-2 because of the Chinese requirement to support
GB18030). And message user agents (as well as any other software) that
run in a GB18030 environment (on Unixoid systems) can easily translate
between UTF-8 and GB18030.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: 18 Jan 2003 06:50:00 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
Sender: [email-masked]

Costello asks how to say ``exactly one UTF-8 character'' in a regexp.
http://cr.yp.to/proto/utf-8-re.txt has the answer. Of course, all of
this has been discussed many times before.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sat, 18 Jan 2003 05:56:48 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

"D. J. Bernstein" <[email-masked]> wrote:

> A byte-by-byte regexp matcher that doesn't know anything about UTF-8,
> such as an ancient version of the UNIX grep program, nevertheless does
> a perfect job of matching a UTF-8 regexp against a UTF-8 string.

I think it will never match something that shouldn't match, which is
indeed a pretty cool feature of UTF-8, but it will sometimes fail to
match something that should match.  For example, the regexp foo.bar will
fail to match foo±bar (because the ± character is two bytes).

AMC

Date: Fri, 17 Jan 2003 06:10:21 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: What's in a "name"?
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> So, for purposes of a document intended to be a standards-track
> RFC, it seems quite clear that a newsgroup "name" is a name in
> the RFC 1958 and RFC 2277 sense.  It is therefore not subject to
> internationalization,

Couldn't you say the same thing about domain names?  They are protocol
elements without a doubt (in email addresses and URLs, for example).
Therefore they are not subject to internationalization, and the IDN
effort should never have begun in the first place?  That argument can
be made, but given that IDNs have been approved, that sets a precedent
for a reinterpretation or reconsideration of whatever principle you are
invoking.

The difficulty here is that some identifiers are intended, from the
start, to be used by both machines and humans.  Domain names and
newsgroup names both fall into that category.

Jean-Marc Desperrier <[email-masked]> wrote:

> The ASCII in "case-independant ASCII" is an extremly important part.
>
> When the name is not ASCII, the case-independance rules are ultimately
> dependant on the locale used.

They can be.  Unicode defines two case-folding operations, a default
one, and one for use with Turkish and Azeri.  The only difference is the
dot on the letter i.

> Case-independance for non-ASCII strings is a bad idea.
> It a whole can of worms there.

It looks like a pretty small can to me.  One letter in two languages.

For case mapping (as opposed to case folding), the SpecialCasing.txt
describes all the non-trivial mappings.  Again the only locale-sensitive
mappings deal with the dot on the letter i, this time in three languages
(Lithuanian, Turkish, and Azeri).

People point to Turkish i as if it's just one example from a large
class, but as far as I can tell, it's the only one.

> Do you know of What's the IETF policy here ? I haven't seen it
> discussed yet.

Unicode specifies a caseless matching operation, and IDNs use it.  See
UAX#21:

http://www.unicode.org/reports/tr21/

AMC

To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
From: Andrew Gierth <[email-masked]>
Date: 18 Jan 2003 03:20:37 +0000
Lines: 29
Sender: [email-masked]

>>>>> "Claus" == Claus Färber <[email-masked]> writes:

 > Russ Allbery <[email-masked]> schrieb/wrote:
 >> comes to Usenet feed patterns.  Right now, it's common to attempt
 >> to exclude certain types of binaries with patterns like @*mp3* and
 >> @*dvd*; so long as all newsgroup names are ASCII words, this works
 >> fairly well,

 Claus> Building filters that way will also catch some innocent
 Claus> discussion group, e.g. a discussion group about CD-ROM and DVD
 Claus> drives.

 Claus> If you filter alt.binaries.*mp3, you already avoid problems
 Claus> with IDNA names.

Plenty of big MP3 binary groups would get through, too.

But that's not really the point - sites choose to exclude *mp3* from
feeds knowing (one hopes) that it will hit discussion groups about mp3
as well as the binaries themselves. It's quite something else for that
to result in them excluding completely unrelated groups just because
of encoded name artifacts.

(It's not just !*mp3*, either. Other common exclusions include !*warez*,
!*teen*, !*pedo*, !*bin* (or *bina*, *binar*, etc.), !*sex* (yes, this
also caught uk.local.essex), and so on.)

-- 
Andrew.

To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 17 Jan 2003 19:16:34 -0800
Lines: 21
Sender: [email-masked]

Claus Färber <[email-masked]> writes:
> Russ Allbery <[email-masked]> schrieb/wrote:

>> comes to Usenet feed patterns.  Right now, it's common to attempt to
>> exclude certain types of binaries with patterns like @*mp3* and @*dvd*; so
>> long as all newsgroup names are ASCII words, this works fairly well,

> Building filters that way will also catch some innocent discussion  
> group, e.g. a discussion group about CD-ROM and DVD drives.

> If you filter alt.binaries.*mp3, you already avoid problems with IDNA  
> names.

And don't get all the MP3 groups.

I understand the problem, but unfortunately until we have a working binary
classification system that can bear the load, it's the only thing that
works without going to quite a bit more work.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 18 Jan 2003 03:25:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Keith Moore <[email-masked]> schrieb/wrote:
> For better or worse, usenet decided long ago to have a more
> restrictive syntax than that of email.  Gateways have been dealing
> with this for a long time.  It's occasionally annoying because a
> message that is gatewayed from email to news and back suffers
> information loss from the gateway.

My guess is that most gateways just don't care. I.e. the gateway will
try to just send the mail as-is to a newsgroup ... and if that fails, it
just fails.
Because most mail user agents actually use a very restrictive syntax  
when generating mail, the mails they generate are compatible with news.  
So these gates will work most of the time.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: 18 Jan 2003 03:30:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
Organization: The Eyrie
Sender: [email-masked]

Russ Allbery <[email-masked]> schrieb/wrote:
> comes to Usenet feed patterns.  Right now, it's common to attempt to
> exclude certain types of binaries with patterns like @*mp3* and @*dvd*; so
> long as all newsgroup names are ASCII words, this works fairly well,

Building filters that way will also catch some innocent discussion  
group, e.g. a discussion group about CD-ROM and DVD drives.

If you filter alt.binaries.*mp3, you already avoid problems with IDNA  
names.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
From: Andrew Gierth <[email-masked]>
Date: 18 Jan 2003 02:04:22 +0000
Lines: 38
Sender: [email-masked]

>>>>> "Keith" == Keith Moore <[email-masked]> writes:

 >> Which is an argument in favor of using UTF-8 newsgroup names on
 >> the wire between news servers, since then a UTF-8-aware wildmat
 >> will work as one expects.  If newsgroup names are decoded into
 >> UTF-8 before matching, wildmat matches will always work as
 >> expected.

 Keith> seems like the code needs to be changed either way.

most of the server (relay and serving agents in USEFOR-speek) does not
need to be changed in any way whatsoever to support raw UTF-8. Note
that many servers are _only_ relay agents and that these form a
fundamental part of the Usenet infrastructure.

The sole server change required is for injecting agents that wish to
support posting to non-ascii moderated groups, and all they require is
a bolt-on addition to the mail-to-moderator operation which is
normally handled in an external program anyway and therefore can
almost certainly be dealt with without having to actually touch the
server code itself.

 Keith> existing expression matchers seem unlikely to do useful things
 Keith> with utf-8 regardless of whether or not the utf-8 is encoded
 Keith> as ascii.  for instance, will the * character match a sequence
 Keith> of octets or a sequence of utf-8 characters?

for raw UTF-8 it doesn't matter; a valid UTF-8 wildmat using '*'
metacharacters (only) will match precisely the same set of UTF-8
strings regardless of whether the matching algorithm understands UTF-8
or not. (The behaviour can only differ if either the wildmat or the
string being matched against is not itself a valid UTF-8 string.)

On the other hand, if the UTF-8 is encoded into some ASCII
representation, then this is no longer guaranteed.

-- 
Andrew.

Date: 18 Jan 2003 02:04:00 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
Sender: [email-masked]

Keith Moore writes:
> existing expression matchers seem unlikely to do useful things with utf-8

You couldn't possibly be more wrong.

A byte-by-byte regexp matcher that doesn't know anything about UTF-8,
such as an ancient version of the UNIX grep program, nevertheless does
a perfect job of matching a UTF-8 regexp against a UTF-8 string.

The relevant features of UTF-8 are that (1) it's compatible with ASCII,
so characters such as * are the same in ASCII and UTF-8; and (2) it's
self-synchronizing, so a UTF-8 character cannot match a UTF-8 string
except at a character boundary.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 17 Jan 2003 17:42:54 -0800
Lines: 21
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> seems like the code needs to be changed either way.  existing expression
> matchers seem unlikely to do useful things with utf-8 regardless of
> whether or not the utf-8 is encoded as ascii.  for instance, will the *
> character match a sequence of octets or a sequence of utf-8 characters?

It turns out that it's not actually that bad.  Yes, * will match a series
of octets rather than a sequence of characters, but because of the
(excellent) structure of UTF-8, it's rather difficult to construct
realistic cases where that doesn't end up amounting to the same thing.
The main damage is to the ? single-character match and character classes,
but those are rather rarely used features in practice, to a degree that
character classes likely won't be standardized in the NNTP revision.

In practice, nearly all of the wildmats one sees are either literal group
names or patterns using only *.  I expect that servers could go quite some
distance with UTF-8 groups without changing their wildmat engine at all.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 17 Jan 2003 19:20:11 -0500
From: Keith Moore <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: rather than argue and bicker about who said what...
Sender: [email-masked]

> > That is actually a very good point. There is an expectation, built into
> > NNTP, that users will be able to filter newsgroup-names using "wildmats"
> > (which are a restricted form of regular expression). The draft of the
> > new NNTP standard (draft-ietf-nntpext-base-15.txt) goes to considerable
> > trouble to define wildmats in such a way that they will work with UTF-8.
> 
> Which is an argument in favor of using UTF-8 newsgroup names on the wire
> between news servers, since then a UTF-8-aware wildmat will work as one
> expects.  If newsgroup names are decoded into UTF-8 before matching,
> wildmat matches will always work as expected.

seems like the code needs to be changed either way.  existing expression
matchers seem unlikely to do useful things with utf-8 regardless of whether or
not the utf-8 is encoded as ascii.   for instance, will the * character match
a sequence of octets or a sequence of utf-8 characters?

Date: Fri, 17 Jan 2003 19:03:12 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Charles Lindsey wrote:

>One of the sad things about the email standards is thet they permit, and
>even encourage, changing the order of headers.
>
Can you cite a current email standard which "encourage[s] changing the order
of header [field]s"?

To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 17 Jan 2003 15:39:34 -0800
Lines: 20
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> That is actually a very good point. There is an expectation, built into
> NNTP, that users will be able to filter newsgroup-names using "wildmats"
> (which are a restricted form of regular expression). The draft of the
> new NNTP standard (draft-ietf-nntpext-base-15.txt) goes to considerable
> trouble to define wildmats in such a way that they will work with UTF-8.

Which is an argument in favor of using UTF-8 newsgroup names on the wire
between news servers, since then a UTF-8-aware wildmat will work as one
expects.  If newsgroup names are decoded into UTF-8 before matching,
wildmat matches will always work as expected.

I can see why Andrew proposed what he did.  Re-encoding at the news/mail
boundary for only those three headers (Newsgroups, Followup-To, and
Control) may be easier than trying to push IDNA all the way down into the
server and dealing with things like wildmat patterns.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 17 Jan 2003 17:07:11 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

> All those optional comments and the like will actually break much of
> the installed base of Usenet (comments in a References-header would be
> exceedingly troublesome, for example). Shouldn't that be a worry to
> you?

For better or worse, usenet decided long ago to have a more restrictive syntax than that of email.  Gateways have been dealing with this for a long time.  It's occasionally annoying because a message that is gatewayed from email to news and back suffers information loss from the gateway.  But that decision was made long ago.  Even if you relax the format of news articles existing news readers will still not know how to parse them.  So for practical purposes you need to continue adhering to those restrictions for a while.

Keith
-- 
I tried enlightenment but it kept crashing.

To: [email-masked]
Xref: clerew local.mime:2321
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: rather than argue and bicker about who said what...
Date: Fri, 17 Jan 2003 12:42:00 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Keith Moore <[email-masked]> writes:

>> it would be interesting to know what the problems are, especially since
>> IDNA or something close to it seems likely to be used in email
>> addresses.

>I'm not at all sure that I agree with Charles's blithe assertions about
>the opinion of USEFOR, but the one objection that was raised to IDNA is
>that it has some potentially somewhat unfortunate side effects when it
>comes to Usenet feed patterns.  Right now, it's common to attempt to
>exclude certain types of binaries with patterns like @*mp3* and @*dvd*; so
>long as all newsgroup names are ASCII words, this works fairly well, but
>once you start using an encoding like IDNA, the chances of odd false
>positives and negatives is quite high.

That is actually a very good point. There is an expectation, built into
NNTP, that users will be able to filter newsgroup-names using "wildmats"
(which are a restricted form of regular expression). The draft of the new
NNTP standard (draft-ietf-nntpext-base-15.txt) goes to considerable
trouble to define wildmats in such a way that they will work with UTF-8.

But for sure they would not work with anything encoded using punycode.

Indeed, that may turn out to be a problem with anyone who wants to filter
out domains of the form "*.æøå.com" from his incoming emails.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2319
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Date: Fri, 17 Jan 2003 12:29:37 GMT
Lines: 41
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>On 16 Jan 2003 16:14:00 +0200
>[email-masked] (Claus Färber) wrote:

>> 
>> Keith Moore <[email-masked]> schrieb/wrote:
>> >  I do believe that migrating to utf-8 in message headers is the way
>> >  to go in the long term, and that a transition strategy similar to what
>> >  Dan is suggesting (support utf-8 on receipt soon, generation of utf-8
>> >  later) is ultimately the way to get there from here.
>> 
>> If we do that, we should also simplify the horrible general format of
>> RFC 2822 messages: No optional comments (or other optional syntax
>> elements), simpler folding rules, etc.

All those optional comments and the like will actually break much of the
installed base of Usenet (comments in a References-header would be
exceedingly troublesome, for example). Shouldn't that be a worry to you?

But Usenet (or Usefor) is not complaining. We have dutifully incorporated
the bulk of that stuff into Usefor, and we expect the installed base to
move to accept it. (We still have problems with a few minor features of
RFC 2822, which we will likely raise if/when it comes up for revision).

>if DRUMS didn't see fit to do those things, I don't know why we should
>revisit the issue now.  those discussions take a *LOT* of time and don't
>accomplish very much. 

I agree (those minor quibbles apart).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2322
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: distinguishing between utf-8 and gb18030
Date: Fri, 17 Jan 2003 12:45:18 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>question:

>in an environment where either utf-8 or gb18030 may appear, how reliably
>can gb18030 and utf-8 strings be identified and distinguished from one another?
>offhand it appears that many gb18030 strings are also valid utf-8 strings.

Andrew Gierth did some experiments on this, using a whole week's worth of
news on supernews. It was found to be far more reliable that any of us had
imagined. I forget the exact false positive rate, but it was exceedingly
low. It did not distinguish the various gb*s from each other or from other
non-UTF-8, of course.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2320
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: distinguishing between utf-8 and gb18030
Date: Fri, 17 Jan 2003 12:34:01 GMT
Lines: 25
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> As for untagged encodings: There can be only one winner in the end

>I do not share your faith that both encodings won't be widely used, or that
>they will converge to utf-8 within our lifetimes.

>but if both encodings do appear without tagging, and the difference can
>reliably be determined by heuristics, I do have faith that vendors will
>implement those heuristics.

It has been reported that Mozilla already implements a suitable heuristic,
as least so far as being able to spot and act upon UTF-8 (I don't think it
claims to be able to distinguish the Chinese stuff from other non-UTF-8).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2318
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Date: Fri, 17 Jan 2003 12:18:46 GMT
Lines: 71
Sender: [email-masked]

In <[email-masked]> "Bart Schaefer" <[email-masked]> writes:

>Advocates of UTF-8 in usenet headers want to take the same chance that
>1341 senders took -- that their content might not get through undamaged
>everywhere, particularly when gatewayed -- rather than adopt a workaround
>of the kind 1342/1522/2047 represents.

>This doesn't seem unreasonable on the face of it, but the difference is
>that RFC 1341 had no effect on the mechanics of message transmission;
>it introduced no changes in any existing headers, and no syntactically
>incompatible changes in the extended headers.  The content might become
>broken, but the process could not.

>Changing existing headers to UTF-8 content *can* break the process; adding
>new headers in UTF-8 *is not* syntactically compatible; and it *is not*
>possible to predict where that breakage will occur or how the unexpected
>syntax will be handled by software that conforms to older standards. Can
>anyone refute any of those statements without resorting to probabilistic
>arguments?  Using anything but 7-bit in headers is a calculated risk.

Actually, we have a means to predict the breakages rather well. There is
already a considerable amount of ("illegal") 8-bit headers being sent
through email. So we just need to see what is currently happening to it.

1. I doubt than any current sofware actually dumps core when that stuff
arrives.
2. Some transports may reject is with an error response. That seems fine.
3. Many transports just pass it through as-is (it is then someone else's
problem). I believe this is the most common case.
4. Some transports drop some offending characters (sendmail is the best
known example).
5. Some transports may truncate to 7 bits (I don't know of any examples).

Are there any other possibilities? Do we have a feel for their relative
preponderance?

Gateways have essentially the same possibilities.

For user agents, we can again observe what they actually do:

1. I doubt than any current UA actually dumps core when that stuff
arrives.
2. Some UAs may report an error, but display as best they are able.
3. Some UAs may report an error, and refuse to display. Do we know of any
examples?
4. Some UAs may discard the message silently. That would be bad, but do we
know of any?
5. Some UAs may display the human-readable texts (phrases, comments, etc)
as gobbledegook, but process other stuff (email addresses, etc) as
intended. I believe this is the most common situation.
6. Some UAs may display those headers as gobbledegook, and also mess up
stuff such as email addresses. That would be bad. Do we know of examples?
7. Some UAs may, by careful configuration or just by good luck, manage to
display (some of) it correctly.

Are there any other possibilities? Do we have a feel for their relative
preponderance?

As I said, there is enough of this stuff around that the above questions
should be answerable.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "David Barr" <[email-masked]>
To: <[email-masked]>, "'Usefor Mailing List'" <[email-masked]>
Subject: RE: For shame (Was: Re: RFC 2047 and gatewaying)
Date: Fri, 17 Jan 2003 10:18:29 -0600
Importance: Normal
Sender: [email-masked]

> We have discussed getting a base document out and then 
> following it up with experimental rfcs. That seems to have 
> been ignored multiple times. It seems some here think they 
> can do it all in one document and they want to get it "right" 
> the first time before we will get a chance to 
> issue anything.  That's a mistake IMHO.  

Um, not exactly.  It's been made very clear to everyone numerous times that
the goal is to get the base doc out, and have experimental rfc's later.
It's been ignored, but it's also been beaten again into people's heads again
regularly when it has.  I think we've done a pretty good job of keeping out
most of the creep, but regardless nobody has been working under the delusion
that we have only one shot at getting everyone's pet projects into one doc
(except the few who quite recently feel we need to start over from scratch).

Perhaps it was a mistake to try the standards document first.  I don't think
so.  It is the most important (though hardest), so naturally it should
first.  Plus, all the other extensions tend to have dependencies on the
first standard.  One thing that could have been done differently was to fork
off an usenet-ext group at the very very beginning with a very narrow
charter and focus on several sequential unrelated extensions.  However there
was at least one aborted attempt to do that but everyone was so preoccupied
with the main group that it never went anywhere.
 
--Dave

From: "Shmuel (Seymour J.) Metz" <[email-masked]>
Date: Fri, 17 Jan 2003 10:09:33 -0500
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: For shame (Was: Re: RFC 2047 and gatewaying)
Organization: Atid/2
Sender: [email-masked]

In <[email-masked]>, on 01/15/2003
   at 04:04 PM, Bruce Lilly <[email-masked]> said:

>That's the same charter which says

My reading of the charter is that we we to produce multiple 
documents, some of which could have been experimental in nature. 
Perhaps we need a base document that says must accept, should not 
generate until a supporting STD is passed and an experimental RFC that
specifies raw UTF-8 plus  gatewaying requirements. If we go that
route, is there any reason that the experimental RFC could not be
submitted first?
 
-- 
-----------------------------------------------------------
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2
     Team OS/2
     Team PL/I
-----------------------------------------------------------

Date: Fri, 17 Jan 2003 09:00:43 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

On Fri, 17 Jan 2003 14:56:29 +0100
Arnt Gulbrandsen <[email-masked]> wrote:

> Keith Moore writes:
> > but if both encodings do appear without tagging, and the difference 
> > can reliably be determined by heuristics, I do have faith that 
> > vendors will implement those heuristics.
> 
> My "reliability" may be your "nightmare".
> 
> Wearing my GUI programmer hat, I have faith that I can provide a good 
> default when I need to ask the user for an encoding (say in a "file 
> load" dialog). It will be right better than nine times out of ten, and 
> when it's wrong the user can try a different encoding anyway.
> 
> Someone writing e.g. an MTA may want a tad more reliability than that.

well, I hope mtas never do tagging of unknown charsets.

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: distinguishing between utf-8 and gb18030
Cc: [email-masked], "D. J. Bernstein" <[email-masked]>
Date: Fri, 17 Jan 2003 14:56:29 +0100
Sender: [email-masked]

Keith Moore writes:
> but if both encodings do appear without tagging, and the difference 
> can reliably be determined by heuristics, I do have faith that 
> vendors will implement those heuristics.

My "reliability" may be your "nightmare".

Wearing my GUI programmer hat, I have faith that I can provide a good 
default when I need to ask the user for an encoding (say in a "file 
load" dialog). It will be right better than nine times out of ten, and 
when it's wrong the user can try a different encoding anyway.

Someone writing e.g. an MTA may want a tad more reliability than that.

--Arnt

Date: Thu, 16 Jan 2003 21:36:05 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

> As for untagged encodings: There can be only one winner in the end

I do not share your faith that both encodings won't be widely used, or that
they will converge to utf-8 within our lifetimes.

but if both encodings do appear without tagging, and the difference can
reliably be determined by heuristics, I do have faith that vendors will
implement those heuristics.

Date: 17 Jan 2003 01:57:55 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

Keith Moore writes:
> in an environment where either utf-8 or gb18030 may appear

What environment are you talking about? Who, exactly, uses GB18030? Try
Google searches for charset=gb18030, charset=utf-8, charset=iso-8859-1.
If there are all these GB18030 users, why can't we find mail from them
in publicly archived mailing lists?

As for untagged encodings: There can be only one winner in the end, and
it's obviously going to be UTF-8. Even if the Chinese government demands
untagged GB18030, people are going to deal with that the same way that
people dealt with the United States government demanding OSI, Ada, etc.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Thu, 16 Jan 2003 17:40:02 -0800 (PST)
From: [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
To: Bart Schaefer <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> On Jan 15, 11:34pm, D. J. Bernstein wrote:
> }
> } Bruce Lilly writes:
> } > header field order can (and does) change, so there's no way of
> } > guaranteeing that a header-encoding header-field would always precede
> } > use of any specified encoding
> }
> } So what? Does your MUA also have trouble displaying From before Subject
> } when the From line in the message comes after the Subject line?

> As I recall the discussion at the time (circa RFC 1342), the issue was not
> the ordering, but rather that each header field needed to be interpreted
> in isolation, because assorted transports (and maybe even some MUAs) could
> not be counted upon to pass through a header field with an unknown name,
> or to pass it undamaged.  (It was some while before senders really had any
> confidence that the RFC 1341 [no, that's not a typo] headers would reach
> all recipients, unless I'm misremembering.)

There actually was a proposal, coauthored, if memory serves, by Bob Smart and
myself, that defined an approach using multiple header fields. 

The consensus at the time was that Keith's approach, which came out later, was
clearly superior, both for the reasons you gave and because it ended up being
quite a bit simpler. Again if memory serves, Bob and I agreed with that
concensus and withdraw our document from consideration.

> Of secondary interest, but another reason for not using a separate field,
> is that the encoding can represent multiple character sets and languages
> in that same single header.  (Obviously that's not so significant if a
> "universal" character set is available, but language information is still
> of interest even if it isn't absolutely essential.)

Our proposal at the time handled this to some extent, but at the cost
of considerable complexity.

> ...

> It's one thing to accept a risk to your own data, but quite another to
> standardize on something that imposes that risk on others, no matter how
> unlikely you think it is that anything "really bad" will happen, and no
> matter how desirable the outcome once all the breakage is "outgrown."
> That, in my view, is why the proposal as it stands is not acceptable, and
> is the issue that has to be addressed before an acceptable proposal is
> possible.

Very nicely put. I agree completely.

				Ned

Date: Thu, 16 Jan 2003 19:58:45 -0500
From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

 > When did you stop beating your dog?

once he stopped yelping, there didn't seem to be a point any longer.

Keith

Date: Thu, 16 Jan 2003 16:37:54 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

Keith,

Thursday, January 16, 2003, 8:20:42 AM, you wrote:

Keith> On Thu, 16 Jan 2003 07:45:55 -0800
Keith> Dave Crocker <[email-masked]> wrote:

>> Keith,
>> 
>> Although well intentioned, you are pursuing a question that is
>> fundamentally irrelevant to an IETF technical effort.

Keith> That's okay, ietf-822 is no longer an IETF working group,

then it's probably a good thing that I said "ietf technical effort"
rather than "ietf working group".  I hope you are not suggesting that
this discussion list pertains to something other than ietf technical
work.  If you are, please explain.

Keith> IETF working
Keith> groups do need to evaluate the feasibility of proposals, and part
Keith> of that is understanding the impact on the installed base.  For
Keith> better or worse, and in spite of language in the standards
Keith> prohibiting such behavior, extensive use of unencoded non-ASCII
Keith> text in message headers has been observed.

1.  And it was discussed ad nauseum 10 years ago.  Nothing we are
hearing now sounds any different.

2.  I apologize for worrying about the efficiency of the discussion.
Deprivation of service attacks, by distraction of the conversation flow,
seems to be the norm on many IETF lists.  I apologize for being
concerned that this one try to reduce the problem.

Keith> Anyway, at this stage I'm just asking a question - not
Keith> making any kind of proposal to be attacked, nor even presuming
Keith> such a proposal.

When did you stop beating your dog?

Please don't get upset.  I'm just asking a question.

Oh.  I guess some questions really do have some negative impact.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

To: [email-masked]
Subject: Re: rather than argue and bicker about who said what...
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 16 Jan 2003 15:41:39 -0800
Lines: 33
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> it would be interesting to know what the problems are, especially since
> IDNA or something close to it seems likely to be used in email
> addresses.

I'm not at all sure that I agree with Charles's blithe assertions about
the opinion of USEFOR, but the one objection that was raised to IDNA is
that it has some potentially somewhat unfortunate side effects when it
comes to Usenet feed patterns.  Right now, it's common to attempt to
exclude certain types of binaries with patterns like @*mp3* and @*dvd*; so
long as all newsgroup names are ASCII words, this works fairly well, but
once you start using an encoding like IDNA, the chances of odd false
positives and negatives is quite high.

This problem may apply to any ASCII encoding system, however.  (I believe
this is mostly an argument for using UTF-8 on the wire and only encoding
at boundaries.)

I don't recall any specific objections to using IDNA as the boundary
encoding for newsgroup names, and it seems a logical fit given the use of
newsgroup names in envelope recipients.  (Although I don't know if the
Usenet . to - mapping for relayed messages to moderators would cause
problems.)

> do we really want usefor to have to deal with 3 or 4 separate schemes
> for encoding text that appears in message headers (2047 for text, IDNA
> for email addresses, something else for newsgroup names?)

I certainly don't.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 16 Jan 2003 18:16:13 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: rather than argue and bicker about who said what...
Sender: [email-masked]

> >1. until the mail spec is revised (which is to say, not in the near
> >future) no unencoded non-ascii characters are permitted in the message
> >header.
> 
> They should not be seen in email. That is not to say that they will not be
> seen in Netnews.

agreed that this is a separate question.  the real trick is keeping the
news articles from leaking into email.

> >3. for other usenet message header protocol elements which need to
> >contain non-ascii characters (e.g. newsgroup names) usefor (or its
> >successor) is encouraged to investigate the suitability of using the idna
> >encoding.
> 
> The view of Usefor is that IDNA is not suitable.

it would be interesting to know what the problems are, especially since
IDNA or something close to it seems likely to be used in email addresses.  do
we really want usefor to have to deal with 3 or 4 separate schemes for encoding
text that appears in message headers (2047 for text, IDNA for email addresses,
something else for newsgroup names?)

Keith

Date: Thu, 16 Jan 2003 18:02:57 -0500
From: Keith Moore <[email-masked]>
To: [email-masked] (Claus =?ISO-8859-1?Q?F?=ärber)
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On 16 Jan 2003 16:14:00 +0200
[email-masked] (Claus Färber) wrote:

> 
> Keith Moore <[email-masked]> schrieb/wrote:
> >  I do believe that migrating to utf-8 in message headers is the way
> >  to go in the long term, and that a transition strategy similar to what
> >  Dan is suggesting (support utf-8 on receipt soon, generation of utf-8
> >  later) is ultimately the way to get there from here.
> 
> If we do that, we should also simplify the horrible general format of
> RFC 2822 messages: No optional comments (or other optional syntax
> elements), simpler folding rules, etc.

if DRUMS didn't see fit to do those things, I don't know why we should
revisit the issue now.  those discussions take a *LOT* of time and don't
accomplish very much. 

To: [email-masked]
Xref: clerew local.mime:2298
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Thu, 16 Jan 2003 16:18:52 GMT
Lines: 53
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> 
>> The primary impact will be on the installed base within Usenet. That is an
>> issue for the Usefor WG to consider, and it has concluded that the impact
>> is manageable.

>That installed base includes gateways and moderators.  While some in the
>Usefor WG may think that the impact of the proposed changes are
>manageable, there are a significant number of others who strongly
>disagree. And the IESG and IETF will have to be convinced that the
>installed base will not be presented with data which is illegal under the
>existing standard  -- that is what "the impact of a new standard on the
>installed base that uses the old standard" means.

Whenever a new feature is added to a protocol, it is likely that the
installed base will not be able to accept the new "illegal" constructions.
If the IETF does not occasionally allow such things to occur, no new
features will ever get added to anything.

So the correct thing is to examine what harm, if any, will arise, and to
decide whether it is tolerable. At the least, all things that currently
work should continue to work.

But, in fact, the IESG and the IETF do, occasionally, allow a change in a
protocol that does not even allow current practice to continue. Let me
give you an example:

The OpenPGP standard made a non-compatible change to the protocol for
signing documents (non-compatible with the de facto situation obtaining
previously with PGP 2.6.x and PGP 5.x). You can indeed make signatures
with PGP 2.6.x that will not verify under OpenPGP (PGP 6.x and later) and
vice versa. The standard for application/pgp-signature had to be changed,
and a new RFC issued.

But did I hear you railing against this? No!

In fact, the change in OpenPGP was a very necessary one, since the
behaviour of the old PGP 2.6.x was stupid and inconsistent - in fact, it
was a BUG. Better to fix it and accept the pain, than to perpetuate it for
ever.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2299
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: prevervation of installed base
Date: Thu, 16 Jan 2003 16:27:13 GMT
Lines: 33
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
> >
> > The question is whether that situation is acceptable in a transitional
> > phase,

>You're talking about gateways to email -- what "transitional phase" -- what
>are you assuming will change in the email protocols?

I am talking about gateways _from_ news. It is proposed (that is what we
are discussing) that Netnews should change. That requires that gateways
should, at some stage, change. Hence it follows that there will be a
transitional phase.

I am therefore assuming that the Netnews protocols will change. I am _not_
assuming that the email protocols will change.

I repeat. The question is whether the situation during the transitional
phase is acceptable, or not. The worst that seems likely to happen is that
some gobbledegook will be seen (and please bear in mind that much
gobbledegook is already seen at present).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2300
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: rather than argue and bicker about who said what...
Date: Thu, 16 Jan 2003 16:31:19 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>maybe we can agree on some basic constraints for gatewaying news into mail?

>e.g.

>1. until the mail spec is revised (which is to say, not in the near
>future) no unencoded non-ascii characters are permitted in the message
>header.

They should not be seen in email. That is not to say that they will not be
seen in Netnews.

>2. 2047-style encoded-words are only permitted for text that is intended
>for human consumption - it is not permitted to use 2047 for machine-readable
>protocol elements.

Agreed.

>3. for other usenet message header protocol elements which need to
>contain non-ascii characters (e.g. newsgroup names) usefor (or its
>successor) is encouraged to investigate the suitability of using the idna
>encoding.

The view of Usefor is that IDNA is not suitable. It has, however, proposed
another encoding for the limited cases where it will be necessary (notably
when communicating with moderators).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 16 Jan 2003 16:14:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Keith Moore <[email-masked]> schrieb/wrote:
>  I do believe that migrating to utf-8 in message headers is the way
>  to go in the long term, and that a transition strategy similar to what
>  Dan is suggesting (support utf-8 on receipt soon, generation of utf-8
>  later) is ultimately the way to get there from here.

If we do that, we should also simplify the horrible general format of
RFC 2822 messages: No optional comments (or other optional syntax
elements), simpler folding rules, etc.

>  But there are several things I don't believe.

>  One, that this will return us to a world where messages are ordinary
>  text and can be treated as such.  For instance, even if we allow utf-8
>  in message headers, there will still be a need for canonicalization of
>  certain fields before sending and/or before comparison of values
>  embedded in those fields,

In general, this are the header field parts that are 'addresses': 
newsgroup names, email addresses, and domain names.
For all of these, a compatible superset of the IDNA encoding could be 
used.

>  Two, that this allows use of utf-8 in addresses or domain names.
>  Those are separate problems,

Yes, this is a completly different problem (and the more serious one: if
UTF-8 text fields are mangled, it's not such a big problem as completly
misdirected messages.

Further, IDNA hast some nice properties that make it a better solution
than UTF-8: The encoding is also a good thing FOR HUMANS. This might
sound strange but it's actually quite simple:

While virtually every human being (except illiterates) has an
understanding of the basic DNS alphabet (Latin alphanumerics plus
hyphen), this is not true with the full Unicode range.

Even if the system would support Unicode and UTF-8 completly, the humans
using it don't. They see an opaque identifier made of symbols they don't
know, so they can't read a simple email address aloud over the phone,
copy it to paper, etc.

With IDNA, they just have to tell their system that they don't want to
see characters from other than certain locales. They will still see an
opaque identifier but this time it is composed of symbols they know, so
they can handle it.

Note that this is even independent from the wire format of the
addresses. It would be sensible to take a string that comes in in UTF-8
format and convert it to IDNA only for display to humans that don't know
the characters used in the string.

>  Three, that the existing ability of some user agents to display utf-8
>  in message headers is sufficient for proper processing of headers
>  containing utf-8,

Well, if the servers can handle UTF-8, I suppose that they will take
care of the canonicalisation of the addresses and the conversion to IDNA
when necessary. With SMTP, this could even mean that the address is kept
in the UTF-8 form until it must be sent to a system that does not
advertise a "UTF8ADDR" extension. For NNTP, this would mean that servers
can accept UTF-8 in the POST command and do the translation.
On the other hand, having UTF-8 on the wire between certain servers is
only an unnecessary gadget.

It's useful to implement it in everything that interacts with humans,
though: config files (could also be automatically translated to UTF-8
and non-UTF-8 local encodings by a command like viactive), command line
for message injection, etc.

Claus
--
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: 16 Jan 2003 16:14:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

David Barr <[email-masked]> schrieb/wrote:
>  However, client support for (and active use of) UTF-8 is far more
>  pervasive than any of the other proposed encoding schemes.

Not the way you need it. Untagged non-UTF-8 charsets will be around for
some time, so you need a newsreader than can switch between UTF-8 and at
least one different 8bit charset automatically. That's only implemented
by very few newsreaders (and the right thing to do even if there was no
draft suggesting it) and all of these newsreaders also support RFC 2047.

This only applies to text for human consumption (i.e. everything where
RFC 2047's encoded-words are allowed). Raw UTF-8 in newsgroup names has
already been proven to cause severe interoperability problems.

Claus
--
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: What's in a "name"?
From: Andrew Gierth <[email-masked]>
Date: 16 Jan 2003 20:38:16 +0000
Lines: 11
Sender: [email-masked]

>>>>> "Henry" == Henry Spencer <[email-masked]> writes:

 Henry> They've been case-sensitive from the start: A News, B News, C
 Henry> News, INN, and various offshoots have all been consistent
 Henry> about that.

I believe one or two servers (including, ironically, Microsoft's) did
fall foul of the long-standing error in RFC 977.

-- 
Andrew.

Date: Thu, 16 Jan 2003 14:58:19 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: What's in a "name"?
Sender: [email-masked]

On Thu, 16 Jan 2003, Russ Allbery wrote:
> ...it's really not an issue for USEFOR, as Usenet is
> case-sensitive pretty much everywhere.  Newsgroup names in particular have
> always been case-sensitive in INN, and I believe in pretty much all other
> news software.

They've been case-sensitive from the start:  A News, B News, C News, INN,
and various offshoots have all been consistent about that.

                                                          Henry Spencer
                                                       [email-masked]

To: [email-masked], USEFOR <[email-masked]>
Subject: Re: What's in a "name"?
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 16 Jan 2003 11:27:48 -0800
Lines: 20
Sender: [email-masked]

Jean-Marc Desperrier <[email-masked]> writes:

> When the name is not ASCII, the case-independance rules are ultimately
> dependant on the locale used.

> Case-independance for non-ASCII strings is a bad idea.  It a whole can
> of worms there.

> What's the IETF policy here ? I haven't seen it discussed yet.

Perhaps because it's really not an issue for USEFOR, as Usenet is
case-sensitive pretty much everywhere.  Newsgroup names in particular have
always been case-sensitive in INN, and I believe in pretty much all other
news software.

If you're wondering about things defined to be case-insensitive (primarily
domain names), see IDNA for how that's dealt with.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 16 Jan 2003 20:27:26 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Bart Schaefer a dit :

>Using anything but 7-bit in headers is a calculated risk.
>  
>
Using anything on internet is a calculated risk, it does fail sometimes.
Sending email *can* fail without warning.

But I think it's possible to work with the restriction that you set.

>It's one thing to accept a risk to your own data, but quite another to
>standardize on something that imposes that risk on others
>
The current draft allows both RFC2047 and UTF-8.

It can be worded so that people who are not willing to take this risk 
are pushed in the RFC2047 direction.
So the people who get the risk are only those who want it.

The negative aspect is the fact some people who are unprepared for it 
would receive 8 bit data they can not handle.

But they are receiving invalid 8 bit data already today.

With that wording, the draft would not increase the amount of 
transmitted 8 bit data, but rather lower it, people havig a clear 
reference about how to send 8 bit instead of the blank of the current 
RFC1036 that leads so many to send raw local encoding, and quite a few 
would prefer RFC2047 to UTF-8.

Date: Thu, 16 Jan 2003 14:19:32 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: What's in a "name"? (Was Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter))
Sender: [email-masked]

On Thu, 16 Jan 2003, Jean-Marc Desperrier wrote:
> The ASCII in "case-independant ASCII" is an extremly important part.
> When the name is not ASCII, the case-independance rules are ultimately  
> dependant on the locale used.

In particular, it is important to note that when you get beyond ASCII,
the case-independence rules change *even for ASCII letters*.

The poster-child example for this is that in Turkish, the uppercase version
of "i" is not "I", and the lowercase version of "I" is not "i".

(Turkish separates dotted and dotless I.  Uppercase "i" is "I" with a dot
above it, U+0130; lowercase "I" is "i" minus the dot, U+0131.)

                                                          Henry Spencer
                                                       [email-masked]

Date: Thu, 16 Jan 2003 20:10:38 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: Bruce Lilly <[email-masked]>
CC: Keith Moore <[email-masked]>, [email-masked], USEFOR <[email-masked]>, [email-masked]
Subject: Re: What's in a "name"? (Was Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter))
Sender: [email-masked]

Bruce Lilly a dit :

> RFC 1958 says
> that names "should be in case-independent ASCII". Case-independence
> is a compatibility issue that would need to be addressed (e.g. by
> noting that newgroups SHOULD be recognized in a case-independent
> manner, but MUST be generated in lower-case only (for backward
> compatibility), with a note that a subsequent editions of the standard
> are expected to require case-independent recognition and permit
> arbitrary case in generation). 

The ASCII in "case-independant ASCII" is an extremly important part.

When the name is not ASCII, the case-independance rules are ultimately  
dependant on the locale used.

Case-independance for non-ASCII strings is a bad idea.
It a whole can of worms there.

What's the IETF policy here ? I haven't seen it discussed yet.

For me, the good thing is to either :
- give up case-independance for things that are not restricted to ASCII
- make this a ASCII restricted case-independance, ie case-independance 
only applies to characters that are part of the ASCII repertoir.

The second choice will be surprising in some respects for unwarned people.
 
But if you try do better than this, you'll discover that there NO 
absolute solution.

In my opinion, the first solution is better, to be used where ASCII is 
the only authorised encoding, when the value belongs to a fixed list all 
of which are ASCII, or when extension to the list are explicitly 
restricted to never being anything but ASCII.
In the last case, someday someone will want to not only use ASCII, so 
it's not a good idea to be case-independant.

I do not see a problem, within this context, on the other hand with 
saying for exemple that the string "poster" will be recognized in all 
variations of casing.

Date: Thu, 16 Jan 2003 14:03:43 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>3. And what about the newly-proposed Mail-Copies-To field? Or Complaints-To?
>>   Existing gateways have no syntax information for those, so could not
>>   encode. Not to mention any header fields that might be added in future.
> 
> 
> Existing gateways do not encode at all.

Precisely; requiring them to do so breaks compatibility with the
installed base of RFC 1036-compliant gateways.

> Headers that might be added in the future are indeed a problem.

There's only a minor problem (remember, we're talking about human-readable
text strings, not protocol elements) in the real world -- that minor
problem is only significant where cut-and-paste or similar operations
are involved.  In the ivory-tower Usefor draft world, where encoding
is expected to magically take place divorced from charset and language
information necessary to the encoding process, it's an insurmountable
problem.

 > They are
> already a problem with email headers and RFC 2047 (and new email headers
> are appearing at a faster rate than is ever likely to be the case with
> Netnews).

Be careful; that's an area that I've been tracking closely. In the
recent past (four years), in reverse chronological order:
   Received-Content-MIC  RFC 3335  September 2002
   Accept-Language  RFC 3282  May 2002
   List-ID  RFC 2919  March 2001
   Content-Features  RFC 2912  September 2000
   Media-Accept-Features  RFC 2530  March 1999
RFC 2047 encoding isn't applicable to Content-Features. All of them
are primarily of interest for automated processing of protocol elements
rather than for human display.

The Usefor draft proposes to establish six new header fields not defined
in any other RFC.  The last time an RFC introduced that many at one time
was RFC 2369 in July 1998; of those, three are described as "core header
fields" and the remainder "are not as widely applicable", and all are also
primarily of interest for automated processing rather than human display.

Date: Thu, 16 Jan 2003 12:56:50 -0500
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked], USEFOR <[email-masked]>, [email-masked]
Subject: What's in a "name"? (Was Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter))
Sender: [email-masked]

Keith Moore wrote:
>>There is a clean solution:
>>"Here's how to do it properly: use RFCs 2047 / 2231."  That (a) is
>>backwards compatible, (b) is consistent with Best Current Practice
>>(RFC 2277), (c) is consistent with the Internet Architecture (RFC
>>1958), (d) is compatible with RFCs 822, 2822 and the MIME RFCs, and
>>(e) is inclusive (GB18030 users are not ostracized). Raw untagged
>>utf-8 does not posess those characteristics.
> 
> 
> it's not a complete solution, because 2047 only applies to human-readable
> text, and the 2231 extensions only apply to parameter values in MIME headers.

2231 also amends 2047 to provide language tagging.  There's also a question
about whether or not 2231 only applies to parameters; that was true until
RFC 3335 used syntax elements (via a convoluted path) which had been
[re]defined by RFC 2231, but outside of a parameter.  But that doesn't
affect the Usefor draft.

As far as usenet article format and the Usefor draft is concerned, 2047/2231
covers everything except possibly newsgroups.  And that might in fact be
a complete solution to header field internationalization for usenet articles
(keep reading).  Under the current standard (RFC 1036), the only issue w.r.t.
newsgroups and the characteristics mentioned seems to be case-independence
vs. Internet Architecture and the related backwards compatibility issue, and
that hinges on whether or not a newsgroup specification is a "name" as that
term is used in RFCs 1958 and 2277

Expanding on the last point: a "name" is a chunk of text used in a
protocol exchange, not necessarily intended to be human-readable.
Newsgroups are clearly used in the protocol exchange[*]. RFC 1958 says
that names "should be in case-independent ASCII". Case-independence
is a compatibility issue that would need to be addressed (e.g. by
noting that newgroups SHOULD be recognized in a case-independent
manner, but MUST be generated in lower-case only (for backward
compatibility), with a note that a subsequent editions of the standard
are expected to require case-independent recognition and permit
arbitrary case in generation).

A "name", per RFC 2277, SHOULD have a language tag (that is a strong
recommendation, but not an mandatory requirement).  So long as newsgroups
will eventually have to be case-independent (unless a waiver for that
is obtained), and that will necessitate some server changes over a
transitionary period, there is a way out of the language-tagging issue
for newsgroups (which has been presented in the Usefor WG), viz. to
consider the newsgroup "name" in the sense of a 1958/2277 name, i.e. as
a protocol element, associating that protocol element with some
language-tagged human readable text (e.g. stored in a server "active"
file) and that human-readable text could be presented to the user.
Indeed, with such a scheme, there is in fact no need for IDNA for
newsgroups (though the two are not incompatible) -- any header
field-compatible string of text could be used to identify a newsgroup
for protocol purposes, and any arbitrary text in any language (or in
multiple languages) could be associated with that newsgroup "name".
While idna has some interesting features, AFAIK is is not inclusive
in the sense of not ostracizing the large proportion of humans who
may wish to use GB18030.

At the root of these issues is the need to make a decision; is a newsgroup
a protocol element ("name" in the 1958/2277 sense), or is is a "text string"?
Here is the relevant 2277 text:
    Internationalization is for humans. This means that protocols are not
    subject to internationalization; text strings are. Where protocol
    elements look like text tokens, such as in many IETF application
    layer protocols, protocols MUST specify which parts are protocol and
    which are text. [WR 2.2.1.1]

    Names are a problem, because people feel strongly about them, many of
    them are mostly for local usage, and all of them tend to leak out of
    the local context at times.
Some have given the opinion that newsgroup "names" should be human-readable
as well as protocol elements, in the same sense that a phrase associated
with a mailbox angle-addr is human-readable.  I.e. neither fish nor fowl,
but both. Note that there is no provision for an element to be both protocol
*and* a text string; that would lead to the logical contradiction "not subject
to internationalization" *and* "subject to internationalization". Newsgroup
"names" are clearly used by servers and are transmitted as protocol in
Newsgroups, Followup-To, and Control header fields (as well as in related
protocols such as NNTP and IMAP).

So, for purposes of a document intended to be a standards-track RFC,
it seems quite clear that a newsgroup "name" is a name in the RFC 1958
and RFC 2277 sense.  It is therefore not subject to internationalization,
That does not mean that the Usefor WG either should not or cannot come
up with an extra-protocol means of internationalized text associated with
a newsgroup "name"; it simply means that attempts to force internationalization
onto the protocol elements are misguided (and that is what has derailed
much of what has required "urgent attention" (and still requires it)).

-----------
* if a newsgroup were just a human-readable text string, 2047 would be
applicable, case-independence would not apply, but internationalization
would be required (including language tagging).

To: [email-masked]
Xref: clerew local.mime:2297
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Thu, 16 Jan 2003 15:58:27 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>3. And what about the newly-proposed Mail-Copies-To field? Or Complaints-To?
>    Existing gateways have no syntax information for those, so could not
>    encode. Not to mention any header fields that might be added in future.

Existing gateways do not encode at all.

Naturally, if/when an encoding capability is added to them, it will
include an encoding capability for all the headers expected to arise in
Netnews. Clearly, that includes Mail-Copies-To and Complaints-To.

Headers that might be added in the future are indeed a problem. They are
already a problem with email headers and RFC 2047 (and new email headers
are appearing at a faster rate than is ever likely to be the case with
Netnews).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 16 Jan 2003 11:20:42 -0500
From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

On Thu, 16 Jan 2003 07:45:55 -0800
Dave Crocker <[email-masked]> wrote:

> Keith,
> 
> Although well intentioned, you are pursuing a question that is
> fundamentally irrelevant to an IETF technical effort.

That's okay, ietf-822 is no longer an IETF working group, hasn't been one for around ten years, and the discussion here doesn't have to strictly stay within the boundaries of IETF technical efforts.

> The IETF does not produce technical specifications that use
> heuristics. 

Tell that to the ftpext working group!

Even assuming that your statement were true, and that your statement were relevant to this discussion -- IETF working groups do need to evaluate the feasibility of proposals, and part of that is understanding the impact on the installed base.  For better or worse, and in spite of language in the standards prohibiting such behavior, extensive use of unencoded non-ASCII text in message headers has been observed.

One thing that we observed in introducing MIME is that there can be considerable expectation and/or demand from users that nonstandard practices continue to work (at least, as well as they ever "worked"), and that failure to consider that can delay adoption of new standards, even when those standards are technically superior.  So it would seem prudent that any proposed extensions to the message format to support a wider range of charcters than US-ASCII be evaluated according to whether they would adversely impact nonstandard practices that appeared to "work", even for a limited community.

Anyway, at this stage I'm just asking a question - not making any kind of proposal to be attacked, nor even presuming such a proposal.  I hope that someone here knows the answer, and I hope that the answer will be interesting to others in this discussion.

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: Thu, 16 Jan 2003 08:02:48 -0800 (PST)
From: [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
To: Keith Moore <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> people often claim that it's okay to use utf-8 without tagging in contexts
> where other charsets may also appear, because sufficiently long strings of
> utf-8 can be distinguished (more or less reliably) from other charsets by
> checking to see if the string is valid utf-8.

> question:

> in an environment where either utf-8 or gb18030 may appear, how reliably
> can gb18030 and utf-8 strings be identified and distinguished from one another?
> offhand it appears that many gb18030 strings are also valid utf-8 strings.

Most of the time it should be easy. Both GB18030 and UTF-8 are ASCII
compatible, but the similarities stop there. Most Chinese characters in GB18030
encode as two bytes >0x07f, same as GB2312 or GBK. The rest of Unicode maps to
a set of four byte sequences. While aligning with valid UTF-8 is possible, it
isn't very likely, and the likelihood drops dramatically the more data you
have.

Distingushing GB18030 from other multibyte encodings of Chiness, Japanese, or
Korean is much trickier, however. You certainly can tell the difference from
the varius iso-2022 derived encodings with ease (and those encodings can be
distinguished from each other fairly easily too), but the structure of some of
these multibyte CJK encodings is very similar. This can change the problem from
one of looking for valid byte sequences in a particular encoding to having to
use frequency count information and the like. Not only is this much more
difficult, requiring large and complex tables, it also requires much more data
in order to be able to guess.

If you actually used GB18030 to encode text written in a script other than
Roman or Kanji you'd have something that might be recognizable as being
uniquely in GB18030.

There are companies that specialize in selling software to do this sort of
thing, you know. I haven't dealt with such software myself but I know it
exists.

				Ned

Date: Thu, 16 Jan 2003 07:45:55 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: distinguishing between utf-8 and gb18030
Sender: [email-masked]

Keith,

Although well intentioned, you are pursuing a question that is
fundamentally irrelevant to an IETF technical effort.

The IETF does not produce technical specifications that use heuristics.
A specification needs to provide a precise (and efficient) means of
distinguishing different data.

The line of question you are asking is based on data analysis that is
either a heuristic or requires an unbounded (or at least far too large a
memory) lexical analyzer.

Given the difficulty of getting clear, precise and workable transition
and operations analysis from the proponents of "just use UTF-8" and
given their failure to attend to the extended discussion of such a model
10 years ago, pursuing your question provides them encouragement where
none should exist.

d/

Thursday, January 16, 2003, 7:19:47 AM, you wrote:

Keith> people often claim that it's okay to use utf-8 without tagging in contexts
Keith> where other charsets may also appear, because sufficiently long strings of
Keith> utf-8 can be distinguished (more or less reliably) from other charsets by
Keith> checking to see if the string is valid utf-8.

Keith> question:

Keith> in an environment where either utf-8 or gb18030 may appear, how reliably
Keith> can gb18030 and utf-8 strings be identified and
Keith> distinguished from one another?

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Thu, 16 Jan 2003 10:19:47 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: distinguishing between utf-8 and gb18030
Sender: [email-masked]

people often claim that it's okay to use utf-8 without tagging in contexts
where other charsets may also appear, because sufficiently long strings of
utf-8 can be distinguished (more or less reliably) from other charsets by
checking to see if the string is valid utf-8.

question:

in an environment where either utf-8 or gb18030 may appear, how reliably
can gb18030 and utf-8 strings be identified and distinguished from one another?
offhand it appears that many gb18030 strings are also valid utf-8 strings.

Keith

Date: Thu, 16 Jan 2003 10:15:51 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

> There is a clean solution:
> "Here's how to do it properly: use RFCs 2047 / 2231."  That (a) is
> backwards compatible, (b) is consistent with Best Current Practice
> (RFC 2277), (c) is consistent with the Internet Architecture (RFC
> 1958), (d) is compatible with RFCs 822, 2822 and the MIME RFCs, and
> (e) is inclusive (GB18030 users are not ostracized). Raw untagged
> utf-8 does not posess those characteristics.

it's not a complete solution, because 2047 only applies to human-readable
text, and the 2231 extensions only apply to parameter values in MIME headers.

Date: Thu, 16 Jan 2003 09:52:34 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Seth Breidbart <[email-masked]> writes:
> 
> 
>>My take is that some people want/need to send 8-bit, so if the
>>standard says "don't" they ignore it (and find that it works,
>>mostly).  If the standard says "Here's how to do it properly" most of
>>them will be perfectly happy doing it that way.
> 
> 
> Exactly so. If the standard stays silent, the present chaos will surely
> increase. If it provides an approved way to do it, then there is at least
> the possibility that people will move over to that way.
> 
> I am still worried that the Chinese will stick with GB18030, though, and
> there is no clean solution to that in sight.

There is a clean solution:
"Here's how to do it properly: use RFCs 2047 / 2231."  That (a) is
backwards compatible, (b) is consistent with Best Current Practice
(RFC 2277), (c) is consistent with the Internet Architecture (RFC
1958), (d) is compatible with RFCs 822, 2822 and the MIME RFCs, and
(e) is inclusive (GB18030 users are not ostracized). Raw untagged
utf-8 does not posess those characteristics.

Date: 16 Jan 2003 10:40:22 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Bart Schaefer writes:
> the people who wanted non-ASCII character sets in headers did not have
> the clout to force most of the established players to go along.  

False. Popular demand forced the last 7-bit MTA implementors to support
ISO 8859-1 throughout messages, including header fields. Try deploying a
new 7-bit MTA and you'll find yourself laughed out of the marketplace.
Welcome to the real world.

> assorted transports (and maybe even some MUAs) could not be counted upon
> to pass through a header field with an unknown name, or to pass it undamaged

Are you claiming that all new fields were bad? Nothing in your argument
has any connection to the semantics of the unknown header field. Unless
you're willing to say

   A new Content-Type field was a bad idea: maybe some clueless MTA
   destroyed all fields it didn't recognize. We should have used tags
   inside message bodies.

then you can't say the same thing about a new Header-Character-Encoding
field versus tags in the message header.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

From: "Bart Schaefer" <[email-masked]>
Date: Thu, 16 Jan 2003 08:08:56 +0000
Comments: In reply to "D. J. Bernstein" <[email-masked]> "Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)" (Jan 15, 11:34pm)
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Jan 15, 11:34pm, D. J. Bernstein wrote:
} 
} Bruce Lilly writes:
} > header field order can (and does) change, so there's no way of
} > guaranteeing that a header-encoding header-field would always precede
} > use of any specified encoding
} 
} So what? Does your MUA also have trouble displaying From before Subject
} when the From line in the message comes after the Subject line?

As I recall the discussion at the time (circa RFC 1342), the issue was not
the ordering, but rather that each header field needed to be interpreted
in isolation, because assorted transports (and maybe even some MUAs) could
not be counted upon to pass through a header field with an unknown name,
or to pass it undamaged.  (It was some while before senders really had any
confidence that the RFC 1341 [no, that's not a typo] headers would reach
all recipients, unless I'm misremembering.)

The interesting feature of RFC 1342/1522/2047 is therefore not that they
maintain a 7-bit encoding, but that they package the character set (and
now language) information in one header field along with the content, in
a form that will not be disrupted by those transports that conform to an
earlier standard and that therefore do not expect a "known" field to be
carrying such a "package."

Of secondary interest, but another reason for not using a separate field,
is that the encoding can represent multiple character sets and languages
in that same single header.  (Obviously that's not so significant if a
"universal" character set is available, but language information is still
of interest even if it isn't absolutely essential.)

Its true that the 7-bit encoding was chosen so that the "package" would
pass undamaged through the greatest proportion of intermediary software,
and that some of that software could arguably be called "broken," but
the politics were such that the people who wanted non-ASCII character
sets in headers did not have the clout to force most of the established
players to go along.  

Guess what:  Those politics haven't changed, though some people are now
on the opposite side of the figurative aisle.  A number of posters have
argued that Usefor should recognize that the UTF-8 headers proposal has
no political future, regardless of its technical merits.

However, political arguments tend to encourage an accusatory tone with a
lot of mud-slinging, so I'm going to try to formulate a social position
instead.

Advocates of UTF-8 in usenet headers want to take the same chance that
1341 senders took -- that their content might not get through undamaged
everywhere, particularly when gatewayed -- rather than adopt a workaround
of the kind 1342/1522/2047 represents.

This doesn't seem unreasonable on the face of it, but the difference is
that RFC 1341 had no effect on the mechanics of message transmission;
it introduced no changes in any existing headers, and no syntactically
incompatible changes in the extended headers.  The content might become
broken, but the process could not.

Changing existing headers to UTF-8 content *can* break the process; adding
new headers in UTF-8 *is not* syntactically compatible; and it *is not*
possible to predict where that breakage will occur or how the unexpected
syntax will be handled by software that conforms to older standards. Can
anyone refute any of those statements without resorting to probabilistic
arguments?  Using anything but 7-bit in headers is a calculated risk.

It's one thing to accept a risk to your own data, but quite another to
standardize on something that imposes that risk on others, no matter how
unlikely you think it is that anything "really bad" will happen, and no
matter how desirable the outcome once all the breakage is "outgrown."
That, in my view, is why the proposal as it stands is not acceptable, and
is the issue that has to be addressed before an acceptable proposal is
possible.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: For shame (Was: Re: RFC 2047 and gatewaying)
From: Andrew Gierth <[email-masked]>
Date: 16 Jan 2003 06:22:52 +0000
Lines: 21
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 > Henry Spencer wrote:
 >> non-ASCII newsgroup names are already in substantial use. 

 Bruce> Really?

to the best of my knowledge, actual use of non-ASCII newsgroup names
is largely confined at present to non-Usenet uses (i.e. private
servers or private subnets). This does mean that there are probably no
good ways to get statistics on how widespread this usage is, since I
can't just dredge relevent statistics out of the raw newsfeed the way
I've done for other issues.  (However, the importance of non-Usenet use
of NNTP and the Netnews format should not be underestimated.)

All the actual usage of non-ASCII names that I've encountered on such
servers has been in the form of untagged usage of the local 8-bit
charset.

-- 
Andrew.

To: [email-masked]
Xref: clerew local.mime:2264
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Date: Wed, 15 Jan 2003 12:40:36 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Seth Breidbart <[email-masked]> writes:

>My take is that some people want/need to send 8-bit, so if the
>standard says "don't" they ignore it (and find that it works,
>mostly).  If the standard says "Here's how to do it properly" most of
>them will be perfectly happy doing it that way.

Exactly so. If the standard stays silent, the present chaos will surely
increase. If it provides an approved way to do it, then there is at least
the possibility that people will move over to that way.

I am still worried that the Chinese will stick with GB18030, though, and
there is no clean solution to that in sight. That might leave them as a
"cooperating subnet" (and that is just a polite way of saying they were
rogue :-) ).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 15 Jan 2003 23:42:14 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Russ Allbery writes:
> In fact, they're a huge distraction and waste of time and effort for
> the few people who are developing *for Usenet* to have to re-solve
> problems already being addressed by other protocols.

Right. That's why you should use the existing general-purpose solutions:
Unicode as a character set and UTF-8 as a character encoding.

There are some other protocols that are faced with buggy software---for
example, Sendmail destroying bytes 128 through 159---and that have made
the mistake of developing ad-hoc workarounds rather than demanding that
the bugs be fixed. There's no reason for USENET to make that mistake.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 15 Jan 2003 23:34:41 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Bruce Lilly writes:
> header field order can (and does) change, so there's no way of
> guaranteeing that a header-encoding header-field would always precede
> use of any specified encoding

So what? Does your MUA also have trouble displaying From before Subject
when the From line in the message comes after the Subject line?

> precludes use of different encodings for different header fields

Use a universal character set and a universal encoding. This is not
rocket science.

> how would you magically change the large installed base of software
> (not to mention networks) overnight?

Nobody is talking about an overnight change. Deployment of 8-bit body
support in sendmail took some time after Allman implemented it in 1993,
for example.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 15 Jan 2003 23:19:34 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: For shame (Was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Bruce Lilly writes:
> As I have always observed and practiced use of the term, "consensus" means:
> + there is at least one proponent of the proposition in question
> + there is no opposition on substantive grounds to the proposition.

The IETF instead uses ``rough consensus,'' which is defined as ``any
situation that the working group chairs think that they can get away
with declaring to be rough consensus.''

For example, the DNSEXT chairs sent the ``axfr-clarify'' document to
the IESG, even though seven people have publicly objected to that
document on various substantive grounds.

As another example, the IDN chairs declared ``rough consensus'' on IDNA
even though more than _three hundred_ people publicly objected to IDNA,
specifically to its failure to prohibit multiple names with identical
semantics. (There were many other objections, but IDNA's mishandling of
semantic similarity is what has really angered users.)

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 15 Jan 2003 22:53:25 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Why we're in this mess
Sender: [email-masked]

Bruce Lilly writes:
> "identify" and "untagged" are diametrically opposed.

An out-of-band specification of all text as UTF-8 is just as informative
as an in-band tag.

I see that Lilly wrote ``Bruce Lilly'' instead of ``=?us-ascii?Q?Bruce?=
=?us-ascii?Q?Lilly?=''. Did he therefore fail to identify the charset of
his name? Does software have to guess whether he's using ASCII or EBCDIC
or something else?

Of course not. All (untagged 7-bit) text is identified, out of band, as
ASCII. See, for example, RFC 822.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Wed, 15 Jan 2003 14:27:32 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: ``ASCII-compatible'' does not mean ``7-bit''
Sender: [email-masked]

Wednesday, January 15, 2003, 1:54:32 PM, you wrote:
D.> RFC 2047 is _not_ compatible with ASCII. RFC 2047 specifies a different
D.> interpretation of some of those strings.

in case anyone wondered whether this thread had any hope of producing
anything productive, Dan's posting should be conclusive.

let's move on.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: 15 Jan 2003 22:12:48 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Why we're in this mess
Sender: [email-masked]

822/2822 specify an extremely flexible Received-line format. At least
for 2822, this flexibility is quite deliberate.

821/2821 specify a more restricted Received-line format to be used
``when an SMTP server receives a message.'' There is no requirement that
the restricted format be used in other contexts.

Lilly is complaining about a perfectly valid Received line that has
nothing to do with SMTP. He is completely wrong when he asserts that
``from'' and ``by'' are required in all Received lines.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Wed, 15 Jan 2003 17:05:18 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Keith Moore <[email-masked]> writes:
> 
> 
>>>Nobody here has been prepared to say that they expect RFC 2047 encoding to
>>>be around in 20 years time. 
> 
> 
>>I expect that in 20 years time there will still be a need to read old messages
>>that use both 2047 encoding and old charsets.
> 
> 
> Sure, software may still be recognizing it as "obsolete syntax", but it
> will not be in regular use for current messages.

The relevant BCP RFC (2277) has this to say:
    Negotiating a charset may be regarded as an interim mechanism that is
    to be supported until support for interchange of UTF-8 is prevalent;
    however, the timeframe of "interim" may be at least 50 years, so
    there is every reason to think of it as permanent in practice.

Date: Wed, 15 Jan 2003 17:03:59 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: ``ASCII-compatible'' does not mean ``7-bit''
Sender: [email-masked]

On 15 Jan 2003 21:54:32 -0000
"D. J. Bernstein" <[email-masked]> wrote:

> RFC 2047 is _not_ compatible with ASCII. 

yawn. 

Date: 15 Jan 2003 21:54:32 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: ``ASCII-compatible'' does not mean ``7-bit''
Sender: [email-masked]

Keith Moore writes:
> Yes, it's probably desirable to migrate to a new format which relies more
> on utf-8 and less on ascii-compatible encoding schemes.

The American Standard Code for Information Interchange is more than
simply the set 0, 1, 2, ..., 127. It specifies an _interpretation_ of
each of those numbers as a (printable or control) character, and thus an
interpretation of strings of those numbers.

RFC 2047 is _not_ compatible with ASCII. RFC 2047 specifies a different
interpretation of some of those strings.

In contrast, UTF-8 is compatible with ASCII. Every ASCII string has the
same interpretation under UTF-8.

The feature of the RFC 2047 encoding that you care about is that it's a
7-bit encoding. In particular, it's compatible with some buggy MTAs. In
contrast, UTF-8 requires some MTAs to be fixed.

You and the other IDNA GoofyCode people make yourselves sound like fools
when you misuse the phrase ``ASCII-compatible'' to mean ``7-bit,'' and
in particular when you refer to blatantly incompatible encodings as
``ASCII-compatible.''

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Wed, 15 Jan 2003 16:45:05 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Henry Spencer <[email-masked]>
CC: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Henry Spencer wrote:

> No, you referred to "newgroup tags", a curious and non-standard phrase
> which has nothing obvious to do with newsgroup names.  You may think that
> was "clearly" a reference to newsgroup names, but it certainly wasn't
> clear to me. 

Beware "name" -- it has implication w.r.t. case-independence (RFC 1958)
and language-tagging (RFC 2277), particularly if one is aiming for a
standards-track RFC.

Perhaps newgroups should be treated in a case-independent manner, and
perhaps they should be language-tagged.  But that has not to date been
the case.

Date: Wed, 15 Jan 2003 16:04:39 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: Henry Spencer <[email-masked]>, [email-masked], Usefor Mailing List <[email-masked]>
Subject: For shame (Was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Henry Spencer wrote:

> non-ASCII newsgroup names are already in substantial use. 

Really?  What percentage of the total number of newsgroups?
20%?  That would barely qualify as "substantial". I'd be
surprised if it amounts to a tiny fraction of that.

More to the point: how many in the millenium in which the
charter referring to "urgent attention" which formed the WG
was written? That's the same charter which says

   The Goal of this working group is to publish a standards-track
   successor to RFC 1036 that with particular attention to backward
   compatibility, formalizes best current practice and best
   proposed practice. The Group shall also aid and/or oversee the
   production of other Usenet related Internet Drafts and Standards.

"Standards-track" has a very specific meaning which precludes
the Usefor draft as it now stands due to a number of issues.
"Particular attention to backward compatibility" also has quite
specific meaning.

So where's the standards-track successor to RFC 1036 that pays
particular attention to backward compatibility? Oh that's
right, there isn't one.  OK, so how many of the "other Usenet
related Internet Drafts and Standards" has the WG produced? None.

Certainly by 1998, to use Bernstein's milestone, all the pieces were
in place to deal with the "urgent attention" issues (possibly
excepting newsgroups, which would not seem to be particularly urgent
considering that some five years later there is a mere handful
of affected newsgroups) in accordance with "The Goal" of
the WG. In 1998 MIME was in it's fifth year of deployment and third
incarnation, complete with full charset and language tagging in
compliance with the BCP RFC 2277.  The specific issue of
newsgroups could have been dealt with as one of the "other"
documents provided for in the charter.  The real shame is that
the pointless bickering has been going on for years in spite of
repeated calls to get on with a document addressing the issues
on which consensus (more on that below) can be reached, deferring
the remaining issues for a second document.  The bickering is
pointless precisely because The Goal of producing a *standards-track*
successor to RFC 1036 *with particular attention to backward
compatibility* ought to have resolved a large number of issues
a long time ago; issues that are still being (pointessly) debated.

Returning to consensus: I was amused and somewhat bewildered to
see that a poll of something like 9:8 was reported as "rough
consensus" of the WG.  As a past chairman of a[n] (IEEE) standards
committee and as present chairman of a section of an international
professional society of engineers, I have a clear concept of what
consensus means. Maybe I'm just an old fogey, but here's what it
*doesn't* mean (_recent_ dictionary definition additions notwithstanding):
- a particular vote count (there are separate terms, e.g. plurality,
   majority, 3/4 vote, etc.)
- there is disagreement on substantive matters
nor is consensus necessarily limited to a synonym for "unanimity",
which is one of the traditional dictionary definitions of consensus.
As I have always observed and practiced use of the term, "consensus"
means:
+ there is at least one proponent of the proposition in question
+ there is no opposition on substantive grounds to the proposition.
Those two simple principles are very powerful.  Consensus, as I have
thus summarized it, permits moving forward with action where there is
no opposition to a proposition, but for which there may be considerable
apathy (unlike requiring a majority of affirmative support to proceed).
It also prevents giving the imprimatur of the group to some proposition
which has serious shortcomings (unlike a majority). That is most
definitely a positive attribute, as it forces open debate on the
substantive issues, and such debate often leads to resolution of
the issues. That is not to say that a spurious objection (i.e. not
on substantive issues) can stall action, nor should it open debate
on frivolous matters -- that is why committes have chairmen; to
focus the group on the important issues, and to remain on track with
the group's purpose and goal.  In rare cases where consensus cannot be
reached, the proposition can be deferred for later discussion, or in
even rarer cases, tabled.

I would therefore like to suggest to the Usefor WG Chair that the WG
again consider extracting from the current Usefor draft, modified
where necessary, those items which are consistent with the goal of
the WG as documented in its charter (including the charter's
stipulation that the 1036 successor be eligible for standards-track
status and that it pay particular attention to backward compatibility),
with the objective of meeting that stated goal of the WG.  Remaining
content to be deferred to later discussion and action once that goal
is on its way to being achieved.

It is clear from recent discussion, much of it on ietf-822 only, that
the current draft has several provisions which preclude standards-track
status and that backward compatibility (in the sense in which that term
is used in the IETF) has not been properly addressed.  It follows that
the draft in its entirety has no chance of meeting the WG goal.

The review of the draft with that objective in mind need not be a lengthy
or difficult process; it should be fairly clear where there are backwards
compatibility issues, where there are standards-track issues (ref. RFC 2026),
and relevant BCP RFCS (e.g. 2277) can provide guidance -- this isn't rocket
science, and there has been considerable discussion both on usenet-format
and ietf-822.

Date: Wed, 15 Jan 2003 14:15:09 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: rather than argue and bicker about who said what...
Sender: [email-masked]

maybe we can agree on some basic constraints for gatewaying news into mail?

e.g.

1. until the mail spec is revised (which is to say, not in the near future) no unencoded non-ascii characters are permitted in the message header.

2. 2047-style encoded-words are only permitted for text that is intended
for human consumption - it is not permitted to use 2047 for machine-readable protocol elements.

3. for other usenet message header protocol elements which need to contain non-ascii characters (e.g. newsgroup names) usefor (or its successor) is encouraged to investigate the suitability of using the idna encoding.

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: Wed, 15 Jan 2003 13:32:27 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>Any raw untagged utf-8 in any header field is an example (that is exactly
>>what one would get if the properly tagged and encoded form which is legal
>>in header fields were decoded).
> 
> 
> "Encoded-word" is a technical term defined in RFC 2047. It encompasses
> objects of the general form
>     =?charset?encoding?data?=

No the *general* form is =?charset*language?encoding?text?=.

> Whilst an encoded-word might indeed be decoded into raw untagged utf-8
> under some circumstance, it is totally illogical to claim that every
> instance of "raw untagged utf-8" that one might encounter must therefore
> have resulted from "decoding some encoded-word".

We're not discussing "every instance [...] that one might encounter";
we're discussing header fields such as To, Cc, Bcc, Reply-To, Subject,
etc., and the *only* way that non-ASCII content can appear in header
fields is via the 2047/2231 mechanisms.

> I have repeatedly invited you to indicate exactly where the Usefor draft
> requires (or even allows) any encoded-word to be decoded (other than for
> final display in a UA).
> 
> That invitation is still open.

You have yourself claimed that encoded-words in header fields can be
decoded for a followup, and the Usefor draft clearly permits such text
resulting from decoding those encoded-words to subsequently be sent in
raw untagged utf-8 to a gateway.

Date: Wed, 15 Jan 2003 13:22:28 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Dave Crocker <[email-masked]> writes:
> 
> 
>>The issue is not whether there is an existence proof for the practise you
>>want to standardize.  The issue is, instead, the impact of a new standard on
>>the installed base that uses the old standard.
> 
> 
> The primary impact will be on the installed base within Usenet. That is an
> issue for the Usefor WG to consider, and it has concluded that the impact
> is manageable.

That installed base includes gateways and moderators.  While some in the
Usefor WG may think that the impact of the proposed changes are
manageable, there are a significant number of others who strongly
disagree. And the IESG and IETF will have to be convinced that the
installed base will not be presented with data which is illegal under the
existing standard  -- that is what "the impact of a new standard on the
installed base that uses the old standard" means.

> The impact on the installed email base is important, but secondary. We
> propose to tackle it by specifying rather strongly what must happen at the
> interface. There will be some impact, but we believe it is manageable.

You've missed the point. It's not a question of saying "abracadabra, change!";
the existing installed base operating under the existing standard cannot
be handed data which that installed base is unprepared for.  "Specifying
what must happen" doesn't address that issue.

Date: Wed, 15 Jan 2003 09:50:49 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Charles Lindsey" <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Charles,

Wednesday, January 15, 2003, 4:05:50 AM, you wrote:
>>I expect that in 20 years time there will still be a need to read old messages
>>that use both 2047 encoding and old charsets.

Charles> Sure, software may still be recognizing it as "obsolete syntax", but it
Charles> will not be in regular use for current messages.

Given that January 1, 2003 marked the 20th anniversary of continuous,
production use of TCP/IP, and that the core technical formats of
Internet email have been in continuous use for 25 years (remember than
RFC822 was merely an upgrade to RFC733)...

You might want to be a bit cautious about such certitude.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Wed, 15 Jan 2003 13:09:48 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Henry Spencer <[email-masked]>
CC: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Henry Spencer wrote:
> On Wed, 15 Jan 2003, Bruce Lilly wrote:
> 
>>>>UTF-8 does not adequately address i18n issues (language-tsgging)...
>>>
>>>As others have noted, 2047-amended has its problems... and modern Unicode
>>>(and thus UTF-8) *does* address language tagging...
>>
>>MIME does not lack higher-level tagging facilities and is specifically
>>mentioned in the Unicode documents as one of those areas where the Unicode
>>language-tagging facility is not to be used.
> 
> 
> Correct but completely irrelevant, since the issue under discussion was
> UTF-8 as an alternative to MIME.  You were claiming that MIME was superior
> because it had language tagging and UTF-8 didn't; that's incorrect.

Please follow the thread. Bernstein claimed that utf-8 should have been
required in 1994 and required as a default in 1998.  Unicode didn't
have a language-tagging facility until Unicode 3.1 introduced in mid-2001.
In 1998, 2047/2231 *did* have language tagging, as required by RFC 2277.
So claims of what Unicode 3.1/3.2 are capable of in contexts where MIME
is not used (which doesn't include news articles; see the Usefor charter)
is irrelevant to the issue discussed, viz. what could be standardized in
1994/1998.

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Wed, 15 Jan 2003 09:47:09 -0800
Lines: 23
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> Yes, and Usenet (or rather Usefor) is quite willing to stick its neck
> out and be the guinea pig. It is an environment that is much more
> tolerant of interoperability failures (or, it you like, it has lived
> with them for years) and so is much better placed to cope with that
> initial disruprion.  And we are quite willing to try to keep that
> disruprion out of email so far as we can.

I usually don't point this out explicitly, since I'd be posting all day to
do so and I figure folks here are smart enough to figure it out, but this
is so blatant that I cannot just let it stand.

Charles, as eloquently as he expresses his personal opinions, does not
speak for Usenet.  He does not even speak for Usefor, although his
opinions seem to be relatively representative of a slim majority on the
group.  He is Usefor document editor, but not the Usefor chair, and there
are some of us on Usefor who are not at all convinced that Usenet is eager
and willing to serve as the guinea pig for testing UTF-8 in a messaging
protocol.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Wed, 15 Jan 2003 12:46:50 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

On Wed, 15 Jan 2003, Bruce Lilly wrote:
> > It provides for non-ASCII character sets in *some* header fields.  (Taken
> > literally, 2047 makes it impossible to use its facilities in, for example, 
> > newsgroup names.) 
> 
> As the part which you didn't quote clearly pointed out as the sole case
> in the current Usefor draft where 2047 is inapplicable...

No, you referred to "newgroup tags", a curious and non-standard phrase
which has nothing obvious to do with newsgroup names.  You may think that
was "clearly" a reference to newsgroup names, but it certainly wasn't
clear to me. 

> and further
> points out that changes to newsgroup names are *not* mentioned in the
> Usefor charter *at all*...

Except under the part about non-ASCII characters in headers.  "Newsgroups"
is a header, and an important one at that.  Or are you arguing that the
charter only refers to some headers?  My understanding is that the charter
wording was most definitely intended to include "Newsgroups" as a header
which needed non-ASCII support -- indeed, one of the most urgent ones,
since non-ASCII newsgroup names are already in substantial use. 

                                                          Henry Spencer
                                                       [email-masked]

Date: Wed, 15 Jan 2003 12:37:43 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> Yes, I am coming round to the view that, in the few cases where a gateway
> cannot determine the proper way to encode some header in RFC 2047, it
> would be better to stick an "X-" in front of that header and then encode
> it as unstructured. That gives the user the best chance of working out
> what was originally intended, and little harm will ensue since it cannot
> have been a header that was essential for the basic News or Email
> protocols.

That still doesn't address backward compatibility, because that's not
what existing gateways do.

Date: Wed, 15 Jan 2003 12:37:18 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Wed, 15 Jan 2003, Bruce Lilly wrote:
> >>UTF-8 does not adequately address i18n issues (language-tsgging)...
> > As others have noted, 2047-amended has its problems... and modern Unicode
> > (and thus UTF-8) *does* address language tagging...
> 
> MIME does not lack higher-level tagging facilities and is specifically
> mentioned in the Unicode documents as one of those areas where the Unicode
> language-tagging facility is not to be used.

Correct but completely irrelevant, since the issue under discussion was
UTF-8 as an alternative to MIME.  You were claiming that MIME was superior
because it had language tagging and UTF-8 didn't; that's incorrect.

> Moreover, as Dan Kohn has
> pointed out (on ietf-822), that language-tagging facility is under review
> (it likely will be deprecated).

Perhaps, and perhaps not.  It is already "strongly discouraged", with
various similar phrases encouraging one to avoid it unless there is no
reasonable alternative.  However, it *is* part of Unicode and is not 
going to go away.

                                                          Henry Spencer
                                                       [email-masked]

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Wed, 15 Jan 2003 09:33:26 -0800
Lines: 25
Sender: [email-masked]

Andrew Gierth <[email-masked]> writes:
>>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

>  Russ> No, I was more thinking that the server could just use GNU
>  Russ> Libidn, which was recently released under the LGPL.  :)

> jeez. Just the .tar.gz for that library is comparable in size to the
> entire unpacked source code of my server. (And even discounting the
> docs, data files, etc., it's something like 50% of the number of lines
> of code.)

Yeah, but the number of places that you have to use it is pretty small, so
you can just link it shared into things like newsgroup creation tools.
It's a larger problem for readers, though.

I wonder how much of that complexity is the Unicode canonicalization.
That will have to be done anyway if we want to use UTF-8 anywhere in any
form.

(Oh, and Libidn also includes code to do other types of canonicalization
than IDNA, such as Kerberos canonicalization, so not all that code is
used.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Xref: clerew local.mime:2263
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Date: Wed, 15 Jan 2003 12:32:53 GMT
Lines: 93
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> The reason people disobey the
>> 7-bit standard is that the 7-bit standard sucks.

>True.  Then again, the obvious alternative also sucks - at least in
>the short term.  Just because something sucks doesn't mean it's not
>the best way to do something.

Though it is unlikely to be the best way to do something if people
routinely disobey it :-( .

But yes, I find myself in agreement with most of what you are saying here.

>I do believe that migrating to utf-8 in message headers is the way
>to go in the long term, and that a transition strategy similar to what
>Dan is suggesting (support utf-8 on receipt soon, generation of utf-8
>later) is ultimately the way to get there from here.  (I suspect we
>have very different ideas about what would be a reasonable interval
>between those two events)

And probably very little control over that interval, too :-( .

>But there are several things I don't believe.

>One, that this will return us to a world where messages are ordinary
>text and can be treated as such.  For instance, even if we allow utf-8
>in message headers, there will still be a need for canonicalization of
>certain fields before sending and/or before comparison of values
>embedded in those fields,

Yes, individual protocols will impose such constraints as are necessary.
The Usefor draft has a lot to say about newsgroup-names, for example.

>Two, that this allows use of utf-8 in addresses or domain names.
>Those are separate problems,

Yes. I think they should eventually be allowed (and will happen anyway).
But that is an issue for the email standards. All addresses and domain
names in Usefor currently follow RFC 2822 exactly, and it is not proposed
to change that.

>Three, that the existing ability of some user agents to display utf-8
>in message headers is sufficient for proper processing of headers 
>containing utf-8,

>Four, that user agents that support utf-8 receipt in the near term will
>receive sufficient testing through ordinary usage to be reliable at
>supporting it once generation of utf-8 in headers is endorsed.
>(so some other testing will be needed - not the way IETF normally works)

Support for and acceptance of Unicode (whether as UTF-16 or UTF-8) seems
to be rising exponentially. It started from a very low level 10 years ago
(as Dan keeps reminding us), has now reached the point of being very
visible, and will therefore (if the exponential is followed) be almost
universal in no time at all. No doubt early implementations got it wrong in
various ways (including normalizations, no doubt) but we can expect those
things to be put right, since the Unicode standards are pretty clear.

>Five, that supporting utf-8 rids user agents of the burden
>of supporting 2047, IDNA, and similar encodings, at least for
>reading/presentation purposes, probably within our lifetimes.

Probably sooner than you would think (or like) if my exponential
hypothesis is correct.

>In other words, utf-8 is the right way to go for the long term - 
>but the devil is in the details and it's not nearly as simple
>as it looks.  And I do think that a two-step transition is necessary -
>we can't just start cramming utf-8 into either email or usenet and
>expect things to work reasonably.

>I also think that usenet can serve as a good test case for email.
>(translation: I'd much rather usenet suffer the initial disruption than
>email, so email could perhaps learn from usenet's experience.)

Yes, and Usenet (or rather Usefor) is quite willing to stick its neck
out and be the guinea pig. It is an environment that is much more tolerant
of interoperability failures (or, it you like, it has lived with them for
years) and so is much better placed to cope with that initial disruprion.
And we are quite willing to try to keep that disruprion out of email so
far as we can.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2260
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Wed, 15 Jan 2003 11:57:20 GMT
Lines: 55
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> You claimed (and your words are still quoted above, preceded by ">>>>>")
>> that there was a design flaw in Usefor which involved "decoding
>> encoded-words for transmission". Will you please explain exactly where in
>> Usefor it is specified that any encoded-word is required (or even allowed)
>> to be "decoded for transmission".

>Any raw untagged utf-8 in any header field is an example (that is exactly
>what one would get if the properly tagged and encoded form which is legal
>in header fields were decoded).

"Encoded-word" is a technical term defined in RFC 2047. It encompasses
objects of the general form
    =?charset?encoding?data?=
Whilst an encoded-word might indeed be decoded into raw untagged utf-8
under some circumstance, it is totally illogical to claim that every
instance of "raw untagged utf-8" that one might encounter must therefore
have resulted from "decoding some encoded-word".

In fact, it is NEVER the case that any raw untagged utf-8 that results
from the Usefor draft will have arisen from decoding some encoded-word.

>  When such untagged 8-bit data (in general,
>utf-8 in particular) is sent by a UA to a gateway (which is subsequently
>expected to magically [re-]encode it), that is what I mean by "decoding
>encoded-words for transmission".

Indeed, untagged 8-bit data can be sent to a gateway, but not "such"
untagged 8-bit data, because your "such" evidently has no antecedent.

So if you are claiming that sending untagged 8-bit data, which has never
been in the form of an encoded-word and has never been decoded from
anything, can then be described as "decoding encoded-words for
transmission", then all I can say is that you are living in
Humpty-Dumpty-land.

I have repeatedly invited you to indicate exactly where the Usefor draft
requires (or even allows) any encoded-word to be decoded (other than for
final display in a UA).

That invitation is still open.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2259
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Wed, 15 Jan 2003 11:28:08 GMT
Lines: 31
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>The issue is not whether there is an existence proof for the practise you
>want to standardize.  The issue is, instead, the impact of a new standard on
>the installed base that uses the old standard.

The primary impact will be on the installed base within Usenet. That is an
issue for the Usefor WG to consider, and it has concluded that the impact
is manageable.

The impact on the installed email base is important, but secondary. We
propose to tackle it by specifying rather strongly what must happen at the
interface. There will be some impact, but we believe it is manageable.

The important difference between Usenet and Email is that Usenet is a
Social thing, within which a certain degree of failed interoperability can
be (and often is) tolerated. We realise that email operates under much
tighter constraints, but we are talking here of the little bit of Usenet
that escapes into email and which, from a social point of view, is really
just an extension of Usenet itself.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2261
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Wed, 15 Jan 2003 12:03:53 GMT
Lines: 29
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> As I have said, the real problems are in generating RFC 2047.
>> You have to second guess what a plausible encoding of some unknown header
>> might be, and then you have to second guess what receiving agents might do
>> if you guessed wrongly.

>I'm very much opposed to having intermediaries trying to guess what
>encoding was used at the source.  Tag it as "unknown", encode it, 
>and move on. 

Yes, I am coming round to the view that, in the few cases where a gateway
cannot determine the proper way to encode some header in RFC 2047, it
would be better to stick an "X-" in front of that header and then encode
it as unstructured. That gives the user the best chance of working out
what was originally intended, and little harm will ensue since it cannot
have been a header that was essential for the basic News or Email
protocols.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2262
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047
Date: Wed, 15 Jan 2003 12:05:50 GMT
Lines: 21
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> Nobody here has been prepared to say that they expect RFC 2047 encoding to
>> be around in 20 years time. 

>I expect that in 20 years time there will still be a need to read old messages
>that use both 2047 encoding and old charsets.

Sure, software may still be recognizing it as "obsolete syntax", but it
will not be in regular use for current messages.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 15 Jan 2003 12:11:07 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Leo Bicknell <[email-masked]>
CC: [email-masked]
Subject: Re: Why we're in this mess
Sender: [email-masked]

Leo Bicknell wrote:
> In a message written on Wed, Jan 15, 2003 at 10:07:23AM -0500, Bruce Lilly wrote:
> 
>>still doesn't comply with RFC 821 (which requires a space character 
>>immediately
>>preceding the semicolon which delimits the time stamp in a Received header),
> 
> 
> I'll note the problem here is that RFC 821, and RFC 822, both define a
> Received header.  821 requires the space, 822 does not.

But SMTP servers, which are what the quote specifically applies to, are
bound by *both* specifications, and as 822 does not prohibit a space,
it is unquestionably required.

Now there are some contradictions between 821 and 822, most of which
(but not all) have been eliminated in 2821 / 2822, but that's another
story.

The point is that in spite of a crystal clear specification of what
an SMTP server is supposed to generate for a time stamp line (a.k.a.
Received header field), more than two decades later, it's still the
case (and the discrepancy isn't exactly a closely-guarded secret) that
most SMTP servers are non-compliant.  Anybody who thinks that SMTP
server implementations are going to change en masse quickly is deluded.

N.B. this isn't a case of "sendmail got it wrong"; in sendmail the
Received field content is specified in the configuration file, and it's
trivially easy to fix the broken standard configuration files via an edit.
Others, including D. J. "Ego bigger than all outdoors" Bernstein got
it wrong too -- and in Bernstein's qmail, the error is hard-coded in
the source code.

Date: Wed, 15 Jan 2003 08:37:25 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Leo Bicknell <[email-masked]>
CC: [email-masked]
Subject: Re: Why we're in this mess
Sender: [email-masked]

Leo,

Wednesday, January 15, 2003, 7:51:22 AM, you wrote:
Leo> I'll note the problem here is that RFC 821, and RFC 822, both define a
Leo> Received header.  821 requires the space, 822 does not.

And the writing of both specs was reasonably well coordinated.

So there is a lesson, here, about having the same thing defined in two
documents.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Wed, 15 Jan 2003 11:22:49 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Henry Spencer <[email-masked]>
CC: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Henry Spencer wrote:
> On Wed, 15 Jan 2003, Bruce Lilly wrote:
> 
>>>  - Use of non-ASCII character sets in article headers and bodies
>>
>>That's subsumed by the part that says
>>   - Standardization of article bodies and the use of MIME in articles.
>>since MIME provides for non-ASCII character sets in header fields and bodies.
> 
> 
> It provides for non-ASCII character sets in *some* header fields.  (Taken
> literally, 2047 makes it impossible to use its facilities in, for example, 
> newsgroup names.) 

As the part which you didn't quote clearly pointed out as the sole case
in the current Usefor draft where 2047 is inapplicable, and further
points out that changes to newsgroup names are *not* mentioned in the
Usefor charter *at all*, and therefore ought to have had no effect on
what should have been a timely adoption of MIME, including 2047, to address
the urgent issues which *are* delineated in the charter.

Date: Wed, 15 Jan 2003 11:16:05 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Henry Spencer <[email-masked]>
CC: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Henry Spencer wrote:
> On Wed, 15 Jan 2003, Bruce Lilly wrote:
> 
>>UTF-8 does not adequately address i18n issues (language-tsgging), as
>>per RFC 2277 (1998).  RFC 2047 as amended by 2231 (1997) *does* address
>>i18n issues, including laguage-tagging.
> 
> 
> As others have noted, 2047-amended has its problems... and modern Unicode
> (and thus UTF-8) *does* address language tagging, using a facility that
> was added specifically to accommodate Internet protocols lacking
> higher-level tagging facilities. 

MIME does not lack higher-level tagging facilities and is specifically
mentioned in the Unicode documents as one of those areas where the Unicode
language-tagging facility is not to be used.  Moreover, as Dan Kohn has
pointed out (on ietf-822), that language-tagging facility is under review
(it likely will be deprecated).

Date: Wed, 15 Jan 2003 11:14:24 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

On Wed, 15 Jan 2003, Bruce Lilly wrote:
> >   - Use of non-ASCII character sets in article headers and bodies
> 
> That's subsumed by the part that says
>    - Standardization of article bodies and the use of MIME in articles.
> since MIME provides for non-ASCII character sets in header fields and bodies.

It provides for non-ASCII character sets in *some* header fields.  (Taken
literally, 2047 makes it impossible to use its facilities in, for example, 
newsgroup names.)  The temptation to diverge from MIME's handling of the
issue would have been considerably reduced if the MIME facilities had been
more general-purpose.

                                                          Henry Spencer
                                                       [email-masked]

Date: Wed, 15 Jan 2003 11:02:43 -0500
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Shmuel (Seymour J.) Metz wrote:
> In <[email-masked]>, on 01/07/2003
>    at 08:45 PM, [email-masked] (Kent Landfield) said:
> 
> 
>>Yeah right...  The real problem is that people didn't keep the focus 
>>on the group's charter
> 
> 
> Including the part that lists as urgent
> 
>   - Use of non-ASCII character sets in article headers and bodies

That's subsumed by the part that says

   - Standardization of article bodies and the use of MIME in articles.

since MIME provides for non-ASCII character sets in header fields and bodies.

Now it's true that the MIME methods (2047 / 2231) aren't directly
applicable to non-ASCII newgroup tag components (and that's the sole
exception), but then (N.B.!) there's no mention of non-ASCII newsgroup
tag components in the charter.  Kent is absolutely correct.

Date: Wed, 15 Jan 2003 10:55:52 -0500
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Russ Allbery wrote:
> Charles Lindsey <[email-masked]> writes:
> 
> 
>>Nobody here has been prepared to say that they expect RFC 2047 encoding
>>to be around in 20 years time.
> 
> 
> I expect to see RFC 2047 encoding still in use in 20 years time.  I
> believe I already said that.

I definitely said so. In fact in <[email-masked]> I wrote:
    we will continue to need mechanisms like 2047 / 2231 [yes, Charles, even
    20 or 30 years hence

> My position is basically this:  Usenet does not have a lot of mind share
> or a lot of market share, but it continues to be very useful for some
> particular areas.

Agreed.

> There are significant problems
> on Usenet that need resolution, such as [...]
 >  improving the linkage between
> mail and news for gatewaying, and the whole specter of authentication and
> authorized article deletion.

Indeed, *improving* the compatibility of mail and news (rather than incompatible
divergence) ought to be addressed.  And implications of any proposed changes on
authentication should be given adequate consideration now -- any proposal that
requires changing message content after signing (e.g. encoding in gateways) will
simply not work, and will only cause severe problems when authentication is fully
addressed.  Just-send-8-bits-and-fix-it-in-the-gateway may make life easy for
lazy UA programmers now (and make life impossible for gateway developers), but
it will result in severe problems.  Note that the Usefor charter lists signed
articles as the first of the "areas [which] need urgent attention".

> Time spent trying to solve problems that are *not* unique to Usenet and
> that don't further any of the things that make Usenet interesting and
> useful as a protocol suite are not nearly as interesting.  In fact,
> they're a huge distraction and waste of time and effort for the few people
> who are developing *for Usenet* to have to re-solve problems already being
> addressed by other protocols.  This is *particularly* true when it comes
> to problems that are already being solved with e-mail, given that one
> reason why there are as many news clients as there are is because it's
> fairly trivial to write a news client once you've already written a decent
> mail client.

 From my perspective, solving problems that are related to the Internet text
message format as extended by MIME and other RFCs is interesting.  Under the
current news article format standard (RFC 1036) that includes news (there are
a few relatively minor special cases -- and one goal of Usefor ought to be
a migration toward *better* compatibility), but there are no insurmountable
problems.  If there is eventually an article format which deviates
sufficiently from the Internet text message format, I have no plan to
adress that format, precisely because Usenet is such a small part of the
"big picture".  Show-stoppers include subtantially-different definitions
for a common header field (e.g. parameters, raw 8-bit content in message
and MIME-part header fields), redefinition of existing MIME media types
such as message/rfc-822; which are in the current Usefor draft.

[some material elided for brevity -- no disagreement]

> In other words, all of these debates are very important, but I think that
> so far as possible, Usenet should simply say "this is a different way of
> transferring e-mail messages with a few additional fields" and be done
> with it, just like RFC 1036 did.  People who feel strongly about
> particular ways of handling such common problems as internationalization
> should contribute actively to the e-mail standards world and Usenet will
> *automatically* follow further developments in e-mail.

Agreed wholeheartedly.

> As an implementor, I would dearly like it if people involved in Usenet
> standardization work would stop trying to get me to work on problems that
> are not *Usenet* problems and instead let me use the mass of existing
> techniques, libraries, and tools for handling the e-mail message format
> and concentrate my work on those parts of Usenet that actually make it
> *Usenet*.  (Like newsgroup names.)

As an implementor of libraries and tools for handling the Internet text
message format (email, and currently -- under 1036 -- news) I would dearly
like it if people involved in Usenet standardization would stop trying to
deviate from that common format. I'm happy to let specialists in respective
areas deal with display issues, GUIs, voice messaging esoterica, and specifics
such as those related to Usenet (e.g. newsgroups and distributions)

Date: Wed, 15 Jan 2003 10:51:22 -0500
From: Leo Bicknell <[email-masked]>
To: [email-masked]
Subject: Re: Why we're in this mess
Organization: United Federation of Planets
Sender: [email-masked]

In a message written on Wed, Jan 15, 2003 at 10:07:23AM -0500, Bruce Lilly wrote:
> still doesn't comply with RFC 821 (which requires a space character 
> immediately
> preceding the semicolon which delimits the time stamp in a Received header),

I'll note the problem here is that RFC 821, and RFC 822, both define a
Received header.  821 requires the space, 822 does not.

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

Date: Wed, 15 Jan 2003 10:07:23 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Why we're in this mess
Sender: [email-masked]

D. J. Bernstein wrote:
> Keith ``7 bits forever!'' Moore writes:
> 
>>your naivete is showing even worse than usual -- about what was
>>technically feasible at the time, what was politically feasible at the
>>time, what would interoperate with the installed base at the time
> 
> 
> The ietf-smtp archives aren't a secret, Keith.
> 
> We can all see that, early in 1991, Robert Ullmann and Andre Pirard and
> others proposed requiring 8-bit-clean mail software. This requirement
> should have been adopted.
> 
> We can all see that the only objection to that requirement was the claim
> that 8-bit support would take a long time to be deployed. You were aware
> of the massive long-term benefits; you simply refused to plan ahead.
> 
> For example, Paul Vixie said that he had some seven-year-old sendmail
> binaries, and concluded ``with near-certainty'' that ``any changes to
> the SMTP spec will take at least a decade to reach 90% of the critical
> server population.''

If knowledgeable people are asked to name those who can speak authoritatively
about sendmail and SMTP, Paul Vixie's name is one of the three that are likely
to top the list (others being Allman and Avolio).  If that was 1991, and here
we are "at least a decade" later and 90% (possibly more) of SMTP software
still doesn't comply with RFC 821 (which requires a space character immediately
preceding the semicolon which delimits the time stamp in a Received header),
the conclusion is that Paul's predictive ability is quite good -- and (N.B.)
that's a fundamental aspect of the syntax, clearly given in ABNF, and one
which hasn't changed until recently (2821 relaxes the requirement to mandatory
CFWS before the semicolon).  Indeed, there is some software that fails to
comply with even the *basic* requirements for an RFC-821 time-stamp line,
such as the *required* "from" and "by" clauses,
e.g.:
   Received: (qmail 13349 invoked by uid 1016); 15 Jan 2003 02:09:13 -0000

The ietf-smtp archives aren't the only things which aren't secret...

> Similarly, UTF-8 support should have been required in all mail software
> when RFC 2277 was published five years ago.

Untagged UTF-8 is incompatible with section 4 of 2277. 2047/2231 (which
predate 2277) are compliant with 2277.  Before invoking RFC 2277, you might
try reading it.

For those still advocating untagged utf-8; note RFC 2277 section
3.1:
    All protocols MUST identify, for all character data, which charset is
    in use.
"identify" and "untagged" are diametrically opposed.

and section 4.2:
    Protocols that transfer text MUST provide for carrying information
    about the language of that text.

 > Instead, according to your
> so-called ``standards,'' it's perfectly acceptable---in 2003---to deploy
> a mailer that throws away everything other than ASCII.

No, according to standards-track RFCs and BCP RFCs it is unacceptable to
have untagged, unencoded non-ASCII (and some ASCII) content in Internet
text message headers and MIME-part headers.  One cannot speak of "throw[ing]
away" something which cannot be there in the first place.

According to your half-baked ideas, it's perfectly acceptable to ignore the
Best Current Practice RFC 2277 -- the very one that you are trumpeting -- which
clearly and unambiguously states that charsets MUST be identified and that
there MUST be provision for carrying language information.

Date: Wed, 15 Jan 2003 09:49:01 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Wed, 15 Jan 2003, Bruce Lilly wrote:
> UTF-8 does not adequately address i18n issues (language-tsgging), as
> per RFC 2277 (1998).  RFC 2047 as amended by 2231 (1997) *does* address
> i18n issues, including laguage-tagging.

As others have noted, 2047-amended has its problems... and modern Unicode
(and thus UTF-8) *does* address language tagging, using a facility that
was added specifically to accommodate Internet protocols lacking
higher-level tagging facilities. 

                                                          Henry Spencer
                                                       [email-masked]

Date: Wed, 15 Jan 2003 09:32:44 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

D. J. Bernstein wrote:
> Keith Moore writes:

>>Which is precisely why 2047 was never intended for anything that needs
>>to be interpreted by machine.
> 
> 
> Even with that (shortsighted) restriction, RFC 2047 was badly designed.
> For example, you should have declared encoding in a separate header
> field, not by inserting bytes into existing header fields.

That wouldn't work. It's OK to specify body encoding in a header field (as
is done with Content-Transfer-Encoding) because the header always precedes
the body.  But header field order can (and does) change, so there's no way
of guaranteeing that a header-encoding header-field would always precede
use of any specified encoding.  It also begs the question of what encoding
is used for such a hypothetical header-encoding header field.  It also
precludes use of different encodings for different header fields (think of
Resent- fields, for example).

Second-guessing past design in hindsight is one thing.  Doing so with an
ill-concieved half-baked purported alternative is another.

 > You should
> have required 8-bit-clean software in 1991;

And what about networks and charset issues?  And how would you magically
change the large installed base of software (not to mention networks)
overnight?

 > pandering to 7-bit software
> was a huge mistake.

It not just the software...

 > You should have required UTF-8 support in 1994. You
> should have required UTF-8 as a default in 1998.

UTF-8 does not adequately address i18n issues (language-tsgging), as
per RFC 2277 (1998).  RFC 2047 as amended by 2231 (1997) *does* address
i18n issues, including laguage-tagging.

Date: Wed, 15 Jan 2003 09:20:59 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

On Wed, 15 Jan 2003 09:12:31 -0500
Bruce Lilly <[email-masked]> wrote:

> 
> Russ Allbery wrote:
> > Using RFC 2047 for the Newsgroups header is probably unworkable for a
> > variety of reasons, most notably being that RFC 2047 is not a unique
> > encoding format (there are multiple ways to encode the same word), and all
> > existing Usenet software requires that a given newsgroup have one and only
> > one textual representation.
> 
> That reason is not a show-stopper; Unicode (and therefore utf-8) has the
> same problem (multiple representations of the same word), and the issue
> can be easily handled by specifying a subset of the general method.

I would like to maintain the convention that 2047 format is only used
for human-readable text, on the assumption that various tools and translators
feel free to decode and/or translate anything that matches =?[BQbq]?[^?]*?=

Date: Wed, 15 Jan 2003 09:18:14 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein wrote:

> The Subject line ends up being created by somebody, encoded as per RFC
> 2047, decoded for editing in a followup, encoded again, decoded again
> for editing in someone else's followup, etc.
> 
> The second encoding often has very different bytes from the first.

What is "very different" (vs. "different") supposed to mean?

If the Subject field is edited, it obviously will (except for the
special case of a null effect) have different bytes from the original.
What is your point (assuming that there is one)?

Date: Wed, 15 Jan 2003 09:12:31 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Russ Allbery wrote:
> Using RFC 2047 for the Newsgroups header is probably unworkable for a
> variety of reasons, most notably being that RFC 2047 is not a unique
> encoding format (there are multiple ways to encode the same word), and all
> existing Usenet software requires that a given newsgroup have one and only
> one textual representation.

That reason is not a show-stopper; Unicode (and therefore utf-8) has the
same problem (multiple representations of the same word), and the issue
can be easily handled by specifying a subset of the general method. For
example, a 2047-like encoding (since 2047 is not directly applicable for
other reasons) simply using =xx (q encoding) could work.  Alternatively,
a 2231-like (or a similar URL-like) encoding using %xx could work.

Which raises an interesting question for the MIME authors: 2231 section
4 states
    Percent signs
    ("%") are used as the encoding flag, which agrees with RFC 2047.
whereas 2047 uses '=' for that purpose.  Presumably the flag character
was originally intended to be the same, but was changed w/o changing
the text -- is that correct?  (an erratum might be in order).

The "one and only one textual representation" is an issue that favors
having the canonical form be the compatible form, as then there is no
need for multiple codec cycles, each of which has the possibility of
an error (e.g. failure to decode or encode -- highly likely as there
is currently no encoding or decoding involved in gateways).  That is
in addition to the issues of
1. compatibility with the Internet text message format, which *never*
    permits 8-bit content in any header fields.
2. compatibility with existing protocols (IMAP, SMTP) which prohibit
    8-bit content in header fields
3. compatibility with existing gateways, which do not modify Newsgroups,
    Followup-To, etc. header fields.

Date: Wed, 15 Jan 2003 08:40:43 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:

> Nobody here has been prepared to say that they expect RFC 2047 encoding to
> be around in 20 years time.

That is untrue. I have said so, clearly and unambiguously (and added that I
expect it to be around longer).  At least one other person has also said so.

So Charles,

1. Did you fail to read the responses to the issue which you raised?  If so,
    what was the point of raising the issue in the first place?

2. Did you read the responses but somehow fail to comprehend them?  If so,
    you may have a serious reading comprehension problem (this seems quite
    likely, given your failure to note the clear and unabiguous wording in
    the MIME RFCs regarding header field content in both message and
    multipart media types).

3. Did you read and understand the responses but nevertheless think that in
    a public forum with an archive that you could get away with prevarication?

Date: Wed, 15 Jan 2003 08:30:38 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: prevervation of installed base
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
>>
>>>No, I am talking primarily about gateways *from* news.
> 
> 
>>Yes, from news and to ... what?  If not "to email", as I said, then why
>>are you discussing it here?  If in fact we *are* still talking about
>>news->mail gateways, then please answer the question.
> 
> 
> Gateways *from* news are a matter to be defined by Usefor. If those
> gateways are *to* mail, then Usefor has to ensure they generate compliant
> mail, or else make it clear that they do not. We are discussing it here to
> see whether or not Usefor has so ensured, and in particular to look at
> some awkward cases to see how they are best resolved.

You have once again evaded the question at hand, which is:

 > Therefore, it is reasonable to proceed on the basis that existing gateways
 > will either drop non-conforming input on the floor, or they will pass it
 > on in a form that will be obviously "wrong" (i.e. will appear as
 > gobbledegook) wherever it is viewed.
 >
 > The question is whether that situation is acceptable in a transitional
 > phase,

You're talking about gateways to email -- what "transitional phase" -- what
are you assuming will change in the email protocols?

In addition, it has also been pointed out (by several people) that handing
illegal input (in this case non-ASCII header field content) to existing
implementations where that is illegal is neither conforming nor backwards
compatible.  That is not a mere matter of *appearance*; is is a protocol
violation.

Perhaps it doesn't really matter that you are unwilling to answer the question;
your assertion that "it is reasonable" for "existing gateways" to "pass on"
"obviously 'wrong'" content in violation of protocols is incorrect.  It is
neither reasonable nor acceptable (full stop).

Date: Wed, 15 Jan 2003 08:15:01 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Keith Moore <[email-masked]> writes:

>>but it only matters for the purpose of display.  if you're using an encoded-word
>>for anything other than purely human-readable text, you're not using them 
>>properly.
> 
> 
> Actually, it does matter for purposes other than display, because people
> want to construct whitelists, blacklists, killfiles, etc so as to filter
> or compartmentalise their mail and news, and they are certainly not going
> to construct those lists using raw encoded-words.

Keith is still correct; the human-readable text isn't reliable for the purposes
cites as they are easily changed / forged / removed.  Instead of a display name,
filtering on the (machine-readable) addr-spec is more reliable.  Subject is
another matter, but filtering on Subject content has never been reliable.

One *could* however filter using encoded-words; there is no technical reason
why that wouldn't work (regexp matching would have to take special characters
into account in any case). It is true that fully-general 2047 encoding does
provide for variation in encoding (Q vs. B, =20 vs. _, etc.), but that is
also true of any Unicode encoding including utf-8 (pre-composed accented
characters vs. non-spacing modifiers, alternative forms, etc.).  If this were
deemed a sufficiently interesting issue, Usefor *could* still use compatible 2047
representation along the same lines as RFC 1036; specify a more restrictive
subset of what is permitted in general Internet text messages (e.g. specify
Q encoding only).

>>and outside of to/cc/from/bcc/reply-to, few structured fields contain human
>>readable text anyway.
> 
> 
> And that I think is the real point that I want to make. So long as
> gateways manage to encode those cases correctly (and that is a MUST in my
> proposed wording) then the few cases in other onscure headers that slip
> through will hardly be a problem in practice.

1. As those header fields are common to mail and news, they should always
    use 2047 for non-ASCII content and may never contain raw 8-bit content.
2. Existing gateways do not need to encode any headers now, because the
    current standard (1036) does not permit raw 8-bit content.  A change
    would break compatibility with those gateways (and with IMAP, etc.).
3. And what about the newly-proposed Mail-Copies-To field? Or Complaints-To?
    Existing gateways have no syntax information for those, so could not
    encode. Not to mention any header fields that might be added in future.
4. Only the generating UA has the charset and language information necessary
    for 2047 tagging; only the generating UA can do the job -- a gateway
    cannot.
5. There remains the issue of picking *one* way; instead of saying use 2047
    or use raw utf-8 (as the Usefor draft currently does), it is far better
    to simply say use 2047:
  a. one (raw utf-8) is not compatible
  b. when there are two ways of doing something, pick one
  c. one (raw utf-8) does not provide for i18n considerations (language-tagging)

Date: Wed, 15 Jan 2003 07:51:09 -0500
From: Keith Moore <[email-masked]>
To: Andrew Gierth <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

> It's clearly easier to implement UTF-8 than to implement UTF-8 _and_ IDN.

actually, it's not - because the IDN libraries already exist, because you
need to do canonicalization anyway, and because "implementing only UTF-8"
introduces additional complexity elsewhere (in the email-news boundary)
that may not be necessary if you use IDN.

To: [email-masked]
Xref: clerew local.mime:2258
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Wed, 15 Jan 2003 11:17:42 GMT
Lines: 22
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>You are probably right, but you are not understanding the religion.

>The religion is called "protect the installed base and do upgrades at the
>edges rather than the infrastructure whenever possible".

Yes, if you regard Usenet as being "at the edges" of email, then that is
what Usefor is trying to do. And it that makes the edges/interfaces more
complex in the short term, then that may be a price worth paying. Anyway,
that is the conclusion that the Usefor WG came to.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Subject: Re: Historical trivia
From: Nick Shelness<[email-masked]>
Date: Wed, 15 Jan 2003 11:32:07 +0000
Sender: [email-masked]

"Rolf E. Sonneveld" <[email-masked]> wrote 

> But alphanumerical timezones has always been ambiguous 
> as I'm quite sure you will be able to find other use of MET and WET. I 
> think the numeric format has the future.

Unfortunately, as we discovered in the iCalendar working group, numeric 
format works for present times and for past times (if one can determine 
what the UTC offset was on that date). Unfortunately, it doesn't work for 
some future times. This is because the DST rules for that location may 
change (for a whole variety of reasons). Thus, for future times, one is 
forced to employ a geographic area designator that in the case of 
iCalendar is registered with IANA. But given geopolitical realities, even 
this binding may change over time.

Nick

Nick Shelness
Independent Technology Consultant <[email-masked]>

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Andrew Gierth <[email-masked]>
Date: 15 Jan 2003 11:20:34 +0000
Lines: 12
Sender: [email-masked]

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 Russ> No, I was more thinking that the server could just use GNU
 Russ> Libidn, which was recently released under the LGPL.  :)

jeez. Just the .tar.gz for that library is comparable in size to the
entire unpacked source code of my server. (And even discounting the
docs, data files, etc., it's something like 50% of the number of lines
of code.)

-- 
Andrew.

Date: Wed, 15 Jan 2003 11:50:06 +0100
From: "Rolf E. Sonneveld" <[email-masked]>
Subject: Re: Historical trivia
To: Nick Shelness <[email-masked]>
Cc: [email-masked]
Reply-to: [email-masked]
Organization: Sonnection B.V.
Sender: [email-masked]

Hi,

Nick Shelness wrote:

>Dave Crocker wrote:
>
>  
>
>>(I'd guess that GDT means Greenwhich Daylight Time, but
>>suspect there really is no such thing.)
>>    
>>
>
>For what it's worth, Britain currently uses two time-zone(?) designators. 
>Greenwich (no "h") Mean Time (GMT) in the winter and British Summer Time 
>(BST) in the summer. During World War II it also used a third (= GMT+2). I 
>have no idea what that was called. Also, since Portugal (among other 
>countries) is in the same time-zone(?). I presume they use another 
>designator for GMT+1 in the summer. I have no idea what that is.
>
AFAIK nearly all of Western European countries (except for the British 
Isles and Ireland and maybe Iceland) use MET (Middle European Time) in 
winter and MET-DST in summer. MET is UTC+0100 while MET-DST is UTC+0200. 
And, I've also seen WET (Western European Time) used for the UK and 
Ireland as well. But alphanumerical timezones has always been ambiguous 
as I'm quite sure you will be able to find other use of MET and WET. I 
think the numeric format has the future.

/rolf

Rolf E. Sonneveld
www.sonnection.nl

To: [email-masked]
Subject: Re: Historical trivia
Sensitivity: 
From: Nick Shelness<[email-masked]>
Date: Wed, 15 Jan 2003 10:24:59 +0000
Sender: [email-masked]

Dave Crocker wrote:

> (I'd guess that GDT means Greenwhich Daylight Time, but
> suspect there really is no such thing.)

For what it's worth, Britain currently uses two time-zone(?) designators. 
Greenwich (no "h") Mean Time (GMT) in the winter and British Summer Time 
(BST) in the summer. During World War II it also used a third (= GMT+2). I 
have no idea what that was called. Also, since Portugal (among other 
countries) is in the same time-zone(?). I presume they use another 
designator for GMT+1 in the summer. I have no idea what that is.

Nick

Nick Shelness
Independent Technology Consultant <[email-masked]>

Date: Wed, 15 Jan 2003 12:20:36 +0200
From: Alan Barrett <[email-masked]>
To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Keith Moore wrote:
> I agree that 2047 is not suitable for this purpose.  My guess is that
> the IDNA stuff is the best starting point.

draft-faerber-i18n-email-netnews-names-00.txt contains a proposal
for newsgroup names based on IDNA.

--apb (Alan Barrett)

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Andrew Gierth <[email-masked]>
Date: 15 Jan 2003 08:58:31 +0000
Lines: 37
Sender: [email-masked]

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 >> This is one of the reasons why my proposal (which was the basis
 >> for what ended up in the draft, other than the fact that I did not
 >> propose using UTF-8 for anything other than newsgroup names, or in
 >> message/rfc822)

 Russ> It lost quite a bit in the translation there; I would have
 Russ> voted for that over the option chosen.

I know.

This, btw, for the ietf-822 crowd, was what I originally proposed to
Usefor a little over a year ago:

  1. Strike all basic syntax and punt it back to the mail standards.
  Remove any changes that would have the effect of allowing UTF-8 in
  headers meaningful to mail. Instead, explicitly provide for the use
  of UTF-8 in Newsgroups and Followup-To, and provide _exactly one_
  mapping to a format compatible with mail, to allow gateways to handle
  it, and to allow servers to reject Newsgroups lines which have been
  munged for mail purposes. As regards UTF-8 in other fields, point out
  that it should be a future direction, that there are currently issues
  with interoperability with mail, but that both standards should be
  aiming to migrate towards UTF-8 (or some such wording).

 Russ> Clients are going to have to implement an encoding whatever way
 Russ> you look at it; either they're going to have to implement
 Russ> UTF-8, or they're going to have to implement something else.
 Russ> I'm not sure that it's really that much easier to implement
 Russ> UTF-8 than IDN.

It's clearly easier to implement UTF-8 than to implement UTF-8 _and_ IDN.
It's not an either-or situation.

-- 
Andrew.

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 21:36:35 -0800
Lines: 70
Sender: [email-masked]

Andrew Gierth <[email-masked]> writes:
>>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

>  Russ> IDNA certainly will be the easiest to implement for the servers.

> I don't think I entirely agree - you seem to be assuming that the server
> can avoid implementing the encoding, but this is not necessarily true
> once you start thinking about administrative issues such as adding or
> removing groups.

No, I was more thinking that the server could just use GNU Libidn, which
was recently released under the LGPL.  :)  The big advantage of using a
standard encoding format is that many software authors won't have to
implement it themselves.

> There is also the 'mp3 problem' - certain sequences of US-ASCII
> characters appearing as the encoded form of a non-ASCII name will have
> undesired consequences on the propagation of groups.

Yeah, that's an issue.  That's likely to apply to pretty much any ASCII
encoding due to the way that newsfeed patterns are currently built.
(Although if that's the worst problem that we have, I think we'd get off
pretty easy.)

> plus, there is the '.' to '-' translation issue which one has to be
> careful of, and the fact that some sites are (bogusly) rejecting '%' in
> local-parts, so there are considerations that apply to this encoding
> that may be awkward in other contexts.

Yeah, although I believe IDN wins on all of those counts (although it does
use -, I believe, and I'm not sure if it does in ways that would cause us
problems).

> This is one of the reasons why my proposal (which was the basis for what
> ended up in the draft, other than the fact that I did not propose using
> UTF-8 for anything other than newsgroup names, or in message/rfc822)

It lost quite a bit in the translation there; I would have voted for that
over the option chosen.

> did not suggest using the encoding everywhere, but strictly at
> mail<->news borders such as the moderation mechanism. (Since only
> newsgroup names are affected, existing group- or hierarchy-specific
> gateways don't need to do anything special.)

The only worry that I have for this is that it adds a lot of encoding
transitions.  Each time one has to recode other than at the end point adds
another potential point of failure (which was one of the things that
people were complaining about with RFC 2047).  I'm not sure that saving
encoding on the wire is worth having news messages *necessarily* involve
8-bit characters in the headers as opposed to that being optional where
user agents are known to support it.

That being said, I could certainly live with this.

> This can be retroactively implemented for many existing injectors simply
> by wrapping the mail-to-moderator script to do the necessary conversions
> to the destination address and the Newsgroups header. This should be
> easier than doing it in all clients.

Clients are going to have to implement an encoding whatever way you look
at it; either they're going to have to implement UTF-8, or they're going
to have to implement something else.  I'm not sure that it's really that
much easier to implement UTF-8 than IDN.  Although UTF-8 does have the
advantage that if you don't implement it at all and just echo raw bytes to
the screen, it's vaguely, distantly possible that the right thing will
happen, at least on some Unix systems.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Andrew Gierth <[email-masked]>
Date: 15 Jan 2003 05:03:39 +0000
Lines: 46
Sender: [email-masked]

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 > Keith Moore <[email-masked]> writes:
 >> I agree that 2047 is not suitable for this purpose.  My guess is
 >> that the IDNA stuff is the best starting point.

 Russ> IDNA certainly will be the easiest to implement for the servers.

I don't think I entirely agree - you seem to be assuming that the
server can avoid implementing the encoding, but this is not
necessarily true once you start thinking about administrative issues
such as adding or removing groups.

There is also the 'mp3 problem' - certain sequences of US-ASCII
characters appearing as the encoded form of a non-ASCII name will have
undesired consequences on the propagation of groups.

 Russ> Note that there is no way to avoid a 7-bit encoding for
 Russ> newsgroup names without significant changes to the Usenet
 Russ> architecture, since posts to moderated groups are sent to
 Russ> <group-name>@moderators.isc.org by default and to some
 Russ> variation on that including the group-name even for hierarchies
 Russ> that don't use the isc.org forwarding system.  This requires
 Russ> embedding the newsgroup name in an SMTP envelope, and I don't
 Russ> expect any 8-bit encoding to work reliably enough there in the
 Russ> near future.

plus, there is the '.' to '-' translation issue which one has to be
careful of, and the fact that some sites are (bogusly) rejecting '%'
in local-parts, so there are considerations that apply to this
encoding that may be awkward in other contexts. This is one of the
reasons why my proposal (which was the basis for what ended up in the
draft, other than the fact that I did not propose using UTF-8 for
anything other than newsgroup names, or in message/rfc822) did not
suggest using the encoding everywhere, but strictly at mail<->news
borders such as the moderation mechanism. (Since only newsgroup names
are affected, existing group- or hierarchy-specific gateways don't
need to do anything special.)

This can be retroactively implemented for many existing injectors
simply by wrapping the mail-to-moderator script to do the necessary
conversions to the destination address and the Newsgroups header. This
should be easier than doing it in all clients.

-- 
Andrew.

Date: Tue, 14 Jan 2003 19:42:15 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Keith,

Tuesday, January 14, 2003, 7:16:59 PM, you wrote:

>> It's always trivial to criticise someone else's design, whereas it is
>> very difficult to work constructively in a consensus-based design
>> process and achieve something productive.
>> 
>> So it's not surprising that some folks choose the role of constant
>> critic, even to the point of damaging the process.

Keith> Well, we need the critics too.

constructive criticism, certainly.

"constant critic" is not the role of a constructive participant.  it is
neurosis.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 19:47:55 -0800
Lines: 82
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> Nobody here has been prepared to say that they expect RFC 2047 encoding
> to be around in 20 years time.

I expect to see RFC 2047 encoding still in use in 20 years time.  I
believe I already said that.

uuencode is still in very widespread use.  BinHex is still in use.  EBCDIC
is still in use.  Encoding formats don't disappear very quickly.

(If it weren't for these interminable discussions about the far future, we
could release new standards every few years, and if it becomes clear that
people are moving towards something else, we could change the standard.)

My position is basically this:  Usenet does not have a lot of mind share
or a lot of market share, but it continues to be very useful for some
particular areas.  In particular, it's much more efficient and effective
as a short-term archive than mailing list archives on the web, it's a much
more easily browsable archive format for long-term archives that one wants
to provide on-line, it handles a large number of messages more easily than
e-mail does (particularly over slow links), it's well-suited for applying
filtering techniques to messages without having to view them all, and it
fosters a culture that I personally happen to like.  It's possible that
IMAP will replace some or even all of these features eventually, but
widespread use of anonymous IMAP isn't really pushing Usenet aside yet.

I'm interested in continuing to explore what potential Usenet has for
those things that it's good at.  As a software implementor, those are the
parts of Usenet that I find interesting.  There are significant problems
on Usenet that need resolution, such as more effectively handling rich
content without taxing disk space on smaller servers, experimenting with
better and more varied filtering mechanisms, improving the linkage between
mail and news for gatewaying, and the whole specter of authentication and
authorized article deletion.

Time spent trying to solve problems that are *not* unique to Usenet and
that don't further any of the things that make Usenet interesting and
useful as a protocol suite are not nearly as interesting.  In fact,
they're a huge distraction and waste of time and effort for the few people
who are developing *for Usenet* to have to re-solve problems already being
addressed by other protocols.  This is *particularly* true when it comes
to problems that are already being solved with e-mail, given that one
reason why there are as many news clients as there are is because it's
fairly trivial to write a news client once you've already written a decent
mail client.

Accordingly, while I'm sure there are all sorts of interesting questions
about the best way to solve these problems, there are *far* more people
thinking about and working on e-mail than there are thinking about or
working on Usenet, and I believe it to be the height of hubris to think
that Usenet is going to be able to charge forward where e-mail is just
beginning to tread.  Usenet simply isn't cool enough, interesting enough,
or well-funded enough to be able to drive world revolution in something as
fundamental as character sets by itself.

I would love to see these debates on how e-mail should be handled to
continue, and to be well-considered and well-debated.  I want to see
continued resources applied to the problem of finding the best way of
allowing all languages to be easily represented in e-mail.

I think that trying to "help" by going off and doing something completely
different on Usenet is self-destructive, since Usenet doesn't have the
resources to strike out on its own.

In other words, all of these debates are very important, but I think that
so far as possible, Usenet should simply say "this is a different way of
transferring e-mail messages with a few additional fields" and be done
with it, just like RFC 1036 did.  People who feel strongly about
particular ways of handling such common problems as internationalization
should contribute actively to the e-mail standards world and Usenet will
*automatically* follow further developments in e-mail.

As an implementor, I would dearly like it if people involved in Usenet
standardization work would stop trying to get me to work on problems that
are not *Usenet* problems and instead let me use the mass of existing
techniques, libraries, and tools for handling the e-mail message format
and concentrate my work on those parts of Usenet that actually make it
*Usenet*.  (Like newsgroup names.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 14 Jan 2003 22:28:59 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> >> If you don't know whether it is structured or unstructured then you are,
> >> officially, stuck.
> 
> >but it only matters for the purpose of display.  if you're using an encoded-word
> >for anything other than purely human-readable text, you're not using them 
> >properly.
> 
> Actually, it does matter for purposes other than display, because people
> want to construct whitelists, blacklists, killfiles, etc so as to filter
> or compartmentalise their mail and news, and they are certainly not going
> to construct those lists using raw encoded-words.

correct.  nor would it work if they did. 

> >so if an implementation doesn't recognize a field and treats it as unstructured,
> >the worst that will happen is that it won't decode some encoded words before
> >display (because it won't recognize them next to some 'special' characters,
> >or it won't recognize that parentheses are comment delimeters).
> 
> Yes, but that is going beyond a strict reading of RFC 2047.

funny, that's what I got from a strict reading of RFC 2047.

> And some
> implementations may treat them as structured by default, or use other
> heuristics.

perhaps, but it's pretty useless to assume that a field is structured 
if you don't know the syntax.  even then, the effect is only that you
will treat some encoded-words as literals rather than as human-readable
text to be decoded.

> As I have said, the real problems are in generating RFC 2047.
> You have to second guess what a plausible encoding of some unknown header
> might be, and then you have to second guess what receiving agents might do
> if you guessed wrongly.

I'm very much opposed to having intermediaries trying to guess what
encoding was used at the source.  Tag it as "unknown", encode it, 
and move on. 

Keith

Date: Tue, 14 Jan 2003 22:24:29 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

> Nobody here has been prepared to say that they expect RFC 2047 encoding to
> be around in 20 years time. 

I expect that in 20 years time there will still be a need to read old messages
that use both 2047 encoding and old charsets.

I expect that for several years after adoption of any new message format
there will still be a need to be compatible with existing MUAs, MTAs, 
UUAs, UTAs, and gateways.

Yes, it's probably desirable to migrate to a new format which relies more
on utf-8 and less on ascii-compatible encoding schemes.  But it's not 
desirable to dismiss the compatibility issues with handwaving.

Date: Tue, 14 Jan 2003 22:16:59 -0500
From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

> It's always trivial to criticise someone else's design, whereas it is
> very difficult to work constructively in a consensus-based design
> process and achieve something productive.
> 
> So it's not surprising that some folks choose the role of constant
> critic, even to the point of damaging the process.

Well, we need the critics too.  Sometimes we can benefit from well-informed,
rational, technical hindsight.  But just as with any other technical work,
we have to evaluate it with whatever wisdom (or lack thereof) we can summon.
Even the wisest of critics is of little use if he cannot get people of less
wisdom to understand and accept his reasoning, and also to understand the 
limitations thereof. And having the wise man make all decisions doesn't scale.

Keith

To: [email-masked]
Xref: clerew local.mime:2214
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Tue, 14 Jan 2003 12:42:46 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> If you don't know whether it is structured or unstructured then you are,
>> officially, stuck.

>but it only matters for the purpose of display.  if you're using an encoded-word
>for anything other than purely human-readable text, you're not using them 
>properly.

Actually, it does matter for purposes other than display, because people
want to construct whitelists, blacklists, killfiles, etc so as to filter
or compartmentalise their mail and news, and they are certainly not going
to construct those lists using raw encoded-words.

>so for instance use of encoded-words for newsgroup names is an improper use.

Yes, that we are agreed on.

>so if an implementation doesn't recognize a field and treats it as unstructured,
>the worst that will happen is that it won't decode some encoded words before
>display (because it won't recognize them next to some 'special' characters,
>or it won't recognize that parentheses are comment delimeters).

Yes, but that is going beyond a strict reading of RFC 2047. And some
implementations may treat them as structured by default, or use other
heuristics. As I have said, the real problems are in generating RFC 2047.
You have to second guess what a plausible encoding of some unknown header
might be, and then you have to second guess what receiving agents might do
if you guessed wrongly.

>and outside of to/cc/from/bcc/reply-to, few structured fields contain human
>readable text anyway.

And that I think is the real point that I want to make. So long as
gateways manage to encode those cases correctly (and that is a MUST in my
proposed wording) then the few cases in other onscure headers that slip
through will hardly be a problem in practice.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2212
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Tue, 14 Jan 2003 12:28:18 GMT
Lines: 41
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
>> 
>> 
>>>Charles Lindsey wrote:
>> 
>> 
>>>>But it doesn't. Unicode 3.2 is the current standard, and defines precisely
>>>>what is allowed in UTF-8. All earlier versions are obsolete.
>> 
>> 
>>>So overnight all of the MS Windows 95, 98, NT, and 2000 implementations
>>>mysteriously vanished?
>> 
>> 
>> That is Bill Gates' problem. However, I doubt any of them ever generated
>> the deprecated (and now illegal) forms of UTF-8.

>Please use a little common sense; Unicode 3.1 was introduced in mid-2001.
>All of the aforementioned "operating" systems predate that (and many are
>still in widespread use).

Indeed so. But do any of them generate UTF-8 that is unacceptable to the
Unicode 3.2 standard? Because if they do not, then they do not pose any
problem.

And if they do, then they are not suitable for generating news articles
conforming to Usefor.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2215
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Date: Tue, 14 Jan 2003 12:47:06 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>Bruce Lilly writes:

>> Famous last words.  From my handy dead-tree copy of Unicode 2.0, page 
>> 2-4, under the "Full Encoding heading":
>>
>> "There are over 18,000 unassigned code positions that are available 
>> for future allocation. This number far exceeds anticipated character 
>> encoding requirements for all world characters and symbols."

>Yep. I have that too. The fact that 18,000 isn't enough doesn't mean 
>that about a million isn't enough.

And even if a million is not enough, the UTF-8 encoding, as currently
defined in RFC 2279, RFC 2279bis and Usefor is quite capable of coping
with the whole of UCS-32.

But perhaps even UCS-32 is not big enough for Bruce?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2217
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047
Date: Tue, 14 Jan 2003 12:59:00 GMT
Lines: 33
Sender: [email-masked]

In <[email-masked]> Brett Kottmann <[email-masked]> writes:

>Is there any (serious) disadvantage to relying on the format of email
>messages as the template for news messages?

>I've seen the discussions go on for some time...is it just me or does anyone
>else think that stating "the format of a news message and header shall
>conform to the current standard for email message bodies and headers" would
>save a great deal of time?

It might save a great deal of time now, but it would only store up trouble
for the future.

Nobody here has been prepared to say that they expect RFC 2047 encoding to
be around in 20 years time. Ergo, something has got to change sooner or
later. Usefor does not want to lock itself in to something that is
exceedingly inconvenient, is already being widely misued or ignored, and
is inevitably going to change at some point.

Granted that Email is not yet in a position to escape from this
straightjacket, that is no reason to prevent other protocols from doing
so, even if there is some price to pay at the interface between the two.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2216
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: prevervation of installed base
Date: Tue, 14 Jan 2003 12:51:33 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> 
>> No, I am talking primarily about gateways *from* news.

>Yes, from news and to ... what?  If not "to email", as I said, then why
>are you discussing it here?  If in fact we *are* still talking about
>news->mail gateways, then please answer the question.

Gateways *from* news are a matter to be defined by Usefor. If those
gateways are *to* mail, then Usefor has to ensure they generate compliant
mail, or else make it clear that they do not. We are discussing it here to
see whether or not Usefor has so ensured, and in particular to look at
some awkward cases to see how they are best resolved.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2213
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Tue, 14 Jan 2003 12:30:45 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> I think you will find that, in the countries where that spam originates
>> from, users are regularly using untagged non-ASCII in both headers and
>> bodies when communicating with each other. I have no way of verifying
>> whether that is true or not in email, but it is most certainly true in the
>> case of Usenet (you just need to look at the cn.* groups).

>And your point is..?

>If it is not "spammers use untagged non-ASCII, so it's OK for Usenet",
>then what precisely is it?

It is that NON-spammers in France, Poland and China (but especially in
China) are currently using untagged non-ASCII.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 14 Jan 2003 21:59:06 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Why we're in this mess
Sender: [email-masked]

> Aren't you even slightly ashamed of yourself?

Hey, I'm not the one trying to reopen arguments for things that were decided
ten years ago; I'm not the one making personal attacks; and I'm not going to
pretend that I had either perfect foresight, the ability to impose my will
on others, or the responsibility to do so, at the times you think I should
have done things in the way that your less-than-20/20 hindsight dictates.

Enjoy your delusions, Dan.  You're certainly putting lots of effort into them,
so you might as well get something out of them.

Keith

Date: Tue, 14 Jan 2003 21:54:25 -0500 (EST)
From: Seth Breidbart <[email-masked]>
To: [email-masked]
Reply-To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

My take is that some people want/need to send 8-bit, so if the
standard says "don't" they ignore it (and find that it works,
mostly).  If the standard says "Here's how to do it properly" most of
them will be perfectly happy doing it that way.

Seth

Date: Tue, 14 Jan 2003 18:25:59 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Keith,

Tuesday, January 14, 2003, 5:43:40 PM, you wrote:
>> Had Dan been participating in the discussions, 10 years ago, no doubt he
>> could have ensured that the design was perfect.

Keith> yep.  and so obviously perfect that he would have easily won consensus.

It's always trivial to criticise someone else's design, whereas it is
very difficult to work constructively in a consensus-based design
process and achieve something productive.

So it's not surprising that some folks choose the role of constant
critic, even to the point of damaging the process.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: 15 Jan 2003 02:09:13 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Why we're in this mess
Sender: [email-masked]

Keith ``7 bits forever!'' Moore writes:
> your naivete is showing even worse than usual -- about what was
> technically feasible at the time, what was politically feasible at the
> time, what would interoperate with the installed base at the time

The ietf-smtp archives aren't a secret, Keith.

We can all see that, early in 1991, Robert Ullmann and Andre Pirard and
others proposed requiring 8-bit-clean mail software. This requirement
should have been adopted.

We can all see that the only objection to that requirement was the claim
that 8-bit support would take a long time to be deployed. You were aware
of the massive long-term benefits; you simply refused to plan ahead.

For example, Paul Vixie said that he had some seven-year-old sendmail
binaries, and concluded ``with near-certainty'' that ``any changes to
the SMTP spec will take at least a decade to reach 90% of the critical
server population.''

In fact, 8-bit _body_ support was deployed much more quickly than that,
and if you hadn't screwed up in 1991 then complete 8-bit support would
have come along with 8-bit body support. But the issue is not the exact
number of years---the issue is how amazingly shortsighted you were.

Similarly, UTF-8 support should have been required in all mail software
when RFC 2277 was published five years ago. Instead, according to your
so-called ``standards,'' it's perfectly acceptable---in 2003---to deploy
a mailer that throws away everything other than ASCII.

Aren't you even slightly ashamed of yourself?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 14 Jan 2003 20:43:40 -0500
From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Tue, 14 Jan 2003 17:14:39 -0800
Dave Crocker <[email-masked]> wrote:

> 
> Folks,
> 
> Tuesday, January 14, 2003, 4:38:02 PM, you wrote:
> D.> Even with that (shortsighted) restriction, RFC 2047 was badly designed.
> D.> For example, you should have declared encoding in a separate header
> 
> 
> Had Dan been participating in the discussions, 10 years ago, no doubt he
> could have ensured that the design was perfect.

yep.  and so obviously perfect that he would have easily won consensus.

Keith

Date: Tue, 14 Jan 2003 17:14:39 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Folks,

Tuesday, January 14, 2003, 4:38:02 PM, you wrote:
D.> Even with that (shortsighted) restriction, RFC 2047 was badly designed.
D.> For example, you should have declared encoding in a separate header

Had Dan been participating in the discussions, 10 years ago, no doubt he
could have ensured that the design was perfect.

Living with imperfection is such a heavy burden.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 14 Jan 2003 20:07:01 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On 15 Jan 2003 00:38:02 -0000
"D. J. Bernstein" <[email-masked]> wrote:

> 
> Keith Moore writes:
> > I don't share your confidence that this happens in every environment,
> > for every program that generates header fields.
> 
> That isn't your problem. If you want C-normalized text on the wire,
> specify C-normalized text on the wire. You don't need to know whether
> software accomplishes this by having text C-normalized in the keyboard
> interface (as everyone does now) or in the MUA (which would work too).

okay, whatever. 

> > > Anyway, your hypothetical Unicode normalization problems would also
> > > arise in normalizing RFC 2047, so you can't use normalization as an
> > > argument against moving to UTF-8.
> > Which is precisely why 2047 was never intended for anything that needs
> > to be interpreted by machine.
> 
> Even with that (shortsighted) restriction, RFC 2047 was badly designed.
> For example, you should have declared encoding in a separate header
> field, not by inserting bytes into existing header fields. You should
> have required 8-bit-clean software in 1991; pandering to 7-bit software
> was a huge mistake. You should have required UTF-8 support in 1994. You
> should have required UTF-8 as a default in 1998.

your naivete is showing even worse than usual -- about what was technically
feasible at the time, what was politically feasible at the time, what would
interoperate with the installed base at the time, what powers "I" had at 
any of these times, etc.  even with ten years of hindsight I'm not sure
how I could have made 1342 much better given the constraints we had to 
work with at the time.

but no matter, 1342/1522/2047 are water under the bridge anyway. 

Keith

Date: 15 Jan 2003 00:38:02 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Keith Moore writes:
> I don't share your confidence that this happens in every environment,
> for every program that generates header fields.

That isn't your problem. If you want C-normalized text on the wire,
specify C-normalized text on the wire. You don't need to know whether
software accomplishes this by having text C-normalized in the keyboard
interface (as everyone does now) or in the MUA (which would work too).

> > Anyway, your hypothetical Unicode normalization problems would also
> > arise in normalizing RFC 2047, so you can't use normalization as an
> > argument against moving to UTF-8.
> Which is precisely why 2047 was never intended for anything that needs
> to be interpreted by machine.

Even with that (shortsighted) restriction, RFC 2047 was badly designed.
For example, you should have declared encoding in a separate header
field, not by inserting bytes into existing header fields. You should
have required 8-bit-clean software in 1991; pandering to 7-bit software
was a huge mistake. You should have required UTF-8 support in 1994. You
should have required UTF-8 as a default in 1998.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 14 Jan 2003 23:52:37 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> > Untagged UTF-8 is good. It is more and more widely supported by readers,
> > and will eventually be the only format produced by writers. It works.
> This is what we heard 10 years ago, too.

False.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 14 Jan 2003 23:48:27 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Russ Allbery writes:
> My point is just that I'd like to see some reason to believe that all
> the people sending unlabelled 8-bit content are interested in moving
> to UTF-8.

It's a question of costs and benefits. How do we minimize the damage
when the user wants to send something more than ASCII?

The simplest thing to do is take 8-bit data from the user---let's say
8859-1---and transmit it as is, hoping that the recipient is using
8859-1 too. Of course, 8859-1 (like ASCII) is a deficient character set,
so it isn't used everywhere, so this strategy has a noticeable failure
rate.

Encoding in RFC 2047 can make the display work for recipients with other
local character sets. However, it screws up the display for recipients
whose software doesn't go to the effort of decoding RFC 2047. For many
users, this is a loss, so many message writers don't do it. (More for
USENET than for email.)

What's happening now is that more and more software is learning to
handle UTF-8. (I've heard a rumor that even sendmail, which has screwed
up bytes 128-159 for years, is finally going to start handling UTF-8.)
At some point, translating to UTF-8 will have a clearly higher success
rate than either 8859-1 or RFC 2047, so message writers will start doing
that. (Sooner for USENET than for email.)

In the long run, UTF-8 will work everywhere. Of course, for a safe
transition, users should stick to ASCII until the UTF-8 support is in
place.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 14 Jan 2003 18:20:47 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On 14 Jan 2003 23:13:19 -0000
"D. J. Bernstein" <[email-masked]> wrote:

> 
> Keith Moore writes:
> > there will still be a need for canonicalization of certain fields
> 
> No. All text is already, at a minimum, C-normalized. Keyboard interfaces
> get it right, and other programs don't randomly switch accent positions.

I don't share your confidence that this happens in every environment,
for every program that generates header fields.

> Anyway, your hypothetical Unicode normalization problems would also
> arise in normalizing RFC 2047, so you can't use normalization as an
> argument against moving to UTF-8.

Which is precisely why 2047 was never intended for anything that needs
to be interpreted by machine.

> Your ``sufficient testing through ordinary usage'' argument is equally
> silly. Testing UTF-8 is much simpler than testing RFC 2047.

Your comparison is silly, because the two solve different problems;
and because for better or worse, the complexity associated with 2047
is going to be around for a long time anyway.

Keith

To: <[email-masked]>, <[email-masked]>
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 15:18:02 -0800
Lines: 17
Sender: [email-masked]

David Barr <[email-masked]> writes:

> If you are right and they are going to continue doing what they are
> doing regardless, then your point is moot (since they wouldn't accept
> either of the proposed routes).

Yes.  That is exactly and precisely my point.  Right now, the existence of
people sending unlabelled 8-bit national character sets on Usenet is being
used in support of UTF-8, which makes no sense to me at all.  Unlabelled
national character sets are not UTF-8.

I believe that what you say above is more likely to be true, namely that
what those folks are doing is basically moot from a standardization
viewpoint since they're doing the one thing that we *can't* standardize.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 14 Jan 2003 23:13:19 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Keith Moore writes:
> there will still be a need for canonicalization of certain fields

No. All text is already, at a minimum, C-normalized. Keyboard interfaces
get it right, and other programs don't randomly switch accent positions.

Anyway, your hypothetical Unicode normalization problems would also
arise in normalizing RFC 2047, so you can't use normalization as an
argument against moving to UTF-8. (The big problem with normalizing RFC
2047, of course, is the horrible mess of character encodings.)

Your ``sufficient testing through ordinary usage'' argument is equally
silly. Testing UTF-8 is much simpler than testing RFC 2047.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 14 Jan 2003 15:10:09 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Folks,

Tuesday, January 14, 2003, 12:36:42 PM, you wrote:
D.> Untagged UTF-8 is good. It is more and more widely supported by readers,
D.> and will eventually be the only format produced by writers. It works.

1.  This is what we heard 10 years ago, too.  If it is a correct
prediction, the small question of "when will it be true" is worth
noting.

2. Predictions of the future that have any serious counter-view have two
consistent characteristics:

           a) they are usually made without offering much or any
           substantiation, and

           b) they are usually wrong.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 14 Jan 2003 15:05:09 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Russ Allbery <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Russ,

Tuesday, January 14, 2003, 1:07:28 PM, you wrote:
Russ> Dave Crocker <[email-masked]> writes:

Russ> I think you're badly misunderstanding my point.  I'm on your side.  :)

damn.  i thought we were working on a really good disagreement.

Russ> My point is that just putting "use UTF-8" in a standard doesn't actually
Russ> solve any problems, nor does it cause people to switch all their
Russ> unlabelled 8-bit content over to UTF-8.

indeed, all this is an exact echo of the 'just use binary' arguments
from 10 years ago, for MIME vs. 8-bit SMTP.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: 14 Jan 2003 22:33:21 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Dave Crocker writes:
> paying attention to the installed base requires worrying about
> compatibility with what has been established practise, not what
> "might" work or what is "frequently" available.

RFC 2047 makes a blatantly incompatible change to the semantics of
various sequences of bytes. It defends this incompatibility by saying

   The syntax of encoded-words is such that they are unlikely to
   "accidentally" appear as normal text in message headers.

Why aren't you screaming about the difference between ``unlikely to''
and ``guaranteed to never''? This isn't a safe extension that merely
assigns semantics to previously prohibited behavior (such as ``8-bit
data is UTF-8''). This is an _incompatible change_ to the semantics of
perfectly _valid_ behavior. Why is that okay? What level of potential
collateral damage are you willing to tolerate?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 14 Jan 2003 22:14:01 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Claus writes:
> It's easy for a user agent to translate the changed data back into
> transmission format.

To the extent that the user sticks to fields recognized by the MUA, such
as Subject, it's certainly possible to do that translation.

The Subject line ends up being created by somebody, encoded as per RFC
2047, decoded for editing in a followup, encoded again, decoded again
for editing in someone else's followup, etc.

The second encoding often has very different bytes from the first. If
the first encoding uses KOI8-R, for example, then it's hardly a surprise
for the second encoding to use UTF-8. But that's a failure when the two
messages are handled by popular message processors that match Subject
lines according to the 822/1036 format.

You could argue that implementations should check for unchanged Subject
lines and use the original encoding if possible (which also means that
implementations have to be structured to keep the original encoding
around instead of hiding RFC 2047 in the message-I/O module). But most
implementations don't do this (even if they're structured that way).

Of course, Subject is only part of the RFC 2047 mess. I noticed that a
recent message to this list has

   In-Reply-To: <001601c2bc0a$2844d650$0200000a@DAVE>
      ("David Barr"'s message of "Tue, 14 Jan 2003 14:18:47 -0600")

   David Barr <[email-masked]> writes:

in the header and body. Does all the reading and writing software work
correctly with an RFC 2047 name instead of David Barr? What happens when
names are copied to and from address books? Where is the line drawn
between RFC 2047 in header fields and UTF-8 in an LDAP database? Exactly
which data structures are supposed to use the RFC 2047 encoding?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

From: "David Barr" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: RE: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Date: Tue, 14 Jan 2003 15:21:47 -0600
Importance: Normal
Sender: [email-masked]

> sympathetic to it.  My point is just that I'd like to see 
> some reason to believe that all the people sending unlabelled 
> 8-bit content are interested in moving to UTF-8.  I don't see 
> much evidence that they're that interested in changing to 
> anything else.

If you are right and they are going to continue doing what they are doing
regardless, then your point is moot (since they wouldn't accept either of
the proposed routes).  Unless you have some scheme that would make what they
are sending legal?

--Dave

To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 13:07:28 -0800
Lines: 29
Sender: [email-masked]

Dave Crocker <[email-masked]> writes:

> Russ,

> Tuesday, January 14, 2003, 11:16:03 AM, you wrote:

> Russ> Given that people have been sending 8-bit data in headers for
> Russ> years without regard to any existing standard, why do you believe
> Russ> that releasing a standard that mandates UTF-8 will cause them to
> Russ> start paying attention to standards again?

> Given that people often drive their cars with no regard for posted speed
> limits, we should just get rid of the speed limits, right?

I think you're badly misunderstanding my point.  I'm on your side.  :)

My point is that just putting "use UTF-8" in a standard doesn't actually
solve any problems, nor does it cause people to switch all their
unlabelled 8-bit content over to UTF-8.  There has to be some additional
reason to believe that UTF-8 is something they want to use.

I understand Dan Bernstein's point as well, and am rather sympathetic to
it.  My point is just that I'd like to see some reason to believe that all
the people sending unlabelled 8-bit content are interested in moving to
UTF-8.  I don't see much evidence that they're that interested in changing
to anything else.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 14 Jan 2003 15:59:46 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> Untagged UTF-8 is good. It is more and more widely supported by
> readers, and will eventually be the only format produced by writers.
> It works.
> 
> In contrast, a mess of multiple character encodings---even with
> tags--- is bad because of its unnecessary complexity. Throw away tags
> and it's a disaster: you can't decode it.

I wish you every success in convincing the government of China that they
shouldn't have their own encoding.

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: Tue, 14 Jan 2003 15:58:28 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

> The reason people disobey the
> 7-bit standard is that the 7-bit standard sucks.

True.  Then again, the obvious alternative also sucks - at least in
the short term.  Just because something sucks doesn't mean it's not
the best way to do something.

I do believe that migrating to utf-8 in message headers is the way
to go in the long term, and that a transition strategy similar to what
Dan is suggesting (support utf-8 on receipt soon, generation of utf-8
later) is ultimately the way to get there from here.  (I suspect we
have very different ideas about what would be a reasonable interval
between those two events)

But there are several things I don't believe.

One, that this will return us to a world where messages are ordinary
text and can be treated as such.  For instance, even if we allow utf-8
in message headers, there will still be a need for canonicalization of
certain fields before sending and/or before comparison of values
embedded in those fields,

Two, that this allows use of utf-8 in addresses or domain names.
Those are separate problems,

Three, that the existing ability of some user agents to display utf-8
in message headers is sufficient for proper processing of headers 
containing utf-8,

Four, that user agents that support utf-8 receipt in the near term will
receive sufficient testing through ordinary usage to be reliable at
supporting it once generation of utf-8 in headers is endorsed.
(so some other testing will be needed - not the way IETF normally works)

Five, that supporting utf-8 rids user agents of the burden
of supporting 2047, IDNA, and similar encodings, at least for
reading/presentation purposes, probably within our lifetimes.

In other words, utf-8 is the right way to go for the long term - 
but the devil is in the details and it's not nearly as simple
as it looks.  And I do think that a two-step transition is necessary -
we can't just start cramming utf-8 into either email or usenet and
expect things to work reasonably.

I also think that usenet can serve as a good test case for email.
(translation: I'd much rather usenet suffer the initial disruption than
email, so email could perhaps learn from usenet's experience.)

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: 14 Jan 2003 20:36:42 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Bruce Lilly writes:
> If you think carefully and objectively about why untagged 8-bit content
> in charsets other than utf-8 is bad, you will realize why untagged utf-8
> is also bad.

Untagged UTF-8 is good. It is more and more widely supported by readers,
and will eventually be the only format produced by writers. It works.

In contrast, a mess of multiple character encodings---even with tags---
is bad because of its unnecessary complexity. Throw away tags and it's
a disaster: you can't decode it.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: <[email-masked]>, <[email-masked]>
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 12:34:23 -0800
Lines: 19
Sender: [email-masked]

David Barr <[email-masked]> writes:

> However, client support for (and active use of) UTF-8 is far more
> pervasive than any of the other proposed encoding schemes.

For Newsgroups, yes.  (Although this is a matter of there being one or two
implementations compared to none, and I'm not sure how significant that
is.)

I believe this is incorrect for everything other than Newsgroups.  In
other words, I believe that you'll find that support for RFC 2047 encoding
in the Subject and From headers is far more common than support for
untagged UTF-8 in those headers.

Both of the examples given of news readers that support untagged UTF-8
also support RFC 2047 for those fields.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 12:31:25 -0800
Lines: 39
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> I agree that 2047 is not suitable for this purpose.  My guess is that
> the IDNA stuff is the best starting point.

IDNA certainly will be the easiest to implement for the servers.  If it
catches on in e-mail, I expect it will also be the easiest to implement
for the clients, since they'll have to implement it anyway for e-mail
addresses.

Note that there is no way to avoid a 7-bit encoding for newsgroup names
without significant changes to the Usenet architecture, since posts to
moderated groups are sent to <group-name>@moderators.isc.org by default
and to some variation on that including the group-name even for
hierarchies that don't use the isc.org forwarding system.  This requires
embedding the newsgroup name in an SMTP envelope, and I don't expect any
8-bit encoding to work reliably enough there in the near future.

Note that several mail systems are involved, including the mail system at
the originating site, the mail system of the various sites around the
world involved in the moderators.isc.org relay service, and the mail
system of the moderator.  Even if we can upgrade the mail systems of the
moderators of non-ASCII groups (likely) and the mail systems of all hosts
volunteering to be part of moderators.isc.org (possible), upgrading the
mail systems of all sites from which posts to a non-ASCII moderated group
may be sent seems pretty intimidating.

Furthermore, it had already been determined that some sort of 7-bit
encoding of the Newsgroups header would be necessary when sending the
messages to the moderators, an encoding that could then be handled by
their software.

Given that a 7-bit encoding is going to be needed *anyway*, I personally
am not particularly fond of the idea of having two separate encodings
floating around, but I can also see the cleanliness argument behind using
pure UTF-8 in the transit system.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 14 Jan 2003 15:29:56 -0500
From: Keith Moore <[email-masked]>
To: "David Barr" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Tue, 14 Jan 2003 14:18:47 -0600
"David Barr" <[email-masked]> wrote:

> 
> [ Sigh.  Ignore my previous email correcting my non-error.  I've been at my
> computer too long today and my eyes are going batty. ]
> 
> > the second does not follow from the first.  or more 
> > specifically, just because someone puts untagged 8bit 
> > characters in a message header does not mean that that person 
> > would support a standard of using utf-8 (or any other 
> > representation other than the one he happened to be using) in 
> > the message header.
> 
> However, client support for (and active use of) UTF-8 is far more > pervasive than any of the other proposed encoding schemes.

not sure about that - after all, the people using other encoding
schemes are presumably finding them satisfactory in their limited 
circumstances.  however I think we'd agree that utf-8 is by far the
most widely supported (by existing user agents) "universal" encoding
scheme.

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: Tue, 14 Jan 2003 12:28:11 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Russ Allbery <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Russ,

Tuesday, January 14, 2003, 11:16:03 AM, you wrote:
Russ> Given that people have been sending 8-bit data in headers for years
Russ> without regard to any existing standard, why do you believe that releasing
Russ> a standard that mandates UTF-8 will cause them to start paying attention
Russ> to standards again?

Given that people often drive their cars with no regard for posted speed
limits, we should just get rid of the speed limits, right?

Oh.  That's right.  There are other consequences to worry about.

It might be useful for "just do utf-8" proponents to start paying attention
to the citations of other consequences.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: 14 Jan 2003 20:26:53 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Russ Allbery writes:
> Given that people have been sending 8-bit data in headers for years
> without regard to any existing standard, why do you believe that
> releasing a standard that mandates UTF-8 will cause them to start
> paying attention to standards again?

People disobey the 7-bit standard. You leap to the conclusion that they
are, as a rule, not paying attention to standards, and therefore won't
pay attention to the next standard. Wrong. The reason people disobey the
7-bit standard is that the 7-bit standard sucks.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 14 Jan 2003 15:23:20 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> Hey, Keith, have you ever heard of ``AXFR clarify''?

> Why aren't you yelling ``Thou shalt not break compatibility with the
> current-IETF-standards-conforming installed base'' in that case?

there's too much brain damage for any one person to deal with.
don't you agree?

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: Tue, 14 Jan 2003 15:22:06 -0500
From: Keith Moore <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

> Using RFC 2047 for the Newsgroups header is probably unworkable for a
> variety of reasons,

I agree that 2047 is not suitable for this purpose.  My guess is that
the IDNA stuff is the best starting point.
--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

From: "David Barr" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: RE: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Date: Tue, 14 Jan 2003 14:18:47 -0600
Importance: Normal
Sender: [email-masked]

[ Sigh.  Ignore my previous email correcting my non-error.  I've been at my
computer too long today and my eyes are going batty. ]

> the second does not follow from the first.  or more 
> specifically, just because someone puts untagged 8bit 
> characters in a message header does not mean that that person 
> would support a standard of using utf-8 (or any other 
> representation other than the one he happened to be using) in 
> the message header.

However, client support for (and active use of) UTF-8 is far more pervasive
than any of the other proposed encoding schemes.

--Dave

Date: Tue, 14 Jan 2003 15:09:41 -0500
From: Keith Moore <[email-masked]>
To: "David Barr" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Tue, 14 Jan 2003 13:56:16 -0600
"David Barr" <[email-masked]> wrote:

> 
> > Given that people have been sending 8-bit data in headers for 
> > years without regard to any existing standard, why do you 
> > believe that releasing a standard that mandates UTF-8 will 
> > cause them to start paying attention to standards again?
> 
> It doesn't take much common sense to see those people putting 8-bit data in
> headers are precisely the ones who think a) 8-bits in headers work (for
> their purposes) and thus b) would be most likely to adopt an 8-bit standard.

the second does not follow from the first.  or more specifically,
just because someone puts untagged 8bit characters in a message
header does not mean that that person would support a standard
of using utf-8 (or any other representation other than the one he
happened to be using) in the message header.

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 12:07:23 -0800
Lines: 30
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:
> Brett Kottmann <[email-masked]> writes:

>> I've seen the discussions go on for some time...is it just me or does
>> anyone else think that stating "the format of a news message and header
>> shall conform to the current standard for email message bodies and
>> headers" would save a great deal of time?

> Yes, it does.  It saves a vast amount of time, which is why almost
> exactly that phrasing was used in RFC 1036, the previous standard for
> news messages.

> The attempted removal of that phrase from the current USEFOR draft has,
> by itself, probably resulted in something on the order of two years of
> argument and debate.

It occurs to me that I failed to mention something important here, which
may not be clear from the discussions so far.  The reason why this topic
came up in the first place is that there is demand for non-ASCII newsgroup
names, and when attempting to find a way to deal with that, one simply
cannot adopt what mail did, since mail has no concept of newsgroup names.

Using RFC 2047 for the Newsgroups header is probably unworkable for a
variety of reasons, most notably being that RFC 2047 is not a unique
encoding format (there are multiple ways to encode the same word), and all
existing Usenet software requires that a given newsgroup have one and only
one textual representation.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 14 Jan 2003 20:05:30 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Keith ``7 bits forever!'' Moore writes:
> the religious issue is 'thou shall not break compatibility with the 
> current-IETF-standards-conforming installed base'.  and it's a damn
> fine religion, if you ask me.

Hey, Keith, have you ever heard of ``AXFR clarify''?

It's an effort by the BIND company to force everybody to imitate BIND 9.
They admit that their draft prohibits the conforming behavior of most
existing DNS server installations---including all BIND 8 installations!
http://cr.yp.to/djbdns/axfr-clarify.html has details.

Why aren't you yelling ``Thou shalt not break compatibility with the
current-IETF-standards-conforming installed base'' in that case?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 12:04:58 -0800
Lines: 42
Sender: [email-masked]

David Barr <[email-masked]> writes:

> It doesn't take much common sense to see those people putting 8-bit data
> in headers are precisely the ones who think a) 8-bits in headers work
> (for their purposes)

Yes.

> and thus b) would be most likely to adopt an 8-bit standard.

I don't see how this necessarily follows.  In fact, I believe that the
people who are currently using only ASCII are more likely to adopt a UTF-8
standard than the people who are currently sending 8-bit data in headers,
which is the exactly opposite of your point (b).

> Most importantly they would likely be the *least* likely to accept going
> back to 7 bits and encoding if they already have experience that 8-bits
> work.

This is probably true, yes.

> Given the history of how many standards have evolved in the past, I
> can't much reason to agree with you.  Protocols are *full* of examples
> of this sort of evolution, whereby technically illegal (but largely
> working) practices evolved into an accepted form.

Yes.  Done by standardizing what people are already doing.

Standardizing UTF-8 in headers is not standardizing what people are
already doing.  There are considerably fewer people using UTF-8 in headers
than there are people who are using RFC 2047.

If you were talking about standardizing sending local 8-bit character sets
in headers, I could see how this argument would apply, but given that
we're talking about going from one workable but somewhat unpopular
standard to a brand-new standard that bears no resemblence to what people
are doing now, I don't believe the "evolution of standards by
standardizing technically invalid but existing practice" argument applies
at all.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: "David Barr" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: RE: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Date: Tue, 14 Jan 2003 14:03:04 -0600
Importance: Normal
Sender: [email-masked]

> It doesn't take much common sense to see those people putting 
> 8-bit data in headers are precisely the ones who think a) 
> 8-bits in headers work (for their purposes) and thus b) would 
> be most likely to adopt an 8-bit standard. Most importantly 
                             ^^^^^

Oops, that should of course read "encoding" (or 7-bit)

--Dave

From: "David Barr" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: RE: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Date: Tue, 14 Jan 2003 13:56:16 -0600
Importance: Normal
Sender: [email-masked]

> Given that people have been sending 8-bit data in headers for 
> years without regard to any existing standard, why do you 
> believe that releasing a standard that mandates UTF-8 will 
> cause them to start paying attention to standards again?

It doesn't take much common sense to see those people putting 8-bit data in
headers are precisely the ones who think a) 8-bits in headers work (for
their purposes) and thus b) would be most likely to adopt an 8-bit standard.
Most importantly they would likely be the *least* likely to accept going
back to 7 bits and encoding if they already have experience that 8-bits
work.

> I must say that I don't find the argument "because I 
> personally like this new standard better than the one that we 
> have now" particularly persuasive when it comes to predicting 
> the behavior of anyone other than you personally.

Given the history of how many standards have evolved in the past, I can't
much reason to agree with you.  Protocols are *full* of examples of this
sort of evolution, whereby technically illegal (but largely working)
practices evolved into an accepted form.  Even Usenet.  For example, the
now-obsolete 14-character newsgroup component limit.  Or the leading-digit
prohibition in early DNS.  People did illegal but common sense things, and
eventually the standards caught up to them.  Once they did, they adopted the
new more sensible stanards.

--Dave

To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 14 Jan 2003 11:16:03 -0800
Lines: 19
Sender: [email-masked]

Shmuel (Seymour J ) Metz <[email-masked]> writes:

> The jinni is already out of the body; people have been sending 8-bit
> data in headers for years. They have had good reasons for doing so, even
> though they broke interoperability. Mandating UTF-8 would help to
> restore interoperability.

Given that people have been sending 8-bit data in headers for years
without regard to any existing standard, why do you believe that releasing
a standard that mandates UTF-8 will cause them to start paying attention
to standards again?

I must say that I don't find the argument "because I personally like this
new standard better than the one that we have now" particularly
persuasive when it comes to predicting the behavior of anyone other than
you personally.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 14 Jan 2003 12:46:01 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> You claimed (and your words are still quoted above, preceded by ">>>>>")
> that there was a design flaw in Usefor which involved "decoding
> encoded-words for transmission". Will you please explain exactly where in
> Usefor it is specified that any encoded-word is required (or even allowed)
> to be "decoded for transmission".

Any raw untagged utf-8 in any header field is an example (that is exactly
what one would get if the properly tagged and encoded form which is legal
in header fields were decoded).  When such untagged 8-bit data (in general,
utf-8 in particular) is sent by a UA to a gateway (which is subsequently
expected to magically [re-]encode it), that is what I mean by "decoding
encoded-words for transmission".  As has been pointed out, UAs need to
handle 2047/2231 for header fields in any event, so permitting (even
encouraging) UAs to instead transmit the "decoded" untagged raw 8-bit
form for transmission amounts to the same thing as decoding. And that
is what the Usefor draft as it is currently written does.

Date: Tue, 14 Jan 2003 09:30:09 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Charles Lindsey" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles,

Tuesday, January 14, 2003, 4:24:25 AM, you wrote:
Charles> So you want to apply the word "decoded" to something that was never
Charles> encoded in the first place? OK, I'll play.

UTF-8 is an encoding of a larger bit-space.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 14 Jan 2003 12:24:47 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: [email-masked]
Subject: Re: Historical trivia
Sender: [email-masked]

Dave Crocker wrote:
> (I'd guess that GDT means Greenwhich Daylight Time, but
> suspect there really is no such thing.)

There's a detailed list of time zone data at ftp://elsie.nci.nih.gov/pub.
But no mention of a GDT.  I suspect that it may have been included for
symmetry with the others (NST, introduced in 733, has no D counterpart
but wasn't in 724 or 561).

By the way, the reason for my questions, aside from historical interest
in 680, is that I have a parser which currently handles the text message
format from 822 forward, and I'm considering implementing backward
compatibility with the earlier standards.  Some parts would probably be
quite difficult to integrate with the current ones (so it's good news that
some of the 733 features were never used -- I can safely ignore them).

To: [email-masked]
Xref: clerew local.mime:2211
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Tue, 14 Jan 2003 12:24:25 GMT
Lines: 55
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>>>>That is
>>>>>clearly the intent of RFC 2047.  The design flaw, as has been already
>>>>>mentioned, is decoding encoded-words for transmission.
>>>>
>>>>
>>>>But nobody is proposing to do that.
>> 
>> 
>>>The Usefor draft permits it by allowing raw untagged 8-bit data to be
>>>transmitted between generating user agents and gateways.
>> 
>> 
>> If some raw untagged 8-bit data is produced then, by definition, it is not
>> in the form of an encoded-word,

>I.e. it is in decoded form.  Header fields must use 2047/2231 for non-ASCII
>characters; any raw untagged 8-bit data is therefore referred to as "decoded"
>for purposes of this discussion.

So you want to apply the word "decoded" to something that was never
encoded in the first place? OK, I'll play.

>> so the question of "decoding it for
>> transmission" cannot possibly arise.

>Yes it can; if a user agent transmits raw untagged 8-bit data (rather than
>the standard 2047/2231 encoded-words).

But now you are saying that something transmitted in "decoded" form must
necessarily have been through a "decoding" process beforehand. Sorry, you
can't have it both ways.

You claimed (and your words are still quoted above, preceded by ">>>>>")
that there was a design flaw in Usefor which involved "decoding
encoded-words for transmission". Will you please explain exactly where in
Usefor it is specified that any encoded-word is required (or even allowed)
to be "decoded for transmission".

You will not be able, because it simply is not there.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 14 Jan 2003 08:57:38 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Charles Lindsey" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles,

Tuesday, January 14, 2003, 4:10:21 AM, you wrote:
Charles> The issue being discussed is whether implementations exist that will
Charles> handle Netnews articles as proposed in the Usefor draft.

And therein lies the exact problem.  You have missed was is really being
discussed by the folks pointing out problems with "just do UTF-8".

The issue is not whether there is an existence proof for the practise you
want to standardize.  The issue is, instead, the impact of a new standard on
the installed base that uses the old standard.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 14 Jan 2003 08:55:39 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Charles Lindsey" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles,

Tuesday, January 14, 2003, 3:47:22 AM, you wrote:
Charles> Currently, the email standards require that 'thou shalt have no gods other
Charles> than US-ASCII' (and that position does seem to be being defended in the
Charles> manner of a religious issue).

You are probably right, but you are not understanding the religion.

The religion is called "protect the installed base and do upgrades at the
edges rather than the infrastructure whenever possible".

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 14 Jan 2003 08:53:59 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Bruce Lilly <[email-masked]>
CC: [email-masked]
Subject: Re: Historical trivia
Sender: [email-masked]

Bruce,

Tuesday, January 14, 2003, 6:03:20 AM, you wrote:

Bruce> 2. RFCs 561 and 724 provided for an alphabetic zone "GDT", which
Bruce>     was dropped in 733.  But neither of the earlier RFCs gives the
Bruce>     semantics for GDT. What were they?

As I recall, the formulation of timezones for RFC 733 did not take much note
of the previous documents. We simply decided which timezones we wanted to
refer to by "name". Offsets (civilian or military) would be used for the
rest. So we decided on all North American timezones for names, along with
Greewhich Mean Time. (I'd guess that GDT means Greenwhich Daylight Time, but
suspect there really is no such thing.)

"North America" included the far east of Canada as well as Alaska and
Hawaii. Alaska swings *very* far west. None of us knew the names of their
zones. It turns out the next zone east of EST is Atlantic Standard Time. I
think I found that in a dictionary, and maybe also the Newfoundland zone.

The western extremes were more challenging.  I could not find any reference
to their names.  I was working at The Rand Corporation at the time.  It was
famous for "inventing" the technical research library and it had a reference
librarian who was legendary for finding difficult answers.

So I gave her a call and asked for her help.  She called back in 10 minutes
with the answer.  I could not believe how quickly she solved this, so I
asked how she came up with the answers.

She said she called the telephone information operators in the relevant
areas and asked them what they called their timezones...

Bruce> 3. 724 and 733 provided for nested lists of addresses, either
Bruce>     angle-bracketed or in groups -- these were dropped in 822.
Bruce>     Were they ever used?

No.  That's why we dropped it.  Offhand, the portions of 733 that were
innovative -- rather than the portions that simply tried to codify existing
practise -- had a mixed track record of actual adoption.

Bruce> 4. Were :atom: address forms other than :Include: and :Postal:
Bruce>     ever defined?

No.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 14 Jan 2003 11:53:49 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
>>
>>>Not so. Untagged data in "any other" charset would be non-compliant with
>>>the standard,
> 
> 
>>Clearly, that is being handled as a religious issue; 'thou shalt have no
>>gods other than utf-8', and would delare those other charsets 'heretical'.
>>If you think carefully and objectively about why untagged 8-bit content
>>in charsets other than utf-8 is bad, you will realize why untagged utf-8
>>is also bad.  But fundamentalist religious zealots never think objectively
>>about their beliefs...
> 
> 
> Currently, the email standards require that 'thou shalt have no gods other
> than US-ASCII' (and that position does seem to be being defended in the
> manner of a religious issue).

That is an inaccurate characterization of (a) the standards, and (b) the
positions being taken.  The standards specify *a subset* of us-ascii due
to a number of historical reasons, including the basis of initial electronic
mail (prior to SMTP) on FTP and TELNET protocols, network limitations, and
some charset conversion issues.  MIME extended that to provide for
representation of the remainder of us-ascii as well as other charsets by
the means which is currently defined in RFC 2047 as amended by RFC 2231
and errata, and which has evolved to include language-tagging capability
as well.  With the exception of providing for us-ascii as a *default*, all
charsets are treated equally. If a default were not provided, one would
need to tag everything. US-ASCII as a default is both consistent with
historical practice (arising out of the origins of the Internet as a US
Department of Defense Advanced Research Projects Agency effort) and as a
common subset of many other charsets (including most of the ISO-8859
series).  In some cases, even with the default, it is necessary to use
tagging, e.g. for =?us-ascii*en?q?boot?=. There is a great deal of
software, widely implemented and interoperating, based on the standards.
The position regarding the standards can be summed up as:
1. compatibility with those earlier standards is critical to maintaining
    interoperability with good-faith existing implementations compliant
    with those standards -- injecting illegal data (w.r.t. existing
    standards) is not conforming.
2. compatibility and interoperability can only be assured by proper
    formulation of a standard. They can not be achieved by a "it sorta
    kinda works (maybe) some of the time" approach.
3. compatibility with existing standards means *true* compatibility,
    not "I'll ignore the part about header content in the MIME-part
    headers of message and multipart composite media types", for example.
4. compatibility and interoperability considerations require looking at
    the "big picture", not just an isolated niche. In this case, that
    means considering gateways to email, which may subsequently hit list
    expanders, travel over diverse networks, etc., and IMAP, which has
    specific features for news as well as email.
5. outright violation of existing draft standards is simply unacceptable
    (whether bald-faced or via a "back-door" approach).
6. a single scheme (e.g. use 2047/2231) is preferable to multiple choice
    in standards
7. one cannot avoid encoding; utf-8 is yet another encoding
8. language tagging is important in many contexts (RFC 2277, sect. 4)
9. scalable interoperability is more important than compatibility with
    existing practices which are non-conformant with existing standards
10. RFC 2047 encoding is intended for items used for human (not machine)
    interpretation
11. existence of a single implementation that handles some input X is
    uninteresting -- what is important (w.r.t. compatibility and
    interoperability) is what happens to implementations which conform
    to existing standards when presented with X
12. "elegance" and "aesthetics" are subordinate to interoperabilty

Exclusive worship of the untagged utf-8 god requires sacrifing
compatibility, interoperability, existing conforming implementations,
scalability, internationalization (language-tagging), and simplicity
of software implementation.  Conversely, observing the principles
expounded above ensures those attributes, but prevents neither free
worship of any of the charset gods nor charset agnosticism.

 > Expanding that, within Netnews, to 'thou
> shalt have no gods other than utf-8' is clearly a viable option,

Bad premise -> bad conclusion. GIGO.

 > whereas
> saying 'thou mayst have any untagged gods thou likest' is not viable
 > because there is no way to tell which god you are invoking, which is going
 > to cause some considerable confusion on the top of Mount Olympus.

On that we can agree.  But I see no problem with permitting any *tagged*
charset, while there are problems with *any* untagged charset (unless
that happens to be a common denominator which is used in  the current
standards (and therefore has no backwards-compatibility issues), such
as the subset of us-ascii called netascii).

> Indeed, some GB18030 heretics may try to get away with it, but at least it
> is easy to detect those heresies when they occur.

I have no objection to properly-tagged use of GB18030 -- it's in the
official IANA charset list.  Ditto for properly-tagged utf-8.

From: "Shmuel (Seymour J.) Metz" <[email-masked]>
Date: Tue, 14 Jan 2003 10:05:13 -0500
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Organization: Atid/2
Sender: [email-masked]

In <[email-masked]>, on 01/11/2003
   at 04:24 PM, Dave Crocker <[email-masked]> said:

>7.  breaking standards is not measured by whether someone's code
>core-dumps. it is measured by whether it violates the specification. 
>sending 8-bit data in a 7-bit environment breaks the specification. 

The jinni is already out of the body; people have been sending 8-bit
data in headers for years. They have had good reasons for doing so,
even though they broke interoperability. Mandating UTF-8 would help to
restore interoperability.
 
-- 
-----------------------------------------------------------
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2
     Team OS/2
     Team PL/I
-----------------------------------------------------------

Date: Tue, 14 Jan 2003 09:23:40 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
>>
>>>In <[email-masked]> Bruce Lilly <[email-masked]> writes:
>>>
>>>
>>>
>>>>....  Drop the raw utf-8 nonsense and move on to somthing that has
>>>>a non-zero chance of baing (a) standardized and (b) implemented.
>>>
>>>
>>>The chances of getting it standardized may or may not be zero. But the
>>>chances of it being implemented are certainbly non-zero, because
>>>implementations already exist.

> No, for the comparatively minor problem of gateways from particular
> newsgroups to mailing lists, implementations do not yet exist

OK, I'll take that as a retraction of your earlier claim of existing
implementations.

 > The problems are certainly
> soluble

No, as currently formulated in the Usefor draft, they cannot be (except
of course by addressing the root problem of untagged raw 8-bit header
field content, which simultaneously addresses backwards compatibility
(with good-faith IMAP 1036-compliant implementations as well as
existing gateways and other software, such as used by moderators) and
interoperability issues with email (as required for moderation)).
Simply requiring user agents to continue to comply with the existing
standard (RFC 1036), i.e. use RFC 2047 / 2231 for non-ASCII and/or
language-tagged header field data, will solve most of the problems in
the current Usefor draft.

> but corners will have to be cut. The question is which corners? I
> have suggested some possibilities, but you seem unwilling to discuss them.

Those horses have been flogged to death.  Gateways cannot pluck charset
and language information from thin air; the only viable solution is as
described above -- viz. to continue to require compliance with RFC 822
as extended by MIME.

Date: Tue, 14 Jan 2003 09:03:20 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Historical trivia
Sender: [email-masked]

I have a few questions of an historical nature:

1. The rfc-index states (correctly) that RFC 680 (between 561
    and 724 in the chain leading up to 2822) is not online. If
    somebody has a copy of 680 that they would be willing to
    (a) scan and email
    or (b) fax
    or (c) photocopy and mail
    to me, I'll do what I can to get it into suitable electronic
    form.  Please email me off-list to discuss.

2. RFCs 561 and 724 provided for an alphabetic zone "GDT", which
    was dropped in 733.  But neither of the earlier RFCs gives the
    semantics for GDT. What were they?

3. 724 and 733 provided for nested lists of addresses, either
    angle-bracketed or in groups -- these were dropped in 822.
    Were they ever used?

4. Were :atom: address forms other than :Include: and :Postal:
    ever defined?

Date: Tue, 14 Jan 2003 08:45:03 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> Currently, the email standards require that 'thou shalt have no gods other
> than US-ASCII' (and that position does seem to be being defended in the
> manner of a religious issue). 

the religious issue is 'thou shall not break compatibility with the 
current-IETF-standards-conforming installed base'.  and it's a damn
fine religion, if you ask me.

To: [email-masked]
Xref: clerew local.mime:2210
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Tue, 14 Jan 2003 12:10:21 GMT
Lines: 39
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
>> 
>> 
>>>....  Drop the raw utf-8 nonsense and move on to somthing that has
>>>a non-zero chance of baing (a) standardized and (b) implemented.
>> 
>> 
>> The chances of getting it standardized may or may not be zero. But the
>> chances of it being implemented are certainbly non-zero, because
>> implementations already exist.

>Please do keep in mind the situation being discussed.

The issue being discussed is whether implementations exist that will
handle Netnews articles as proposed in the Usefor draft. So far as User
Agents are concerned, yes they do. In fact, it appears that the latest
Mozilla can even distinguish automatically between genuine UTF-8 and other
8bit charsets, which both surprises and pleases me.

No, for the comparatively minor problem of gateways from particular
newsgroups to mailing lists, implementations do not yet exist (Rome wasn't
built in a day), which is why we are discussing the best way for those
gateways to handle some marginal cases. The problems are certainly
soluble, but corners will have to be cut. The question is which corners? I
have suggested some possibilities, but you seem unwilling to discuss them.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2209
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Tue, 14 Jan 2003 11:47:22 GMT
Lines: 35
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> 
>> Not so. Untagged data in "any other" charset would be non-compliant with
>> the standard,

>Clearly, that is being handled as a religious issue; 'thou shalt have no
>gods other than utf-8', and would delare those other charsets 'heretical'.
>If you think carefully and objectively about why untagged 8-bit content
>in charsets other than utf-8 is bad, you will realize why untagged utf-8
>is also bad.  But fundamentalist religious zealots never think objectively
>about their beliefs...

Currently, the email standards require that 'thou shalt have no gods other
than US-ASCII' (and that position does seem to be being defended in the
manner of a religious issue). Expanding that, within Netnews, to 'thou
shalt have no gods other than utf-8' is clearly a viable option, whereas
saying 'thou mayst have any untagged gods thou likest' is not viable
because there is no way to tell which god you are invoking, which is going
to cause some considerable confusion on the top of Mount Olympus.

Indeed, some GB18030 heretics may try to get away with it, but at least it
is easy to detect those heresies when they occur.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Subject: Re: UTF-8 over RFC 2047
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 13 Jan 2003 17:31:09 -0800
Lines: 30
Sender: [email-masked]

Brett Kottmann <[email-masked]> writes:

> Is there any (serious) disadvantage to relying on the format of email
> messages as the template for news messages?

That pretty much cuts to the heart of the matter, doesn't it?

The people who strongly dislike RFC 2047 and want to use UTF-8 in headers
without additional encoding would say yes, and that RFC 2047 is
unnecessarily complex.  The people who figure that one has to deal with
RFC 2047 anyway and that it's not likely to go away any time soon would
say no, nothing is really gained.

You're seeing extended posts from both sides of that question.

> I've seen the discussions go on for some time...is it just me or does
> anyone else think that stating "the format of a news message and header
> shall conform to the current standard for email message bodies and
> headers" would save a great deal of time?

Yes, it does.  It saves a vast amount of time, which is why almost exactly
that phrasing was used in RFC 1036, the previous standard for news
messages.

The attempted removal of that phrase from the current USEFOR draft has, by
itself, probably resulted in something on the order of two years of
argument and debate.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Mon, 13 Jan 2003 18:56:19 -0500
Subject: Re: UTF-8 over RFC 2047
From: Brett Kottmann <[email-masked]>
To: <[email-masked]>
Sender: [email-masked]

Is there any (serious) disadvantage to relying on the format of email
messages as the template for news messages?

I've seen the discussions go on for some time...is it just me or does anyone
else think that stating "the format of a news message and header shall
conform to the current standard for email message bodies and headers" would
save a great deal of time?

I think we may be spending a great deal of time on a format that will go the
way of gopher unless we make it as "web" and email compliant as possible.
 
Brett Kottmann
bkottmann(at)webteamone.com
Visit the Sac and Fox global community:
http://groups.yahoo.com/group/sac_and_fox   

Date: Mon, 13 Jan 2003 18:13:03 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked], Usefor Mailing List <[email-masked]>
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

On Mon, 13 Jan 2003, Bruce Lilly wrote:
> > B News predates RFC1036.
> 
> And B news was still in use many years after RFC 1036 was
> published (Dec. 1987).

B News use probably peaked around the time 1036 was published.  It was
already in decline by the time of the first data I have (about a year
later).  After the first polished C News release in 1989, B News use
dwindled rather spectacularly; a lot of people had been hurting from its
limitations.  It had only a small share of Usenet by 1992, when INN
delivered the coup de grace.  That was about the time when we stopped
paying much attention to B News compatibility as an issue in further
C News development. 

> It was certainly still in use in the mid-90s.

In minor backwaters of the net only.  (Even A News was still in use, here
and there, into the 90s.)  When O'Reilly and I agreed on the first outline
for what eventually became "Managing Usenet", in summer 1993, B News was
mentioned only in the history chapter; Usenet was overwhelmingly dominated
by C News and INN then.

Nobody in the mid-90s was concerned about B News compatibility any more. 
As a constraint on (e.g.) 8-bit content, it was years dead by then.

                                                          Henry Spencer
                                                       [email-masked]

Date: Mon, 13 Jan 2003 17:18:42 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Jean-Marc Desperrier <[email-masked]>
CC: Ralph Babel <[email-masked]>, [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Jean-Marc Desperrier wrote:
> Ralph Babel a dit :
> 
>> Dan Kohn wrote:
>>  
>>
>>> Jean-Marc Desperrier wrote:
>>>   
>>>
>>>> The big difference is that Usenet transport has
>>>> almost since the very start been fully 8 bit.
>>>>     
>>
>> B News didn't handle 8-bit data even in the body.
>>
>> See toascii(), isprint() etc. in inews.c.
>>  
>>
> "almost since the very start". This wording was excluding B News.
> 
> B News predates RFC1036.

And B news was still in use many years after RFC 1036 was
published (Dec. 1987).  It was certainly still in use in
the mid-90s.  According to my recollection of Usenet history,
it started ca. 1980.  That means that there have been 8-bit
issues for more than 70% of the time that Usenet has been
around, which means that the "almost since the very start"
statement is clearly false.

Date: Mon, 13 Jan 2003 22:00:01 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: Ralph Babel <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047
Sender: [email-masked]

Ralph Babel a dit :

>Dan Kohn wrote:
>  
>
>>Jean-Marc Desperrier wrote:
>>    
>>
>>>The big difference is that Usenet transport has
>>>almost since the very start been fully 8 bit.
>>>      
>>>
>B News didn't handle 8-bit data even in the body.
>
>See toascii(), isprint() etc. in inews.c.
>  
>
"almost since the very start". This wording was excluding B News.

B News predates RFC1036.

Date: Mon, 13 Jan 2003 09:35:04 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> If you see the header
> 	Foorbar: XXX (=?iso8859-1?q?=A3-=A3?=) YYY
> then if it is a structured header you are supposed to decode the
> encoded-word, and if it is an unstructured header you are not supposed to
> decode it.
> 
> If you don't know whether it is structured or unstructured then you are,
> officially, stuck.

but it only matters for the purpose of display.  if you're using an encoded-word
for anything other than purely human-readable text, you're not using them 
properly.

so for instance use of encoded-words for newsgroup names is an improper use.

so if an implementation doesn't recognize a field and treats it as unstructured,
the worst that will happen is that it won't decode some encoded words before
display (because it won't recognize them next to some 'special' characters,
or it won't recognize that parentheses are comment delimeters).

and outside of to/cc/from/bcc/reply-to, few structured fields contain human
readable text anyway.

Keith

Date: Mon, 13 Jan 2003 08:16:37 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: prevervation of installed base
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>You're talking about gateways to email -- what "transitional phase" -- what
>>are you assuming will change in the email protocols?
> 
> 
> No, I am talking primarily about gateways *from* news.

Yes, from news and to ... what?  If not "to email", as I said, then why
are you discussing it here?  If in fact we *are* still talking about
news->mail gateways, then please answer the question.

Date: Mon, 13 Jan 2003 08:11:07 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> I think you will find that, in the countries where that spam originates
> from, users are regularly using untagged non-ASCII in both headers and
> bodies when communicating with each other. I have no way of verifying
> whether that is true or not in email, but it is most certainly true in the
> case of Usenet (you just need to look at the cn.* groups).

And your point is..?

If it is not "spammers use untagged non-ASCII, so it's OK for Usenet",
then what precisely is it?

Date: Mon, 13 Jan 2003 08:07:20 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
> 
> 
>>>But it doesn't. Unicode 3.2 is the current standard, and defines precisely
>>>what is allowed in UTF-8. All earlier versions are obsolete.
> 
> 
>>So overnight all of the MS Windows 95, 98, NT, and 2000 implementations
>>mysteriously vanished?
> 
> 
> That is Bill Gates' problem. However, I doubt any of them ever generated
> the deprecated (and now illegal) forms of UTF-8.

Please use a little common sense; Unicode 3.1 was introduced in mid-2001.
All of the aforementioned "operating" systems predate that (and many are
still in widespread use).

Date: Mon, 13 Jan 2003 08:04:17 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
>>
>>>But for some purposes (followups) the header gets used as the basis of
>>>headers in a new message, and at that point the starting point has to be
>>>the decoded form (since how else is the user to edit the new message).
> 
> 
>>I don't follow your argument; could you please explain the circumstances
>>under which some individual would need to edit an encoded-word which is
>>part of another person's display name in a header field.  There is no
>>need to decode and re-encode in order to use a From field content to generate
>>a new To, Cc, etc. field.
> 
> 
> The situation is going to arise most commonly within Subject-headers,
> which the user will most certainly expect to be able to edit.

Here there are two issues:
1. "editing" dealt with below
2. the "Re: " hack, which effectively makes Subject structured to some
    extent.

> It will need to be copied into To or Cc fields if the user wants to email
> his followup as well as to post it. I expect most user agents will expect
> the user to edit his To or Cc fields too.

"Editing" doesn't necessarily mean inserting and removing random characters
at random places.  One might want to insert or remove entire mailboxes from
a list of mailboxes, but that does not require decoding and re-encoding.

>>>>That is
>>>>clearly the intent of RFC 2047.  The design flaw, as has been already
>>>>mentioned, is decoding encoded-words for transmission.
>>>
>>>
>>>But nobody is proposing to do that.
> 
> 
>>The Usefor draft permits it by allowing raw untagged 8-bit data to be
>>transmitted between generating user agents and gateways.
> 
> 
> If some raw untagged 8-bit data is produced then, by definition, it is not
> in the form of an encoded-word,

I.e. it is in decoded form.  Header fields must use 2047/2231 for non-ASCII
characters; any raw untagged 8-bit data is therefore referred to as "decoded"
for purposes of this discussion.

> so the question of "decoding it for
> transmission" cannot possibly arise.

Yes it can; if a user agent transmits raw untagged 8-bit data (rather than
the standard 2047/2231 encoded-words).

 > If, at some gateway, it gets encoded,

Can't be done there. Encoding requires charset and (optionally) language
information plus header field syntax information.  Charset and language
information isn't available to gateways, and syntax information might not
be available.

> then it is now in RFC 2047 form (hopefully correct), and then it will
> remain so

That is why the correct form (2047/2231) must be produced by the user
agent (which does have the necessary information).

Date: Mon, 13 Jan 2003 07:21:49 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>....  Drop the raw utf-8 nonsense and move on to somthing that has
>>a non-zero chance of baing (a) standardized and (b) implemented.
> 
> 
> The chances of getting it standardized may or may not be zero. But the
> chances of it being implemented are certainbly non-zero, because
> implementations already exist.

Please do keep in mind the situation being discussed.  Now can you
provide examples of implementations of general-purpose news->mail
gateways which will always produce legal, properly charset- and
language-tagged header fields, including experimental fields, from
input with a variety of untagged 8-bit charsets (as currently abounds
on Usenet)?  Of course you cannot, because it is impossible.

Date: Mon, 13 Jan 2003 07:14:58 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>That would only work if it could be absolutely guaranteed that no untagged
>>data in any other character set might appear.  No such guarantee is possible,
>>and in fact Usenet abounds with untagged charsets, of which, according to
>>Andrew Gierth, "no significant amount" is utf-8.
> 
> 
> Not so. Untagged data in "any other" charset would be non-compliant with
> the standard,

Clearly, that is being handled as a religious issue; 'thou shalt have no
gods other than utf-8', and would delare those other charsets 'heretical'.
If you think carefully and objectively about why untagged 8-bit content
in charsets other than utf-8 is bad, you will realize why untagged utf-8
is also bad.  But fundamentalist religious zealots never think objectively
about their beliefs...

 > Usefor does provide a _suggested_ way out which, if followed,
> will result in nothing worse that the present unsatisfactory state of
> affairs).

I.e. it will remain an unsatisfactory state of affairs.  Unsatisfactory
because of lack of compatibility with the installed base which has remained
compliant with the RFC 1036 article format.

> Having said that, if someone can propose a tagging scheme for such headers
> (and that might well include language tags), then there is no reason why
> Usefor could not consider it.

It's in RFC 2047 as amended by RFC 2231 and errata.  That will work for
everything except newsgroup component tags, for which several workable,
compatible methods have been proposed.

To: [email-masked]
Xref: clerew local.mime:2181
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 17:42:30 GMT
Lines: 54
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> 
>> But for some purposes (followups) the header gets used as the basis of
>> headers in a new message, and at that point the starting point has to be
>> the decoded form (since how else is the user to edit the new message).

>I don't follow your argument; could you please explain the circumstances
>under which some individual would need to edit an encoded-word which is
>part of another person's display name in a header field.  There is no
>need to decode and re-encode in order to use a From field content to generate
>a new To, Cc, etc. field.

The situation is going to arise most commonly within Subject-headers,
which the user will most certainly expect to be able to edit.

It will need to be copied into To or Cc fields if the user wants to email
his followup as well as to post it. I expect most user agents will expect
the user to edit his To or Cc fields too. A clever agent might well spot
that the field was unchanged since it was taken from an earlier
From/Reply-To, and thus preserve the original 2047 encoding. But I rather
doubt that agents are going to be that clever in practice. So they will
re-encode it.

>>> That is
>>>clearly the intent of RFC 2047.  The design flaw, as has been already
>>>mentioned, is decoding encoded-words for transmission.
>> 
>> 
>> But nobody is proposing to do that.

>The Usefor draft permits it by allowing raw untagged 8-bit data to be
>transmitted between generating user agents and gateways.

If some raw untagged 8-bit data is produced then, by definition, it is not
in the form of an encoded-word, so the question of "decoding it for
transmission" cannot possibly arise. If, at some gateway, it gets encoded,
then it is now in RFC 2047 form (hopefully correct), and then it will
remain so, so the question of "decoding it for transmission" still cannot
arise. Hence your claim, as quoted above, is plain wrong.

As I said.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2182
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 18:03:56 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> But it doesn't. Unicode 3.2 is the current standard, and defines precisely
>> what is allowed in UTF-8. All earlier versions are obsolete.

>So overnight all of the MS Windows 95, 98, NT, and 2000 implementations
>mysteriously vanished?

That is Bill Gates' problem. However, I doubt any of them ever generated
the deprecated (and now illegal) forms of UTF-8.

>When precisely did RFC 2279 become obsoleted?

It didn't (though it soon will).

However, it forbade generation of the deprecated forms, and discouraged
(should) attempts to interpret them. It did not address the surrogate
issue but, since the codes now forbidden would map into unallocated
Unicode code points, it did not provide any justification for generating
them either.

But all that is moot in the present context, because the Usefor draft
explicitly limits the UTF-8 that may be generated of interpreted in
compliant Netnews Articles in a manner consistent with the latest Unicode
standards.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2185
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: prevervation of installed base
Date: Sun, 12 Jan 2003 19:51:53 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Keld Jørn Simonsen <[email-masked]> writes:

>I agree that there is probably not many general news-mail gateways
>around, but there are a lot of specific mail-news/news-mail gateways,
>and many of them involve constituencies where non-ascii newsgroup names
>may become relevant. The new USEFOR scheme should cater for standard 
>gatewaying for all these individual cases.

Indeed, and where the constituency knows it has a need for non-ascii
newsgroup names it will make sure that whoever does the gatewaying does
the proper conversions. Usefor will indeed make it clear what those proper
conversions are.

But other constituencies which do not have those particular needs should
be able to carry on with their existing gateways until such time as they
find the problem is pressing.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2187
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 20:05:48 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>For what it's worth, I agree with this.  I think that this is the best way
>to move forward in the long term.  However, client authors don't seem to
>be rushing in this direction, and it's unclear to me how best to encourage
>them.  I also am a bit at a loss as to how best to handle these issues in
>the short to medium term before all of the client authors have updated
>their software.

It's a chicken and egg problem. People who care about using the new
features will ensure that suitable clients are made available. As those
clients become available, more people will start using them.

Indeed, comcerned users are already experimenting. The newsgroup
dk.test.utf-æøå is where you will find them.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2186
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: prevervation of installed base
Date: Sun, 12 Jan 2003 19:58:28 GMT
Lines: 31
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> The question is whether that situation is acceptable in a transitional
>> phase,

>You're talking about gateways to email -- what "transitional phase" -- what
>are you assuming will change in the email protocols?

No, I am talking primarily about gateways *from* news. I assume nothing
about changes in the email protocols. I am speaking of transitional
arrangements arising from changes in the news protocols. These are
carefully discussed in the Usefor draft, and the principle conclusion
there is that user agents will need to upgrade if they expect to see the
benefit of the new functionalities (transports are largely unaffected).

Insofar as some user agents will live on the far side of
news->mailing-list gateways, it is the gateways that will need to upgrade,
again only if they care about the new functionalities.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2183
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 19:34:24 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Not really, since the mainstream mail user agents (mwny of which also
>happen to be news user agents) already provide proper encoding and
>tagging in header phrases and comments using the RFC 2047 methods. The
>untagged non-ASCII content is primarily confined to spam; spammers
>tend to use software which is writeen by largely clueless people as
>is evident by the large number of syntax errors in spam. Indeed, those
>errors (including untagged non-ASCII content) are a good indicator of
>spam.

I think you will find that, in the countries where that spam originates
from, users are regularly using untagged non-ASCII in both headers and
bodies when communicating with each other. I have no way of verifying
whether that is true or not in email, but it is most certainly true in the
case of Usenet (you just need to look at the cn.* groups).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2184
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 19:42:27 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> [email-masked] (Claus Färber) writes:

>D. J. Bernstein <[email-masked]> schrieb/wrote:

>> The crucial advantage of UTF-8 is its generality: it can and will be
>> used everywhere.

>That can actually be a problem w.r.t. addresses (domain names, email
>addresses, newsgroup names): An editor does not guarantee anything about
>the normalisation form of its output.

Actually, I believe that decent editors will ensure normalised output.
Whether all editors will turn out to be decent, remains to be seen though.
And actually, it is keyboard drivers that need to get it right in the
main. They should never let the editor see an unnormalised form.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 13 Jan 2003 07:01:12 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
>>
>>>The problem is not so much with software that decodes RFC 2047 (though
>>>even that gets it wrong sometimes), as with software that encodes it in
>>>the first place and which gets stuck as soon as it encounters a header
>>>that it does not know how to parse.
> 
> 
>>An entity (software or human) must have syntax information in order to
>>correctly generate *any* header with *any* content (including RFC 2047
>>encoded-words).  Your sentence quoted directly above therefore makes no
>>sense -- an entity cannot generate a header with unknown syntax.
> 
> 
> A human can quite well generate such a header in full accord with some
> recently published standards-track RFC that allows it. He will of course
> generate it in whichever charset his editor is configured to use at that
> point.

And he will necessarily use RFC 2047/2231 to tag that charset and ensure
that the header field complies with the standards-track RFC, which forbids
non-ASCII characters which do not use 2047/2231.  And there is no need for
a gateway to subsequently change that header field.

> If it arises in Netnews, and his User Agent manages to produce UTF-8 for
> it, then it is some gateway that may get stuck.- at which point that
> gateway needs either to apply my heuristic, or apply one of the other
> solutions suggested, e.g. drop that header, or stick an "X-" in front of
> it.

No, if the field is properly encoded, as it must be per RFC 1036, no
gateway needs to do anything with that header field other than pass it along.
And if a "header field" contains illegal characters, sticking an "X-" in
front of the field name won't make it legal.  And as discussed at length,
the proposed "heuristic" simply cannot work in the scenario given (viz. no
syntax information available).

Date: Mon, 13 Jan 2003 06:51:56 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Philip Hazel <[email-masked]>
CC: [email-masked]
Subject: Re: Unicode principles (Was Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying))
Sender: [email-masked]

Philip Hazel wrote:
> On Sun, 12 Jan 2003, Bruce Lilly wrote:
> 
> 
>>people don't need to look up strings of musical notes in a
>>disctionary, arrange them in a collating sequence, etc.
> 
> 
> They do need to look them up in thematic musical indexes (e.g. an index
> of Beethoven's sonatas).

Which is unrelated to text and therefore shouldn't involve Unicode.

Date: Mon, 13 Jan 2003 09:03:39 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: [email-masked]
Subject: Re: Unicode principles (Was Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying))
Sender: [email-masked]

On Sun, 12 Jan 2003, Bruce Lilly wrote:

> people don't need to look up strings of musical notes in a
> disctionary, arrange them in a collating sequence, etc.

They do need to look them up in thematic musical indexes (e.g. an index
of Beethoven's sonatas).

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

To: [email-masked]
Subject: Re: Unicode principles (Was Re: UTF-8 versions (was: Re: RFC 2047 and  gatewaying))
From: Andrew Gierth <[email-masked]>
Date: 13 Jan 2003 03:36:47 +0000
Lines: 27
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 Bruce> No, please read the entire "Full Encoding" section of Unicode
 Bruce> 3.0 or earlier (it's only a few psragraphs).  The number of
 Bruce> code points (including Unicode 3.0 and earlier surrogate
 Bruce> pairs) has not changed.  What has changed is that the
 Bruce> surrogate pairs (somewhat analagous to shift codes in some
 Bruce> character sets) have been effectively converted to "native"
 Bruce> 32-bit codes.  And that changed Unicode from a uniform 16-bit
 Bruce> code set; look at the number of changes in 3.1 that altered
 Bruce> the "16-bit" text of earlier Unicode.

reading the Unicode docs, the change seems unsurprising given that
Unicode has not been what I would call a "uniform 16-bit code set"
(which I would define as one in which all characters were represented
with exactly 16 bits) for as long as it has had surrogate pairs.

As long as there were no characters assigned in the range above
U+FFFF, the Unicode spec could continue to ignore this discrepancy;
but the allocation of such codes obviously makes that impossible.

I suspect this is historical; Unicode seems to have been designed on
the assumption that 16 bits was enough, and that UTF-16 would be the
primary interchange format.

-- 
Andrew.

To: [email-masked]
Xref: clerew local.mime:2180
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 17:22:12 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>On Fri, 10 Jan 2003 15:26:03 GMT
>"Charles Lindsey" <[email-masked]> wrote:

>> The problem is not so much with software that decodes RFC 2047 (though
>> even that gets it wrong sometimes), as with software that encodes it in
>> the first place and which gets stuck as soon as it encounters a header
>> that it does not know how to parse.

>Does this same software "get stuck" as soon as it encounters some other
>syntax error in the input?  After all, a non-ascii character in the message
>header of an input message is a violation of the message syntax.
>Some errors are recoverable with a reasonable degree of safety - so for
>instance you can often make sense of a malformed rfc822 style date
>even if the syntax is completely wrong.  Similarly, you can make sense of
>some header fields which contain non-ascii characters by encoding them
>per 2047.  But that doesn't mean you can recover from all inappropriate
>uses of non-ascii characters.  

Sure. There are a variety of ways software may "get stuck" if it receives
unexpected input. The implementor then has to use his judgement as to
whether to attempt a repair or whether to drop possibly useful
information. I am speaking of gateways and receiving User Agents here.

If the software is close to the poster/author, then sending it back to the
author with an error message is a third possibility.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2177
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 16:53:16 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> 
>> The problem is not so much with software that decodes RFC 2047 (though
>> even that gets it wrong sometimes), as with software that encodes it in
>> the first place and which gets stuck as soon as it encounters a header
>> that it does not know how to parse.

>An entity (software or human) must have syntax information in order to
>correctly generate *any* header with *any* content (including RFC 2047
>encoded-words).  Your sentence quoted directly above therefore makes no
>sense -- an entity cannot generate a header with unknown syntax.

A human can quite well generate such a header in full accord with some
recently published standards-track RFC that allows it. He will of course
generate it in whichever charset his editor is configured to use at that
point. His User-Agent, which was written before that RFC was published
will the get "stuck" - just like I said. So it can happen, and it can
happen in email.

If it arises in Netnews, and his User Agent manages to produce UTF-8 for
it, then it is some gateway that may get stuck.- at which point that
gateway needs either to apply my heuristic, or apply one of the other
solutions suggested, e.g. drop that header, or stick an "X-" in front of
it. There is indeed room to discuss which of these things would be the
best for it to do.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2178
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 17:04:33 GMT
Lines: 34
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>D. J. Bernstein wrote:
>> 
>> Specifying ``UTF-8 everywhere'' removes the need to add charset tags to
>> individual pieces of data. The message writer converts the local charset
>> to UTF-8 and then discards the local charset information.

>That would only work if it could be absolutely guaranteed that no untagged
>data in any other character set might appear.  No such guarantee is possible,
>and in fact Usenet abounds with untagged charsets, of which, according to
>Andrew Gierth, "no significant amount" is utf-8.

Not so. Untagged data in "any other" charset would be non-compliant with
the standard, so the standard is not obliged to cater for that case
(however, Usefor does provide a _suggested_ way out which, if followed,
will result in nothing worse that the present unsatisfactory state of
affairs).

Having said that, if someone can propose a tagging scheme for such headers
(and that might well include language tags), then there is no reason why
Usefor could not consider it. I see many technical problems, but they
could be discussed.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2176
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 16:41:18 GMT
Lines: 37
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>D. J. Bernstein wrote:
>> In a world that makes any use of RFC 2047, displaying text correctly
>> means identifying the locations where RFC 2047 encoding is used.

>Display (as opposed to generation) does not require information about
>header field syntax; RFC 2047 explicitly provides for considering
>unrecognized fields as unstructured *for the purpose of display*.

Not so.

If you see the header
	Foorbar: XXX (=?iso8859-1?q?=A3-=A3?=) YYY
then if it is a structured header you are supposed to decode the
encoded-word, and if it is an unstructured header you are not supposed to
decode it.

If you don't know whether it is structured or unstructured then you are,
officially, stuck. In practice, most implementations will take a heuristic
decision to decode it, but they are not complying with the standard if
they do so.

Some people consider that the fact that RFC 2047 puts implementors into
these impossible situations is a "design flaw" in RFC 2047. Note that this
is not a Usefor problem. It exists in email already.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2179
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Sun, 12 Jan 2003 17:10:52 GMT
Lines: 19
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>....  Drop the raw utf-8 nonsense and move on to somthing that has
>a non-zero chance of baing (a) standardized and (b) implemented.

The chances of getting it standardized may or may not be zero. But the
chances of it being implemented are certainbly non-zero, because
implementations already exist.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Sun, 12 Jan 2003 18:01:39 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Jean-Marc Desperrier <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Jean-Marc Desperrier wrote:
> Dave Crocker writes :

>>>> If this is is an example of the way discussions have gone on usefor, 
>>>> then
>>>> it's pretty clear why the technical work is in trouble.
>>>>
> The current usefor text is very technically sound .

Actually, no. Please see the discussions about gateways and RFC 2047
on the ietf-822 archives.  Summary: the Usefor draft is technically flawed;
it requires gateways to do the impossible.

> You might not like the direction where it goes, but there has been a lot 
> of serious work on every aspect.

Much is at best experimental, and often untested.  Sometimes not even
clearly thought out.

>> 3.  the logic behind that goal is applicable to any other application 
>> having
>> a similar installed base history.
>>
> The big difference is that Usenet transport has almost since the very 
> start been fully 8 bit.

Now that is an example of the way discussions have gone on usefor. Dave
wrote about the logic behind the goal of compatibility with installed
base (including moderation methods, combined user agents, etc.). Suddenly
there's an abrupt change of topic to "Usenet has always had 8-bit clean
transport".  Whether or not that statement is factually correct is being
addressed by others, I see. My point is that the issue isn't solely about
transport. Compatibilty is also about user agent software, moderation
methods and software, etc.

>> 7.  breaking standards is not measured by whether someone's code 
>> core-dumps.
>> it is measured by whether it violates the specification.  sending 
>> 8-bit data
>> in a 7-bit environment breaks the specification.  sending valid 
>> strings of 7-bit
>> data that might need further interpretation (to obtain the semantics) 
>> does
>> not.
>>
> Usenet is a 8 bit environnement.

No Usenet necessarily uses email for moderation, and there are established
prohibitions against 8-bit data in headers (incidentally, that also applies
to news, as RFC 1036 uses the same message format with the same header content
restrictions).  Once again, there is confusion between transport and the larger
scope of compatibility in toto.

Date: Sun, 12 Jan 2003 17:45:09 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: [email-masked]
Subject: Unicode principles (Was Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying))
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> That change was basically an admission that 64k wasn't enough. It is 
> still possible that some bigger number is enough. The unicode consortium 
> believes that 17*64k is enough, and I agree.

No, please read the entire "Full Encoding" section of Unicode 3.0 or
earlier (it's only a few psragraphs).  The number of code
points (including Unicode 3.0 and earlier surrogate pairs) has not
changed.  What has changed is that the surrogate pairs (somewhat
analagous to shift codes in some character sets) have been effectively
converted to "native" 32-bit codes.  And that changed Unicode from a
uniform 16-bit code set; look at the number of changes in 3.1 that
altered the "16-bit" text of earlier Unicode.

>> "There are over 18,000 unassigned code positions that are available 
>> for future allocation. This number far exceeds anticipated character 
>> encoding requirements for all world characters and symbols."
> 
> 
> Yep. I have that too. The fact that 18,000 isn't enough doesn't mean 
> that about a million isn't enough.

Now you've conflated *unassigned* code points with *total* code
points.  And as noted above, that's 16-bit native code positions; the
text continues to point out that even early Unicode had over a million
code points available via surrogate pairs.

> That doesn't 
> add up to much either

A piece of straw doesn't weigh much -- ever hear of "the straw that broke
the camel's back"?
.
>> "Graphologies unrelated to text, such as musical and dance notations, 
>> are outside the scope of the Unicode Standard."
> 
> 
> "Unrelated to text". If something like that is used in books 
> intermingled with English text, it's hard to say that it's unrelated to 
> English text.

Had the Unicode Consortium started with the premise that it was encoding glyphs
and that anything that had ever appeared on a display, on paper, carved in clay
or stone, or scratched in sand was fair game, that would be one thing. But it
didn't; it started with the principle of encoding text (not limited to English)
*characters*, not glyphs.

So we now have Western musical notes which had been deemed out of scope. How
long do you suppose it will be before some group that uses non-Latin text
characters pipes up and says "wait a minute, that's *Western* musical
notation; we use something different, so you have to include our notation
also"? So we'll have a handful of Asian musical notations, a bunch of Native
American musical notations, some Australian aboriginal musical notations,
and so on.  All unrelated to text.

Publishing in books shouldn't be the main criterion for whether something
is part of Unicode.  The criteria, first and formost, should be "is it text?".
Then "is it fundamentally different from what is already in Unicode, or it
it a sytlistic variant?"; e.g. different font styles or different han
sytles.  Musical notation never belonged in Unicode, as originally stated.
It does not fall within the scope of the issues that Unicode was intended
to address -- people don't need to look up strings of musical notes in a
disctionary, arrange them in a collating sequence, etc.

In any event, we're getting pretty far off topic here.

Date: Sun, 12 Jan 2003 14:19:15 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Henry Spencer <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Henry,

Sunday, January 12, 2003, 12:43:17 PM, you wrote:
Henry> The 7-bit software issues occur only at the edges of Usenet, in gateways
Henry> to mail in particular, not in Usenet's own transport code. 

thanks for the clarification.

in fact, I mean "transport" in an entirely loose sense, and would have done
better to say something explicitly loose like "the global usenet
infrastructure" or somesuch.

One still might take exception, noting that Henry said "edges" but it is
important to note that anything that is prevalent among at the edge
constitutes a defact part of the infrastructure.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Sun, 12 Jan 2003 15:43:17 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: Dan Kohn <[email-masked]>
cc: [email-masked], [email-masked]
Subject: RE: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Sun, 12 Jan 2003, Dan Kohn wrote:
> > The big difference is that Usenet transport has almost since the very
> > start been fully 8 bit.
> 
> No, no, no.  Usenet has always supported 8 bit message bodies, as HTTP
> and BEEP have, and unlike SMTP.  But RFC 1036 (and RFC 850 before that)
> do *not* allow 8 bit headers...

Careful here.  The two statements are not in conflict.  Usenet transport
has never enforced this aspect of RFC 1036.  The *transport software* has
been 8-bit-clean essentially from the start, simply because none of the
software authors saw any reason to add code to enforce an artificial 7-bit
restriction.  (There were sometimes restrictions on specific items of data,
like newsgroup names, but typically not on headers in general.)

The 7-bit software issues occur only at the edges of Usenet, in gateways
to mail in particular, not in Usenet's own transport code. 

                                                          Henry Spencer
                                                       [email-masked]

content-class: urn:content-classes:message
Importance: normal
Subject: RE: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Date: Sun, 12 Jan 2003 12:01:50 -0800
From: "Dan Kohn" <[email-masked]>
To: "Jean-Marc Desperrier" <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

Dave Crocker wrote:

>> 2.  compatibility with netascii is an explicit goal when working
>> with email. 
>> 3.  the logic behind that goal is applicable to any other
>> application having a similar installed base history.

Jean-Marc Desperrier writes:

> The big difference is that Usenet transport has almost since the very
> start been fully 8 bit.

No, no, no.  Usenet has always supported 8 bit message bodies, as HTTP
and BEEP have, and unlike SMTP.  But RFC 1036 (and RFC 850 before that)
do *not* allow 8 bit headers (and neither do HTTP, BEEP, and SMTP).

To quote RFC 1036:

    The primary consideration in choosing a message format is that it
    fit in with existing tools as well as possible.  Existing tools
    include implementations of both mail and news....  Therefore, the
    rule is adopted that all USENET news messages must be formatted as
    valid Internet mail messages, according to the Internet standard
    RFC-822....  However, it should always be possible
    to use a tool expecting an Internet message to process a news
    message.  In any situation where this standard conflicts with the
    Internet standard, RFC-822 should be considered correct and this
    standard in error.

>> 4.  paying attention to the installed base requires worrying about
>> compatibility with what has been established practise, not what
>> "might" work or what is "frequently" available.

> We have a problem here with the definition of what constitutes
> established practise.
> In my opinion, in every respect 8 bits in headers is established
> practise in usenet.

Yes, in practice, many usenet gateways and user agents support 8 bit
headers (though very few today understand UTF-8).  But many others
(including mail gateways and IMAP servers) do not, and those were
written in good faith to be compatible with RFC 1036.

> A person who receives a usenet message is never the destinary of that
> message, and it is his job to make that it takes to receive them.
> There's  never a guarantee you'll receive a message, it just happens
> sometimes that a specific message does not arrive, on very loaded
> binary groups it's even very frequent.
> If that happens and you're not happy, you have alternatives and can
> try to get it from another source.

Obviously, the goal of a usefor (or rechartered working group) RFC
should be a standard that, when followed, drastically improves the
reliability of usenet, including support for i18n headers as well as
multimedia and i18n message bodies.  It should also be a goal that the
standard does not worsen reliability for gateways and agents that don't
(yet) follow it.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Sun, 12 Jan 2003 20:01:49 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Dave Crocker writes :

>Erland> Dave Crocker <[email-masked]> writes:
>  
>
>>>If this is is an example of the way discussions have gone on usefor, then
>>>it's pretty clear why the technical work is in trouble.
>>>
The current usefor text is very technically sound .
You might not like the direction where it goes, but there has been a lot 
of serious work on every aspect.

>2.  compatibility with netascii is an explicit goal when working with email.
>
The current USEFOR draft does a lot to insure that.
I'm certain it's possible to find a solution to the very last remaining 
problem.

>3.  the logic behind that goal is applicable to any other application having
>a similar installed base history.
>
The big difference is that Usenet transport has almost since the very 
start been fully 8 bit.

>4.  paying attention to the installed base requires worrying about
>compatibility with what has been established practise, not what "might" work
>or what is "frequently" available.
>
We have a problem here with the definition of what constitutes 
established practise.
In my opinion, in every respect 8 bits in headers is established 
practise in usenet.

>7.  breaking standards is not measured by whether someone's code core-dumps.
>it is measured by whether it violates the specification.  sending 8-bit data
>in a 7-bit environment breaks the specification.  sending valid strings of 7-bit
>data that might need further interpretation (to obtain the semantics) does
>not.
>
Usenet is a 8 bit environnement.

Also, this discussion remembers me that there's a difference in the 
philosophy of usenet and email that explains why the japan related ML 
that I participate in stay in netascii, while the 
fr.lettre.langue.japonaise newsgroup switched to using UTF8, both choice 
being coherent in their own context.

A person who receives a usenet message is never the destinary of that 
message, and it is his job to make that it takes to receive them.
There's  never a guarantee you'll receive a message, it just happens 
sometimes that a specific message does not arrive, on very loaded binary 
groups it's even very frequent.
If that happens and you're not happy, you have alternatives and can try 
to get it from another source.

In the case of the current recommendation of Usefor, people could get it 
through Google Group, it already works today :
http://groups.google.fr/groups?oe=UTF-8&selm=3C430CFC.1080804%40club-internet.fr

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Date: Sun, 12 Jan 2003 19:22:27 +0100
Sender: [email-masked]

The below basically argues that Unicode (and UTF-8) is big enough for 
the foreseeable future, so that no change to the UTF-8 specification 
will be necessary.

If you don't care about Unicode, just stop reading here.

Bruce Lilly writes:
> Arnt Gulbrandsen wrote:
>> Why would you expect Unicode to change substantively?
>
> The 3.0->3.1 experience. A.k.a. "once burned, twice shy".

That change was basically an admission that 64k wasn't enough. It is 
still possible that some bigger number is enough. The unicode 
consortium believes that 17*64k is enough, and I agree.

>> The number of characters used for human communication desn't seem to 
>> be rising much, and there's plenty of space left in the current 
>> specification. IIRC Unicode still uses less than 200,000 of the 
>> million-odd possible code points.
>
> Famous last words.  From my handy dead-tree copy of Unicode 2.0, page 
> 2-4, under the "Full Encoding heading":
>
> "There are over 18,000 unassigned code positions that are available 
> for future allocation. This number far exceeds anticipated character 
> encoding requirements for all world characters and symbols."

Yep. I have that too. The fact that 18,000 isn't enough doesn't mean 
that about a million isn't enough.

> Cough, cough.  It is nearly a universal truth that things tend to 
> expand to fill the available space (and/or time).  Why do you 
> (apparently) think that Unicode is exempt?

I don't. I do think that people's ability/willingness to learn 
characters is a (much) stricter limitation than the number of available 
code points in Unicode.

Some people will invent new scripts for some languages, but I doubt 
_many_ characters will be added in this way. The costs of teaching kids 
big alphabets are too high, for a start.

Some people will take books which mix a "dance notation" font with 
English, write up a proposal adding those characters, and submit it. Or 
the chess notation used in the newspaper's chess column. That doesn't 
add up to much either - the number of characters added in that way is 
limited to what the font vendors and publishers use, and what the 
audience(s) will learn.

>> I suppose you could argue that Unicode adds alphabets. But do you 
>> think Unicode still hasn't reached the 20% mark?
>
> They add more than "alphabets", and that's part of the problem. Again 
> quoting Unicode 2.0 (page 1-3 this time):
>
> "Graphologies unrelated to text, such as musical and dance notations, 
> are outside the scope of the Unicode Standard."

"Unrelated to text". If something like that is used in books 
intermingled with English text, it's hard to say that it's unrelated to 
English text.

--Arnt

Date: Sun, 12 Jan 2003 12:06:44 -0500
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Bruce Lilly wrote:
> Arnt Gulbrandsen wrote:
> 
>> Why would you expect Unicode to change substantively?
> 
> 
> The 3.0->3.1 experience. A.k.a. "once burned, twice shy".

A postscript: for those who haven't looked recently, the Unicode
Consortium web site at http://www.unicode.org mentions a Unicode
4.0 Alpha -- given the sweeping changes during the *minor* number
increment from 3.0 to 3.1, I shudder to think that the *major*
version number increment portends.

Date: Sun, 12 Jan 2003 11:52:27 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Why would you expect Unicode to change substantively?

The 3.0->3.1 experience. A.k.a. "once burned, twice shy".

> The number of characters used for human communication desn't seem to be 
> rising much, and there's plenty of space left in the current 
> specification. IIRC Unicode still uses less than 200,000 of the 
> million-odd possible code points.

Famous last words.  From my handy dead-tree copy of Unicode 2.0, page
2-4, under the "Full Encoding heading":

"There are over 18,000 unassigned code positions that are available for
future allocation. This number far exceeds anticipated character encoding
requirements for all world characters and symbols."

Cough, cough.  It is nearly a universal truth that things tend to expand
to fill the available space (and/or time).  Why do you (apparently) think
that Unicode is exempt?

> I suppose you could argue that Unicode adds alphabets. But do you think 
> Unicode still hasn't reached the 20% mark?

They add more than "alphabets", and that's part of the problem. Again
quoting Unicode 2.0 (page 1-3 this time):

"Graphologies unrelated to text, such as musical and dance notations, are
outside the scope of the Unicode Standard."

Yeah, right. What did the Unicode Consortium add in 3.1? -- musical notation.
So, as I said, I will not (anymore) be surprised when the Unicode
Consortium adds (literally) chicken scratching, probably shortly after
they add (if they haven't already) dance notation (human and otherwise),
every possible organic chemistry carbon ring symbol, Feynman diagrams,
CAD/CAM symbols, traffic sign symbols, trademarks, logos, etc. ad nauseum.
N.B. no smiley. [that reminds me; they'll probably add all of the smiley
variants too (if they haven't already)]

I was once a Unicode fan -- the above quotations from earlier Unicode
Standards and "The Ten Unicode Design Principles" that were also in the
earlier standards were sound principles.  Unfortunately the Unicode
Consortium appears to have abandoned many of those principles.

Date: Sun, 12 Jan 2003 11:10:51 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: [email-masked], [email-masked]
Subject: Re: Call for Usefor to recharter
Sender: [email-masked]

Dan Kohn wrote:
> I would say that punycode is compatible with all MIME,
> DNS, and Internet application protocols, because it is 7 bits, and so it
> transits them unchanged.  (In fact, it only uses LDH - letters, digit,
> hyphens.)

The latter (use of ldh only), in conjuction with length limits, is
*precisely* what is required for compatibility, as that is what
conventional domain names use.  It is not coincidental that
punycode is compatible and is moving forward rapidly through the
standards process and is expected to be widely and quickly
implemented.  The same was (and remains) true of MIME; RFC 2047 /
2231 encoding was designed for compatibility (with known gateway
issues and with RFC 822 parsing).

From: Arnt Gulbrandsen <[email-masked]>
To: Bruce Lilly <[email-masked]>
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Cc: [email-masked], Andrew Gierth <[email-masked]>
Date: Sun, 12 Jan 2003 17:01:58 +0100
Sender: [email-masked]

Bruce Lilly wrote:
> Andrew Gierth wrote:
>> since there have been no substantive changes made to UTF-8 _ever_ 
>> since its adoption as any sort of standard, why would you expect 
>> future changes to introduce incompatibilities?
>
> Bad premise; every time Unicode changes substantively, the utf-8 
> specification necessarily also changes since it is a Unicode to/from 
> octet stream transformation.

Why would you expect Unicode to change substantively?

The number of characters used for human communication doesn't seem to be 
rising much, and there's plenty of space left in the current 
specification. IIRC Unicode still uses less than 200,000 of the 
million-odd possible code points.

I suppose you could argue that Unicode adds alphabets. But do you think 
Unicode still hasn't reached the 20% mark?

> When the Unicode consortium decides to include chicken scratching as 
> "characters" and extends the maximum width to beyond 32 bits, even 
> the 5- and 6-byte sequences (which exist in some "utf-8" 
> specifications on the octet-stream side, but not in others) will have 
> to change.

"Will have to change"... There's a little more than 5,000 languages on 
the globe. If every one of them were to invent its own kanji-like 
writing system with about 100,000 characters, that still wouldn't fill 
32 bits.

--Arnt

Date: Sun, 12 Jan 2003 10:27:48 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Andrew Gierth wrote:

> Or do you expect things like this:
> 
> 
>>----Original message----
>>Date: 11 Jan 2003 20:23:00 +0200
>>From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
>>To: [email-masked]
>>Subject: Re: RFC 2047 and gatewaying
> 
> 
> to appear in encoded form as above?

[Hmmm, RFC 2822 strongly discourages comments in address fields]

Short answer:
I expect that the comment would be placed in the body which would
be quoted-printable encoded and in the iso-8859-1 charset, i.e. the
line with the encoded word in your example would be
 >From: [email-masked] (Claus_F=E4rber)
in the body text, which would display as expected in a MIME-compliant
user agent.  It would not affect the message transmission in any way.
Some user agents will do that automatically if the 8-bit raw iso-8850-1
text is entered, hopeully that is what Mozilla will do with this message
(so what appears above should *display* as what would appear verbatim in
the message body, while what appears below is what you would see if you
received a message with that attribution as described above, assuming
that Mozilla does the right thing)
 >From: [email-masked] (Claus_Färber)
That is not quite full decoding since it merely involves stripping the
RFC 2047 lead-in, glue, and trailer from the encoded-word text.

Longer answer:
That depends on a number of factors.  Decoding is only possible if
1a. the body has a suitable domain as specified in a
     Content-Transfer-Encoding field (8bit in this case, which further
     presumes that an 8bit transfer path is available)
or
1b. the body is suitably encoded and tagged via a Content-Transfer-Encoding
     field (e.g using quoted-printable encoding, which is similar to the Q
     encoding used in this case; in general it may be necessary to decode
     and re-encode to use this method)
and either
2a. the body charset is the same as is used in the header field(s)
or
2b. it is possible and practical to translate the charset in the encoding
     used in the header field(s) to the charset of the body.
and
3.  A Content-Type header field is provided with an appropriate charset
     parameter (except in the unlikely case that the default us-ascii
     charset applies to the entire body (including the decoded encoded-word)).

Some of the necessary condition(s) might not be satisfied in a given instance.
In the particular case given, the most generally compatible method would be
to use quoted-printable transfer encoding in the body with an iso-8859-1
charset (and corresponding header fields).  That is compatible with all
known mail and news transmission paths and gateways. It is possible, however,
that the recipient(s) might not be using a MIME-compliant reader, in which
case the quoted-printable encoding will be visible.  If that is a concern,
one could of course leave the RFC 2047 encoded-word verbatim, in which
case no special considerations need to be given to body encoding (at least
w.r.t. that copied content) in which case the 2047 encoded word probably
will not be decoded for display (it should not be decoded under those
conditions).  Under specific circumstances other alternatives from the above
list might be possible; for example if it is known that all possible mail
exchangers for all recipients support the 8BITMIME ESMTP extension and that
that extension is also supported by the sender's ESMTP client and any
SMTP relays which might be involved, 1a could be used provided that conditions
2 and 3 were also satisfied and that 8bit transfer were not prohibited by
the message itself (due to MIME requirements; see below).  In practice even
experts in mail protocols would find it difficult to be absolutely certain
of meeting those conditions except for trivial cases (e.g. somebody sending
over a LAN, with a single MX supporting ESMTP and 8BITMIME, and no mailing
list expanders); network and server outages or congestion (which may result
in use of an unusual MX alternative), mailing list expansion, gateways, etc.
all complicate matters.  A user who is ignorant of the mail protocols would
be unable to be certain of the necessary conditions.  A user agent could
not ascertain whether those conditions were satisfied either, also because of
the possibility of list expansion, etc.  The only safe courses of action
for user agent software in this example are
a. (trivial case) elide the comment. It is after all only a comment, not a
    display name
b. use quoted-printable encoding as described above
c. leave it as RFC 2047 encoded-words (the lazy programmer's approach)

It is important to note that the decoded (or not) content will be placed in
body text, not in message or MIME-part header fields, so with an appropriate
Content-Transfer-Encoding field in an appropriate context with a suitable
transmission path, that is acceptable.  Unlike the case for message and MIME-
part header fields, which can never have unencoded non-ASCII content (nor
can they contain ASCII NUL, a lone ASCII CR, or a lone ASCII LF).

It is also important to note that there are MIME restrictions on where the
various transfer encodings may be used. For example if the body text in
question were part of a composite MIME type, quoted-printable transfer
encoding would have to be applied at the innermost level (RFC 2045 section
6.4). See also RFC 2046 section 5.2.1.  If the message is part of a sufficiently
lengthy encapsulated message that it might be split into pieces, note that
RFC 2046 section 5.2.2 prohibits any transfer encoding other than 7bit for
the encapsulating message/partial type, and that in turn prohibits use of
8bit or binary transfer encoding in the encapsulated message.

> While the individual may not strictly need to edit them, they
> nevertheless become part of an entity that the user is editing.  Of
> course, they are not then part of a header field, but if, say, I need
> to reply to someone who sent a message which has been sent on to me by
> a third party in this way, I have no real option but to copy+paste from
> the message body back into the headers.

Since the only things that can be encoded by RFC 2047 (display names and
comments) are transparent to the transmission protocols, one *need* not
copy those parts *to* header fields at all; in the case of the diplay name
phrase of a mailbox specification, the address w/o the display name would
suffice, and in the case of a named group, any display name could be used.
And of course comments could be replaced by linear whitespace.

> Expecting RFC2047 encoded-words never to be decoded except for display
> is not realistic.

They should never be decoded in headers (or for SMTP "envelope" transactions)
for transmission.  RFC 2047 does not specifically apply to body text (other
than MIME-part header fields and DSN and MDN fields).

[note: Andrew, the In-Reply-To header field of your message did not contain
a message-id as required by RFC 2822 section 3.6.4]

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Andrew Gierth <[email-masked]>
Date: 12 Jan 2003 12:26:55 +0000
Lines: 36
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 Bruce> I don't follow your argument; could you please explain the
 Bruce> circumstances under which some individual would need to edit
 Bruce> an encoded-word which is part of another person's display name
 Bruce> in a header field.  There is no need to decode and re-encode
 Bruce> in order to use a From field content to generate a new To, Cc,
 Bruce> etc. field.

Leaving aside the question of editing Subject lines, From, To,
etc. fields are routinely decoded in order to incorporate their data
into the body of the message in the form of an attribution line or
quoted message headers.

Or do you expect things like this:

>----Original message----
>Date: 11 Jan 2003 20:23:00 +0200
>From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
>To: [email-masked]
>Subject: Re: RFC 2047 and gatewaying

to appear in encoded form as above?

While the individual may not strictly need to edit them, they
nevertheless become part of an entity that the user is editing.  Of
course, they are not then part of a header field, but if, say, I need
to reply to someone who sent a message which has been sent on to me by
a third party in this way, I have no real option but to copy+paste from
the message body back into the headers.

Expecting RFC2047 encoded-words never to be decoded except for display
is not realistic.

-- 
Andrew.

Date: Sun, 12 Jan 2003 06:56:53 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein wrote:
> Bruce Lilly writes:
> 
>>could you please explain the circumstances under which some individual
>>would need to edit an encoded-word which is part of another person's
>>display name in a header field
> 
> 
> I press ``f'' in my mailer. It creates a tentative followup to the
> message I'm looking at, and invokes my text editor. I then edit the
> message to taste.
> 
> Sometimes I edit header fields. For example, I might change the Subject
> line.
<snip>

You haven't answered the question which was asked.

From: "Shmuel (Seymour J.) Metz" <[email-masked]>
Reply-to: [email-masked], [email-masked]
Date: Fri, 10 Jan 2003 14:43:44 -0500
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Atid/2
Sender: [email-masked]

In <[email-masked]>, on 01/07/2003
   at 08:45 PM, [email-masked] (Kent Landfield) said:

>Yeah right...  The real problem is that people didn't keep the focus 
>on the group's charter

Including the part that lists as urgent

  - Use of non-ASCII character sets in article headers and bodies
 
-- 
-----------------------------------------------------------
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2
     Team OS/2
     Team PL/I
-----------------------------------------------------------

Date: 12 Jan 2003 00:09:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein <[email-masked]> schrieb/wrote:
> Sometimes I edit header fields. For example, I might change the Subject
> line. What do you think happens if I correct the spelling of an RFC 2047
> French word with an accent? That's right: I'm editing an encoded word.

> Of course, display is critical. I need to read what I'm editing. I can't
> read RFC 2047 gobbledygook. My text editor is not going to support RFC
> 2047 and IDNA GoofyCode and all the other complicated ad-hoc ``7 bits
> forever!'' character encodings.

It's the task of the user agent to present the message to the user in a
friendly format. This can easily be done for header fields whose format
is known. Data that can't be represented on the local system can be kept
in encoded form, so this also works on systems that don't support
Unicode.

It's easy for a user agent to translate the changed data back into
transmission format. For header fields not known to the user agent, the
user is responsible for the correct encoding.

> The crucial advantage of UTF-8 is its generality: it can and will be
> used everywhere.

That can actually be a problem w.r.t. addresses (domain names, email
addresses, newsgroup names): An editor does not guarantee anything about
the normalisation form of its output. A user might enter a character
that has some compatibility decomposition and will never understand why
his message bounces, does not appear in a newsgroup, etc.
When the user agent does a conversion from a display and edit format to
a transmission format, it can normalise the data at the same time.

Of course, if we were designing a completly new message format, UTF-8  
would be the right choice. But we are not. Of course, adding more and  
more kludges like RFC 2047, RFC 2231, Punycode, etc. to maintain  
compatibility are not the most elegant solution... but it's hardly the  
worst part of the RFC 2822/1036 message format.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: Sat, 11 Jan 2003 16:24:43 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: [email-masked] (Erland Sommarskog)
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Erland,

Saturday, January 11, 2003, 2:22:37 PM, you wrote:

Erland> Dave Crocker <[email-masked]> writes:
>> Erland> Why let it stop with 7 bits? Why not cram it into one bit, while you're
>> Erland> at it?
>>
>> If this is is an example of the way discussions have gone on usefor, then
>> it's pretty clear why the technical work is in trouble.
>>
>> 7-bit netascii is a long-standing internet norm.  that's why.

Erland> If you were unimpressed by the argument, that was kind of the point.

I know the feeling.

Mine was that your thinking that carrying the point ad absurdem would
somehow be useful shows a lack of understanding for what was being said, as
does your latest response.

1.  netascii has a 30-year installed base.

2.  compatibility with netascii is an explicit goal when working with email.

3.  the logic behind that goal is applicable to any other application having
a similar installed base history.

4.  paying attention to the installed base requires worrying about
compatibility with what has been established practise, not what "might" work
or what is "frequently" available.

5.  encodings are different from representations.  representation is the
real, "native" data.  encoding is a way of transporting that data over a
constrained environment.

6.  the fact that one 7-bit encoding scheme was not successful does not
condemn all 7-bit encoding schemes.

7.  breaking standards is not measured by whether someone's code core-dumps.
it is measured by whether it violates the specification.  sending 8-bit data
in a 7-bit environment breaks the specification.  sending valid strings of 7-bit
data that might need further interpretation (to obtain the semantics) does
not.

>> it would be extremely helpful for this topic to benefit from attention being
>> paid to deployed, real-world standards-based installed base.

Erland> Yes it would. So why don't you do so, rather than referring to
Erland> theoretical 7-bit editors?

the reference has been to standard, deployed text editors and display
capabilities.  that is practical reality, not theory.

So,

Unless I am missing something, the line being pursued is identical to the
"just send 8-bits" argument that was used 10 years ago for email.  It was
misguided then and nothing has changed.

When you folks want to pay more attention to established standards practise
and to the real installed base, you might make some standards progress.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Sun, 12 Jan 2003 01:16:55 +0100
From: Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Tue, Jan 07, 2003 at 06:58:40PM -0500, Keith Moore wrote:
> 
> > > No, because the only standardised 7-bit encoding for unicode is UTF-7 
> > > that has so many drawbacks it's strongly deprecated.
> > 
> > There is another standardized encoding for iso-10646, namely mnemonic in
> > RFC 1345.
> 
> RFC 1345 is an informational document.

yes, but you can still use the charsets defined it it as mime charsets.

Kind regards
keld

Date: 11 Jan 2003 22:33:21 -0000
To: [email-masked], [email-masked]
Subject: RE: Call for Usefor to recharter
From: [email-masked] (Erland Sommarskog)
Sender: [email-masked]

"Dan Kohn" <[email-masked]> writes:
> Our disagreement comes down to the definition we're each using for
> "compatible".  I would say that punycode is compatible with all MIME,
> DNS, and Internet application protocols, because it is 7 bits, and so it
> transits them unchanged.  (In fact, it only uses LDH - letters, digit,
> hyphens.)  Now, it's true that the non-LDH characters encoded in
> punycode look like gobbledygook on non-punycode-aware readers.  But, the
> punycode text is not going to be corrupted by gateways or non-punycode
> aware user agents.  User agents that understand it can provide good
> i18n; the rest of the agents still work fine.

Of course, it depends on where one has the priorities.

If all one want to say is "look, here is a possibility for those
funny people who don't speak to English to have newsgroups in their
own language" and then roll over to other side and fall back to
sleep, something like Punycode perfectly meets the needs.

However, being one of those funny people myself, I am also interested 
in a solution that works. And I don't see that Punycode does that. It is
fact so useless, that it's better to stay at the current situation.

For Punycode to be widely supported in newsreaders, there has to be
demand. And for there to be demand, there has to be newsgruops. And
for there to be newsgroups, there has to be sufficiently many potential
users that can understand the group name.

Note also that Punycode has the theoretical problem that the encoding
of a name could coincide with the name of an existing group. Not bloodly
likely, but it should bother die-hard standards buffs.

> And though no news reader currently supports punycode, it's difficult to
> imagine that it will not be ubiquitously deployed within a year or two,

Did people say the same when MIME was introduced? It was just recently
the popular news reader Free Agent got MIME support. And I am reading
this mainling list in a mailer in which I can read unencoded UTF-8
easily, but not Quoted Printable or Base-64.

> By contrast, raw UTF-8 may work fine for some newsreaders (like
> Mozilla), but it is *incompatible* for others, and especially for
> gateways (like IMAP servers).  Incompatible means that it breaks
> software that had been correctly designed to work with previous versions
> of the standard.
>
> Note that, as in this case, elegance and compatibility are often at
> odds.

Yes, and if one is interested in the elegance of the standards it is
an easy choice.

If one is interested in the stuff of actually being useful, then it is
not so easy. Then you may have to make a trade-off between compatibility
and usability.
--
Erland Sommarskog, Stockholm, [email-masked]

Date: 11 Jan 2003 22:22:37 -0000
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: [email-masked] (Erland Sommarskog)
Sender: [email-masked]

Dave Crocker <[email-masked]> writes:
> Erland> Why let it stop with 7 bits? Why not cram it into one bit, while you're
> Erland> at it?
>
> If this is is an example of the way discussions have gone on usefor, then
> it's pretty clear why the technical work is in trouble.
>
> 7-bit netascii is a long-standing internet norm.  that's why.

If you were unimpressed by the argument, that was kind of the point.

It is quite clear from your comment that your knowledge on Unicode
is shallow. Believing that getting Unicode into a 7-bit world is
just to cram it into one less bit, is not very intelligent.

As already has been pointed out there existed the UTF-7 encoding for
this purpose. It has been withdrawn, because there too many problems
with it.

You're seven bit alternative appears to be RFC2047, but that is a
huge difference to a UTF-8 encoding. There is no editor, or any
other general-purpose tool that understands RFC2047. RFC2047 is
constrained to a much smaller realm.

Therefore going from raw UTF-8 to RFC2047 is a much huger leap than
stripping one more bit.

> Erland> Incorrect. If you have an 8-bit editor that does not understand UTF-8,
> Erland> you will see the text, but it will look "ugly".
>
> it would be extremely helpful for this topic to benefit from attention being
> paid to deployed, real-world standards-based installed base.

Yes it would. So why don't you do so, rather than referring to
theoretical 7-bit editors?

> Erland> The same applies if you have a any-bit editor which does not understand
> Erland> RFC2047.
>
> Erland> But:
> Erland> 1) UTF-8 looks less ugly than RFC2047 when incorrectly displayed.
>
> Unfortunately, this effort is not a computer science theoretical exercise.

Good. Let's then aim at a solution that is practically useful, and which
have the least bad impacts when used with legacy software.

The theoretical ides about it could break some arcane software, I leave
to someone else.

--
Erland Sommarskog, Stockholm, [email-masked]

Date: 11 Jan 2003 22:12:18 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Dave Crocker writes:
> 7-bit netascii is a long-standing internet norm.

The American Standard Code for Information Interchange encodes only
128 different characters: !"#$%&'()*+,-./0123456789:;<=>?@[\]^_`{|}~
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz plus some control
characters. Millions of users need other characters.

Yes, ASCII is a standard. That doesn't mean it's adequate. It isn't.

Mail readers and news readers have to be able to display o-umlaut, for
example. Yes, this is a change from the ASCII world. The ASCII world is
obsolete.

Yes, changing software is expensive. But this is a case where huge costs
are outweighed by huge benefits. Furthermore, sensible design---settling
on Unicode instead of a mess of ad-hoc character sets, and settling on
UTF-8 instead of a mess of ad-hoc character encodings---has considerably
reduced the costs. A huge amount of software has already been upgraded.
Many users are working comfortably in Unicode environments. 

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

content-class: urn:content-classes:message
Importance: normal
Subject: RE: Call for Usefor to recharter
Date: Sat, 11 Jan 2003 13:37:58 -0800
From: "Dan Kohn" <[email-masked]>
To: "Erland Sommarskog" <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Erland Sommarskog wrote:

> Punycode is useless, because it is not compatible with anything. Sure,
> nothing breaks, but a newsgroup name must be display well and a name
> in Punycode gets wrecked beyond recognition, even for Latin scripts.
> Unless, of course, you have a reader that understands Punycode, but
> there is no such thing today. So it is not very likely that any
> hierarchy would actually use it.

Our disagreement comes down to the definition we're each using for
"compatible".  I would say that punycode is compatible with all MIME,
DNS, and Internet application protocols, because it is 7 bits, and so it
transits them unchanged.  (In fact, it only uses LDH - letters, digit,
hyphens.)  Now, it's true that the non-LDH characters encoded in
punycode look like gobbledygook on non-punycode-aware readers.  But, the
punycode text is not going to be corrupted by gateways or non-punycode
aware user agents.  User agents that understand it can provide good
i18n; the rest of the agents still work fine.

And though no news reader currently supports punycode, it's difficult to
imagine that it will not be ubiquitously deployed within a year or two,
as it will be necessary to display international domain names (IDNs).
IDNs will be supported widely before usefor becomes a proposed standard,
especially at usefor's current rate of progress.

By contrast, raw UTF-8 may work fine for some newsreaders (like
Mozilla), but it is *incompatible* for others, and especially for
gateways (like IMAP servers).  Incompatible means that it breaks
software that had been correctly designed to work with previous versions
of the standard.

Note that, as in this case, elegance and compatibility are often at
odds.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Sat, 11 Jan 2003 12:57:27 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: [email-masked] (Erland Sommarskog)
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Erland,

Saturday, January 11, 2003, 12:22:56 PM, you wrote:
>> Cramming those bits into a 7-bit environment is just one more cramming
>> effort.  It stands equal to the others as an alternative that has benefits
>> and detriments.

Erland> Why let it stop with 7 bits? Why not cram it into one bit, while you're
Erland> at it?

If this is is an example of the way discussions have gone on usefor, then
it's pretty clear why the technical work is in trouble.

7-bit netascii is a long-standing internet norm.  that's why.

it would be extremely helpful for this topic to benefit from attention being
paid to deployed, real-world standards-based installed base.

>> If you do not have UTF-8 or UTF-16 tools, you cannot view the data at all.

Erland> Incorrect. If you have an 8-bit editor that does not understand UTF-8,
Erland> you will see the text, but it will look "ugly".

it would be extremely helpful for this topic to benefit from attention being
paid to deployed, real-world standards-based installed base.

Erland> The same applies if you have a any-bit editor which does not understand
Erland> RFC2047.

Erland> But:

Erland> 1) UTF-8 looks less ugly than RFC2047 when incorrectly displayed.

Unfortunately, this effort is not a computer science theoretical exercise.

it would be extremely helpful for this topic to benefit from attention being
paid to deployed, real-world standards-based installed base.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Sat, 11 Jan 2003 21:51:48 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: Call for Usefor to recharter
Sender: [email-masked]

Erland Sommarskog a écrit:
> Furthermore, there are newsreaders today that understands UTF-8.

Today for all headers except the newsgroups one, Mozilla does exactly 
what is suggested by the current USEFOR draft.
It will interpret the header as UTF-8 if it's valid UTF-8, and then try 
a local encoding, with a very configurable algorithm and possible 
autodetection (even without counting the UTF-8 aspect, only 
Emacs-GNUS/Agent are as configurable, and they don't have the universal 
autodetection available in Mozilla).

Many other readers can be configured to understand UTF-8, but usually 
will not be able to do that and simultanessly be able handle data in 
local encoding.
Such data is so common that it is an annoying restriction.

It might unnecessary to continue the ietf-822 cross-adressing in this 
discussion ? (I keeped it mostly because I've already said the above in 
the usefor ML)

Date: 11 Jan 2003 20:44:09 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Bruce Lilly writes:
> could you please explain the circumstances under which some individual
> would need to edit an encoded-word which is part of another person's
> display name in a header field

I press ``f'' in my mailer. It creates a tentative followup to the
message I'm looking at, and invokes my text editor. I then edit the
message to taste.

Sometimes I edit header fields. For example, I might change the Subject
line. What do you think happens if I correct the spelling of an RFC 2047
French word with an accent? That's right: I'm editing an encoded word.

Of course, display is critical. I need to read what I'm editing. I can't
read RFC 2047 gobbledygook. My text editor is not going to support RFC
2047 and IDNA GoofyCode and all the other complicated ad-hoc ``7 bits
forever!'' character encodings. The crucial advantage of UTF-8 is its
generality: it can and will be used everywhere.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 11 Jan 2003 20:22:56 -0000
To: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: [email-masked] (Erland Sommarskog)
Sender: [email-masked]

[Usefor folks: Apologies for sending this twice, but the first attempt
didn't make it to ietf-822, as I am not a subscriber there.]

Dave Crocker <[email-masked]> writes:
> Hence, UTF-16 and UTF-8 are methods of encoding a larger bit space into a
> smaller representation space, producing variable-length strings.  One crams
> the larger space into a 16-bit world.  The other crams it into an 8-bit
> world.
>
> There depending on the situation, there can be processing or space
> efficiencies gained by one encoding over another.
>
> But there is no theoretical or aesthetic superiority that can be claimed by
> one over the other.
>
> Cramming those bits into a 7-bit environment is just one more cramming
> effort.  It stands equal to the others as an alternative that has benefits
> and detriments.

Why let it stop with 7 bits? Why not cram it into one bit, while you're
at it?

> The confusion on this issue probably stems from the fact that you can use
> existing data viewers -- such as text editors -- to view the result of a
> 7-bit encoding and cannot use such "legacy" services for viewing UTF-8 or
> UTF-16.
>
> If you do not have UTF-8 or UTF-16 tools, you cannot view the data at all.

Incorrect. If you have an 8-bit editor that does not understand UTF-8,
you will see the text, but it will look "ugly".

The same applies if you have a any-bit editor which does not understand
RFC2047.

But:

1) UTF-8 looks less ugly than RFC2047 when incorrectly displayed.
2) UTF-8 editors are commonly available. RFC2047-capable editors are not.

And if you only have a 7-bit editor? Then you are grossly out of date
anyway.
--
Erland Sommarskog, Stockholm, [email-masked]

Date: 11 Jan 2003 20:22:40 -0000
To: [email-masked], [email-masked]
Subject: RE: Call for Usefor to recharter
From: [email-masked] (Erland Sommarskog)
Sender: [email-masked]

[Usefor folks: Apologies for sending this twice, but the first attempt
didn't make it to ietf-822, as I am not a subscriber there.]

"Dan Kohn" <[email-masked]> writes:
> By contrast, if i18n of news were accomplished with 2047/2231 encoding
> (plus punycode for newsgroup names), all existing gateways (including
> those used by moderators!) would keep working as usual, 

Yes, it would, but there is even simpler path to achieve the same
effect: do nothing.

Punycode is useless, because it is not compatible with anything. Sure,
nothing breaks, but a newsgroup name must be display well and a name
in Punycode gets wrecked beyond recognition, even for Latin scripts.
Unless, of course, you have a reader that understands Punycode, but
there is no such thing today. So it is not very likely that any
hierarchy would actually use it.

UTF-8 names does of course look garbled in a reader that does not under-
stand UTF-8, but for languages using Latin scripts, the name is fairly
likely to be comprehensible. For other scripts, though, the UTF-8 name
is completely opaque.

Furthermore, there are newsreaders today that understands UTF-8. I can
read UTF-8 from trn through my SSH client. Not because trn knows UTF-8,
but because my SSH client does.

And that's an important thing that the mail people have difficulties to
understand: UTF-8 is a much more widespread encoding that is used far
beyond the realm of news and mail, whereas RFC2047 or Punycode is not.

We are not looking for a token solution that provides internationalization
in theory, but once that can be used in reality.
--
Erland Sommarskog, Stockholm, [email-masked]

Date: 11 Jan 2003 20:23:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein <[email-masked]> schrieb/wrote:
> Bruce Lilly writes:
>> and in fact Usenet abounds with untagged charsets
> Obviously we can't make all of them work simultaneously. The way out of
> this mess is for message readers to support UTF-8---as many implementors
> have already done---so that message writers can safely use UTF-8.

Remember that UTF-8 and RFC 2047 are completly orthagonal features,
which solve different problems.

UTF-8 (or any other Unicode transformation) solves the multiple charset
problem: Instead of a large number of different charset that have to be
supported, you only have one.
RFC 2047 solves two other problems: Charset (and language) tagging and
encoding 8bit data in 7bit.

This gives us four different possibilities:

1. Unencoded leagacy 8bit charsets: Clearly a mess because of the
missing charset tagging. Users have to set the charset on a per-news-
group basis; unworkable for email with contacts from different locales.

2. RFC 2047-encoded leagacy 8bit charsets: An improvement over #1
because user agents can detect the charset automatically. (I assume that
most current user agents already support RFC 2047, whose ancestor is
nearly a decade old.) However, user agents that don't know a charset
can't display an encoded character even if it's available on the current
system.

3. RFC 2047-encoded UTF-8: Similar to #2 but it solves the problem of
having multiple charsets. It can coexist with #2. Most recent user
agents already support both UTF-8 and RFC 2047, so they don't have a
problem with this.

4. Unencoded UTF-8: Granted, the most appealing solution. It can coexist
with #2 and #3. It could also coexist with #1 because UTF-8 has a
distinctive structure that could be described as self-tagging.
The problem is that this is not currently implemented: Current
newsreaders have a setting for a 'default' or fallback charset (i.e. the
legacy charset from #1) but they can't automatically detect UTF-8 from
leagacy encodings. This also rules out MIME- and charset-ignorant user
agents: These can't handle two different charsets -- leagacy and UTF-8
-- at the same time. In mail, unencoded UTF-8 will further cause more
problems than other 8bit charsets because it makes use of the range
0x80..0x9F.
Then, of course, it's incompatible with current standards and there's no
way to define a confined environment (like the 8BITMIME world) as a
translation without potential data loss is impossible.
Further, the Unicode language tags are deprecated in favour of other  
means of language tagging.

So #3 finally wins over #4: It's already standardised, it's implemented
and it does not cause problems with software that can't handle the full
8bit range.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: 11 Jan 2003 20:29:00 +0200
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Bruce Lilly
Sender: [email-masked]

Bruce Lilly <[email-masked]> schrieb/wrote:
> I don't follow your argument; could you please explain the circumstances
> under which some individual would need to edit an encoded-word which is
> part of another person's display name in a header field.  There is no
> need to decode and re-encode in order to use a From field content to generate
> a new To, Cc, etc. field.

Note that this is a pseudo problem: If a user agent copies data from a
header field of the original message, that header field is already known
to the implementors of that user agent. So the user agent has enough
information on the syntax of this header to do a conversion between to  
encoded and a self-defined display format.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: Sat, 11 Jan 2003 09:08:01 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Charles Lindsey wrote:

> [utf-8] plays well with NNTP, and it plays well in the bodies of articles in
> SMTP where 8BITMIME is supported.

The 8ITMIME *extension* is not supported everywhere.  And while utf-8, or
any other 8-bit charset, can be used in the bodies of articles and in
the bodies of MIME message parts. it cannot be used without encoding via
the established RFC 2047/2231 methods in message headers or in MIME entity
headers.

Date: Sat, 11 Jan 2003 08:57:56 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: prevervation of installed base
Sender: [email-masked]

Charles Lindsey wrote:

> Therefore, it is reasonable to proceed on the basis that existing gateways
> will either drop non-conforming input on the floor, or they will pass it
> on in a form that will be obviously "wrong" (i.e. will appear as
> gobbledegook) wherever it is viewed.
> 
> The question is whether that situation is acceptable in a transitional
> phase,

You're talking about gateways to email -- what "transitional phase" -- what
are you assuming will change in the email protocols?

 > and my argument is that it is acceptable insofar as it only arises
> where the new functionality has been used, that the buildup of the new
> functionality will be slow,

What new functionality? Oh right, there isn't any.

 > that it can be tolerated to a degree in some
> environments, and that where it cannot be tolerated the gateways can be
> upgraded before the scale of the problem becomes serious.

The issue is not confined to gateways; the current Usefor draft has
serious implications for user agents, vaious bits of news-specific
software, and gateways.  Worse, the Usefor draft's proposed scheme is
completely unworkable w.r.t. gateways, so there is absolutely no way
that those "gateways can be upgraded" -- there is no magic.

Date: Sat, 11 Jan 2003 08:48:51 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:

> And you have completely missed the point, which is that these situations,
> which are not supposed to happen, WILL happen in the real world.

Please be precise about which "situations" you are talking about.  Are
you now saying that, in contradiction to your earlier claim, "The
occurrence in a Netnews article of a Return-Path containing non-ASCII
within a comment" is a cause for concern?

> So you cannot say "this will never happen, so we need do nothing about
> it". You have to specify _something_.
> 
> So the possibilities are:
> 1. Use some heuristic that works most of the time
> 2. Drop it on the floor (i.e. it works none of the time)
> and another possibility that occurs to me:
> 3. Convert it to an X- header (stick and "X-" on the front) and convert
> that according to RFC 2047. At least that way no information gets lost.

I'm sorry Charles, but if you wish to have any sort of meaningful discussion,
you'll have to provide some relevant context.  Are you talking about some
scenario which hasn't been presented -- if so, please specify.  OTOH, if
this is more of the same "untagged data from UA, magic happens in gateway
to provide proper tags", forget it -- that has been shown to be unworkable.

Date: Sat, 11 Jan 2003 08:40:37 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> The _real_ problem is that they are not using UTF-8. They are using
> whatever character set they fancy, and too bad if the recipient cannot
> guess which it is.

I.e. the *real* problem is lack of tagging.  Pretending that all untagged
content is utf-8 isn't consistent with reality.

> However, that is a problem for the email community.

Not really, since the mainstream mail user agents (mwny of which also
happen to be news user agents) already provide proper encoding and
tagging in header phrases and comments using the RFC 2047 methods. The
untagged non-ASCII content is primarily confined to spam; spammers
tend to use software which is writeen by largely clueless people as
is evident by the large number of syntax errors in spam. Indeed, those
errors (including untagged non-ASCII content) are a good indicator of
spam.

> My concern is to see
> that Usenet does not get dragged down into the same pit, by laying out a
> coherent plan for Usenet.

It appears that Usenet is being dragged into a deeper, darker pit, and
I have not yet seen any sort of coherent plan in the Usefor drafts --
only untested, half-baked, and/or unworkable proposals.

> Yes, fixing this situation for either Usenet or for Email is going to
> cause some pain, but sooner or later that pain is gping to have to be
> faced. Better to do it sooner, and try to minimize it so far as can be.
> But do not kid yourselves that it can be avoided. It cannot.

Leaving aside spam, the solution is to bring rogue software into
compliance with the standards.  Substituting one untagged charset for
another untagged charset and/or substituting an untagged encoding (utf-8
is both a charset and an encoding) for an existing mechanism which
provides for tagged charset and encoding, while preventing language tagging
which is provided for by the existing mechanisms certainly will cause pain,
but cannot "fix" anything.  It can only make matters worse.

Date: Sat, 11 Jan 2003 08:21:39 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

>>Obviously if the Unicode consortium states unequivocally that there are
>>multiple utf-8 specifications which differ, there cannot be "precicely
>>one" utf-8 specification.
> 
> 
> But it doesn't. Unicode 3.2 is the current standard, and defines precisely
> what is allowed in UTF-8. All earlier versions are obsolete.

So overnight all of the MS Windows 95, 98, NT, and 2000 implementations
mysteriously vanished?

When precisely did RFC 2279 become obsoleted?

Date: Sat, 11 Jan 2003 08:18:34 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>RFC 2047 encoded-words should be decoded for diplay only, and non-ASCII
>>text should always remain in encoded form in header fields.
> 
> 
> Yes, we agree that is the ideal situation, and that any semantic
> interpretation should use the original form. Actually, that is an
> irrelevance because RFC 2047 is only used for things (phrases, comments,
> etc) which do not need to be semantically interpreted.
> 
> But for some purposes (followups) the header gets used as the basis of
> headers in a new message, and at that point the starting point has to be
> the decoded form (since how else is the user to edit the new message).

I don't follow your argument; could you please explain the circumstances
under which some individual would need to edit an encoded-word which is
part of another person's display name in a header field.  There is no
need to decode and re-encode in order to use a From field content to generate
a new To, Cc, etc. field.

>> That is
>>clearly the intent of RFC 2047.  The design flaw, as has been already
>>mentioned, is decoding encoded-words for transmission.
> 
> 
> But nobody is proposing to do that.

The Usefor draft permits it by allowing raw untagged 8-bit data to be
transmitted between generating user agents and gateways.

content-class: urn:content-classes:message
Importance: normal
Subject: RE: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Date: Fri, 10 Jan 2003 21:31:04 -0800
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>
Sender: [email-masked]

Bruce Lilly wrote:

> 3. it illustrates the horribly baroque Unicode 3.x encoding of
>     characters making up ISO 639 language tags. One of the objections
>     to RFC 2047 voiced has been that it uses some additional octets
>     compared to raw 8-bit data -- Unicode 3.x langauage tags are much
>     worse as (via utf-8) they transform the 7-bit characters which
>     comprise ISO 639 language tags into long sequences of octets.
> 4. If there is a proposal to use untagged utf-8 instead of properly
>     tagged (RFC 2047 / 2231) charsets (including but not limited to
>     utf-8), then the entirety of the repercussions of such a proposal
>     ought to be considered, and language tagging has long been part
>     of MIME, but is a recent incompatible (among Unicode versions)
>     addition to Unicode, and one which is also incompatible (per
>     Unicode standards) with MIME.

At <http://www.unicode.org/review/>, the Unicode consortium has for
public review to "Deprecate the Plane 14 Language Tags".

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 10 Jan 2003 19:31:38 -0800
Lines: 33
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> "D. J. Bernstein" <[email-masked]> writes:

>> Here's a better transition plan:

>>   (1) Teach every message reader to accept UTF-8 in message/rfc822.
>>       Some readers do this already.
>>   (2) Allow message writers to generate UTF-8 in message/rfc822.

> It's not a question of "allowing" them to generate such thing. The plain
> fact of the matter is that they WILL write such things.

> No, that is not right either. They already ARE writing such things, and
> they are not going to stop just because the IETF wrings its hands.

> The _real_ problem is that they are not using UTF-8.

You just contradicted yourself.

Whether or not clients are inserting random untagged 8-bit character sets
into message/rfc822 messages is another matter entirely.  I have seen no
evidence presented that they're currently putting UTF-8 there; in fact,
use of UTF-8 in news messages and I expect e-mail messages as well (which
are not the same thing; maybe you don't understand what message/rfc822 is)
seems quite small.

The reason why they're not is because, as djb points out, there's a
transition that has to happen.  *First* you upgrade readers to support
UTF-8, and *then* you allow message writers to generate it.  The readers
have, by and large, not yet been upgraded.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Xref: clerew local.mime:2127
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: prevervation of installed base
Date: Fri, 10 Jan 2003 16:47:56 GMT
Lines: 45
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>>>With respect to the appearance of new capabilities (such as new data formats
>>>and encodings) in the installed base, your use of the term "gobbledegook"
>>>needs to be made more precise. If it means that that a legacy system
>>>receives the data and the data are legal but meaningless -- MIME base64 is
>>>example of this approach -- then it is fine. It permits incremental adoption
>>>without breaking existing systems. The downside for existing systems is that
>>>they do not get the benefit of the new feature, but everything else
>>>continues to work fine.

>Charles> Yes, that is exactly what I mean. If someone in the newsgroup suddenly
>Charles> starts using raw UTF-8 in his name (a phrase) or in his Subject, the
>Charles> existing gateway is unlikely to fall over and collapse.

>unlikely.

Yes, if someone has written a gateway that dumps core whenever it receives
illegal input, then I have no sympathy with him. And if such gateways
currently exist, then they must already be dumping core on a regular
basis, because there is already a very considerable amount of 8-bit
headers travelling around on Usenet.

Therefore, it is reasonable to proceed on the basis that existing gateways
will either drop non-conforming input on the floor, or they will pass it
on in a form that will be obviously "wrong" (i.e. will appear as
gobbledegook) wherever it is viewed.

The question is whether that situation is acceptable in a transitional
phase, and my argument is that it is acceptable insofar as it only arises
where the new functionality has been used, that the buildup of the new
functionality will be slow, that it can be tolerated to a degree in some
environments, and that where it cannot be tolerated the gateways can be
upgraded before the scale of the problem becomes serious.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2126
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Fri, 10 Jan 2003 16:31:49 GMT
Lines: 37
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>>  The occurrence in a Netnews article of a
>> Return-Path containing non-ASCII within a comment is as unlikely as they
>> come.

>You have completely missed the point.  That point is that your
>"heuristic" is inherently unworkable. Any scheme that requires
>encoding content via standard (e.g RFC 2047, 2231, 2396) methods
>for header fields requires that the syntax of the header field
>be known. 

And you have completely missed the point, which is that these situations,
which are not supposed to happen, WILL happen in the real world.

So you cannot say "this will never happen, so we need do nothing about
it". You have to specify _something_.

So the possibilities are:
1. Use some heuristic that works most of the time
2. Drop it on the floor (i.e. it works none of the time)
and another possibility that occurs to me:
3. Convert it to an X- header (stick and "X-" on the front) and convert
that according to RFC 2047. At least that way no information gets lost.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2125
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Fri, 10 Jan 2003 15:58:28 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>Here's a better transition plan:

>   (1) Teach every message reader to accept UTF-8 in message/rfc822.
>       Some readers do this already.
>   (2) Allow message writers to generate UTF-8 in message/rfc822.

It's not a question of "allowing" them to generate such thing. The plain
fact of the matter is that they WILL write such things.

No, that is not right either. They already ARE writing such things, and
they are not going to stop just because the IETF wrings its hands.

The _real_ problem is that they are not using UTF-8. They are using
whatever character set they fancy, and too bad if the recipient cannot
guess which it is. There is just a chance that, if some coherent plan were
to be laid out involving the use of UTF-8, it _might_ just be in time to
stop the rot. But it will not be stopped by continually preaching RFC
2047/2232/punycode/whatever-the-next-fudge-is to them.

However, that is a problem for the email community. My concern is to see
that Usenet does not get dragged down into the same pit, by laying out a
coherent plan for Usenet.

Yes, fixing this situation for either Usenet or for Email is going to
cause some pain, but sooner or later that pain is gping to have to be
faced. Better to do it sooner, and try to minimize it so far as can be.
But do not kid yourselves that it can be avoided. It cannot.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2123
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Fri, 10 Jan 2003 15:40:58 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>RFC 2047 encoded-words should be decoded for diplay only, and non-ASCII
>text should always remain in encoded form in header fields.

Yes, we agree that is the ideal situation, and that any semantic
interpretation should use the original form. Actually, that is an
irrelevance because RFC 2047 is only used for things (phrases, comments,
etc) which do not need to be semantically interpreted.

But for some purposes (followups) the header gets used as the basis of
headers in a new message, and at that point the starting point has to be
the decoded form (since how else is the user to edit the new message).

>  That is
>clearly the intent of RFC 2047.  The design flaw, as has been already
>mentioned, is decoding encoded-words for transmission.

But nobody is proposing to do that.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2128
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Date: Fri, 10 Jan 2003 16:53:18 GMT
Lines: 32
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>Charles> There are some practical superiorities. It transports well over many
>Charles> Internet protocols, because it does not do nasty thungs with CR, LF and
>Charles> NUL. It has the _whole_ of ASCII as a strict subset (which is not true of
>Charles> UTF-7).

>Usually, the "plays well with other protocols" argument is given for use of
>netAscii, which is a subset of 7-bit.  I have not seen it used for 8-bit
>before.

It plays well with NNTP, and it plays well in the bodies of articles in
SMTP where 8BITMIME is supported. I.e. you can use CTE 8bit where the
charset is UTF-8 where you could not use it if the charset was UTF-16.

>Charles> But UTF-8 would be a pain for internal use in Operating systems,

>I thought we were talking about an exchange protocol, not o/s data storage.

Sure. I was just pointing out that there were other places (not
tnansmission) where UTF-16 was more suitable than UTF-8.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2124
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Fri, 10 Jan 2003 15:45:52 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles, in spite of having been shown the differences in the past, you persist
>in claiming that there are none.  RFC 2044 is not a Unicode document, and has
>long been obsoleted.

Yes, I should have RFC 2279 rather than 2044.

>  One clue that you are wrong is the following quotation
>from Unicode Technical report 28:

>"Most notable among the corrigenda to the Standard is a further tightening of the definition of UTF-8, to eliminate irregular UTF-8 and to bring the Unicode specification of UTF-8 more completely into line with other specifications of UTF-8. "

>Obviously if the Unicode consortium states unequivocally that there are
>multiple utf-8 specifications which differ, there cannot be "precicely
>one" utf-8 specification.

But it doesn't. Unicode 3.2 is the current standard, and defines precisely
what is allowed in UTF-8. All earlier versions are obsolete.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
From: Andrew Gierth <[email-masked]>
Date: 11 Jan 2003 02:29:45 +0000
Lines: 91
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 Bruce> U+DBC0 U+DC01 U+DBC0 U+DC65 U+DBC0 U+DC6E U+0066 U+006F U+006F
 Bruce> U+DBC0 U+DC01 U+DBC0 U+DC7F
 
 Bruce> That has 5 surrogate pairs which were not in the original
 Bruce> Unicode 3.1 text.

 >> but those surrogate pairs are precisely the ones that represent the
 >> original codepoints. i.e. if you merely converted the original
 >> character sequence into UTF-16, you would get the same result.
 
 >> The fact that those codepoints may not be known to the receiving
 >> application is to do with the difference in the Unicode specification
 >> itself, and nothing to do with UTF-8.

 Bruce> It has to do with the difference in the two utf-8
 Bruce> specifications, specifically for 4-byte sequences, which in
 Bruce> one case (but not the other) involve surrogate pairs.  The
 Bruce> original had 8 "characters" (actually at most 5 could really
 Bruce> be called characters in any meaningful sense; the others are
 Bruce> syntactic glue), whereas the result has 13 16-bit
 Bruce> "characters", 10 of which are "garbage".

The original, if expressed in 16-bit form (i.e. in UTF-16), would also
have had the exact same sequence of 16-bit values.

The difference in the number of characters and whether or not they are
"garbage" is solely due to the difference in Unicode versions and not on
the transport format used; there is no way to encode a Unicode 3.1 text
that contains codepoints not used in Unicode 2.0 in such a way that this
will not happen.

 Bruce> One "utf-8" encoding which transforms between uniform-width
 Bruce> 16-bit "characters" and a sequence of bytes cannot be said to
 Bruce> be "the same as" another "utf-8" specification which
 Bruce> transforms between variable-width "characters" and a series of
 Bruce> bytes; they are different because the domain of one side of
 Bruce> the transformation differs.

But this is a difference of _description_ not a genuine difference in
the underlying definition. If I can transform A<->C by one definition
and B<->C by another, where the A<->B transformation is known and
well-defined, then the two definitions are logically equivalent even
though they have different domains and describe different functions.
Either definition can be obtained from the other merely by composing
it with the A<->B transformation or its inverse.

You can describe UTF-8 either in terms of its relationship to UTF-16
or in terms of its relationship to UCS-4. (The older definition you
quoted from the Unicode specs does the former; other definitions,
including the later Unicode specs do the latter.) These descriptions
will look different, but only until you also factor in the UCS-4 to
UTF-16 transformation at which point they become equivalent.

Which transformation function you actually _use_ then of course
depends on what your desired domain is.

 >> [snip language-tagging stuff irrelevent to this issue]

 Bruce> The language tagging is relevant for a number of reasons:

it's irrelevent to my argument and to the question of whether there
is more than one version of UTF-8 (independently of the question of
whether there is more than one version of Unicode which is obviously
true).

I am not trying to argue for use of utf-8 language tags (or even for
raw utf-8 in headers shared with email - I'll leave that position for
others to take).

 >> it encodes the exact same sequence of codepoints regardless of
 >> specification. The fact that some of those codepoints are not defined
 >> in earlier unicode versions is nothing to do with the use of UTF-8.

 Bruce> It is related to the different utf-8 specifications inasmuch
 Bruce> as the different specifications in conjunction with the change
 Bruce> from a 16-bit pre-3.1 character width and the newly-introduced
 Bruce> codepoints (including language tags) together result in
 Bruce> obfuscated content with a different number of bits.  It's
 Bruce> difficult to cleanly separate those issues, since if musical
 Bruce> notes, baroque encoding of language tags etc. hadn't been
 Bruce> added (deviating from the original Unicode Design Priciples),
 Bruce> the character width wouldn't have had to be increased beyond
 Bruce> 16 bits

sure, and the addition of about 43 thousand more Han ideographs had
nothing to do with the increase in number of bits. Right.

-- 
Andrew.

Date: Fri, 10 Jan 2003 20:08:21 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

On Fri, 10 Jan 2003 15:26:03 GMT
"Charles Lindsey" <[email-masked]> wrote:

> 
> In <[email-masked]> Keith Moore <[email-masked]> writes:
> 
> >> There are many more pieces of software than Usefor gateways which get
> >> stuck because they do not know the correct way to parse some field. The
> >> design flaw was in RFC 2047 which allowed that situation to be gotten into
> >> in the first place.
> 
> >rfc 2047 doesn't affect how fields are parsed.  it only affects how
> >fields are displayed.  if a viewer doesn't realize that a field 
> >is structured and/or doesn't know how it should be parsed, the worst
> >that should happen is that encoded-words are not decoded before display.
> 
> The problem is not so much with software that decodes RFC 2047 (though
> even that gets it wrong sometimes), as with software that encodes it in
> the first place and which gets stuck as soon as it encounters a header
> that it does not know how to parse.

Does this same software "get stuck" as soon as it encounters some other
syntax error in the input?  After all, a non-ascii character in the message
header of an input message is a violation of the message syntax.
Some errors are recoverable with a reasonable degree of safety - so for
instance you can often make sense of a malformed rfc822 style date
even if the syntax is completely wrong.  Similarly, you can make sense of
some header fields which contain non-ascii characters by encoding them
per 2047.  But that doesn't mean you can recover from all inappropriate
uses of non-ascii characters.  

For example, until we have a specification for use of non-ascii characters
in email addresses, there's no way that you can convert an input message
that contains a non-ascii character in an email address into a valid message.
"getting stuck" (or hopefully, reporting an error) is the best you can do.

> The standard of implementation of RFC 2047 in current software is
> exceedingly poor. The reason seems to be that implementors find that doing
> it correctly is so burdensome that they decide to cut corners. You would
> say that it is all the wicked inplementors' fault. Other would say it
> illustrates that it is a bad standard.

it may be a bad standard, and at the same time, it may be (close to) optimal.
the two are not mutually exclusive.

Keith

Date: Fri, 10 Jan 2003 19:45:30 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: old format date fields (was Re: RFC 2047 and gatewaying)
Sender: [email-masked]

> >>Non-numeric offsets have been in the "do not generate" category (see RFC 2822)
> >>for more than 20 months.
> > 
> > 
> > well, your software still needs to be able to read the things - otherwise how are
> > you going to read old email?
> 
> Yes, it does need to do so, and it does do so.  Nevertheless, messages generated
> recently, now, and in the future, including Charles', are supposed to use
> numeric offsets 

uh, no.  software that claims to conform to RFC 2822 is supposed to use
numeric offsets. 

Keith

Date: Fri, 10 Jan 2003 19:29:28 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Andrew Gierth wrote:

> since there have been no substantive changes made to UTF-8 _ever_
> since its adoption as any sort of standard, why would you expect
> future changes to introduce incompatibilities?

Bad premise; every time Unicode changes substantively, the utf-8
specification necessarily also changes since it is a Unicode to/from
octet stream transformation.  That definitely happened when Unicode
went from 3.0 to 3.1, and there is no guarantee that it won't happen
again.

When the Unicode consortium decides to include chicken scratching as
"characters" and extends the maximum width to beyond 32 bits, even
the 5- and 6-byte sequences (which exist in some "utf-8" specifications
on the octet-stream side, but not in others) will have to change.
In light of the 3.0->3.1 changes, somehow a smiley doesn't seem
appropriate...

Date: Fri, 10 Jan 2003 19:12:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Andrew Gierth wrote:
>>>>>>"Bruce" == Bruce Lilly <[email-masked]> writes:

> that was hardly an answer to my question of Sep 30th which was part
> of a different branch of the discussion.

An example of a difference serves as that, and I was not inclided to waste
bandwidth repeating the same example in such a short span of time.  In any
event you now have it again.

>  Bruce> U+E0001 U+E0065 U+E006E U+0066 U+006F U+006F U+E0001 U+E007F
> 
>  Bruce> Transformed to Unicode 3.1 UTF-8 (hex values for 8-bit codes):
> 
>  Bruce> F3 A0 80 81 F3 A0 81 A5 F3 A0 81 AE foo F3 A0 80 81 F3 A0 81 8F
> 
>  Bruce> The Unicode 2.0 UTF-8 reverse transformation yields:
> 
>  Bruce> U+DBC0 U+DC01 U+DBC0 U+DC65 U+DBC0 U+DC6E U+0066 U+006F U+006F
>  Bruce> U+DBC0 U+DC01 U+DBC0 U+DC7F
> 
>  Bruce> That has 5 surrogate pairs which were not in the original
>  Bruce> Unicode 3.1 text.
> 
> but those surrogate pairs are precisely the ones that represent the
> original codepoints. i.e. if you merely converted the original
> character sequence into UTF-16, you would get the same result.
 >
> The fact that those codepoints may not be known to the receiving
> application is to do with the difference in the Unicode specification
> itself, and nothing to do with UTF-8.

It has to do with the difference in the two utf-8 specifications, specifically
for 4-byte sequences, which in one case (but not the other) involve surrogate
pairs.  The original had 8 "characters" (actually at most 5 could really be
called characters in any meaningful sense; the others are syntactic glue),
whereas the result has 13 16-bit "characters", 10 of which are "garbage".

One "utf-8" encoding which transforms between uniform-width 16-bit "characters"
and a sequence of bytes cannot be said to be "the same as" another "utf-8"
specification which transforms between variable-width "characters" and a series
of bytes; they are different because the domain of one side of the transformation
differs.  Now you're saying that with yet another transformation, one can
get from a to c by way of an intermediate q, but that is not the same as saying
that a->b (Unicode 3.1+ to "utf-8") and b->c ("utf-8" to Unicode 3.0 and earlier)
are parts of the "same" transformation.  The "utf-8" transformation could be
said to be "the same" in that case if and only if a == c, but since a and c are
quite different, the two "utf-8" transformations are necessarily different.

>  [snip language-tagging stuff irrelevent to this issue]

The language tagging is relevant for a number of reasons:
1. it bears on the disadvantages of untagged utf-8 (vs. properly
    RFC 2047/2231 tagged utf-8)
2. it is an example of the differences in the utf-8 specifications
    (since the 4-byte sequences which differ are involved)
3. it illustrates the horribly baroque Unicode 3.x encoding of
    characters making up ISO 639 language tags. One of the objections
    to RFC 2047 voiced has been that it uses some additional octets
    compared to raw 8-bit data -- Unicode 3.x langauage tags are much
    worse as (via utf-8) they transform the 7-bit characters which
    comprise ISO 639 language tags into long sequences of octets.
4. If there is a proposal to use untagged utf-8 instead of properly
    tagged (RFC 2047 / 2231) charsets (including but not limited to
    utf-8), then the entirety of the repercussions of such a proposal
    ought to be considered, and language tagging has long been part
    of MIME, but is a recent incompatible (among Unicode versions)
    addition to Unicode, and one which is also incompatible (per
    Unicode standards) with MIME.

>  Bruce> That of course is not a parsing issue, but one of semantic
>  Bruce> interpretation of the "utf-8".  According to one of the many
>  Bruce> "utf-8" specifications, the "utf-8" stream encodes a
>  Bruce> language-tagged string, while in at least one other "utf-8"
>  Bruce> specification it encodes something quite different.
> 
> it encodes the exact same sequence of codepoints regardless of
> specification. The fact that some of those codepoints are not defined
> in earlier unicode versions is nothing to do with the use of UTF-8.

It is related to the different utf-8 specifications inasmuch as the
different specifications in conjunction with the change from a 16-bit
pre-3.1 character width and the newly-introduced codepoints (including
language tags) together result in obfuscated content with a different
number of bits.  It's difficult to cleanly separate those issues, since
if musical notes, baroque encoding of language tags etc. hadn't been
added (deviating from the original Unicode Design Priciples), the
character width wouldn't have had to be increased beyond 16 bits and the
utf-8 4-byte sequences wouldn't need to be specified differently. It
is precisely because the Unicode versions are different that the "utf-8"
transformations specifications are necessarily different.

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Andrew Gierth <[email-masked]>
Date: 11 Jan 2003 00:03:27 +0000
Lines: 24
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 >> and as I also pointed out on USEFOR, the untagged utf-8 that
 >> _does_ appear can be distinguished from the other untagged 8-bit
 >> charsets by means of a trivial heuristic with an extremely low
 >> error rate (no false negatives, very few false positives).

 Bruce> Two observations:
 Bruce> 1. if only an insignificant amount (of untagged utf-8)
 Bruce> currently appears, extrapolating the rate of error rates is
 Bruce> quite risky; i.e. one might find much higher error rates if
 Bruce> more untagged utf-8 were used.

the error rate of false _positives_ (data assumed to be utf-8 which in
fact is not) is not dependent on the amount of utf-8 data present; in
fact precisely the reverse, it depends on the amount of non-utf-8
unlabelled 8-bit, of which I had a large sample to work from.

The false-negative error rate would depend on the amount of utf-8 data
present if it were not for the fact that it is zero by definition (since
the heuristic is "assume the data is utf-8 if in fact it is valid utf-8").

-- 
Andrew.

Date: Fri, 10 Jan 2003 18:15:06 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Andrew Gierth wrote:
>>>>>>"Bruce" == Bruce Lilly <[email-masked]> writes:
> 
> 
>  Bruce> That would only work if it could be absolutely guaranteed that
>  Bruce> no untagged data in any other character set might appear.  No
>  Bruce> such guarantee is possible, and in fact Usenet abounds with
>  Bruce> untagged charsets, of which, according to Andrew Gierth, "no
>  Bruce> significant amount" is utf-8.
> 
> and as I also pointed out on USEFOR, the untagged utf-8 that _does_
> appear can be distinguished from the other untagged 8-bit charsets by
> means of a trivial heuristic with an extremely low error rate (no
> false negatives, very few false positives).

Two observations:
1. if only an insignificant amount (of untagged utf-8) currently appears,
    extrapolating the rate of error rates is quite risky; i.e. one might
    find much higher error rates if more untagged utf-8 were used.
2. non-zero error rates are probably acceptable for non-critical
    purposes (e.g. display), but are generally unacceptable for critical
    use (as in transmission via gateways to/from domains where strict
    transmission protocols are in effect).

Date: Fri, 10 Jan 2003 18:07:43 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein wrote:
> Bruce Lilly writes:
> 
>>But that is only for display -- hardly a critical issue.
> 
> 
> Display is a critical issue. The reason that we aren't satisfied with
> ASCII is that it doesn't let us display most of the glyphs that people
> want to see.

I didn't say it was unimportant; I said it was not critical in the
same way that transmission protocol issues are critical. Display
issues can be resolved largely independently from other issues and
infrequently involve compatibility or interoperation issues
(unfortunately due to differences in utf-8 specifications and Unicode
versions, such issues *do* affect utf-8 encoded Unicode, but
they do not affect transmission protocol issues in Internet text
messages).

>>user-defined fields are required to begin with "X-"
> 
> 
> False. See RFC 822.

False only if one is willing to have a user-defined field pre-empted
by a properly-standardized extension field with the same name. "X-"
is used specifically to prevent name space clashes of that sort. And
the distiction between user-defined and extension fields is not
present in RFC 2822.

 > Anyway, this is irrelevant to the discussion. As
> you've now admitted, the locations of RFC 2047 encoded words are not
> stable: they are affected by RFCs specifying new header fields.

So what's your point? If a newly-defined Internet text message header
field isn't recognized by a particular reader UA but contains 2047/2231
encoded-words:
1. no transmission issues are affected
2. there are no gateway issues
3. that UA can be upgraded without affecting any other software,
    without affecting message delivery, gateway operations, etc.
4. since 2047 applies only to comments and phrases for display,
    there can be no adverse effect on any protocol-related issues;
    parameters in currently-defined header fields which permit
    parameters and can use encoded-words aren't supposed to be
    automatically used for anything (i.e. w/o user confirmation),
    and it would be unwise for any newly defined header to set
    such a precedent.
5, The reader UA can either assume the field is unstructured and look
    for encoded-words accordingly or it can present the header field
    text verbatim. Or (mind you, we're talking about for *display* only)
    it could even use a Charles Lindsey-style heuristic. It's not
    *critical*.
6. the (human) reader can in any event look at the raw content and
    determine (a) the charset, (b) the language (where applicable),
    (c) the encoding method, and (d) the encoded text.  In raw
    (untagged) presumed-utf-8, he only can determine the encoded-text
    which might not be utf-8 at all.
7. Congratulations -- you have just added to the argument that the
    Usefor draft is unworkable; every time a new field is introduced
    all gateways would have to be instantly updated for the Usefor
    draft proposal to be workable, and that is clearly impossible.

How frequently are new structured fields that might use encoded
words standardized?  Not terribly often -- certainly a subset
of structured fields (e.g. Content-Features cannot have encoded-words),
which in turn is a subset of all new fields.  How frequently are
ambiguities w.r.t. 2047/2231 encodings introduced by such new structured
field definitions? Rarely -- the only example I can think of which *might*
be ambiguous is RFC 3335's Received-Content-MIC field, which uses the ABNF
element "value" (by way of RFC 1847 "micalg") outside of a MIME
parameter (for which "value" was defined in the MIME base RFCs and
changed by RFC 2231; "micalg" is not used elsewhere other than in
MIME parameters); as such "value" as modified by RFC 2231 might include
an RFC 2231 extended values.  Does Received-Content-MIC affect Usenet? No.
Is it likely that an RFC 2231 extended value will have to be used in a
Received-Content-MIC field due to (a) charset for a micalg, (b) langauge,
or (c) parameter length? No, no, and no.  Is it likely that an end user
(as opposed to a developer of application software using Received-Content-MIC
fields) will need to display Received-Content-MIC fields decoded? Never.

Date: Fri, 10 Jan 2003 17:15:48 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein wrote:
> Bruce Lilly writes:
> 
>>and in fact Usenet abounds with untagged charsets
> 
> 
> Obviously we can't make all of them work simultaneously. The way out of
> this mess is for message readers to support UTF-8---as many implementors
> have already done---so that message writers can safely use UTF-8.

We can't make *any* untagged non-ascii charsets work simultaneously for
all users.  Message writers can safely use utf-8 now, as with any other
charset, by specifying "utf-8" as the charset with the mechanisms defined
in RFC 2047 as amended by RFC 2231 and errata. Message readers ought to
support RFC 2047 / 2231 in any event, and are expected to support properly-
tagged utf-8, as they are used with email, which is a much larger application
base than Usenet. Neither Usenet nor Usefor can hope to force raw untagged
utf-8 on that application base, which includes gateways and combined mail/news
user agents.  Drop the raw utf-8 nonsense and move on to somthing that has
a non-zero chance of baing (a) standardized and (b) implemented.

>>Who said anything about gateways having to use language information?
> 
> 
> You said that the user agent ``must preserve ... language information in
> some way that it can be used by gateways'' or must use RFC 2047.

... in order for gateways to be able to reconstruct the language information
for RFC 2047 encoding as required by the Usefor draft, where desired for
header information.
   Subject: =?iso-8859-1*en?q?boot?=
and
   Subject: =?iso-8859-1*de?q?boot?=
mean quite different things.  They are pronounced differently, and that is
a consideration for the sight-impaired users who use screen reader software.
Clearly it is simple, practical, and compatible to use proper 2047 / 2231
tagging including language where applicable than attempt the unworkable,
circuitous, complex Usefor-draft-recommended method of dropping the charset
and language information in the generating UA for untagged utf-8, then
attempting to recreate that information out of thin air in a gateway which
has no access to the charset amd language information known to the generating
UA, and might not have header field syntax information.  The gateway itslef
never has any direct need for language information, and I never claimed
otherwise.  The language information should be preserved end-to-end from
generating UA to receiving UA, and untagged utf-8 in compliance with Unicode
standards and across different Unicode versions provides no way to do so. RFC
2047 / 2231 do provide such a mechanism.

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 10 Jan 2003 13:57:03 -0800
Lines: 15
Sender: [email-masked]

D J Bernstein <[email-masked]> writes:

> Obviously we can't make all of them work simultaneously. The way out of
> this mess is for message readers to support UTF-8---as many implementors
> have already done---so that message writers can safely use UTF-8.

For what it's worth, I agree with this.  I think that this is the best way
to move forward in the long term.  However, client authors don't seem to
be rushing in this direction, and it's unclear to me how best to encourage
them.  I also am a bit at a loss as to how best to handle these issues in
the short to medium term before all of the client authors have updated
their software.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 10 Jan 2003 21:33:01 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Bruce Lilly writes:
> and in fact Usenet abounds with untagged charsets

Obviously we can't make all of them work simultaneously. The way out of
this mess is for message readers to support UTF-8---as many implementors
have already done---so that message writers can safely use UTF-8.

> Who said anything about gateways having to use language information?

You said that the user agent ``must preserve ... language information in
some way that it can be used by gateways'' or must use RFC 2047. This is
silly.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 10 Jan 2003 21:24:33 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Bruce Lilly writes:
> But that is only for display -- hardly a critical issue.

Display is a critical issue. The reason that we aren't satisfied with
ASCII is that it doesn't let us display most of the glyphs that people
want to see.

Maybe you've never used anything beyond ASCII. Maybe you think of all
these character-set extensions as abstract games having nothing to do
with text displayed on the screen. Millions of users disagree with you.

> user-defined fields are required to begin with "X-"

False. See RFC 822. Anyway, this is irrelevant to the discussion. As
you've now admitted, the locations of RFC 2047 encoded words are not
stable: they are affected by RFCs specifying new header fields.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Andrew Gierth <[email-masked]>
Date: 10 Jan 2003 20:57:31 +0000
Lines: 15
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 Bruce> That would only work if it could be absolutely guaranteed that
 Bruce> no untagged data in any other character set might appear.  No
 Bruce> such guarantee is possible, and in fact Usenet abounds with
 Bruce> untagged charsets, of which, according to Andrew Gierth, "no
 Bruce> significant amount" is utf-8.

and as I also pointed out on USEFOR, the untagged utf-8 that _does_
appear can be distinguished from the other untagged 8-bit charsets by
means of a trivial heuristic with an extremely low error rate (no
false negatives, very few false positives).

-- 
Andrew.

Date: Fri, 10 Jan 2003 15:55:16 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein wrote:
> Bruce Lilly writes:
> 
>>Display (as opposed to generation) does not require information about
>>header field syntax
> 
> 
> False. See RFC 2047, section 6.1. Encoded words can appear as
> 
>    * substrings (up to 75 bytes) of fields defined as unstructured,
>    * words inside phrases in fields defined as structured, and
>    * substrings of comments in fields defined as structured.

It is true (as RFC 2047 states) that encoded-words may appear in
those places and that parsing recognized header fields per RFC 822
rules is necessary to locate those encoded words unambiguously. But
that is only for display -- hardly a critical issue.

> False. You are confusing extension fields with user-defined fields; see
> RFC 822. Every structured extension field changes the set of locations
> where RFC 2047 encoded words can appear.

I assure you *I* am not in the least confused by the distinction. That
some may be confused is another matter; e.g. to use the terminology of
your earlier message, if "some nitwit defines and deploys a new header
field", such as "Mail-Followup-To", that would be an extension field
*if and only if standardized in an RFC*, and is clearly not a user-defined
field since user-defined fields are required to begin with "X-".

For purpose of display, it is perfectly acceptable to treat an unrecognized
field as unstructured when looking for encoded-words; the worst that can
happen is the display might show some garbage which was not intended to be
an encoded-word but which matched the syntax for an encoded-word in *text
context.  It's also acceptable to simply ignore looking for encoded-words
in unrecognized fields (for purpose of display).  The issue is strictly
limited to display. It is also limited to structured fields that might
contain encoded words (structured fields that provide for neither phrase
nor comments cannot contain encoded words). And implementation issues are
strictly limited to user agents.

All of which is entirely different from the unworkable Usefor draft which
as currently worded requires gateways to do the impossible, viz. to construct
correct header fields containing encoded-words for unrecognized header fields
for transmission.

To: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
From: Andrew Gierth <[email-masked]>
Date: 10 Jan 2003 20:52:21 +0000
Lines: 47
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 Bruce> I did not decline; indeed in
 Bruce> <[email-masked]> posted to the Usefor list
 Bruce> Sun, 22 Sep 2002 12:36:13 -0400 I worote:

that was hardly an answer to my question of Sep 30th which was part
of a different branch of the discussion.

 [snip]

 Bruce> U+E0001 U+E0065 U+E006E U+0066 U+006F U+006F U+E0001 U+E007F

 Bruce> Transformed to Unicode 3.1 UTF-8 (hex values for 8-bit codes):

 Bruce> F3 A0 80 81 F3 A0 81 A5 F3 A0 81 AE foo F3 A0 80 81 F3 A0 81 8F

 Bruce> The Unicode 2.0 UTF-8 reverse transformation yields:

 Bruce> U+DBC0 U+DC01 U+DBC0 U+DC65 U+DBC0 U+DC6E U+0066 U+006F U+006F
 Bruce> U+DBC0 U+DC01 U+DBC0 U+DC7F

 Bruce> That has 5 surrogate pairs which were not in the original
 Bruce> Unicode 3.1 text.

but those surrogate pairs are precisely the ones that represent the
original codepoints. i.e. if you merely converted the original
character sequence into UTF-16, you would get the same result.

The fact that those codepoints may not be known to the receiving
application is to do with the difference in the Unicode specification
itself, and nothing to do with UTF-8.

 [snip language-tagging stuff irrelevent to this issue]

 Bruce> That of course is not a parsing issue, but one of semantic
 Bruce> interpretation of the "utf-8".  According to one of the many
 Bruce> "utf-8" specifications, the "utf-8" stream encodes a
 Bruce> language-tagged string, while in at least one other "utf-8"
 Bruce> specification it encodes something quite different.

it encodes the exact same sequence of codepoints regardless of
specification. The fact that some of those codepoints are not defined
in earlier unicode versions is nothing to do with the use of UTF-8.

-- 
Andrew.

Date: Fri, 10 Jan 2003 15:18:52 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein wrote:
> Bruce Lilly writes:
> 
>>For gateways to work at all, either the user agent must preserve charset
>>and language information in some way that it can be used by gateways
> 
>   [ or it must use RFC 2047 ]
> 
> Specifying ``UTF-8 everywhere'' removes the need to add charset tags to
> individual pieces of data. The message writer converts the local charset
> to UTF-8 and then discards the local charset information.

That would only work if it could be absolutely guaranteed that no untagged
data in any other character set might appear.  No such guarantee is possible,
and in fact Usenet abounds with untagged charsets, of which, according to
Andrew Gierth, "no significant amount" is utf-8.

> As for languages: Exactly which gateways use language information?
> Anyway, the notion that a gateway _has to_ use language information---
> information that almost no messages have---is patently absurd.

Who said anything about gateways having to use language information? I
certainly did not.  But the receiving UA may make use of such information
(e.g. in a Subject field), and if the language information is discarded by
a generating UA, no gateway can hope to reconstruct that information unless
it is conveyed to the gateway by the generating UA by some mechanism. Hence
the requirements excerpted at the top of this message.

Date: Fri, 10 Jan 2003 11:56:49 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Friday, January 10, 2003, 11:33:24 AM, you wrote:
D.> Specifying ``UTF-8 everywhere'' removes the need to add charset tags to
D.> individual pieces of data.

Except in any of the places in the installed base that are not expecting
UTF-8.

Oh.  That's right.  That means every place using current standards.

oops.

d/

-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: 10 Jan 2003 19:33:24 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Bruce Lilly writes:
> For gateways to work at all, either the user agent must preserve charset
> and language information in some way that it can be used by gateways
  [ or it must use RFC 2047 ]

Specifying ``UTF-8 everywhere'' removes the need to add charset tags to
individual pieces of data. The message writer converts the local charset
to UTF-8 and then discards the local charset information.

As for languages: Exactly which gateways use language information?
Anyway, the notion that a gateway _has to_ use language information---
information that almost no messages have---is patently absurd.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 10 Jan 2003 19:07:35 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Bruce Lilly writes:
> Display (as opposed to generation) does not require information about
> header field syntax

False. See RFC 2047, section 6.1. Encoded words can appear as

   * substrings (up to 75 bytes) of fields defined as unstructured,
   * words inside phrases in fields defined as structured, and
   * substrings of comments in fields defined as structured.

This is, as I said, an extremely complicated set of locations. Even
worse, as I said, it is an unstable set of locations.

> RFC 2047 explicitly provides for considering
> unrecognized fields as unstructured *for the purpose of display*.

False. You are confusing extension fields with user-defined fields; see
RFC 822. Every structured extension field changes the set of locations
where RFC 2047 encoded words can appear.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Fri, 10 Jan 2003 14:04:06 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Keith Moore <[email-masked]> writes:
> 
> 
>>>There are many more pieces of software than Usefor gateways which get
>>>stuck because they do not know the correct way to parse some field. The
>>>design flaw was in RFC 2047 which allowed that situation to be gotten into
>>>in the first place.
> 
> 
>>rfc 2047 doesn't affect how fields are parsed.  it only affects how
>>fields are displayed.  if a viewer doesn't realize that a field 
>>is structured and/or doesn't know how it should be parsed, the worst
>>that should happen is that encoded-words are not decoded before display.
> 
> 
> The problem is not so much with software that decodes RFC 2047 (though
> even that gets it wrong sometimes), as with software that encodes it in
> the first place and which gets stuck as soon as it encounters a header
> that it does not know how to parse.

An entity (software or human) must have syntax information in order to
correctly generate *any* header with *any* content (including RFC 2047
encoded-words).  Your sentence quoted directly above therefore makes no
sense -- an entity cannot generate a header with unknown syntax.

The root problem is once again the Usefor draft, which is unworkable
because:
1. it permits user agents to generate "header" fields with untagged raw
    non-ASCII content, discarding any charset and language information
    which the user agent ought to know. [obviously, the user agent must
    have syntax information to be able to construct such a "header", unless
    it permits the user to input raw text for "header" fields, in which
    case the user must know the relevant syntax.]
2. it then requires gateways to magically construct a valid RFC 2822 + MIME
    header field from that gobbldeygook, with no charset or language
    information, and possibly for some "header" field unrecognized by the
    gateway.

That can't be done. End of the line. Back to the drawing board.

For gateways to work at all, either the user agent must preserve charset
and language information in some way that it can be used by gateways *and*
gateways need to have syntax information for any possible header that might
have non-ASCII content generated by any such user agent, *or* the user agent
must use a compatible means of generating the "header" field in the first
place (viz. RFC 2047 / 2231), in which case there is no additional effort --
and no magic -- required of the gateways.  In the first case, there is the
additional issue of existing gateways, which requires some reasonable
transition plan whereby the mechanism for conveying preserved charset and
language information and the header field syntax information can be
incorporated into those gateways.  Obviously the second method is far
simpler to implement and is compatible with existing gateways and is
consistent with the current article format standard (RFC 1036) and is
compatible with existing software which handles mail and news (including
combined user agents). But "the Usefor WG chose not to do that"; well,
the method in the current Usefor draft simply cannot work, and the Usefor
WG will have to reconsider. Any scheme that requires encoding separate
from header field generation will have the same problems, for the same
reasons.

Date: Fri, 10 Jan 2003 13:36:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Andrew Gierth wrote:
>>>>>>"Bruce" == Bruce Lilly <[email-masked]> writes:
> 
> 
> I repeatedly asked you to be specific on the USEFOR list when you
> started making claims about differing UTF-8 versions there; since
> you declined to respond, I will deal with it here instead.

I did not decline; indeed in <[email-masked]>
posted to the Usefor list Sun, 22 Sep 2002 12:36:13 -0400
I worote:

"It should also be noted that there is not a single definition of
UTF-8. For example, the definition and implementation of UTF-8
described in Unicode <= 3.0 has no provision for Unicode values
  > 16 bits, so cannot represent the language tags. That means
that if a Unicode 3.1 application inserts such tags, which are
then transmitted as Unicode 3.1 UTF-8 to a Unicode 3.0 or earlier
platform, the latter will see "garbage characters" as described in
Unicode tr20. Example: English text "foo" tagged per Unicode 3.1:

U+E0001 U+E0065 U+E006E U+0066 U+006F U+006F U+E0001 U+E007F

Transformed to Unicode 3.1 UTF-8 (hex values for 8-bit codes):

F3 A0 80 81 F3 A0 81 A5 F3 A0 81 AE foo F3 A0 80 81 F3 A0 81 8F

The Unicode 2.0 UTF-8 reverse transformation yields:

U+DBC0 U+DC01 U+DBC0 U+DC65 U+DBC0 U+DC6E U+0066 U+006F U+006F U+DBC0 U+DC01 U+DBC0 U+DC7F

That has 5 surrogate pairs which were not in the original
Unicode 3.1 text.

Some equivalent RFC 2047 codings of English "foo", suitable for use
in common headers (Subject, display name in From, Reply-To, etc.) are:

1. =?us-ascii*en?q?foo?=
2. =?utf-8?q?=F3=A0=80=81=F3=A0=81=A5=F3=A0=81=AEfoo=F3=A0=80=81=F3=A0=81=8F?=

Obviously the first is
a) much more compact
b) much easier for a reader with no Unicode or MIME support to
    determine the language
c) much easier for a reader with no Unicode or MIME support to read

Note also that the second just fits in the 75-character limit for an
encoded-word; if a text string longer than 3 characters were encoded,
or if a longer language tag were used, multiple encoded-words would be
required.  Note also that any header line containing an encoded-word
must be 76 characters or shorted (not including CRLF), so if the
second version were part of any header, it would have to appear alone
on a continuation line.  But the second form should never appear,
since Unicode 3.1 prohibits use of language tags in conjunction with
MIME or in the absence of ACAP. "

That of course is not a parsing issue, but one of semantic interpretation
of the "utf-8".  According to one of the many "utf-8" specifications, the
"utf-8" stream encodes a language-tagged string, while in at least one
other "utf-8" specification it encodes something quite different.

Date: Fri, 10 Jan 2003 13:21:41 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Andrew Gierth wrote:
>>>>>>"Bruce" == Bruce Lilly <[email-masked]> writes:

>  Bruce>  From the point of view of parsing some stream of octets,
>  Bruce> according to one "utf-8" specification a certain sequence *is*
>  Bruce> a utf-8 sequence, and according to other "utf-8"
>  Bruce> specifications is is *not* a utf-8 sequence.
> 
> If instead you ask the question "is this a sequence of Unicode (or
> ISO10646) characters encoded in UTF-8", then the difference between
> versions disappears. (Given a 5-byte sequence, you can either reject
> it as not in the spec, or decode it to the appropriate codepoint and
> then reject _that_ as being out of range.)

That's an irrelevant question to the issue of parsing a stream of
octets.  You're talking about semantic interpretation of what has
presumably already been parsed.

> The distinction is solely in whether you regard utf-8 as a
> representation of Unicode characters, or as a way to encode arbitrary
> 31-bit values.

That too is irrelevant; for parsing an octet stream, what is
represented by the stream (again, semantic interpretation rather
than syntactic) is not of concern.

>  Bruce> I stand by my assertion that (from the aforementioned point of
>  Bruce> view) there are multiple incompatible "utf-8" specifications.
> 
> You are splitting a theoretical hair that is irrelevent in practice.

It is quite relevant to parsing, which precedes semantic interpretation.

Date: Fri, 10 Jan 2003 13:08:33 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D. J. Bernstein wrote:
> In a world that makes any use of RFC 2047, displaying text correctly
> means identifying the locations where RFC 2047 encoding is used.

Display (as opposed to generation) does not require information about
header field syntax; RFC 2047 explicitly provides for considering
unrecognized fields as unstructured *for the purpose of display*.

> Even worse, the definition is unstable. Consider a mail client that does
> a perfect job of decoding RFC 2047 in mail messages today. As soon as
> some nitwit defines and deploys a new header field using the RFC 2047
> encoding, that mail client will start displaying messages incorrectly.

Not necessarily.  Even if that were correct, "incorrect display" is
not nearly as serious as a protocol violation.

> This is the design flaw under discussion.

No, the design flaw is an attempt to use RFC 2047 in gateways to modify
header field content *for transmission* (which is not a *display* issue),
based on untagged raw 8-bit content.  RFC 2047 encoding must be used
during header field generation, where field syntax, charset, and language
(if relevant) are known or can be unambiguously determined.

To: [email-masked]
Xref: clerew local.mime:2122
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Fri, 10 Jan 2003 15:26:03 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> There are many more pieces of software than Usefor gateways which get
>> stuck because they do not know the correct way to parse some field. The
>> design flaw was in RFC 2047 which allowed that situation to be gotten into
>> in the first place.

>rfc 2047 doesn't affect how fields are parsed.  it only affects how
>fields are displayed.  if a viewer doesn't realize that a field 
>is structured and/or doesn't know how it should be parsed, the worst
>that should happen is that encoded-words are not decoded before display.

The problem is not so much with software that decodes RFC 2047 (though
even that gets it wrong sometimes), as with software that encodes it in
the first place and which gets stuck as soon as it encounters a header
that it does not know how to parse.

The standard of implementation of RFC 2047 in current software is
exceedingly poor. The reason seems to be that implementors find that doing
it correctly is so burdensome that they decide to cut corners. You would
say that it is all the wicked inplementors' fault. Other would say it
illustrates that it is a bad standard.

The plain fact of the matter is that standards are no use unless the world
can be persuaded that there is benefit to be gained by adhering to them.
Ultimately, it is the world, not the standards making bodies, that
decides.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 10 Jan 2003 09:43:13 +0100
From: Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: prevervation of installed base
Sender: [email-masked]

On Thu, Jan 09, 2003 at 10:43:52PM -0800, Russ Allbery wrote:
> 
> Andrew Gierth <[email-masked]> writes:
> 
> > the most important technical difference (as I personally see it - my
> > opinions may differ from others involved in USEFOR) is that gateways for
> > individual groups or hierarchies do not have to contend with the use of
> > non-ASCII _newsgroup names_ unless they specifically choose to do
> > so. (In such cases there is generally no particular need to preserve the
> > Newsgroups header accurately in the case of crossposts to groups not
> > handled by the same gateway; furthermore, if the gateway is implemented
> > via the moderation mechanism, this becomes a non-issue anyway).

I agree that there is probably not many general news-mail gateways
around, but there are a lot of specific mail-news/news-mail gateways,
and many of them involve constituencies where non-ascii newsgroup names
may become relevant. The new USEFOR scheme should cater for standard 
gatewaying for all these individual cases.

Kind regards
Keld

To: [email-masked]
Subject: Re: prevervation of installed base
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 09 Jan 2003 22:43:52 -0800
Lines: 32
Sender: [email-masked]

Andrew Gierth <[email-masked]> writes:

> the most important technical difference (as I personally see it - my
> opinions may differ from others involved in USEFOR) is that gateways for
> individual groups or hierarchies do not have to contend with the use of
> non-ASCII _newsgroup names_ unless they specifically choose to do
> so. (In such cases there is generally no particular need to preserve the
> Newsgroups header accurately in the case of crossposts to groups not
> handled by the same gateway; furthermore, if the gateway is implemented
> via the moderation mechanism, this becomes a non-issue anyway).

This is an extremely good point, which I'd like to highlight as well.

In general, the only existing news to mail gatewaying that would
potentially need to deal with non-ASCII *newsgroup names* is the
moderation relaying.  (Although there is some possibility that non-ASCII
newsgroup names would show up for other gateways in crossposts or in a
header like Followup-To, those headers are generally meaningless in mail
and therefore losing them or mangling them causes no major and likely few
minor problems.)

The case for mail compatibility is much, much stronger for Subject, From,
and other such shared headers than it is for Newsgroups, Followup-To,
Control, and other similar Usenet-only headers that contain newsgroup
names.  (I think there is still some case for mail compatibility with the
latter headers as well, on the basis that it's very convenient to write
software that works seamlessly with both mail and news, but there's a lot
more room for various approaches other than trying to strictly follow the
same rules one would follow for a mail header.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
From: Andrew Gierth <[email-masked]>
Date: 10 Jan 2003 06:32:20 +0000
Lines: 30
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 >> that is because Unicode does not have any codepoints above
 >> U+10FFFF, and therefore such sequences could not be generated. I
 >> understand that there are also no assigned ISO 10646 codepoints
 >> above 10FFFF, and also that there will be no future assignments of
 >> such.

 Bruce>  From the point of view of parsing some stream of octets,
 Bruce> according to one "utf-8" specification a certain sequence *is*
 Bruce> a utf-8 sequence, and according to other "utf-8"
 Bruce> specifications is is *not* a utf-8 sequence.

If instead you ask the question "is this a sequence of Unicode (or
ISO10646) characters encoded in UTF-8", then the difference between
versions disappears. (Given a 5-byte sequence, you can either reject
it as not in the spec, or decode it to the appropriate codepoint and
then reject _that_ as being out of range.)

The distinction is solely in whether you regard utf-8 as a
representation of Unicode characters, or as a way to encode arbitrary
31-bit values.

 Bruce> I stand by my assertion that (from the aforementioned point of
 Bruce> view) there are multiple incompatible "utf-8" specifications.

You are splitting a theoretical hair that is irrelevent in practice.

-- 
Andrew.

To: [email-masked]
Subject: Re: prevervation of installed base
From: Andrew Gierth <[email-masked]>
Date: 10 Jan 2003 06:26:09 +0000
Lines: 73
Sender: [email-masked]

>>>>> "Dave" == Dave Crocker <[email-masked]> writes:

 >>> To add to Ned's correction, news-> email gatewaying has been
 >>> around for at least 20 years, as far as I can
 >>> recall. Organizations have often wanted to plug mailing lists
 >>> into newsgroups. To permit the newsgroup readers to participate
 >>> in the mailing list, a news->email gateway is required. In other
 >>> environments, the 2-way gatewaying is simply part of a model that
 >>> lets users decide how they want to receive and process their
 >>> group discussion messages.

 Charles> Yes indeed, but those are not GENERAL-PURPOSE gateways.

 Dave> I have no idea was technical differences you think exist
 Dave> between whatever it is you have in mind and the examples I
 Dave> described.

 Dave> So, please characterize those technical differences, in enough
 Dave> detail to evalute their impact on the current work.

the most important technical difference (as I personally see it - my
opinions may differ from others involved in USEFOR) is that gateways
for individual groups or hierarchies do not have to contend with the
use of non-ASCII _newsgroup names_ unless they specifically choose to
do so. (In such cases there is generally no particular need to
preserve the Newsgroups header accurately in the case of crossposts to
groups not handled by the same gateway; furthermore, if the gateway is
implemented via the moderation mechanism, this becomes a non-issue
anyway). Furthermore, articles gatewayed from news to email via such
gateways are generally not expected to be gatewayed back into news _as
the same article_ (i.e. in the same group and with the same
message-id), and therefore the gateway does not need to satisfy the
requirements of a news transport. (If an article is to be gatewayed
news->email->news and arrive as the same article from the news
system's perspective, then the two gateways combined constitute a
relaying agent and must satisfy the requirements for such, which
include preserving all headers and body intact and unmolested except
for specific exemptions such as the Path header.)

The use of (unlabelled) 8-bit character sets in unstructured headers
like Subject and in phrases or comments in From headers is already
endemic in Usenet (and probably in email too, but it is harder to get
good statistics for that). Specifically, when I last measured,
something ike 17% of all text posts (excluding detectable spam) to
generally-propagated Usenet groups contained octets >127 in either
Subject or From. That figure increased to 30% if one excluded the
primary English-language hierarchies. The usage of RFC2047 was
substantially less; 2.7% and 4.6% respectively. (The actual split
between unlabelled 8-bit and RFC2047 varies quite widely by hierarchy,
but only in the Japanese-language hierarchies does RFC2047 truly
dominate.)

(Note, however, that no significant amount of this existing usage of
unlabelled 8-bit currently involves the use of utf-8; usually it is a
popular local charset. Introducing more unlabelled utf-8 would not
make this situation significantly worse; the "if it's valid utf-8 then
assume that's what it is" heuristic works extremely well.)

My own view is that I would be happy to see USEFOR's use of utf-8
limited to newsgroup names, with RFC2047 specified for use in those
headers (shared with email) in which it is valid. However, I don't
expect this to have much effect on existing usage; thus, it would be
fundamentally dishonest of USEFOR to in any way suggest that
implementors can disregard the issue of octets >127; such an
implementation would be useless in the real world.

(The question of what to do with newsgroup names themselves has been
extensively thrashed out on USEFOR; I strongly recommend that people
not try and jump into that one without looking at the previous
debate.)

-- 
Andrew.

Date: 10 Jan 2003 04:48:32 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Leo Bicknell writes:
> the subset of people who need to read news in e-mail

I don't care what set of messages you're talking about. My point is that

   (1) changing the readers of those messages to accept UTF-8 in a new
       message/utf8 and then
   (2) allowing the writers of those messages to send UTF-8 in
       message/utf8

is not as good as

   (1) changing the readers of those messages to accept UTF-8 in
       message/rfc822, as some readers do already, and then
   (2) allowing the writers of those messages to send UTF-8 in
       message/rfc822.

The second plan achieves the same effect at lower cost.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Thu, 09 Jan 2003 23:45:01 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Andrew Gierth wrote:
>>>>>>"Bruce" == Bruce Lilly <[email-masked]> writes:

>  Bruce> Clearly RFC 2279 provides for 5- and 6-byte utf-8 sequences,
>  Bruce> which are not provided for by Unicode through 3.2.
> 
> that is because Unicode does not have any codepoints above U+10FFFF,
> and therefore such sequences could not be generated. I understand that
> there are also no assigned ISO 10646 codepoints above 10FFFF, and also
> that there will be no future assignments of such.

 From the point of view of parsing some stream of octets, according to one
"utf-8" specification a certain sequence *is* a utf-8 sequence, and according
to other "utf-8" specifications is is *not* a utf-8 sequence.  I.e. one
cannot design a parser to recognize "utf-8" from a sequence of octets
unless one specifies *which* of the mutually-incompatible "utf-8"
specifications is to be used, viz. whether or not the 5- and 6-byte
sequnces are or are not "utf-8".

I stand by my assertion that (from the aforementioned point of view) there
are multiple incompatible "utf-8" specifications.

Date: Thu, 9 Jan 2003 19:46:39 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Charles Lindsey" <[email-masked]>
CC: [email-masked]
Subject: Re: prevervation of installed base
Sender: [email-masked]

Charles,

Wednesday, January 8, 2003, 2:58:57 PM, you wrote:
Charles> In <[email-masked]> Dave Crocker <[email-masked]> writes:
>>To add to Ned's correction, news-> email gatewaying has been around for at
>>least 20 years, as far as I can recall. Organizations have often wanted to
>>plug mailing lists into newsgroups. To permit the newsgroup readers to
>>participate in the mailing list, a news->email gateway is required. In other
>>environments, the 2-way gatewaying is simply part of a model that lets users
>>decide how they want to receive and process their group discussion messages.

Charles> Yes indeed, but those are not GENERAL-PURPOSE gateways.

I have no idea was technical differences you think exist between whatever it
is you have in mind and the examples I described.

So, please characterize those technical differences, in enough detail to
evalute their impact on the current work.

>>With respect to the appearance of new capabilities (such as new data formats
>>and encodings) in the installed base, your use of the term "gobbledegook"
>>needs to be made more precise. If it means that that a legacy system
>>receives the data and the data are legal but meaningless -- MIME base64 is
>>example of this approach -- then it is fine. It permits incremental adoption
>>without breaking existing systems. The downside for existing systems is that
>>they do not get the benefit of the new feature, but everything else
>>continues to work fine.

Charles> Yes, that is exactly what I mean. If someone in the newsgroup suddenly
Charles> starts using raw UTF-8 in his name (a phrase) or in his Subject, the
Charles> existing gateway is unlikely to fall over and collapse.

unlikely.

you seem to miss the point that the IETF does not make new standards that
break old standards.  parallel effort is fine, but you do not lay down a new
thing that violates an old one.

your use of "unlikely" basically ends this discussion.

standards work uses statistics to make guesses about needs, but it does not
use it to decide whether a technology is valid.  either it breaks things or
it doesn't.  "unlikely" is not in the vocabulary.  (it breaks the
discussion.)

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Thu, 9 Jan 2003 19:27:06 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

D.,

Thursday, January 9, 2003, 11:44:10 AM, you wrote:
D.> The crucial advantage of UTF-8 is that it will be used _everywhere_. The
D.> correct display algorithm is simple and stable.

I remember hearing that 10 years ago.  It still isn't true.

making decisions that rely on outside events going down a certain path,
within a certain time, establishes a critical dependency that frequently
proves fatal.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

To: [email-masked]
Subject: UTF-8 versions (was: Re: RFC 2047 and gatewaying)
From: Andrew Gierth <[email-masked]>
Date: 09 Jan 2003 23:25:57 +0000
Lines: 54
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

I repeatedly asked you to be specific on the USEFOR list when you
started making claims about differing UTF-8 versions there; since
you declined to respond, I will deal with it here instead.

 Bruce> "Most notable among the corrigenda to the Standard is a
 Bruce> further tightening of the definition of UTF-8, to eliminate
 Bruce> irregular UTF-8 and to bring the Unicode specification of
 Bruce> UTF-8 more completely into line with other specifications of
 Bruce> UTF-8. "

 Bruce> Obviously if the Unicode consortium states unequivocally that
 Bruce> there are multiple utf-8 specifications which differ, there
 Bruce> cannot be "precicely one" utf-8 specification.

The difference is solely between "illegal" and "irregular" sequences;
earlier versions of the UTF-8 definition in the Unicode spec made a
distinction between the two, though neither could be legally generated;
later versions remove the distinction.

 Bruce> Clearly RFC 2279 provides for 5- and 6-byte utf-8 sequences,
 Bruce> which are not provided for by Unicode through 3.2.

that is because Unicode does not have any codepoints above U+10FFFF,
and therefore such sequences could not be generated. I understand that
there are also no assigned ISO 10646 codepoints above 10FFFF, and also
that there will be no future assignments of such.

 Bruce> And some 4-byte sequences differ in different Unicode versions
 Bruce> (particulary those corresponding to surrogate pairs).

no, there are no 4-byte sequences that differ in any way between the
different versions.

The only difference has been in the wording describing isolated
constituents of a surrogate pair (i.e. values D800-DFFF, which would
encode into 3 bytes if that were allowed) which have been illegally
encoded. It has never been legal in any version of the Unicode spec to
encode the constituents of a surrogate pair into UTF-8 separately;
they must be encoded together as a single character (i.e.  decoded
into UCS-4 and then encoded as a 4-byte sequence). The only difference
is in the description of whether such sequences are "illegal" or
"irregular" or "ill-formed" or some such term.

 Bruce> Whether or not Unicode 4.0 and/or the draft mentioned above
 Bruce> will introduce additional variants is another matter.

since there have been no substantive changes made to UTF-8 _ever_
since its adoption as any sort of standard, why would you expect
future changes to introduce incompatibilities?

-- 
Andrew.

Date: Thu, 9 Jan 2003 15:48:52 -0500
From: Leo Bicknell <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: United Federation of Planets
Sender: [email-masked]

In a message written on Thu, Jan 09, 2003 at 08:15:38PM -0000, D. J. Bernstein wrote:
> To summarize your transition plan:
> 
>    (1) Teach every message reader how to parse message/utf8, just like
>        message/rfc822 but allowing UTF-8.

No, teach every message reader for the subset of people who need to read
news in e-mail.

>    (2) Allow message writers to generate UTF-8 using message/utf8.

This is only a requirement for message writers who need non-US-ASCII
characters, as the e-mail->news gateway can easily do a US-ASCII-UTF-8
conversion.

> Here's a better transition plan:
> 
>    (1) Teach every message reader to accept UTF-8 in message/rfc822.
>        Some readers do this already.
>    (2) Allow message writers to generate UTF-8 in message/rfc822.

No, that's a bad plan.  I will submit there are at least an order of
magnitude, and perhaps two or three more e-mail clients in the world
than 1) people who read news at all, or 2) people who read news
gatewayed to e-mail.

Any proposal dependant on all things that parse rfc822 today being able
to parse UTF-8 to make this work will fail.  If they don't fail in the
standards bodies, they will fail in the real world.  There's no way
Usenet/Usefor has enough leverage to change the e-mail world.

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

Date: 9 Jan 2003 20:15:38 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Leo Bicknell writes:
> Here's an interesting path forward

To summarize your transition plan:

   (1) Teach every message reader how to parse message/utf8, just like
       message/rfc822 but allowing UTF-8.
   (2) Allow message writers to generate UTF-8 using message/utf8.

Here's a better transition plan:

   (1) Teach every message reader to accept UTF-8 in message/rfc822.
       Some readers do this already.
   (2) Allow message writers to generate UTF-8 in message/rfc822.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 9 Jan 2003 19:44:10 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

In a world that makes any use of RFC 2047, displaying text correctly
means identifying the locations where RFC 2047 encoding is used. (Never
mind all the other complications in RFC 2047.) The definition of those
locations is extremely complicated---so complicated that, in practice,
nobody gets it right.

Even worse, the definition is unstable. Consider a mail client that does
a perfect job of decoding RFC 2047 in mail messages today. As soon as
some nitwit defines and deploys a new header field using the RFC 2047
encoding, that mail client will start displaying messages incorrectly.
This is the design flaw under discussion.

The crucial advantage of UTF-8 is that it will be used _everywhere_. The
correct display algorithm is simple and stable.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Thu, 09 Jan 2003 13:29:06 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:

>  The occurrence in a Netnews article of a
> Return-Path containing non-ASCII within a comment is as unlikely as they
> come.

You have completely missed the point.  That point is that your
"heuristic" is inherently unworkable. Any scheme that requires
encoding content via standard (e.g RFC 2047, 2231, 2396) methods
for header fields requires that the syntax of the header field
be known.  That will not be true for uny experimental header
under the currently-proposed Usefor draft scheme, which requires
gateways to encode untagged utf-8.  The issue is not, and never
has been, whether or not any particular header with specific
content might or might not appear in a news article.

Date: Thu, 09 Jan 2003 12:29:13 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>It's worse than that; there are at least 3 different versions of UTF-8.
>>They differ in the longer multi-byte sequences.
> 
> 
> No, there is precisely one, as defined by the relevant Unicode documents.
> See RFC 2044 and draft-yergeau-rfc2279bis-02.txt, with all of which Usefor
> is fully compatible.

Charles, in spite of having been shown the differences in the past, you persist
in claiming that there are none.  RFC 2044 is not a Unicode document, and has
long been obsoleted.  One clue that you are wrong is the following quotation
from Unicode Technical report 28:

"Most notable among the corrigenda to the Standard is a further tightening of the definition of UTF-8, to eliminate irregular UTF-8 and to bring the Unicode specification of UTF-8 more completely into line with other specifications of UTF-8. "

Obviously if the Unicode consortium states unequivocally that there are
multiple utf-8 specifications which differ, there cannot be "precicely
one" utf-8 specification.

Here, once again:

Unicode 2.0, table A-3 (applies through Unicode 3.0):

  Unicode Value     1st Byte  2nd Byte  3rd Byte  4th Byte
000000000xxxxxxx   0xxxxxxx
00000yyyyyxxxxxx   110yyyyy  10xxxxxx
zzzzyyyyyyxxxxxx   1110zzzz  10yyyyyy  10xxxxxx
110110wwwwzzzzyy +
110111yyyyxxxxxx   11110uuu  10uuzzzz  10yyyyyy  10xxxxxx

RFC 2044 is obsolete; here's the table from RFC 2279:

    UCS-4 range (hex.)           UTF-8 octet sequence (binary)
    0000 0000-0000 007F   0xxxxxxx
    0000 0080-0000 07FF   110xxxxx 10xxxxxx
    0000 0800-0000 FFFF   1110xxxx 10xxxxxx 10xxxxxx

    0001 0000-001F FFFF   11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
    0020 0000-03FF FFFF   111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
    0400 0000-7FFF FFFF   1111110x 10xxxxxx ... 10xxxxxx

Unicode 3.2, Unicode Technical Report #28:

  Code Points   1st Byte   2nd Byte   3rd Byte   4th Byte
U+0000..U+007F 00..7F
U+0080..U+07FF C2..DF 80..BF
U+0800..U+0FFF E0 A0..BF 80..BF
U+1000..U+CFFF E1..EC 80..BF 80..BF
U+D000..U+D7FF ED 80..9F 80..BF
U+D800..U+DFFF ill-formed
U+E000..U+FFFF EE..EF 80..BF 80..BF
U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
U+100000..U+10FFFF F4 80..8F 80..BF  80..BF

Clearly RFC 2279 provides for 5- and 6-byte utf-8 sequences, which are
not provided for by Unicode through 3.2.  And some 4-byte sequences
differ in different Unicode versions (particulary those corresponding
to surrogate pairs).  Whether or not Unicode 4.0 and/or the draft
mentioned above will introduce additional variants is another matter.

To: [email-masked]
Xref: clerew local.mime:2107
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Thu, 9 Jan 2003 10:46:29 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>> Now if I had
>> 	Return-Path: <(?-?-?-?)>
>> where "?-?-?-?" should be regarded as a placeholder for some sensible
>> comment in Hungarian, Arabic or Chinese, then my proposed workaround
>> would transform it into
>> 	Return-Path: =?iso-8859-1?q?<(=3F-=3F-=3F-=3F>)?=
>> which, I will grant you, is not a valid Return-Path

>As you may recall, that was in response to the lastest variant of your
>gateway "heuristic", in which you claimed that URIs should be bracketed
>by '<' and '>' on the supposed grounds that that would prevent
>misinterpreting parentheses in URIs as comments, further supposing
>(incorrectly) that comments could not appear within angle bracketed
>constructs.  So your gateway process (which you now call a "workaround")
>would in fact entirely miss the content within the angle brackets. And
>that proposed gateway heuristic simply won't work precisely because
>it does not take into account full syntax.

Exactly so. A "heuristic" is, by definition, a method that will get the
right answer in a sufficiently high proportion of cases to be useful.
Inevitably, it will fail in a few cases. The art of designing a heuristic
lies in knowing which cases are likely to happen in practice, so that the
benefit can be maximised. The occurrence in a Netnews article of a
Return-Path containing non-ASCII within a comment is as unlikely as they
come.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2106
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Call for Usefor to recharter
Date: Thu, 9 Jan 2003 10:40:48 GMT
Lines: 35
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>I have to say I find the recent claim that "in the reverse direction
>News->Email, such general-purpose gateways just Do Not Exist" to be pretty
>amazing, if for no other reason that I have years of direct personal experience
>with such gateways. For example, as many of you know, I used to be a frequent
>contributor to the info-pmdf mailing list, and that list was for several years
>gatewayed bidirectionally between netnews and email. (This particular gateway
>was discontinued in the news->email direction at some point, but other gateways
>of this sort continue to operate.)

Then please read my lips.

The important words in my claim were "such GENERAL-PURPOSE gateways".

The gateway into the info-pmdf mailing list that you mentioned was not a
"GENERAL-PURPOSE gateway". It was a special-purpose gateway for the
benefit of one newsgroup and one small group of users (and see my
respponse to Dave Crocker for the effect on such gateways).

A GENERAL-PURPOSE gateway would be one which was capable of handling and
Usenet newsgroup that was thrown at it. I repeat that such gateways do not
exist in practice (though they do exist in the mail->news direction, where
the problems are easier).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 09 Jan 2003 11:14:00 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: old format date fields (was Re: RFC 2047 and gatewaying)
Sender: [email-masked]

Keith Moore wrote:
> 
>>By the way, could you *please* fix or replace the software which is generating
>>the illegal Date headers in your messages, e.g.
>>Date: Mon, 6 Jan 2003 17:07:33 GMT
>>Non-numeric offsets have been in the "do not generate" category (see RFC 2822)
>>for more than 20 months.
> 
> 
> well, your software still needs to be able to read the things - otherwise how are
> you going to read old email?

Yes, it does need to do so, and it does do so.  Nevertheless, messages generated
recently, now, and in the future, including Charles', are supposed to use
numeric offsets (which in any event have been strongly recommended since at
least RFC 1123 ca. 1989).  One might hope that an upgrade would also correct
the annoying tendency of Charles' (and nobody else's) messages showing up with
Date fields that predate their origination by many hours (in some cases days).

Since Charles is using a news reader (viz. NN version 6.5.2 (NOV)), one wonders
whether or not there is more than lip service being paid to compliance with the
Internet text message standards and extensions.

Date: Thu, 09 Jan 2003 10:57:47 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Jean-Marc Desperrier <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Jean-Marc Desperrier wrote:
> Dan Kohn wrote :
> 
>> I wrote:
>>
>>>> I have a simple question.  What can a UTF-8 subject header
>>>> communicate that an RFC 2047 one can't?  Other than inelegance,
>>>> what's the downside of 2047, when the upside is a huge increase in
>>>> backward compatibility? 
> 
> 
> I do not know where this discussion took place, but I have an answer to it.

There has been substantial discussion on the ietf-822 mailing list. See
http://www.imc.org/ietf-822/mail-archive/maillist.html for the archive.

> It's a simple fact.
> In every single thread with non US-ASCII data in subject encoded by 
> RFC2047 (sorry I wrote 2049 by error in my last mail) I've seen, the 
> subject turned to garbage after 5 or 6 messages.
> 
> The reason for that is that all implementations of RFC2047 around are 
> full of implementation errors.

There is one specific error that could cause that, namely improper use
of RFC 2047 for any purpose other than for display.  Once a header
field has been generated with RFC 2047 encoded-words, those
encoded-words should never be modified -- the content may be decoded for
display in the specified charset (and optionally, language), but the
header content should not be modified.

Surely the solution to that problem is to sorrect the faulty implementations
which are responsible for inappropriately modifying header content.

> And here raw UTF-8 is a clear winner. No complex implementation rules, 
> no border cases, one string will always have one and only one 
> representation.

That is not correct, simply because Unicode has multiple representations,
including non-spcing modifiers.  Therefore a utf-8 encoding of unicode
will also have multiple representations. Unless, of course some additional
"complex implementation rules" are applied for normalization.  And the
longer utf-8 sequences are effectively border cases, since different utf-8
specifications have different rules.

Date: Thu, 9 Jan 2003 07:48:05 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Charles Lindsey" <[email-masked]>
CC: [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Sender: [email-masked]

Charles,

Wednesday, January 8, 2003, 3:12:47 PM, you wrote:
Charles> UTF-8 is space efficient if you still expect to have a large proportion of
Charles> ASCII mixed in with it. It compares quite well with UTF-16 for European
Charles> languages, but I can imagine it would be less efficient in Chinese.

yes, it is good to mention the language that has more native speakers than
any other language on the planet.

>>But there is no theoretical or aesthetic superiority that can be claimed by
>>one over the other.

Charles> There are some practical superiorities. It transports well over many
Charles> Internet protocols, because it does not do nasty thungs with CR, LF and
Charles> NUL. It has the _whole_ of ASCII as a strict subset (which is not true of
Charles> UTF-7).

Usually, the "plays well with other protocols" argument is given for use of
netAscii, which is a subset of 7-bit.  I have not seen it used for 8-bit
before.

Charles> But UTF-8 would be a pain for internal use in Operating systems,

I thought we were talking about an exchange protocol, not o/s data storage.

>>The confusion on this issue probably stems from the fact that you can use
>>existing data viewers -- such as text editors -- to view the result of a
>>7-bit encoding and cannot use such "legacy" services for viewing UTF-8 or
>>UTF-16.

Charles> There are many editors that will display _something_ useful for any 8-bit
Charles> stuff.

The concern is not whether there is an existence proof for the capability,
or even whether there is already a strong support base. The concern is what
happens to the platforms that do not yet support it.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Thu, 09 Jan 2003 10:45:05 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: prevervation of installed base
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Dave Crocker <[email-masked]> writes:

>>To add to Ned's correction, news-> email gatewaying has been around for at
>>least 20 years, as far as I can recall. Organizations have often wanted to
>>plug mailing lists into newsgroups. To permit the newsgroup readers to
>>participate in the mailing list, a news->email gateway is required. In other
>>environments, the 2-way gatewaying is simply part of a model that lets users
>>decide how they want to receive and process their group discussion messages.
> 
> 
> Yes indeed, but those are not GENERAL-PURPOSE gateways. They only have to
> be capable of meeting the needs of the newsgroup and the community of
> users they were designed to serve. That is a simpler problem than a
> gateway that will take any article whatsoever out of Usenet and produce a
> compliant email out of it.

Surely if the Usefor draft as it is currently formulated will have dire
consequenses for limited-purpose gateways, general-purpose gateways will
be at least as adversely affected.

>>With respect to the appearance of new capabilities (such as new data formats
>>and encodings) in the installed base, your use of the term "gobbledegook"
>>needs to be made more precise. If it means that that a legacy system
>>receives the data and the data are legal but meaningless -- MIME base64 is
>>example of this approach -- then it is fine. It permits incremental adoption
>>without breaking existing systems. The downside for existing systems is that
>>they do not get the benefit of the new feature, but everything else
>>continues to work fine.
> 
> 
> Yes, that is exactly what I mean. If someone in the newsgroup suddenly
> starts using raw UTF-8 in his name (a phrase) or in his Subject, the
> existing gateway is unlikely to fall over and collapse.

But that is *not* what the above quoted part of Dave's message says;
rather it corresponds to his text:

"If it means that the legacy system receives data that are illegal, with
respect to the existing service, then this is not at all fine.
Interoperability requires conforming to a standard, rather than conforming
to it whenever it is convenient. Injecting illegal data is not conforming.
"

 > But it will surely
> pass on something that has been munged, or will be munged when it arrives
> at some User Agent as a mail to be displayed. So the reader will see
> something that is obviously not correct (hence my use of the term
> "gobbledegook"). It will stick out like a sore thumb. So the sky does not
> fall in, but you do not get the benefit of the new functionality.

No, there simply is no new functionality -- indeed there is significanly *less*
functionality compared to 2822 + MIME, as RFC 2047/2231 encodings provide
charset tags and language tagging, which obviously raw (i.e. untagged) utf-8
encoding does not.  And since utf-8 is only a charset as far as this use is
concerned, anything that could potentially be done in those header fields with
raw utf-8 could instead be done in a standards-compliant, interoperable,
compatible manner by using RFC 2047 encoding, tagged with the charset "utf-8",
and optionally tagged with a language tag.

> So what happens? Perhaps they say "This is an English speaking group; why
> are these people trying to inflict a foreign language on us?". Not a very
> polite response, but Usenet is not always a very polite place :-( .

More likely they'll say "use the standard RFC 2047 encoding", and in the case
of Subject field, "please use a language tag".

> Or else they say "There are only a few posters doing this, and even though
> we cannot decipher their names, their email addresses are still OK, and
> their Subjects have a few odd characters, but we can work out what is
> going on from the bodies". And so they tolerate the small amount of stuff
> coming through.
> 
> Or else they say "We should be using this new functionality, and the
> amount of it is likely to increase. So we will ask our gateway
> administrator to upgrade in line with the new standard".

But *there is no new functionality*!  So in this case, gateway administrators
would be pestered for no good reason.

> So, gradually, the new functionality is seen throughout the net, starting
> with the places where it is most needed.

But *THERE IS NO NEW FUNCTIONALITY*!!

>>If it means that the legacy system receives data that are illegal, with
>>respect to the existing service, then this is not at all fine.
>>Interoperability requires conforming to a standard, rather than conforming
>>to it whenever it is convenient. Injecting illegal data is not conforming.
> 
> 
> You will only get data that are illegal with respect to the existing
> service if someone is trying to use the new funtionality.

Bullshit! Tere is no new functionality.

>>Infrastructure changes rarely permit incremental upgrade. Hence, a change to
>>the infrastructure often must be done to the *entire* infrastructure before
>>there can be benefit to any users.
> 
> 
> The main advantage that Usenet has over email is that is main
> infrastructure - the backbone of relaying agents that carry it worldwide -
> is already capable of supporting UTF-8 (indeed any charset) in headers.
> Hence the cost of the new functionality need only be born by those who
> want to use it - namely the users who may need to upgrade their
> newsreaders in order to see the benefits. There are some problems with
> moderators, which we have carefully addressed, and some issues with
> gateways, as I have described above.

Once again, there simply is no new functionality involved w.r.t the examples
given here.  The existing Usenet transport is also fully capable of supporting
standard (i.e. RFC 2047/2231) encoding mechanism (as opposed to raw utf-8
encoding), which has functionality which would be removed.  And the most widely
used newsreaders are in fact combined mail user agents / news readers, which
already support RFC 2047/2231 (yes, there are some bugs).  The Usefor draft's
requirement of raw utf-8 encoding means not only that gateways will have to
be substantially modified (and in fact some of the proposed changes are not
possible to implement), but it causes difficulty for moderators and news
system administrators, and would require those combined mail user agent/ news
reader programs to be modified as well -- with absolutely no new functionality
except for extended newsgroup component tags (which could be implemented by
a variety of standards-compliant, backward-compatible, interoperable
mechanisms).  Given the importance of email to commerce (vs. the negative
commercial impact of supposed "workers" wasting their employers' time on
Usenet during business hours), in fact what is likely to happen is that
what are now combined mail user agents / news readers will likely drop support
for news -- it is simply not worth turning the entire world upside down
because a handful of individuals dislikw RFC 2047.   Moreover the infrastructure
of Usenet includes the email infrastructure, since Usenet *requires* use of email
for moderated postings.

Date: Thu, 09 Jan 2003 10:06:25 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>The design defect is in the Usefor draft proposal, which requires
>>gateways to perform processing which in turn requires header field
>>syntax to be known, but which is *not* known in all cases, and cannot
>>be known in advance in the case of experimental header fields.
> 
> 
> There are many more pieces of software than Usefor gateways which get
> stuck because they do not know the correct way to parse some field. The
> design flaw was in RFC 2047 which allowed that situation to be gotten into
> in the first place.

RFC 2047 encoded-words should be decoded for diplay only, and non-ASCII
text should always remain in encoded form in header fields.  That is
clearly the intent of RFC 2047.  The design flaw, as has been already
mentioned, is decoding encoded-words for transmission.  In the case of
Usefor, that flaw in the current draft, which does not exist in the
current standard, can be remdied by complying with the Internet text
message format and with the Multipurpose Internet Mail Extensions.

To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Nick Shelness <[email-masked]>
Date: Thu, 9 Jan 2003 14:03:37 +0000
Sender: [email-masked]

Charles

> There are many more pieces of software than Usefor gateways which get
> stuck because they do not know the correct way to parse some field. The
> design flaw was in RFC 2047 which allowed that situation to be gotten 
into
> in the first place.

This claim keeps being made with little supporting evidence. RFC 2047 
encoding has a number of features.

1. It employs an escape sequence "=?" which is only valid in certain 
   RFC [2]822 header contexts.
2. This is followed by an explicit character set and language tag which 
   allows a set of glyphs to be associated with code points.
   It is recommended that members of the ISO-8859-* series be used in
   preference to other character sets,but this often not the case with
   non-latin (i.e., Asian) languages.
3. This is followed by an encoding indicator ?Q? or ?B?
4. This is followed by the encoded text which is terminated by a "?=" 
sequence

In your opinion, which of these is/are problematic? 

I find the selection of an appropriate character set when generating 
headers somewhat problematic. This though could be addressed by moving to 
using UTF-8 instead of ISO-8859-* as the preferred character set.

Nick

Nick Shelness
Independent Technology Consultant <[email-masked]>

Date: Thu, 9 Jan 2003 08:37:13 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> There are many more pieces of software than Usefor gateways which get
> stuck because they do not know the correct way to parse some field. The
> design flaw was in RFC 2047 which allowed that situation to be gotten into
> in the first place.

rfc 2047 doesn't affect how fields are parsed.  it only affects how
fields are displayed.  if a viewer doesn't realize that a field 
is structured and/or doesn't know how it should be parsed, the worst
that should happen is that encoded-words are not decoded before display.

even the, encoded-words were specifically chosen to be unlikely to appear
in ordinary input.  so simply matching =\?[^?]+\?[^?]+\?[^?]+\?= 
(or something like that), while it does have a potential for error,
has a low failure rate in practice.

most of the problems result from using rfc 2047 for purposes other
than display.

Keith

To: [email-masked]
Xref: clerew local.mime:2105
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Thu, 9 Jan 2003 10:34:53 GMT
Lines: 31
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> 
>> Yes, there is certainly a design defect, which accounts for why so many
>> implementations have chosen to fudge their RFC 2047 handling in so many
>> ways.

>The design defect is in the Usefor draft proposal, which requires
>gateways to perform processing which in turn requires header field
>syntax to be known, but which is *not* known in all cases, and cannot
>be known in advance in the case of experimental header fields.

There are many more pieces of software than Usefor gateways which get
stuck because they do not know the correct way to parse some field. The
design flaw was in RFC 2047 which allowed that situation to be gotten into
in the first place.

As it is, many such systems currently use heuristics such as the one I
have proposed.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2098
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: prevervation of installed base
Date: Wed, 8 Jan 2003 22:58:57 GMT
Lines: 117
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>Charles,

>I have included segments from your posting that pertain to the question of
>impact on installed base.  This issue is fundamental.

>To add to Ned's correction, news-> email gatewaying has been around for at
>least 20 years, as far as I can recall. Organizations have often wanted to
>plug mailing lists into newsgroups. To permit the newsgroup readers to
>participate in the mailing list, a news->email gateway is required. In other
>environments, the 2-way gatewaying is simply part of a model that lets users
>decide how they want to receive and process their group discussion messages.

Yes indeed, but those are not GENERAL-PURPOSE gateways. They only have to
be capable of meeting the needs of the newsgroup and the community of
users they were designed to serve. That is a simpler problem than a
gateway that will take any article whatsoever out of Usenet and produce a
compliant email out of it.

>With respect to the appearance of new capabilities (such as new data formats
>and encodings) in the installed base, your use of the term "gobbledegook"
>needs to be made more precise. If it means that that a legacy system
>receives the data and the data are legal but meaningless -- MIME base64 is
>example of this approach -- then it is fine. It permits incremental adoption
>without breaking existing systems. The downside for existing systems is that
>they do not get the benefit of the new feature, but everything else
>continues to work fine.

Yes, that is exactly what I mean. If someone in the newsgroup suddenly
starts using raw UTF-8 in his name (a phrase) or in his Subject, the
existing gateway is unlikely to fall over and collapse. But it will surely
pass on something that has been munged, or will be munged when it arrives
at some User Agent as a mail to be displayed. So the reader will see
something that is obviously not correct (hence my use of the term
"gobbledegook"). It will stick out like a sore thumb. So the sky does not
fall in, but you do not get the benefit of the new functionality.

So what happens? Perhaps they say "This is an English speaking group; why
are these people trying to inflict a foreign language on us?". Not a very
polite response, but Usenet is not always a very polite place :-( .

Or else they say "There are only a few posters doing this, and even though
we cannot decipher their names, their email addresses are still OK, and
their Subjects have a few odd characters, but we can work out what is
going on from the bodies". And so they tolerate the small amount of stuff
coming through.

Or else they say "We should be using this new functionality, and the
amount of it is likely to increase. So we will ask our gateway
administrator to upgrade in line with the new standard".

So, gradually, the new functionality is seen throughout the net, starting
with the places where it is most needed. But, because the quantity is
initially small, it is not a huge problem, and all existing stuff continues
to work as before. So I would describe the problem as "containable".

>If it means that the legacy system receives data that are illegal, with
>respect to the existing service, then this is not at all fine.
>Interoperability requires conforming to a standard, rather than conforming
>to it whenever it is convenient. Injecting illegal data is not conforming.

You will only get data that are illegal with respect to the existing
service if someone is trying to use the new funtionality.

>The difference between an upgrade that protects the installed base, versus
>an upgrade that requires creating a new installed base is absolutely
>fundamental.  Each can be appropriate but explicitly choosing between them
>is essential.

>The importance of preserving the installed base -- by way of doing an
>incremental upgrade rather than a parallel replacement -- is frequently
>missed. However the market power of an installed base is massive. Engineers
>and vendors ignore it at their peril.

Indeed so. Market forces will win in the end. And Usenet is indeed a
marketplace that is well used to making its opinion felt (IETF or No IETF,
Usefor or No Usefor).

>The very strong IETF philosophy has been to try to preserve the installed
>base, through optional, incremental upgrade.  It is part of the reason that the
>architectural history has been to keep the core infrastructure as simple as
>possible.

>Infrastructure changes rarely permit incremental upgrade. Hence, a change to
>the infrastructure often must be done to the *entire* infrastructure before
>there can be benefit to any users.

The main advantage that Usenet has over email is that is main
infrastructure - the backbone of relaying agents that carry it worldwide -
is already capable of supporting UTF-8 (indeed any charset) in headers.
Hence the cost of the new functionality need only be born by those who
want to use it - namely the users who may need to upgrade their
newsreaders in order to see the benefits. There are some problems with
moderators, which we have carefully addressed, and some issues with
gateways, as I have described above.

>It is certainly true that this approach creates solutions that are ugly. The
>encoding aspect of MIME is the epitome of ugliness, in my opinion. And, yes,
>incremental ugliness can create an aggregate messiness that eventually
>requires complete replacement.  And it is important to look for that
>critical mass of cumbersomeness.

There is a feeling (obviously not shared by all on this list) that that
critical mass of cumbersomeness has already been reached.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2099
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)
Date: Wed, 8 Jan 2003 23:12:47 GMT
Lines: 52
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>Correct me if I am wrong, but I believe the "native" representation for
>Unicode is something like 24 bits.  (No, folks, please don't correct me it
>is any other number over 16.)

>Hence, UTF-16 and UTF-8 are methods of encoding a larger bit space into a
>smaller representation space, producing variable-length strings.  One crams
>the larger space into a 16-bit world.  The other crams it into an 8-bit
>world.

>There depending on the situation, there can be processing or space
>efficiencies gained by one encoding over another.

UTF-8 is space efficient if you still expect to have a large proportion of
ASCII mixed in with it. It compares quite well with UTF-16 for European
languages, but I can imagine it would be less efficient in Chinese.

>But there is no theoretical or aesthetic superiority that can be claimed by
>one over the other.

There are some practical superiorities. It transports well over many
Internet protocols, because it does not do nasty thungs with CR, LF and
NUL. It has the _whole_ of ASCII as a strict subset (which is not true of
UTF-7).

But UTF-8 would be a pain for internal use in Operating systems, and maybe
even in file storage. UTF-16 is reasonable for such internal use until you
come to the code points beyond 0xFFFF, but those points are little used
(well, that's the theory, and Bill Gates seems to have committed himself
to that theory).

>The confusion on this issue probably stems from the fact that you can use
>existing data viewers -- such as text editors -- to view the result of a
>7-bit encoding and cannot use such "legacy" services for viewing UTF-8 or
>UTF-16.

There are many editors that will display _something_ useful for any 8-bit
stuff. There are a smaller number that will actually show you and let you
edit the full Unicode characters.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2097
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Wed, 8 Jan 2003 22:22:32 GMT
Lines: 62
Sender: [email-masked]

In <[email-masked]> Leo Bicknell <[email-masked]> writes:

>In a message written on Tue, Jan 07, 2003 at 03:20:55PM -0800, Russ Allbery wrote:
>> > * Extend MIME (eg, define types) to allow for "message/rfcM".  Boom,
>> >   e-mail is good.
>> 
>> You've lost me here.  How does defining message/rfcM let you deal with
>> UTF-8 headers in e-mail?

>message/multipart
> - text/plain
>    "This is a news article you need to approve/disapprove."
>    or
>    "This is a news article that has been gatewayed to e-mail."
> - message/rfcM
>    - UTF-8 article, base64 encoded

The beast you are describing exists already. It is called
application/news-transmission, and it is perfectly suitable for tunneling
any news article (with UTF-8 in its headers) through any mail system.

It is indeed an ideal tool for sending articles to moderators (it even
comes with a parameter saying "this is for you to moderate"). It is exactly
what Usefor wanted to use.

>> This sounds like you're talking about encapsulating submissions to
>> moderators rather than gatewaying them.  This breaks almost all existing
>> moderation software.

>I submit the number of moderators is /vastly/ smaller than the number of
>people who read e-mail, or read news, or are on a list gatewayed between
>them.  If the fallout is that moderators need to update their software
>I'd say that's affecting the smallest group possible.

>They are also people who are likely to update to the latest and greatest
>with any news change, as I think they are likely to want new bells and
>whistles, or at least be able to understand them as they moderate.

And that is what we thought, but sadly it did not turn out to be so (even
though it is already used to some extent in Usenet). The moderators just
did not want to know, and so we had to backtrack and write detailed rules
on how to send email to moderators without encapsulation.

In the longer term, I hope that the encapsulation method (which we all
agree is technically superior) will win out.

And in principle it would also work for other gateways, except that User
Agents currently do not know what to do with it except store it into a
file for later examination. So it does not solve the present problem,
except maybe in a few very specialized applications.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 8 Jan 2003 14:36:48 -0500 (EST)
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Usefor repoll justified? (was Re: Call for Usefor to rec
Reply-To: "Forrest J. Cavalier III" <[email-masked]>
CC: [email-masked]
Sender: [email-masked]

(Original sent to [email-masked] was spam trapped, thanks
for adding the posting address.)

----------------Forwarded from Usefor.  

"Dan Kohn" <[email-masked]> wrote in part:

> Here, I believe, is the crux of the disagreement between the working
> consensus of usefor and the unanimous agreement of email folks from
> ietf-822.
> 
> Usefor recently conducted a straw poll [1] in which raw UTF-8 headers
> barely won out [2] over using 2047/2231.  (Specifically, in the #1 vs.
> #5 vote, the tally was 9 to 8.)  The chair, Dave Barr, declared "rough
> consensus" [3] and suggested the group should move forward on that
> basis.
> 
> I suggest to the usefor chair that the group should conduct the poll
> again, based on a new piece of information.  Specifically, they've seen
> the applicable area director react in such a viscerally negative fashion
> [4] that it is nearly impossible to imagine anything resembling the
> current usefor approach being approved by the IESG.  I.e., the backward
> compatibility issue is nowhere near as small as the group seems to
> believe.

I disagree with the consensus of the recent Usefor poll, too.
But, in the recent Usefor poll, the implementation options and
their consequences were very cleary explained.  

The likely IETF responses were discussed on list as well. The poll
participants were expected to factor in their predictions of what
the IETF would do.  (I think the poll results definitely reflect that,
since the poll ended at a different conclusion than the
August poll.  Better conclusion IMHO, too.)

But I think the poll result is not questionable. The poll results must
be the guide for the attempt at making progress, unless a repoll
can be justified.  Some reasons for that might be:

   1. a flaw in the poll (an option was not in fact as it was presented)

   2. a flaw in the voting (voters not understanding the system,
      which I think did happen in one case, based on my understanding
      of his previous postings to the WG list, but the outcome would not
      change) or a flaw in someone's reading of the text at each option.

   3. rejection of the result by a higher authority (which is not the same
      as a strong comment from an AD who knows what the draft
      is by hearsay or in summary.)

   4. clear evidence that poll results are not an accurate
      reflection of the larger Internet community beyond Usefor.
      (And this may be the case.  How can we tell?)

The Usefor poll result may be surprising, or not match what
you think is obvious (I think so too) but covering
already covered discussion (e.g. what gateways
will/will not do, what the IESG will/will not approve)
is not useful.  Usefor did all that already (more than
once for most issues) and most importantly, the recent poll
text accurately reflected the major issues and implications
of each choice. (I am very glad we decided to use that
format for the poll.)

Forrest

Subject: Re: Usefor repoll justified? (was Re: Call for Usefor to recharter)
From: Dave Barr <[email-masked]>
To: [email-masked], [email-masked]
Organization: 
Date: 08 Jan 2003 13:10:35 -0600
Sender: [email-masked]

I agree with Forrest here.  The complaints being lodged against the
results of our work are not new.  We've been struggling with all these
issues for quite some time ourselves, and we voted this way having
already having been told many times that the IESG as well as the email
folks wanted encoding over moving towards UTF-8 in headers.

It's not like encoding is a magic fix.  If it were, it would have been
an easy decision and it would have been made two years ago.  The issues
against encoding go beyond the fact that the standards and
implementations are highly flawed.

I've asked paf to step in and help out.

--Dave

Date: Wed, 8 Jan 2003 09:37:11 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: old format date fields (was Re: RFC 2047 and gatewaying)
Sender: [email-masked]

> By the way, could you *please* fix or replace the software which is generating
> the illegal Date headers in your messages, e.g.
> Date: Mon, 6 Jan 2003 17:07:33 GMT
> Non-numeric offsets have been in the "do not generate" category (see RFC 2822)
> for more than 20 months.

well, your software still needs to be able to read the things - otherwise how are
you going to read old email?

Keith

Date: Wed, 08 Jan 2003 09:06:36 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> OK, so you have no problem with
> 	Content-Disposition: attachment; filename="?-?-?-?"
> where "?-?-?-?" is a placeholder for some filename in Hungarian, Arabic or
> Chinese. Just so long as that usage is confined to Netnews, and is not
> seen in Email except in the corresponding RFC 2231 form?
> 
> And, naturally, that usage will be seem in the body part headers of a
> multipart/mixed (again within Netnews, not Email)? For that is precisely
> where you would expect "attachments" to be found.
> 
> So I am allowed to reuse the existing media type multipart/mixed.

No, no, no. RFC 2046 section 5.1.1 (Common Syntax for multipart media
types) *clearly and unambiguously* states:

    However, in no event are headers (either message
    headers or body part headers) allowed to contain anything other than
    US-ASCII characters.

By the way, could you *please* fix or replace the software which is generating
the illegal Date headers in your messages, e.g.
Date: Mon, 6 Jan 2003 17:07:33 GMT
Non-numeric offsets have been in the "do not generate" category (see RFC 2822)
for more than 20 months.

Date: Wed, 08 Jan 2003 08:54:19 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>It is insufficient to say "comply with 2822 / 2047 / 2231); there is
>>a design defect which prevents that, unless a specific mechanism is
>>provided to ensure compliance.  As it stands,
>>1. it is impossible for a gateway to determine the syntax of an
>>   experimental header which the gateway author is unaware of
>>2. Without header field syntax information, it is impossible to
>>   encode non-ASCII "header" content.
> 
> 
> Yes, there is certainly a design defect, which accounts for why so many
> implementations have chosen to fudge their RFC 2047 handling in so many
> ways.

The design defect is in the Usefor draft proposal, which requires
gateways to perform processing which in turn requires header field
syntax to be known, but which is *not* known in all cases, and cannot
be known in advance in the case of experimental header fields.

>>The design defect can be eliminated by:
> 
> 
> You cannot eliminate a desgin defect, but you can maybe work around it:

Since the aforementioned defect does not exist in mail or in the current
Usenet standard (RFC 1036), but only in the paper design in the Usefor
draft, it can be eliminated by correcting the defective design in that
draft.

>>a. forbidding non-ASCII content in all but a well-defined set which
>>   all gateways are required to recognize, *and* providing a practical
>>   transition plan so that existing gateways can be upgraded while
>>   ensuring that the mail protocols are not violated
> 
> 
> Which is what Usefor is trying to do. But Usefor cannot prevent future
> extensions to itself from adding to the set. So you still need a fudge.

I see no evidence of a practical transistion plan in the draft; in fact
it is clear from the draft and from discussion in the Usefor mailing list
that there essentially is no transition plan.  And rather than restricting
non-ASCII content to a set of well-defined Usenet-specific headers, the
draft instead seeks to introduce non-ASCII content into existing headers
which have always forbidden such content.

>>or
>>b. simply using the RFC 822 / 2822 / 2047 / 2231 Internet text message
>>   format, which avoids the encoding problem in gateways and adds no
>>   new restrictions on gateway transformations (thereby obviating that
>>   aspect of a transition plan).
> 
> 
> Yes, but Usefor has chosen a different route.

And therein lies the defect.

>>or
>>c. require that unrecognized "headers" with non-ASCII content be elided
>>   or moved to an encoded attachment.
> 
> 
> And of your various suggestions, that is the one which deserves serious
> consideration. It was #3 in my list IIRC. The question to address is
> whether it will do more, or less, harm than my workaround proposal. It is
> not clear.

Your "workaround proposal", as you now call it, simply won't work because
it requires header syntax to be known in advance for all Usefor "header"
fields which might contain non-ASCII content, including experimental ones,
and that can only be achieved by one of the above three methods, not by
your "proposed workaround".  Of the three, "b" is the cleanest approach,
which is fully compatible with existing gateways and with the existing
standard, does not present difficulties w.r.t. IETF standardization, and
has been strongly recommended, independedntly, by several respondents in
this thread.

Date: Wed, 08 Jan 2003 08:37:19 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>2. Comments may well exist within angle brackets, e.g.:
> 
> 
>>  Return-Path: <(=?us-ascii?q?foo?=)>
> 
> 
> But there is no non-ASCII there, so no problem.
> 
> Now if I had
> 	Return-Path: <(?-?-?-?)>
> where "?-?-?-?" should be regarded as a placeholder for some sensible
> comment in Hungarian, Arabic or Chinese, then my proposed workaround
> would transform it into
> 	Return-Path: =?iso-8859-1?q?<(=3F-=3F-=3F-=3F>)?=
> which, I will grant you, is not a valid Return-Path

As you may recall, that was in response to the lastest variant of your
gateway "heuristic", in which you claimed that URIs should be bracketed
by '<' and '>' on the supposed grounds that that would prevent
misinterpreting parentheses in URIs as comments, further supposing
(incorrectly) that comments could not appear within angle bracketed
constructs.  So your gateway process (which you now call a "workaround")
would in fact entirely miss the content within the angle brackets. And
that proposed gateway heuristic simply won't work precisely because
it does not take into account full syntax.

content-class: urn:content-classes:message
Importance: normal
Subject: RE: Call for Usefor to recharter
Date: Wed, 8 Jan 2003 00:20:56 -0800
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Thanks for the update.  I agree that the next logical step is for
Patrick to chime in, or say that he's not planning to.  (Note that my
error was because Ned, not Patrick, is listed as the Applications Area
Advisor at <http://www.ietf.org/html.charters/usefor-charter.html>.)

It's certainly a point of contention of whether there are only "several
hundred" gateways, that are all "special-purpose".  My impression is
that a significant number of extant IMAP servers also gateway
newsgroups, and that all of this RFC 1036-compliant software would
suddenly become non-compliant with usefor news messages (due to raw
UTF-8 in headers), potentially resulting in breakages for end-users.
For example, see [1] or [2].

By contrast, if i18n of news were accomplished with 2047/2231 encoding
(plus punycode for newsgroup names), all existing gateways (including
those used by moderators!) would keep working as usual, while providing
full i18n to anyone with an updated client (and most clients already
support 2047/2231, and will be getting punycode support for IDNA).

I'm cc'ing ietf-822 to look at concluding the thread there, but I agree
that this conversation belongs on usenet-format going forward.

[1] http://www.acm.org/chapters/bombay/news/archives/200002.html "Usenet
News can be fed into a shared folder and read as email"
[2]
http://www.washington.edu/imap/meeting.2nd/present/shakib/tsld010.htm
"Same environment as mail!"

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

-----Original Message-----
From: Charles Lindsey [mailto:[email-masked]] 
Sent: Tuesday, January 07, 2003 15:30
To: [email-masked]
Subject: Re: Call for Usefor to recharter

In <[email-masked]> "Dan
Kohn" <[email-masked]> writes:

>I wrote:

>>> I have a simple question.  What can a UTF-8 subject header
>>> communicate that an RFC 2047 one can't?  Other than inelegance,
>>> what's the downside of 2047, when the upside is a huge increase in
>>> backward compatibility? 

>Charles Lindsey responded:

>> No, it's just the inelegance. Plus the fact that the backward
>> compatibility issue is nowhere so huge as you imagine. In fact, it is
>> rather small.

>Here, I believe, is the crux of the disagreement between the working
>consensus of usefor and the unanimous agreement of email folks from
>ietf-822.

>Usefor recently conducted a straw poll [1] in which raw UTF-8 headers
>barely won out [2] over using 2047/2231.  (Specifically, in the #1 vs.
>#5 vote, the tally was 9 to 8.)  The chair, Dave Barr, declared "rough
>consensus" [3] and suggested the group should move forward on that
>basis.

OK, time to give some background as to what has been going on.

1. We conducted a straw poll and the conclusion reached was that we
would
proceed on the basis that:

  a) We would continue to use UTF-8 in headers, with RFC 2047/2231 as an
  alternative (and special considerations for newsgroup-names).

  b) Whenever anything had to go via Email, there MUST be
transformations so
  that the Email is compliant (the transformations would involve RFC
  2047/2231 and 5.5.2).

2. As part of the implementation of that decision, I wrote some text on
those transformations and posted it here. There has been no comment here
on that text.

3. Because I wanted to be sure that the transformations would indeed
produce compliant email, I also posted that text to the ietf-822 mailing
list (RFC 2047 is a difficulat beast to comply with, and there were some
awkward corners I had to address).

4. All Hell broke loose, much of it based on gut reaction rather than
careful appraisal of what I was proposing. You need to understand that
the
ietf-822 list is peopled by the authors of the MIME-standards, and by
those who claim to have great knowledge of, and influence on, the IESG.

5. And so there were predictions, prophecies and threats that, it we
proceeded on our present course, it would never get past the IESG. There
were essentially three issues:

6. Issue #1. My transformation for the awkward corners (which were
advisory rather than obligatory) could lead to non-compliant email. This
is true, but only in the event of a long chain of improbable
happeninggs.
I considered my suggestions to be better, on average, than the
alternaitve
which was to drop information on the floor. But I have no problem with
revisiting that part of the proposal.

7. Issue #2. News->Mail gateways that had not upgraded to our standard
would fail as soon as they saw some UTF-8 character, because they would
not know how to do those transformations. I tried to explain that this
was
a small and containable problem, because GENERAL-PURPOSE gateways from
news->mail do not exist. By "GENERAL-PURPOSE" I mean a gateway that is
set
up to handle input from _every_ group on Usenet and to produce
corresponding email. The gateways that are around (and there are not
that
many of them - a few hundred maybe) are all SPECIAL-PURPOSE, being
created to gateway particular newsgroups for particular classes of
people
(usually a mailing list). Since the language used by most of them is
English and the charset is ASCII, they are not going to notice. In the
cases where it may matter, they will start to see a small trickle of
articles with strange things in their headers, whereupon they may decide
to tolerate them or else decide that it is time to update the gateway.
My
point is that the problem is _containable_. But I completely failed to
get
across to them the sifnificance of the phrase "GENERAL-PURPOSE" (I had
not
thought it necessary to shout it then) and so my arguments were falling
on
deaf ears. But I still think this problem is manageable.

8. Issue #3. This is the real crunch. We have destroyed the purity of
MIME, by permitting UTF-8 characters in two places:
   a) in parameters in some of the Content-* headers
   b) in the headers of message/rfc822 objects (because we propose to
use
   that Content-Type rather than the to-be-obsoleted message/news for
Netnews
   articles that get encapsulated within other Netnews articles).
Note that in both cases, those relaxations are ONLY to be allowed within
Netnews articles - if you want to move that article, or that
message/rfc822 object, into mail, then you MUST transform as above. I
have
tried to claim that the existing MIME definitions apply only within
Email
(see RFC 2045) and that it is therefore lawful to augment them for use
within Netnews only (and subject to transformation before gatewaying to
mail). I have also pointed out that the HTTP spec was allowed to take
similar liberties with some MIME definitions. But they will have none of
it.

>I suggest to the usefor chair that the group should conduct the poll
>again, based on a new piece of information.  Specifically, they've seen
>the applicable area director react in such a viscerally negative
fashion
>[4] that it is nearly impossible to imagine anything resembling the
>current usefor approach being approved by the IESG.  I.e., the backward
>compatibility issue is nowhere near as small as the group seems to
>believe.

>(Ned, if I'm overstating your viewpoint and its implications, my
>apologies, and please correct my error.)

No, Ned is NOT the applicable Area Director (though he is a member of
the
IESG). The AD is Pat Falstrom, who has not yet appeared in the
dicussion.
I think a sueful next move would be for our Chairman to speak to Pat.

>Based on this new information, I would suggest that the group consider
>rechartering with a more precise plan and schedule, both approved by
the
>IESG, in such a matter that successful publication of a standards-track
>RFC is a likely (or at least possible) outcome.

>Alternatively, if the group decides it's not interested in the IESG
>imprimatur, I'd suggest that the group consider reforming outside the
>aegis of the IETF.

Yes, that is one possibility, but there are many other possibilities.

One is to press ahead, and deal with the IESG when we have to, which is
not yet.

Another would be to write the standard for RFC 2047/2231 only (but
leaving
newsgroup-names with 5.5.2) and then immediately to issue an extension
allowing UTF-8 as an Experimental Protocol (the marketplace would then
determine whether or not to join in the experiment).

And there are doubtless many other possibilities.

But the fundamental problem remains this: There is a tension between the
"elegant" solution and the "backward-compatible" solution. The IESG has,
in the past, always favoured the "backward-compatible" solution for fear
of offending some piece of legacy software out there. That might be a
fine
thing, but it sure is a bar to progress in the long-term. And usenet, as
an institution, does not have to be 100% perfect. It is much more able
to
tolerate the occasional backward-compatibility failure. More so than
email
would be, and certainly more so than DNS (which is why we are now faced
with IDNA).

-- 
Charles H. Lindsey ---------At Home, doing my own
thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web:
http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8
3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14
A4 AB A5

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 07 Jan 2003 22:03:15 -0800
Lines: 86
Sender: [email-masked]

Leo Bicknell <[email-masked]> writes:

> Ok, I'll admit I don't have all the facts, but I don't see how this
> situation can exist with what has been proposed.

Here, maybe this will help:

 * News servers currently always use one method to forward postings to
   moderators, namely they add a To header to the message as posted and
   dump it into e-mail.  (There are a very small handful of exceptions,
   but not enough to be worth thinking about much.)

 * Most moderated groups use ASCII, but not all.  But even the ones that
   don't at present use the above method, which means that in hierarchies
   where using unencoded 8-bit characters is common, they get them via
   e-mail as described above.  It seems likely that the only hierarchies
   in which there are moderated groups and unencoded 8-bit data in headers
   are using character sets that avoid the range of characters that get
   mangled.

 * Moderation software is often running nearly on autopilot, and there is
   no standard package that everyone uses.  A bunch of people use STUMP,
   some people use DMOD, and then most moderators have rolled their own.
   Many moderators just use a regular e-mail client and post things
   manually, which works fine for low-traffic groups.

> If news is UTF-8, raw, there must be some mechanism to e-mail that
> content to a moderator,

News is not UTF-8, raw.  This is just a proposal, which Charles is
advocating and which some people on USEFOR think is the best solution.
How to deal with the implications of this position for moderation is not
something that has yet been fully fleshed out, although the current theory
(as I understand it) seems to be that everything will get translated into
RFC 2047 by the news server before being sent to the moderator.

This is probably doable, but I'm skeptical.

> have them be able to read it at least well enough to decide how to
> moderate it, and then be able to send it back in some format that
> preserves the content with their additional approval (which most likely
> implies UTF-something).

It is *not* necessary for moderators to be able to read or understand all
the content in the headers to be able to moderate a group.  Consider, for
example, the (likely increasingly common) case where the non-ASCII content
in the headers is restricted just to the person's name, because they don't
want to use an ASCII transcription of their real name.  I don't need to be
able to read the From header to be able to moderate the group; all I have
to do is be able to get it into news without mangling it.

But that being said, the problem isn't just what to do about postings that
contain non-ASCII headers.  The problem is how to allow news servers to
continue to use *one* method to forward messages to moderators rather than
making moderators handle two possibilities depending on what the message
looks like, and simultaneously not breaking the moderation of all the
existing ASCII moderated groups.

That leads to a desire to be able to continue to send messages to
moderators using the current method.

Another possible alternative would be to handle moderation differently for
groups with non-ASCII character sets, although I don't think this actually
scales well because of the above example of someone who just wants to use
their correct name.

Any solution that changes the way that posts are sent to moderators for
the existing ASCII moderated groups is going to break a *lot* of software,
and the proposal of that in the past caused massive pushback by
moderators.

> IMHO moderation is broken anyway, because it must go to e-mail and back.

I doubt very many people would disagree with you that it's less than
ideal, although I'm not sure that's the right point to try to fix it.
Fixing this is *way* outside the scope of any current working group,
though, and isn't something that would happen at all quickly, even if we
could come up with a better protocol.

(It would certainly be better from a cleanliness perspective if all posts
sent to moderators were encapsulated rather than gatewayed, for example,
but this unfortunately is likely to be something that we're too late to
fix.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

content-class: urn:content-classes:message
Importance: normal
Subject: RE: RFC 2047 and gatewaying
Date: Tue, 7 Jan 2003 18:58:45 -0800
From: "Dan Kohn" <[email-masked]>
To: "Leo Bicknell" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

I think that encapsulating messages with 8-bit headers as
application/news-transmission is a marginally feasible way to transmit
them to moderators.  Moderators at least have an incentive to upgrade
their tools.

However, the proposal fails utterly at supporting gatewayed mailing
lists, where mail users who previously had full access to news content
start getting an unknown attachment that they have no idea how or why to
open.

By contrast, sticking with 7-bit headers enables both moderators and
gatewayed mailing list users to continue with full functionality.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

-----Original Message-----
From: Leo Bicknell [mailto:[email-masked]] 
Sent: Tuesday, January 07, 2003 13:29
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying

Here's an interesting path forward, although it's probably going to get
blasted for attempting to rewrite how e-mail works, that said:

* Define a new standard for Unicode based "messages".  I'll call this
  "rfcM".  For the moment I'll suggest we think about a UTF-8 format
  similar to what e-mail and Usenet look like today, headers, body,
  UTF-8 everywhere.  (Could also be UTF-16, UTF-32, or whatever, I'm
  going to use UTF-8 just as a point of discussion.)

* Extend MIME (eg, define types) to allow for "message/rfcM".  Boom,
  e-mail is good.  On 7 bit transports we can base64 encode the part, 
  on 8 bit clean transports they can be sent raw.  At least for now
  everyone will presumably use the base64 method.

  - At this point e-mail transport RFC's should be updated for 8 bit
    clean transport (assuming UTF-8), so at some point in the future
    that can be the default choice.

* Define the new usenet format as an extension to "rfcM", adding the
  special headers and other junk necessary to support news.

Now, mail<->news is easy.  Moderators would need e-mail clients that
understand message/rfcM, but other than that there are no transport
issues.  Over time, message/rfcM would become part of more and more
mail clients, eventually allowing message/rfc822 to be depreferenced,
leaving us with Unicode clean e-mail and news.  Going the other way
is easy, e-mail that came in as message/rfcM can be turned directly
into news, things that came in as message/rfc822 would go through a
ISO-8859-1->UTF-8 translation and then be put into news.

Granted, I've left out a lot of details, but wouldn't this sort of
high level direction allow e-mail and usenet to reconverge in a 
compatable format in the future, and make the interim time not too
painful for either group?

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

Date: Tue, 7 Jan 2003 21:55:22 -0500
From: Keith Moore <[email-masked]>
To: Leo Bicknell <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> If news is UTF-8, raw, there must be some mechanism to e-mail that
> content to a moderator, have them be able to read it at least well
> enough to decide how to moderate it, and then be able to send it
> back in some format that preserves the content with their additional
> approval (which most likely implies UTF-something).

it's certainly possible to do this if you can get moderators
to run new code. however expecting this functionality to already be 
implemented in existing software is hopelessly unrealistic.

Date: Tue, 7 Jan 2003 21:16:45 -0500
From: Leo Bicknell <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: United Federation of Planets
Sender: [email-masked]

In a message written on Tue, Jan 07, 2003 at 05:32:56PM -0800, Russ Allbery wrote:
> No, they're not; in fact, the exact opposite is true.  Personal experience
> speaking there.  Usenet moderators on the whole are the most likely to
> still be using software that was written in the early 1990s, because they
> have a disproportionate representation of old Unix hackers.

Ok, I'll admit I don't have all the facts, but I don't see how this
situation can exist with what has been proposed.

If news is UTF-8, raw, there must be some mechanism to e-mail that
content to a moderator, have them be able to read it at least well
enough to decide how to moderate it, and then be able to send it
back in some format that preserves the content with their additional
approval (which most likely implies UTF-something).

I would love to hear how if news is UTF-8 that could be done round trip
allowing all existing software to work.  It seems like a complete
impossibility to me.

If anything, my proposal I think would make it easier.  I can imagine,
for instance, a moderation gateway sending out:

multipart/mixed
  - text/plain
    "This is a message you must moderate."
  - multipart/alternative
     -text/plain
      "All US-ASCII characters extracted from the UTF-8 message."
     -message/rfcM
      UTF-8 message in base64

So now, Mr Still Runs SunOS on a 3/80 gets a message that he can tell
he has to moderate (first text/plain part), can probably mostly read
(since Mr 3/80 is so in the past, he almost must speak US-ASCII, so
it is extremely unlikely he's moderating a foreign language group)
in the text/plain section.  Finally, he can copy back the message/rfcM
part in a new outgoing message, something you can even do with awk/sed
with limited tallent (or say perl, with readily availble MIME library,
etc).

Indeed, this would allow someone moderating a mostly english newsgroup
to preserve things like "From:" lines with foreign language characters
never needing a mailer/terminal understanding UTF-8 or any of those
foreign characters.  He can moderate based on the text/plain content,
in english.

Does he need a new "approve" script to pipe the message to in order to
make this work?  Sure.  However I submit it would be relatively trivial
to write such a thing in /bin/sh or perl, and if someone did just that
with the same calling conventions as the existing scripts they would
probably happly install it and use it glad they never need to know about
UTF-8.

Meanwhile, those who need UTF-8 can upgrade to mailers that display
all the characters they need to have newsgroups in their native language
moderated in the native language.

IMHO moderation is broken anyway, because it must go to e-mail and back.
Moderation should be built into NNTP, that is a command should be added
like:

"MODERATE my.group mypassword"

At which point the server shows you all submitted but not yet moderated
article.  They could then be moderated with:

"MODERATE my.group <article-number> mypassword"

Triggered by something in the news client software.  Again, newsreaders
would have to be updated -- but if the group is set on UTF-8 that's a
given anyway and it would get rid news<->mail issues for moderation
once and for all.

A huge oversight in the original NNTP protocol, IMHO.

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 07 Jan 2003 17:32:56 -0800
Lines: 22
Sender: [email-masked]

Leo Bicknell <[email-masked]> writes:

> I submit the number of moderators is /vastly/ smaller than the number of
> people who read e-mail, or read news, or are on a list gatewayed between
> them.  If the fallout is that moderators need to update their software
> I'd say that's affecting the smallest group possible.

We've been over this already; the USEFOR archives have the fallout of the
flamewar.  I think it's a nonstarter.

And no, it's probably not smaller than the number of gateways.

> They are also people who are likely to update to the latest and greatest
> with any news change,

No, they're not; in fact, the exact opposite is true.  Personal experience
speaking there.  Usenet moderators on the whole are the most likely to
still be using software that was written in the early 1990s, because they
have a disproportionate representation of old Unix hackers.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 7 Jan 2003 19:36:55 -0500
From: Leo Bicknell <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: United Federation of Planets
Reply-By: Tue Jan 14 19:26:07 EST 2003
Sender: [email-masked]

In a message written on Tue, Jan 07, 2003 at 03:20:55PM -0800, Russ Allbery wrote:
> > * Extend MIME (eg, define types) to allow for "message/rfcM".  Boom,
> >   e-mail is good.
> 
> You've lost me here.  How does defining message/rfcM let you deal with
> UTF-8 headers in e-mail?

message/multipart
 - text/plain
    "This is a news article you need to approve/disapprove."
    or
    "This is a news article that has been gatewayed to e-mail."
 - message/rfcM
    - UTF-8 article, base64 encoded

Is it pretty?  No.  Do the headers directly translate, no.

For the moderation issue, I don't think this is a problem.  The real
issue is an e-mail<->news gateway for Joe Average, and this gives
mailers today a way to ignore things they don't understand.  I see
some minor problems with translating some headers (eg, a From: line
with UTF-8 in it), however again during the interim a gateway can
just set "From: [email-masked]".

Then, in one magical day in the future when UTF-8 mailers are common
the gateways can change to just sending a single part message, format
message/rfcM.

> This sounds like you're talking about encapsulating submissions to
> moderators rather than gatewaying them.  This breaks almost all existing
> moderation software.

I submit the number of moderators is /vastly/ smaller than the number of
people who read e-mail, or read news, or are on a list gatewayed between
them.  If the fallout is that moderators need to update their software
I'd say that's affecting the smallest group possible.

They are also people who are likely to update to the latest and greatest
with any news change, as I think they are likely to want new bells and
whistles, or at least be able to understand them as they moderate.

If anything, the moderation could become quite simple, take the multipart I
described above, and send back:

message/multipart
  - text/plain
    "approve my_password"
  - text/rfcM
    - The UTF-8 article, 100% unchanged from the first message.

Since the second part is copied verbatim, any MIME tools today could
probably trivially be modified to construct a reply in this format, the
only real issue for the moderator is finding something that will display
the UTF-8 message, but as I see it that's going to be a problem
regardless of the format of the e-mail he gets (UTF-8 raw, UTF-8 in
base64 MIME, some other wacky new encoding), so it's a noop.

I realize it's not quite how it works today (although it's been a 
while since I did anything serious with news), but it seems to be a
way to do it that existing tools are 90% of the way to supporting,
and that also in the long run might help move e-mail to UTF-8, which
I have to believe will be a battle at some point.

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

Date: Tue, 7 Jan 2003 16:26:40 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Folks,

The most amusing things about this line of discussion is how similar it
sounds to claims made during the MIME discussions, 10 years ago.

Unicode has taken over.  8-bit is prevalent.  etc. etc.

Tuesday, January 7, 2003, 3:52:35 PM, you wrote:

>> My point was that the argument that it's bad and evil to make an 
>> incompatible change that would mean that 8 bit data would circulate 
>> somewhere where it's not allowed in the current standards is weak 
>> because the actual situation is that a lot of 8 bit data is being sent.

Keith> no, it doesn't follow.  yes, a lot of 8-bit data is being sent,
Keith> but there's also a very good chance that that data is not being
Keith> displayed correctly at the recipient's end.  most mail handling
Keith> tools have learned not to crash when presented with 8-bit header
Keith> text, but there is no good convention for how to interpret such 
Keith> text.  (and simply saying "use utf-8" doesn't work 

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 7 Jan 2003 18:58:40 -0500
From: Keith Moore <[email-masked]>
To: Keld =?ISO-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

> > No, because the only standardised 7-bit encoding for unicode is UTF-7 
> > that has so many drawbacks it's strongly deprecated.
> 
> There is another standardized encoding for iso-10646, namely mnemonic in
> RFC 1345.

RFC 1345 is an informational document.

Date: Tue, 7 Jan 2003 18:52:35 -0500
From: Keith Moore <[email-masked]>
To: Jean-Marc Desperrier <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> My point was that the argument that it's bad and evil to make an 
> incompatible change that would mean that 8 bit data would circulate 
> somewhere where it's not allowed in the current standards is weak 
> because the actual situation is that a lot of 8 bit data is being sent.

no, it doesn't follow.  yes, a lot of 8-bit data is being sent,
but there's also a very good chance that that data is not being
displayed correctly at the recipient's end.  most mail handling
tools have learned not to crash when presented with 8-bit header
text, but there is no good convention for how to interpret such 
text.  (and simply saying "use utf-8" doesn't work 

> And that the current endorsed solution for sending non US-ASCII data 
> (RFC2047) has so many problems that in many situations, messages are 
> more likely be handled correctly by sending them in raw 8 bits that by 
> using RFC2047.

strongly disagree.  yes, text that is repeatedly unencoded and
re-encoded by lots of different user agents is likely to get corrupted
at some point.  but that situation seems to be improving.  and 2047
is more likely to be presented correctly.

Keith

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 07 Jan 2003 15:20:55 -0800
Lines: 18
Sender: [email-masked]

Leo Bicknell <[email-masked]> writes:

> * Extend MIME (eg, define types) to allow for "message/rfcM".  Boom,
>   e-mail is good.

You've lost me here.  How does defining message/rfcM let you deal with
UTF-8 headers in e-mail?

> Now, mail<->news is easy.  Moderators would need e-mail clients that
> understand message/rfcM, but other than that there are no transport
> issues.

This sounds like you're talking about encapsulating submissions to
moderators rather than gatewaying them.  This breaks almost all existing
moderation software.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 7 Jan 2003 23:44:55 +0100
From: Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>
To: Jean-Marc Desperrier <[email-masked]>
Cc: Dave Crocker <[email-masked]>, [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

On Tue, Jan 07, 2003 at 09:57:58PM +0100, Jean-Marc Desperrier wrote:
> 
> No, because the only standardised 7-bit encoding for unicode is UTF-7 
> that has so many drawbacks it's strongly deprecated.

There is another standardized encoding for iso-10646, namely mnemonic in
RFC 1345.

Kind regards
Keld

Date: Tue, 7 Jan 2003 16:28:49 -0500
From: Leo Bicknell <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: United Federation of Planets
Reply-By: Tue Jan 14 16:17:19 EST 2003
Sender: [email-masked]

Here's an interesting path forward, although it's probably going to get
blasted for attempting to rewrite how e-mail works, that said:

* Define a new standard for Unicode based "messages".  I'll call this
  "rfcM".  For the moment I'll suggest we think about a UTF-8 format
  similar to what e-mail and Usenet look like today, headers, body,
  UTF-8 everywhere.  (Could also be UTF-16, UTF-32, or whatever, I'm
  going to use UTF-8 just as a point of discussion.)

* Extend MIME (eg, define types) to allow for "message/rfcM".  Boom,
  e-mail is good.  On 7 bit transports we can base64 encode the part, 
  on 8 bit clean transports they can be sent raw.  At least for now
  everyone will presumably use the base64 method.

  - At this point e-mail transport RFC's should be updated for 8 bit
    clean transport (assuming UTF-8), so at some point in the future
    that can be the default choice.

* Define the new usenet format as an extension to "rfcM", adding the
  special headers and other junk necessary to support news.

Now, mail<->news is easy.  Moderators would need e-mail clients that
understand message/rfcM, but other than that there are no transport
issues.  Over time, message/rfcM would become part of more and more
mail clients, eventually allowing message/rfc822 to be depreferenced,
leaving us with Unicode clean e-mail and news.  Going the other way
is easy, e-mail that came in as message/rfcM can be turned directly
into news, things that came in as message/rfc822 would go through a
ISO-8859-1->UTF-8 translation and then be put into news.

Granted, I've left out a lot of details, but wouldn't this sort of
high level direction allow e-mail and usenet to reconverge in a 
compatable format in the future, and make the interim time not too
painful for either group?

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

content-class: urn:content-classes:message
Subject: RE: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Importance: normal
Date: Tue, 7 Jan 2003 13:22:38 -0800
From: "Dan Kohn" <[email-masked]>
To: "Jean-Marc Desperrier" <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

It took place <http://www.imc.org/ietf-822/mail-archive/msg02674.html>
and <http://www.imc.org/ietf-822/mail-archive/msg02696.html>.

If 2047 is broken, then it should be fixed or replaced (perhaps with
nameprep+punycode) with something that can provide adequate i18n while
still preserving backward compatibility.  Most likely, it should be
fixed, since (due to email) it will quite likely be around forever and
all newsreaders will need to be able to at least decode it for the
foreseeable future.

One solution would be to profile 2047 use for news along the lines of
"legal to generate/must understand" in sections 3 & 4 of RFC 2822.  All
newsreaders would be required to understand 2047, but compliant readers
would only generate 2047 text that actually used Q or B encoding of
UTF-8.

I'm not sure if this is the right solution, but I stand by suggestion
that rechartering would provide the context for evaluating the solution
space.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600> 

-----Original Message-----
From: Jean-Marc Desperrier [mailto:[email-masked]] 
Sent: Tuesday, January 07, 2003 10:15
To: [email-masked]
Cc: [email-masked]
Subject: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)

Dan Kohn wrote :
> I wrote:
>>>I have a simple question.  What can a UTF-8 subject header
>>>communicate that an RFC 2047 one can't?  Other than inelegance,
>>>what's the downside of 2047, when the upside is a huge increase in
>>>backward compatibility? 

I do not know where this discussion took place, but I have an answer to
it.

It's a simple fact.
In every single thread with non US-ASCII data in subject encoded by 
RFC2047 (sorry I wrote 2049 by error in my last mail) I've seen, the 
subject turned to garbage after 5 or 6 messages.

The reason for that is that all implementations of RFC2047 around are 
full of implementation errors.

The reason for that is that the RFC2047 encoding is full of specific 
cases, hard to understand rules, and enables an amazing number of 
different possibilities for the encoding of the same string.
The analyses of it that was done during recent discussion in the usefor 
mailing list led to the discovery of incredibly obscure border cases, 
that can only result in an implementor getting it wrong, or having to 
choose between respecting the standard, or refusing that other will 
produce, which will make it look like it is the one that gets it wrong 
given the number of software that will produce the incorrect encodings.

And here raw UTF-8 is a clear winner. No complex implementation rules, 
no border cases, one string will always have one and only one 
representation.

Another choice would be throwing away RFC2047 and devising a new 7 bits 
encoding that does not have all the inconveniences.

This has been debated, but not choosen.

In my opinion, here are the main reasons that justify that :
- reserves against producing yet another encoding (that might itself 
have defaults that are immediatly apparent)
- the fact that with a very wide majority the non US-ASCII world has 
*already* choosen raw 8 bit against 7 bit (RFC2047) encoded data.[1]
- such an encoding would have no support at first in the installed base 
of softwares, whereas both RFC2047 and raw utf-8 at least are already 
supported by a part of them.

[1] This excludes the parts of the world where the standard encoding for

email is seven bit.

content-class: urn:content-classes:message
Subject: RE: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Importance: normal
Date: Tue, 7 Jan 2003 13:14:29 -0800
From: "Dan Kohn" <[email-masked]>
To: "Jean-Marc Desperrier" <[email-masked]>, "Dave Crocker" <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Sender: [email-masked]

IDNA has defined nameprep+punycode as a new 7 bit ACE (ASCII-compatible encoding).  In fact, every i18n newsreader will need to implement punycode to correctly display domains.

I suspect that the better answer for usenet is to tighten the specification of 2047, if that's what's causing interoperability problems.  But, using punycode encoding of Subject headers would enable compliant software to take advantage of the i18n without disrupting existing software, similar to MIME's deployment.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600> 

-----Original Message-----
From: Jean-Marc Desperrier [mailto:[email-masked]] 
Sent: Tuesday, January 07, 2003 12:58
To: Dave Crocker
Cc: [email-masked]; [email-masked]
Subject: Re: UTF-8 over RFC 2047 (Re: Call for Usefor to recharter)

Dave Crocker a écrit:
> [...] (No, folks, please don't correct me it
> is any other number over 16.)

It's now 0x10FFFF.

> Cramming those bits into a 7-bit environment is just one more cramming
> effort.  It stands equal to the others as an alternative that has benefits
> and detriments.

No, because the only standardised 7-bit encoding for unicode is UTF-7 
that has so many drawbacks it's strongly deprecated.

> If you have a text editor, you can view the 7-bit encoding.  The fact that
> it is "ugly", therefore is actually a feature, not a bug.

Don't worry, the "pretty" aspect of UTF-8 is not that, it's truly that 
it's already implemented in most any client that is able to support more 
than one encoding (many more than the one that would support UTF-16).
If it were not, a new seven bit encoding would be better.

Date: Tue, 07 Jan 2003 22:01:08 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Keith Moore a écrit:
>>Well then maybe the IETF needs a reality check too.
> 
> there seems to be a lot of need for reality checks.

It's always a good thing to do a reality check.

[...]
> widespread current practice is to do things that work on a local
> level, even if that practice diverges from one user to another,
> or one group to another. [...]
> 
> in standards work, it's more important to specify something that can
> be made to scalably interoperate than to specify something that is
> compatible with existing practice.

I fully agree with all of that.	
I spend time discussing USEFOR, because I'd love the current 
non-scalable situation to change.

My point was that the argument that it's bad and evil to make an 
incompatible change that would mean that 8 bit data would circulate 
somewhere where it's not allowed in the current standards is weak 
because the actual situation is that a lot of 8 bit data is being sent.

And that the current endorsed solution for sending non US-ASCII data 
(RFC2047) has so many problems that in many situations, messages are 
more likely be handled correctly by sending them in raw 8 bits that by 
using RFC2047.

It was not my point to support any solution that does not scale.
I would 100% approve that a change for something that does not scale is 
bad and evil.

But by saying that I have misrepresented Usefor which I regret.
The current version is almost 100% compatible with email standard.
Maybe it can be made 100% compatible by enhancing the part that is still 
problematic.
I just intended to voice why usefor is favoring an 8 bit solution in 
usenet, not to say change incompatible with email are a good thing if 
there's any way to avoid them.

Date: Tue, 07 Jan 2003 21:57:58 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: Dave Crocker <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Dave Crocker a écrit:
> [...] (No, folks, please don't correct me it
> is any other number over 16.)

It's now 0x10FFFF.

> Cramming those bits into a 7-bit environment is just one more cramming
> effort.  It stands equal to the others as an alternative that has benefits
> and detriments.

No, because the only standardised 7-bit encoding for unicode is UTF-7 
that has so many drawbacks it's strongly deprecated.

> If you have a text editor, you can view the 7-bit encoding.  The fact that
> it is "ugly", therefore is actually a feature, not a bug.

Don't worry, the "pretty" aspect of UTF-8 is not that, it's truly that 
it's already implemented in most any client that is able to support more 
than one encoding (many more than the one that would support UTF-16).
If it were not, a new seven bit encoding would be better.

Date: Tue, 7 Jan 2003 15:12:15 -0500 (EST)
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
From: "Keith Moore" <[email-masked]>
To: <[email-masked]>
Importance: Normal
Cc: <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

you may be correct about the unwillingness of implementors to change.
however, any new encoding scheme would bear the same burden, and would
further increase complexity due to the need to support both 2047 and the
new scheme.

Jean-Marc Desperrier said:
> Keith Moore a écrit:
>> In my experience the problem with RFC 2047 implementations is usually
>> that implementations try to "decode" RFC 2047 for the purpose of
>> internal representation and then to "encode" again for retransmission,
>> rather than to "decode" RFC 2047 only for the purpose of display.
>
> Unfortunately, this is the way most work and not likely to change.
>
> If I were to write the ideal specification for the developpment of a new
>  news/mail reader, yes, it would have exactly the behaviour of keeping
> the subject intact by not decoding/recoding it.
>
> Unfortunately most developper do not see the advantage of it, and do not
>  want to make the extra effort this means.
> Being able to do it requires to test if the subject of the reply has or
> has not been edited by user, and reusing the older subject if there was
> no change.
>
> This is extra code to circumvent the problems with RFC2047, and does not
>  even solve everything, even if it would reduce the worst cases.

Date: Tue, 07 Jan 2003 21:12:06 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Keith Moore a écrit:
> In my experience the problem with RFC 2047 implementations is usually
> that implementations try to "decode" RFC 2047 for the purpose of
> internal representation and then to "encode" again for retransmission,
> rather than to "decode" RFC 2047 only for the purpose of display.

Unfortunately, this is the way most work and not likely to change.

If I were to write the ideal specification for the developpment of a new 
news/mail reader, yes, it would have exactly the behaviour of keeping 
the subject intact by not decoding/recoding it.

Unfortunately most developper do not see the advantage of it, and do not 
want to make the extra effort this means.
Being able to do it requires to test if the subject of the reply has or 
has not been edited by user, and reusing the older subject if there was 
no change.

This is extra code to circumvent the problems with RFC2047, and does not 
even solve everything, even if it would reduce the worst cases.

Date: Tue, 7 Jan 2003 12:01:52 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Jean-Marc Desperrier <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Jean-Marc,

Tuesday, January 7, 2003, 10:15:12 AM, you wrote:
Jean-Marc> And here raw UTF-8 is a clear winner. No complex implementation rules,
Jean-Marc> no border cases, one string will always have one and only one 
Jean-Marc> representation.

Correct me if I am wrong, but I believe the "native" representation for
Unicode is something like 24 bits.  (No, folks, please don't correct me it
is any other number over 16.)

Hence, UTF-16 and UTF-8 are methods of encoding a larger bit space into a
smaller representation space, producing variable-length strings.  One crams
the larger space into a 16-bit world.  The other crams it into an 8-bit
world.

There depending on the situation, there can be processing or space
efficiencies gained by one encoding over another.

But there is no theoretical or aesthetic superiority that can be claimed by
one over the other.

Cramming those bits into a 7-bit environment is just one more cramming
effort.  It stands equal to the others as an alternative that has benefits
and detriments.

The confusion on this issue probably stems from the fact that you can use
existing data viewers -- such as text editors -- to view the result of a
7-bit encoding and cannot use such "legacy" services for viewing UTF-8 or
UTF-16.

If you do not have UTF-8 or UTF-16 tools, you cannot view the data at all.

If you have a text editor, you can view the 7-bit encoding.  The fact that
it is "ugly", therefore is actually a feature, not a bug.

Unless you think that "invisible or broken" is better than "ugly".

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

To: [email-masked], [email-masked]
Subject: Re: Call for Usefor to recharter
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 07 Jan 2003 11:07:35 -0800
Lines: 33
Sender: [email-masked]

ned freed <[email-masked]> writes:

> I have to say I find the recent claim that "in the reverse direction
> News->Email, such general-purpose gateways just Do Not Exist" to be
> pretty amazing, if for no other reason that I have years of direct
> personal experience with such gateways.

I don't know what the phrase "general-purpose" is intended to mean, but
there are a very large number of news->mail gateways in operation today.
I run one here at Stanford for any of our local groups that anyone wishes
gated to a mailing list, so that could be described as general-purpose.  I
believe comp.lang.python is gated back to e-mail, as are multiple
newsgroups in the gnu.* hierarchy.  And those are just the ones that occur
to me off-hand.

The functionality is even built-in to one mailing list manager (Mailman),
and is a common enough request that software to do this comes with INN.

>> Alternatively, if the group decides it's not interested in the IESG
>> imprimatur, I'd suggest that the group consider reforming outside the
>> aegis of the IETF.

> That's also an option, of course.

As an implementor, I'm pretty uninterested in any "standardization" effort
for Usenet that doesn't take place under the imprimatur of the IETF.  If
no coherent IETF working group can be formed, I think we'd still be better
off continuing on the way that we have for the past decade than to try to
follow a process that doesn't have the sorts of checks that the IETF
process has.  I think the structure adds something valuable.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: Jean-Marc Desperrier <[email-masked]>
Subject: Re: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
cc: [email-masked], [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Date: Tue, 07 Jan 2003 13:52:36 -0500
Sender: [email-masked]

> It's a simple fact.
> In every single thread with non US-ASCII data in subject encoded by 
> RFC2047 (sorry I wrote 2049 by error in my last mail) I've seen, the 
> subject turned to garbage after 5 or 6 messages.
> 
> The reason for that is that all implementations of RFC2047 around are 
> full of implementation errors.
> 
> The reason for that is that the RFC2047 encoding is full of specific 
> cases, hard to understand rules, and enables an amazing number of 
> different possibilities for the encoding of the same string.

I don't dispute the observation.  However, I do dispute the reason.

In my experience the problem with RFC 2047 implementations is usually
that implementations try to "decode" RFC 2047 for the purpose of
internal representation and then to "encode" again for retransmission,
rather than to "decode" RFC 2047 only for the purpose of display.

It's the multiple encoding/decoding steps that cause problems,
especially when there's often no clear separation between text that
was originally encoded and text that was originally unencoded
in the same header field.

Keith

To: Jean-Marc Desperrier <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
cc: [email-masked], [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Date: Tue, 07 Jan 2003 13:48:12 -0500
Sender: [email-masked]

> Well then maybe the IETF needs a reality check too.

there seems to be a lot of need for reality checks.
 
> A gateway that will not transparently support 8 bit data in
> headers will _not_ be able to transport the usenet traffic as it
> is now.

and a gateway that sends 8bit traffic in email is sending traffic
that - no matter what character encoding it uses - will not be 
correctly interpreted by the majority of email user agents in 
use today.  such traffic *might* be read correctly by the moderator
of a news group (if that person happens to use the same convention
used by the sender of that message), or it might not.

widespread current practice is to do things that work on a local
level, even if that practice diverges from one user to another,
or one group to another.  but the purpose of standards is to encourage uniform practice, because the interoperability of divergent practice doesn't scale.

in standards work, it's more important to specify something that can
be made to scalably interoperate than to specify something that is
compatible with existing practice.

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: Tue, 07 Jan 2003 19:15:12 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: UTF-8 over RFC 2047  (Re: Call for Usefor to recharter)
Sender: [email-masked]

Dan Kohn wrote :
> I wrote:
>>>I have a simple question.  What can a UTF-8 subject header
>>>communicate that an RFC 2047 one can't?  Other than inelegance,
>>>what's the downside of 2047, when the upside is a huge increase in
>>>backward compatibility? 

I do not know where this discussion took place, but I have an answer to it.

It's a simple fact.
In every single thread with non US-ASCII data in subject encoded by 
RFC2047 (sorry I wrote 2049 by error in my last mail) I've seen, the 
subject turned to garbage after 5 or 6 messages.

The reason for that is that all implementations of RFC2047 around are 
full of implementation errors.

The reason for that is that the RFC2047 encoding is full of specific 
cases, hard to understand rules, and enables an amazing number of 
different possibilities for the encoding of the same string.
The analyses of it that was done during recent discussion in the usefor 
mailing list led to the discovery of incredibly obscure border cases, 
that can only result in an implementor getting it wrong, or having to 
choose between respecting the standard, or refusing that other will 
produce, which will make it look like it is the one that gets it wrong 
given the number of software that will produce the incorrect encodings.

And here raw UTF-8 is a clear winner. No complex implementation rules, 
no border cases, one string will always have one and only one 
representation.

Another choice would be throwing away RFC2047 and devising a new 7 bits 
encoding that does not have all the inconveniences.

This has been debated, but not choosen.

In my opinion, here are the main reasons that justify that :
- reserves against producing yet another encoding (that might itself 
have defaults that are immediatly apparent)
- the fact that with a very wide majority the non US-ASCII world has 
*already* choosen raw 8 bit against 7 bit (RFC2047) encoded data.[1]
- such an encoding would have no support at first in the installed base 
of softwares, whereas both RFC2047 and raw utf-8 at least are already 
supported by a part of them.

[1] This excludes the parts of the world where the standard encoding for 
email is seven bit.

Date: Tue, 7 Jan 2003 09:45:27 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Pete Resnick <[email-masked]>
CC: Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Running code (Was: RFC 2047 and gatewaying)
Sender: [email-masked]

Pete,

I don't know whether anyone else is having this reaction, but I suspect this
sub-thread is a deep, dark rat-hole.

When the IETF starts a new standards effort, for something that has been in
operation pre-IETF, an explicit question is the amount of attention to be
paid to preservation of the installed base, versus "merely" taking prior
experience as guidance to what is really a new engineering effort.

For any substantial installed base, we tend to err in the direction of
preserving that installed base -- ie, documenting running code -- rather
than starting fresh.

But not always.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 7 Jan 2003 09:30:25 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Charles Lindsey" <[email-masked]>
CC: [email-masked]
Subject: prevervation of installed base
Sender: [email-masked]

Charles,

I have included segments from your posting that pertain to the question of
impact on installed base.  This issue is fundamental.

To add to Ned's correction, news-> email gatewaying has been around for at
least 20 years, as far as I can recall. Organizations have often wanted to
plug mailing lists into newsgroups. To permit the newsgroup readers to
participate in the mailing list, a news->email gateway is required. In other
environments, the 2-way gatewaying is simply part of a model that lets users
decide how they want to receive and process their group discussion messages.

With respect to the appearance of new capabilities (such as new data formats
and encodings) in the installed base, your use of the term "gobbledegook"
needs to be made more precise. If it means that that a legacy system
receives the data and the data are legal but meaningless -- MIME base64 is
example of this approach -- then it is fine. It permits incremental adoption
without breaking existing systems. The downside for existing systems is that
they do not get the benefit of the new feature, but everything else
continues to work fine.

If it means that the legacy system receives data that are illegal, with
respect to the existing service, then this is not at all fine.
Interoperability requires conforming to a standard, rather than conforming
to it whenever it is convenient. Injecting illegal data is not conforming.

The difference between an upgrade that protects the installed base, versus
an upgrade that requires creating a new installed base is absolutely
fundamental.  Each can be appropriate but explicitly choosing between them
is essential.

The importance of preserving the installed base -- by way of doing an
incremental upgrade rather than a parallel replacement -- is frequently
missed. However the market power of an installed base is massive. Engineers
and vendors ignore it at their peril.

The very strong IETF philosophy has been to try to preserve the installed
base, through optional, incremental upgrade.  It is part of the reason that the
architectural history has been to keep the core infrastructure as simple as
possible.

Infrastructure changes rarely permit incremental upgrade. Hence, a change to
the infrastructure often must be done to the *entire* infrastructure before
there can be benefit to any users. Think about the effort -- and delay --
this requires for a large infrastructure, like the IP Internet, or Usenet. A
delay that is effectively infinite becomes probable when the infrastructure
is very large.

It is certainly true that this approach creates solutions that are ugly. The
encoding aspect of MIME is the epitome of ugliness, in my opinion. And, yes,
incremental ugliness can create an aggregate messiness that eventually
requires complete replacement.  And it is important to look for that
critical mass of cumbersomeness.

But the reason for choosing the MIME ugliness was to permit incremental
upgrade while preserving the installed base -- and to make the upgrade be
strictly in the leaves, with no changes at all to the email infrastructure.

There were previous attempts to introduce multi-media email to the Internet,
through operation of an independent, parallel service that had a multi-media
core.  The installed base ignored those efforts.

d/

Monday, January 6, 2003, 6:39:19 AM, you wrote:
>>This is perhaps Ned's most important statement.  At countless times in
>>the last 20 years, the IETF and it's predecessors have chosen backward
>>compatibility over the "efficient" solution.  ...
>>  The economic calculation was that clients that care about i18n
>>can implement IDNA, but that the IETF would not break the "social
>>contract" made with DNS software that was (and now still will be)
>>compatible with pre-IDNA standards.

Charles> Yes, it's the tension between the "elegant" solution and the "ugly but
Charles> compatible solution". But there is the added problem that, every time you
Charles> adopt an "ugly" solution you make it yet harder again to use the "elegant"
Charles> solution next time, ...

Charles>  Breaking
Charles> compatibility for Email could be a serious matter. Breaking compatibility
Charles> for Usenet is far far less so...

>>Similarly, a usefor solution that does not immediately cause existing
>>software (*including* gateways) to "break" on usefor messages, seems
>>1000 times more likely to pass IESG muster.  To quote Ned again:

>>> The issue is instead that your approach in effect declares a large
>>> body of software as being no longer compliant with long established
>>> standards. This is something we try very hard not to do.

Charles> Which shows that people have a fundamental misunderstanding of the role of
Charles> gateways in Usenet. ...

Charles> But in the reverse direction News->Email, such general-purpose gateways
Charles> just Do Not Exist. ...

Charles> They will appear gradually, in groups where the
Charles> customary language will benefit from them. Users on the far side of
Charles> gateways will notice strange gobbledegook appearing. ...

Charles> No, it's just the inelegance. Plus the fact that the backward
Charles> compatibility issue is nowhere so huge as you imagine. In fact, it is
Charles> rather small.

d/
-- 
 Dave <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.408.850.1850

Date: Tue, 07 Jan 2003 18:23:46 +0100
From: Jean-Marc Desperrier <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

[email-masked] a écrit:
> Like it or not, one of the things RFC 1036 standardizes is the interoperability
> of and ability to gateway between netnews and email. And this capability has
> been used to advantage in IETF protocols like IMAP4, which has facilities
> explicitly intended to accomodate netnews.  This has led to countless
> implementations of news<-->email gatways and IMAP4 servers that can access
> news. [...] They exist in abundance and are based
> on IETF standards, and therefore cannot simply be dismissed as irrelevant.

Well then maybe the IETF needs a reality check too.

A gateway that will not transparently support 8 bit data in headers will 
_not_ be able to transport the usenet traffic as it is now.

This gateways either can't fulfill their duty, or can not be based on 
strict conformance to IETF standards.

For exemple, the tw.bbs.* subnetwork is fully relayed to a network of 
bbs, but supports 8 bit data everywhere in header, even in the 0x80-0x9F 
  range ! About every single message subject is using 8 bits data.

So it would be useful for the IETFT to consider that some of the 
standards are not used today (namely RFC2047) and doing this would in my 
opinion show some very clear reasons _why_ they are not used.

From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: re: running code (was: rfc 2047 and gatewaying)
Date: Tue, 07 Jan 2003 11:27:39 -0500
Sender: [email-masked]

[resend: forgot to cc list in original reply]

> >I realize I'm taking this out of context, but...
> >the criteria for standardization levels are documented in RFC 2026.
> >words like "no known technical omissions" are used.  it doesn't say anything
> >about standards documenting the state of running code.
> 
> 2026 does say things about requiring interoperable implementations. 

yes, but they're not required for proposed standards - and the 
requiremnt for interoperable implementations is to test the
clarity of the spec, *not* for the spec to document implementations.
if the implementations don't agree with the spec, we don't automatically
change the spec to reflect the implementations.  instead, we try to 
understand whether the difference is due to a lack of clarity in the
spec (in which case we clarify it) or whether it is due to the failure
of the spec to specify what is really needed (in which case we fix the
spec and typically recycle at proposed).  but we don't assume that the 
running code is correct, and we don't constrain ourselves to make
the revised spec be consistent with running code.  

> And let's be very clear about what 2026 actually says: As far as "no 
> known technical omissions" goes, 2026 says that this requirement can 
> be waived under certain conditions. 

yes, but in practice this is an exceptional case, not the normal case.
IESG rarely waives the requirement.

>"Interoperable implementations" 
> and "operational experience" (which, of course, don't kick in as 
> requirements until Draft Standard) have no exceptions whatsoever 
> mentioned in 2026. Moreover, those requirements are *precursors* to 
> standardization: You don't get to move along the standards track 
> unless in the standard you are documenting actual running code.

not past proposed standard.   though, again, if errors are found
in the proposed standard document the IESG will sometimes approve
fixes to the draft standard version that aren't implemented
in running code.  (if those fixes won't interoperate with running
code, then IESG is more likely to reset to proposed, but that's not
the same thing as having them implemented first.)

> >in my experience, trying to document running code is often a good 
> >idea, and trying to define a standard is often a good idea, but 
> >trying to do both in the same exercise leads to disaster - there is 
> >inevitably a conflict between what meets IETF criteria for 
> >standardization and what is actually done in running code.  the 
> >result is usually that people are tempted to lie about either what 
> >the running code does, or worse, to  lie about whether the running 
> >code meets standardization criteria.
> 
> My experience is that more often, people fudge in the opposite 
> direction: They write the standard for the way they *want* the 
> running code to be, rather than having the standard represent what 
> the actual interoperable implementations are doing.

that's the way it should be.  the problem occurs when people get
confused between whether they're documenting existing practice or
writing a standard.

> >>When we work on Proposed Standards where there is no running code, 
> >>we sometimes propose ways of doing things and ask people to make 
> >>their implementations a certain way, but we don't tell people who 
> >>are following a current standard that they are hereafter wrong 
> >>without pretty impressive reasons. We do "de facto" standards, not 
> >>"de jure" ones.
> >
> >it's true that we don't often tell people it's wrong to follow a 
> >current standard - at least, not without a good reason, and 
> >especially not when it's one of *our* standards.  but our standards 
> >are closer to de jure standards than de facto ones.  documenting 
> >running code is NOT what we do in our standards process, at least, not when we're following our rules.
> 
> I disagree. Most other standards bodies create de jure standards: Not 
> a lick of code is written before standardization, and no operational 
> experience is required for standardization. What we do (in comparison 
> to others) is *much* more like de facto standards: We require, as 
> part of standardization, interoperability and operational experience.

we do require that to get to draft standard status.  but
since most of our standards never make it past proposed, 
it's not required in practice.

and even when we do require running code before standardization, the purpose of the standard is still to document how things _should_ work, not to describe how running code does work.  if the code doesn't implement the standard, the code is suspect before the standard is.

> None of this is meant to say that the what the IETF does is purely 
> documentation; certainly we do original work all of the time. But 
> what we put in standards track RFCs better be, for the most part, a 
> description of running code, not just rules for how we think running 
> code should go. The latter are called "Experimental RFCs". They're 
> different.

no, you have it backwards.

standards-track RFCs describe how things should work (as best we
understand "should"); informational/experimental RFCs can be used to describe existing practice.

--
Keith Moore                              http://www.cs.utk.edu/~moore/
27 February 1933                                     11 September 2001

Date: Tue, 7 Jan 2003 09:57:41 -0600
To: Keith Moore <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Running code (Was: RFC 2047 and gatewaying)
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

On 1/7/03 at 7:35 AM -0500, Keith Moore wrote:

>On Mon, 6 Jan 2003 23:31:20 -0600 Pete Resnick <[email-masked]> wrote:
>
>>Just to make sure this doesn't get by without comment:
>>
>>On 1/4/03 at 2:06 AM -0500, Leo Bicknell wrote:
>>
>>>Change the RFC, then complain about the program.
>>
>>Sorry, we don't do that in the IETF. The IETF standards process 
>>doesn't make up rules and then make everybody fix their code. In 
>>the IETF, standards simply document the current state of affairs of 
>>interoperable running code.
>
>I realize I'm taking this out of context, but...
>the criteria for standardization levels are documented in RFC 2026.
>words like "no known technical omissions" are used.  it doesn't say anything
>about standards documenting the state of running code.

2026 does say things about requiring interoperable implementations. 
And let's be very clear about what 2026 actually says: As far as "no 
known technical omissions" goes, 2026 says that this requirement can 
be waived under certain conditions. "Interoperable implementations" 
and "operational experience" (which, of course, don't kick in as 
requirements until Draft Standard) have no exceptions whatsoever 
mentioned in 2026. Moreover, those requirements are *precursors* to 
standardization: You don't get to move along the standards track 
unless in the standard you are documenting actual running code.

>in my experience, trying to document running code is often a good 
>idea, and trying to define a standard is often a good idea, but 
>trying to do both in the same exercise leads to disaster - there is 
>inevitably a conflict between what meets IETF criteria for 
>standardization and what is actually done in running code.  the 
>result is usually that people are tempted to lie about either what 
>the running code does, or worse, to  lie about whether the running 
>code meets standardization criteria.

My experience is that more often, people fudge in the opposite 
direction: They write the standard for the way they *want* the 
running code to be, rather than having the standard represent what 
the actual interoperable implementations are doing.

>>When we work on Proposed Standards where there is no running code, 
>>we sometimes propose ways of doing things and ask people to make 
>>their implementations a certain way, but we don't tell people who 
>>are following a current standard that they are hereafter wrong 
>>without pretty impressive reasons. We do "de facto" standards, not 
>>"de jure" ones.
>
>it's true that we don't often tell people it's wrong to follow a 
>current standard - at least, not without a good reason, and 
>especially not when it's one of *our* standards.  but our standards 
>are closer to de jure standards than de facto ones.  documenting 
>running code is NOT what we
>do in our standards process, at least, not when we're following our rules.

I disagree. Most other standards bodies create de jure standards: Not 
a lick of code is written before standardization, and no operational 
experience is required for standardization. What we do (in comparison 
to others) is *much* more like de facto standards: We require, as 
part of standardization, interoperability and operational experience.

None of this is meant to say that the what the IETF does is purely 
documentation; certainly we do original work all of the time. But 
what we put in standards track RFCs better be, for the most part, a 
description of running code, not just rules for how we think running 
code should go. The latter are called "Experimental RFCs". They're 
different.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Tue, 7 Jan 2003 07:35:15 -0500
From: Keith Moore <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

On Mon, 6 Jan 2003 23:31:20 -0600
Pete Resnick <[email-masked]> wrote:

> 
> Just to make sure this doesn't get by without comment:
> 
> On 1/4/03 at 2:06 AM -0500, Leo Bicknell wrote:
> 
> >Change the RFC, then complain about the program.
> 
> Sorry, we don't do that in the IETF. The IETF standards process 
> doesn't make up rules and then make everybody fix their code. In the 
> IETF, standards simply document the current state of affairs of 
> interoperable running code. 

I realize I'm taking this out of context, but...
the criteria for standardization levels are documented in RFC 2026.
words like "no known technical omissions" are used.  it doesn't say anything
about standards documenting the state of running code.

in my experience, trying to document running code is often a good idea,
and trying to define a standard is often a good idea, but trying to do
both in the same exercise leads to disaster - there is inevitably a 
conflict between what meets IETF criteria for standardization and what
is actually done in running code.  the result is usually that people are
tempted to lie about either what the running code does, or worse, to 
lie about whether the running code meets standardization criteria.

> When we work on Proposed Standards where 
> there is no running code, we sometimes propose ways of doing things 
> and ask people to make their implementations a certain way, but we 
> don't tell people who are following a current standard that they are 
> hereafter wrong without pretty impressive reasons. We do "de facto" 
> standards, not "de jure" ones.

it's true that we don't often tell people it's wrong to follow a current
standard - at least, not without a good reason, and especially not when
it's one of *our* standards.  but our standards are closer to de jure
standards than de facto ones.  documenting running code is NOT what we
do in our standards process, at least, not when we're following our rules.

Keith

Date: Mon, 06 Jan 2003 22:20:14 -0800 (PST)
From: [email-masked]
Subject: Re: Call for Usefor to recharter
To: Dan Kohn <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> I wrote:

> >> I have a simple question.  What can a UTF-8 subject header
> >> communicate that an RFC 2047 one can't?  Other than inelegance,
> >> what's the downside of 2047, when the upside is a huge increase in
> >> backward compatibility?

> Charles Lindsey responded:

> > No, it's just the inelegance. Plus the fact that the backward
> > compatibility issue is nowhere so huge as you imagine. In fact, it is
> > rather small.

> Here, I believe, is the crux of the disagreement between the working
> consensus of usefor and the unanimous agreement of email folks from
> ietf-822.

> Usefor recently conducted a straw poll [1] in which raw UTF-8 headers
> barely won out [2] over using 2047/2231.  (Specifically, in the #1 vs.
> #5 vote, the tally was 9 to 8.)  The chair, Dave Barr, declared "rough
> consensus" [3] and suggested the group should move forward on that
> basis.

> I suggest to the usefor chair that the group should conduct the poll
> again, based on a new piece of information.  Specifically, they've seen
> the applicable area director react in such a viscerally negative fashion
> [4] that it is nearly impossible to imagine anything resembling the
> current usefor approach being approved by the IESG.  I.e., the backward
> compatibility issue is nowhere near as small as the group seems to
> believe.

> (Ned, if I'm overstating your viewpoint and its implications, my
> apologies, and please correct my error.)

It overstates things, but only slightly. First, I'm not the apps AD presently
responsible for usefor; Patrik Falstrom is. However, I am on the IESG, and at
some point this will be something I have to vote on.

My current position here is first that the use of message/rfc822 in the
proposed way is unacceptable, period. As for the general use of utf-8 in the
context of news, I'm having a great deal of trouble buying the argument that
this doesn't disenfranchise any number of gateways between the two
environments.

I have to say I find the recent claim that "in the reverse direction
News->Email, such general-purpose gateways just Do Not Exist" to be pretty
amazing, if for no other reason that I have years of direct personal experience
with such gateways. For example, as many of you know, I used to be a frequent
contributor to the info-pmdf mailing list, and that list was for several years
gatewayed bidirectionally between netnews and email. (This particular gateway
was discontinued in the news->email direction at some point, but other gateways
of this sort continue to operate.)

> Based on this new information, I would suggest that the group consider
> rechartering with a more precise plan and schedule, both approved by the
> IESG, in such a matter that successful publication of a standards-track
> RFC is a likely (or at least possible) outcome.

An excellent idea, one I would fully support.

> Alternatively, if the group decides it's not interested in the IESG
> imprimatur, I'd suggest that the group consider reforming outside the
> aegis of the IETF.

That's also an option, of course.

				Ned

Date: Mon, 06 Jan 2003 21:25:15 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> However, looking at the precedents established in the case of HTTP, I see
> the use of phraseology such as "MIME-like" and "different use of Internet
> Media Types than is typically found in Internet mail". More specifically,
> I see that the media type text/plain is different in that naked CR and LF
> are allowed, and even line endings that are totally different in the event
> that multi-byte charsets are used. And, again, in their usage of
> multipart/mixed they allow the use of HTTP headers as body part headers,
> with no suggestion that headers not of the form "Content-*" might get
> discarded en route.

> And naturally they warn that precautions are necessary if ever the HTTP
> object is to be moved into a pure (Email) MIME environment, which is
> exactly the same as I am proposing to do with Netnews.

There were several reasons for allowing this in HTTP: (1) Text media types were
already widely deployed in HTTP with different semantics than in email at the
time this was added to the HTTP standard, (2) HTTP and email are sufficiently
disjoint applications that incompatibilities in their interpretation of media
types are unlikely to cross over, and (3) To the extent there was crossover
gateways and converters were aware from the get-go of the need to convert
between the two.

None of these considerations apply to use of message/rfc822 in news: (1) You
yourself have argued that MIME elements aren't deployed in the context of
netnews, (2) Cases abound where material passes back and forth between netnews
and email, (3) Existing gateways aren't going to know what to do and therefore
will not handle this well if at all.

But even if the cases were comparable, this barely squeaked by in HTTP at the
time. And knowing what we know now about this stuff, many people, myself
included, now believe it was a mistake to have allowed this. So if we had it to
do over again I very much doubt it would be allowed.

> >> 8. Insofar as MIME introduces the use of headers within the bodies of
> >> articles/messages, it is only to be expected that within Netnews the
> >> general conventions regarding Netnews headers should apply to them. Again,
> >> any resulting messiness is to be confined to the gateways.
> >
> >And that's fine. Just don't reuse an existing media type in the process.

> OK, so you have no problem with
> 	Content-Disposition: attachment; filename="?-?-?-?"
> where "?-?-?-?" is a placeholder for some filename in Hungarian, Arabic or
> Chinese. Just so long as that usage is confined to Netnews, and is not
> seen in Email except in the corresponding RFC 2231 form?

I assume the ?-?-?-? is intended to represent a utf-8 string. If so, then
of course I have a problem with this. It is a different problem than
the one with message/rfc822 usage, but a problem nevertheless.

> And, naturally, that usage will be seem in the body part headers of a
> multipart/mixed (again within Netnews, not Email)? For that is precisely
> where you would expect "attachments" to be found.

> So I am allowed to reuse the existing media type multipart/mixed.

> But not message/rfc822?

Sigh. I again repeat: You can use any media type in any way you want as long as
you do not change its definition. What you cannot do is redefine it. Media
types are supposed to define content independent of transport. If I have
a handler for message/rfc822 objects I should be able to call it from a
any sort of application that deals with media types and it should be able
to work.

This completely transcends the issue of gateways and their ability to convert
between netnews and email.

> >> 9. In fact, there are only two places where MIME in Netnews will differ
> >> from MIME in Email. One is in allowing full UTF-8 within quoted-strings
> >> within parameters in the Content-* headers (and probably in comments
> >> within those headers too). The other is in "message/rfc822". Both these
> >> cases therefore require attention by gateways.

> It seems you are accepting the first difference, but not the second?

I never said anything of the sort. I accept neither but for different reasons.

				Ned

Date: Mon, 6 Jan 2003 23:31:20 -0600
To: Leo Bicknell <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Cc: Russ Allbery <[email-masked]>, [email-masked]
Sender: [email-masked]

Just to make sure this doesn't get by without comment:

On 1/4/03 at 2:06 AM -0500, Leo Bicknell wrote:

>Change the RFC, then complain about the program.

Sorry, we don't do that in the IETF. The IETF standards process 
doesn't make up rules and then make everybody fix their code. In the 
IETF, standards simply document the current state of affairs of 
interoperable running code. When we work on Proposed Standards where 
there is no running code, we sometimes propose ways of doing things 
and ask people to make their implementations a certain way, but we 
don't tell people who are following a current standard that they are 
hereafter wrong without pretty impressive reasons. We do "de facto" 
standards, not "de jure" ones.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Mon, 06 Jan 2003 21:13:01 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Keith Moore <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> > However, looking at the precedents established in the case of HTTP, I see
> > the use of phraseology such as "MIME-like" and "different use of Internet
> > Media Types than is typically found in Internet mail". More specifically,
> > I see that the media type text/plain is different in that naked CR and LF
> > are allowed, and even line endings that are totally different in the event
> > that multi-byte charsets are used.

> please don't cite past accidents as justifications for deliberately
> introducing needless complexity in the future.

Exactly. We blew it badly in this case; if anything it should serve to make us
more diligent now.

				Ned

content-class: urn:content-classes:message
Importance: normal
Subject: Call for Usefor to recharter
Date: Mon, 6 Jan 2003 20:42:24 -0800
From: "Dan Kohn" <[email-masked]>
To: "Charles Lindsey" <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

I wrote:

>> I have a simple question.  What can a UTF-8 subject header
>> communicate that an RFC 2047 one can't?  Other than inelegance,
>> what's the downside of 2047, when the upside is a huge increase in
>> backward compatibility? 

Charles Lindsey responded:

> No, it's just the inelegance. Plus the fact that the backward
> compatibility issue is nowhere so huge as you imagine. In fact, it is
> rather small.

Here, I believe, is the crux of the disagreement between the working
consensus of usefor and the unanimous agreement of email folks from
ietf-822.

Usefor recently conducted a straw poll [1] in which raw UTF-8 headers
barely won out [2] over using 2047/2231.  (Specifically, in the #1 vs.
#5 vote, the tally was 9 to 8.)  The chair, Dave Barr, declared "rough
consensus" [3] and suggested the group should move forward on that
basis.

I suggest to the usefor chair that the group should conduct the poll
again, based on a new piece of information.  Specifically, they've seen
the applicable area director react in such a viscerally negative fashion
[4] that it is nearly impossible to imagine anything resembling the
current usefor approach being approved by the IESG.  I.e., the backward
compatibility issue is nowhere near as small as the group seems to
believe.

(Ned, if I'm overstating your viewpoint and its implications, my
apologies, and please correct my error.)

Based on this new information, I would suggest that the group consider
rechartering with a more precise plan and schedule, both approved by the
IESG, in such a matter that successful publication of a standards-track
RFC is a likely (or at least possible) outcome.

Alternatively, if the group decides it's not interested in the IESG
imprimatur, I'd suggest that the group consider reforming outside the
aegis of the IETF.

[1] http://www.landfield.com/usefor/2002/Nov/0022.html
[2] http://www.landfield.com/usefor/2002/Dec/0004.html
[3] http://www.landfield.com/usefor/2002/Dec/0009.html
[4] http://www.landfield.com/usefor/2003/Jan/0000.html

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Mon, 6 Jan 2003 23:12:00 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

> However, looking at the precedents established in the case of HTTP, I see
> the use of phraseology such as "MIME-like" and "different use of Internet
> Media Types than is typically found in Internet mail". More specifically,
> I see that the media type text/plain is different in that naked CR and LF
> are allowed, and even line endings that are totally different in the event
> that multi-byte charsets are used.

please don't cite past accidents as justifications for deliberately 
introducing needless complexity in the future.

Keith

To: [email-masked]
Xref: clerew local.mime:2065
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 14:39:19 GMT
Lines: 115
Sender: [email-masked]

In <[email-masked]> "Dan Kohn" <[email-masked]> writes:

>Ned said:

>> Yawn. Heard all these arugments before back with MIME. This doesn't
>> address the backwards compatibility issue, and like it or not this is
>> an issue you are going to have to address.

>This is perhaps Ned's most important statement.  At countless times in
>the last 20 years, the IETF and it's predecessors have chosen backward
>compatibility over the "efficient" solution.  The (IESG-approved) IDNA
>drafts epitomize this approach, by promulgating an "ugly-looking"
>transfer encoding (punycode) rather than the "elegant" solution of
>UTF-8.  The economic calculation was that clients that care about i18n
>can implement IDNA, but that the IETF would not break the "social
>contract" made with DNS software that was (and now still will be)
>compatible with pre-IDNA standards.

Yes, it's the tension between the "elegant" solution and the "ugly but
compatible solution". But there is the added problem that, every time you
adopt an "ugly" solution you make it yet harder again to use the "elegant"
solution next time, until eventually you reach a saturation point from
which there is no escape. That is the essence of the lecture Kai has been
giving us. From the view of the outsider, Email seems already to have
reached that point, and they want to protect Netnews from following that
same path. Again, they perceive the recent IDNA as the final nail in that
coffin.

However, the ills (perceived or real) of IDNA and Email are not my
problem, except insofar as Usefor have committeed me to ensuring that any
Email generated by Netnews is compliant, at some cost to gateways if
necessary.

Usefor wants to go the "elegant" route, insofar as contraints will allow.
There is a downside to this (e.g. in extra work for gateways); the
question is whether it is a better compromise. And what is a bad
compromise for one protocol is not necessarily so for another. Breaking
compatibility for Email could be a serious matter. Breaking compatibility
for Usenet is far far less so, and hence the elegant solution is in with a
better chance.

>Similarly, a usefor solution that does not immediately cause existing
>software (*including* gateways) to "break" on usefor messages, seems
>1000 times more likely to pass IESG muster.  To quote Ned again:

>> The issue is instead that your approach in effect declares a large
>> body of software as being no longer compliant with long established
>> standards. This is something we try very hard not to do.

Which shows that people have a fundamental misunderstanding of the role of
gateways in Usenet. In the direction Email->News there are indeed lots of
general purpose gateways that expect to work with arbitrary emails and
convert them into articles in arbitrary groups. But gatewaying in that
direction is easy (relatively speaking).

But in the reverse direction News->Email, such general-purpose gateways
just Do Not Exist. Nobody is trying to take the whole of Usenet and
convert it into email. Instead, what you have is a large number of small
gateways each serving a very specific community of users - usually by
converting a single newsgroup into a mailing list. They tend to be written
by ad hoc means to serve their particular communities, which usually means
that they are concerned only with a single language and a single character
set. If the rest of Usenet suddenly starts using UTF-8, they will neither
notice nor care. If they do care, then they will lean on the adminisitrator
of their gateway and he will fix it.

The great myth that is being put around is that, overnight, myriads of
articles in strange character sets will suddenly descend on every Usenet
group. They won't. They will appear gradually, in groups where the
customary language will benefit from them. Users on the far side of
gateways will notice strange gobbledegook appearing. Either will they
ignore it, or they will flame the senders of it, or they will ask the
gateway owner to fix it. Said gateway owner will observe the there is a
new standard out, and will take steps to comply with it.

That is how change takes place on Usenet. We have been through it all
before, for example when RFC 2047 started appearing in headers, and then
wheh quoted-printable started appearing in bodies. No, that approach would
not work for Email (well, not as well) but on Usenet things are much more
relaxed. Moreover, Usenet has a built-in mechanism for propagating
information about the system itself (no, it does not work instantaneously,
but ultimately everybody on Usenet is connected to everybody else - things
get around).

>Kai's argument is quite similar to the arguments made in IDN for "just
>use UTF-8".  A strong consensus was reached that the elegance of UTF-8
>was a far lower priority than backward compatibility.  For RFC 2822, the
>value of backward compatibility over elegance was even more obvious.  If
>you care about wasting bits in RFC 2047 encoding of UTF-8, start
>worrying about the ATM cell tax of every DSL line, or the obnoxious
>SONET overhead.  But if you want to deploy a standard in the IETF, focus
>on backward compatibility.

As you say, it is a matter of balance. My point is that the balance on
Usenet is quite different, because it is a different sort of place.

>I have a simple question.  What can a UTF-8 subject header communicate
>that an RFC 2047 one can't?  Other than inelegance, what's the downside
>of 2047, when the upside is a huge increase in backward compatibility?

No, it's just the inelegance. Plus the fact that the backward
compatibility issue is nowhere so huge as you imagine. In fact, it is
rather small.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2071
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 17:07:33 GMT
Lines: 102
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> In <[email-masked]> [email-masked] writes:

>> 6. MIME is a set of protocols defined for the Email world, as is
>> explicitly stated in RFC 2045. Officially speaking, there is no such thing
>> as MIME within Netnews (which accounts for its extremely poor uptake
>> within Usenet).

>Wrong on several counts. First, MIME is a format in addition to being a
>protocol. This has serious implications in regards to how tightly you can draw
>boundaries around a particular use of MIME in a specific application transport.

>Second, while MIME was originally defined and targetted at email, it lost that
>focus long ago -- certainly before RFCs 2045-2049 came out. The current MIME
>documents try to make it clear what parts are email-specific and what parts are
>more generally applicable. And media types are one of things that are more
>generally applicable than just to email.

Well that is far from clear from a reading of RFC 204[56] (I grant you
that the converse is not clear either).

However, looking at the precedents established in the case of HTTP, I see
the use of phraseology such as "MIME-like" and "different use of Internet
Media Types than is typically found in Internet mail". More specifically,
I see that the media type text/plain is different in that naked CR and LF
are allowed, and even line endings that are totally different in the event
that multi-byte charsets are used. And, again, in their usage of
multipart/mixed they allow the use of HTTP headers as body part headers,
with no suggestion that headers not of the form "Content-*" might get
discarded en route.

And naturally they warn that precautions are necessary if ever the HTTP
object is to be moved into a pure (Email) MIME environment, which is
exactly the same as I am proposing to do with Netnews.

>> 7. Usefor rectifies this by defining the usage of the MIME protocols
>> within Netnews.

>And that's a perfectly legitimate thing to do. But you cannot break the
>definition of existing media type as part of this process. Pick another media
>type, for heaven's sake!

>And to the extent that netnews is not a world unto itself, you have to take the
>effects of the sort of MIME you select on other things into account.

Yes, we are aware of that.

>> 8. Insofar as MIME introduces the use of headers within the bodies of
>> articles/messages, it is only to be expected that within Netnews the
>> general conventions regarding Netnews headers should apply to them. Again,
>> any resulting messiness is to be confined to the gateways.
> 
>And that's fine. Just don't reuse an existing media type in the process.

OK, so you have no problem with
	Content-Disposition: attachment; filename="?-?-?-?"
where "?-?-?-?" is a placeholder for some filename in Hungarian, Arabic or
Chinese. Just so long as that usage is confined to Netnews, and is not
seen in Email except in the corresponding RFC 2231 form?

And, naturally, that usage will be seem in the body part headers of a
multipart/mixed (again within Netnews, not Email)? For that is precisely
where you would expect "attachments" to be found.

So I am allowed to reuse the existing media type multipart/mixed.

But not message/rfc822?

>> 9. In fact, there are only two places where MIME in Netnews will differ
>> from MIME in Email. One is in allowing full UTF-8 within quoted-strings
>> within parameters in the Content-* headers (and probably in comments
>> within those headers too). The other is in "message/rfc822". Both these
>> cases therefore require attention by gateways.

It seems you are accepting the first difference, but not the second?

If that is the case, then there are things that could be done with
message/rfc822. One of them is to revert to the type message/news which we
were proposing to obsolete. The problem with that is that current MUAs do
not recognise it (it was never implemented AFAIK). Or we might insist that
the gateway conversions were done _before_ constructing the message/rfc822
object, but that is hardly necessary within the pure Netnews environment
(not that we expect encapsulated news articles in message/anything to be
at all common - currently they are virtually unknown - but part of the
intent in all of this is to encourage more sensible usages of MIME within
Netnews).

So, yes, this is an issue that maybe I take back to the Usefor WG, but I
would need a little more convincing and understanding of your position
first.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2069
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Mon, 6 Jan 2003 15:47:33 GMT
Lines: 86
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>So you're saying that given

>    List-Owner: <mailto:[email-masked]%3E>

>you would construct

I see no non-ASCII characters in there, so I see no problem relevant to
this discussion. Pete Resnick has explained fully why it is irrelevant.

>   To: [email-masked]%3E

I've no idea. I would apply whatever rules are specified by the relevant
RFCs. It that's what results, then it is correct. If it doesn't, then it
isn't.

>Good luck.

>> Now all this could change if IRIs (draft-duerst-iri-02.txt) should get to
>> the standard stage.

>No, it would not change one whit. List-Owner et al. specifically refer
>to URIs, and that would remain the case unless and until the defining
>RFC is superseded.

You really need to understand the difference between "could" and "would".

I said it "could" change, and you said it "would not" unless and until the
rules for List-Owner changed. In that case it presumably "would". Ergo, it
"could".

Now whilst there is little likelihood that List-Owner will change in that
way, it is highly likely that, if draft-duerst-iri-02.txt is accepted,
_some_ protocols will start to use it, and certainly some future extension
to Usefor might then do so. So it is worth giving a little thought to it
at this stage, because it is probable that a transformation back to URI
format would need be be included in the Usefor gatewaying rules in that
case.

>2. Comments may well exist within angle brackets, e.g.:

>   Return-Path: <(=?us-ascii?q?foo?=)>

But there is no non-ASCII there, so no problem.

Now if I had
	Return-Path: <(?-?-?-?)>
where "?-?-?-?" should be regarded as a placeholder for some sensible
comment in Hungarian, Arabic or Chinese, then my proposed workaround
would transform it into
	Return-Path: =?iso-8859-1?q?<(=3F-=3F-=3F-=3F>)?=
which, I will grant you, is not a valid Return-Path (I never claimed that
my workaround would work in all cases). It _might_, nevertheless, be
displayed correctly by a tolerant User-Agent. Indeed, I just tried it on
my Netscape, and it did just that.

But before you throw up your hand in horror, you have to account for how
that header came to be in a Usenet article in the first place. It is not
compliant with Usefor (it is not a Usefor-defined header). It is not
supposed even to be a Mail header, except after final RFC 2821 delivery,
and then it would not contain any Non-ASCII character. So how did it get
there? Some perverse poster put it there by hand, perhaps?

No, before you can complain about my workaround, you have to posit some
plausible circumstance in which it could arise. The only case I can
imagine is where someone in the future has published an extension to the
Usefor standard AND there is some gateway that understands Usefor but does
not know how to parse that extension header AND that someone produced such
a header containing some UTF-8 character AND that the gateway produced a
bad guess at how that header should look in email AND that that header
somehow mattered in the email version (even though it was designed as a
Netnews header).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2070
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 16:11:50 GMT
Lines: 57
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>It is insufficient to say "comply with 2822 / 2047 / 2231); there is
>a design defect which prevents that, unless a specific mechanism is
>provided to ensure compliance.  As it stands,
>1. it is impossible for a gateway to determine the syntax of an
>    experimental header which the gateway author is unaware of
>2. Without header field syntax information, it is impossible to
>    encode non-ASCII "header" content.

Yes, there is certainly a design defect, which accounts for why so many
implementations have chosen to fudge their RFC 2047 handling in so many
ways.

>The design defect can be eliminated by:

You cannot eliminate a desgin defect, but you can maybe work around it:

>a. forbidding non-ASCII content in all but a well-defined set which
>    all gateways are required to recognize, *and* providing a practical
>    transition plan so that existing gateways can be upgraded while
>    ensuring that the mail protocols are not violated

Which is what Usefor is trying to do. But Usefor cannot prevent future
extensions to itself from adding to the set. So you still need a fudge.

As to those existing gateways, I have answered this elsewhere. You would
do well to study a few actual gateways before asserting what would need to
be done to them.

>or
>b. simply using the RFC 822 / 2822 / 2047 / 2231 Internet text message
>    format, which avoids the encoding problem in gateways and adds no
>    new restrictions on gateway transformations (thereby obviating that
>    aspect of a transition plan).

Yes, but Usefor has chosen a different route.

>or
>c. require that unrecognized "headers" with non-ASCII content be elided
>    or moved to an encoded attachment.

And of your various suggestions, that is the one which deserves serious
consideration. It was #3 in my list IIRC. The question to address is
whether it will do more, or less, harm than my workaround proposal. It is
not clear.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2066
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 14:46:47 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Per contra, so long as the WG insists on incompatibility and actively
>attempts to break mail standards because "we have to force mail to use utf-8"
>(paraphrased), there is absolutely zero chance of the Usefor draft ever
>becoming a standard.

You have a strange way of paraphrasing things. Even though we may (and do)
stand back in disbelief at the failure of Email (and now IDNA) to use
UTF-8, we have NO intent to "force mail to use utf-8", so I don't know
where you got that idea from. We are committed to ensuring that any Email
that comes out of Netnews is compliant, insofar as it is possible to do
so.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2067
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 14:51:23 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>If you could wave a magic wand and make sendmail completely 8-bit
>transparent, there would still be network and character set issues
>that would prevent use of certain characters (many of them within
>the US-ASCII set) without encoding.

>Unlike Usenet, mail is vital to many businesses and individuals.  One
>cannot simply "change the RFC, then complain" when things stop
>working en masse.

And there you have hit the nail on the head. Mail is vital to many
businesses and individuals.

Usenet is not.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2068
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 14:58:28 GMT
Lines: 19
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>It's worse than that; there are at least 3 different versions of UTF-8.
>They differ in the longer multi-byte sequences.

No, there is precisely one, as defined by the relevant Unicode documents.
See RFC 2044 and draft-yergeau-rfc2279bis-02.txt, with all of which Usefor
is fully compatible.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Marc Mutz <[email-masked]>
Organization: KDE
To: [email-masked]
Subject: [OT] mUTF-7 in IMAP
Date: Mon, 6 Jan 2003 19:06:28 +0100
Sender: [email-masked]

--Boundary-02=_kWcG+BHeYAm3VlA
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Monday 06 January 2003 13:32, Charles Lindsey wrote:
<snip>
> I think it is agreed that UTF-7 was an experiment that failed. Its
> usage will die out (and that includes in IMAP where it was not
> obligatory, anyway).
<snip>

What's the alternative for i18n-ized folder names in IMAP, then, if=20
mUTF-7 wasn't obligatory?

Marc

=2D-=20
Silent leges inter arma      -- Cicero

--Boundary-02=_kWcG+BHeYAm3VlA
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_kWcG+BHeYAm3VlA--

To: [email-masked]
Xref: clerew local.mime:2064
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 12:32:04 GMT
Lines: 61
Sender: [email-masked]

In <[email-masked]> Leo Bicknell <[email-masked]> writes:

>While I know little of Usenet's problems, or of NSI's motivations,
>I can speak as a sysadmin who has found simple harmony to the fact
>that all the internet protocols use US-ASCII, or ISO-8859-1.  I
>also see a future developing where if I want to code up a perl
>script to do something I'll have to understand UTF-8 to read a
>Netnews article, some weirdo NSI encoding to DNS resolve names from
>that article, and then know how to base64 encode things for e-mail.
>And that's just for three services I have been following.

The way the world seems to be moving is that UTF-16 will be used internally
by OSes, and UTF-8 will be used for external communications, and maybe for
file storage. OSes will be able to convert between the two (often with the
user being unaware, as when dragging or pasting).

UTF-16 as an internal code is not perfect (it will be a pain for the
Private Use planes, musical notes, etc. but is fine for almost everything
else). UTF-8 is fine for internet use 'on the wire' because it does not
misuse CR, LF and NUL, and it does not have any -endian problems.

The Bad News is that this combination will not satisfy the Chinese, though
it will satisfy almost everybody else. And some OSes will try to use UTF-8
internally (including some or all UNICES).

>This is already bad, but knowing that OS vendors are dealing with
>the UTF-7/UTF-8/UTF-16 problem, and other protocol groups are
>dealing with the same issues (indeed, IMAP uses UTF-7) seems to
>show a very fragmented future.

I think it is agreed that UTF-7 was an experiment that failed. Its usage
will die out (and that includes in IMAP where it was not obligatory,
anyway).

>What's my point?  Today I can "grep" a newsgroup article (ok,
>depends on server software and format), pass that to "dig" (ok,
>maybe with some sed and other work) and find out DNS information,
>and then use "sendmail" (ok, with a wrapper to generate a real
>e-mail) to mail someone.  I hope we all agree this is a good thing.
>I see a future developing where a custom filter will be needed
>between each of those steps to preserve "international" characters.
>I hope we can all see why that is bad.

Today you can grep, using UTF-8, in a file that is written in ISO8859-1,
or UTF-16 or whatever. What you can't do is grep in a file that contains
stuff encoded in RFC 2047. So if you want to grep in your news spool for
articles From: "Claus Färber" you might find it if the headers were in
UTF-8 (as Usefor prefers). But it would be hopeless even if you grepped
for the corresponding RFC 2047 string, because there are many ways that
"Claus Färber" can be encoded according to RFC 2047.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2063
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 12:03:20 GMT
Lines: 39
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Kai Henningsen <[email-masked]> writes:
>> [email-masked] (Russ Allbery) wrote:

>>> that all software will soon expect any untagged 8-bit data to be in
>>> Unicode,

>> That part is baseless exaggeration.

>Quite to the contrary, that's a summary of the basic foundation of the
>current draft, and is exactly what has been argued by multiple people on
>the USEFOR list on multiple occasions.

>To be more precise, the expectation is that any untagged 8-bit data will
>be in UTF-8.

And to be even more precise, the expectation is that software will check
to see whether the untagged data looks like UTF-8, and treat it as such if
it is.  If it isn't then the data will be non-compliant, but a reasonable
workaround would be to treat it as whatever default charset has been set
up by the user.

Another possibility I could be persuaded of is a Header-Charset: header
which indicated the actual charset used in the headers (defaulting to
UTF-8). There are problems with that (e.g. if the first header containing
Non-ASCII comes earlier than that header), but it might fly well enough to
be useful.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2062
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 6 Jan 2003 11:51:50 GMT
Lines: 43
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> 6. MIME is a set of protocols defined for the Email world, as is
>> explicitly stated in RFC 2045. Officially speaking, there is no such
>> thing as MIME within Netnews (which accounts for its extremely poor
>> uptake within Usenet).

>I doubt this.  I believe that the slow uptake of MIME within Usenet is due
>to other issues entirely, primarily that many news servers have had to
>block attachments of any kind because of space constraints and abuse and
>MIME makes it easier for people to attach documents, and even more because
>of some horrible misuse of multipart/alternative back in the early days.

It is certainly true that misuse of multipart/alternative has turned
people off the use of multipart/mixed, which could have some reasonable
applications on Usenet (particularly in the form of multipart/digest).

>I really don't think the lack of official standardization for Usenet had
>anything to do with it, and I believe that if you look, you'll find that
>many, if not most, Usenet messages have Content-Type and MIME-Version
>headers.  I don't think that's extremely poor uptake.

On the contrary, Content-Type, and particularly its charset parameter, is
about the only feature of MIME that HAS been taken up on Usenet. The only
other MIME feature with noticeable exposure has been RFC 2047 for poster's
names.

And Content-Type: image/* is hardly used (I agree it should be, in places
where binaries are welcome). All we have is bad jokes about what Microsoft
software does with lines in text/plain that start with "begin " :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Sun, 05 Jan 2003 17:38:34 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Pete Resnick wrote:
> On 12/24/02 at 12:43 PM -0500, Bruce Lilly wrote:
> 

>> That depends on the context. In the case of the List-Owner example and 
>> a number of fairly widely-used MUAs, the URL is properly decoded when 
>> using the List-Owner field content to generate a message to the list 
>> owner, and some even decode the 2047 encoding for display.
> 
> 
> Absolutely irrelevant. As far as 822/2822/2047 is concerned, List-Owner 
> does not contain a phrase.
[...]
> Now, 
> the contents of a List-Owner field can be passed to a 1738/2396 parser 
> and, upon discovering a mailto: URL (as against any other kind of URL), 
> one might parse the URL according to 2368 and discover within the 
> scheme-specific part of the URL something that is a mailbox list. The 
> *result* could then be passed to a 822/2822 parser and *then* to a 2047 
> parser for display purposes.

We're saying the same thing different ways.

>>>  And as far as I can tell, you can't have anything in a URL in a 
>>> header field that a 2047 parser would recognize as a phrase or a 
>>> comment.
>>
>>
>> URLs can contain parentheses
> 
> 
> But URLs can't contain "=" or "?", so a 2047 parser is not going to find 
> anything interesting in the parentheses.

Per 2396, an opaque_part can contain '='and/or '?', and opaque_part
can be part of an absoluteURI.  Also, '?' can delimit a query as part
of a relativeURI, and a query can contain '=' and/or '?'.

>> Examples have been given in earlier messages in this thread; a 
>> simplistic matching of header field text using regular expressions 
>> might incorrectly match a "comment" where there is none.
> 
> 
> But that wasn't the issue. The question that was being asked was whether 
> a simplistic regexp "parser" would accidentally find text which it 
> thought was 2047 syntax. Your claim was that such text could occur in a 
> field which contained a URL.

Yes, and I stand by that claim.

 > Since "=" and "?" can't appear in a URL
> according to 1738 and 2396, a 2047 parser should never be tripped up by 
> a URL in a header field.

See 2396 as referenced above.

> If you can come up with a serious example of where one might find 
> something that looked like 2047 text in a field where it shouldn't find 
> any, I'd be significantly more concerned.

I've already given several.  The bottom line is that simplistic matching
via regular expressions simply is inadequate by itself for parsing header
fields sufficiently to correctly identify RFC 2047 encoded-words. To
repeat an earlier example (slightly modified):

    Content-Location: http://users.erols.com/blilly/mailparse/(=?us-ascii?q?=3D?=)

That contains a valid RFC 2396 URI. A simplistic regular expression match
as proposed in Charles' Usefor draft would incorrectly identify a comment
and an encoded-word (neither exist in the example).  A *correct* grammar-based
parser would not identify a comment (presuming that the inherent ambiguity
in RFC 2557 is resolved by changing CFWS to FWS in the ABNF); it would
identify a URI.  The URI is parsed per 2396 as follows:

http              -> scheme                                           \
:                 -> ":"                                               |
//                -> "//"                                 \            |
users.erols.com   -> authority                             |           |
/                 -> "/"                      \            |           |
blilly            -> segment \                 |           |           |
/                 -> "/"      |                 > abs_path  > net_path  > absoluteURI
mailparse         -> segment   > path_segments |           |           |
/                 -> "/"      |                |           |           |
(=                -> segment /                /           /            |
?                 -> "?"                                               |
us-ascii?q?=3D?=) -> query                                            /

Date: Sun, 5 Jan 2003 10:24:53 -0600
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Cc: [email-masked], Charles Lindsey <[email-masked]>, Bruce Lilly <[email-masked]>
Sender: [email-masked]

On 12/24/02 at 12:43 PM -0500, Bruce Lilly wrote:

>Pete Resnick wrote:
>>On 12/24/02 at 9:06 AM -0500, Bruce Lilly wrote:
>>
>>>There most certainly is a phrase, part of the mailbox.
>>
>>No, sorry, it doesn't work that way. Just because 2368 (via 1738) 
>>specifies that some structured part of an encoded URL follows 
>>822/2822 syntax for "mailbox" does not mean that a mail header 
>>field that contains such a URL therefore contains a mailbox. The 
>>parser for 822/2822/2047 does not know about URL parsing and cannot 
>>get past the "%" encoded bits. There is no intervening 1738 parser.
>
>That depends on the context. In the case of the List-Owner example 
>and a number of fairly widely-used MUAs, the URL is properly decoded 
>when using the List-Owner field content to generate a message to the 
>list owner, and some even decode the 2047 encoding for display.

Absolutely irrelevant. As far as 822/2822/2047 is concerned, 
List-Owner does not contain a phrase. It might contain something with 
a lot of "%" characters, but the parser is not going to see a phrase 
in there. Now, the contents of a List-Owner field can be passed to a 
1738/2396 parser and, upon discovering a mailto: URL (as against any 
other kind of URL), one might parse the URL according to 2368 and 
discover within the scheme-specific part of the URL something that is 
a mailbox list. The *result* could then be passed to a 822/2822 
parser and *then* to a 2047 parser for display purposes. But that's 
*at least* a third pass over the data (and probably a bunch more 
depending on how you count). And what that last 2047 parse is seeing 
is most emphatically NOT the contents of a List-Owner field. The 
"context" (as you put it) does not pass from parser to parser.

>Surely you're not suggesting that correct use of the List-Owner etc. 
>header fields to generate messages to the list owner etc. would be 
>to use the raw URL %-encoded text in the headers?

No. I'm suggesting that you can't do it in one pass and expect 
legitimate results.

>It seems that you're implying that a To header generated from a 
>mailto URL doesn't contain a mailbox even if it matches the mailbox 
>syntax and although a To header generated from any other mechanism 
>may contain a mailbox -- that doesn't sound right either.

No, of course the To field generated from a mailto URL contains a 
mailbox. It's that a List-Owner field containing a mailto URL doesn't 
contain a mailbox.

>>  And as far as I can tell, you can't have anything in a URL in a 
>>header field that a 2047 parser would recognize as a phrase or a 
>>comment.
>
>URLs can contain parentheses

But URLs can't contain "=" or "?", so a 2047 parser is not going to 
find anything interesting in the parentheses.

>Examples have been given in earlier messages in this thread; a 
>simplistic matching of header field text using regular expressions 
>might incorrectly match a "comment" where there is none.

But that wasn't the issue. The question that was being asked was 
whether a simplistic regexp "parser" would accidentally find text 
which it thought was 2047 syntax. Your claim was that such text could 
occur in a field which contained a URL. Since "=" and "?" can't 
appear in a URL according to 1738 and 2396, a 2047 parser should 
never be tripped up by a URL in a header field.

If you can come up with a serious example of where one might find 
something that looked like 2047 text in a field where it shouldn't 
find any, I'd be significantly more concerned.
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

content-class: urn:content-classes:message
Subject: RE: RFC 2047 and gatewaying
Importance: normal
Date: Sat, 4 Jan 2003 14:53:24 -0800
From: "Dan Kohn" <[email-masked]>
To: "Kai Henningsen" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Kai Henningsen wrote:

> I don't want to go into that whole debate right now, but I *am* of the
> opinion that the IETF is going exactly the wrong way there, exchanging
> short-term pain for long-term pain. That solution is *never* the right
> one.

OK, I get that you want to move usenet to the right long-term solution,
and that you think that is raw UTF-8 in headers.  Personally, I believe
that raw UTF-8 headers will never make it through the IESG, and that it
is unlikely that large numbers of usenet clients or servers will
implement a proposal without an IETF imprimatur.

> Frankly, while it is certainly possible to argue about the relative
> merits of various solutions, I really doubt that any other solution
> has *significantly* lower impact than [UTF-8 in headers].

This is the question I want to pursue with you.  At the bottom, I'd like
to understand what impact this proposal has that raw UTF-8 doesn't.

> [2047] is a combination of wasting programmer's times (without any
hope of
> this ever getting better), of (as a consequence) introducing
> additional bugs, and of (partly as a consequence) irritating users.

> Oh, and nobody please try to tell me that 2047 "just works". I've
> seen it break far too often for that.

> 2047 *must go away*, not be perpetuated forever. It is an abomination.
> (And really, the same arguments hold for 2231.) And now we get
> punicode. 

> The Internet is increasingly feeling like typical MS code - patches
> upon patches upon patches.

The Internet is feeling that way because that's exactly the design
approach that has been practiced.  HTTP/1.1, for example, made all kinds
of absurd design decisions (down to misspelling referrer) to perpetuate
backward compatibility.

Now, I don't claim that 2047/2231 is anywhere near perfect, though I do
agree that it has advantages over raw UTF-8 outside of backward
compatibility (namely, language tagging).

But ignoring that, what programmer time will be saved by adding support
for raw UTF-8 headers?  Won't 2047/2231 headers still be present in
messages?  Won't any reasonable client need to be able to decode them?
Further, won't those clients also need to be able to decode punycode,
since domains will be encoded in it?

> Well, the problem is that it seems you cannot be backwards compatible
> to both current mail standards and current mail usage by current
> Usenet, as the two are *already* incompatible. Or at least not do
> that and actually have an even halfway sane method of i18n. Or at
> least I haven't seen any such proposal.

I'm suggesting that redefining the usefor article format to be RFC
2822+2047+2231 + a bunch of 1036 headers +
<http://www.normos.org/ietf/draft/draft-faerber-i18n-email-netnews-names
-00.txt> is backward compatible with both current mail and current news
formats, and provides full i18n of headers.

Let's see how it stacks up:

> 1. Must be able to support non-ASCII newsgroup names.

Supports full Unicode repertoire, encoded in punycode.  E.g.:

   se.test.zq--rksmrgs-5wao1o

> 2. Because of that is how the installed base of servers works,
>    newsgroup names (while in Usenet) *can* use non-7bit characters.

Encoded newsgroup names with more than 7 bits would be deprecated.  Note
that this wouldn't break any current server or client, but would serve
to show the direction for proper i18n going forward.

Conformant servers could (if they can identify the charset of the 8-bit
newsgroup name), convert it to Unicode and punycode encode it, although
this might cause more problems than it solves.

> 3. By the same argument, the identity relation on newsgroup names
>    *must* work without needing any form of normalization (because no
>    such form is deployed).

One huge advantage of using punycode is that usenet could adopt nameprep
normalization at the same time.  Note that usenet clients will *already*
need to adopt nameprep+punycode to support entry and display of IDNs.
The identical code could be reused for each component of a newsgroup
name.

Normalization would drastically improve the chances that a newsgroup
name a user types in would bring them to the newsgroup they want.

> 4. Names that fit in ASCII must still be in ASCII, for obvious
> reasons. 

Check

> 5. Because of moderated groups, news articles *will* be sent to
>    moderators via mail.
> 6. Because of the installed base, this *will* (currently) happen in
>    most cases without changing any header at all; we have a small
>    chance of using attachments instead.

No need to use attachments, since all usefor headers will also be legal
RFC 2822 headers.  (Note that an attachment of
application/news-transmission works OK for moderators, but has a
horrendously negative effect for cross-postings to mailing lists.)

> 7. Moderators, in the vast majority, refuse to do anything complicated
>    with these articles before injecting them to news servers. (See the
>    relevant flamewar in the USEFOR archives.) Most of them use tools
>    that are barely adequate to the job as-is, or at least that's the
>    impression I get listening to them.

They just add their Approved header like always.

> 8. Because of crossposts, non-ASCII names *will* make their way to
>    people who are not all that interested in groups with those names
>    themselves. This must not make anything break.

Many of those headers will be correctly displayed by any mailer that
supports 2047 and 2231.  The rest (Newsgroup, Control) will require
support for punycode newsgroup names, but all existing clients can at
least deal with the (ugly) ACE versions.

> While I dislike 2047, I do think a solution could demand it for any
> other fields; however, I do not see how it could possibly work for
> newsgroup names (Newsgroups: and Followup-To: header fields).

Agreed, you need punycode for that.

>> I have a simple question.  What can a UTF-8 subject header
>> communicate that an RFC 2047 one can't?  Other than inelegance,
>> what's the downside of 2047, when the upside is a huge increase in
>> backward compatibility? 

> The downside is exactly lack of backwards compatibility. See above for
> details.

Kai, other than bad aesthetics, how is this proposal not backward
compatible with both current mail and current news practices?

You listed your requirements for an i18n proposal.  Here are some of
mine, which this proposal supports, and which raw UTF-8 headers don't:

1) Provides i18n of all user-meaningful headers with support for the
full Unicode repertoire

2) Support for normalization so as to increase chances that what a user
enters is what they mean

3) Support for language tagging, as mandated by Section 4 of RFC 2277
<http://www.normos.org/ietf/rfc/rfc2277.txt>.

4) Does not break any current clients, servers, or gateways, either for
mail or news.

5) Incrementally deployable, with conformant clients getting full i18n
and non-conformant ones having an ugly though usable fallback.

6) No change to servers required.  (At least to support i18n.)

7) Least effort for coders.  This may seem controversial, but I argue
that i18n news clients already need to implement 2047/2231 due to mail
leakage, and nameprep+punycode to deal with IDNs.  So, supporting raw
UTF-8 in headers would be an *additional*, incremental piece of work.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

  Randomly generated quote:
As the British Constitution is the most subtle organism which has
proceeded from the womb and long gestation of progressive history, so
the American Constitution is, so far as I can see, the most wonderful
work ever struck off at a given time by the brain and purpose of man. 
- W.E. Gladstone    

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sat, 04 Jan 2003 11:31:46 -0800
Lines: 24
Sender: [email-masked]

D J Bernstein <[email-masked]> writes:
> Russ Allbery writes:

>> However, UTF-8 penalizes non-ASCII characters spacewise, and is
>> somewhat more complex to parse and reason about than a pure multibyte
>> character set.

> Have you ever written a program to handle Unicode characters correctly?
> Do you realize that UTF-16 is not a ``pure multibyte'' encoding outside
> the Basic Multilingual Plane?

I'm sorry, I should have been clearer.  The intended comparison was not to
UTF-16, which combines the worst of both worlds due to surrogate pairs,
but to UTF-32, which is (so far as I know) a pure multibyte encoding.

> Do you realize that Unicode has zero-width accents, so any ``byte count
> equals width'' rule can't possibly work?

Yes.  I know about combining marks and other similar characters, and I'm
not saying that even UTF-32 is simple, just that UTF-32 is somewhat
simpler to parse and reason about than UTF-8.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 4 Jan 2003 19:28:19 -0000
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Russ Allbery writes:
> However, UTF-8 penalizes non-ASCII characters spacewise, and is
> somewhat more complex to parse and reason about than a pure multibyte
> character set.

Have you ever written a program to handle Unicode characters correctly?
Do you realize that UTF-16 is not a ``pure multibyte'' encoding outside
the Basic Multilingual Plane? Do you realize that Unicode has zero-width
accents, so any ``byte count equals width'' rule can't possibly work?

The notion that UTF-16 is simpler than UTF-8 seems to come from broken
programs that (1) don't handle zero-width characters, (2) don't handle
double-width characters, and (3) are limited to the BMP.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

content-class: urn:content-classes:message
Subject: RE: RFC 2047 and gatewaying
Importance: normal
Date: Sat, 4 Jan 2003 11:21:41 -0800
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>, "Sam Roberts" <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

Here's a decent discussion of the challenges of GB 18030:

http://www-106.ibm.com/developerworks/unicode/library/u-china.html?dwzon
e=unicode

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>  

  Randomly generated quote:
As the British Constitution is the most subtle organism which has
proceeded from the womb and long gestation of progressive history, so
the American Constitution is, so far as I can see, the most wonderful
work ever struck off at a given time by the brain and purpose of man.  -
W.E. Gladstone

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Saturday, January 04, 2003 08:11
To: Sam Roberts
Cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying

> Quoteing [email-masked], on Fri, Jan 03, 2003 at 08:03:02PM
-0800:
> > > OTOH, one hears rumours that the Chinese are not altogether happy.
> >
> > Hardly a rumor. The Chinese dislike Unicode, or perhaps more
accurately the
> > UTF-8 and UTF-16 encodings of Unicode, sufficiently to have defined
a new

> Why?

I suppose the reason was to have something that is backwards compatible
with
existing, widely used multibyte encodings for simplified Chinese. I've
heard
GB18080 referred to as UTF-GBK on occasion.

But I'm really not the right person to ask about this. As Russ already
pointed
out, the entire charset space is hugely political. And I'm not privy to
the
political discussions that led to this being done. I just happened to
have had
to write code to support the outcome.

				Ned

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sat, 04 Jan 2003 11:12:58 -0800
Lines: 50
Sender: [email-masked]

Kai Henningsen <[email-masked]> writes:
> [email-masked] (Russ Allbery) wrote:

>> Quite to the contrary, that's a summary of the basic foundation of the
>> current draft,

> Don't be silly.

I'm not.

>> and is exactly what has been argued by multiple people on the USEFOR
>> list on multiple occasions.

> Not where I have seen it, and I've been in a number of those threads.

I suppose we'll have to agree to disagree, then, as I've also been in a
number of those threads.  It doesn't really matter whether or not my
characterization is accurate, but I'm not going to withdraw it, as it
remains my honest opinion.

> What *has* been argued is that *if* USEFOR says naked UTF-8 in headers
> is ok, then interpreting unencoded news headers as UTF-8 is going to be
> pretty much universal pretty fast - note the sharply limited domain here
> - and that, as the idea has now been around for a few years, some people
> already started writing their software along those lines.

> That is *very* far from "any untagged 8-bit data is UTF-8".

The same was also argued about article bodies, and if you put headers
together with bodies, you have, precisely, "any untagged 8-bit data."

There is, of course, very little point in not tagging the body data, and I
don't believe anyone was arguing in favor of leaving off the charset tag.
Maybe that's where you're disagreeing with me.

The part of this with which I disagree is the belief that if some standard
is issued stating that unencoded UTF-8 is allowed in headers, software
will immediately start changing to send and interpret that.  I don't
believe the level of influence that a new standard would have at this
point is that high, and use of other character sets (without labelling) is
extremely common right now.  It is possible to attempt to distinguish
between various untagged character sets, but I'm rather less optimistic
that anyone's going to rush out to write that code because we suggest it
in a standard.

I realize that the Turnpike folks have already done this, so perhaps that
indicates that I'm wrong.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sat, 04 Jan 2003 14:10:30 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Russ Allbery wrote:

>    It's best to think of all 8-bit character data as encoded.  UTF-8 is
>    just as much an encoding as RFC 2047.
[...]
>  * Assuming that the world agrees on Unicode (there don't seem to be any
>    other viable options for a universal character set used all over the
>    world), there are at least three major encodings (UTF-8, UTF-16, and
>    UTF-32) that can be used with it, as well as a bunch of other minor
>    ones.

It's worse than that; there are at least 3 different versions of UTF-8.
They differ in the longer multi-byte sequences.

Date: Sat, 04 Jan 2003 14:02:48 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Lawrence Greenfield <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Lawrence Greenfield wrote:

> Newsgroup names are a very hard problem. Note that the way Usenet works 
> right now requires these get passed gracefully through e-mail: in 
> moderated newsgroups posts are e-mailed to the moderator.

Not at all difficult when one realizes that the newsgroup "name" isn't
really a name at all; it's just a goup of tags that identify a place
in a hierarchy.  Many of the existing newsgroup component tags are
derived from abbreviations, contractions, or acronyms.

> I'm not sure I understand the NEED for the "canonical" name to be pure 
> UTF-8 (that is, encoded Unicode) instead of some other encoding.

Exactly -- there is no real "need", any more than there is a "need" for
ISO 639-2 language tags to be anything other than what they are; comprised
of a set of characters which are amenable to being used in (among other
places) email header fields.  Nobody -- at least nobody worth paying any
attention to -- is throwing tantrums because the language tag "ja" isn't
comprised of some utf-8 (or JIS) sequence.  For display / UI purposes
there is no reason why client software couldn't display the appropriate
Kanji in a menu, etc. where the language tag "ja" is recognized, and it
is precisely to the user agent (clent) software where such matters should
be constrained.  Ditto for charset tags, header field names, various
keywords that appear in various headers, etc.

Indeed, the existing unencoded, non-Unicode tags are quite convenient
for news administrators, as typical filesystem tools will work fine with
the tags on a wide variety of systems.  That is not necessarily the case
for utf-8 encoded tags, and the Usefor WG would be well advised to consider
the IMAP experience (elided here for brevity).

Date: Sat, 04 Jan 2003 13:44:59 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> 5. Insofar as these choices conflict with practice in the Email world,
> there was a consciously taken decision to compress any resultant messiness
> into the gateways. The number of these is small compared to the vast
> number of Unsnet servers worldwide and the even vaster number of clients.

The fact is that the decision to use untagged, unencoded utf-8 in
newsgroup component tags has implications for software complexity in
servers and clients and intermediate software (e.g. IMAP implementations)
as well as gateways.

> It is already the case that 2.5% [1] of Usenet articles use raw Non-ASCII
> characters in their headers. I have not heard of any damage or chaos
> arising from gateways that cannot cope.
[...]
> And that problem is not limited to Netnews. 30% of the Email that I
> receive has raw Non-ASCII in its headers.

There is a very important difference between
A. Some user entered some illegal information and some software implementation
    incorrectly transmitted it verbatim in violation of standards
and
B. A draft ecourages violation of standards.

In the former case, the user and/or software author are at fault.  In the
latter case, the draft has no hope of being approved by the IETF.

Date: Sat, 04 Jan 2003 13:37:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Kai Henningsen <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Kai Henningsen wrote:

> (And, of course, 1036 did _not_ use the Internet text message format, it  
> defined its own that looked similar but had lots of important  
> differences.)

You may wish to review RFC 1036 section 2, specifically:

     Therefore, the rule is adopted that
     all USENET news messages must be formatted as valid Internet mail
     messages, according to the Internet standard RFC-822.  The USENET
     News standard is more restrictive than the Internet standard,
     placing additional requirements on each message and forbidding use
     of certain Internet features.  However, it should always be possible
     to use a tool expecting an Internet message to process a news
     message.  In any situation where this standard conflicts with the
     Internet standard, RFC-822 should be considered correct and this
     standard in error.

Date: Sat, 04 Jan 2003 13:18:52 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> It should be noted that, despite the use of the numbers "822", a
> "message/rfc822" entity isn't restricted to material in strict conformance
> to RFC822, nor are the semantics of "message/rfc822" objects restricted to
> the semantics defined in RFC822. More specifically, a "message/rfc822"
> message could well be a News article or a MIME message. 
> 
> Clearly, strict RFC [2]822 compliance is not a requirement, and use for
> News articles is explicitly encouraged (which is why Usefor makes it the
> official way to encapsulate News).

The reference in 2046 is to "News article" as defined at the time 2046
was written, viz. RFC 1036, which uses the 822 message format.  message/rfc822
always means headers comply with the 822 header syntax, no matter how
the Usefor WG tries to finagle things.

> The only requirement is that the syntax should agree with RFC [2]822, and
> I have no problem with that since the proposed Usefor wording for gateways
> that we are discussing explicitly requires that it should be so on the
> Email side of the gateway, and tells you what transformations are
> necessary to ensure it. So I still do not see what Larry's problem is.

The draft does not provide a mechanism to ensure compliance.

Date: Sat, 04 Jan 2003 13:09:28 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

>>If you want to update RFC 2822 to allow unencoded UTF-8 in headers, do
>>so. Don't try to slip it in the backdoor.
> 
> 
> But I don't. The text I posted made it quite clear that, on the Email side
> of the gateway, all UTF-8 would have been removed, by one means or
> another,
> 
> from the headers of the article
> from the headers of any contained message/rfc822
> from the headers of any contained MIME body part
> 
> and that if a particular implementor chose not to do that, and thus
> produced a non-compliant Email object, then that would be his risk and
> responsibility.

It is insufficient to say "comply with 2822 / 2047 / 2231); there is
a design defect which prevents that, unless a specific mechanism is
provided to ensure compliance.  As it stands,
1. it is impossible for a gateway to determine the syntax of an
    experimental header which the gateway author is unaware of
2. Without header field syntax information, it is impossible to
    encode non-ASCII "header" content.

The design defect can be eliminated by:
a. forbidding non-ASCII content in all but a well-defined set which
    all gateways are required to recognize, *and* providing a practical
    transition plan so that existing gateways can be upgraded while
    ensuring that the mail protocols are not violated
or
b. simply using the RFC 822 / 2822 / 2047 / 2231 Internet text message
    format, which avoids the encoding problem in gateways and adds no
    new restrictions on gateway transformations (thereby obviating that
    aspect of a transition plan).
or
c. require that unrecognized "headers" with non-ASCII content be elided
    or moved to an encoded attachment.

Date: Sat, 04 Jan 2003 12:54:52 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>Surely you're not suggesting that correct use of the List-Owner etc. header
>>fields to generate messages to the list owner etc. would be to use the raw
>>URL %-encoded text in the headers?
> 
> 
> That is exactly what I would suggest for general use.

So you're saying that given

    List-Owner: <mailto:[email-masked]%3E>

you would construct

   To: [email-masked]%3E

Good luck.

> Now all this could change if IRIs (draft-duerst-iri-02.txt) should get to
> the standard stage.

No, it would not change one whit. List-Owner et al. specifically refer
to URIs, and that would remain the case unless and until the defining
RFC is superseded.

>>URLs can contain parentheses (which is why there's a problem with the
>>RFC 2557 ABNF and CFWS).
> 
> 
> Which is a good reason for putting URLs inside <...>, and ensuring that
> nobody goes looking for comments inside there.

1. You are of course free to suggest bracketing URIs to the authors of RFCs
    2557 and 2017, but the current situation is that unadorned URIs may
    appear in a number of places, and those URIs may conntain parentheses.
    and that the only way to determine whether or not something bracketed in
    matching parenthese is or is not a comment is by consideration of the
    header field ABNF in conjunction with any relevant field component
    structure specification.

2. Comments may well exist within angle brackets, e.g.:

   Return-Path: <(=?us-ascii?q?foo?=)>

    See RFC 2822 section 3.6.7.  [BTW, do you make up these "rules" about
    where to look for comments as you go along, or have you bothered to
    think about the issue?]

Date: Sat, 04 Jan 2003 12:27:14 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Leo Bicknell <[email-masked]>
CC: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Leo Bicknell wrote:
> In a message written on Fri, Jan 03, 2003 at 09:01:42PM -0800, Russ Allbery wrote:

>> * sendmail still is not fully 8-bit clean in the latest analysis that
>>   I've seen, which means that 7-bit issues continue to plague the mail
>>   system.
> 
> 
> While I run sendmail, it is not the only mail system.  qmail,
> vmmail, etc all are widely used.  Change the RFC, then complain
> about the program.

While sendmail-bashing may be a popular pastime in some circles, sendmail
isn't the entirety of the issue.  There are a number of issues which have
affected the mail protocols:
1. the need for certain uniform conventions, e.g. end-of-line.
2. software constraints, e.g. use of ASCII NUL to terminate character
    strings in some library functions in some programming languages.
3. character set limitations, e.g. EBCDIC.
4. network limitations, e.g. Bitnet.

If you could wave a magic wand and make sendmail completely 8-bit
transparent, there would still be network and character set issues
that would prevent use of certain characters (many of them within
the US-ASCII set) without encoding.

Unlike Usenet, mail is vital to many businesses and individuals.  One
cannot simply "change the RFC, then complain" when things stop
working en masse.  *Before* the protocols can be changed, it is
necessary to ensure that software *and* networks, *as affected
by their operating environments* are compatible.  Before considering
the considerable amount of research required even to determine
what the various network and software constraints are, it is prudent
to consider the cost/benefit ratio.  Mail works fine now -- what
benefit is to be achieved by changing the protocol?  In the case
of the characters set limitations, any benefits are likely to be
minimal and are constrained by a number of factors:
1. There is no universal character set (don't point to Unicode; there
    are a number of incompatible Unicode versions, and as noted in
    this thread, Unicode is unacceptable to some)
2. It is not possible to identitfy a character set reliably without
    some external indication (a.k.a. tagging).
3. Low-level protocols are based on octets, and any character set
    with any hope of being a universal character set will have
    "characters" wider than one octet (once upon a time, Unicode
    was guaranteed to be 16-bits, but that was when Unicode was for
    characters -- before it was extended to include things which it
    specifically excluded, such as musical notes).  Therefore, *some*
    encoding will be required to split the N bits comprising a
    character into some sequence of octets. E.g. there are several
    varieties of utf-8.  Of course, there's also utf-7, which avoids
    the network and software issues in the first place...
4. There is often a need for language tagging, as provided for by
    RFCs 2047 / 2231. E.g. =?iso-8859-1*en?q?boot?= is very different
    from =?iso-8859-1*de?q?boot?= even though the character set and
    encoding is invariant.  A very recent change to Unicode provides
    for language tagging (via an *encoded* set of tags), but that
    tagging feature is specifically disallowed with "higher-level protocols",
    specifically mentioning MIME.
The key points are:
a. encoding will still be required
b. language tagging is still desirable in many situations
c. character set tagging will still be required.
Proponents of utf-8 and opponents of RFC 2047 / 2231 (often the same
individuals) focus on the encoding issue and lose sight of the charset
tagging and language tagging issues, which are equally important.

The bottom line is that even if it were possible to eliminate the
encdoding isse, 2047 / 2231 would still be needed for charset and
language tagging.  Since it isn't possible to avoid the encoding
issue (at least until such time as the "character" width stabilizes
and is no wider than the low-level protocol transmission width), we
will continue to need mechanisms like 2047 / 2231 [yes, Charles, even
20 or 30 years hence (though hopefully the RFCs will have been updated
to consolidate requirements, conform to current nomenclature, and
correct errors)].

Date: Sat, 04 Jan 2003 11:27:29 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Russ Allbery wrote:
> Bruce Lilly <[email-masked]> writes:
> 
> 
>>I expect IETF standards which update or replace earlier standards to
>>address compatibility issues.  As discussed on the Usefor list, the
>>current draft expresses requirements which are diametrically opposed to
>>what the current standard (RFC 1036) requires (e.g. 1036 makes
>>processing of articles with Subject: cmsg ... as a control message,
>>whereas the current draft forbids such processing).  Failure to provide
>>for a graceful transition is a recipe for disaster.
> 
> 
> This specific case is a bad example, as major news software dropped
> support for the obsolete control message format some time ago.

It is just *an* example; there are other instances.

> In general, while I agree with you that graceful transition is important,
> I disagree that a graceful transition from RFC 1036 *in particular* is
> important.  One of the very difficult parts about working on Usenet
> standardization is that RFC 1036 is functionally obsolete.  It does not
> describe how Usenet actually works today.

RFC 1036 *is* the current official standard, whether or not current
practice follows it.  The Usefor WG ought to have issued an update to
1036 long ago, in accordance with the task that the WG was formed to
address.  Even at this late date, it is still possible for the WG to
update those non-controversial issues where current practice has
deviated from 1036.  Such a revision could be one step in the "graceful
transition" to some goal, and could be approved through the standards
process fairly quickly.

Per contra, so long as the WG insists on incompatibility and actively
attempts to break mail standards because "we have to force mail to use utf-8"
(paraphrased), there is absolutely zero chance of the Usefor draft ever
becoming a standard.

[...]

> A graceful transition from existing software and existing practice is
> important, and a graceful transition from RFC 1036 is a reasonable
> fallback in those cases where we cannot characterize existing practice,
> but RFC 1036 really shouldn't have much in the way of special standing in
> this process given its age and the accumulated differences between it and
> standard practice.

Much of what is in the current Usefor draft is clearly untested (or
inadequately tested) experimental work, and that certainly includes
the utf-8-everywhere and parameters-everywhere parts.  Those [experimental,
inadequately-tested] parts shouldn't be part of the draft; without them,
a draft could have been finished, approved, and codified as a 1036
successor ages ago.

Then, after appropriate consideration of the totality of the issues
involved, a carefully-designed and adeqautely-tested scheme for
handling i18n issues which cannot be handled via the existing 2047
and 2231 standards could become a successor to (or extension of) the
1036 successor.

Date: Sat, 04 Jan 2003 08:11:07 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Sam Roberts <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Quoteing [email-masked], on Fri, Jan 03, 2003 at 08:03:02PM -0800:
> > > OTOH, one hears rumours that the Chinese are not altogether happy.
> >
> > Hardly a rumor. The Chinese dislike Unicode, or perhaps more accurately the
> > UTF-8 and UTF-16 encodings of Unicode, sufficiently to have defined a new

> Why?

I suppose the reason was to have something that is backwards compatible with
existing, widely used multibyte encodings for simplified Chinese. I've heard
GB18080 referred to as UTF-GBK on occasion.

But I'm really not the right person to ask about this. As Russ already pointed
out, the entire charset space is hugely political. And I'm not privy to the
political discussions that led to this being done. I just happened to have had
to write code to support the outcome.

				Ned

Date: 04 Jan 2003 13:55:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Russ Allbery)  wrote on 03.01.03 in <[email-masked]>:

> Kai Henningsen <[email-masked]> writes:
> > [email-masked] (Russ Allbery) wrote:
>
> >> that all software will soon expect any untagged 8-bit data to be in
> >> Unicode,
>
> > That part is baseless exaggeration.
>
> Quite to the contrary, that's a summary of the basic foundation of the
> current draft,

Don't be silly.

> and is exactly what has been argued by multiple people on
> the USEFOR list on multiple occasions.

Not where I have seen it, and I've been in a number of those threads.

What *has* been argued is that *if* USEFOR says naked UTF-8 in headers is  
ok, then interpreting unencoded news headers as UTF-8 is going to be  
pretty much universal pretty fast - note the sharply limited domain here -  
and that, as the idea has now been around for a few years, some people  
already started writing their software along those lines.

That is *very* far from "any untagged 8-bit data is UTF-8".

Though, of course, I'm sure there already is software which tries this.  
Oh, yes: XML has a rule that sounds very similar to this (it's either UTF- 
8 or UTF-16; mail and news obviously can't use UTF-16 that way - 8bit  
clean is not binary clean).

MfG Kai

Date: 04 Jan 2003 13:43:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 03.01.03 in <[email-masked]>:

> > 2. But in fact, if you look at the headers we have defined so far, you
> > will see that the only places 8bit freedom is allowed is in comments,
> > phrases, parameters and newsgroup-names.
>
> you obviously don't get it.  the definition of message/rfc822 is for
> the content-type, not for the use of that content-type in email.
> content-type definitions are independent of where they are used.
> (the only exception is text/plain where the default charset is
> "iso-8859-1" in http and "us-ascii" in email - but this was only a
> hack for backward compatibility with pre-standards http.)
>
> it's absolutely unacceptable for usefor to redefine the meaning
> of message/rfc822 - even if they only pretend to do so for usenet.

Frankly, from past discussions on both USEFOR and this list, it appears  
that using message/rfc822 as-is has unacceptabable constraints(1),  
allowing a workaround is unacceptable(2), using another media type does  
not work(3), and sending news via mail is not going to stop(4).

Now what?

>From what I can see, (2) is *by far* the easiest "problem" to ignore.

Of course, if this debate continues to stall at every option, I can  
guarantee you that what you get is what we have today, which is  
essentially (4) stomping on *all* these solutions - i.e. sending a naked  
news article as-is with only an added To: header. USEFOR nearly had a  
revolt of the Usenet moderators just for daring to suggest that that had  
problems. (See the archives.)

And frankly, the holy restriction you're arguing for with (2) is already a  
very silly idea in the first place.

MfG Kai

Date: 04 Jan 2003 14:20:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Leo Bicknell)  wrote on 04.01.03 in <[email-masked]>:

> In a message written on Fri, Jan 03, 2003 at 09:01:42PM -0800, Russ Allbery
> >    wrote: It's best to think of all 8-bit character data as encoded.
> >    UTF-8 is just as much an encoding as RFC 2047.  UTF-16 is an encoding.
> >    UTF-32 is an encoding.  You're going to have to deal with an encoding,
> >    no
>
> Per unicode.org (gotta love google) there are 13 formal standards
> for encoding unicode, and at least a half dozen more.  That's an

Reality check: most of them are irrelevant to the real world.

> order of magnitude too many.  While I would like one, the world
> has existed fine with two (US-ASCII & ISO-8859-1), and three might

Reality check: the world *has not* existed fine with US-ASCII & ISO 8859- 
1. Even when US-ASCII was relatively new as the US variant of ISO 646,  
there were a lot of other national variants around (I'd think 13 wouldn't  
be far off), and that was only North America and Western Europe. If you  
start looking at the ISO 2022 area and beginning to really offer solutions  
to *the world*, I think you'll get closer to 50 or so charsets *in*  
*actual* *use*. That *is* what the world exists with.

> work.  13, no way no how give up now if you think that might work.

13 - even if anyone would suggest supporting all of those, which nobody  
does AFAICT - is still *vastly* better than what we have today. Have a  
look at your iconv library one of these days.

> The world can deal with a fairly small set of encodings.  The world
> cannot deal with a large (> 5) number of the.

Your definition of "world" is way off.

> I hate to be an arrogant american in this, but I will take that
> role.  If, after we adopt something better (unicode?  something
> else?  I dunoo) you can't speak a character set on the Internet
> you don't deserve to be online.

So go offline already. *You* are the one claiming to be unable to cope  
with a solution that is significantly easier that what we have today.

> world does not speak english.  So, I will adopt software that allows
> the majority of foreign (to me) languages to work.  If some obscure
> language with 1,000 speakers who's country GDP is less than my
> sallary is excluded excuse me if I don't care.

Well, there is exactly one technology available to meet that spec, and  
it's Unicode. And it *does* tend to solve the 1000 speakers problem - if  
you can show such a community with so-far unencoded characters which  
actually wants to use them on computers, your chances of getting those  
characters assigned are pretty good.

(Some people claim that 2022 is another such solution. Except it isn't -  
all it does is label the differences.)

> If there is anything technology has proved it is that standards
> are good.  I'm not going to hold my breath for one worldwide
> "language", but the ones on the fringes can die and go away.

The ones on the fringe are exactly those supporting the big standards. The  
ones not exactly on the fringe (like the US) are typically the ones  
supporting standards like US-ASCII that work only locally.

> While I run sendmail, it is not the only mail system.  qmail,
> vmmail, etc all are widely used.  Change the RFC, then complain
> about the program.

And the IETF seems to reject every effort to make mail use more than  
US_ASCII without imposing additional encodings on top of whatever you  
already have.

That, IMO, is the real cause for the problem here.

> At least, in general, mail has MIME, which has
> a number of ways to encode unicode in a standard.  That said, "raw"
> mail is still of value, and should be made to "just work".

You think so, I think so.

> > Universal flag days, as you point out, aren't going to work.  I'm
> > therefore becoming increasingly resigned to having to deal with encodings
> > pretty much forever, or at least for the forseeable future.  So in answer
>
> If we have to use (many) encodings then all of this is going to fail.
> Period.  Full stop.  Whatever the "." character means in your language.

There seems to be no other option available.

> other is saying?  I am 100% sure that worldwide communication will
> lead to one, and only one worldwide language.  In my lifetime?
> Doubtful.  In my childrens?  Likely.

Now you're out in fantasy land. The way to bet is that "one language"  
won't happen even in several centuries. Humans just aren't wired that way.

MfG Kai

Date: 04 Jan 2003 15:26:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Dan Kohn)  wrote on 03.01.03 in <[email-masked]>:

> Charles, as someone with absolutely no political or religious stakes in
> this discussion, I'd like to point out to you that I've found Ned Freed
> to be one of the most level-headed, open-minded IETF oldtimers.

Well, I cannot make that description match with his current mails. They  
seem needlessly confrontational and ...

> I've
> seen him go out of his way to offer constructive criticism, and be far
> more open to new ideas than many.

... perfectly free of anything that could be construed as constructive.

> This is perhaps Ned's most important statement.  At countless times in
> the last 20 years, the IETF and it's predecessors have chosen backward
> compatibility over the "efficient" solution.  The (IESG-approved) IDNA
> drafts epitomize this approach, by promulgating an "ugly-looking"
> transfer encoding (punycode) rather than the "elegant" solution of
> UTF-8.  The economic calculation was that clients that care about i18n
> can implement IDNA, but that the IETF would not break the "social
> contract" made with DNS software that was (and now still will be)
> compatible with pre-IDNA standards.

I don't want to go into that whole debate right now, but I *am* of the  
opinion that the IETF is going exactly the wrong way there, exchanging  
short-term pain for long-term pain. That solution is *never* the right  
one.

> I believe you're radically underestimating the IESG interest in that
> backward compatibility, specifically in how it relates to mail gateways.

I increasingly have the impression that the people here but not on the  
USEFOR list do not actually *understand* the backward compatibility issues  
involved. For example, they seem to think that RFC 1036 actually describes  
how Usenet looks today. This is not true.

> Further, I don't think anyone on the IESG will consider "UTF-8 in
> headers") as an essential feature of "best proposed practice".  The IESG
> will demand i18n of headers, but it will equally push for the lowest
> impact way of achieving this.

Frankly, while it is certainly possible to argue about the relative merits  
of various solutions, I really doubt that any other solution has  
*significantly* lower impact than this one.

> Kai Henningsen said:
>
> > Very shortly put, there is a long tradition on Usenet of passionately
> > hating 2047 encoding in at least some quarters. Note the word
> > "passionately"; it is not exaggerated. Long flamewars have been fought
> > about this before anyone even thought about USEFOR.
>
> > My only personal stance on this is that I consider it a serious,
> > inexcusable bug that 2821/2822 do not allow naked UTF-8 in headers.
> > Frankly, "we once made the mistake to spec 7 bits so this must remain
> > 7 bit forever" is absolutely and inexcusably insane. But that is a
> > mail problem, not a news problem.
>
> Not only is it a news problem, but it is the news problem that will
> likely prevent the usefor draft from being approved.  What I don't get
> is that news clients will need to understand 2047 and 2231 syntax
> anyway, since such things are bound to leak in (from email if nowhere
> else).
>
> Kai's argument is quite similar to the arguments made in IDN for "just
> use UTF-8".  A strong consensus was reached that the elegance of UTF-8
> was a far lower priority than backward compatibility.  For RFC 2822, the
> value of backward compatibility over elegance was even more obvious.  If

Frankly, it was totally nonobvious to me at the time; it was so before,  
and it still is. I strongly believe exchanging short-term pain for long- 
term pain is not a reasonable decision.

> you care about wasting bits in RFC 2047 encoding of UTF-8, start

If you think this is about wasting bits, you haven't even started to  
understand the problems.

It is a combination of wasting programmer's times (without any hope of  
this ever getting better), of (as a consequence) introducing additional  
bugs, and of (partly as a consequence) irritating users.

Oh, and nobody please try to tell me that 2047 "just works". I've seen it  
break far too often for that.

2047 *must go away*, not be perpetuated forever. It is an abomination.  
(And really, the same arguments hold for 2231.) And now we get punicode.

The Internet is increasingly feeling like typical MS code - patches upon  
patches upon patches.

> SONET overhead.  But if you want to deploy a standard in the IETF, focus
> on backward compatibility.

Well, the problem is that it seems you cannot be backwards compatible to  
both current mail standards and current mail usage by current Usenet, as  
the two are *already* incompatible. Or at least not do that and actually  
have an even halfway sane method of i18n. Or at least I haven't seen any  
such proposal.

Obviously, abandoning the Usenet side of this means making a standard  
that's irrelevant to the real world, as Usenet will just ignore it the  
same way it actually does much of that today. Backwards compatibility to  
existing Usenet practice is a pretty absolute MUST. And specifically,  
compatibility with the installed server base is much more critical than  
for mail, because of the flood fill nature of Usenet. That limits  
implementation choices a lot.

Now, I personally am not a member of the crowd that starts flaming when  
they see 2047 in headers; I just rant a lot when I have to implement that  
mess (as I do right now).

So from my point of view, the really important i18n considerations are the  
following - maybe someone can come up with that missing ideal solution:

1. Must be able to support non-ASCII newsgroup names.
2. Because of that is how the installed base of servers works, newsgroup
   names (while in Usenet) *can* use non-7bit characters.
3. By the same argument, the identity relation on newsgroup names *must*
   work without needing any form of normalization (because no such form is
   deployed).
4. Names that fit in ASCII must still be in ASCII, for obvious reasons.
5. Because of moderated groups, news articles *will* be sent to moderators
   via mail.
6. Because of the installed base, this *will* (currently) happen in most
   cases without changing any header at all; we have a small chance of
   using attachments instead.
7. Moderators, in the vast majority, refuse to do anything complicated
   with these articles before injecting them to news servers. (See the
   relevant flamewar in the USEFOR archives.) Most of them use tools that
   are barely adequate to the job as-is, or at least that's the impression
   I get listening to them.
8. Because of crossposts, non-ASCII names *will* make their way to people
   who are not all that interested in groups with those names themselves.
   This must not make anything break.

While I dislike 2047, I do think a solution could demand it for any other  
fields; however, I do not see how it could possibly work for newsgroup  
names (Newsgroups: and Followup-To: header fields).

> I have a simple question.  What can a UTF-8 subject header communicate
> that an RFC 2047 one can't?  Other than inelegance, what's the downside
> of 2047, when the upside is a huge increase in backward compatibility?

The downside is exactly lack of backwards compatibility. See above for  
details.

MfG Kai

Date: 04 Jan 2003 13:33:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked]  wrote on 03.01.03 in <[email-masked]>:

> > 5. Insofar as these choices conflict with practice in the Email world,
> > there was a consciously taken decision to compress any resultant messiness
> > into the gateways. The number of these is small compared to the vast
> > number of Unsnet servers worldwide and the even vaster number of clients.
>
> Yawn. Heard all these arugments before back with MIME. This doesn't
> address the backwards compatibility issue, and like it or not this is
> an issue you are going to have to address.

Which backwards compatibility issue? The ones I'm aware of are all on the  
news side, and *are* addressed (though one can certainly debate if each  
solution is the right one, as is always possible).

> And that's a perfectly legitimate thing to do. But you cannot break the
> definition of existing media type as part of this process. Pick another
> media type, for heaven's sake!

USEFOR *did* that, originally. This(!) mailing list pointed out that that  
wouldn't work.

I guess the only way to find out is to actually try for last call and see  
what happens; listening to individual people is going to stall the  
standard forever.

> I would refrain from arguing on the basis of what your charter currently
> says if I were you. A WG whose current charter includes no goals and
> milestones position in the IETF can only be described as "precarious". At a
> minimum a rechartering exercise could be in order, and given recent
> experience with charters and the IESG the resulting charter that would
> emerge would be very likely to constrain you in ways you really would not
> like.

Oh?

In which case I suspect that the Usenet people would say "Ok, so the RFC  
standards process is a dead end", and will continue outside of that  
process.

I don't see how that could be in the interests of IETF or IESG, but  
whatever ...

> The issue is instead that your approach in effect declares a large body of
> software as being no longer compliant with long established standards. This
> is something we try very hard not to do.

Actually, it does no such thing, as there *are currently no standard RFCs*  
for news (article format, that is, there are for NNTP). 1036 was never a  
standard, and doesn't adequately describe current Usenet anyway.

There *is no* "large body of software" that is compliant with any "long  
established standard" that would then no longer be compliant with those  
standards. It may be noncompliant with the news standard, but, again,  
there never was a news standard it could have been compliant to before,  
and dozens of programmers have noticed that complying with 1036 is not  
actually all that useful in the real world.

This argument is utterly bogus.

> The argument of "but everyone breaks the rules so we can too" has been made
> countless times. I cannot recall a single case, however, where such an
> argument made it past the IESG.

Of course, that *is not* the argument being made, as I suspect you are  
very well aware.

> > I am not expecting getting Usefor past the IETF to be an easy task. But
> > then getting Usefore to make up its collective mind was not an easy task
> > either :-( .
>
> Well, you're clearly set on this course and it seems unlikely that anything
> I say is going to change it. So this wil be my final response on this
> general set of issues.

Much more important, it seems unlikely that any other course is even  
*possible*. Whatever you think of any actual WG consensus, anything *but*  
WG consensus obviously doesn't even have a snowball's chance in hell.

MfG Kai

Date: Sat, 4 Jan 2003 02:06:38 -0500
From: Leo Bicknell <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: United Federation of Planets
Reply-By: Sat Jan 11 01:41:10 EST 2003
Sender: [email-masked]

In a message written on Fri, Jan 03, 2003 at 09:01:42PM -0800, Russ Allbery wrote:
>    It's best to think of all 8-bit character data as encoded.  UTF-8 is
>    just as much an encoding as RFC 2047.  UTF-16 is an encoding.  UTF-32
>    is an encoding.  You're going to have to deal with an encoding, no

Per unicode.org (gotta love google) there are 13 formal standards
for encoding unicode, and at least a half dozen more.  That's an
order of magnitude too many.  While I would like one, the world
has existed fine with two (US-ASCII & ISO-8859-1), and three might
work.  13, no way no how give up now if you think that might work.

The world can deal with a fairly small set of encodings.  The world
cannot deal with a large (> 5) number of the.

>  * Your goal above therefore reduces to "can we all agree on an encoding
>    for characters?"

If we agree Unicode has all available characters (which I'm not
sure we do) then it seems quite strange to me that we can't agree
how to encode them.  Again, possibly to a couple of standards (eg,
big endian little endian, not 13).

>  * The development of character sets is highly political.  It's not clear
>    that development of a single character set is going to succeed,
>    unfortunately not for technical reasons but for political reasons.  But

I hate to be an arrogant american in this, but I will take that
role.  If, after we adopt something better (unicode?  something
else?  I dunoo) you can't speak a character set on the Internet
you don't deserve to be online.  I want, desperate, to do this
today with US-ASCII, but no matter how much I want it much of the
world does not speak english.  So, I will adopt software that allows
the majority of foreign (to me) languages to work.  If some obscure
language with 1,000 speakers who's country GDP is less than my
sallary is excluded excuse me if I don't care.

If there is anything technology has proved it is that standards
are good.  I'm not going to hold my breath for one worldwide
"language", but the ones on the fringes can die and go away.

>  * Lots of people don't use Unicode at all and continue to not use
>    Unicode.  Are those people all going to switch to Unicode?  I really

Lot's of people don't care.  If their xterm/dos prompt/securecrt
prompt displays {insert favorite unusual language" they will be
happy.  If it displays "garbage" they will be happy.  If it displays
"you cannot read this" they will be happy.

The fact that most people know a single, or perhaps two languages
are irrevelant.  Display it as it was intended.  If I buy a book
in Chinese and can't read it that doesn't change the fact that it
is in Chinese.  If the software works, no one will care.

>  * UTF-16 breaks Internet protocols and all the programs that manipulate
>    them horribly because UTF-16 includes characters that C libraries think
>    are embedded NULs and therefore end-of-string markers.  So I have a
>    hard time imagining the Internet standardizing on UTF-16.  You can't
>    even store files in a Unix file system with names encoded in UTF-16
>    without major surgery to the entire C API.

You have just suggested a good reason UTF-16 should be removed as a
standard.  1 down, 12 to go.  Next?

>  * Microsoft Windows uses UTF-16, as do many other early adopters of
>    Unicode.

5 billion flies every day eat feces, why don't you?

Don't let an early adopter get in the way of the right solution.
Remember, at one time the entire ARAPnet did not run TCP/IP, that
didn't stop them from converting.

>  * UTF-8 seems to be the obvious choice for Internet protocols, since

Yes, it does, at least to me the uninformed in so many issues.

>  * sendmail still is not fully 8-bit clean in the latest analysis that
>    I've seen, which means that 7-bit issues continue to plague the mail
>    system.

While I run sendmail, it is not the only mail system.  qmail,
vmmail, etc all are widely used.  Change the RFC, then complain
about the program.  At least, in general, mail has MIME, which has
a number of ways to encode unicode in a standard.  That said, "raw"
mail is still of value, and should be made to "just work".

> Universal flag days, as you point out, aren't going to work.  I'm
> therefore becoming increasingly resigned to having to deal with encodings
> pretty much forever, or at least for the forseeable future.  So in answer

If we have to use (many) encodings then all of this is going to fail.
Period.  Full stop.  Whatever the "." character means in your language.

> If it's any comfort, the only reason why you and I have been spared this
> so far is because we both use English and are therefore happy with
> US-ASCII.  The rest of the world has been dealing with this for years.

I would consider you right, but for the wrong reason.  We all
communicate for one, and only one reason.  To communicate ideas.
In the end ideas of character sets and encodings and all that are
rather esoteric.  The real question is do we understand what each
other is saying?  I am 100% sure that worldwide communication will
lead to one, and only one worldwide language.  In my lifetime?
Doubtful.  In my childrens?  Likely.  Will there be a lot of pain
in the middle?  Yes.  Pockets of various languages will exisit for a
long time, and generally communicate only internally to themselves.

To that end, I suspect most of the world isn't dealing with this.
Speakers of language "x" use a similar set of tools that use a 
similar set of encodings and character sets.  The problem is when
someone who speaks language x communicates with someone using language
y, and expects them to view something like they do (eg, the book
analogy).  While I think it would be good for us all to view the same
thing (like a book), it really does no good as we can't understand it.
Ever see foreign language books in your local bookstore?  No?  There's
a good reason.

So, protocols being language independant == good, let's support them
all until the world decides on one.  Expecting anyone else to understand
your character set == bad.  For instance, for me something that decodes
chinese might as well display a row of "xxxxxxxxxxx"'s.  Displaying the
characters will do me no goood.

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 03 Jan 2003 22:12:52 -0800
Lines: 42
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> I expect IETF standards which update or replace earlier standards to
> address compatibility issues.  As discussed on the Usefor list, the
> current draft expresses requirements which are diametrically opposed to
> what the current standard (RFC 1036) requires (e.g. 1036 makes
> processing of articles with Subject: cmsg ... as a control message,
> whereas the current draft forbids such processing).  Failure to provide
> for a graceful transition is a recipe for disaster.

This specific case is a bad example, as major news software dropped
support for the obsolete control message format some time ago.

In general, while I agree with you that graceful transition is important,
I disagree that a graceful transition from RFC 1036 *in particular* is
important.  One of the very difficult parts about working on Usenet
standardization is that RFC 1036 is functionally obsolete.  It does not
describe how Usenet actually works today.

Accordingly, Usenet standardization work is half updating of RFC 1036 and
half throwing it out entirely and then going through the gruelling process
of attempting to measure and characterize current practice and them
formalize it in a standard.

When it comes to control message processing, the latter is definitely more
needed than the former.  For example, of the control messages listed in
RFC 1036, sendsys and version have been obsolete and essentially unheard
of for longer than I've been working on Usenet software (at least four
years), ihave and sendme are only useful for UUCP sites and are
practically unheard of given the wholesale conversion of Usenet to NNTP,
and checkgroups, newgroup, and rmgroup are routinely PGP-signed and
verified via a protocol RFC 1036 makes no mention of at all.

A graceful transition from existing software and existing practice is
important, and a graceful transition from RFC 1036 is a reasonable
fallback in those cases where we cannot characterize existing practice,
but RFC 1036 really shouldn't have much in the way of special standing in
this process given its age and the accumulated differences between it and
standard practice.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

content-class: urn:content-classes:message
Subject: RE: RFC 2047 and gatewaying
Importance: normal
Date: Fri, 3 Jan 2003 22:03:35 -0800
From: "Dan Kohn" <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

Charles, as someone with absolutely no political or religious stakes in
this discussion, I'd like to point out to you that I've found Ned Freed
to be one of the most level-headed, open-minded IETF oldtimers.  I've
seen him go out of his way to offer constructive criticism, and be far
more open to new ideas than many.  You've already spent several years of
your life working on usefor; I would suggest that you ignore his counsel
now at your work's peril.  The fact that he is the IESG member with the
most email/news/MIME/gateway experience only reinforces the weight his
opinion (positive or negative) would anyway hold.

Most specifically, I would immediately drop the idea of a message/rfc822
with anything other than 7 bit headers, as the idea is clearly dead on
arrival.  But, I think you need to suggest to usefor that they
reconsider the entire current approach.  Charles, you said:

>> 1. Netnews is to be regarded fundamentally as an 8-bit clean medium.
>> It is a _different_ medium than Email, although there are strong
>> links that must be preserved.

This is not a novel concept.  HTTP and BEEP are application-level
protocols that are 8-bit-clean, use MIME, and yet have been able to
survive just fine with 7-bit headers.

Charles said:

>> 5. Insofar as these choices conflict with practice in the Email
>> world, there was a consciously taken decision to compress any
>> resultant messiness into the gateways. The number of these is small
>> compared to the vast number of Unsnet servers worldwide and the even
>> vaster number of clients. 

Ned said:

> Yawn. Heard all these arugments before back with MIME. This doesn't
> address the backwards compatibility issue, and like it or not this is
> an issue you are going to have to address.

This is perhaps Ned's most important statement.  At countless times in
the last 20 years, the IETF and it's predecessors have chosen backward
compatibility over the "efficient" solution.  The (IESG-approved) IDNA
drafts epitomize this approach, by promulgating an "ugly-looking"
transfer encoding (punycode) rather than the "elegant" solution of
UTF-8.  The economic calculation was that clients that care about i18n
can implement IDNA, but that the IETF would not break the "social
contract" made with DNS software that was (and now still will be)
compatible with pre-IDNA standards.

Similarly, a usefor solution that does not immediately cause existing
software (*including* gateways) to "break" on usefor messages, seems
1000 times more likely to pass IESG muster.  To quote Ned again:

> The issue is instead that your approach in effect declares a large
> body of software as being no longer compliant with long established
> standards. This is something we try very hard not to do.

You say:

>> This is
>> why the WG sought, and was given in its charter, permission to
>> include extensions to the existing protocols.

But, usefor's charter says:

"The Goal of this working group is to publish a standards-track
successor to RFC 1036 that with particular attention to backward
compatibility, formalizes best current practice and best proposed
practice."

I believe you're radically underestimating the IESG interest in that
backward compatibility, specifically in how it relates to mail gateways.
Further, I don't think anyone on the IESG will consider "UTF-8 in
headers") as an essential feature of "best proposed practice".  The IESG
will demand i18n of headers, but it will equally push for the lowest
impact way of achieving this.

Kai Henningsen said:

> Very shortly put, there is a long tradition on Usenet of passionately
> hating 2047 encoding in at least some quarters. Note the word
> "passionately"; it is not exaggerated. Long flamewars have been fought
> about this before anyone even thought about USEFOR.

> My only personal stance on this is that I consider it a serious,
> inexcusable bug that 2821/2822 do not allow naked UTF-8 in headers.
> Frankly, "we once made the mistake to spec 7 bits so this must remain
> 7 bit forever" is absolutely and inexcusably insane. But that is a
> mail problem, not a news problem.

Not only is it a news problem, but it is the news problem that will
likely prevent the usefor draft from being approved.  What I don't get
is that news clients will need to understand 2047 and 2231 syntax
anyway, since such things are bound to leak in (from email if nowhere
else).

Kai's argument is quite similar to the arguments made in IDN for "just
use UTF-8".  A strong consensus was reached that the elegance of UTF-8
was a far lower priority than backward compatibility.  For RFC 2822, the
value of backward compatibility over elegance was even more obvious.  If
you care about wasting bits in RFC 2047 encoding of UTF-8, start
worrying about the ATM cell tax of every DSL line, or the obnoxious
SONET overhead.  But if you want to deploy a standard in the IETF, focus
on backward compatibility.

I have a simple question.  What can a UTF-8 subject header communicate
that an RFC 2047 one can't?  Other than inelegance, what's the downside
of 2047, when the upside is a huge increase in backward compatibility?
Given that any reasonable news agent is going to need to be able to do
"pretty display" of the following address from
<http://www.normos.org/ietf/draft/draft-faerber-i18n-email-netnews-names
-00.txt>:

  From: =?ISO-8859-1?Q?Claus_F=E4rber?=
<[email-masked]>

and that RFC 1958 says that "If there are several ways of doing the same
thing, choose one",

why require an additional encoding of UTF-8 at all?

In summary, I would strongly consider redefining the usefor article
format to be RFC 2822+2047+2231 + a bunch of 1036 headers +
faerber-i18n.  Unless and until you can make a compelling case for UTF-8
*and* you can explain how a large number of gateways will not break on
usefor messages, I would give up on UTF-8 headers.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

  Randomly generated quote:
As the British Constitution is the most subtle organism which has
proceeded from the womb and long gestation of progressive history, so
the American Constitution is, so far as I can see, the most wonderful
work ever struck off at a given time by the brain and purpose of man. 
- W.E. Gladstone    

Date: Sat, 04 Jan 2003 00:25:21 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:

>>>It receives an article with UTF8-xtra-chars in some unrecognized header.
>>>For sure it was not a header known to the Netnews protocol, so it was
>>>superfluous for news propagation.
> 
> 
>>Not necessarily; it may be one of the newfangled Usefor "headers" and the
>>"gateway" may be one operating according to the current specification which
>>does not define such a "header".
> 
> 
> Naturally, a gateway that has not been upgraded to conform to the new
> standard will not conform to the new standard. Do you really expect such
> tautologies to be included in IETF standards?

I expect IETF standards which update or replace earlier standards to
address compatibility issues.  As discussed on the Usefor list, the
current draft expresses requirements which are diametrically
opposed to what the current standard (RFC 1036) requires (e.g. 1036
makes processing of articles with Subject: cmsg ... as a control
message, whereas the current draft forbids such processing).  Failure
to provide for a graceful transition is a recipe for disaster.

>> Or it may be an experimental "header", the
>>syntax of which is only known to those participating in the experiment [in
>>email, such expermental headers would begin with "X-", but current practice
>>in Usenet articles appears to be to make up some tag without consideration
>>of collisions with existing standards (e.g. Supersedes)].
> 
> 
> Experimental headers are, by definition, those beginning with "X-". They
> are always treated as unstructured by RFC 2047, which is also what my text
> says.

RFC 2047 encoding is used for parts of message header fields which are
used for display, i.e. they are not interpreted as part of the message
protocol -- display name phrases used with mailboxes and named groups,
comments, and unstructured headers (Subject: and Comments: among the
standard headers).  In all cases, the header text remains unaltered; UAs
may decode the text *for display*. 2047 does not directly address gateway
transformations; the only mention of gateways in 2047 is in reference to
the design of the encoding.  2047 does not provide for altering header
fields, unstructured or not, unlike your text.

In RFC 822/2822 Internet text messages, by definition no [experimental]
header field can contain any non-ASCII octet.  So under the current standard
(RFC 1036), Usenet gateways need not be concerned with experimetal headers
as 1036 uses the 822 Internet text message format. If the Usenet article
format is to be changed to include non-ASCII octets, then all Usenet
gateways to email will have to be modified.  And unless experimental
Usenet headers are forbidden non-ASCII content, that means that those
gateways will need to be able to provide appropriate encoding, which
in  turn means that all gateways will need to be able to determine the
syntax of all experimental headers in advance -- clearly impractical.

There are only a few practical options:
1. stay with the Internet text message format, which avoids the problem.
      As there is then no relevant change from 1036 w.r.t. gateway
      transformations, there are no transition issues w.r.t. those gateway
      transformations.
2. deviate from the Internet text message format only for a specific,
      well-defined set of Usefor headers which must be recognized by all
      news->mail gateways (a graceful transition plan will be required).
3. standardize processing of unrecognized Usefor headers which
      contain non-ASCII octets in such a way as to guarantee that the
      resulting mail message fully complies with all relevant mail
      standards -- and a gracful transition plan will be required.

>>You are presuming (incorrectly) that there are no other possible options.
> 
> 
> Well you have failed to provide any others that were not already in my
> text.

I have indeed provided others, quoted below.

   > And you have failed to answer the question. Which would you
> recommend?

"Answer the question: have you stopped beating your wife -- yes or no?"

I have in fact answered that your item #1 is unacceptable and that #4
cannot be implemented.  That leaves #2, #3, and other possiblites such
as I described.

>>The best solution would be to continue the RFC 1036 practice of using the
>>Internet text message format, i.e. there would be no untagged, unencoded
>>illegal octets or superfuous "parameters", 
> 
> 
> Yes, but the Rough Consensus on the Usefor Group is not to do that.

As noted by Russ, that is in large measure because those who are not
raw-untagged-utf-8 chauvinists have stopped wasting thier time patiently
explaining the issues to the latter, and/or have tired of the seven-plus
year project which was supposed to address several "urgent" issues...

>>Returning to your four suggestions:
>>#1 would violate Internet RFCs (822 / 2822 and probably 2821), so is unacceptable.
> 
> 
> So you have never seen a violation of RFC 2822? Lucky you! No Korean spam!

Spammers violating technical standards as well as ethical ones is one
matter (and in fact at least *some* Asian spam *does* use properly encoded
and tagged header content), but an IETF draft that encourages violation of
IETF technical standards is an entirely different matter.

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 03 Jan 2003 21:01:42 -0800
Lines: 92
Sender: [email-masked]

Leo Bicknell <[email-masked]> writes:

> What's my point?  Today I can "grep" a newsgroup article (ok, depends on
> server software and format), pass that to "dig" (ok, maybe with some sed
> and other work) and find out DNS information, and then use "sendmail"
> (ok, with a wrapper to generate a real e-mail) to mail someone.  I hope
> we all agree this is a good thing.  I see a future developing where a
> custom filter will be needed between each of those steps to preserve
> "international" characters.  I hope we can all see why that is bad.

Right.  I think we've all been here and all had these same thoughts.

Here's where my thought processes went after those thoughts, roughly:

 * I realized that, due to the way that 8-bit character sets developed,
   there is no "canonical" 8-bit character encoding and certainly no
   canonical character encoding that handles the full range of characters
   people want to use.  The realization that this leads to is that there's
   simply no such thing as "unencoded" 8-bit data.

   It's best to think of all 8-bit character data as encoded.  UTF-8 is
   just as much an encoding as RFC 2047.  UTF-16 is an encoding.  UTF-32
   is an encoding.  You're going to have to deal with an encoding, no
   matter what happens, until some mythical far future in which everyone
   uses exactly the same data representation, and then we might be able to
   stop thinking about encoding like we did with US-ASCII.

 * Your goal above therefore reduces to "can we all agree on an encoding
   for characters?"

 * The development of character sets is highly political.  It's not clear
   that development of a single character set is going to succeed,
   unfortunately not for technical reasons but for political reasons.  But
   if any is going to succeed, Unicode has the best shot.  Unicode is,
   however, a character *set*, not a unique character *encoding*.  That's
   just half the puzzle, namely "what characters exist."

 * Lots of people don't use Unicode at all and continue to not use
   Unicode.  Are those people all going to switch to Unicode?  I really
   don't know; again, see the above point about politics.  I don't know
   very much about politics, particularly those politics, so I don't feel
   qualified to even guess.  That means that assuming that Unicode is
   going to win seems pretty risky to me.

 * Assuming that the world agrees on Unicode (there don't seem to be any
   other viable options for a universal character set used all over the
   world), there are at least three major encodings (UTF-8, UTF-16, and
   UTF-32) that can be used with it, as well as a bunch of other minor
   ones.  So "use Unicode" doesn't answer what encoding we should try to
   use.

 * UTF-16 breaks Internet protocols and all the programs that manipulate
   them horribly because UTF-16 includes characters that C libraries think
   are embedded NULs and therefore end-of-string markers.  So I have a
   hard time imagining the Internet standardizing on UTF-16.  You can't
   even store files in a Unix file system with names encoded in UTF-16
   without major surgery to the entire C API.

 * Microsoft Windows uses UTF-16, as do many other early adopters of
   Unicode.

 * UTF-8 seems to be the obvious choice for Internet protocols, since
   transforming US-ASCII into UTF-8 is an identity transform, and NUL
   still means NUL.  However, UTF-8 penalizes non-ASCII characters
   spacewise, and is somewhat more complex to parse and reason about than
   a pure multibyte character set.  So there are good reasons to use it
   and good reasons not to use it.

 * sendmail still is not fully 8-bit clean in the latest analysis that
   I've seen, which means that 7-bit issues continue to plague the mail
   system.

When you put all of that together, I don't see any clear character
encoding winning, and furthermore I think that the reasons why no
character encoding is winning are outside the scope of issues that the
IETF can manage.

I accordingly then became much less optimistic about UTF-8 providing the
sort of universal tool access that you talk about, and have been unable to
see any clear path that will get us to there from where we are now.
Universal flag days, as you point out, aren't going to work.  I'm
therefore becoming increasingly resigned to having to deal with encodings
pretty much forever, or at least for the forseeable future.  So in answer
to Charles's question, yes, I expect to see RFC 2047 ten or even twenty
years from now.  I hope I won't, but I do expect that.

If it's any comfort, the only reason why you and I have been spared this
so far is because we both use English and are therefore happy with
US-ASCII.  The rest of the world has been dealing with this for years.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 3 Jan 2003 23:35:29 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Fri, Jan 03, 2003 at 08:03:02PM -0800:
> > OTOH, one hears rumours that the Chinese are not altogether happy.
> 
> Hardly a rumor. The Chinese dislike Unicode, or perhaps more accurately the
> UTF-8 and UTF-16 encodings of Unicode, sufficiently to have defined a new

Why?

Sam

> charset called GB18030 that is backwards compatible with GBK and GB2312 but can
> handle the full Unicode repetoire as well. And it is my understanding that
> there may come a day where GB18030 support will be required in order to sell
> some software in China.
> 
> Writing code to support GB18030 isn't fun at all. I know; I've done it.
> 
> 				Ned

Date: Fri, 03 Jan 2003 20:03:02 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> > The majority of folks on USEFOR also seem to me to have bought
> > wholeheartedly into the concept of Unicode and the belief that Unicode
> > will displace all other character sets in the near future, that all
> > software will soon expect any untagged 8-bit data to be in Unicode, and
> > that handling Unicode is all that's necessary to move forward into the
> > non-ASCII world.

> Well it seems that Microsoft have bought into Unicode pretty thoroughly,
> and that may well be the determining factor. Much as it grieves me to
> admit it, this may be one thing that Microsoft did right.

The fact that the Windows OS offers native Unicode support through various APIs
doesn't mean all applications on Windows handle Unicode well if at all. Nor
does the presence of APIs that handle UTF-16 automatically translate into
support for UTF-8.

And even if the presence of Unicode support in Windows did mean all
applications on Windows handled Unicode well, this doesn't translate into
everyone starting to use Unicode encodings on the wire.

> OTOH, one hears rumours that the Chinese are not altogether happy.

Hardly a rumor. The Chinese dislike Unicode, or perhaps more accurately the
UTF-8 and UTF-16 encodings of Unicode, sufficiently to have defined a new
charset called GB18030 that is backwards compatible with GBK and GB2312 but can
handle the full Unicode repetoire as well. And it is my understanding that
there may come a day where GB18030 support will be required in order to sell
some software in China.

Writing code to support GB18030 isn't fun at all. I know; I've done it.

				Ned

Date: Fri, 3 Jan 2003 22:35:18 -0500
From: Leo Bicknell <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: United Federation of Planets
Reply-By: Fri Jan 10 22:17:30 EST 2003
Sender: [email-masked]

I wanted to jump in here for a moment to draw a few things together.

Recently on Nanog (information and archives at
http://www.nanog.org/mailinglist.html) there was an announcement
about NSI moving forward with "international" domain names, eg a
hack to DNS to allow non US-ASCII (or maybe non ISO-8859-1)
characters.

After reading the thread on here, and the information on the NSI
proposal I must admit I am alarmed.  After doing some more research
it appears the problem of non US-ASCII character sets is appearing
across the board, which is no surprise.  Being only a US-ASCII
speaker, the problem is of little concern to me, but the proposed
solutions I have seen so far are of great concern.

While I know little of Usenet's problems, or of NSI's motivations,
I can speak as a sysadmin who has found simple harmony to the fact
that all the internet protocols use US-ASCII, or ISO-8859-1.  I
also see a future developing where if I want to code up a perl
script to do something I'll have to understand UTF-8 to read a
Netnews article, some weirdo NSI encoding to DNS resolve names from
that article, and then know how to base64 encode things for e-mail.
And that's just for three services I have been following.

This is already bad, but knowing that OS vendors are dealing with
the UTF-7/UTF-8/UTF-16 problem, and other protocol groups are
dealing with the same issues (indeed, IMAP uses UTF-7) seems to
show a very fragmented future.

I have no idea who's right or wrong, but I do know that if every
protocol/service decides on a protocol/service specific method of
encoding non US-ASCII characters that we're going to make a monumental
step backwards.  I also understand that the Internet is probably
too large today for a "flag day" to UTF-8 (or UTF-7, those are
probably the only two options network wide).

What's my point?  Today I can "grep" a newsgroup article (ok,
depends on server software and format), pass that to "dig" (ok,
maybe with some sed and other work) and find out DNS information,
and then use "sendmail" (ok, with a wrapper to generate a real
e-mail) to mail someone.  I hope we all agree this is a good thing.
I see a future developing where a custom filter will be needed
between each of those steps to preserve "international" characters.
I hope we can all see why that is bad.

If you should take anything from this e-mail, it's that I would
support an IETF group to develop a meta-standard for how we migrate
from US-ASCII or ISO-8859-1 to Unicode, done in such a way that
all protocols and services can use a similar, and most importantly
compatable scheme.  This starts at the OS (layer 3.5ish?) and moves
all the way up to the application.  Having new protocols/services
deployed with "proprietary" solutions is, in my opinion, a step
backwards.

-- 
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

Date: Fri, 03 Jan 2003 22:20:32 -0500
From: Lawrence Greenfield <[email-masked]>
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

--On Friday, January 03, 2003 1:14 PM +0000 Charles Lindsey 
<[email-masked]> wrote:

>
> In <[email-masked]> Russ Allbery <[email-masked]>
> writes:
>
>
>> I never understood what was wrong with RFC 2047 encoding, at least for
>> everything except the Newsgroups header (which poses some special
>> challenges), but many other people on USEFOR seem to feel that it would
>> be a catastrophically bad decision, and as I only speak English, I don't
>> feel particularly well-qualified to comment.
>
> I think it is a general belief that cramming 8 bits into 7 was never going
> to be the solution for the long term. Does anyone really believe that we
> shall still be using $RFC 2047 encoding in 20 years time? There is just
> too much effort being expended in writing code to encode and decode
> things. That may be acceptable for mainstream servers and user agents, but
> it is an utter pain for people who write scripts and filters for special
> purpose uses. Yes, there are some nice facilities in the Perl CPAN
> library, but it is still hard work to use them.

UTF-8 is an encoding. What Windows uses (UTF-16) is an encoding.

I'll leave it as a question to the reader whether or not UCS-4 is an 
encoding. (Big endian or little endian?)

Larry

To: [email-masked]
Xref: clerew local.mime:2021
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Fri, 3 Jan 2003 13:14:06 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>I never understood what was wrong with RFC 2047 encoding, at least for
>everything except the Newsgroups header (which poses some special
>challenges), but many other people on USEFOR seem to feel that it would be
>a catastrophically bad decision, and as I only speak English, I don't feel
>particularly well-qualified to comment.

I think it is a general belief that cramming 8 bits into 7 was never going
to be the solution for the long term. Does anyone really believe that we
shall still be using $RFC 2047 encoding in 20 years time? There is just
too much effort being expended in writing code to encode and decode
things. That may be acceptable for mainstream servers and user agents, but
it is an utter pain for people who write scripts and filters for special
purpose uses. Yes, there are some nice facilities in the Perl CPAN
library, but it is still hard work to use them.

The fact is that people will use whatever works for them. There is already
a considerable amount of Non-ASCII around in both Usenet and Email. Email
transports are now mostly 8bit clean, and even Sendmail are committeed to
that in their next major release, if not before. The IETF can indeed try
to enforce standards such as RFC 2047, but the IETF also needs to remember
the fate of King Canute.

>The majority of folks on USEFOR also seem to me to have bought
>wholeheartedly into the concept of Unicode and the belief that Unicode
>will displace all other character sets in the near future, that all
>software will soon expect any untagged 8-bit data to be in Unicode, and
>that handling Unicode is all that's necessary to move forward into the
>non-ASCII world.

Well it seems that Microsoft have bought into Unicode pretty thoroughly,
and that may well be the determining factor. Much as it grieves me to
admit it, this may be one thing that Microsoft did right.

OTOH, one hears rumours that the Chinese are not altogether happy.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 03 Jan 2003 19:09:55 -0800
Lines: 17
Sender: [email-masked]

Kai Henningsen <[email-masked]> writes:
> [email-masked] (Russ Allbery) wrote:

>> that all software will soon expect any untagged 8-bit data to be in
>> Unicode,

> That part is baseless exaggeration.

Quite to the contrary, that's a summary of the basic foundation of the
current draft, and is exactly what has been argued by multiple people on
the USEFOR list on multiple occasions.

To be more precise, the expectation is that any untagged 8-bit data will
be in UTF-8.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 03 Jan 2003 14:59:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Russ Allbery)  wrote on 02.01.03 in <[email-masked]>:

> I never understood what was wrong with RFC 2047 encoding, at least for
> everything except the Newsgroups header (which poses some special
> challenges), but many other people on USEFOR seem to feel that it would be
> a catastrophically bad decision, and as I only speak English, I don't feel
> particularly well-qualified to comment.

Very shortly put, there is a long tradition on Usenet of passionately  
hating 2047 encoding in at least some quarters. Note the word  
"passionately"; it is not exaggerated. Long flamewars have been fought  
about this before anyone even thought about USEFOR.

My only personal stance on this is that I consider it a serious,  
inexcusable bug that 2821/2822 do not allow naked UTF-8 in headers.  
Frankly, "we once made the mistake to spec 7 bits so this must remain 7  
bit forever" is absolutely and inexcusably insane. But that is a mail  
problem, not a news problem.

> The majority of folks on USEFOR also seem to me to have bought
> wholeheartedly into the concept of Unicode and the belief that Unicode
> will displace all other character sets in the near future, that all

Incidentally, so has the IETF, judging by the charsets-in-RFCs RFC.

> software will soon expect any untagged 8-bit data to be in Unicode, and

That part is baseless exaggeration.

> that handling Unicode is all that's necessary to move forward into the

Not exactly "all that's necessary": necessary, and would solve all charset  
problems except the one that people do still use other charsets.

> non-ASCII world.  I part company some on that point; these days, Unicode
> looks to me a lot more like IPv6, namely a better technical concept that
> may eventually get quietly deployed but that isn't going to change the
> world any time soon.

It's already started changing the world, and is already deployed in  
absolutely fucking *HUGE* numbers. Remember, for example, *every* WinNT- 
based machine has it in the OS interfaces (in the form of UTF-16). Given  
that XP is WinNT-based, that's a lot. (And for Win95-based machines  
there's a standard upgrade.) Also, Linux distributions seem to be  
switching to use UTF-8 as base character set, too. MacOS X, ditto UTF-16.

There's a lot more applied Unicode around than IPv6.

MfG Kai

Date: 03 Jan 2003 15:04:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Bruce Lilly)  wrote on 24.12.02 in <[email-masked]>:

> The best solution would be to continue the RFC 1036 practice of using the
> Internet text message format, i.e. there would be no untagged, unencoded
> illegal octets or superfuous "parameters", and gateway header

That, however, is not the existing real world practice. Real world  
practice among those actually using non-ASCII characters is, not for  
everyone but for huge numbers of messages, to have untagged non-ASCII  
stuff in headers in whatever the local charset happens to be.

(And, of course, 1036 did _not_ use the Internet text message format, it  
defined its own that looked similar but had lots of important  
differences.)

MfG Kai

Date: Fri, 03 Jan 2003 16:42:56 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

there was also a recent draft (draft-faerber-i18n-email-netnews-names) 
on using the IDN algorithms for newsgroup and various other news- and 
email-related names.

	Tony Hansen
	[email-masked]

Lawrence Greenfield wrote:
> 
> --On Friday, January 03, 2003 11:38 AM -0800 [email-masked] 
> wrote:
>> Newsgroup names are an interesting case because they aren't something 
> email
>> cares about. While an 8bit newsgroup name in some news-specific header
>> field may not survive a trip into and out of email, at least it doesn't
>> trash any of the email-specific pparts of the message.
>>
>> An argument to have this one thing be in 8bit would still be a very tough
>> sell, but far far easier than arguing that it is OK to have 8bit fields
>> that are shared between email and netnews.
> 
> Newsgroup names are a very hard problem. Note that the way Usenet works 
> right now requires these get passed gracefully through e-mail: in 
> moderated newsgroups posts are e-mailed to the moderator.

Date: Fri, 03 Jan 2003 15:55:51 -0500
From: Lawrence Greenfield <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

--On Friday, January 03, 2003 11:38 AM -0800 [email-masked] wrote:

[...]
> Newsgroup names are an interesting case because they aren't something 
email
> cares about. While an 8bit newsgroup name in some news-specific header
> field may not survive a trip into and out of email, at least it doesn't
> trash any of the email-specific pparts of the message.
>
> An argument to have this one thing be in 8bit would still be a very tough
> sell, but far far easier than arguing that it is OK to have 8bit fields
> that are shared between email and netnews.

Newsgroup names are a very hard problem. Note that the way Usenet works 
right now requires these get passed gracefully through e-mail: in moderated 
newsgroups posts are e-mailed to the moderator.

I'm not sure I understand the NEED for the "canonical" name to be pure 
UTF-8 (that is, encoded Unicode) instead of some other encoding.

A similiar problem happened in IMAP4 when people wanted to internationalize 
mailbox names (which function similiarly or identically to newsgroups, 
depending on perspective). In IMAP4's case, we decided on a modified UTF-7 
encoding over the wire. This means servers have an option of the underlying 
canonical name: they can store the mUTF-7, they can store UTF-8, they can 
store UTF-16, they can store in a legacy charset (and refuse all mailboxes 
with names that can't be encoding in the legacy charset). IMAP4 mailboxes 
can also appear in IMAP URIs, in which case they are encoded in UTF-8 and 
then hex-escaped. The IMAP4 URI spec included C code to do this conversion.

I don't think that introducing an all new encoding is definitely the right 
choice for Usenet---of course, the draft currently does introduce a new 
encoding but it's only used some of the time.

Larry

Date: Fri, 03 Jan 2003 11:38:58 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Keith Moore <[email-masked]> writes:

> > Too many WGs have made the mistake in thinking that their consensus
> > would be acceptable to the wider community, or could be made acceptable
> > without major changes.  More WGs need to find out what is acceptable to
> > the wider community long before they go to Last Call.

> On top of that, the current consensus, such as it is within USEFOR, is a
> consensus of exhaustion, and somewhat marginal even at that.  It's been
> years, we still don't have any sort of base document, and arguments like
> this one have driven many people off of the mailing list entirely.

Yes, this is unfortunately not an uncommon problem. It is all the more
unfortunate because all those stray chickens tend to come home to roost during
last call. Or worse, even later.

> I've personally just ran out of energy to argue about the right way to do
> non-ASCII newsgroup names.  There's some huge amount of complexity that's
> going to get shoved somewhere, and there are fewer gateways than there are
> news clients or news servers, so I suppose it makes some sense to shove
> the complexity there.  It seems unlikely to me that the current approach
> is going to actually work that well, but I ran out of anything better to
> offer that's not unacceptable to some other group of people.

Newgroup names are an interesting case because they aren't something email
cares about. While an 8bit newsgroup name in some news-specific header
field may not survive a trip into and out of email, at least it doesn't
trash any of the email-specific pparts of the message.

An argument to have this one thing be in 8bit would still be a very tough sell,
but far far easier than arguing that it is OK to have 8bit fields that are
shared between email and netnews.

> I never understood what was wrong with RFC 2047 encoding, at least for
> everything except the Newsgroups header (which poses some special
> challenges), but many other people on USEFOR seem to feel that it would be
> a catastrophically bad decision, and as I only speak English, I don't feel
> particularly well-qualified to comment.

There was plenty of input from non-English speakers during the design of MIME.
Indeed, the original motivation that started the MIME work was to
"internationalize email". We ended up going a bit further than that...

> The majority of folks on USEFOR also seem to me to have bought
> wholeheartedly into the concept of Unicode and the belief that Unicode
> will displace all other character sets in the near future, that all
> software will soon expect any untagged 8-bit data to be in Unicode, and
> that handling Unicode is all that's necessary to move forward into the
> non-ASCII world.  I part company some on that point; these days, Unicode
> looks to me a lot more like IPv6, namely a better technical concept that
> may eventually get quietly deployed but that isn't going to change the
> world any time soon.

I hate to have to say it, but I'm with you on this as well. I believe Unicode
and more specifically UTF-8 is the best answer we have in hand at present and
as such constitute a preferred solution of sorts, but this doesn't mean it has
the ability to magically displace stuff that's already out there. On the
contrary, use of all sorts of other charsets continues unabated, and speaking
as someone who deals with this stuff regularly as part of my day job, attempts
to actually deploy UTF-8 as a solution are unsuccessful on a fairly regular
basis. The necessary will just isn't there.

My only hopes for significant displacement of other charsets arise from the
develope of UTF-8 only solutions in other protocol spaces. These uses put
pressure on older protocols to trade up. But I have to admit the results of
such pressure are taking a long time to show up.

				Ned

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 03 Jan 2003 10:40:07 -0800
Lines: 68
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> 6. MIME is a set of protocols defined for the Email world, as is
> explicitly stated in RFC 2045. Officially speaking, there is no such
> thing as MIME within Netnews (which accounts for its extremely poor
> uptake within Usenet).

I doubt this.  I believe that the slow uptake of MIME within Usenet is due
to other issues entirely, primarily that many news servers have had to
block attachments of any kind because of space constraints and abuse and
MIME makes it easier for people to attach documents, and even more because
of some horrible misuse of multipart/alternative back in the early days.

I really don't think the lack of official standardization for Usenet had
anything to do with it, and I believe that if you look, you'll find that
many, if not most, Usenet messages have Content-Type and MIME-Version
headers.  I don't think that's extremely poor uptake.

There continues to be a somewhat irrational bias against any sort of MIME
multipart since (largely due to multipart/alternative) they're considered
synonymous on Usenet with massive, unnecessary bloat that looks horrible
in news readers, even though most news readers cope fine now.

There were also some secondary problems with quoted-printable encoding and
clients that were too aggressive about going to an encoding rather than
just figuring out the right charset tag to use, since quoted-printable
pretty much destroys source code excerpts for a client that doesn't know
how to decode it.  Technical mailing lists have had this same problem.

> 9. In fact, there are only two places where MIME in Netnews will differ
> from MIME in Email. One is in allowing full UTF-8 within quoted-strings
> within parameters in the Content-* headers (and probably in comments
> within those headers too).

I think this is a really unfortunate decision to try to push.  There are
already two encoding methods in widespread use for parameters in MIME
headers, namely the correct RFC 2231 encoding and the incorrect but widely
used and understood mangling of RFC 2047.  Adding yet a third encoding
that, on top of everything else, is unlabelled really isn't doing anyone
any favors.

> It is already the case that 2.5% [1] of Usenet articles use raw
> Non-ASCII characters in their headers. I have not heard of any damage or
> chaos arising from gateways that cannot cope.

Gateways pretty much universally pass those characters into e-mail
unencoded, which for the most part works because nearly all widely used
encodings avoid the no-man's-land of unassigned ISO 8859-1 codes that are
mangled by sendmail.  UTF-8 does not.

However, I think one key point that you're missing is that the arrival of
those messages at a destination mailbox without any changes made to the
bytes contained in the message isn't success in and of itself.  Those
bytes also have to be *interpreted* correctly.  As long as they're not
correctly interpreted by the end-user's mail client, the communication has
failed.

I daresay that you're likely not seeing Korean characters in all those
8-bit e-mail messages that you receive, indicating that the encoding that
those messages are using is failing to actually communicate.  The fact
that you received the raw bytes as they were sent is really rather
uninteresting by comparison.  And the reason why you're not seeing Korean
characters is that your client doesn't have enough information to know
what character set to use, since the Korean spam is, pretty much without
exception, unlabelled 8-bit content.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 03 Jan 2003 09:33:40 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> [email-masked] writes:

> >> Clearly, strict RFC [2]822 compliance is not a requirement, and use for
> >> News articles is explicitly encouraged (which is why Usefor makes it the
> >> official way to encapsulate News).

> >Strict conformance is not a requirement, but restriction of the headers
> >to US-ASCII is most certainly a requirement. See the third paragraph
> >of the section.

> It is a requirement for a "message/rfc822" that is to be conveyed as an
> email in accordance with RFC 204[56]. No quarrel with that, and the
> wording I proposed will bring it about.

More sophistry. The type is clearly defined and the rules for its content
exist independent of the transport being used to carry it around.

> Time to stand back and give a broader picture of what Usefor thinks it is
> trying to do.

> 1. Netnews is to be regarded fundamentally as an 8-bit clean medium. It is
> a _different_ medium than Email, although there are strong links that must
> be preserved.

> 2. But in fact, if you look at the headers we have defined so far, you
> will see that the only places 8bit freedom is allowed is in comments,
> phrases, parameters and newsgroup-names. That may not be so as new headers
> are defined in future extensions, but it is where we are at now.

> 3. There is a choice, for comments, phrases and parameters. Either they
> may be presented as raw UTF-8 _or_ they may be encoded using RFC
> 2047/2231. Conformant reading agents MUST support both. The document takes
> a neutral position as to which should be used (ultimately market forces
> will decide). It does point out that some existing software will break
> whichever is used.

This is going to be a tough sell in and of itself. People are going to ask why
you didn't pick a single scheme and stick to it.

> 4. The canonical form of newsgroup-names MUST be UTF-8. A special encoding
> is provided for transient use during moderation and for other specific
> purposes.

> 5. Insofar as these choices conflict with practice in the Email world,
> there was a consciously taken decision to compress any resultant messiness
> into the gateways. The number of these is small compared to the vast
> number of Unsnet servers worldwide and the even vaster number of clients.

Yawn. Heard all these arugments before back with MIME. This doesn't
address the backwards compatibility issue, and like it or not this is
an issue you are going to have to address.

> 6. MIME is a set of protocols defined for the Email world, as is
> explicitly stated in RFC 2045. Officially speaking, there is no such thing
> as MIME within Netnews (which accounts for its extremely poor uptake
> within Usenet).

Wrong on several counts. First, MIME is a format in addition to being a
protocol. This has serious implications in regards to how tightly you can draw
boundaries around a particular use of MIME in a specific application transport.

Second, while MIME was originally defined and targetted at email, it lost that
focus long ago -- certainly before RFCs 2045-2049 came out. The current MIME
documents try to make it clear what parts are email-specific and what parts are
more generally applicable. And media types are one of things that are more
generally applicable than just to email.

> 7. Usefor rectifies this by defining the usage of the MIME protocols
> within Netnews.

And that's a perfectly legitimate thing to do. But you cannot break the
definition of existing media type as part of this process. Pick another media
type, for heaven's sake!

And to the extent that netnews is not a world unto itself, you have to take the
effects of the sort of MIME you select on other things into account.

> 8. Insofar as MIME introduces the use of headers within the bodies of
> articles/messages, it is only to be expected that within Netnews the
> general conventions regarding Netnews headers should apply to them. Again,
> any resulting messiness is to be confined to the gateways.
 
And that's fine. Just don't reuse an existing media type in the process.

> 9. In fact, there are only two places where MIME in Netnews will differ
> from MIME in Email. One is in allowing full UTF-8 within quoted-strings
> within parameters in the Content-* headers (and probably in comments
> within those headers too). The other is in "message/rfc822". Both these
> cases therefore require attention by gateways.

> 10. RFC 1036 is now so far behind the times that it provides but a poor
> description of the Usnet of today. Considerable developments in practice
> have taken place since that time, all without the backup of formal
> standards. Some of these developments have worked well. Others have been
> haphazard and are going to be hard to undo. This is why the WG sought, and
> was given in its charter, permission to include extensions to the existing
> protocols.

I would refrain from arguing on the basis of what your charter currently says
if I were you. A WG whose current charter includes no goals and milestones
position in the IETF can only be described as "precarious". At a minimum a
rechartering exercise could be in order, and given recent experience with
charters and the IESG the resulting charter that would emerge would be very
likely to constrain you in ways you really would not like.

> 11. Insofar as our extensions introduce new and desirable functionality
> into Usenet, it is accepted that the benefits will be seen only as upgraded
> user agents come into use. Those users who care for the new functionality
> will take the trouble to upgrade. Those who do not will struggle on as they
> are. Nevertheless, we have taken trouble to ensure that the present
> transport backbone of Usenet will work as it stands (though it will work
> better with a little tweaking).

> >Second, to the extent that the article format is extended to allow utf-8 in
> >headers, you have to explain how the many gateways of the world between netnews
> >and email, gateways which are good faith implementation of various RFCs, are
> >going to be able to accomodate the changes you have made and not cause damage
> >to the infrastructure when they are presented with something they do not
> >expect. If you cannot come up with a satisfactory explanation of how this
> >damage will be minimized past history indicates that you will not be allowed to
> >make such changes in an RFC.

> It is already the case that 2.5% [1] of Usenet articles use raw Non-ASCII
> characters in their headers. I have not heard of any damage or chaos
> arising from gateways that cannot cope. The sky has not fallen in. No
> doubt some very odd-looking headers have emerged from the email side of
> those gateways and, by and large, users have ignored them and Just Hit
> DELETE.

Strawman argument since nobody is saying the sky will fall in. This was a
fairly serious issue at the time MIME was developed but I see no evidence it is
one any longer. The existance of vast amounts of illegally formatted mail has
more or less insured that everything out there can cope with 8bit material in
every possible part of a message. Of course coping doesn't imply that things
work correctly; clearly they frequently do not.

The issue is instead that your approach in effect declares a large body of
software as being no longer compliant with long established standards. This
is something we try very hard not to do.

> The Bad News, of course, is that these headers are mostly not in UTF-8.
> They are in every character set imagineable (and often with the explicit
> approval of the administrators of their hierarchies). So such articles
> will not be compliant with Usefor, but Usefor has had to take note of the
> problem and to suggest a workaround.

> And that problem is not limited to Netnews. 30% of the Email that I
> receive has raw Non-ASCII in its headers. I have to do a lot of Hitting
> DELETE :-( .

It is a useful criteria for detecting spam, isn't it? That alone should
give you pause...

> [1] That 2.5% figure is from memory. If you want the exact figure (the
> count has indeed been done) I will have to delve back into the Usefor
> archives. I suspect it was actually higher than 2.5%.

The argument of "but everyone breaks the rules so we can too" has been made
countless times. I cannot recall a single case, however, where such an argument
made it past the IESG.

> >The question you need to be asking is whether or not coming up with something
> >that is extremely unlikely to ever be approved as-is and which it is very
> >likely will have to be completely redesigned is a useful way for the WG to
> >spend its time.

> I am not expecting getting Usefor past the IETF to be an easy task. But
> then getting Usefore to make up its collective mind was not an easy task
> either :-( .

Well, you're clearly set on this course and it seems unlikely that anything I
say is going to change it. So this wil be my final response on this general set
of issues.

				Ned

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying 
Date: Fri, 03 Jan 2003 12:52:57 -0500
Sender: [email-masked]

> 2. But in fact, if you look at the headers we have defined so far, you
> will see that the only places 8bit freedom is allowed is in comments,
> phrases, parameters and newsgroup-names.

you obviously don't get it.  the definition of message/rfc822 is for
the content-type, not for the use of that content-type in email.
content-type definitions are independent of where they are used.
(the only exception is text/plain where the default charset is
"iso-8859-1" in http and "us-ascii" in email - but this was only a
hack for backward compatibility with pre-standards http.)

it's absolutely unacceptable for usefor to redefine the meaning
of message/rfc822 - even if they only pretend to do so for usenet.

To: [email-masked]
Xref: clerew local.mime:2020
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Fri, 3 Jan 2003 12:59:02 GMT
Lines: 129
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> Clearly, strict RFC [2]822 compliance is not a requirement, and use for
>> News articles is explicitly encouraged (which is why Usefor makes it the
>> official way to encapsulate News).

>Strict conformance is not a requirement, but restriction of the headers
>to US-ASCII is most certainly a requirement. See the third paragraph
>of the section.

It is a requirement for a "message/rfc822" that is to be conveyed as an
email in accordance with RFC 204[56]. No quarrel with that, and the
wording I proposed will bring it about.

Time to stand back and give a broader picture of what Usefor thinks it is
trying to do.

1. Netnews is to be regarded fundamentally as an 8-bit clean medium. It is
a _different_ medium than Email, although there are strong links that must
be preserved.

2. But in fact, if you look at the headers we have defined so far, you
will see that the only places 8bit freedom is allowed is in comments,
phrases, parameters and newsgroup-names. That may not be so as new headers
are defined in future extensions, but it is where we are at now.

3. There is a choice, for comments, phrases and parameters. Either they
may be presented as raw UTF-8 _or_ they may be encoded using RFC
2047/2231. Conformant reading agents MUST support both. The document takes
a neutral position as to which should be used (ultimately market forces
will decide). It does point out that some existing software will break
whichever is used.

4. The canonical form of newsgroup-names MUST be UTF-8. A special encoding
is provided for transient use during moderation and for other specific
purposes.

5. Insofar as these choices conflict with practice in the Email world,
there was a consciously taken decision to compress any resultant messiness
into the gateways. The number of these is small compared to the vast
number of Unsnet servers worldwide and the even vaster number of clients.

6. MIME is a set of protocols defined for the Email world, as is
explicitly stated in RFC 2045. Officially speaking, there is no such thing
as MIME within Netnews (which accounts for its extremely poor uptake
within Usenet).

7. Usefor rectifies this by defining the usage of the MIME protocols
within Netnews.

8. Insofar as MIME introduces the use of headers within the bodies of
articles/messages, it is only to be expected that within Netnews the
general conventions regarding Netnews headers should apply to them. Again,
any resulting messiness is to be confined to the gateways.
 
9. In fact, there are only two places where MIME in Netnews will differ
from MIME in Email. One is in allowing full UTF-8 within quoted-strings
within parameters in the Content-* headers (and probably in comments
within those headers too). The other is in "message/rfc822". Both these
cases therefore require attention by gateways.

10. RFC 1036 is now so far behind the times that it provides but a poor
description of the Usnet of today. Considerable developments in practice
have taken place since that time, all without the backup of formal
standards. Some of these developments have worked well. Others have been
haphazard and are going to be hard to undo. This is why the WG sought, and
was given in its charter, permission to include extensions to the existing
protocols.

11. Insofar as our extensions introduce new and desirable functionality
into Usenet, it is accepted that the benefits will be seen only as upgraded
user agents come into use. Those users who care for the new functionality
will take the trouble to upgrade. Those who do not will struggle on as they
are. Nevertheless, we have taken trouble to ensure that the present
transport backbone of Usenet will work as it stands (though it will work
better with a little tweaking).

>Second, to the extent that the article format is extended to allow utf-8 in
>headers, you have to explain how the many gateways of the world between netnews
>and email, gateways which are good faith implementation of various RFCs, are
>going to be able to accomodate the changes you have made and not cause damage
>to the infrastructure when they are presented with something they do not
>expect. If you cannot come up with a satisfactory explanation of how this
>damage will be minimized past history indicates that you will not be allowed to
>make such changes in an RFC.

It is already the case that 2.5% [1] of Usenet articles use raw Non-ASCII
characters in their headers. I have not heard of any damage or chaos
arising from gateways that cannot cope. The sky has not fallen in. No
doubt some very odd-looking headers have emerged from the email side of
those gateways and, by and large, users have ignored them and Just Hit
DELETE.

The Bad News, of course, is that these headers are mostly not in UTF-8.
They are in every character set imagineable (and often with the explicit
approval of the administrators of their hierarchies). So such articles
will not be compliant with Usefor, but Usefor has had to take note of the
problem and to suggest a workaround.

And that problem is not limited to Netnews. 30% of the Email that I
receive has raw Non-ASCII in its headers. I have to do a lot of Hitting
DELETE :-( .

[1] That 2.5% figure is from memory. If you want the exact figure (the
count has indeed been done) I will have to delve back into the Usefor
archives. I suspect it was actually higher than 2.5%.

>The question you need to be asking is whether or not coming up with something
>that is extremely unlikely to ever be approved as-is and which it is very
>likely will have to be completely redesigned is a useful way for the WG to
>spend its time.

I am not expecting getting Usefor past the IETF to be an easy task. But
then getting Usefore to make up its collective mind was not an easy task
either :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 02 Jan 2003 09:57:30 -0800
Lines: 38
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> Too many WGs have made the mistake in thinking that their consensus
> would be acceptable to the wider community, or could be made acceptable
> without major changes.  More WGs need to find out what is acceptable to
> the wider community long before they go to Last Call.

On top of that, the current consensus, such as it is within USEFOR, is a
consensus of exhaustion, and somewhat marginal even at that.  It's been
years, we still don't have any sort of base document, and arguments like
this one have driven many people off of the mailing list entirely.

I've personally just ran out of energy to argue about the right way to do
non-ASCII newsgroup names.  There's some huge amount of complexity that's
going to get shoved somewhere, and there are fewer gateways than there are
news clients or news servers, so I suppose it makes some sense to shove
the complexity there.  It seems unlikely to me that the current approach
is going to actually work that well, but I ran out of anything better to
offer that's not unacceptable to some other group of people.

I never understood what was wrong with RFC 2047 encoding, at least for
everything except the Newsgroups header (which poses some special
challenges), but many other people on USEFOR seem to feel that it would be
a catastrophically bad decision, and as I only speak English, I don't feel
particularly well-qualified to comment.

The majority of folks on USEFOR also seem to me to have bought
wholeheartedly into the concept of Unicode and the belief that Unicode
will displace all other character sets in the near future, that all
software will soon expect any untagged 8-bit data to be in Unicode, and
that handling Unicode is all that's necessary to move forward into the
non-ASCII world.  I part company some on that point; these days, Unicode
looks to me a lot more like IPv6, namely a better technical concept that
may eventually get quietly deployed but that isn't going to change the
world any time soon.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 02 Jan 2003 07:11:08 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> [email-masked] writes:

> >> In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

> >> >The draft proposes identifying MIME parts with message/rfc822 that do not
> >> >conform to RFC [2]822.

> >> So? There is no requirement for message/rfc822 to conform to RFC [2]822,
> >> as I showed. So please explain your problem more clearly.

> >Sure there is. See RFC 2046 section 5.2.1.

> Yes, I have read section 5.2.1. Indeed I posted a large chunk of it here a
> couple of days back. Here it is again:

> A media type of "message/rfc822" indicates that the body contains an
> encapsulated message, with the syntax of an RFC 822 message.
>                       ^^^^^^^^^^^^^^^
> ...............

The syntax of message/rfc822 does not include utf-8 in headers. Usefor does. As
such, they are not compatible. And this means that usefor messages are not
compatible with the message/rfc822 media type.

> It should be noted that, despite the use of the numbers "822", a
> "message/rfc822" entity isn't restricted to material in strict conformance
> to RFC822, nor are the semantics of "message/rfc822" objects restricted to
> the semantics defined in RFC822. More specifically, a "message/rfc822"
> message could well be a News article or a MIME message.

> Clearly, strict RFC [2]822 compliance is not a requirement, and use for
> News articles is explicitly encouraged (which is why Usefor makes it the
> official way to encapsulate News).

Strict conformance is not a requirement, but restriction of the headers
to US-ASCII is most certainly a requirement. See the third paragraph
of the section.

You are engaging in sophistry here, and it will not fly.

> The only requirement is that the syntax should agree with RFC [2]822, and
> I have no problem with that since the proposed Usefor wording for gateways
> that we are discussing explicitly requires that it should be so on the
> Email side of the gateway, and tells you what transformations are
> necessary to ensure it. So I still do not see what Larry's problem is.

There are at least two issues here. First, section 6.21.2.2 of the news format
document explicitly calls for this use of the media type message/rfc822 to
label news messages that contain utf-8 in their headers. This violates the
definition of message/rfc822 given in RFC 2046, a draft standard, and hence is
simply unacceptable. There are sure to be media type handlers out there that
expect message/rfc822 to only contain US-ASCII headers.

Second, to the extent that the article format is extended to allow utf-8 in
headers, you have to explain how the many gateways of the world between netnews
and email, gateways which are good faith implementation of various RFCs, are
going to be able to accomodate the changes you have made and not cause damage
to the infrastructure when they are presented with something they do not
expect. If you cannot come up with a satisfactory explanation of how this
damage will be minimized past history indicates that you will not be allowed to
make such changes in an RFC.

> >It seems USEFOR is in need of a reality check. The IETF process doesn't consist
> >of "WG reaches consensus, standards track RFC is published, done". There are
> >also the pesky matters of AD review, IETF last call, and IESG review. And a
> >document that makes incompatible changes to any number of standards, changes
> >that break implementations done in good faith to those standards, is extremely
> >unlikely to pass all those steps, regardless of whether or not it wins
> >enthusiastic approval in the WG itself.

> Yes, I know all that. But WG consensus comes first. Then what the WG
> produces has to be sold to the wider community, and then maybe the IESG
> asks for reconsideration of some issues.

The question you need to be asking is whether or not coming up with something
that is extremely unlikely to ever be approved as-is and which it is very
likely will have to be completely redesigned is a useful way for the WG to
spend its time.

> >Like it or not, one of the things RFC 1036 standardizes is the interoperability
> >of and ability to gateway between netnews and email. And this capability has
> >been used to advantage in IETF protocols like IMAP4, which has facilities
> >explicitly intended to accomodate netnews.  This has led to countless
> >implementations of news<-->email gatways and IMAP4 servers that can access
> >news. It matters not at all if the USEFOR group unanimously dislikes such
> >things and thinks they are of no value. They exist in abundance and are based
> >on IETF standards, and therefore cannot simply be dismissed as irrelevant.

> Actually, RFC 1036 does not standardize anything, because it is not a
> standard.

More sophistry. While it is true the document doesn't have standard status in
the IETF, its title is "Standard for the interchange of USENET messages".

> But, it never was the case, either in theory or in practice,
> that RFC 1036 objects were fully compatible with RFC 822 objects. In fact,
> Usefor is much closer to the mail standards than RFC 1036 ever was (for
> example, comments were never a part of RFC 1036 headers, except for a few
> specific cases, and much current Netnews software will barf when it sees
> them).

Perhaps, but in most ways RFC 1036 was a subset, not a superset, of the
messaging standards. RFC 2046 only found it necessary to loosen a couple of
rules on RFC 822 to allow for netnews articles. What you are now proposing is a
very significant and incompatible extension.

> You point about IMAP4 may need attention though. We were aware that some
> IMAP servers attempted to provide a news capability, but not that there
> were any such requirements in the IMAP standards. I need to look into
> that.

You need to look in to a lot more than that.

				Ned

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying 
Date: Thu, 02 Jan 2003 10:22:21 -0500
Sender: [email-masked]

> Yes, I know all that. But WG consensus comes first.

Too many WGs have made the mistake in thinking that their consensus
would be acceptable to the wider community, or could be made acceptable
without major changes.  More WGs need to find out what is acceptable
to the wider community long before they go to Last Call.

Date: Thu, 02 Jan 2003 09:31:11 -0500
From: Lawrence Greenfield <[email-masked]>
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

--On Thursday, January 02, 2003 12:01 PM +0000 Charles Lindsey 
<[email-masked]> wrote:

> Yes, I have read section 5.2.1. Indeed I posted a large chunk of it here a
> couple of days back. Here it is again:
>
> A media type of "message/rfc822" indicates that the body contains an
> encapsulated message, with the syntax of an RFC 822 message.
>                       ^^^^^^^^^^^^^^^
> ...............
>
> It should be noted that, despite the use of the numbers "822", a
> "message/rfc822" entity isn't restricted to material in strict conformance
> to RFC822, nor are the semantics of "message/rfc822" objects restricted to
> the semantics defined in RFC822. More specifically, a "message/rfc822"
> message could well be a News article or a MIME message.
>
> Clearly, strict RFC [2]822 compliance is not a requirement, and use for
> News articles is explicitly encouraged (which is why Usefor makes it the
> official way to encapsulate News).

Please read the next paragraph. Here's the paragraph you aren't quoting:

   No encoding other than "7bit", "8bit", or "binary" is permitted for
   the body of a "message/rfc822" entity.  The message header fields are
   always US-ASCII in any case, and data within the body can still be
   encoded, in which case the Content-Transfer-Encoding header field in
   the encapsulated message will reflect this.  Non-US-ASCII text in the
   headers of an encapsulated message can be specified using the
   mechanisms described in RFC 2047.

enjoy,
Larry

To: [email-masked]
Xref: clerew local.mime:2015
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Thu, 2 Jan 2003 12:01:03 GMT
Lines: 80
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

>> >The draft proposes identifying MIME parts with message/rfc822 that do not
>> >conform to RFC [2]822.

>> So? There is no requirement for message/rfc822 to conform to RFC [2]822,
>> as I showed. So please explain your problem more clearly.

>Sure there is. See RFC 2046 section 5.2.1.

Yes, I have read section 5.2.1. Indeed I posted a large chunk of it here a
couple of days back. Here it is again:

A media type of "message/rfc822" indicates that the body contains an
encapsulated message, with the syntax of an RFC 822 message.
                      ^^^^^^^^^^^^^^^
...............

It should be noted that, despite the use of the numbers "822", a
"message/rfc822" entity isn't restricted to material in strict conformance
to RFC822, nor are the semantics of "message/rfc822" objects restricted to
the semantics defined in RFC822. More specifically, a "message/rfc822"
message could well be a News article or a MIME message. 

Clearly, strict RFC [2]822 compliance is not a requirement, and use for
News articles is explicitly encouraged (which is why Usefor makes it the
official way to encapsulate News).

The only requirement is that the syntax should agree with RFC [2]822, and
I have no problem with that since the proposed Usefor wording for gateways
that we are discussing explicitly requires that it should be so on the
Email side of the gateway, and tells you what transformations are
necessary to ensure it. So I still do not see what Larry's problem is.

>It seems USEFOR is in need of a reality check. The IETF process doesn't consist
>of "WG reaches consensus, standards track RFC is published, done". There are
>also the pesky matters of AD review, IETF last call, and IESG review. And a
>document that makes incompatible changes to any number of standards, changes
>that break implementations done in good faith to those standards, is extremely
>unlikely to pass all those steps, regardless of whether or not it wins
>enthusiastic approval in the WG itself.

Yes, I know all that. But WG consensus comes first. Then what the WG
produces has to be sold to the wider community, and then maybe the IESG
asks for reconsideration of some issues.

>Like it or not, one of the things RFC 1036 standardizes is the interoperability
>of and ability to gateway between netnews and email. And this capability has
>been used to advantage in IETF protocols like IMAP4, which has facilities
>explicitly intended to accomodate netnews.  This has led to countless
>implementations of news<-->email gatways and IMAP4 servers that can access
>news. It matters not at all if the USEFOR group unanimously dislikes such
>things and thinks they are of no value. They exist in abundance and are based
>on IETF standards, and therefore cannot simply be dismissed as irrelevant.

Actually, RFC 1036 does not standardize anything, because it is not a
standard. But, it never was the case, either in theory or in practice,
that RFC 1036 objects were fully compatible with RFC 822 objects. In fact,
Usefor is much closer to the mail standards than RFC 1036 ever was (for
example, comments were never a part of RFC 1036 headers, except for a few
specific cases, and much current Netnews software will barf when it sees
them).

You point about IMAP4 may need attention though. We were aware that some
IMAP servers attempted to provide a news capability, but not that there
were any such requirements in the IMAP standards. I need to look into
that.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 01 Jan 2003 09:10:54 -0800 (PST)
From: [email-masked]
Subject: Re: RFC 2047 and gatewaying
To: Charles Lindsey <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

> >--On Tuesday, December 31, 2002 4:44 PM +0000 Charles Lindsey
> ><[email-masked]> wrote:

> >> No, the Usefor text is not recommending violating anything. The text I
> >> posted here is entirely consistent the RFC 2047/2231 as regards all cases
> >> that those RFCs appear to cover (and if it is not, then please tell me -
> >> that is why it was posted here).

> >The draft proposes identifying MIME parts with message/rfc822 that do not
> >conform to RFC [2]822.

> So? There is no requirement for message/rfc822 to conform to RFC [2]822,
> as I showed. So please explain your problem more clearly.

Sure there is. See RFC 2046 section 5.2.1.

It seems USEFOR is in need of a reality check. The IETF process doesn't consist
of "WG reaches consensus, standards track RFC is published, done". There are
also the pesky matters of AD review, IETF last call, and IESG review. And a
document that makes incompatible changes to any number of standards, changes
that break implementations done in good faith to those standards, is extremely
unlikely to pass all those steps, regardless of whether or not it wins
enthusiastic approval in the WG itself.

Like it or not, one of the things RFC 1036 standardizes is the interoperability
of and ability to gateway between netnews and email. And this capability has
been used to advantage in IETF protocols like IMAP4, which has facilities
explicitly intended to accomodate netnews.  This has led to countless
implementations of news<-->email gatways and IMAP4 servers that can access
news. It matters not at all if the USEFOR group unanimously dislikes such
things and thinks they are of no value. They exist in abundance and are based
on IETF standards, and therefore cannot simply be dismissed as irrelevant.

Nor does the sometimes heard claim of "you should have participated in the WG
in order to express your opinion, now it is too late to do so" pass muster. The
review steps in the process that lie outside the group itself are there for a
reason. It is a task for the group to gain the support of the community for
what it wants to do, not for the rest of the community to have to partipicate
in the group in order for their opinions to be heard.

USEFOR is not being singled out here. Examples abound of cases where other WGs
have reached consensus on a document only to find out during one or more of
these external steps that the document isn't acceptable in its current form and
that their entire approach has to be reconsidered. Recent examples that come to
mind among my own WGs are the FAX CONNEG proposal, which received substantial
pushback during last call and is now being extensively revised, and the WebDAV
ACL proposal, which got tripped up on security issues during IESG review.

I'm afraid USEFOR needs to reconsider its entire approach in light of what the
reaction of the larger community to their proposal is certain to be.

				Ned

To: [email-masked]
Xref: clerew local.mime:2013
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Wed, 1 Jan 2003 12:11:33 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

>--On Tuesday, December 31, 2002 4:44 PM +0000 Charles Lindsey 
><[email-masked]> wrote:

>> No, the Usefor text is not recommending violating anything. The text I
>> posted here is entirely consistent the RFC 2047/2231 as regards all cases
>> that those RFCs appear to cover (and if it is not, then please tell me -
>> that is why it was posted here).

>The draft proposes identifying MIME parts with message/rfc822 that do not 
>conform to RFC [2]822.

So? There is no requirement for message/rfc822 to conform to RFC [2]822,
as I showed. So please explain your problem more clearly.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5