Subject: Re: RFC 2047 and gatewaying
From: "Charles Lindsey" <[email-masked]>
To: [email-masked]
Date: Tue, 31 Dec 2002 17:06:03 GMT
Newsgroups: local.mime
List-Id: <ietf-822.imc.org>

In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

>   From: "Charles Lindsey" <[email-masked]>
>   Date: Mon, 30 Dec 2002 14:50:09 GMT
>[...]
>   Anyway, the Usefor list is the place to discuss these issues. All we are
>   concerned with here is whether my text covers the use of RFC 2047/2231
>   well enough.

>I think a document that encourages people to send non-RFC 2822
>messages as "message/rfc822" is well in scope here.

To quote from RFC 2046:

A media type of "message/rfc822" indicates that the body contains an
encapsulated message, with the syntax of an RFC 822 message.
                      ^^^^^^^^^^^^^^^
...............

It should be noted that, despite the use of the numbers "822", a
"message/rfc822" entity isn't restricted to material in strict conformance
to RFC822, nor are the semantics of "message/rfc822" objects restricted to
the semantics defined in RFC822. More specifically, a "message/rfc822"
message could well be a News article or a MIME message.

>If you want to update RFC 2822 to allow unencoded UTF-8 in headers, do
>so. Don't try to slip it in the backdoor.

But I don't. The text I posted made it quite clear that, on the Email side
of the gateway, all UTF-8 would have been removed, by one means or
another,

from the headers of the article
from the headers of any contained message/rfc822
from the headers of any contained MIME body part

and that if a particular implementor chose not to do that, and thus
produced a non-compliant Email object, then that would be his risk and
responsibility.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 31 Dec 2002 12:34:08 -0500
From: Lawrence Greenfield <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

--On Tuesday, December 31, 2002 4:44 PM +0000 Charles Lindsey 
<[email-masked]> wrote:

>> if the usefor spec is recommending violating either [2]822 or MIME
>> then you'd better discuss it here now - because otherwise you're going
>> to get shot down at last call.  and you'd bloody well deserve it.
>
> No, the Usefor text is not recommending violating anything. The text I
> posted here is entirely consistent the RFC 2047/2231 as regards all cases
> that those RFCs appear to cover (and if it is not, then please tell me -
> that is why it was posted here).

The draft proposes identifying MIME parts with message/rfc822 that do not 
conform to RFC [2]822.

Larry

To: [email-masked]
Xref: clerew local.mime:2010
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Tue, 31 Dec 2002 16:44:33 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> Anyway, the Usefor list is the place to discuss these issues. All we are
>> concerned with here is whether my text covers the use of RFC 2047/2231
>> well enough.

The issues I proposed should be taken to the Usefor list did not relate to
RFC 2047/2231.

>if the usefor spec is recommending violating either [2]822 or MIME
>then you'd better discuss it here now - because otherwise you're going
>to get shot down at last call.  and you'd bloody well deserve it.

No, the Usefor text is not recommending violating anything. The text I
posted here is entirely consistent the RFC 2047/2231 as regards all cases
that those RFCs appear to cover (and if it is not, then please tell me -
that is why it was posted here).

It is the cases that RFC 2047 does not cover that are at issue (namely
when a header is seen that the agent does not know how to parse). What do
present mail agents do when they see such things? Answer: they fudge it as
best they can. My text merely "suggested" (not "recommended", because that
word has special meaning) a course of action broadly in line with the
fudges that are already around.

The point is that such situations, though rare, will inevitably happen.
What is an agent to do in such cases? I proposed four alternatives, of
which the fudge was one, and invited people to say which they would do.
That invitation is still open.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying 
Date: Mon, 30 Dec 2002 16:47:15 -0500
Sender: [email-masked]

> Anyway, the Usefor list is the place to discuss these issues. All we are
> concerned with here is whether my text covers the use of RFC 2047/2231
> well enough.

if the usefor spec is recommending violating either [2]822 or MIME
then you'd better discuss it here now - because otherwise you're going
to get shot down at last call.  and you'd bloody well deserve it.

it's one thing to screw up usenet, but people actually use email.

Ketih

Date: Mon, 30 Dec 2002 16:16:30 -0500
From: Lawrence Greenfield <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

   From: "Charles Lindsey" <[email-masked]>
   Date: Mon, 30 Dec 2002 14:50:09 GMT
[...]
   Anyway, the Usefor list is the place to discuss these issues. All we are
   concerned with here is whether my text covers the use of RFC 2047/2231
   well enough.

I think a document that encourages people to send non-RFC 2822
messages as "message/rfc822" is well in scope here.

If you want to update RFC 2822 to allow unencoded UTF-8 in headers, do
so. Don't try to slip it in the backdoor.

Larry

To: [email-masked]
Xref: clerew local.mime:2007
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Mon, 30 Dec 2002 15:05:45 GMT
Lines: 65
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Pete Resnick wrote:
>> 
>> No, sorry, it doesn't work that way. Just because 2368 (via 1738) 
>> specifies that some structured part of an encoded URL follows 822/2822 
>> syntax for "mailbox" does not mean that a mail header field that 
>> contains such a URL therefore contains a mailbox. The parser for 
>> 822/2822/2047 does not know about URL parsing and cannot get past the 
>> "%" encoded bits. There is no intervening 1738 parser.

Exactly so. If you have an encoding inside an encoding, then the main
news/mail protocols need only concern themselves with the outer one. Only
the parts of the implementation that specifically understand that header
(we are talking of headers that contain URIs here) need to know what to do
with the inner one.

>That depends on the context. In the case of the List-Owner example and
>a number of fairly widely-used MUAs, the URL is properly decoded when
>using the List-Owner field content to generate a message to the list
>owner, and some even decode the 2047 encoding for display.  It is
>true that for transport, there is no decoding, but that's axiomatic as
>2047 is clearly intended for display purposes and not for processing
>during transport.

>Surely you're not suggesting that correct use of the List-Owner etc. header
>fields to generate messages to the list owner etc. would be to use the raw
>URL %-encoded text in the headers?

That is exactly what I would suggest for general use. URIs are already
7-bit clean, so there is no need for gateways to touch them. If some
user-agent that understands the List-Owner-header cares to look inside and
do clever things, then it is welcome, of course.

Now all this could change if IRIs (draft-duerst-iri-02.txt) should get to
the standard stage. That draft contains provisions for encoding them back
into URIs, which is probably what should be done by a gateway that spots
one (and maybe I have to include that case in my text at some stage).

>  It seems that you're implying that a
>To header generated from a mailto URL doesn't contain a mailbox even if it
>matches the mailbox syntax and although a To header generated from any other
>mechanism may contain a mailbox -- that doesn't sound right either. 

Yes, that would be my understanding too, at least so far as what RFC 2047
is supposed to do about deeply nested comments/phrases (unless Keith can
convince us otherwise).

>URLs can contain parentheses (which is why there's a problem with the
>RFC 2557 ABNF and CFWS).

Which is a good reason for putting URLs inside <...>, and ensuring that
nobody goes looking for comments inside there.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2006
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 30 Dec 2002 14:50:09 GMT
Lines: 58
Sender: [email-masked]

In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

>I think the current text on gatewaying such messages to mail is completely 
>insufficient. The algorithm given in 5.5.2 won't display in any clients 
>without modifications, despite those clients having UTF-8 support.

It was never expected that clients would decode it. Moderators may need to
arrange to do so, but Usefor discusses that elsewhere. The only case the
average mail user will encounter is in a posted-and-mailed article in
which the Newsgroups-header may look a little odd. But in an email that
header is informational only, so not semantic harm will ensue.

>The majority of e-mail clients interpret untagged 8-bit in their local 
>character set, which is likely not UTF-8, so ignoring the standards is not 
>a reasonable or acceptable answer.

Fine! My text does not suggest that. I merely pointed to that as one of the
things that gateways _might_ do. I think you just said you preferred one
of my other three possibilities. So do I.

> Using message/rfc822 to identify 
>articles (as in 6.21.2.2) with unencoded 8-bit headers is highly harmful 
>for processing messages.

Why? You want us to bring back the obsoleted message/news which no known
agent currently recognizes?

>Section 6.21.3 says
>   Injecting and relaying agents MUST NOT change the encoding of
>   articles passed to them. Gateways SHOULD NOT change the encoding
>   unless absolutely necessary.

>Please reconcile this with RFC 2822; you are already identifying non-RFC 
>822 messages are "message/rfc822" _and_ you say that gateways shouldn't 
>change this.

"unless absolutely necessary". Sometimes it will be absolutely necessary.

> What about clients saving messages to a local filesystem? 
>Should the mime type be identified as "message/rfc822"?

I don't see why not if that is the format of the saved object.

Anyway, the Usefor list is the place to discuss these issues. All we are
concerned with here is whether my text covers the use of RFC 2047/2231
well enough.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2005
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 30 Dec 2002 14:26:45 GMT
Lines: 169
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> That is not so, because Usefor explicitly extends RFC 2047 to apply to
>> Netnews (but it does not actually change anything in the 2047 protocol).

>Those extensions may well need to be presented here before one can reasonably
>expect a detailed critique of the bigger picture.

You can read the gory details in section 4.4.1 of
<http://www.landfield.com/usefor/draft-ietf-usefor-article-08.01.unpaged>
but all it does is to define a little terminology to allow the wording of
RFC 2047 to apply to Netnews articles.

>There is an example of the need for clarity; does "the header" refer to a
>Usefor draft "header" or to the RFC 822 / 2822 Internet text message header
>that the gateway is supposed to construct? 

Since the text we are discussing is part of Usefor, it should be obvious
to you that "header" means a Usefor header.

>[likewise for other parts of
>the text, such as section 5 below.]  It's also unclear, at least from what
>has been posted here, what the rules are for Usefor "headers". For example,
>it is unclear whether or not content in a structured Usefor "header" enclosed
>within U+207D and U+207E (or within U+208D and U+208E) is considered to be a
>"comment" [or, for that matter, whether or not U+2474 etc. are "comments"].

The Usefor definition of CFWS is identical to the RFC 2822 definition,
except for the extended character set allowed in ctext.

>Such considerations don't arise in Internet text message headers because
>those characters cannot exist in headers.  But since they may exist in
>Usefor "headers", the issue does arise.  And there is the more serious
>matter of whether or not one is expected to use the consistent RFC 822 /
>2822 / 2047 / 2231 definitions of "comment", "phrase", "quoted-string", etc.
>or the quite different Usefor draft definitions (which are different because
>the Usefor draft uses different definitions of "text", "ctext" "qtext", etc.).

Again, it should be obvious to you that the purpose of the text under
discusssion is to tell you what do do when you encounter a Usefor comment
(etc.) in a Usefor header in order to ensure that it will become a valid
Email header-field.

>> It receives an article with UTF8-xtra-chars in some unrecognized header.
>> For sure it was not a header known to the Netnews protocol, so it was
>> superfluous for news propagation.

>Not necessarily; it may be one of the newfangled Usefor "headers" and the
>"gateway" may be one operating according to the current specification which
>does not define such a "header".

Naturally, a gateway that has not been upgraded to conform to the new
standard will not conform to the new standard. Do you really expect such
tautologies to be included in IETF standards?

>  Or it may be an experimental "header", the
>syntax of which is only known to those participating in the experiment [in
>email, such expermental headers would begin with "X-", but current practice
>in Usenet articles appears to be to make up some tag without consideration
>of collisions with existing standards (e.g. Supersedes)].

Experimental headers are, by definition, those beginning with "X-". They
are always treated as unstructured by RFC 2047, which is also what my text
says.

>>  So what is the gateway
>> to do? I see four possibilities:
>> 
>> 1. It leaves it as raw 8-bit and hopes that it survives. Indeed, many mail
>> transports will pass it on untouched, but it is liable to be munged as
>> soon as it hits a Sendmail. Maybe that is a reasonable risk. Maybe not.
>> 
>> 2. It drops that particular header (presumably it was not an essential one
>> for delivering mail on the mailing list). But that would be a pity,
>> because useful information might be lost.
>> 
>> 3. It drops the article entirely. That is hardly providing a decent
>> service to the readers of the mailing list.
>> 
>> 4. It tries to encode it using RFC 2047/2231. Maybe it succeeds. Maybe it
>> doesn't. If it doesn't, at worst some representation will survive in the
>> email on the mailing list which a human might be able to decipher. At
>> best, some over-liberal user agent will decode the 2047/2231 stuff and
>> produce something sensible.
>> 
>> Now which of those four would you recommend the gatewayer to do? My text
>> recommends #4 as the least of the evils.

>You are presuming (incorrectly) that there are no other possible options.

Well you have failed to provide any others that were not already in my
text. And you have failed to answer the question. Which would you
recommend?

>The best solution would be to continue the RFC 1036 practice of using the
>Internet text message format, i.e. there would be no untagged, unencoded
>illegal octets or superfuous "parameters", 

Yes, but the Rough Consensus on the Usefor Group is not to do that.

>Failing that ideal solution, there are still other possibilities.  For
>example, the gateway could take the offending "header" (or the entire article),
>encode it using an established mechanism (e.g. base64), and package it as
>application/octet-stream.

Encoding the entire article (as an application/news-transmission) is
already one of the possibilities in my text, but sadly is not applicable
for moderators (not in the short term, anyway), and would hardly be
appreciated by the readers of the mailing lists.

Encoding a single header is fine, but if you invent Yet Another Encoding
you can hardly expect the average MUA to understand it, which is why the
suggestion was to force it into an RFC 2047 encoding. That way, there is a
reasonable chance that a MUA _Might_ manage to decode it, and if not
then you have not lost anything.

One has to face the fact that there IS no perfect solution. It is simply a
question of which will best serve the users' needs.

>Returning to your four suggestions:
>#1 would violate Internet RFCs (822 / 2822 and probably 2821), so is unacceptable.

So you have never seen a violation of RFC 2822? Lucky you! No Korean spam!

The fact is that, when faced with unpalatable choices, breaking the
standard may be the least worst thing to do. However, my text did not
actually suggest that option. But I expect a lot of gateways will
actually do it.

The other three options at least produce something compatible with RFC
2821.

>#4 presents several problems. One is that as the hypothetical header field
>    syntax is unknown, one cannot determine what to encode, or which mechanism
>    should be used.

A mechanism was suggested that will produce displayable text some of the
time, and leave the viewer looking at the undecoded text the rest of the
time. That is surely better that leaving the viewer with the undecoded text
all of the time.

>> It is however quite easy to detect when non-UTF-8 charsets have
>> been used

>  It is *not* easy to *reliably*
>detect charset when untagged, as in Usefor "headers".  One can detect that
>an octet stream is not a valid utf-8 stream, but it is possible that an
>untagged non-utf-8 octet stream may correspond to a valid utf-8 sequence
>even though it is not utf-8.

The experiment that Andrew Gierth did demonstrated a false positive rate
of 0.05% IIRC.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Arnt Gulbrandsen <[email-masked]>
To: Russ Allbery <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Cc: [email-masked]
Date: Sat, 28 Dec 2002 00:09:03 +0100
Sender: [email-masked]

Russ Allbery writes:
> Arnt Gulbrandsen <[email-masked]> writes:
>> Search.
>
> That's a better example. That's not a systemic flaw, though; you could 
> easily add a search command to NNTP from a protocol standpoint, and 
> there have been various proposals for that.

I agree.

I don't know whether the search command was in the very first IMAP 
version. I suspect not. Doesn't matter really. My point is that once 
you add the features that are sensible for reading mail, you'll have 
something roughly as complex as IMAP, and I conjecture that it would 
have roughly as many warts.

IMAP's complexity is not a fault of IMAP, it's a necessary function of 
its complex goal... or so I think. I'm open to explanations of why the 
protocol is 'baroque', however.

> I'm not adverse to the argument that IMAP is superior to NNTP for 
> reading clients in at least some situations.

Perhaps most importantly, clients for which IMAP support is necessary 
for other reasons.

1. It's possible right now. 2. It makes sense in some cases. Shouldn't 
that add up to "the protocol and message format shouldn't break it"?

> I've had that discussion with people in person before. I think the 
> IMAP protocol itself is unnecessarily complex for that particular 
> application, and I'm just not particularly fond of the protocol 
> having had to implement pieces of it,

Well, implementing IMAP sure beats implementing IMAP+NNTP ;)

> but at this point that's kind of irrelevant given the quantity of 
> software that speaks it.
>
>>  Speaking as (former) NNTP client author: Servers aren't always 
>>  well-run, users don't always direct their complaints in the right 
>>  direction, and I really, really, really hate complaints for a 
>>  problem I can't either solve or mitigate.
>
> True. But this is true of any protocol. Clients *can* locally stash 
> message IDs and use them to adjust to renumberings; they just don't 
> because renumberings shouldn't happen and are quite rare.

I had better reasons, but it's rather off-topic.

--Arnt

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 27 Dec 2002 13:31:00 -0800
Lines: 47
Sender: [email-masked]

Arnt Gulbrandsen <[email-masked]> writes:
> Russ Allbery <[email-masked]>

>> I think you need to find a better example.  NNTP has both article
>> numbers and unique identifiers.

> You mean message-id? Good point. I'd forgotten that on usenet, those are
> dependably unique.

> Search.

That's a better example.  That's not a systemic flaw, though; you could
easily add a search command to NNTP from a protocol standpoint, and there
have been various proposals for that.

The main reason why NNTP still doesn't have much search functionality is
more because of performance issues.  Maintaining decent indexes, or
opening every article and trying to do searches against it with the type
of traffic volume NNTP normally handles would be difficult from a
performance standpoint.  (But it might make a lot of sense for a text-only
feed.)

A better example is server-side state, which goes much more to the heart
of the difference between the NNTP and IMAP protocols.

I'm not adverse to the argument that IMAP is superior to NNTP for reading
clients in at least some situations.  I've had that discussion with people
in person before.  I think the IMAP protocol itself is unnecessarily
complex for that particular application, and I'm just not particularly
fond of the protocol having had to implement pieces of it, but at this
point that's kind of irrelevant given the quantity of software that speaks
it.

I think NNTP gets a lot of mindshare and neat software written for it just
because it's so simple and easy to deal with.

> Speaking as (former) NNTP client author: Servers aren't always well-run,
> users don't always direct their complaints in the right direction, and I
> really, really, really hate complaints for a problem I can't either
> solve or mitigate.

True.  But this is true of any protocol.  Clients *can* locally stash
message IDs and use them to adjust to renumberings; they just don't
because renumberings shouldn't happen and are quite rare.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 27 Dec 2002 22:04:34 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Russ Allbery <[email-masked]>
> Arnt Gulbrandsen <[email-masked]> writes:
> 
> > NNTP was really simple to understand and implement, but that simplicity
> > is bought at the expense of completeness. For example, IMAP has
> > UIDVALIDITY while NNTP has a pretense that article numbers never change.
> 
> I think you need to find a better example.  NNTP has both article numbers
> and unique identifiers.

You mean message-id? Good point. I'd forgotten that on usenet, those are
dependably unique.

Search.

> (And on a well-run server, article numbers
> *don't* ever change.)

Speaking as (former) NNTP client author: Servers aren't always well-run,
users don't always direct their complaints in the right direction, and I
really, really, really hate complaints for a problem I can't either solve
or mitigate.

--Arnt

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 27 Dec 2002 12:36:43 -0800
Lines: 12
Sender: [email-masked]

Arnt Gulbrandsen <[email-masked]> writes:

> NNTP was really simple to understand and implement, but that simplicity
> is bought at the expense of completeness. For example, IMAP has
> UIDVALIDITY while NNTP has a pretense that article numbers never change.

I think you need to find a better example.  NNTP has both article numbers
and unique identifiers.  (And on a well-run server, article numbers
*don't* ever change.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: Arnt Gulbrandsen <[email-masked]>
To: Kai Henningsen <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Cc: [email-masked]
Date: Fri, 27 Dec 2002 21:11:22 +0100
Sender: [email-masked]

Kai Henningsen writes:
> Well, I would not so much disbelieve that this is done, as much as 
> that doing this is particuölarly sane. Reading the IMAP RFC makes for 
> an interesting experience, but I wouldn't call it particularly 
> pleasant.
>
> IMAP has some useful features, yes, but the protocol that implements them
> is more than baroque.

Could you elaborate that opinion?

I've implemented both (NNTP as both client and server) and my experience 
is that IMAP is about as good as it reasonably could be. The task of 
managing e-mail is complex, and old protocols and APIs have warts - 
that's just how it is.

NNTP was really simple to understand and implement, but that simplicity 
is bought at the expense of completeness. For example, IMAP has 
UIDVALIDITY while NNTP has a pretense that article numbers never 
change.

--Arnt

From: Keith Moore <[email-masked]>
To: Leo Bicknell <[email-masked]>
cc: [email-masked]
Subject: Re: Domains to try in double bounce processing. 
Date: Wed, 25 Dec 2002 23:06:28 -0500
Sender: [email-masked]

trying to notify postmaster of anything these days is probably hopeless.
and under NO circumstances should any automatically generated notice be 
sent to any address from the message header (other than return-path)

Keith

Date: Wed, 25 Dec 2002 20:36:20 -0500
From: Leo Bicknell <[email-masked]>
To: [email-masked]
Subject: Domains to try in double bounce processing.
Organization: United Federation of Planets
Reply-By: Wed Jan  1 20:19:21 EST 2003
Sender: [email-masked]

--AhhlLboLdkugWU4S
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I'm working on updating contrib/doublebounce.pl in the sendmail
distribution.  If you have specific questions about my work there
please e-mail me off list, but I do have one on-list topic I'd like
to discuss.

A quick summary, a "double bounce" is when an e-mail bounces, and
then the bounce notification also bounces.  This is a common
occurrence for people like me, I run www.tmbg.org which offers your
choice of aliases @tmbg.org.  These are in fact just forward rules
to a users real e-mail (think also pobox.com).  Generally speaking,
the box doing the relaying must accept the mail, discover that the
actual recipient is unavailable, and then finds itself unable to
deliver the failure message.

I'm sure it will come as no surprise that many of these are spam.
This is my motivation to do two things, first get it out of my in
box and second notify someone who might be able to do something
about it.

Anyway, sendmail (and probably other MTA's) allows these messages
to go to a special place.  In my case, a perl script inspired by
contrib/doublebounce.pl.  The idea is to automatically figure out
which postmaster to notify of the problem, and send them a notice.

That is the crux of my question.  Who should be notified?  That
information must come from the header of the original message,
so the options are fairly limited.  I don't know of any RFC that
covers this specific case (but if there is one I would love to
know), so I'd like to find some community consensus.

Basically, there are two schools of thought:

  1 - Notify postmaster@ for the domains in the normal fields
      you would use to reply to a message (Sender, Reply-To,
      From, Errors-To).

  2 - Notify postmaster@ for the domains in the Received headers,
      starting with the first one the mail passed through and
      working forward until notification succeeds.

When looking at each case, you can also then apply a depth first
(eg, try foo.bar.com, then bar.com, then foo.baz.com, then baz.com)
for bredth first (foo.bar.com, foo.baz.com, bar.com, baz.com)
method.

There are drawbacks to both.  Addresses in #1 are often forged,
causing postmaster@ the forged domain to get all sorts of unwanted
notices about people using his domains.  Method #2 would seem to fix it,
but often the first postmaster@ is owned by the spammer, and is in
fact a black hole.

So, if I'm going to rewrite the script what logic should I use?
If there are any RFC's that pertain to this problem I would love
to know what they are so I can follow them, but I suspect there
are not any.  I might even be able to live with two choices of
how the script should work, if that seems like the right thing
for people to do.

Thank you for your input.

--=20
       Leo Bicknell - [email-masked] - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [email-masked], www.tmbg.org

--AhhlLboLdkugWU4S
Content-Type: application/pgp-signature
Content-Disposition: inline

--AhhlLboLdkugWU4S--

Date: 26 Dec 2002 00:02:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Russ Allbery)  wrote on 24.12.02 in <[email-masked]>:

> I and others have also tried to raise the issue of IMAP as a protocol for
> reading Usenet messages, and have been met with disbelief that anyone does
> this (even though we'd pointed out that we knew it was done) and complete
> lack of concern for the effects on such systems.

Well, I would not so much disbelieve that this is done, as much as that   
doing this is particuölarly sane. Reading the IMAP RFC makes for an  
interesting experience, but I wouldn't call it particularly pleasant.

IMAP has some useful features, yes, but the protocol that implements them  
is more than baroque.

MfG Kai

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 24 Dec 2002 15:05:49 -0800
Lines: 22
Sender: [email-masked]

Lawrence Greenfield <[email-masked]> writes:

> I think the current text on gatewaying such messages to mail is
> completely insufficient. The algorithm given in 5.5.2 won't display in
> any clients without modifications, despite those clients having UTF-8
> support.

I encourage you to raise these issues in the USEFOR working group.  I and
a few others have been trying for some months, but in a recent straw poll,
this viewpoint lost in favor of raw UTF-8 in headers, and the current
prevailing belief is that the issues are not significant enough to warrant
another course of action.

I and others have also tried to raise the issue of IMAP as a protocol for
reading Usenet messages, and have been met with disbelief that anyone does
this (even though we'd pointed out that we knew it was done) and complete
lack of concern for the effects on such systems.

At this point, I'm too discouraged to continue this aspect of the fight.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 24 Dec 2002 17:53:54 -0500
From: Lawrence Greenfield <[email-masked]>
To: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

--On Tuesday, December 24, 2002 11:10 AM -0800 Russ Allbery 
<[email-masked]> wrote:

> In any event, there is widespread feeling on the USEFOR list that failure
> to provide for Unicode newsgroup names would basically make the entire
> effort of producing a new standard pointless, and all of the solutions
> seem to have significant problems.  There seems to be a consensus that
> using raw UTF-8 in the Newsgroups header for Usenet articles is the least
> bad choice.

I think the current text on gatewaying such messages to mail is completely 
insufficient. The algorithm given in 5.5.2 won't display in any clients 
without modifications, despite those clients having UTF-8 support.

My gateway is an IMAP server which is used by clients to read netnews. IMAP 
is a superior protocol to NNTP for end client access (it has server-side 
seen state, secure authentication, and other features) and it would be nice 
not to lock this out.

The majority of e-mail clients interpret untagged 8-bit in their local 
character set, which is likely not UTF-8, so ignoring the standards is not 
a reasonable or acceptable answer. Using message/rfc822 to identify 
articles (as in 6.21.2.2) with unencoded 8-bit headers is highly harmful 
for processing messages.

Section 6.21.3 says
   Injecting and relaying agents MUST NOT change the encoding of
   articles passed to them. Gateways SHOULD NOT change the encoding
   unless absolutely necessary.

Please reconcile this with RFC 2822; you are already identifying non-RFC 
822 messages are "message/rfc822" _and_ you say that gateways shouldn't 
change this. What about clients saving messages to a local filesystem? 
Should the mime type be identified as "message/rfc822"?

Larry

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 24 Dec 2002 11:14:38 -0800
Lines: 43
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> Although the distinction does not directly affect the issue of
> transformations, technically moderation is not news->mail->news, since
> the proto-article which is mailed to the moderator is not news, as it
> has not been injected into the news system and lacks some news-specific
> headers (at least Approved).

I don't consider this a useful distinction.  The only functional
difference between the proto-article and a real Usenet article is that it
doesn't have a Path header.  It otherwise looks pretty much exactly like a
regular Usenet article, except that the process of sending it through
e-mail adds a bunch of other headers.

(We would have been much better off, in retrospect, to have always
encapsulated articles sent to moderators rather than just dumping the
article into e-mail, but hindsight is 20/20 and it's very difficult to fix
this now.)

> Email submission to a moderator does not require all of the checks that
> are necessary for news->mail gateways w.r.t. loops etc.,

This is not the case; moderated groups have pretty much exactly the same
problems as any other gateway.  If the group is hand-moderated, those
problems can be addressed by the human moderator, but for automatically
moderated groups, the same issues apply.

> and in an RFC 1036 environment there are no transformations necessary
> (any minor transformations such as changinig ':' to ": " can be
> performed at the time of injection).

This is true.

> It is only the major departure from Internet text message article format
> introduced by the Usefor draft's inclusion of untagged, unencoded utf-8
> that necessitates transformations during the process of submission for
> moderation (and in gateways).

This is not the case.  At the least, Received headers have to be removed
from the message before posting with many existing news servers.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 24 Dec 2002 11:10:29 -0800
Lines: 44
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> Another major difference between Usefor draft "headers" and Internet
> text message headers is that the former may include semicolon delimited
> parameters for some fields (e.g. Date) where the syntax and ABNF for the
> corresponding Internet text message header field do not permit such a
> construct.

I believe this is being removed.

> And there are some relatively minor differences, such as the difference
> between the delimiter marking the end of a header field tag (the single
> character ':' in Internet text messages vs. the two-character string ": "
> in Usefor "headers").

You'll find that this difference is also present in RFC 1036, so if you
feel that's sufficient to make news headers not RFC 2822 headers, that was
already the case as of RFC 1036.

You'll also find that RFC 1036 restricted the format of the From header
relative to RFC 822 (although not particularly clearly or formally).

> Not necessarily; it may be one of the newfangled Usefor "headers" and
> the "gateway" may be one operating according to the current
> specification which does not define such a "header".  Or it may be an
> experimental "header", the syntax of which is only known to those
> participating in the experiment [in email, such expermental headers
> would begin with "X-", but current practice in Usenet articles appears
> to be to make up some tag without consideration of collisions with
> existing standards (e.g. Supersedes)].

*sigh*  We already have this discussion about Supersedes.  Widespread use
of the Supersedes header in news predates any standardization of the
header in e-mail by something on the order of eight years.

In any event, there is widespread feeling on the USEFOR list that failure
to provide for Unicode newsgroup names would basically make the entire
effort of producing a new standard pointless, and all of the solutions
seem to have significant problems.  There seems to be a consensus that
using raw UTF-8 in the Newsgroups header for Usenet articles is the least
bad choice.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 24 Dec 2002 12:43:39 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: [email-masked], Charles Lindsey <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Pete Resnick wrote:
> On 12/24/02 at 9:06 AM -0500, Bruce Lilly wrote:
> 
>> There most certainly is a phrase, part of the mailbox.
> 
> 
> No, sorry, it doesn't work that way. Just because 2368 (via 1738) 
> specifies that some structured part of an encoded URL follows 822/2822 
> syntax for "mailbox" does not mean that a mail header field that 
> contains such a URL therefore contains a mailbox. The parser for 
> 822/2822/2047 does not know about URL parsing and cannot get past the 
> "%" encoded bits. There is no intervening 1738 parser.

That depends on the context. In the case of the List-Owner example and
a number of fairly widely-used MUAs, the URL is properly decoded when
using the List-Owner field content to generate a message to the list
owner, and some even decode the 2047 encoding for display.  It is
true that for transport, there is no decoding, but that's axiomatic as
2047 is clearly intended for display purposes and not for processing
during transport.

Surely you're not suggesting that correct use of the List-Owner etc. header
fields to generate messages to the list owner etc. would be to use the raw
URL %-encoded text in the headers?  It seems that you're implying that a
To header generated from a mailto URL doesn't contain a mailbox even if it
matches the mailbox syntax and although a To header generated from any other
mechanism may contain a mailbox -- that doesn't sound right either. [as an
aside, technically the mailto URL syntax in RFC 2368 should probably specify
address rather than mailbox for compatibility with the To header syntax
unless there is some particular reason to exclude named groups]

 > And as far as I
> can tell, you can't have anything in a URL in a header field that a 2047 
> parser would recognize as a phrase or a comment.

URLs can contain parentheses (which is why there's a problem with the
RFC 2557 ABNF and CFWS).  A parser based on header syntax (revised in the
case of 2557) correctly identifies the URL as a URL and not a comment.
Examples have been given in earlier messages in this thread; a simplistic
matching of header field text using regular expressions might incorrectly
match a "comment" where there is none.

Date: Tue, 24 Dec 2002 11:52:44 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Russ Allbery wrote:

> news -> mail -> news gatewaying is integral to the Usenet protocol and is
> actually as common if not more common than mail -> news -> mail
> gatewaying, since moderation is just a special case of news -> mail ->
> news gatewaying.

Although the distinction does not directly affect the issue of transformations,
technically moderation is not news->mail->news, since the proto-article
which is mailed to the moderator is not news, as it has not been injected
into the news system and lacks some news-specific headers (at least
Approved).  Also, injection by the moderator is not necessarily exactly the
same as a mail->news gateway, though a moderator might use such a gateway
for that purpose. Email submission to a moderator does not require all of the
checks that are necessary for news->mail gateways w.r.t. loops etc., and
in an RFC 1036 environment there are no transformations necessary (any
minor transformations such as changinig ':' to ": " can be performed at
the time of injection).  It is only the major departure from Internet
text message article format introduced by the Usefor draft's inclusion of
untagged, unencoded utf-8 that necessitates transformations during the
process of submission for moderation (and in gateways).

Date: Tue, 24 Dec 2002 11:37:31 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> You are really being rather disengenuous.

Not at all; the point is that -- especially when discussing gateways -- clarity
requires specifying whether "headers" refers to RFC 822 / 2822 Internet text
message headers or to something different.  Per RFC 1036, Usenet article
headers are the same as 822 headers; 1036 explicitly says so and uses the same
Internet text message format for articles.  The Usefor draft changes that; its
articles are not Internet text messages (precisely because of the presence of
untagged, unencoded characters forbidden by the Internet text message RFCs (822
and 2822) in Usefor "headers", which for the same reason are not 822 / 2822
headers even though there is a superficial similarlty).

Another major difference between Usefor draft "headers" and Internet text message
headers is that the former may include semicolon delimited parameters for some
fields (e.g. Date) where the syntax and ABNF for the corresponding Internet text
message header field do not permit such a construct.  Obviously gateways into
email would have to elide the extraneous content, though that is not mentioned
in the text which you have presented here (which is not unreasonable, as it does
not bear on RFC 2047 issues).

There is yet another major difference related to the definitions of structural
elements, mentioned below.

And there are some relatively minor differences, such as the difference between
the delimiter marking the end of a header field tag (the single character ':'
in Internet text messages vs. the two-character string ": " in Usefor "headers").

>>Reference to RFC 2047 presumes that one is dealing with RFC 822 Internet
>>text messages; if that is not the case (as seems to be w.r.t. the Usefor
>>draft), all bets are off.
> 
> 
> That is not so, because Usefor explicitly extends RFC 2047 to apply to
> Netnews (but it does not actually change anything in the 2047 protocol).

Those extensions may well need to be presented here before one can reasonably
expect a detailed critique of the bigger picture.

>>>8.8.1.1.  Gatewaying into email
>>>
> 
> 
>>>   2. If the header is unstructured, any word(s) which is contained
>>>      within a comment ...
> 
> 
>>Unstructured RFC 822 / 2822 header fields cannot be said to contain comments;
> 
> 
> Oops! Thank you for spotting my typo.

There is an example of the need for clarity; does "the header" refer to a
Usefor draft "header" or to the RFC 822 / 2822 Internet text message header
that the gateway is supposed to construct? [likewise for other parts of
the text, such as section 5 below.]  It's also unclear, at least from what
has been posted here, what the rules are for Usefor "headers". For example,
it is unclear whether or not content in a structured Usefor "header" enclosed
within U+207D and U+207E (or within U+208D and U+208E) is considered to be a
"comment" [or, for that matter, whether or not U+2474 etc. are "comments"].
Such considerations don't arise in Internet text message headers because
those characters cannot exist in headers.  But since they may exist in
Usefor "headers", the issue does arise.  And there is the more serious
matter of whether or not one is expected to use the consistent RFC 822 /
2822 / 2047 / 2231 definitions of "comment", "phrase", "quoted-string", etc.
or the quite different Usefor draft definitions (which are different because
the Usefor draft uses different definitions of "text", "ctext" "qtext", etc.).

>>>   5. If the header is not one defined by this standard or by any Email
>>>      standard known to the gateway (so that it cannot be determined
>>>      whether it is unstructured, or otherwise where comments and
>>>      phrases occur within it), then it is not possible to encode it
>>>      according to a strict interpretation of [RFC 2047].  Nevertheless,
>>>      it is preferable to attempt an encoding than to discard that
>>>      header or to allow the gatewaying to fail. It is therefore
>>>      suggested that, outside of regions contained within properly
>>>      matched DQUOTEs, <...> or [...], any word(s) contained within
>>>      properly nested "(" and ")" be treated as being within a comment
>>>      and any other word(s) be treated as being within a phrase.
>>>
>>>      Likewise, following any ";", anything of the syntactic form of a
>>>      parameter should be treated as such.
> 
> 
>>This has recently been discussed here; for display, such an error-prone heuristic
>>may be marginally acceptable (user cut-and-paste is likely to result in
>>problems), and it will fail on a number of not-uncommon constructs, but use
>>of such an unreliable mechanism for *gateways* is highly inadvisable, to say the
>>least.
> 
> 
> Well consider a gatway that gates some newsgroup to a mailing list.
> 
> It receives an article with UTF8-xtra-chars in some unrecognized header.
> For sure it was not a header known to the Netnews protocol, so it was
> superfluous for news propagation.

Not necessarily; it may be one of the newfangled Usefor "headers" and the
"gateway" may be one operating according to the current specification which
does not define such a "header".  Or it may be an experimental "header", the
syntax of which is only known to those participating in the experiment [in
email, such expermental headers would begin with "X-", but current practice
in Usenet articles appears to be to make up some tag without consideration
of collisions with existing standards (e.g. Supersedes)]. There may well be
implications for news->mail->news paths w.r.t. such experimental headers,
which might or might not be structured.

 > Maybe, however, it might be meaningful
> when seen on the mailing list (though one would expect such gateways to
> recognize at least the common email header fields). So what is the gateway
> to do? I see four possibilities:
> 
> 1. It leaves it as raw 8-bit and hopes that it survives. Indeed, many mail
> transports will pass it on untouched, but it is liable to be munged as
> soon as it hits a Sendmail. Maybe that is a reasonable risk. Maybe not.
> 
> 2. It drops that particular header (presumably it was not an essential one
> for delivering mail on the mailing list). But that would be a pity,
> because useful information might be lost.
> 
> 3. It drops the article entirely. That is hardly providing a decent
> service to the readers of the mailing list.
> 
> 4. It tries to encode it using RFC 2047/2231. Maybe it succeeds. Maybe it
> doesn't. If it doesn't, at worst some representation will survive in the
> email on the mailing list which a human might be able to decipher. At
> best, some over-liberal user agent will decode the 2047/2231 stuff and
> produce something sensible.
> 
> Now which of those four would you recommend the gatewayer to do? My text
> recommends #4 as the least of the evils.

You are presuming (incorrectly) that there are no other possible options.
The best solution would be to continue the RFC 1036 practice of using the
Internet text message format, i.e. there would be no untagged, unencoded
illegal octets or superfuous "parameters", and gateway header transformations
would ne unnecessary (though gateways may need to add or elide header fields).
Many of the other problems associated with this particular deviation from
RFC 1036 practice (viz. article format change from Internet text message to
something incompatible with RFCs 822 / 2822), several of which have been
enumerated elsewhere, would also disappear.  The entire gateway conundrum is
solely the result of deviation from that RFC 1036 practice; i.e. the change
in article format has added additional burdens to fateways above and beyond
the unrelated issues that gateways have always had to deal with.

Failing that ideal solution, there are still other possibilities.  For
example, the gateway could take the offending "header" (or the entire article),
encode it using an established mechanism (e.g. base64), and package it as
application/octet-stream.  That would preserve the content (with no possibility
of munging) and would comply with the relevant Internet RFCs for email.

There may well be other options.

Returning to your four suggestions:
#1 would violate Internet RFCs (822 / 2822 and probably 2821), so is unacceptable.
#2 would generate syntactically legal email, but might be considered inferior
    to packaging the content as described above.
#3 in fact might be a considerable service -- as you note, the offending
    article may well be Korean spam...
#4 presents several problems. One is that as the hypothetical header field
    syntax is unknown, one cannot determine what to encode, or which mechanism
    should be used.  The particular methods described in the quoted section
    5 have been specifically discussed (though at the time you stated that it
    was for display only, not for gateways).  With the exception of content
    within square brackets, the methods enumerated have been demonstrated by
    example to be unreliable (and I suspect examples involving square brackets
    could be provided, but that is unnecessary as the principle is that it is
    necessary to consider the header syntax when encoding/decoding; guessing
    based on superficial examination of characters is unreliable, particularly
    bad for gateways).

>>>   In all cases, there are additional restrictions imposed by [RFC 2047]
>>>   regarding the size, placement and contents of encoded-words which
>>>   MUST be observed.

In the hypothetical situation proposed, the "header" syntax is unknown,
ergo it is not possible to observe the RFC 2047 rules, as those rules
vary depending on whether or not a header field content is structured
or unstructured (or indeed, within the field in the case of mixed
structured/unstructured fields). Moreover, the rules for structured
fields specifically reference the structural elements, and therefore
cannot be applied if the structure is unknown.

 >>>   Moreover, these transformations MUST be applied
>>>   both within the header of the article and within any body part
>>>   headers (including the headers of any message/rfc822).

That is likely to be a problem for digitally signed messages and for
encrypted messages.  Of course, if the article uses the Internet text
message format, there's no need to apply any transformation here, and
hence no problem.

 >>>   It is
>>>   generally preferable for encodings to use the charset UTF-8, although
>>>   it might be wise first to confirm that that is indeed the charset
>>>   which had been used (see 4.4.1).
> 
> 
>>That raises the issue of round-trip gateways; what mechanism is used to convey
>>charset (and language) information through a mail -> Usefor -> mail transformation
>>to ensure that the charset (and language) information is maintained?  If no
>>such mechanism is provided, one cannot "confirm that that is [...] the charset
>>which had been used".
> 
> 
> If the chain is mail->news->mail, then there should be no 8bit stuff in
> headers anyway, so the problem does not arise.

You are presuming (probably incorrectly) that there will never exist any
mail->news gateway designed and/or operated by an 8-bit thug to transform
tagged and encoded email header content into untagged raw utf-8 Usefor
"header" content.  Bear in mind that the mail->news and news->mail
gateways may be operated by different entities (e.g. one part may involve
submission to a moderator via email).

> The nasty case is the chain news -> mail -> news. That sort of gatewaying
> is _always_ difficult. There are all sorts of things that can go wrong,
> and Usefor warns against them. If the 8bit stuff in the headers is all in
> UTF-8, then it should work provided newsgroup-names are restored to their
> canonical form (that is a MUST).

The hypothetical unrecognized header will still be a problem so long as
there is any transformation involved.  Bearing in mind once again that
the news->mail and mail-> news trasformation may be done at different
places by different software based on different assumptions by different
authors.  There are roughly two types of potential problems; those related
to content transformations and those related to news-specific issues (loops,
etc.). The transformation issues essentially go away if the Internet text
message format is used for articles, as in RFC 1036.

> It is however quite easy to detect when non-UTF-8 charsets have
> been used

It is trivial in the Internet text message format, since charset is explicitly
tagged (as is language, where relevant).  It is *not* easy to *reliably*
detect charset when untagged, as in Usefor "headers".  One can detect that
an octet stream is not a valid utf-8 stream, but it is possible that an
untagged non-utf-8 octet stream may correspond to a valid utf-8 sequence
even though it is not utf-8.

Date: Tue, 24 Dec 2002 08:27:37 -0600
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Cc: Charles Lindsey <[email-masked]>, [email-masked], Bruce Lilly <[email-masked]>
Sender: [email-masked]

On 12/24/02 at 9:06 AM -0500, Bruce Lilly wrote:

>There most certainly is a phrase, part of the mailbox.

No, sorry, it doesn't work that way. Just because 2368 (via 1738) 
specifies that some structured part of an encoded URL follows 
822/2822 syntax for "mailbox" does not mean that a mail header field 
that contains such a URL therefore contains a mailbox. The parser for 
822/2822/2047 does not know about URL parsing and cannot get past the 
"%" encoded bits. There is no intervening 1738 parser. And as far as 
I can tell, you can't have anything in a URL in a header field that a 
2047 parser would recognize as a phrase or a comment.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Tue, 24 Dec 2002 09:06:37 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>   List-Owner: <mailto:[email-masked]%3E?Subject=list>
[...]
>>As mentioned in the earlier message, it's URI-encoded as required by
>>RFC 2368 (section 2). Depending on UA, you may be able to see it
>>simply by positioning your cursor over the mailto URI (at least
>>that works with Mozilla 1.2.1).
> 
> 
> Yes, but if the URI is decoded URI-wise, then I see nothing which suggests
> that it should then be further subjected to RFC 2047 decoding. If it is a
> structured header (I guess List-Owner is structured) then RFC 2047
> instructs you to identify comments and phrases within it. In your example
> there are none.

There most certainly is a phrase, part of the mailbox.  Please read RFC 2368
section 2 as referenced above. Here is a pertinent excerpt:

    Following the syntax conventions of RFC 1738 [RFC1738], a "mailto"
    URL has the form:

      mailtoURL  =  "mailto:" [ to ] [ headers ]
      to         =  #mailbox
      headers    =  "?" header *( "&" header )
      header     =  hname "=" hvalue
      hname      =  *urlc
      hvalue     =  *urlc

    "#mailbox" is as specified in RFC 822 [RFC822]. This means that it
    consists of zero or more comma-separated mail addresses, possibly
    including "phrase" and "comment" components.

Please note the last part of the last sentence quoted above!

RFC 2369 section 2 gives the syntax (but no BNF) for List- headers,
which are unquestionably structured [a comma-separated list (structure)
of angle bracketed (more structure) URIs (yet more structure)]. Section
3.5 gives List-Owner examples, all of which have comments.

And of course RFC 822 gives the formal mailbox ABNF:

      mailbox     =  addr-spec                    ; simple address
                  /  phrase route-addr            ; name & addr-spec

                     ^^^^^^

RFC 2047 does apply to a phrase within a structured header (technically,
there should probably be another %20 between the colon and the first
%3D in my example).

The point is that (via mailto URLs at least) phrases and/or comments
which could potentially contain RFC 2047 encoded-words may appear
within angle brackets (as in the RFC 2369 List- headers), so your
suggestion of ignoring anything within angle brackets when looking
for potential encoded-words leads to failure to recognize encoded-words.
That might be acceptable for display (though it completely misses the
point of RFC 2047 encoding in the first place, which is precisely for
display), but it could lead to gross errors if such a scheme were used
in a gateway.

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 23 Dec 2002 19:30:44 -0800
Lines: 15
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> The nasty case is the chain news -> mail -> news. That sort of
> gatewaying is _always_ difficult. There are all sorts of things that can
> go wrong, and Usefor warns against them. If the 8bit stuff in the
> headers is all in UTF-8, then it should work provided newsgroup-names
> are restored to their canonical form (that is a MUST).

news -> mail -> news gatewaying is integral to the Usenet protocol and is
actually as common if not more common than mail -> news -> mail
gatewaying, since moderation is just a special case of news -> mail ->
news gatewaying.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Xref: clerew local.mime:1985
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Date: Mon, 23 Dec 2002 15:52:10 GMT
Lines: 152
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> In Usefor, where headers in Netnews will be in UTF-8, we have to say how
>> to gateway into email.

>"Headers" is somewhat ambiguous; HTML has "headers", but those are not
>Internet text message (RFC 822 / 2822) headers.  RFC 822 / 2822 preclude
>8th-bit-high octets implied by UTF-8 in Internet text message header fields.
>So presumably the new Usefor format is completely separate from RFC 822
>Internet text message format, unlike RFC 1036, which used the same message
>format as RFC 822 Internet text messages, and "headers" w.r.t. the Usefor
>draft is presumably not the same as "headers" in RFC 822 / 2822 Internet
>text messages (unlike RFC 1036 Usenet messages, where they were the same
>except for some additional restrictions imposed by 1036).

You are really being rather disengenuous. You know perfectly well (because
you have been on the Usefor list) that "Headers", as defined in Usefor, are
broadly similar to those in Email, the major difference being that UTF-8
is allowed in certain places. In order to ensure that gatewaying from news
to mail can proceed, it is necessary to specify transformations that need
to be made at the boundary. The text I posted covers that, and was posted
here for comment because if makes use of RFC 2047/2231 to do most of the
work.

>Reference to RFC 2047 presumes that one is dealing with RFC 822 Internet
>text messages; if that is not the case (as seems to be w.r.t. the Usefor
>draft), all bets are off.

That is not so, because Usefor explicitly extends RFC 2047 to apply to
Netnews (but it does not actually change anything in the 2047 protocol).

>> Your comments would be appreciated.
>> 
>> 8.8.1.1.  Gatewaying into email
>> 

>>    2. If the header is unstructured, any word(s) which is contained
>>       within a comment ...

>Unstructured RFC 822 / 2822 header fields cannot be said to contain comments;

Oops! Thank you for spotting my typo.

>>    5. If the header is not one defined by this standard or by any Email
>>       standard known to the gateway (so that it cannot be determined
>>       whether it is unstructured, or otherwise where comments and
>>       phrases occur within it), then it is not possible to encode it
>>       according to a strict interpretation of [RFC 2047].  Nevertheless,
>>       it is preferable to attempt an encoding than to discard that
>>       header or to allow the gatewaying to fail. It is therefore
>>       suggested that, outside of regions contained within properly
>>       matched DQUOTEs, <...> or [...], any word(s) contained within
>>       properly nested "(" and ")" be treated as being within a comment
>>       and any other word(s) be treated as being within a phrase.
>> 
>>       Likewise, following any ";", anything of the syntactic form of a
>>       parameter should be treated as such.

>This has recently been discussed here; for display, such an error-prone heuristic
>may be marginally acceptable (user cut-and-paste is likely to result in
>problems), and it will fail on a number of not-uncommon constructs, but use
>of such an unreliable mechanism for *gateways* is highly inadvisable, to say the
>least.

Well consider a gatway that gates some newsgroup to a mailing list.

It receives an article with UTF8-xtra-chars in some unrecognized header.
For sure it was not a header known to the Netnews protocol, so it was
superfluous for news propagation. Maybe, however, it might be meaningful
when seen on the mailing list (though one would expect such gateways to
recognize at least the common email header fields). So what is the gateway
to do? I see four possibilities:

1. It leaves it as raw 8-bit and hopes that it survives. Indeed, many mail
transports will pass it on untouched, but it is liable to be munged as
soon as it hits a Sendmail. Maybe that is a reasonable risk. Maybe not.

2. It drops that particular header (presumably it was not an essential one
for delivering mail on the mailing list). But that would be a pity,
because useful information might be lost.

3. It drops the article entirely. That is hardly providing a decent
service to the readers of the mailing list.

4. It tries to encode it using RFC 2047/2231. Maybe it succeeds. Maybe it
doesn't. If it doesn't, at worst some representation will survive in the
email on the mailing list which a human might be able to decipher. At
best, some over-liberal user agent will decode the 2047/2231 stuff and
produce something sensible.

Now which of those four would you recommend the gatewayer to do? My text
recommends #4 as the least of the evils.

>>    In all cases, there are additional restrictions imposed by [RFC 2047]
>>    regarding the size, placement and contents of encoded-words which
>>    MUST be observed. Moreover, these transformations MUST be applied
>>    both within the header of the article and within any body part
>>    headers (including the headers of any message/rfc822).  It is
>>    generally preferable for encodings to use the charset UTF-8, although
>>    it might be wise first to confirm that that is indeed the charset
>>    which had been used (see 4.4.1).

>That raises the issue of round-trip gateways; what mechanism is used to convey
>charset (and language) information through a mail -> Usefor -> mail transformation
>to ensure that the charset (and language) information is maintained?  If no
>such mechanism is provided, one cannot "confirm that that is [...] the charset
>which had been used".

If the chain is mail->news->mail, then there should be no 8bit stuff in
headers anyway, so the problem does not arise.

The nasty case is the chain news -> mail -> news. That sort of gatewaying
is _always_ difficult. There are all sorts of things that can go wrong,
and Usefor warns against them. If the 8bit stuff in the headers is all in
UTF-8, then it should work provided newsgroup-names are restored to their
canonical form (that is a MUST).

If the charset used is not UFTF-8, then the article is not-compliant with
Usefor (nor with any mail standard), in which case all bets are off.

Sadly, such non-compliance does not stop people from doing it (there is a
considerable amount of both Netnews and Email floating around with strange
oriental charsets in the headers - mostly, it must be admitted, Korean
spam). It is however quite easy to detect when non-UTF-8 charsets have
been used, and if systems detect that and are able to work out what to do,
then good luck to them. Usefor draws attention to that possibility, but
does not condone it. It applies equally to Netnews and Email, hence the
remark about confirming the charset. 

>Of course, with RFC 1036, which uses RFC 822 Internet text messages, one would
>simply use the Internet text message mechanisms, including RFC 2047, ...

>This has been discussed on the Usefor mailing list; 

It has indeed. However, Usefor has recently come to a "Rough Cnsensus" on
how to proceed, and the text we are discussing arises from that consensus.
As I said, it is posted here for the RFC 2047/2231 experts to check it
over. It is also, of course, being discussed on the Usefor list.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1986
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Mon, 23 Dec 2002 16:09:06 GMT
Lines: 42
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>>Excluding content bracketed in <> would also be an error. Consider RFCs
>>>2368 and 2369 (not to be confused with 2396, which is also applicable) and:
>> 
>> 

>    List-Owner: <mailto:[email-masked]%3E?Subject=list>

>>>That does contain an RFC2047 encoded-word within the <>.
>> 
>> 
>> Eh? I don't see one.

>As mentioned in the earlier message, it's URI-encoded as required by
>RFC 2368 (section 2). Depending on UA, you may be able to see it
>simply by positioning your cursor over the mailto URI (at least
>that works with Mozilla 1.2.1).

Yes, but if the URI is decoded URI-wise, then I see nothing which suggests
that it should then be further subjected to RFC 2047 decoding. If it is a
structured header (I guess List-Owner is structured) then RFC 2047
instructs you to identify comments and phrases within it. In your example
there are none. End of story, so far as RFC 2047 is concerned. You now
hand it over to the URI department, which may or may not be able to find a
valid mailing address within it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Sun, 22 Dec 2002 04:40:29 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Charles Lindsey wrote:

> In Usefor, where headers in Netnews will be in UTF-8, we have to say how
> to gateway into email.

"Headers" is somewhat ambiguous; HTML has "headers", but those are not
Internet text message (RFC 822 / 2822) headers.  RFC 822 / 2822 preclude
8th-bit-high octets implied by UTF-8 in Internet text message header fields.
So presumably the new Usefor format is completely separate from RFC 822
Internet text message format, unlike RFC 1036, which used the same message
format as RFC 822 Internet text messages, and "headers" w.r.t. the Usefor
draft is presumably not the same as "headers" in RFC 822 / 2822 Internet
text messages (unlike RFC 1036 Usenet messages, where they were the same
except for some additional restrictions imposed by 1036).

> So I am here showing you the text that is currently proposed, primarily
> because I want the RFC 2047 experts here to check that what I say is
> correct, or at least as correct as it is possible tom be with RFC 2047.

Reference to RFC 2047 presumes that one is dealing with RFC 822 Internet
text messages; if that is not the case (as seems to be w.r.t. the Usefor
draft), all bets are off.

> Your comments would be appreciated.
> 
> 8.8.1.1.  Gatewaying into email
> 
>    Although headers containing non-ASCII characters may well be conveyed

"headers containing non-ASCII characters" in the context of Internet text
messages (RFC 822 / 2822) is analogous to "cows with testicles" or "bulls
with udders"; there's no such critter.

[...]

>    2. If the header is unstructured, any word(s) which is contained
>       within a comment and  is delimited by FWS or by the "(" or ")"
>       delimiting that comment

Unstructured RFC 822 / 2822 header fields cannot be said to contain comments;
comments imply structure, and cannot exist in an unstructured field (or in
unstructured parts of mixed structured / unstructured fields). In
RFC 822 / 2822 Internet text messages, the Subject header field is
unstructured;

    Subject: foo (bar)

does not contain a comment, because the header field is unstructured.

    Date: Fri, 13 Dec 2002 12:34:56 -0000 (no triskadekaphobia here)

is a structured RFC 822 / 2822 header field which contains a comment.
If Usefor draft "header" means something different from RFC 822 / 2822
"header", then whether or not an "unstructured" Usefor "header" can be
said to contain a "comment" depends strongly on the Usefor definitions
of "header", "unstructured", and "comment".

[...]

>    5. If the header is not one defined by this standard or by any Email
>       standard known to the gateway (so that it cannot be determined
>       whether it is unstructured, or otherwise where comments and
>       phrases occur within it), then it is not possible to encode it
>       according to a strict interpretation of [RFC 2047].  Nevertheless,
>       it is preferable to attempt an encoding than to discard that
>       header or to allow the gatewaying to fail. It is therefore
>       suggested that, outside of regions contained within properly
>       matched DQUOTEs, <...> or [...], any word(s) contained within
>       properly nested "(" and ")" be treated as being within a comment
>       and any other word(s) be treated as being within a phrase.
> 
>       Likewise, following any ";", anything of the syntactic form of a
>       parameter should be treated as such.

This has recently been discussed here; for display, such an error-prone heuristic
may be marginally acceptable (user cut-and-paste is likely to result in
problems), and it will fail on a number of not-uncommon constructs, but use
of such an unreliable mechanism for *gateways* is highly inadvisable, to say the
least.

>    In all cases, there are additional restrictions imposed by [RFC 2047]
>    regarding the size, placement and contents of encoded-words which
>    MUST be observed. Moreover, these transformations MUST be applied
>    both within the header of the article and within any body part
>    headers (including the headers of any message/rfc822).  It is
>    generally preferable for encodings to use the charset UTF-8, although
>    it might be wise first to confirm that that is indeed the charset
>    which had been used (see 4.4.1).

That raises the issue of round-trip gateways; what mechanism is used to convey
charset (and language) information through a mail -> Usefor -> mail transformation
to ensure that the charset (and language) information is maintained?  If no
such mechanism is provided, one cannot "confirm that that is [...] the charset
which had been used".

Of course, with RFC 1036, which uses RFC 822 Internet text messages, one would
simply use the Internet text message mechanisms, including RFC 2047, verbatim
and there would be no problem. The problems with the Usefor draft proposal all
relate to the attempt to force untagged (w.r.t. both charset and language) raw
utf-8 into a message format which has evolved to use charset tags and a robust
encoding mechanism for anything other than the historical default 7-bit ascii
charset, and which provides for language tags as well (since charset is
insufficient to provide language context).

This has been discussed on the Usefor mailing list; RFC 1036 has the considerable
advantage of using the same Internet text message format as email -- making
combined mail/news user agents and gateways practical.  The news part of such
combined applications and gateways merely need to be able to deal with news-specific
concepts such as newsgroups, distributions, and follow-ups.  If the common Internet
text message format is maintained (viz. no unencoded 8-bit-high octets), that can
continue to be the case.  Conversely, if the new Usefor format is to deviate from
822 / 2822, gateways become considerably more difficult to implement, one cannot
strictly apply RFC 2047 / 2231 directly (as those are based on 822), combined
mail/news user agents become impractical, etc.

Date: Sat, 21 Dec 2002 17:33:18 +0100
From: Marc Mutz <[email-masked]>
Subject: Re: Interpretation of RFC 2047
To: [email-masked], Bruce Lilly <[email-masked]>
Organization: KDE
Sender: [email-masked]

--Boundary-02=_bfJB+6Go7LW+BFa
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Saturday 21 December 2002 00:45, Bruce Lilly wrote:
<snip>
> Kmail 1.4.3: hovering over the link simply displays it verbatim on
> the status line.

1.5.0rc5 shows "mailto:J=FCrgen <[email-masked]>?Subject=3Dlist"

> Clicking opens a composition window which gets the
> To header correct (and 2047-encoding decoded for display), but leaves
> the Subject field blank.

=46ixed locally. Will hopefully yet go into 1.5.0.

Thanks,
Marc

=2D-=20
Ein Grundrecht auf Sicherheit steht bewusst nicht in der Verfassung.
  -- Sabine Leutheusser-Schnarrenberger (ehem. Bundesjustizministerin)

--Boundary-02=_bfJB+6Go7LW+BFa
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_bfJB+6Go7LW+BFa--

Date: Fri, 20 Dec 2002 18:45:30 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>>   To: empty-list:;, =?iso-8859-1?Q?J=FCrgen?= [email-masked]

I made an error; that should be

   To: empty-list:;, =?iso-8859-1?Q?J=FCrgen?= <[email-masked]>

> Yes, groups in addresses are a pain

It's not a problem for a grammar-based parser, nor for relatively simple
parsing (the group addresses are bracketed by the colon/semicolon
pair).  Even a regular expression could be crafted to handle groups.

>>Excluding content bracketed in <> would also be an error. Consider RFCs
>>2368 and 2369 (not to be confused with 2396, which is also applicable) and:
> 
> 
>>   List-Owner: <mailto:[email-masked]?Subject=list>

That also is an error, I meant

    List-Owner: <mailto:[email-masked]%3E?Subject=list>

>>That does contain an RFC2047 encoded-word within the <>.
> 
> 
> Eh? I don't see one.

As mentioned in the earlier message, it's URI-encoded as required by
RFC 2368 (section 2). Depending on UA, you may be able to see it
simply by positioning your cursor over the mailto URI (at least
that works with Mozilla 1.2.1).

Somewhat off-topic digression: I checked handling of that in a few MUAs.
Here's what happens:

Mozilla 1.2.1: positioning the cursor over the URI displays the URI-
decoded version on the status line.  Clicking on the URI opens a
message composition window which seems to do the right things (mailbox
appears as To header, display name is decoded for display, subject
header is correct).

Evolution 1.2.1: right-clicking on the URI sort-of displays the URI-
decoded version (the angle brackets around the addr-spec are missing).
Clicking on the URI opens a composition window with a syntactically
and semantically correct To header, but the 2047 encoding is not
decoded for display. The Subject header is correct.

Kmail 1.4.3: hovering over the link simply displays it verbatim on
the status line. Clicking opens a composition window which gets the
To header correct (and 2047-encoding decoded for display), but leaves
the Subject field blank.

Microsoft Outlook Express: hovering over the URI displays the URI-
decoded version on the status line. Clicking on it opens a
composition window, but the To header is botched pretty badly.

Date: Fri, 20 Dec 2002 21:29:59 +0000 (GMT)
From: Charles Lindsey <[email-masked]>
Reply-To: Charles Lindsey <[email-masked]>
Subject: RFC 2047 and gatewaying
To: [email-masked]
Content-MD5: AqaaMNkdQZ3Q3bi74sAxvA==
Sender: [email-masked]

There has been a lot of discussion of RFC 2047 in recent days. Please
bear with me while I introduce a little more.

In Usefor, where headers in Netnews will be in UTF-8, we have to say how
to gateway into email. Whereas a random gateway set up for a specific
purpose can do pretty well whatever it likes (and its customers can put
up with), gateways that generate mail that is to be readable anywhere
have to be more careful. This applies especially when articles are both
posted and emailed, and when mailing articles to moderators.

So I am here showing you the text that is currently proposed, primarily
because I want the RFC 2047 experts here to check that what I say is
correct, or at least as correct as it is possible tom be with RFC 2047.

Your comments would be appreciated.

8.8.1.1.  Gatewaying into email

   Although headers containing non-ASCII characters may well be conveyed
   intact by many (if not most) current mail transport agents, that
   ability is not a requirement of some transport protocols, notably of
   SMTP [RFC 2821].  Likewise, although many mail user agents may
   currently display (or be configurable to display) such headers
   correctly, or at least adequately, messages containing such headers
   are not compliant with the current Email standards, notably with [RFC
   2822].  Note that non-ASCII body part headers [RFC 2046] (including
   non-ASCII headers of a message/rfc822) are equally at variance with
   the current Email standards.

   If, at some future time, the Email standards should be updated so as
   to allow such headers, it would then become possible to transport
   Netnews articles containing them over Email without further ado.
   Until such a time, however, if a Netnews article is to be gatewayed
   into Email with the intention that it be received and accepted by any
   arbitrarily chosen destination, and if it contains any UTF8-xtra-char
   in any of its headers or body part headers, then it MUST first be
   transformed so as to conform to [RFC 2822] and/or [RFC 2046].  In
   particular, articles emailed to moderators (8.2.2) MUST be so
   transformed.

        NOTE: It is not precluded that a gatewayer who knows, or is able
        to control, the capabilities of the particular sites for which
        an article is destined and of the transport paths leading to
        those sites, may choose to send the article without
        transformation, or at least without transformation of any
        contained body part headers.

   The surest way to transport an article containing non-ASCII headers
   through Email is by encapsulation as an application/news-transmission
   (6.21.6.1).  However this method is not currently available for
   sending to moderators for reason explained in section 8.2.2 step 12.
   Until this method is considered safe to use, therefore,
   transformation of those headers will be necessary. This can be
   accomplished in the following steps:

   1. If the header is unstructured, or is an experimental header
      (4.2.5.1), any word(s) which is delimited by FWS or by the
      start/end of the header-content is encoded according to [RFC
      2047].

   2. If the header is unstructured, any word(s) which is contained
      within a comment and  is delimited by FWS or by the "(" or ")"
      delimiting that comment is encoded according to [RFC 2047], and
      likewise any word(s) which is contained within a phrase and is
      delimited by FWS or by the start/end of the header-content.

   3. If the header contains a (MIME-style) parameter with a non-ASCII
      value, the whole parameter is encoded according to [RFC 2231].

   4. If the header is a Newsgroups-header or a Followup-To-header (or
      any other header that contains a newsgroup-name), each newsgroup-
      name is encoded according to section 5.5.2.  Even if it is not
      decoded at the far end, it is preferable to display that encoded
      form than to display nothing at all. Note, however, that such
      encoded newsgroup-names MUST be restored to their canonical form
      before reinjection into any Netnews system.

   5. If the header is not one defined by this standard or by any Email
      standard known to the gateway (so that it cannot be determined
      whether it is unstructured, or otherwise where comments and
      phrases occur within it), then it is not possible to encode it
      according to a strict interpretation of [RFC 2047].  Nevertheless,
      it is preferable to attempt an encoding than to discard that
      header or to allow the gatewaying to fail. It is therefore
      suggested that, outside of regions contained within properly
      matched DQUOTEs, <...> or [...], any word(s) contained within
      properly nested "(" and ")" be treated as being within a comment
      and any other word(s) be treated as being within a phrase.

      Likewise, following any ";", anything of the syntactic form of a
      parameter should be treated as such.

   In all cases, there are additional restrictions imposed by [RFC 2047]
   regarding the size, placement and contents of encoded-words which
   MUST be observed. Moreover, these transformations MUST be applied
   both within the header of the article and within any body part
   headers (including the headers of any message/rfc822).  It is
   generally preferable for encodings to use the charset UTF-8, although
   it might be wise first to confirm that that is indeed the charset
   which had been used (see 4.4.1).

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 20 Dec 2002 01:20:14 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Keith Moore wrote:

> for the purpose of this discussion the relevant point is that for DSNs, 
> encoded-words are only recognized within comments, per the language in 2047.

OK.

> one important difference between some 1894 fields and 822 fields is that
> several 1894 fields consist of a structured portion followed by a
> non-structured portion. 

As noted in my response to Alan Barrett's query regarding parentheses
not indicating comments; the non-structured part can't be said to contain
comments.

> I'd forgotten that in 1894 I had permitted use of encoded-words in comments.  

Thanks for the clarification.

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: Keith Moore <[email-masked]>
Subject: Re: Interpretation of RFC 2047 
Date: Thu, 19 Dec 2002 23:51:49 -0500
Sender: [email-masked]

> Distinction noted, but that doesn't clarify matters. 1894 also states:
> 
>     Since these fields are defined according to the rules of RFC 822, the
>     same conventions for continuation lines and comments apply.
>     Notification fields may be continued onto multiple lines by beginning
>     each additional line with a SPACE or HTAB.  Text which appears in
>     parentheses is considered a comment and not part of the contents of
>     that notification field.  Field names are case-insensitive, so the
>     names of notification fields may be spelled in any combination of
>     upper and lower case letters.  Comments in DSN fields may use the
>     "encoded-word" construct defined in [7].
> 
> Reference 7 is RFC 1522, an earlier part of the MIME RFCs (the current
> set includes 2047, which is the part dealing with encoded words).

but 2047 is the authoritative definition here, not 822.

> > As author of 3 of the 4 DSN RFCs I can confidently state that 
> > the structuring conventions of RFC 822 header fields were not intended 
> > to apply to DSNs fields.
> 
> In light of the above quote from section 2.1.1 of RFC 1894, could you
> please clarify which parts of 822 header fields structuring conventions
> do not apply to the DSN fields; certainly the ABNF is used, and 2.1.1
> refers to continuation lines and comments.  

for the purpose of this discussion the relevant point is that for DSNs, 
encoded-words are only recognized within comments, per the language in 2047.

one important difference between some 1894 fields and 822 fields is that
several 1894 fields consist of a structured portion followed by a
non-structured portion. 

> > And as author of RFC 2047 I can confidently
> > state that encoded-words are not intended to be used or interpreted in 
> > message bodies  (except for message bodies that by their definition
> > contain 822 messages or message headers)
> 
> That would seem to be a repudiation of 1894's reference to 1522
> specifically mentioning encoded-words.  It seems that I must be
> missing something terribly important or there's a contradiction
> between what you and G.V. wrote in 1894 section 2.1.1 and what you
> now say w.r.t. encoded words and DSN fields -- have I misunderstood?

I'd forgotten that in 1894 I had permitted use of encoded-words in comments.  

Keith

Date: Thu, 19 Dec 2002 23:45:51 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Keith Moore wrote:
>>MDNs and DSNs are rather unusual MIME multipart messages. The fields
>>(N.B. they are described as fields in those RFCs) are in fact defined
>>using RFC 822 syntax (both RFCs predate RFC 2822):
> 
>  
> no, they are defined using ABNF notation, which is not the same thing.

Distinction noted, but that doesn't clarify matters. 1894 also states:

    Since these fields are defined according to the rules of RFC 822, the
    same conventions for continuation lines and comments apply.
    Notification fields may be continued onto multiple lines by beginning
    each additional line with a SPACE or HTAB.  Text which appears in
    parentheses is considered a comment and not part of the contents of
    that notification field.  Field names are case-insensitive, so the
    names of notification fields may be spelled in any combination of
    upper and lower case letters.  Comments in DSN fields may use the
    "encoded-word" construct defined in [7].

Reference 7 is RFC 1522, an earlier part of the MIME RFCs (the current
set includes 2047, which is the part dealing with encoded words).

> As author of 3 of the 4 DSN RFCs I can confidently state that 
> the structuring conventions of RFC 822 header fields were not intended 
> to apply to DSNs fields.

In light of the above quote from section 2.1.1 of RFC 1894, could you
please clarify which parts of 822 header fields structuring conventions
do not apply to the DSN fields; certainly the ABNF is used, and 2.1.1
refers to continuation lines and comments.  Perhaps whitespace may not
be freely inserted between lexical tokens?  From a practical point of
view in parsing header fields and parsing DSN fields, what are the
substantive differences?

 > And as author of RFC 2047 I can confidently
> state that encoded-words are not intended to be used or interpreted in 
> message bodies  (except for message bodies that by their definition
> contain 822 messages or message headers)

That would seem to be a repudiation of 1894's reference to 1522
specifically mentioning encoded-words.  It seems that I must be
missing something terribly important or there's a contradiction
between what you and G.V. wrote in 1894 section 2.1.1 and what you
now say w.r.t. encoded words and DSN fields -- have I misunderstood?

To: [email-masked]
Xref: clerew local.mime:1974
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Thu, 19 Dec 2002 19:20:05 GMT
Lines: 92
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> I think a reasonable heuristic, which would nearly always do the "right
>> thing" would be:
>> 

>As header field contents are defined by a grammar, attempts to
>decode using only regular expressions (as opposed to a parser
>which accepts the defined grammar) are doomed to failure. Failures
>include both false positives and false negatives, as illustrated
>below.

Sure there will be failures. What I said was 'a reasonable heuristic,
which would nearly always do the "right thing"'. The question is what is
the probability of hitting one of those awkward cases (and of finding a
recognizable "=?...?...?=" within it), as opposed to the probability of
finding a "=?...?...?=" in a genuine comment or phrase. My guess (which
seems to be supported by others in this thread) is that it will do the
"right thing" far far more often than it will do the "wrong thing".

Remember that we are merely talking about the display of headers the agent
does not know about. So it will be taking no semantic action, just letting
the reader read it.

And realize also that many (maybe most) current agents already recognize
many of those extra-2047 cases simply because their implementors couldn't
be bothered to parse every obscure header that might turn up. And by and
large those agents do an acceptable job (except that they all work to
slightly different heuristics).

>One cannot recognise a comment unless the header field syntax is known.

Well there seems to be some disagreement about that :-( .

>    Foobar: http://users.erols.com/blilly/mailparse/(=?us-ascii?q?=3D?=)

>Either could just as well be a Content-Location header. Both would be
>attempted to be decoded using a simple regular expression matching
>heuristic.  If thus inappropriately decoded, they would yield

>    Foobar: file:=
>    Foobar: http://users.erols.com/blilly/mailparse/=

>which are clearly not what was intended.

And what do you suppose _was_ intended. Nobody in the real world is going
to write a real URI like that. The only people who write that sort of
stuff are spammers who are trying to obfuscate things, and if a few of
their obfuscations get misinterpreted, that might even be regarded as a
Good Thing :-) .

>treating the header as unstructured may result in decoding for display.
>Gateways should not attempt to transform unrecognized header fields;

Exactly so. I never suggested otherwise.

>Excluding content after a semicolon would fail to decode the RFC 2047
>encoded-word in the header

>    To: empty-list:;, =?iso-8859-1?Q?J=FCrgen?= [email-masked]

Yes, groups in addresses are a pain, which is one of the reasons for
disallowing MIME-style parameters in some headers in Usefor. So it is more
work than just ignoring stuff after a ";", but probably still with the
capabilities of REs.

>Excluding content bracketed in <> would also be an error. Consider RFCs
>2368 and 2369 (not to be confused with 2396, which is also applicable) and:

>    List-Owner: <mailto:[email-masked]?Subject=list>

>That does contain an RFC2047 encoded-word within the <>.

Eh? I don't see one.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1975
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Thu, 19 Dec 2002 19:21:28 GMT
Lines: 18
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>these are not message header fields; they are part of specific
>content-types.  surely you don't think that the 822 syntax should
>be applied to message bodies also?

I would expect it to be applied to body part headers.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: Keith Moore <[email-masked]>
Subject: Re: Interpretation of RFC 2047 
Date: Thu, 19 Dec 2002 12:40:38 -0500
Sender: [email-masked]

> MDNs and DSNs are rather unusual MIME multipart messages. The fields
> (N.B. they are described as fields in those RFCs) are in fact defined
> using RFC 822 syntax (both RFCs predate RFC 2822):
 
no, they are defined using ABNF notation, which is not the same thing.

As author of 3 of the 4 DSN RFCs I can confidently state that 
the structuring conventions of RFC 822 header fields were not intended 
to apply to DSNs fields.  And as author of RFC 2047 I can confidently 
state that encoded-words are not intended to be used or interpreted in 
message bodies  (except for message bodies that by their definition
contain 822 messages or message headers)

Keith

Date: Thu, 19 Dec 2002 12:34:35 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked], Alan Barrett <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Keith Moore wrote:
>> That includes RFC 1894 Original-Envelope-ID,
>>Reporting-MTA, DSN-Gateway, Received-From-MTA, Remote-MTA,
>>Original-Recipient, Final-Recipient; RFC 2298 Diagnostic-Code,
>>MDN-Gateway, and probably Reporting-UA (there is an ambiguity in the
>>Reporting-UA ABNF, but that does not affect this issue).
> 
> 
> these are not message header fields; they are part of specific
> content-types.  surely you don't think that the 822 syntax should
> be applied to message bodies also?

RFC 822 does give a syntax for message bodies, though not as succinctly
as 2822 section 3.5.  But that's not the point, and I suspect wasn't
what your question was addressing.

MDNs and DSNs are rather unusual MIME multipart messages. The fields
(N.B. they are described as fields in those RFCs) are in fact defined
using RFC 822 syntax (both RFCs predate RFC 2822):

1894:

    The body of a message/delivery-status consists of one or more
    "fields" formatted according to the ABNF of RFC 822 header "fields"
    (see [6]).

2298:

    The body of a message/disposition-notification consists of one or
    more "fields" formatted according to the ABNF of RFC 822 header
    "fields" (see [2]).

So the short answer is that in the specific case of MDN and DSN
content in the second part of the multipart/report (viz. the
message/delivery-status or message/disposition notification), RFC
822 syntax is applicable[*].  Given a hypothetical parser capable
of parsing RFC 822 headers, that parser could be capable of parsing
those MDN and DSN fields (excepting the Reporting-UA ambiguity).

Regarding the issue of identifying comments, the 822 rules for
comments would be applicable -- clearly one can't have a comment per
se in the parts of those fields which are specified as *text, but
comments might appear next to parts defined as atom, for example.

The problem with the Reporting-UA specification in 2298 and drafts
of its successor is that the specification

      reporting-ua-field = "Reporting-UA" ":" ua-name
                           [ ";" ua-product ]

      ua-name = *text

      ua-product = *text

is ambiguous, as it is impossible to determine whether

   Reporting-UA: foo ; bar ; baz

does or does not have the optional ua-product, and if so, which
of the semicolons delimits its start. That is because *text does not
exclude semicolons, so those might be part of the ua-name. The
issue could be resolved by redefining

   ua-name = domain / phrase

which provides for the usage recommended in the text of 2298 (via domain)
and also provides for any special characters which might be used
under the current specification, provided that they are quoted as
required for a phrase. [actually, the "domain" part of the proposed
redefinition would become superfluous at such time that 2822 is revised
to formalize '.' in a phrase as suggested in 2822 section 4]

The use of phrase could be interpreted as permitting 2047 encoded-words.
It might even be desirable to redefine ua-product as phrase, as it is
not inconcievable that a product name might need to be encoded and/or
specify charset or language. Likewise for many of the other MDN and
DSN field parts currently defined as *text, as well as the Usefor
draft's proposed User-Agent header field.

-----------
* that is the case because the MDN and DSN fields explicitly specify use
of 822 header field sysntax. It would not apply to the "fields" in a
text/directory media type, for example, since those so-called "fields"
are not defined in accordance with RFC 822 header field syntax.

Date: Thu, 19 Dec 2002 09:06:37 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: Keith Moore <[email-masked]>
cc: [email-masked]
Subject: Re: Interpretation of RFC 2047 
Sender: [email-masked]

On Wed, 18 Dec 2002, Keith Moore wrote:

> so you probably could write a regular expression that would recognize
> rfc 822 in practice (e.g. just be prepared to handle at most 20
> levels of nested parentheses).

Totally off-topic, but people might be interested to know that the next
release of the PCRE regex engine has a recursion feature that enables it
to recognize nested things such as parens. (And other cool stuff - I
have a regex that recognizes palindromes!) It won't be out for some
months because I still have other work to do on it.

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: Alan Barrett <[email-masked]>
Subject: Re: Interpretation of RFC 2047 
Date: Wed, 18 Dec 2002 14:03:30 -0500
Sender: [email-masked]

> Various header fields are defined as structured fields but with
> unstructured portions; those portions may of course contain parentheses
> (not necessarily balanced) which should not be interpreted as comments
> (or, in fact, at all). That includes RFC 1894 Original-Envelope-ID,
> Reporting-MTA, DSN-Gateway, Received-From-MTA, Remote-MTA,
> Original-Recipient, Final-Recipient; RFC 2298 Diagnostic-Code,
> MDN-Gateway, and probably Reporting-UA (there is an ambiguity in the
> Reporting-UA ABNF, but that does not affect this issue).

these are not message header fields; they are part of specific
content-types.  surely you don't think that the 822 syntax should
be applied to message bodies also?

Keith

Date: Wed, 18 Dec 2002 13:29:30 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Alan Barrett <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Alan Barrett wrote:
> On Wed, 18 Dec 2002, Keith Moore wrote:

>>comments are only valid in structured fields.  so in order to
>>recognize a comment you have to know the set of structured fields.

To be more precise, comments are only valid in certain parts of
structured fields. RFC 2822 is quite explicit about where comments
are and are not permitted for the header fields which it defines.
Several other recent RFCs do likewise, using 2822's CFWS notation.

[...]

> OK, all structured header fields.  The entire lexical analyser described
> in RFC 822 section 3.1.4, and RFC 2822 section 3.2 (read in conjunction
> with section 2.2.2), seems to be intended to apply to all structured
> header fields.  I have always assumed that this included any structured
> fields that might be defined in the future.

I think the problem is the assumption. 2822 section 3.6.8 requires only
that extension header fields' content conform to the 2822 definition of
"unstructured" syntax, and specifically states that such fields are
uninterpreted as far as 2822 formal syntax is concerned.  The specified
interpretation of parenthesized comments in 2822 refers only to those
header fields specifically defined in 2822 (and of course by other RFCs
which explicitly reference CFWS and RFC 2822).

> Apart from RFC 2912 Content-Encoding, what else violates this assumption?

As previously mentioned, because URIs (RFC 2396) can include parentheses
(not necessarily balanced), any header field which includes a URI may
also have parentheses which are not part of a comment. That includes
Content-Location (RFC 2557), Content-Type (RFC 2017 provides for a URL
parameter with the message/external-body media type), and the obsolete
Content-Base (RFC 2110). RFC 2156 headers Discarded-X400-IPMS-Extensions,
Discarded-X400-MTS-Extensions, and X400-Received also have parenthesized
productions with specified structure within the parentheses (i.e. not a
comment), as does the Received header as defined in RFC 2821 (blame RFC
1123).  Various header fields are defined as structured fields but with
unstructured portions; those portions may of course contain parentheses
(not necessarily balanced) which should not be interpreted as comments
(or, in fact, at all). That includes RFC 1894 Original-Envelope-ID,
Reporting-MTA, DSN-Gateway, Received-From-MTA, Remote-MTA,
Original-Recipient, Final-Recipient; RFC 2298 Diagnostic-Code,
MDN-Gateway, and probably Reporting-UA (there is an ambiguity in the
Reporting-UA ABNF, but that does not affect this issue).

Of these, 2557 will have to be revised to exclude comments, as the
current ABNF leads to ambiguities, and a case could be made for a
revision to 2821, since as it stands, it's impossible to distinguish
a parenthesized "TCP-Info" production from a comment, both of which
may appear after a domain or address literal in the from or by clauses
of a Received header [and unfortunately, the most reliable trace
information gets relegated to the inside of parentheses while the
easily-forged content is in well-structured parts of the header]. 2298
needs revision due to an unrelated ambiguity.

The revisions to 2557 and 2298's Reporting-UA should not cause problems;
indeed they are necessary to resolve problems in the existing ABNF syntax
specifications. Revising 2821 would probably cause a certain amount of
wailing and gnashing of teeth, but would at least (if done properly)
provide more reliable information suitable for automated tracing of
mail paths (e.g. for tracking the sources of spam) than is currently
possible.  It would be difficult to present a compelling case for
revision to the other headers, let alone actually coming up with
replacement syntax that doesn't break something -- and there does
not appear to be any potential benefit to doing so.

Date: Wed, 18 Dec 2002 20:12:58 +0200
From: Alan Barrett <[email-masked]>
To: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

On Wed, 18 Dec 2002, [email-masked] wrote:
> I therefore question the appropriateness of making this assumption and
> certainly do not make it in the code I write.

I intended to make the assumption (about lexical analysis of as yet
undefined header fields) while considering proposals for the definition
of new header fields, and perhaps in code where failure would not matter
(for example, a user interface feature to elide all comments when
displaying the header).

--apb (Alan Barrett)

Date: Wed, 18 Dec 2002 20:03:23 +0200
From: Alan Barrett <[email-masked]>
To: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

On Wed, 18 Dec 2002, Bruce Lilly wrote:
> >I agree on this point.  However, lexical analysis plus some guessing
> >will often be good enough.
> 
> Lexical analysis is equivalent to using regular expressions

Yes.

> (and hence insufficiently powerful)

That depends on the goal.  I still think that lexing plus heuristics
is often good enough.  For example, it might be good enough for a user
interface to choose whether to display the RFC 2047 decoded form or the
raw form by default (with an option to show the other form in either
case), but not good enough for a gateway.

--apb (Alan Barrett)

From: Keith Moore <[email-masked]>
To: [email-masked]
Subject: Re: Interpretation of RFC 2047 
Date: Wed, 18 Dec 2002 12:56:22 -0500
Sender: [email-masked]

> Lexical analysis is equivalent to using regular expressions (and hence
> insufficiently powerful) -- indeed many lexical analyzers are build by
> constructing a finite automaton from a set of regular expressions.

well, for that matter, almost the entirety of rfc 822 can be parsed
with a finite automaton - the main exception is nested parentheses.
so you probably could write a regular expression that would recognize
rfc 822 in practice (e.g. just be prepared to handle at most 20 
levels of nested parentheses).

But I think the point is that it's hard to find a _simple_ regex/FSM 
that will do an adequate job of recognizing 2047 for all fields.  

Keith

Date: Wed, 18 Dec 2002 08:52:51 -0800 (PST)
From: [email-masked]
Subject: Re: Interpretation of RFC 2047
To: Alan Barrett <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> On Wed, 18 Dec 2002, Keith Moore wrote:
> > > One can recognise a comment from lexical analysis alone.
> >
> > comments are only valid in structured fields.  so in order to
> > recognize a comment you have to know the set of structured fields.

> Yes, that's true (both in RFC 822 section 3.1.3 and RFC 2822 section
> 2.2.1).

> Does anybody claim that that RFC 2912 "Content-Encoding" is an
> unstructured field?

I think you meant content-features, not content-encoding. In any case, I
suppose one of the ways out of this is indeed to call this an unstructured
field as far as RFC 2822 is concerned; it certainly has been done before.

> > and it is (perhaps unfortunately) the cases that some fields have
> > a syntax that uses parenthesis as other than comment delimiters.
> > if I'm not mistaken this has been the case ever since rfc 987,
> > which used constructs like (a) to order to encode things like @ in
> > PrintableString fields.

> By my reading of RFC 987, if a PrintableString is used in a context
> where something like unquoted "(a)" could be misinterpreted as a
> comment, then the entire PrintableString must be further encoded in an
> RFC 822 quoted-string.  See the second paragraph on page 58 of RFC 987,
> where it says "word may be encoded as 822.atom (which has a restricted
> character set) or as 822.quoted-string, which can handle all ASCII
> characters."

Right idea, wrong specific item and wrong RFC. RFC 987 is obsolete; the current
version is RFC 2156. And the case in RFC 2156 where parentheses are used
isn't the encoding of printablestrings, it is the encoding of object
identifiers defined in section 3.3.7:

    joint-iso-ccitt(2) mhs (6) ipms (1) ep (11) ia5-text (0) 

RFC 2156 attempts to finess the conflict with RFC 822 comments by claiming this
is only used in unstructured fields as far as RFC 822 is concerned. This always
struck me as a bit of a stretch, however; several of the fields, such as
original-encoded-information-types clearly contain structured information.

> > > I submit that the RFC 2822 section 3.2.3 definition of a comment was
> > > intended to apply to all header fields
> >
> > I don't think so - that would break too many things already in
> > existence.

> OK, all structured header fields.  The entire lexical analyser described
> in RFC 822 section 3.1.4, and RFC 2822 section 3.2 (read in conjunction
> with section 2.2.2), seems to be intended to apply to all structured
> header fields.  I have always assumed that this included any structured
> fields that might be defined in the future.

> Apart from RFC 2912 Content-Encoding, what else violates this assumption?

I don't know of any offhand, but given that there are already several examples,
it seems wise to assume there might be more. I therefore question the
appropriateness of making this assumption and certainly do not make it
in the code I write.

				Ned

Date: Wed, 18 Dec 2002 12:22:58 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Alan Barrett <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Alan Barrett wrote:
> On Tue, 17 Dec 2002, Bruce Lilly wrote:
> 
>>One cannot recognise a comment unless the header field syntax is known.
> 
> 
> One can recognise a comment from lexical analysis alone.  This was true
> in RFC 822, and should still be true in RFC 2822 unless something went
> wrong.
[...]
>>Other examples could be given, but the above show that it is necessary
>>to fully parse header field content in order to determine whether
>>or not there is an encoded-word; use of regular expressions (or the
>>equivalent) is inadequate.
> 
> 
> I agree on this point.  However, lexical analysis plus some guessing
> will often be good enough.

Lexical analysis is equivalent to using regular expressions (and hence
insufficiently powerful) -- indeed many lexical analyzers are build by
constructing a finite automaton from a set of regular expressions.

Date: Wed, 18 Dec 2002 18:08:53 +0200
From: Alan Barrett <[email-masked]>
To: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

On Wed, 18 Dec 2002, Keith Moore wrote:
> > By my reading of RFC 987, if a PrintableString is used in a context
> > where something like unquoted "(a)" could be misinterpreted as a
> > comment, then the entire PrintableString must be further encoded in an
> > RFC 822 quoted-string.  
> 
> perhaps.  but this was not done in practice.

OK.  I managed to avoid encountering such cases.

--apb (Alan Barrett)

From: Keith Moore <[email-masked]>
To: Alan Barrett <[email-masked]>
cc: [email-masked]
Subject: Re: Interpretation of RFC 2047 
Date: Wed, 18 Dec 2002 11:01:42 -0500
Sender: [email-masked]

> By my reading of RFC 987, if a PrintableString is used in a context
> where something like unquoted "(a)" could be misinterpreted as a
> comment, then the entire PrintableString must be further encoded in an
> RFC 822 quoted-string.  

perhaps.  but this was not done in practice.

> Apart from RFC 2912 Content-Encoding, what else violates this assumption?

the real problem is knowing which fields are structured and which 
are unstructured. 

Keith

Date: Wed, 18 Dec 2002 17:37:03 +0200
From: Alan Barrett <[email-masked]>
To: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

On Wed, 18 Dec 2002, Keith Moore wrote:
> > One can recognise a comment from lexical analysis alone.  
> 
> comments are only valid in structured fields.  so in order to
> recognize a comment you have to know the set of structured fields.

Yes, that's true (both in RFC 822 section 3.1.3 and RFC 2822 section
2.2.1).

Does anybody claim that that RFC 2912 "Content-Encoding" is an
unstructured field?

> and it is (perhaps unfortunately) the cases that some fields have
> a syntax that uses parenthesis as other than comment delimiters.
> if I'm not mistaken this has been the case ever since rfc 987,
> which used constructs like (a) to order to encode things like @ in
> PrintableString fields.

By my reading of RFC 987, if a PrintableString is used in a context
where something like unquoted "(a)" could be misinterpreted as a
comment, then the entire PrintableString must be further encoded in an
RFC 822 quoted-string.  See the second paragraph on page 58 of RFC 987,
where it says "word may be encoded as 822.atom (which has a restricted
character set) or as 822.quoted-string, which can handle all ASCII
characters."

> > I submit that the RFC 2822 section 3.2.3 definition of a comment was
> > intended to apply to all header fields
>
> I don't think so - that would break too many things already in
> existence.

OK, all structured header fields.  The entire lexical analyser described
in RFC 822 section 3.1.4, and RFC 2822 section 3.2 (read in conjunction
with section 2.2.2), seems to be intended to apply to all structured
header fields.  I have always assumed that this included any structured
fields that might be defined in the future.

Apart from RFC 2912 Content-Encoding, what else violates this assumption?

--apb (Alan Barrett)

From: Keith Moore <[email-masked]>
To: Alan Barrett <[email-masked]>
cc: [email-masked]
Subject: Re: Interpretation of RFC 2047 
Date: Wed, 18 Dec 2002 09:35:53 -0500
Sender: [email-masked]

> One can recognise a comment from lexical analysis alone.  

comments are only valid in structured fields.  so in order to
recognize a comment you have to know the set of structured fields.
and it is (perhaps unfortunately) the cases that some fields
have a syntax that uses parenthesis as other than comment delimiters.
if I'm not mistaken this has been the case ever since rfc 987,
which used constructs like (a) to order to encode things like @
in PrintableString fields.

> I submit that the RFC 2822 section 3.2.3 definition of a comment was
> intended to apply to all header fields

I don't think so - that would break too many things already in existence.

Keith

Date: Wed, 18 Dec 2002 11:57:20 +0200
From: Alan Barrett <[email-masked]>
To: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

On Tue, 17 Dec 2002, Bruce Lilly wrote:
> One cannot recognise a comment unless the header field syntax is known.

One can recognise a comment from lexical analysis alone.  This was true
in RFC 822, and should still be true in RFC 2822 unless something went
wrong.

>    Content-Features: (& (Type="text/plain") (charset=US-ASCII) )
>
> contains no comments.

RFC 822 was absolutely clear that it contains a comment.  By my reading
of RFC 2822 section 3.2.3, it still contains a comment.

RFC 2912 suggests that the above Content-Features header field contains
no comments.  But RFC 2912 was published before RFC 2822, so cannot use
any sophistry about RFC 2822 perhaps having unintentionally changed the
definition of a comment.  Instead, RFC 2912 claims to depend on RFC 822,
where the definition of a comment is absolutely clear, so RFC 2912 would
have had no excuse at all for trying to modify it.

RFC 822 and 2822 did not deliberately leave open the possibility for
future header fields to redefine the comment syntax.  RFC 2912 does not
even discuss the fact that it attempts to redefine the comment syntax.
This is a fatal flaw in RFC 2912, and it's somewhat surprising that it
was not noticed before.

>    Foobar: (& (Type="text/plain") (charset=US-ASCII) )
> 
> might or might not contain comments depending on the definition of
> the Foobar header field.  I submit that
> 
>    Foobar: file:(=?us-ascii?q?=3D?=)
> 
> does not contain a comment.  It does have matched parentheses.  It
> does not contain an RFC 2047 encoded-word and does not encode any
> 8-bit characters It does contain a syntactically valid absolute URI.

I submit that the RFC 2822 section 3.2.3 definition of a comment was
intended to apply to all header fields, including those defined in RFC
2822's future; that it was a mistake for RFC 2912's "Content-Features:"
header field to contain stuff that looks like a comment but is not
intended as a comment; and that it would be a mistake for the definition
of the Foobar: header field to try to say that the above example does
not contain a comment.

> Other examples could be given, but the above show that it is necessary
> to fully parse header field content in order to determine whether
> or not there is an encoded-word; use of regular expressions (or the
> equivalent) is inadequate.

I agree on this point.  However, lexical analysis plus some guessing
will often be good enough.

--apb (Alan Barrett)

Date: Tue, 17 Dec 2002 13:27:46 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:

> I think a reasonable heuristic, which would nearly always do the "right
> thing" would be:
> 
> NOT to decode anything within properly matched "...", <...> or [...] or
> which follows a ';' which looks like the start of some MIME parameters.
> And otherwise decode anything enclosed by WS or within properly matched
> and nested (...).

As header field contents are defined by a grammar, attempts to
decode using only regular expressions (as opposed to a parser
which accepts the defined grammar) are doomed to failure. Failures
include both false positives and false negatives, as illustrated
below.

> But there is a more interesting question, which is what agents that create
> unrecognized headers with 8bit stuff in them could usefully do. I.e. a
> user tries to create a Foobar: header with such stuff in it. This could be
> a problem in news to mail gatewaying. Treating all such headers as
> unstructured is possible, but might not do the right thing. Trying to
> recognise comments might be better (not within "...", <...> or [...]
> though).

One cannot recognise a comment unless the header field syntax is known.

    Content-Features: (& (Type="text/plain") (charset=US-ASCII) )

contains no comments.

    Foobar: (& (Type="text/plain") (charset=US-ASCII) )

might or might not contain comments depending on the definition of
the Foobar header field.  I submit that

    Foobar: file:(=?us-ascii?q?=3D?=)

does not contain a comment. It does have matched parentheses.  It does
not contain an RFC 2047 encoded-word and does not encode any 8-bit
characters  It does contain a syntactically valid absolute URI.

    Foobar: http://users.erols.com/blilly/mailparse/(=?us-ascii?q?=3D?=)

does not contain a comment. It does have matched parentheses.  It does
not contain an RFC 2047 encoded-word and does not encode any 8-bit
characters  It contains a valid absolute URI with a query.  You are
welcome to try the URI; it does work (though the query is ignored).

Either could just as well be a Content-Location header. Both would be
attempted to be decoded using a simple regular expression matching
heuristic.  If thus inappropriately decoded, they would yield

    Foobar: file:=
    Foobar: http://users.erols.com/blilly/mailparse/=

which are clearly not what was intended.  You are welcome to try the
last one as a URI, you will get a 404 not found error.  N.B. you might
(in general) instead have stumbled upon a valid URI which was different
from the intended one.

Treating an unrecognized header field as unstructured in the above
examples would not mangle the URIs, for display or otherwise.  If the
example were instead

    Foobar: =?us-ascii?q?-3D?=

treating the header as unstructured may result in decoding for display.
Gateways should not attempt to transform unrecognized header fields; it
is unknown whether or not the above example really contains an encoded-word.
If an unrecognized header field has content which is forbidden in the
destination network, the header could be elided. If the content is not
forbidden, the unrecognized header field should be passed unaltered.  A
network which would forbid RFC 2047 encoded-word content would be rather
unusual, to say the least.  A gateway should never decode RFC 2047
encoded-words in header fields, as the decoded word may have octets or
combinations of octets which are illegal in header fields (e.g. NUL, DEL,
8-bit-set, lone CR).  Such decoding might be acceptable if both of the
following conditions apply:
1. the destination network uses content in some format other than RFC
    2822 header fields (otherwise, there's no need for transformation).
2. it is guaranteed that a reverse transformation from the destination
    network to Internet mail is possible and produces content equivalent
    to the original (i.e. equivalent to leaving the header field unaltered),
    or that no reverse gateway will attempt to regenerate the header field
    (i.e. equivalent to eliding the header in the forward gateway
    transformation).

Excluding content after a semicolon would fail to decode the RFC 2047
encoded-word in the header

    To: empty-list:;, =?iso-8859-1?Q?J=FCrgen?= [email-masked]

Excluding content bracketed in <> would also be an error. Consider RFCs
2368 and 2369 (not to be confused with 2396, which is also applicable) and:

    List-Owner: <mailto:[email-masked]?Subject=list>

That does contain an RFC2047 encoded-word within the <>. Decoding the
content within <> first requires decoding URI encoding to obtain the
same mailbox as specified in the To header example above.

Further note that List-Owner provides for specifying additional header
fields (as with Subject in the example above), and of course
Content-Location and the hypothetical Foobar are not excluded.

Other examples could be given, but the above show that it is necessary
to fully parse header field content in order to determine whether or
not there is an encoded-word; use of regular expressions (or the
equivalent) is inadequate.

To: [email-masked]
Xref: clerew local.mime:1959
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Date: Tue, 17 Dec 2002 10:46:39 GMT
Lines: 80
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> Would it not a sensible rule be to say that you should decode any occurrence
>> of =?<charset>?[BQ]?...?= (subject to the 76 character limit) in any
>> header provided:
>>     (a) it was immediately preceded by '(' or by CFWS
>>     (b) it was immediately followed by ')' or by CFWS
>>     (c) it was not contained within a quoted-string

>(d) it was not part of a MIME parameter (RFC 2047 expressly forbids 2047
>     encoding in MIME parameters; RFC 2231 provides a mechanism for parameters
>     and also extends 2047 to include language tags)

>... and more (see below)

>> Actually, there is a parsing required, because an encoded word in an
>> unstructured header must have LWS (i.e. CFWS) on either side of it, whereas
>> it can also have '(' and ')' immediately next to it in a strutured header.

I think a reasonable heuristic, which would nearly always do the "right
thing" would be:

NOT to decode anything within properly matched "...", <...> or [...] or
which follows a ';' which looks like the start of some MIME parameters.
And otherwise decode anything enclosed by WS or within properly matched
and nested (...).

>Other areas that immediately come to mind are:
>1. RFC 2557 Content-Location, which permits URIs, which in turn (RFC 2396)
>    permit parentheses.  That's in a structured field, but a URI, not a
>    comment. [there are issues with 2557 and CFWS vs. the URIs, and these
>    have been discussed on the MHTML list]

But URIs are not supposed to contain 8bit stuff, so the question should
not arise. And if IRIs should ever get into the standard, then there is a
special downgrading to URI built in (yes, that is Yet Another Encoding for
us to have to worry about :-( ). Mind you, if the URIs are enclosed within
<...>, then my rule above would cover them.

>I'm not certain, but I don't believe that the filter syntax permits anything
>resembling 2047 encoding.  URIs probably do, but again, I haven't checked
>thoroughly.

We are talking only of headers which the agent does not already know how
to parse, so presumably it is not going to do more than display them. But
displaying is probably more useful than leaving them alone as far as being
helpful to the reader is concerned.

>Strictly speaking, one can only decode if one knows the relevant header
>syntax.  Display is a relatively minor issue, subject to the above
>caveat.  But transformations by gateways may result in fouling up content
>beyond all recognition unless the header syntax is known.  Ideally,
>gateways shouldn't decode encoded-words -- if they're left in encoded
>form there is no chance that they'll be garbled, which is the likely
>outcome unless strict syntax of headers is known and applied rigorously.

Agreed.

But there is a more interesting question, which is what agents that create
unrecognized headers with 8bit stuff in them could usefully do. I.e. a
user tries to create a Foobar: header with such stuff in it. This could be
a problem in news to mail gatewaying. Treating all such headers as
unstructured is possible, but might not do the right thing. Trying to
recognise comments might be better (not within "...", <...> or [...]
though).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Subject: Re: The sad state of MIME non-compliance
From: "Adrian 'Dagurashibanipal' von Bidder" <[email-masked]>
To: [email-masked]
Organization: 
Date: 13 Dec 2002 22:56:31 +0100
Sender: [email-masked]

--=-GanXbSZ6C53QDLjkWXTE
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Fri, 2002-12-13 at 19:00, Bruce Lilly wrote:

> multipart/mixed
>     text/plain (introductory text)
>     multipart/alternative
>        multipart/related
>           text/plain (page 1 text)
>           text/plain (page 2 text)
>        multipart/related
>           application/pdf (page 1)
>           application/pdf (page 2)
>=20
> Sadly, many of the MUAs currently on the market fail to display
> such a message properly.  Of course, non-MIME text-only UAs will
> display the message body.  Kmail 1.4.3 seems to do a reasonable
> job of displaying the message correctly.  But Netscape/Mozilla,
> MS Outlook [Express], Eudora, and probably others do not. Most
> of these display only the introductory text.

Just to add a data point: evolution (1.2) does fine, displaying the two
pdf files (and not displaying the two text parts).

cheers
-- vbi

--=20
this email is protected by a digital signature:  http://fortytwo.ch/gpg

NOTE: keyserver bugs! get my key here: https://fortytwo.ch/gpg/92082481

--=-GanXbSZ6C53QDLjkWXTE
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

Signature policy: http://fortytwo.ch/legal/gpg/email.20020822

--=-GanXbSZ6C53QDLjkWXTE--

Date: Fri, 13 Dec 2002 13:19:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked]
Subject: Re: RFC2231 encoding in parameters.
Sender: [email-masked]

[email-masked] wrote:

> 1) Are any MUAs "in the wild" currently actually using the 2047-style encoding
> in parameters rather than the 2231 syntax?  If so, who are they, and who wants
> to send the authors a note? ;)

As of a couple of months ago, Netscape/Mozilla did so by default.
There is an obscure, undocumented configuration parameter that
can be placed in the configuration file to change that to use
2231 encoding, but (again, as of a couple of months ago) it did
not work:
1. 8-bit file names were first 2047-encoded, then that encoded via
    the 2231 mechanism
2. instead of a single language tag, a language list was used under
    some circumstances.

I think a bug report was submitted, but a quick search at bugzilla
today didn't show anything relevant.

With the obscure configuration parameter in place, Mozilla 1.2a
produced the following (which, as the filename is us-ascii, does not
show problem #1):

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910
Content-Type: application/excel;
  name*=ISO-8859-1'en-us, en, fr, ru, ja'membership.xls

Date: Fri, 13 Dec 2002 13:00:48 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: The sad state of MIME non-compliance
Sender: [email-masked]

Recently I had occasion to send a message consisting of
introductory text plus two pages of content in two formats
(text/plain and application/pdf). The obvious MIME structure
would be

multipart/mixed
    text/plain (introductory text)
    multipart/alternative
       multipart/related
          text/plain (page 1 text)
          text/plain (page 2 text)
       multipart/related
          application/pdf (page 1)
          application/pdf (page 2)

Sadly, many of the MUAs currently on the market fail to display
such a message properly.  Of course, non-MIME text-only UAs will
display the message body.  Kmail 1.4.3 seems to do a reasonable
job of displaying the message correctly.  But Netscape/Mozilla,
MS Outlook [Express], Eudora, and probably others do not. Most
of these display only the introductory text.

A sample message has been submitted with Mozilla bug report
#184869, viewable at http://bugzilla.mozilla.org/show_bug.cgi?id=184869

It appears in most cases that it is the nested MIME multipart
structure that is not being correctly parsed; changing individual
media types seems to have little or no effect.

I suspect that part of the problem is what RFC 2822 has remedied
w.r.t. non-MIME messages, viz. the standards are spread out over
multiple documents, which makes it difficult for developers to
refer to *the* standard.  Worse, there are errors in some of the
documents, and the errata page at www.rfc-editor.org is not
exactly well-publicized (IMO, it should be mentioned in
every RFC).  Still worse, some of the RFCs contradict others (e.g.
while RFC 2046 states that the domain of an entity in the absence
of a Content-Transfer-Encoding header defaults to 7bit, RFC
2425 [5.5] claims that it is 8bit.  2822 also clarified the syntax
relative to 822.

It would be nice if the MIME documents were similarly brought up to
date with current ABNF, retrieval of the text which has been lost
in troff-comment-land these many years, and consolidation into a
small set of consistent documents.

Date: Fri, 13 Dec 2002 12:25:30 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: Charles Lindsey <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

Charles Lindsey wrote:

> Would it not a sensible rule be to say that you should decode any occurrence
> of =?<charset>?[BQ]?...?= (subject to the 76 character limit) in any
> header provided:
>     (a) it was immediately preceded by '(' or by CFWS
>     (b) it was immediately followed by ')' or by CFWS
>     (c) it was not contained within a quoted-string

(d) it was not part of a MIME parameter (RFC 2047 expressly forbids 2047
     encoding in MIME parameters; RFC 2231 provides a mechanism for parameters
     and also extends 2047 to include language tags)

... and more (see below)

> Actually, there is a parsing required, because an encoded word in an
> unstructured header must have LWS (i.e. CFWS) on either side of it, whereas
> it can also have '(' and ')' immediately next to it in a strutured header.

That's not accurate: first, LWS and CFWS are different "(a) =?se2?q?x?="
(quotes for legibility only) is legal whereas " (a)=?se2?q?x?=" is not;
both have CFWS immediately before what looks like an encoded-word, but
only the former has LWS immediately before an encoded-word. And there
are many issues with parentheses; ")=?se2?q?x?=(" in a structured
header which contains no other parentheses does not contain an
encoded-word.

Other areas that immediately come to mind are:
1. RFC 2557 Content-Location, which permits URIs, which in turn (RFC 2396)
    permit parentheses.  That's in a structured field, but a URI, not a
    comment. [there are issues with 2557 and CFWS vs. the URIs, and these
    have been discussed on the MHTML list]
2. RFC 2533 "filters" have more nested parentheses than a technical paper
    at a LISP convention.  They're not comments and they appear in
    structured MIME extension headers (RFC 3297 Content-Alternative,
    RFC 2912 Content-Features).
3. URIs can also appear in other MIME extension headers; IIRC one of the
    RFCs provides for a URI in a parameter.
4. URIs also appear in headers which are not MIME extension headers, e.g.
    many of the List- headers.

I'm not certain, but I don't believe that the filter syntax permits anything
resembling 2047 encoding.  URIs probably do, but again, I haven't checked
thoroughly.

Misinterpreting something as encoded when it is in fact not an encoded-word
can have consequences.  Even if not changed in the protocol, but only for
display, there could be problems e.g. with cut-and-paste of URIs.

Strictly speaking, one can only decode if one knows the relevant header
syntax.  Display is a relatively minor issue, subject to the above
caveat.  But transformations by gateways may result in fouling up content
beyond all recognition unless the header syntax is known.  Ideally,
gateways shouldn't decode encoded-words -- if they're left in encoded
form there is no chance that they'll be garbled, which is the likely
outcome unless strict syntax of headers is known and applied rigorously.