Subject: Re: Interpretation of RFC 2047
From: Keith Moore <[email-masked]>
To: [email-masked], [email-masked]
Date: Tue, 22 Oct 2002 13:47:59 -0400
List-Id: <ietf-822.imc.org>

> >> I think when the time comes to revise 2047, you need to take account of
> >> what real implementations actually do, and try to move in their direction.
> 
> >so we make the implementation bugs a part of the standard?  that's nuts.
> 
> I think if an implementation has made a useful improvement to what the
> standard demands (and if that 'improvement' is widely followed by other
> implementations), then it is reasonable for the next standard to take it
> on board.

perhaps.  but those 'useful improvements' are a tiny subset of
'what real implementations actually do' that deviate from the standard.

Keith

To: [email-masked]
Xref: clerew local.mime:1897 local.usefor:15257
Newsgroups: local.mime,local.usefor
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Reply-To: [email-masked], [email-masked]
Date: Mon, 21 Oct 2002 10:56:52 GMT
Lines: 69
Sender: [email-masked]

In <[email-masked]> "Bart Schaefer" <[email-masked]> writes:

>On Oct 17, 11:32am, Charles Lindsey wrote:
>} Subject: Re: Interpretation of RFC 2047
>}
>} 
>} Eh? Where does it say that unknown headers are to be treated as
>} unstructured? If that were the case, things would be much simpler, but
>} I don't think it is.

>RFC 2822, 3.6.8. Optional fields:

>It's true that RFC 822 is not this explicit, but it's pretty clear that
>any field that doesn't have a defined structure is unstructured; RFC 822
>appendix D:

Quite so. Reading between the lines, that may well be the intended meaning
but, as you say, it is not explicit.

And treating unknown headers as 'unstructured' for RFC 2047 purposes does
not work. For example:

	Foobar: (=?ISO-8859-1?Q?Claus_F=E4rber?=) Fabrik

is not 2047-compliant regarded as an unstructured header (it _might_ be if
regarded as a structured one).

>} It is not a question of "acting on it". All I want is for it to be
>} displayed (or displayable) correctly. Most usage of encoded-words is only
>} intended for human consumption anyway.

>You're confusing user interface with protocol.

But Section 6 of RFC 2047 explicitly addresses display issues, so it is in
the protocol (just - that section is decidedly vague).

But generally speaking, if you regard the purpose of a protcol is to
deliver a communication from a sender to receiver, then the user-agent is
indeed a part of the protocol.

Suppose I send you a message
	"I will meet you on Tuesday"
and it is delivered correctly to your MUA, which then displays it as
	"I will meet you on Wednesday"
are you going to claim that no standard has been broken? That the MUA is
fully compliant?

>So a user agent is "allowed" (because it is outside the scope of the
>standard), for purposes of display, to treat any string, anywhere, that
>begins with =? and ends with ?= as an RFC 2047 token, and decode it.

>What the user agent is NOT allowed to do is, for example, when asked to
>forward the message, decode and/or encode 2047 tokens in field bodies for
>which it does not know the semantics.

Yes, I agree fully with that last bit.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1896 local.usefor:15256
Newsgroups: local.mime,local.usefor
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Reply-To: [email-masked], [email-masked]
Date: Mon, 21 Oct 2002 10:38:51 GMT
Lines: 25
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> I think when the time comes to revise 2047, you need to take account of
>> what real implementations actually do, and try to move in their direction.

>so we make the implementation bugs a part of the standard?  that's nuts.

I think if an implementation has made a useful improvement to what the
standard demands (and if that 'improvement' is widely followed by other
implementations), then it is reasonable for the next standard to take it
on board.

That is, after all, the way most of the features you find in present
standards originally came about.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Interpretation of RFC 2047 
Date: Sat, 19 Oct 2002 08:37:33 -0400
Sender: [email-masked]

> I think when the time comes to revise 2047, you need to take account of
> what real implementations actually do, and try to move in their direction.

so we make the implementation bugs a part of the standard?  that's nuts.  

From: Keith Moore <[email-masked]>
To: "Bart Schaefer" <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: Interpretation of RFC 2047 
Date: Sat, 19 Oct 2002 08:36:05 -0400
Sender: [email-masked]

> So a user agent is "allowed" (because it is outside the scope of the
> standard), for purposes of display, to treat any string, anywhere, that
> begins with =? and ends with ?= as an RFC 2047 token, and decode it.

seems like a stretch.  and 2047 takes exception to that.

while it's generally the case that IETF standards do not specify UIs,
I don't recall any rule that limits IETF standards' applicability in this way.
people keep citing it (without references) as if it were one.

in general I think it's wise for IETF to avoid specifying UIs, but occasionally
it's also wise to make an exception to that.  

Keith

To: [email-masked]
Xref: clerew local.mime:1892 local.usefor:15222
Newsgroups: local.mime,local.usefor
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Reply-To: [email-masked], [email-masked]
Date: Fri, 18 Oct 2002 09:39:29 GMT
Lines: 78
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> And that's what worries me (and worries others on the Usefor list). It
>> seems that every time someone invents a new email (or news) extension,
>> lots of gateways, and user agents also, suddenly become out-of-date.

>True, but I don't know what to do about it.  Usenet and email are 
>fundamentally different beasts serving different communities and they 
>will inevitably diverge even if we tried to force them not to do so.
>In a way it's too bad that they use such similar message formats -
>it invites confusion for both humans and protocol implementations.

Indeed, but this isn't just a Usenet problem (I merely used that example
because I have an immediate concern with it).

Suppose that someone has a bright idea for a new email header. For example

    Alternative-Address: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>

with the meaning "if the To: address is undeliverable, then send the email
to the alternative". So he writes an internet-draft, it is discussed, and
eventually becomes a standards-track RFC. Naturally, it wants to use RFC
2047 in that 'phrase', and expects user agents to decode and display
"Färber". The problems it faces in getting recognized by user agents are
exactly the same as with my Mail-Copies-To header.

>well, 2047 doesn't say so explicitly but it would be somewhat
>reasonable for an implementation to treat any unrecognized field
>as a user-defined field for the purpose of display.  that's one
>reason that the rules for recognizing encoded-words are looser
>than the rules for generating enocded-words.

Yes, but not looser enough. A reasonable approach would be:

If you are configured to understand this header, then parse it and deal
with encoded-words accordingly.

If you are not configured to understand this header (in which case there
is no semantic action you can possibly take, other than to display it),
then decode and display anything that looks remotely like an encoded word.

This might have the odd property that a non-upgraded agent might display
something that it would reject as erroneous and fail to display after it
had been upgraded, but I could live with that. I suspect that many real
world agents already do something like that, but it would be much nicer of
RFC 2047 had made that behaviour explicitly allowable, for the benefit of
future headers such as Alternative-Address or Mail-Copies-To.

>> It applies to unstructured fields also, since you need to known whether or
>> not you are allowed to decode "(=?ISO-8859-1?Q?Claus_F=E4rber?=)".

>strictly speaking, there's no parsing to an unstructured field.  you 
>just treat it as *text.

Actually, there is a parsing required, because an encoded word in an
unstructured header must have LWS (i.e. CFWS) on either side of it, whereas
it can also have '(' and ')' immediately next to it in a strutured header.

>however at this point I'd be reluctant to change 2047 - partially because
>stability of the message format is perhaps more important and partially 
>because we've been over all this before and not found a way to significantly
>improve on the current situation.

I think when the time comes to revise 2047, you need to take account of
what real implementations actually do, and try to move in their direction.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "Bart Schaefer" <[email-masked]>
Date: Fri, 18 Oct 2002 15:17:34 +0000
Comments: In reply to "Charles Lindsey" <[email-masked]> "Re: Interpretation of RFC 2047" (Oct 17, 11:32am)
To: [email-masked], [email-masked]
Subject: Re: Interpretation of RFC 2047
Sender: [email-masked]

On Oct 17, 11:32am, Charles Lindsey wrote:
} Subject: Re: Interpretation of RFC 2047
}
} In <[email-masked]> "Bart Schaefer" <[email-masked]> writes:
} 
} >On Oct 14,  1:01pm, Charles Lindsey wrote:
} 
} >The user agent is not supposed to know.  The user agent is not supposed
} >to look for or apply encodings/decodings inside header fields for which
} >it does not know the semantics.  All fields for which the semantics are
} >unknown are to be treated as unstructured upon receipt.
} 
} Eh? Where does it say that unknown headers are to be treated as
} unstructured? If that were the case, things would be much simpler, but
} I don't think it is.

RFC 2822, 3.6.8. Optional fields:

   Fields may appear in messages that are otherwise unspecified in this
   standard.  They MUST conform to the syntax of an optional-field.
   This is a field name, made up of the printable US-ASCII characters
   except SP and colon, followed by a colon, followed by any text which
   conforms to unstructured.

   The field names of any optional-field MUST NOT be identical to any
   field name specified elsewhere in this standard.

   optional-field  =       field-name ":" unstructured CRLF

It's true that RFC 822 is not this explicit, but it's pretty clear that
any field that doesn't have a defined structure is unstructured; RFC 822
appendix D:

     field-body-contents =
                   <the ASCII characters making up the field-body, as
                    defined in the following sections, and consisting
                    of combinations of atom, quoted-string, and
                    specials tokens, or else consisting of texts>

} >If you know what a header means, you may parse it and act on it.  If you
} >don't know, you should ignore it completely (pass it through unchanged,
} >hide it from the user, whatever).
} 
} It is not a question of "acting on it". All I want is for it to be
} displayed (or displayable) correctly. Most usage of encoded-words is only
} intended for human consumption anyway.

You're confusing user interface with protocol.

In general, IETF standards don't restrict the behavior of UIs, they only
restrict the behavior of protocol engines.  (Some of the biggest arguments
in the process of producing 2822 involved cases where the distinction was
not clear -- but I don't think that's the case in this instance.)

So a user agent is "allowed" (because it is outside the scope of the
standard), for purposes of display, to treat any string, anywhere, that
begins with =? and ends with ?= as an RFC 2047 token, and decode it.

What the user agent is NOT allowed to do is, for example, when asked to
forward the message, decode and/or encode 2047 tokens in field bodies for
which it does not know the semantics.

To: [email-masked]
Xref: clerew local.mime:1890 local.usefor:15208
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Reply-To: [email-masked], [email-masked]
Date: Thu, 17 Oct 2002 11:32:46 GMT
Lines: 92
Sender: [email-masked]

In <[email-masked]> "Bart Schaefer" <[email-masked]> writes:

>On Oct 14,  1:01pm, Charles Lindsey wrote:

>}    OTOH, both those views of Interpetation B seem to presuppose that the
>}    user agent was familiar with the syntax of Usefor.

>If the user agent is not familiar with the syntaxt of Usefor, it has no
>business assigning any semantics at all to the Mail-Copies-To field, so
>whether or not encoded words appear there is irrelevant.

A sending agent does not assign semantics to anything (but, in any case,
sending agents are not the problem, because they would not usually be
generating that header if they were not already Usefor-aware).

The real problem is email receiving agents. The only semantics expected in
that case is for the encoded-word to be decoded and displayed (assuming
the charset is known). It is highly desirable that should be done, but it
seems impossible unless the agent either recognizes the syntax, or it is
prepared to decode things that it is not required to decode by RFC 2047
(yes, I know that most agents do decode considerably more than RFC 2047
requires, but I don't like having to rely upon such liberality).

>The user agent is not supposed to know.  The user agent is not supposed
>to look for or apply encodings/decodings inside header fields for which
>it does not know the semantics.  All fields for which the semantics are
>unknown are to be treated as unstructured upon receipt.

Eh? Where does it say that unknown headers are to be treated as
unstructured? If that were the case, things would be much simpler, but I
don't think it is.
> 

>} Interpretation D:
>} 
>} The wording "rules in RFC 822" includes the rules for 'extension-field'
>} and 'user-defined-field'.

>This is the correct interpretation.

OK.

>} Hence "must parse the message" means that the rules in the document
>} defining the extension are to be applied.

>Exactly.

>} OTOH, Interpretation D seems to require that all user agents be
>} magically aware of all new extension headers as soon as their defining
>} documents are published.

>No.  It requires only that user agents DO NOT ATTEMPT TO PARSE header
>fields for which they do not know the semantics.

>If you know what a header means, you may parse it and act on it.  If you
>don't know, you should ignore it completely (pass it through unchanged,
>hide it from the user, whatever).

It is not a question of "acting on it". All I want is for it to be
displayed (or displayable) correctly. Most usage of encoded-words is only
intended for human consumption anyway.

I cannot accept that "hiding a header from the user" can ever be a proper
thing to do (yes, the user should be able to select which headers he sees,
but if he wants to see them all, then that MUST be possible).

>The intent of 2047 is that an application that chooses NOT to implement
>2047 can still parse the resulting tokens according to their definition
>in the appropriate document.  Thus most of the restrictions in 2047 are
>on applications that GENERATE 2047 header fields, to assure that they
>remain interoperable with non-2047-compliant applications

No problems with that.

>Whether an application that receives a 2047 encoded-word chooses to decode
>it is a quality of implementation issue.

But my problem is that 2047 places seemingly unnecessary restrictions on
what is decoded (restrictions which most implementations seem to ignore,
because they make life too difficult when keeping up with new extensions).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Interpretation of RFC 2047 
Date: Thu, 17 Oct 2002 14:43:36 -0400
Sender: [email-masked]

> >Interpretation B is intended.  
> 
> OK.
> 
> >>    Mail-Copies-To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>
> 
> >only if that portion of the field is defined as 'text'
> 
> No, it is defined as 'phrase' (it is part of an 'address-list' in fact).

okay, if it's defined as 'phrase' then 2047 should apply.  OTOH it's
a bit unreasonable to expect a mail user agent to understand the syntax
of a usenet message header field. 

> >  so it would
> >be left to a different spec to define how 2047 applies to mail-copies-to.
> 
> I think that means we have to say explicitly that 'phrases' in that
> header can be encoded as per RFC 2047, rather than the vaguer hand-waving
> in the present Usefor draft. No problem with doing that.

being explicit seems like a good idea.

> >no, it presupposes that when gateways take things that aren't email
> >messages (this includes usenet messages) and sends them via email,
> >it makes them compliant with email message standards at that time.
> >gateways have to track standards on both sides.  that's life.
> 
> And that's what worries me (and worries others on the Usefor list). It
> seems that every time someone invents a new email (or news) extension,
> lots of gateways, and user agents also, suddenly become out-of-date.

True, but I don't know what to do about it.  Usenet and email are 
fundamentally different beasts serving different communities and they 
will inevitably diverge even if we tried to force them not to do so.
In a way it's too bad that they use such similar message formats -
it invites confusion for both humans and protocol implementations.

> Whilst people who maintain gateways might conceivably "track the
> standards", there is no way that the installed base of user agents can
> possibly do so.

certainly they can, just as any other software can get updated.
it just takes a long time.

> >> Here is another example from Usefor:
> >> 
> >>    Organization: =?ISO-8859-1?Q?Claus_F=E4rber Fabrik?=
> >> 
> >> Q: Is that one RFC 2047-compliant?
> 
> >I'd say probably so.  though I don't know if it's formally defined anywhere,
> >organization is quite naturally 'text'.
> 
> It's formally defined as 'unstructured' (and 'unstructured' is essentially
> as in RFC 2822). But again it seems that you are saying that gateways and
> MUAs would have to be "tracking the standards" in order to legitimize it.

well, 2047 doesn't say so explicitly but it would be somewhat
reasonable for an implementation to treat any unrecognized field
as a user-defined field for the purpose of display.  that's one
reason that the rules for recognizing encoded-words are looser
than the rules for generating enocded-words.

> >> One would expect section 6 to require the recognition of anything that
> >> was allowed to appear under section 5, but that seems not to be the case
> >> because there is no mention of "extension message header fields".
> >> 
> >> In 6.1 I find:
> >> 
> >>    A mail reader must parse the message and body part headers according
> >>    to the rules in RFC 822 to correctly recognize 'encoded-word's.
> 
> >this is intended to apply to structured fields - the point is that
> >for structured fields you actually have to parse them to distinguish
> >between places where encoded words are valid (e.g. a word before a 
> >phrase) and places where they are not valid (e.g. a word in a local-part
> >or a domain).
> 
> It applies to unstructured fields also, since you need to known whether or
> not you are allowed to decode "(=?ISO-8859-1?Q?Claus_F=E4rber?=)".

strictly speaking, there's no parsing to an unstructured field.  you 
just treat it as *text.

> Would it not a sensible rule be to say that you should decode any occurrence
> of =?<charset>?[BQ]?...?= (subject to the 76 character limit) in any
> header provided:
>     (a) it was immediately preceded by '(' or by CFWS
>     (b) it was immediately followed by ')' or by CFWS
>     (c) it was not contained within a quoted-string

offhand I don't remember the reasoning for the current language.
I think part of the problem was that you can't expect either ( and ) 
or quotes to be meaningful except in a structured field.  e.g. There's
nothing at all wrong with

Subject: this"subject"has"five"quotes"! :)

also Q-encoded encoded-words are allowed to contain '(', ')' 
or '"' when they're used in unstructured fields. so it's tricky
to allow them to be used as delimiters.

however at this point I'd be reluctant to change 2047 - partially because
stability of the message format is perhaps more important and partially 
because we've been over all this before and not found a way to significantly
improve on the current situation.

> >> 3. Is it possible to go further and introduce header-fields with
> >> explicit 'encoded-word's in them, for example:
> 
> >yes.  I don't exactly like doing that (there's a lot of overlap between
> >'token' and 'encoded-word', for instance) but it's a widespread practice.
> 
> You say "widespread". Can you give examples of actual extensions which
> have done so?

I wasn't saying that including explicit encoded-words was widespread,
but rather we have a lot of protocols defined using productions of the 
form

a = b / c

where 'b' is a subset of 'c' and/or vice versa.

in this specific case encoded-word is a subset of both atom and token.

Keith

To: [email-masked]
Xref: clerew local.mime:1889 local.usefor:15207
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Interpretation of RFC 2047
Reply-To: [email-masked], [email-masked]
Date: Thu, 17 Oct 2002 11:04:59 GMT
Lines: 134
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> 1. Where can encoded-words legitimately appear (RFC 2047 section 5).
>> -------------------------------------------------------------------
>> 
>> Interpretaion B:
>> 
>> It means you can use an encoded-word in
>> 	any Subject                        )
>> 	any Comments                       ) for which the field body is
>> 	any extension message header field ) defined as '*text'
>> 	any MIME body part field           )
>> 	any X-header

>Interpretation B is intended.  

OK.

>>    Mail-Copies-To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>

>only if that portion of the field is defined as 'text'

No, it is defined as 'phrase' (it is part of an 'address-list' in fact).

>  so it would
>be left to a different spec to define how 2047 applies to mail-copies-to.

I think that means we have to say explicitly that 'phrases' in that
header can be encoded as per RFC 2047, rather than the vaguer hand-waving
in the present Usefor draft. No problem with doing that.

>> Q: Is an email message containing that header-field (or should I say the
>> user agent which permitted it to be sent as an email) RFC
>> 2047-compliant?

>not sure if you mean before or after 2047 encoding.   2047 doesn't
>forbid use in other structured fields, but neither does it require
>use in structured fields that aren't listed in 2047.

I meant after RFC 2047 encoding.

>>    OTOH, both those views of Interpetation B seem to presuppose that the
>>    user agent was familiar with the syntax of Usefor. 

>no, it presupposes that when gateways take things that aren't email
>messages (this includes usenet messages) and sends them via email,
>it makes them compliant with email message standards at that time.
>gateways have to track standards on both sides.  that's life.

And that's what worries me (and worries others on the Usefor list). It
seems that every time someone invents a new email (or news) extension,
lots of gateways, and user agents also, suddenly become out-of-date.
Whilst people who maintain gateways might conceivably "track the
standards", there is no way that the installed base of user agents can
possibly do so.

>> Here is another example from Usefor:
>> 
>>    Organization: =?ISO-8859-1?Q?Claus_F=E4rber Fabrik?=
>> 
>> Q: Is that one RFC 2047-compliant?

>I'd say probably so.  though I don't know if it's formally defined anywhere,
>organization is quite naturally 'text'.

It's formally defined as 'unstructured' (and 'unstructured' is essentially
as in RFC 2822). But again it seems that you are saying that gateways and
MUAs would have to be "tracking the standards" in order to legitimize it.

Anyway, as regards user agents that generate (or permit to be generated)
such newly-defined applications of RFC 2047 there is not any real
problem, since they would presumably not be generating such headers unless
they has already been upgraded to do so (and the issue of users inserting
'unknown' headers manually is a minor one).

The real problem is with agents receiving such headers:

>> 2. Where are encoded-words required to be recognized (RFC 2047 section 6).
>> --------------------------------------------------------------------------
>> 
>> One would expect section 6 to require the recognition of anything that
>> was allowed to appear under section 5, but that seems not to be the case
>> because there is no mention of "extension message header fields".
>> 
>> In 6.1 I find:
>> 
>>    A mail reader must parse the message and body part headers according
>>    to the rules in RFC 822 to correctly recognize 'encoded-word's.

>this is intended to apply to structured fields - the point is that
>for structured fields you actually have to parse them to distinguish
>between places where encoded words are valid (e.g. a word before a 
>phrase) and places where they are not valid (e.g. a word in a local-part
>or a domain).

It applies to unstructured fields also, since you need to known whether or
not you are allowed to decode "(=?ISO-8859-1?Q?Claus_F=E4rber?=)".

Would it not a sensible rule be to say that you should decode any occurrence
of =?<charset>?[BQ]?...?= (subject to the 76 character limit) in any
header provided:
    (a) it was immediately preceded by '(' or by CFWS
    (b) it was immediately followed by ')' or by CFWS
    (c) it was not contained within a quoted-string

That is actually sufficient to prevent it happening inside a domain
(except obs-domain), a local-part or a msgid. In fact, a large number of
existing user agents would already accept that, and more. Can anyone think
of any existing header that would actually break?

Anyway, I offer it as a thought for RFC 2047bis, when the time comes.

>> 3. Is it possible to go further and introduce header-fields with
>> explicit 'encoded-word's in them, for example:

>yes.  I don't exactly like doing that (there's a lot of overlap between
>'token' and 'encoded-word', for instance) but it's a widespread practice.

You say "widespread". Can you give examples of actual extensions which
have done so?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "Bart Schaefer" <[email-masked]>
Date: Mon, 14 Oct 2002 18:06:54 +0000
Comments: In reply to Charles Lindsey <[email-masked]> "Interpretation of RFC 2047" (Oct 14,  1:01pm)
To: [email-masked]
Subject: Re: Interpretation of RFC 2047
Cc: [email-masked]
Sender: [email-masked]

On Oct 14,  1:01pm, Charles Lindsey wrote:
}
}    An 'encoded-word' may replace a 'text' token (as defined by RFC 822)
}    in any Subject or Comments header field, any extension message header
}    field, or any MIME body part field for which the field body is
}    defined as '*text'. An 'encoded-word' may also appear in any
}    user-defined ("X-") message or body part header field.
} 
} That is ambiguous, depending on how you interpret the commas in the first
} sentence:

If you use interpretation "A" then extension and body part fields would
have to be replaced in their entirety by a _single_ encoded-word:

    "_An_ 'encoded-word' may replace any extension message header field"

As this is a practical impossibility, it seems pretty obvious that the
intended interpretation is "B", and in fact every implementation I've ever
seen appears to have chosen "B" ("may replace a 'text' _token_").

} [...] I if I had wanted
} Interpretation B, I would have written "... any extension message header
} field or any MIME body part field, for which the field body is defined
} as '*text'".

The intent is NOT to require that the field body is defined as '*text'.
The field body may have ANY definition that includes a 'text' token as
part of that definition.  However, ONLY the 'text' tokens may be replaced
by 'encoded-word'.  Other tokens in the field body MUST NOT be encoded.

}    Mail-Copies-To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>
} 
} Q: Is an email message containing that header-field (or should I say the
} user agent which permitted it to be sent as an email) RFC 2047-compliant?

Yes.

} A: Under Interpretation A, Yes. Because it is an extension-field which
}    satisfies the requirements of Rule 5(1).
}    
}    Under Interpretation B, No. Because the field body is not defined as
}    '*text'.

Wrong.  Under Interpretation B, yes, because the phrase is a text token
and may be encoded, but the address is not a text token and must not be
(and is not).
 
}    However, even with Interpretation B, it might get by under Rule 5(3)
}    because, under the Usefor syntax, it is within a 'phrase'.

That is the intent of 2047.
 
}    OTOH, both those views of Interpetation B seem to presuppose that the
}    user agent was familiar with the syntax of Usefor.

If the user agent is not familiar with the syntaxt of Usefor, it has no
business assigning any semantics at all to the Mail-Copies-To field, so
whether or not encoded words appear there is irrelevant.

}    Organization: =?ISO-8859-1?Q?Claus_F=E4rber Fabrik?=
} 
} Q: Is that one RFC 2047-compliant?
} 
} A: Yes, under both Interpretations A and B (though under B one might
}    wonder how the user agent was supposed to know that it was
}    unstructured).

The user agent is not supposed to know.  The user agent is not supposed
to look for or apply encodings/decodings inside header fields for which
it does not know the semantics.  All fields for which the semantics are
unknown are to be treated as unstructured upon receipt.
 
} In 6.1 I find:
} 
}    A mail reader must parse the message and body part headers according
}    to the rules in RFC 822 to correctly recognize 'encoded-word's.
} 
} Again, I see two interpretaions:
} 
} Interpretation C:
} 
} The wording "rules in RFC 822" means that only the headers explicitly
} defined in RFC 822 are required to be examined for the presence of
} 'encoded-word's.

As you noted, that's not a reasonable interpretation, because body part
headers are not described in RFC 822.

} Interpretation D:
} 
} The wording "rules in RFC 822" includes the rules for 'extension-field'
} and 'user-defined-field'.

This is the correct interpretation.

} Hence "must parse the message" means that the rules in the document
} defining the extension are to be applied.

Exactly.

} OTOH, Interpretation D seems to require that all user agents be
} magically aware of all new extension headers as soon as their defining
} documents are published.

No.  It requires only that user agents DO NOT ATTEMPT TO PARSE header
fields for which they do not know the semantics.

If you know what a header means, you may parse it and act on it.  If you
don't know, you should ignore it completely (pass it through unchanged,
hide it from the user, whatever).
 
} Thus the best interpretation I can place on section 6 is that a
} compliant mail reader MUST recognize and decode 'encoded-word's that
} occur in the headers explicitly defined in RFC 2822, and that it
} MAY/SHOULD/MUST/SOMETHING-ELSE recognize all 'encoded-word's produced by
} a compliant agent (as in section 5).

The intent of 2047 is that an application that chooses NOT to implement
2047 can still parse the resulting tokens according to their definition
in the appropriate document.  Thus most of the restrictions in 2047 are
on applications that GENERATE 2047 header fields, to assure that they
remain interoperable with non-2047-compliant applications.

Whether an application that receives a 2047 encoded-word chooses to decode
it is a quality of implementation issue.

I'm not going to attempt to address the document-authoring questions.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: Interpretation of RFC 2047 
Date: Mon, 14 Oct 2002 13:09:38 -0400
Sender: [email-masked]

> [I originally sent this last Thursday, but omitted to include a Subject.
> That may have caused several people not to see it, so here it is again.
> 
> It is sent primarily to the ietf-822 list (which is where Reply-To
> is set) because I am anxious to hear the views of the email experts.
> However, it is CCed for information to the Usefor list.]
> 
> I am having problems understanding RFC 2047. In particular, I want to
> establish what a user agent would be REQUIRED to do in order to claim
> minimal compliance with RFC 2047 (yes, I know actual implementations are
> likely to be more 'liberal' than that, but I after a fundamental
> baseline which other documents can *rely* on).
> 
> The problem splits into two parts:
> 
> 1. Where can encoded-words legitimately appear (RFC 2047 section 5).
> 2. Where are encoded-words required to be recognized (RFC 2047 section 6).
> 
> 
> 1. Where can encoded-words legitimately appear (RFC 2047 section 5).
> -------------------------------------------------------------------
> 
> In 5 (1) I find:
> 
>    An 'encoded-word' may replace a 'text' token (as defined by RFC 822)
>    in any Subject or Comments header field, any extension message header
>    field, or any MIME body part field for which the field body is
>    defined as '*text'. An 'encoded-word' may also appear in any
>    user-defined ("X-") message or body part header field.
> 
> That is ambiguous, depending on how you interpret the commas in the first
> sentence:
> 
> Interpretation A:
> 
> It means you can use an encoded-word in
> 	any Subject
> 	any Comments
> 	any extension message header field
> 	any MIME body part field for which the field body is defined as '*text'
> 	any X-header
> 
> Interpretaion B:
> 
> It means you can use an encoded-word in
> 	any Subject                        )
> 	any Comments                       ) for which the field body is
> 	any extension message header field ) defined as '*text'
> 	any MIME body part field           )
> 	any X-header

Interpretation B is intended.  

> In Usefor, that is a structured header with a pretty obvious syntax in
> which "Claus Färber" is clearly a 'phrase'. In the email version (even
> if not in the news version also) that has to be encoded as:
> 
>    Mail-Copies-To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>

only if that portion of the field is defined as 'text' (which would be
difficult because 'text' allows '<' so it wouldn't be recognized as a
terminator)  OTOH, if that portion of the mail-copies-to field isn't 
defined as 'text' you still have the burden of encoding it in ASCII somehow,
since octet 0xE4 isn't valid in an email message header.  so it would
be left to a different spec to define how 2047 applies to mail-copies-to.

> Q: Is an email message containing that header-field (or should I say the
> user agent which permitted it to be sent as an email) RFC
> 2047-compliant?

not sure if you mean before or after 2047 encoding.   2047 doesn't
forbid use in other structured fields, but neither does it require
use in structured fields that aren't listed in 2047.

>    OTOH, both those views of Interpetation B seem to presuppose that the
>    user agent was familiar with the syntax of Usefor. 

no, it presupposes that when gateways take things that aren't email
messages (this includes usenet messages) and sends them via email,
it makes them compliant with email message standards at that time.
gateways have to track standards on both sides.  that's life.

> But maybe it was
>    just a simple (non-Usefor-aware) MUA and the user had inserted that
>    header manually (as is sometimes done by users emailing directly to a
>    moderator).  So we would have the seemingly absurd situation that one
>    user agent would be compliant when sending that message, but another
>    user agent which sent that same message would be non-compliant.

it's not clear to what degree user agents are expected to prevent users
from generating invalid messages, but this isn't a 2047 issue.

> Here is another example from Usefor:
> 
>    Organization: Färber Fabrik
> 
> That is, of course, an unstructured header, and would be encoded as
> 
>    Organization: =?ISO-8859-1?Q?Claus_F=E4rber Fabrik?=
> 
> Q: Is that one RFC 2047-compliant?

I'd say probably so.  though I don't know if it's formally defined anywhere,
organization is quite naturally 'text'.

> It should be noted in passing that Rule 5(1) contains no requirement for
> an encoded-word to be preceded (or followed) by 'linear white space',
> although section 7 does seem to enforce such a requirement.

the rule is different for structured and unstructured (text) fields.

> 2. Where are encoded-words required to be recognized (RFC 2047 section 6).
> --------------------------------------------------------------------------
> 
> One would expect section 6 to require the recognition of anything that
> was allowed to appear under section 5, but that seems not to be the case
> because there is no mention of "extension message header fields".
> 
> In 6.1 I find:
> 
>    A mail reader must parse the message and body part headers according
>    to the rules in RFC 822 to correctly recognize 'encoded-word's.

this is intended to apply to structured fields - the point is that
for structured fields you actually have to parse them to distinguish
between places where encoded words are valid (e.g. a word before a 
phrase) and places where they are not valid (e.g. a word in a local-part
or a domain).

> 2. If someone is writing a standards-track document (whether for news or
> email) and wishes to introduce some new header-fields that can make use
> of RFC 2047, what does he have to say? 

just say which elements are to be encoded and interpreted according to 2047.

> 3. Is it possible to go further and introduce header-fields with
> explicit 'encoded-word's in them, for example:

yes.  I don't exactly like doing that (there's a lot of overlap between
'token' and 'encoded-word', for instance) but it's a widespread practice.

Keith

p.s. granted 2047 could use some clarification.

Date: Mon, 14 Oct 2002 13:01:50 +0100 (BST)
From: Charles Lindsey <[email-masked]>
Reply-To: [email-masked]
Subject: Interpretation of RFC 2047
To: [email-masked]
Cc: [email-masked]
Content-MD5: 51gpVTdtT5VcrGhS4ql6Wg==
Sender: [email-masked]

[I originally sent this last Thursday, but omitted to include a Subject.
That may have caused several people not to see it, so here it is again.

It is sent primarily to the ietf-822 list (which is where Reply-To
is set) because I am anxious to hear the views of the email experts.
However, it is CCed for information to the Usefor list.]

I am having problems understanding RFC 2047. In particular, I want to
establish what a user agent would be REQUIRED to do in order to claim
minimal compliance with RFC 2047 (yes, I know actual implementations are
likely to be more 'liberal' than that, but I after a fundamental
baseline which other documents can *rely* on).

The problem splits into two parts:

1. Where can encoded-words legitimately appear (RFC 2047 section 5).
2. Where are encoded-words required to be recognized (RFC 2047 section 6).

1. Where can encoded-words legitimately appear (RFC 2047 section 5).
-------------------------------------------------------------------

In 5 (1) I find:

   An 'encoded-word' may replace a 'text' token (as defined by RFC 822)
   in any Subject or Comments header field, any extension message header
   field, or any MIME body part field for which the field body is
   defined as '*text'. An 'encoded-word' may also appear in any
   user-defined ("X-") message or body part header field.

That is ambiguous, depending on how you interpret the commas in the first
sentence:

Interpretation A:

It means you can use an encoded-word in
	any Subject
	any Comments
	any extension message header field
	any MIME body part field for which the field body is defined as '*text'
	any X-header

Interpretaion B:

It means you can use an encoded-word in
	any Subject                        )
	any Comments                       ) for which the field body is
	any extension message header field ) defined as '*text'
	any MIME body part field           )
	any X-header

I am inclined to believe Interpretation A, because I if I had wanted
Interpretation B, I would have written "... any extension message header
field or any MIME body part field, for which the field body is defined
as '*text'". Of course, the difference only arises in the case of
extension message header fields.

Now suppose I am writing a standards-track document and I want to
introduce a new header-field. Under RFC 822, it would be regarded as an
"extension-field" (under 2822 it would be an "optional-field").

Let us take a specific example from Usefor (note that any news article
is potentially also an email message, because it may have been
posted-and-mailed, or it may be en route to a moderator). So I can
write:

   Mail-Copies-To: Claus Färber <[email-masked]>

In Usefor, that is a structured header with a pretty obvious syntax in
which "Claus Färber" is clearly a 'phrase'. In the email version (even
if not in the news version also) that has to be encoded as:

   Mail-Copies-To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>

Note that a news user agent has some semantic duties to perform when it
sees that header, but all an email user agent is expected to do is not
to munge or delete it, and to enable it to be displayed to the user (at
least if the user asks to see it).

Q: Is an email message containing that header-field (or should I say the
user agent which permitted it to be sent as an email) RFC
2047-compliant?

A: Under Interpretation A, Yes. Because it is an extension-field which
   satisfies the requirements of Rule 5(1).
   
   Under Interpretation B, No. Because the field body is not defined as
   '*text'.

   However, even with Interpretation B, it might get by under Rule 5(3)
   because, under the Usefor syntax, it is within a 'phrase'.

   OTOH, both those views of Interpetation B seem to presuppose that the
   user agent was familiar with the syntax of Usefor. But maybe it was
   just a simple (non-Usefor-aware) MUA and the user had inserted that
   header manually (as is sometimes done by users emailing directly to a
   moderator).  So we would have the seemingly absurd situation that one
   user agent would be compliant when sending that message, but another
   user agent which sent that same message would be non-compliant.

Here is another example from Usefor:

   Organization: Färber Fabrik

That is, of course, an unstructured header, and would be encoded as

   Organization: =?ISO-8859-1?Q?Claus_F=E4rber Fabrik?=

Q: Is that one RFC 2047-compliant?

A: Yes, under both Interpretations A and B (though under B one might
   wonder how the user agent was supposed to know that it was
   unstructured).

It should be noted in passing that Rule 5(1) contains no requirement for
an encoded-word to be preceded (or followed) by 'linear white space',
although section 7 does seem to enforce such a requirement.

2. Where are encoded-words required to be recognized (RFC 2047 section 6).
--------------------------------------------------------------------------

One would expect section 6 to require the recognition of anything that
was allowed to appear under section 5, but that seems not to be the case
because there is no mention of "extension message header fields".

In 6.1 I find:

   A mail reader must parse the message and body part headers according
   to the rules in RFC 822 to correctly recognize 'encoded-word's.

Again, I see two interpretaions:

Interpretation C:

The wording "rules in RFC 822" means that only the headers explicitly
defined in RFC 822 are required to be examined for the presence of
'encoded-word's.

Interpretation D:

The wording "rules in RFC 822" includes the rules for 'extension-field'
and 'user-defined-field'. The RFC 822 syntax for 'extension-field' is
     extension-field =
                   <Any field which is defined in a document
                    published as a formal extension to this
                    specification; none will have names beginning
                    with the string "X-">
Hence "must parse the message" means that the rules in the document
defining the extension are to be applied.

Ad Interpretation C:-

The text I quoted above refers to "message and body part headers". But
since RFC 822 does not define any body part headers, Interpretation C
would not permit any body part header field to be examined (body part
header fields are introduced in section 5.1 of RFC 2046, and are all
supposed to be of the form "Content-*"). For example, you could not
recognize an encoded-word in a "Content-Description", let alone in
"Organisation" or "Mail-Copies-To". So I cannot see how Interpretation C
could have been the intended one.

Ad Interpretation D:-

OTOH, Interpretation D seems to require that all user agents be
magically aware of all new extension headers as soon as their defining
documents are published.

Moreover, you cannot have an agent which attempts to recognize anything
that just happens to look like an encoded-word because it needs to know,
at the very least, whether some unknown header is "unstructured" or not
(i.e. is defined as '*text'). For example,
"(=?ISO-8859-1?Q?Claus_F=E4rber?=)" can occur and should be recognised
in a structured field (if it is in a context where a comment would be
allowed), but it cannot occur in an unstructured field and should
therefore be displayed in its un-decoded form (as is explained in the
examples in section 8 of RFC 2047).

Thus the best interpretation I can place on section 6 is that a
compliant mail reader MUST recognize and decode 'encoded-word's that
occur in the headers explicitly defined in RFC 2822, and that it
MAY/SHOULD/MUST/SOMETHING-ELSE recognize all 'encoded-word's produced by
a compliant agent (as in section 5).

3. And finally ...........
--------------------------

The questions I am unsure about:

1. Which of my "Interpretations" is correct, or are there other possible
Interpretations that I have missed?

2. If someone is writing a standards-track document (whether for news or
email) and wishes to introduce some new header-fields that can make use
of RFC 2047, what does he have to say? Clearly, he defines syntax that
shows whether those fields are unstructured or not, and that introduces
'phrase's and 'comment's in the proper manner. But does he have to
include a remark to the effect that "RFC 822 (or RFC 2822) is hereby
augmented to include these new header-fields, and RFC 2047 is to be
construed accordingly"?

3. Is it possible to go further and introduce header-fields with
explicit 'encoded-word's in them, for example:

    ueser-agent-header = "User-Agent" ":" 1*( product ["/" token] )
                         ; OK, it also needs to show where WS goes
    product = token / quoted-string / encoded-word

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 11 Oct 2002 16:35:32 -0400
From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Subject: Re: how to exchange EPP content using SMTP transport 
BCC:
Sender: [email-masked]

------- Blind-Carbon-Copy

To: Dave Crocker <[email-masked]>
cc: Eric Brunner-Williams in Portland Maine <[email-masked]>,
    [email-masked], brunner
Subject: Re: how to exchange EPP content using SMTP transport 
In-Reply-To: Your message of "Fri, 11 Oct 2002 12:14:48 PDT."
             <[email-masked]> 
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <[email-masked]>
Date: Fri, 11 Oct 2002 16:35:32 -0400
From: Eric Brunner-Williams in Portland Maine <[email-masked]>

Dave,

(ediint and 822 bcc'd to target discussion on the host list, provreg.)

This started as "Applicability Statement for EPP over Transport other than
TCP", with the comment:

   The original motivation for this memo is a casual figure contained in
   a presentation made by James Seng to the PROVREG WG face-to-face
   meeting at IETF-51.  This figure purported to show a loop-free store-
   and-forward transport topology, which unfortunately contained loops.

I then slogged grimly through the rainy grey ruins of connection and HOL
blocking, session semantics, authentication mechanism scaling, state, and
the semantics of EPP session/read/write commands and responses to find no
real case for transport-over-TCP, no there there, at least outside of the
ICANN sandbox, and even there, the real use case seems to be minimum-delay
exact match write races -- a QoS guarantee not actually specified in the
TCP mapping.

I next gamboled blithly over the sunny meadows of {SMTP,NEWS}/{TCP,UUCP},
and HTTP/TCP and concluded that:

>           ... the requirements for epp/smtp are the same as for any MIME
> object needing the same security services.

Once I caught MIME, everything looked like a frenchman wearing thick makeup.

I decided that it was more useful to write a Foo-over-OneTP memo than a
MetaFoo-over-ManyTP memo.

> Having all those examples is really excellent.

Thanks. Laborious. Error-certain.

> The table is interesting, but I am not clear what purpose it servces in the
> current specification.  Please clarify.

Personel taste. A weakness for taxonomy.

> what does "receiver message structures" mean?  are these the replies?

Yup.

> "Standard for the Format of ARPA Internet Text Messages"
> 
> probably should also cite rfc2822, resnick, "Internet Message Format" and
> rfc2821, klensin, "Simple Mail Transfer Protocol".

Thank. I like the oldies. I've a nit with the moderns I haven't got my head
around.

Eric

------- End of Blind-Carbon-Copy

Date: Fri, 11 Oct 2002 12:14:48 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: TribalWise
To: [email-masked], Eric Brunner-Williams in Portland Maine <[email-masked]>
CC: [email-masked]
Subject: Re: how to exchange EPP content using SMTP transport
Sender: [email-masked]

Eric,

(ediint and 822 bcc'd to target discussion on the host list, provreg.)

MANY thanks for doing this.

Comments inline.

Friday, October 11, 2002, 10:13:47 AM, you wrote:
EBWiPM>    This memo defines mechanisms to secure (sign or encrypt or sign and
EBWiPM>    encrypt) one or more EPP messages in any SMTP message.  This memo
EBWiPM>    does not define mechanism to secure (sign or encrypt or sign and
EBWiPM>    encrypt) individual XML elements within an EPP message.

Are there any requirements for epp/smtp security that are distinctive?  My
guess is that the requirements for epp/smtp are the same as for any MIME
object needing the same security services.

That is, if we say that these objects may need privacy, authentication and
signed receipt, then it should the security-related work of this specification
should be done... as long as we can cite an external reference that defines
the common way(s) to achieve those services.

>From work in IMPP I've come to believe that we need a MIMEsec specification
that is common to all MIME objects, for these common services.  (Well, ok,
signed receipt is not yet common.)

Such a document should, essentially, take existing text from various special
efforts, like ediint and impp, as you have noted.  Would this be reasonable?

If yes, then the current specification should be able to cite a MIMEsec
document, for security basic security issues.  Given that it is a transaction
service, perhaps it needs to mention something about man-in-the-middle and DOS
attacks?

Otherwise, the normative (specification) portion could then focus on
definition of the MIME type and, perhaps, profile the bulk carriage details.

Having all those examples is really excellent.

EBWiPM> 2. Background Summary
EBWiPM> 
EBWiPM>    Some language about these three registry protocols, and the
EBWiPM>    applicability of SMTP as a transport protocol.  TBD.
EBWiPM> 
EBWiPM>    This table compares the salient characteristics of EPP, RRP and SRS.

The table is interesting, but I am not clear what purpose it servces in the
current specification.  Please clarify.

EBWiPM> 
EBWiPM> 13. Receiver Message Structures
EBWiPM> 
EBWiPM>    This section is intentionally blank, pending implementation.

what does "receiver message structures" mean?  are these the replies?

EBWiPM>    [3]   Crocker, D., "Simple Mail Transfer Protocol", RFC 822, STD 11,
EBWiPM>          August 1982.

"Standard for the Format of ARPA Internet Text Messages"

probably should also cite rfc2822, resnick, "Internet Message Format" and
rfc2821, klensin, "Simple Mail Transfer Protocol".

d/

To: [email-masked], [email-masked], [email-masked]
cc: [email-masked]
Subject: how to exchange EPP content using SMTP transport
Date: Fri, 11 Oct 2002 13:13:47 -0400
From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Sender: [email-masked]

Appologies for the cross-posting.

As Jon Crowcroft wrote to the ed2end list Monday, this is somewhat 1/2
baked, but comments are welcome.

Available via ftp from [216.220.241.233]
	user=anonymous
	pswd=guest (or whatever)
	file=pub/draft-brunner-epp-smtp-00.txt,html

Comments to me please, I like writing acknowledgements.

Eric

Date: Thu, 10 Oct 2002 15:58:08 +0100 (BST)
From: Charles Lindsey <[email-masked]>
Reply-To: [email-masked]
To: [email-masked]
Cc: [email-masked]
Content-MD5: PXgnG6vPMrveJAehiGLh5Q==
Sender: [email-masked]

{This message is sent primarily to the ietf-822 mailing list, because I
want to hear the views of the mail experts. But it is Cced also to the
usefor list. Reply-To is set to ietf-822.}

I am having problems understanding RFC 2047. In particular, I want to
establish what a user agent would be REQUIRED to do in order to claim
minimal compliance with RFC 2047 (yes, I know actual implementations are
likely to be more 'liberal' than that, but I after a fundamental
baseline which other documents can *rely* on).

The problem splits into two parts:

1. Where can encoded-words legitimately appear (RFC 2047 section 5).
2. Where are encoded-words required to be recognized (RFC 2047 section 6).

1. Where can encoded-words legitimately appear (RFC 2047 section 5).
-------------------------------------------------------------------

In 5 (1) I find:

   An 'encoded-word' may replace a 'text' token (as defined by RFC 822)
   in any Subject or Comments header field, any extension message header
   field, or any MIME body part field for which the field body is
   defined as '*text'. An 'encoded-word' may also appear in any
   user-defined ("X-") message or body part header field.

That is ambiguous, depending on how you interpret the commas in the first
sentence:

Interpretation A:

It means you can use an encoded-word in
	any Subject
	any Comments
	any extension message header field
	any MIME body part field for which the field body is defined as '*text'
	any X-header

Interpretaion B:

It means you can use an encoded-word in
	any Subject                        )
	any Comments                       ) for which the field body is
	any extension message header field ) defined as '*text'
	any MIME body part field           )
	any X-header

I am inclined to believe Interpretation A, because I if I had wanted
Interpretation B, I would have written "... any extension message header
field or any MIME body part field, for which the field body is defined
as '*text'". Of course, the difference only arises in the case of
extension message header fields.

Now suppose I am writing a standards-track document and I want to
introduce a new header-field. Under RFC 822, it would be regarded as an
"extension-field" (under 2822 it would be an "optional-field").

Let us take a specific example from Usefor (note that any news article
is potentially also an email message, because it may have been
posted-and-mailed, or it may be en route to a moderator). So I can
write:

   Mail-Copies-To: Claus Färber <[email-masked]>

In Usefor, that is a structured header with a pretty obvious syntax in
which "Claus Färber" is clearly a 'phrase'. In the email version (even
if not in the news version also) that has to be encoded as:

   Mail-Copies-To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>

Note that a news user agent has some semantic duties to perform when it
sees that header, but all an email user agent is expected to do is not
to munge or delete it, and to enable it to be displayed to the user (at
least if the user asks to see it).

Q: Is an email message containing that header-field (or should I say the
user agent which permitted it to be sent as an email) RFC
2047-compliant?

A: Under Interpretation A, Yes. Because it is an extension-field which
   satisfies the requirements of Rule 5(1).
   
   Under Interpretation B, No. Because the field body is not defined as
   '*text'.

   However, even with Interpretation B, it might get by under Rule 5(3)
   because, under the Usefor syntax, it is within a 'phrase'.

   OTOH, both those views of Interpetation B seem to presuppose that the
   user agent was familiar with the syntax of Usefor. But maybe it was
   just a simple (non-Usefor-aware) MUA and the user had inserted that
   header manually (as is sometimes done by users emailing directly to a
   moderator).  So we would have the seemingly absurd situation that one
   user agent would be compliant when sending that message, but another
   user agent which sent that same message would be non-compliant.

Here is another example from Usefor:

   Organization: Färber Fabrik

That is, of course, an unstructured header, and would be encoded as

   Organization: =?ISO-8859-1?Q?Claus_F=E4rber Fabrik?=

Q: Is that one RFC 2047-compliant?

A: Yes, under both Interpretations A and B (though under B one might
   wonder how the user agent was supposed to know that it was
   unstructured).

It should be noted in passing that Rule 5(1) contains no requirement for
an encoded-word to be preceded (or followed) by 'linear white space',
although section 7 does seem to enforce such a requirement.

2. Where are encoded-words required to be recognized (RFC 2047 section 6).
--------------------------------------------------------------------------

One would expect section 6 to require the recognition of anything that
was allowed to appear under section 5, but that seems not to be the case
because there is no mention of "extension message header fields".

In 6.1 I find:

   A mail reader must parse the message and body part headers according
   to the rules in RFC 822 to correctly recognize 'encoded-word's.

Again, I see two interpretaions:

Interpretation C:

The wording "rules in RFC 822" means that only the headers explicitly
defined in RFC 822 are required to be examined for the presence of
'encoded-word's.

Interpretation D:

The wording "rules in RFC 822" includes the rules for 'extension-field'
and 'user-defined-field'. The RFC 822 syntax for 'extension-field' is
     extension-field =
                   <Any field which is defined in a document
                    published as a formal extension to this
                    specification; none will have names beginning
                    with the string "X-">
Hence "must parse the message" means that the rules in the document
defining the extension are to be applied.

Ad Interpretation C:-

The text I quoted above refers to "message and body part headers". But
since RFC 822 does not define any body part headers, Interpretation C
would not permit any body part header field to be examined (body part
header fields are introduced in section 5.1 of RFC 2046, and are all
supposed to be of the form "Content-*"). For example, you could not
recognize an encoded-word in a "Content-Description", let alone in
"Organisation" or "Mail-Copies-To". So I cannot see how Interpretation C
could have been the intended one.

Ad Interpretation D:-

OTOH, Interpretation D seems to require that all user agents be
magically aware of all new extension headers as soon as their defining
documents are published.

Moreover, you cannot have an agent which attempts to recognize anything
that just happens to look like an encoded-word because it needs to know,
at the very least, whether some unknown header is "unstructured" or not
(i.e. is defined as '*text'). For example,
"(=?ISO-8859-1?Q?Claus_F=E4rber?=)" can occur and should be recognised
in a structured field (if it is in a context where a comment would be
allowed), but it cannot occur in an unstructured field and should
therefore be displayed in its un-decoded form (as is explained in the
examples in section 8 of RFC 2047).

Thus the best interpretation I can place on section 6 is that a
compliant mail reader MUST recognize and decode 'encoded-word's that
occur in the headers explicitly defined in RFC 2822, and that it
MAY/SHOULD/MUST/SOMETHING-ELSE recognize all 'encoded-word's produced by
a compliant agent (as in section 5).

3. And finally ...........
--------------------------

The questions I am unsure about:

1. Which of my "Interpretations" is correct, or are there other possible
Interpretations that I have missed?

2. If someone is writing a standards-track document (whether for news or
email) and wishes to introduce some new header-fields that can make use
of RFC 2047, what does he have to say? Clearly, he defines syntax that
shows whether those fields are unstructured or not, and that introduces
'phrase's and 'comment's in the proper manner. But does he have to
include a remark to the effect that "RFC 822 (or RFC 2822) is hereby
augmented to include these new header-fields, and RFC 2047 is to be
construed accordingly"?

3. Is it possible to go further and introduce header-fields with
explicit 'encoded-word's in them, for example:

    ueser-agent-header = "User-Agent" ":" 1*( product ["/" token] )
                         ; OK, it also needs to show where WS goes
    product = token / quoted-string / encoded-word

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 08 Oct 2002 13:48:25 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: [email-masked]
Subject: Re: empty quoted strings and other oddities
Sender: [email-masked]

Jacob Palme wrote:

> Should you for example accept
> From: John F. Kennedy <[email-masked]>
> 
> Standards have a long time accepted the "K." even
> if the user-friendly name is not quoted, but implementors
> have not been told that accepting such a "." followed
> by space is in the liberal category of what you should
> accept but not produce.

That is in the obs-phrase syntax of RFC 2822.  It is a
bit unusual compared to most of the rest of the obs-
syntax; it was *not* allowed under 822's rules and is
still a MUST NOT generate case, but there's a note in
section 4.1 that implies that it may be formally permitted
in the future.  And of course, 2822-conforming implementations
of receivers MUST recognize it as a valid phrase.

IMO, formally permitting generation would be good.  While

   Wilt "The Stilt" Chamberlain

is a valid phrase and is as would normally be written in
text not neccessarily specific to header formats,

   Bullwinkle J"." Moose
   Bullwinkle "J." Moose
   Bullwinkle "J, Moose"
   "Bullwinkle J." Moose
   "Bullwinkle J. Moose"
   "Bullwinkle" "J""." "Moose"
   ...
   Bullwinkle =?us-ascii*en-us?Q?J=2E?= Moose

are valid phrases under the RFC 2822 generate syntax rules
but are unlikely to be typed as such by a user. In the
first case the double quotes are part of the name, whereas
in the variants of the second example which use double quotes
they are mere syntactic sugar coating [the 2047/2231 encoded
variant is more like syntactic scaffolding in the Gothic
tradition].

I.e. permitting a phrase with an unquoted dot would be
a recognition of existing practice which serves user desires.
Unlike some of the relaxations of syntax permitted by 2822,
such as the oxymoronic but 2822-valid (under obs- rules)
time-stamp line

   Received:

That was not legal under 821/822 syntax, and there's no point
in a time-stamp line with no time stamp.

Date: 08 Oct 2002 08:57:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: empty quoted strings and other oddities
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 02.10.02 in <[email-masked]>:

> > > so? there are lots of addresses which are valid syntax but
> > > not valid because the domain is not defined or the IP
> > > address (in the domain literal) does not exist or is not
> > > assigned to a host.
> >
> > I'm not sure I buy that reasoning.  You seem to be saying that since it's
> > not possible to detect all invalid addresses syntactically that the
> > ability to trap some syntactically (or more than currently trapped) is
> > irrelevant.
>
> no, I'm saying that
>
> - there is useful separation of function here between detecting syntax
>   errors and detecting addresses that are invalid for other reasons that
>   should not be discarded

Aren't you both missing a point here?

These particular bad domain literals certainly *should* be detected  
syntactically - not by the SMTP checker, but by the convert-to-IP-address  
routine. (Or by the convert-to-other-transport-address routine, if  
applicable. And then it might even be legal.)

That is exactly the same case as for the "" localpart - only those parties  
which actually need to understand it should decide if it is syntactically  
valid, the rest of the world can just as well handle this as an opaque  
token.

MfG Kai

Date: 08 Oct 2002 09:02:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: empty quoted strings and other oddities
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Bruce Lilly)  wrote on 03.10.02 in <[email-masked]>:

> An MTA or MUA that does not handle [] well may indeed cause
> problems.  That might range from (multiple) DNS lookups of
> garbage, or malformed DNS lookups to a program crash.

How exactly would that happen? I cannot even imagine the chain of bugs  
(which would presumably be libc or resolver bugs, anyway) which would lead  
to that consequence.

MfG Kai

Date: Mon, 07 Oct 2002 09:54:15 -0700 (PDT)
From: [email-masked]
Subject: Re: empty quoted strings and other oddities
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <p0510031db9c2ed7c528f@[130.237.161.114]> Jacob Palme <[email-masked]> writes:

> >In early mail systems, it was allowed to specify a
> >recipient address with only the local name. This would
> >then default to the sender's domain. Thus,

> >From: [email-masked]
> >To: Mary

> >would default to

> >From: [email-masked]
> >To: [email-masked]

> >This is not allowed any more, but some local UA-MTA
> >might still allow this technique.

> Are you sure this is not allowed? My local UA-MTA (dtmail-sendmail)
> certainly allows if, and sendmail.cf automatically cleans it up before
> letting it be seen in the outside world.

Do not confuse the myriad illegal formats allowed and fixed up by sendmail
(or most any other MTA for that matter) with 

> The incomplete form never gets seen on any "wire". So is any standard
> violated in that case?

There's considerable value in having a standardized submission protocol in
addition to having a standardized transfer protocol. RFC 2476 provided the
first step along this path.

				Ned

Date: Mon, 7 Oct 2002 14:17:16 +0200
To: IETF mailing list on MIME and e-mail  <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Sender: [email-masked]

At 09:41 +0000 02-10-07, Charles Lindsey wrote:
>The incomplete form never gets seen on any "wire". So is any standard
>violated in that case?

Well, the communication between a client and the first
MTA also goes over some kind of wire. Standards are
needed so that products of different manufacture
can co-work, and since client and the first MTA are
often of different manufacture, this should be clarified
in the standard.

I quote the RFC 2822 syntax below, it does not seem
to allow a local-part without a domain ("obs-addr-list"
does not add this format either):

address         =       mailbox / group

mailbox         =       name-addr / addr-spec

name-addr       =       [display-name] angle-addr

angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr

group           =       display-name ":" [mailbox-list / CFWS] ";"
                         [CFWS]

display-name    =       phrase

mailbox-list    =       (mailbox *("," mailbox)) / obs-mbox-list

address-list    =       (address *("," address)) / obs-addr-list

to              =       "To:" address-list CRLF

addr-spec       =       local-part "@" domain

local-part      =       dot-atom / quoted-string / obs-local-part

domain          =       dot-atom / domain-literal / obs-domain

-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

To: [email-masked]
Xref: clerew local.mime:1880
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Date: Mon, 7 Oct 2002 09:41:32 GMT
Lines: 34
Sender: [email-masked]

In <p0510031db9c2ed7c528f@[130.237.161.114]> Jacob Palme <[email-masked]> writes:

>In early mail systems, it was allowed to specify a
>recipient address with only the local name. This would
>then default to the sender's domain. Thus,

>From: [email-masked]
>To: Mary

>would default to

>From: [email-masked]
>To: [email-masked]

>This is not allowed any more, but some local UA-MTA
>might still allow this technique.

Are you sure this is not allowed? My local UA-MTA (dtmail-sendmail)
certainly allows if, and sendmail.cf automatically cleans it up before
letting it be seen in the outside world.

The incomplete form never gets seen on any "wire". So is any standard
violated in that case?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Sat, 5 Oct 2002 05:50:42 +0200
To: "Gary Feldman" <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Cc: <[email-masked]>
Sender: [email-masked]

At 07:54 -0400 02-10-04, Gary Feldman wrote:
>I don't understand the question.  Computer science students have
>long been taught that they must reasonably handle invalid input, at
>the same time that they document precisely what the input must be.
>How is this different from web servers still accepting http 1.0 or
>C compilers still accepting K&R C?

The problem is that the golden rule actually specifies three
standards:

A conservative standard, which implementors should use when
producing new data.

A liberal standard which implementors should use when
accepting incoming data.

The real standards text, which is somewhat inbetween.

It is difficult for an implementor to understand the
difference between these three standards, since most
standards text does not say what is the difference
between them.

Should you for example accept
From: John F. Kennedy <[email-masked]>

Standards have a long time accepted the "K." even
if the user-friendly name is not quoted, but implementors
have not been told that accepting such a "." followed
by space is in the liberal category of what you should
accept but not produce.

The obs- syntax of RFC 2822 is an attempt at clarifying
the difference between these three standards.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

From: Keith Moore <[email-masked]>
To: Bruce Campbell <[email-masked]>
cc: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Fri, 04 Oct 2002 14:49:56 -0400
Sender: [email-masked]

> > > any software
> > > generating a domain literal would need to take some appropriate
> > > steps.
> >
> > has anyone here actually seen a significant operational problem
> > associated with this?
> 
> 'Not yet', but give the spammers time (although I don't think they've
> finished making '<>' fully useless yet).

spammers might or might not try using ""@[] but whether they do so
will not affect their ability to send spam.

Date: Fri, 4 Oct 2002 20:23:52 +0200 (CEST)
From: Bruce Campbell <[email-masked]>
To: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Sender: [email-masked]

On Fri, 4 Oct 2002, Keith Moore wrote:

> > any software
> > generating a domain literal would need to take some appropriate
> > steps.
>
> has anyone here actually seen a significant operational problem
> associated with this?

'Not yet', but give the spammers time (although I don't think they've
finished making '<>' fully useless yet).

--==--
Bruce.

From: Keith Moore <[email-masked]>
To: "Gary Feldman" <[email-masked]>
cc: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Fri, 04 Oct 2002 10:49:40 -0400
Sender: [email-masked]

> Let me also add that the idea of a huge impact on thousands or millions of
> users because of a minor change is overblown.

the harm done by not explicitly forbidding [] is also overblown.

Keith

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Fri, 04 Oct 2002 10:46:53 -0400
Sender: [email-masked]

> > second, your conclusion doesn't follow even if
> > it is a boundary condition - all that is necessary is that it not cause
> > any problems.
> 
> An MTA or MUA that does not handle [] well may indeed cause
> problems.

sure, but the same is true of an MTA or MUA that doesn't handle
any invalid address well.  on the other hand, passing the "" 
within the [] to inet_aton() should return an error code for invalid
address,  just as for any other invalid address.  if that's not the
case, the problem is in the system library, not the MTA or MUA.

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Fri, 04 Oct 2002 10:44:24 -0400
Sender: [email-masked]

> any software
> generating a domain literal would need to take some appropriate
> steps.

look, this is ridiculous.  checking for this one special case
solves such an infintesimal number of problems that it's really
not worth doing.

has anyone here actually seen a significant operational problem 
associated with this?

From: "Gary Feldman" <[email-masked]>
To: <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Date: Fri, 4 Oct 2002 07:54:17 -0400
Sender: [email-masked]

>From: "Jacob Palme" <[email-masked]>
>Sent: Friday, October 04, 2002 3:08 AM

> I have a long time complained that IETF standards did not
> tell implementors how to practically implement this golden
> rule. What is the difference between the conservative
> standards interpretation (what you should send) and the
> liberal standards interpretation (what you should accept).

I don't understand the question.  Computer science students have
long been taught that they must reasonably handle invalid input, at
the same time that they document precisely what the input must be.
How is this different from web servers still accepting http 1.0 or
C compilers still accepting K&R C?

Gary

From: "Gary Feldman" <[email-masked]>
To: <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Date: Fri, 4 Oct 2002 07:51:24 -0400
Sender: [email-masked]

I think Bruce has correctly pointed out that there is an existing mechanism
whose
precise purpose in life is to accomodate such obsolescence changes.

Let me also add that the idea of a huge impact on thousands or millions of
users
because of a minor change is overblown.  People don't wake up one morning
and
discover there's a new RFC or an RFC in a new state, delete their existing
copies
of Outlook or Eudora or sendmail or whatever, and sit around without
email for weeks or months waiting for the world to catch up.

What happens is that the folks at Qualcomm and Microsoft and the open source
community and whereever else put out new releases.  In some cases, they use
it
as a selling point to make more money.  In other cases, such as the open
source
community, they use it to help prove their faithfulness and quality.  They
do so
without breaking existing stuff catastrophically, just as existing web
servers still
support HTTP 1.0 in spite of incompatibilities with HTTP 1.1.

Eventually things get to be so obsolete that people are forced to upgrade,
e.g.
version 1 of Netscape or IE.  Or they continue to work with rough edges,
e.g.
the folks still using version 4.  But the world doesn't end and email won't
break
down.

Gary

Date: Fri, 4 Oct 2002 09:08:38 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Sender: [email-masked]

At 19:02 -0400 02-10-03, Bruce Lilly wrote:
>The suggestion was to relegate empty domain literals to an obs-
>contruct, which means that parsing would not change, but it would
>be come a MUST NOT generate construct.  There need not be any
>syntax checks for any parser w.r.t. this issue; any software
>generating a domain literal would need to take some appropriate
>steps.
>
>Moving it to obs- syntax doesn't kill it, but it does declare it
>as a wart.

One of the IETF golden rules is "Be conservative in what you
produce, be liberal in what you accept". According to this
rule, it is obvious that certain uses of empty strings are
things you should accept in input, but avoid producing.

I have a long time complained that IETF standards did not
tell implementors how to practically implement this golden
rule. What is the difference between the conservative
standards interpretation (what you should send) and the
liberal standards interpretation (what you should accept).

The "obs-" syntax of RFC 2822 is an attempt to codify this,
which I think is good. However, one can discuss whether
all "obs-" constructs have to be "MUST NOT". In some cases,
it may be reasonable to allow production of them. Whether
this applies to empty strings, I am not sure.

--- ---

In early mail systems, it was allowed to specify a
recipient address with only the local name. This would
then default to the sender's domain. Thus,

From: [email-masked]
To: Mary

would default to

From: [email-masked]
To: [email-masked]

This is not allowed any more, but some local UA-MTA
might still allow this technique.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Thu, 03 Oct 2002 19:19:52 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: empty quoted strings and other oddities
Sender: [email-masked]

Keith Moore wrote:

> first of all, it's really not a boundary condition - it's just one example 
> of an invalid address

It's a boundary condition as it's a unique case (zero content).

 > second, your conclusion doesn't follow even if
> it is a boundary condition - all that is necessary is that it not cause
> any problems.

An MTA or MUA that does not handle [] well may indeed cause
problems.  That might range from (multiple) DNS lookups of
garbage, or malformed DNS lookups to a program crash.

> well, it's not like those thousands of existing implementations of email
> software are going to change their implementation techniques just to
> accomodate this case.  

If it becomes a MUST NOT generate construct, damage can be
prevented even if an existing defective implementation of
a receiver does not change.

> as for use of lex, I've tried it.  it's not a good fit.  2822 syntax analysis 
> is somewhat context-dependent (is "Mon" a day or an atom?),  good error 
> recovery is difficult, and you need to have the lexical analyzer recover from
> common syntax errors (such as putting dot in a phrase) in situations where
> look-ahead is required to disambiguate that kind of error from other 
> errors.
> 
> Having done both, I can confidently say that it's easier to write correct 
> C code to do the scanning than to write correct lex code to do it.

Having done both, I've come to the opposite conclusion, but
that's a bit off topic.

> adding cruft to the standard (or to code) to detect cases which
> never happen is not progress.

We're not talking about detection, but rather about relegating
a useless and potentially troublesome construct to MUST NOT
generate status.

> progress is increasing the reliability
> of mail delivery.

Is that relevant to the message format (as opposed to transport
methods and protocols)?

Date: Thu, 03 Oct 2002 19:02:35 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: [email-masked]
Subject: Re: empty quoted strings and other oddities
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Well, assume for the sake of argument that it's relevant. Is it also
> valuable enough to justify changing email after 25 years of production
> use, with a user base presumably in the hundreds of millions, using god
> knows how many thousands of different programs?
> 
> I personally think this particular syntax wart should've been killed at
> birth... but now it's too late.
[...]
> Each extra syntax check involves writing at least one more test in the
> code, at least two more test cases to verify the code's correctness, some
> documentation and some (UI and documentation) translations. And then
> there's the (slightly) increased cost of learning the code for the
> subsequent maintainers, and the UI for the users.

The suggestion was to relegate empty domain literals to an obs-
contruct, which means that parsing would not change, but it would
be come a MUST NOT generate construct.  There need not be any
syntax checks for any parser w.r.t. this issue; any software
generating a domain literal would need to take some appropriate
steps.

Moving it to obs- syntax doesn't kill it, but it does declare it
as a wart.

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Thu, 03 Oct 2002 09:18:19 -0400
Sender: [email-masked]

> >well, what is valid for the message format is not necessarily
> >valid for SMTP - since the message format is transport-independent
> >while SMTP is associated with specific transports.
> 
> Indeed. In some transport systems, the domain literal might be interpreted
> as the serial number of the loft that the carrier pigeon was supposed to
> fly to.

nah, they could implment RFC 1149 and use IP address literals.

Keith

From: Keith Moore <[email-masked]>
To: "Gary Feldman" <[email-masked]>
cc: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Thu, 03 Oct 2002 09:16:19 -0400
Sender: [email-masked]

> The main problem with this analysis is that it ignores the comparison with
> semantic side catching of the problem.  Also, the issue at hand is a
> boundary condition, and hence the test cases need to be there, regardless 
> of how it's defined.

first of all, it's really not a boundary condition - it's just one example 
of an invalid address.   second, your conclusion doesn't follow even if
it is a boundary condition - all that is necessary is that it not cause
any problems.

(""@example.com is not inherently invalid, nor should we restrict strings
to having 1 or more characters - because there are cases where some strings 
need to be zero length)

> Finally, whether or not it actually causes that extra effort and cost
> of learning depends on how it's implemented in the first place.  

well, it's not like those thousands of existing implementations of email
software are going to change their implementation techniques just to
accomodate this case.  

as for use of lex, I've tried it.  it's not a good fit.  2822 syntax analysis 
is somewhat context-dependent (is "Mon" a day or an atom?),  good error 
recovery is difficult, and you need to have the lexical analyzer recover from
common syntax errors (such as putting dot in a phrase) in situations where
look-ahead is required to disambiguate that kind of error from other 
errors.

Having done both, I can confidently say that it's easier to write correct 
C code to do the scanning than to write correct lex code to do it.

> Granted, many implementations won't be so eloquent, but again, the 
> extent to which that should be allowed to hamper progress should be limited.

adding cruft to the standard (or to code) to detect cases which
never happen is not progress.  progress is increasing the reliability
of mail delivery.  if you want to make that kind of progress, you
need to do something about stupid spam filtering, DNS misconfiguration,
and MTA misconfiguration.  of course, it's much easier to think of 
solutions to problems that don't exist.

Keth

To: [email-masked]
Xref: clerew local.mime:1866
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Date: Thu, 3 Oct 2002 10:00:40 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> Also, in this case 2822 seems to be alone in permitting
>> that syntax, which is not permitted by 2821 or 1123.

>well, what is valid for the message format is not necessarily
>valid for SMTP - since the message format is transport-independent
>while SMTP is associated with specific transports.

Indeed. In some transport systems, the domain literal might be interpreted
as the serial number of the loft that the carrier pigeon was supposed to
fly to.

And jbloggs@[] could be taken as an indication to the transport (not an
RFC 2821-compliant one, of course) that "I haven't a clue how jbloggs is
to be reached, but if you can figure out a way to get it to him, then
please do so".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "Gary Feldman" <[email-masked]>
To: <[email-masked]>
Subject: Re: empty quoted strings and other oddities
Date: Wed, 2 Oct 2002 22:28:34 -0400
Sender: [email-masked]

>From: "Arnt Gulbrandsen" <[email-masked]>
>Sent: Wednesday, October 02, 2002 10:36 AM

>
> Well, assume for the sake of argument that it's relevant. Is it also
> valuable enough to justify changing email after 25 years of production
> use, with a user base presumably in the hundreds of millions, using god
> knows how many thousands of different programs?

I don't buy that argument either.  People upgrade routinely.  Supporting
obscure and
old programs, while not entirely a non-issue, nevertheless should have a
limited and
controlled ability to hamper progress.

> Each extra syntax check involves writing at least one more test in the
> code, at least two more test cases to verify the code's correctness, some
> documentation and some (UI and documentation) translations. And then
> there's the (slightly) increased cost of learning the code for the
> subsequent maintainers, and the UI for the users.

The main problem with this analysis is that it ignores the comparison with
semantic side catching of the problem.  Also, the issue at hand is a
boundary
condition, and hence the test cases need to be there, regardless of how it's
defined.  Finally, whether or not it actually causes that extra effort and
cost
of learning depends on how it's implemented in the first place.  It could,
for
example, be done with (a descendant of) lex, or a homegrown table driven
lexical analyzer, or with a Perl pattern, or any number of other
well-structured
lexical or syntactic analysis techniques.  Granted, many implementations
won't
be so eloquent, but again, the extent to which that should be allowed to
hamper progress should be limited.

I apologize for the depth of this digression.  I realize that it's far more
an
issue of philosophy than the relatively minor issue of this point of syntax.

Gary

Date: Wed, 2 Oct 2002 16:36:10 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: empty quoted strings and other oddities
Sender: [email-masked]

Gary Feldman <[email-masked]>
> I'm not sure I buy that reasoning.  You seem to be saying that since it's
> not possible to detect all invalid addresses syntactically that the ability
> to trap some syntactically (or more than currently trapped) is irrelevant.

Well, assume for the sake of argument that it's relevant. Is it also
valuable enough to justify changing email after 25 years of production
use, with a user base presumably in the hundreds of millions, using god
knows how many thousands of different programs?

I personally think this particular syntax wart should've been killed at
birth... but now it's too late.

> I disagree.  I believe it's usually faster, simpler, and more reliable to
> catch errors syntactically than relying on subsequent semantics.

Usually?

Each extra syntax check involves writing at least one more test in the
code, at least two more test cases to verify the code's correctness, some
documentation and some (UI and documentation) translations. And then
there's the (slightly) increased cost of learning the code for the
subsequent maintainers, and the UI for the users.

Add it up¹ and multiply it by the number of MUAs/MTAs in use. What you get
is the cost of retrofitting this check now.

--Arnt

1: Believe in the goodness of your fellow man. Pretend that a significant
   portion of those MUAs/MTAs have test cases and documentation. ;)

From: Keith Moore <[email-masked]>
To: "Gary Feldman" <[email-masked]>
cc: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Wed, 02 Oct 2002 09:06:13 -0400
Sender: [email-masked]

> > so? there are lots of addresses which are valid syntax but
> > not valid because the domain is not defined or the IP
> > address (in the domain literal) does not exist or is not
> > assigned to a host.
> 
> I'm not sure I buy that reasoning.  You seem to be saying that since it's
> not possible to detect all invalid addresses syntactically that the ability
> to trap some syntactically (or more than currently trapped) is irrelevant.

no, I'm saying that 

- there is useful separation of function here between detecting syntax
  errors and detecting addresses that are invalid for other reasons that 
  should not be discarded 

- being able to detect that a few very-seldom-used addresses are
  illegal by means of syntax is of infintesimal value

- trying to detect a significant number of invalid addresses by means
  of syntax is likely to both increase errors (because there are more
  things that can go wrong) and to impair the utility of email addresses 
  at some future date.  

in other words, it's a fool's errand.

Keith

From: "Gary Feldman" <[email-masked]>
To: <[email-masked]>
Subject: Re: empty quoted strings and other oddities 
Date: Wed, 2 Oct 2002 08:33:36 -0400
Sender: [email-masked]

>From: "Keith Moore" <[email-masked]>
>Sent: Tuesday, October 01, 2002 11:13 AM

> > Likewise, a domain literal can consist solely of the
> > square brackets.
>
> so? there are lots of addresses which are valid syntax but
> not valid because the domain is not defined or the IP
> address (in the domain literal) does not exist or is not
> assigned to a host.

I'm not sure I buy that reasoning.  You seem to be saying that since it's
not possible to detect all invalid addresses syntactically that the ability
to trap some syntactically (or more than currently trapped) is irrelevant.
I disagree.  I believe it's usually faster, simpler, and more reliable to
catch errors syntactically than relying on subsequent semantics.

Gary

From: Keith Moore <[email-masked]>
To: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Wed, 02 Oct 2002 08:26:10 -0400
Sender: [email-masked]

> >>Likewise, a domain literal can consist solely of the
> >>square brackets.
> > 
> > 
> > so? there are lots of addresses which are valid syntax but
> > not valid because the domain is not defined or the IP
> > address (in the domain literal) does not exist or is not
> > assigned to a host.
> 
> I'm not sure that there aren't MTAs or MUAs that would
> break or pass bizarre requests to DNS when trying to
> deal with [] as a domain literal.

if the M*A fails to deliver the message because of "no such host"
or some similar reason, that's fine.  if the M*A fails claiming
that the address is invalid syntax, it's broken.
not that this bug is likely to make a difference in practice...

> Also, in this case 2822 seems to be alone in permitting
> that syntax, which is not permitted by 2821 or 1123.

well, what is valid for the message format is not necessarily
valid for SMTP - since the message format is transport-independent
while SMTP is associated with specific transports.

Date: Wed, 02 Oct 2002 06:37:20 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
CC: [email-masked]
Subject: Re: empty quoted strings and other oddities
Sender: [email-masked]

Keith Moore wrote:
>>RFC 2822 permits (as did 822) a quoted string which
>>consists solely of a pair of DQUOTE characters.  It's
>>not clear what the semantics of that are or ought to be.
> 
> 
> it doesn't have to have semantics, since quoted-strings
> can be used in places where they are opaque.  if someone wants their 
> email address to be ""@example.com this is perfectly legal.

OK, I'll buy that (at least 2821 and 2822 are consistent
in permitting it).

>>Likewise, a domain literal can consist solely of the
>>square brackets.
> 
> 
> so? there are lots of addresses which are valid syntax but 
> not valid because the domain is not defined or the IP
> address (in the domain literal) does not exist or is not
> assigned to a host.

I'm not sure that there aren't MTAs or MUAs that would
break or pass bizarre requests to DNS when trying to
deal with [] as a domain literal.

Also, in this case 2822 seems to be alone in permitting
that syntax, which is not permitted by 2821 or 1123.

From: Keith Moore <[email-masked]>
To: [email-masked]
Subject: Re: empty quoted strings and other oddities 
Date: Tue, 01 Oct 2002 11:13:46 -0400
Sender: [email-masked]

> RFC 2822 permits (as did 822) a quoted string which
> consists solely of a pair of DQUOTE characters.  It's
> not clear what the semantics of that are or ought to be.

it doesn't have to have semantics, since quoted-strings
can be used in places where they are opaque.  if someone wants their 
email address to be ""@example.com this is perfectly legal.

> Likewise, a domain literal can consist solely of the
> square brackets.

so? there are lots of addresses which are valid syntax but 
not valid because the domain is not defined or the IP
address (in the domain literal) does not exist or is not
assigned to a host.

Keith