Subject: [EAI] clarify Received header format

From: [email-masked]
Date: Tue, 02 May 2006 18:09:48 +0900 (JST)

draft-yao-ima-smtpext-02.txt section 2.5.2.  Trace Fields is ambiguous.

  Addresses in "for" clauses need further examination and might be
  treated differently depending on [IMA-utf8header].  The reasoning
  in the introductory portion of [IMA-overview] strongly suggests
  that these addresses be in UTF-8 form, rather than some
  specialized encoding.

This Received header format may be defined in utf8header draft and
smtpext draft refer it.

Without clear Received header definition, implementation is impossible.

In my opinion, addresses in "for" clauses should be IMA itself and
should be UTF-8 encoded.

And more, I want to record ALT-ADDR, ATOMIC parameter information.

"i18n-email: 1.0" header may be added by MTA
as a result of adding Received header or IMA-Downgraded-* header.

--
Kazunori Fujiwara, JPRS

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Tue, 02 May 2006 07:17:07 -0400

--On Tuesday, 02 May, 2006 18:09 +0900 [email-masked] wrote:

>   Addresses in "for" clauses need further examination and
> might be   treated differently depending on [IMA-utf8header].
> The reasoning   in the introductory portion of [IMA-overview]
> strongly suggests   that these addresses be in UTF-8 form,
> rather than some   specialized encoding.
> 
> This Received header format may be defined in utf8header draft
> and smtpext draft refer it.
> 
> In my opinion, addresses in "for" clauses should be IMA itself
> and should be UTF-8 encoded.

FWIW, this is my fault from a very early draft.  I started
working through the case analyses (not just my desires) and came
to the conclusion that the problem was hard and there was no
clear answer.  There are clear advantages to having the
in-transit MTA insert only pure-ASCII trace fields into the
headers.  For example, suppose a trace field containing UTF-8
information is to be inserted and we are to require a header
field that indicates that UTF-8 information is present. The MTA
must perform a header scan to determine whether that
UTF-8-indicating header is present and add it if it is not.  We
generally try to avoid requiring MTAs to perform such header
scans -- the historical model says that that, other than
submission and delivery MTAs, they don't do it.   This is,
incidentally, one of several reasons I've been reluctant to
agree to the requirement for such a header field -- it does have
side-effects.

Since we haven't, I think, agreed on the syntax or semantics of
that UTF-8-indicating header field yet, I don't know if one is
supposed to supply it when the SMTP extension is used at all,
even if the headers don't actually contain UTF-8 information.
That decision obviously interacts with the discussion above.

Since "for" is optional, one possibility would be to carry "for"
information in UTF-8 up to the point that downgrade needed to
occur, then drop it from all trace fields in which it appears.
Of course, that requires changing existing trace fields,
something else we prohibit (and turning "downgrading" into even
more of a gateway/ protocol converter -like operation).  I
believe changing "for <UTF-8-address>" into "for
<alternate-address>" in a _previous_ header to be a bad idea --
the "for" field should reflect, IMO, the address to which that
particular MTA thought it was sending, not some later
correction.  One could, I suppose, replace "for <UTF-8-address>"
with "(previous for field deleted in downgrading" without
distorting information.  But see the rather strong MUST NOT
language in 2821 about alterations to existing Received fields;
language that is there for reasons based on nasty past
experiences.

> And more, I want to record ALT-ADDR, ATOMIC parameter

Putting this information into a comment is no problem.  Doing it
as an authoritative subfield raises the usual questions: since
the subfield names are not defined in RFC 2821, you would
presumably have to remove them, retroactively, since 2821 does
not permit them and the result of downgrading must be
2821/2822-compatible.

> as a result of adding Received header or IMA-Downgraded-*

Yes, see comment about scanning header fields above.

I really don't know how to best handle this, although the degree
to which "for" has fallen into disfavor for security reasons may
give us some hints.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 02 May 2006 19:20:00 +0100

> agree to the requirement for such a header field -- it does have

I think an intermediate MTA that needs to know whether it is dealing with  
an extended email or not has a choice. Either it looks for the presence of  
the UTF-8-indicating header, or it looks for the presence of a bit-8 (or  
maybe it does a proper UTF-8 test, as we have been discussing).  
Implementor's choice.

However, as regards the Received header it would, in principle, be nice to  
have it in UTF-8 wherever possible, but there are snags. But I am not sure  
this is one of them. The case we are considering (possible necessity to  
insert a UTF-8-indicating header if we have just created a "for clause")  
will be rare: it only arises if the RCPT address is UTF-8 and the To:  
header is not (otherwise the UTF-8-indicating header would already be  
present). Moreover, the "for" clause can safely be left in UTF-8 if we are  
forwarding to another IMA-compliant MTA, and if we are not, then we shall  
have to use the alternate (or a downgraded) address, in which case the  
"for" clause might as well contain the alternate/downgraded address.

Tne real problem, of course, is that a Received header with UTF-8 in it  
will eventually arrive at a non-compliant MTA, and therfore we need to  
define a means to downgrade Received headers.

> occur, then drop it from all trace fields in which it appears.

What is current practice with regard to inserting the "for" clause? If a  
message passes through a chain of intermediate MTAs, then surely the "for"  
clause needs to be written only once, except in the case where the  
envelope address gets altered en route (e.g. by a mail-list expander).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Tue, 02 May 2006 21:54:12 -0400

--On Tuesday, 02 May, 2006 19:20 +0100 Charles Lindsey
<[email-masked]> wrote:

>>> In my opinion, addresses in "for" clauses should be IMA
>>> itself and should be UTF-8 encoded.
>> 
>> working through the case analyses (not just my desires) and
>> came to the conclusion that the problem was hard and there
>> was no clear answer.  There are clear advantages to having the
>> field that indicates that UTF-8 information is present. The
>> MTA must perform a header scan to determine whether that
>> UTF-8-indicating header is present and add it if it is not.
>> We generally try to avoid requiring MTAs to perform such
>> header scans -- the historical model says that that, other
>> than submission and delivery MTAs, they don't do it.   This
>> is, incidentally, one of several reasons I've been reluctant
>> to agree to the requirement for such a header field -- it
>> does have side-effects.
> 
> I think an intermediate MTA that needs to know whether it is
> dealing with  an extended email or not has a choice. Either it
> looks for the presence of  the UTF-8-indicating header, or it
> looks for the presence of a bit-8 (or  maybe it does a proper
> UTF-8 test, as we have been discussing).  Implementor's choice.
> 
> However, as regards the Received header it would, in
> principle, be nice to  have it in UTF-8 wherever possible, but
> there are snags. But I am not sure  this is one of them. The
> case we are considering (possible necessity to  insert a
> UTF-8-indicating header if we have just created a "for
> clause")  will be rare: it only arises if the RCPT address is
> UTF-8 and the To:  header is not (otherwise the
> UTF-8-indicating header would already be  present).

Or the MAIL address is UTF-8 and the "From:" header is not,
which might occur at least as often.

> Moreover,
> the "for" clause can safely be left in UTF-8 if we are
> forwarding to another IMA-compliant MTA, and if we are not,
> then we shall  have to use the alternate (or a downgraded)
> address, in which case the  "for" clause might as well contain
> the alternate/downgraded address.
> 
> Tne real problem, of course, is that a Received header with
> UTF-8 in it  will eventually arrive at a non-compliant MTA,
> and therfore we need to  define a means to downgrade Received
> headers.

That is, indeed, the key problem.  And downgrading Received
headers violates the very explicit 2821 prohibition on messing
with prior Received headers, so this is not a step to be taken
lightly.

>> Since "for" is optional, one possibility would be to carry
>> "for" information in UTF-8 up to the point that downgrade
>> needed to occur, then drop it from all trace fields in which
>> it appears.
> 
> What is current practice with regard to inserting the "for"
> clause? If a  message passes through a chain of intermediate
> MTAs, then surely the "for"  clause needs to be written only
> once, except in the case where the  envelope address gets
> altered en route (e.g. by a mail-list expander).

The current practice is, more or less, what 2821 specifies.  As
a crude summary:

(i) Insertion of "for" is _always_ optional.

(ii) Insertion for "for" is discouraged when there is more than
one recipient and strongly discouraged if there are recipients
who are not explicitly listed in the headers.  Either can be a
security problem in terms of giving away too much information;
the second will almost certainly be such a problem.

(iii) MTAs that insert "for" do so from the envelope information
available to them.   I don't know of any MTA (although there are
probably some out there) that decides to not put a "for" in
because one was there in a previous trace header.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 03 May 2006 11:41:11 +0100

> with prior Received headers, so this is not a step to be taken

When it comes to downgrading headers, there are broadly two cases to  
consider:

1. Headers which need to play some part in the protocol. E.g.  
From/Reply-To, which will be used when generating replies; Content-Type,  
which will determine how the MUA displays the message; References, Date  
and Subject, which may be used by the MUA to determine the ordering of  
messages for presentation (and maybe for filtering). These headers need to  
be recognizable (and even processible) in their downgraded form by various  
agents

2. Headers which are primarily for human consumption (usually for  
troubleshooting purposes). Keywords; the various Resent-*s; X-headers of  
all sorts; and of course Received. If the human sees one of these in  
downgraded form, then he may need to have a handy upgrading script to hand  
(though if he is the ultimate recipient his MUA maye have been kind enough  
to do the upgrade for him).

For this 2nd class, a one-size-fits-all or last-resort downgrading  
mechanism is quite sufficient. For example, we could invent a new header  
'Downgraded:', with syntax <unstructured> (hence use of RFC 2047 to encode  
any UTF-8 within it would be in order). For any header with no specific  
downgrade rule of its own, you would simply encapsulate the whole header  
within a Downgraded: header, using encoded-words as needed. A final MUA  
might well automatically upgrade it, restoring its original form.

So a downgraded Received header would look like:

Downgraded: Received: from ... by ... for  
<=?utf-8?Q?)&"$%&+!?=@example.com>

which is easily reconstitutable to its original form. Obviously, encoding  
"Q" will usually be preferred to encoding "B", especially if the  
encoded-word includes more of the header than strictly necessary.

---

From: Chris Newman <[email-masked]>
Date: Fri, 05 May 2006 16:25:16 -0700

I agree it's absolutely critical to have the UTF-8 address in the Received 
header for diagnostic purposes.  But 7-bit downgrading of Received headers must 
be as non-destructive and mechanically simple as possible.  Tampering with 
diagnostic trace information is very risky (but so is providing incomplete 
information).

I suggest we require 7-bit in all syntactic parts of the Received headers 
except for comments where we permit UTF-8.  On down-conversion, the comments 
with UTF-8 characters are 2047 encoded; no other changes to the Received header 
are permitted.  This permits the UTF-8 address to be carried in canonical form 
as a comment in a pure UTF-8 environment, and the only permitted 
down-conversion is one that's non-destructive.

All domain names in the Received header will be IDN encoded (but the 8-bit 
domain should be included in a comment), and the for clause will contain either 
the 7-bit alternate address or the 7-bit down-conversion (with the true UTF-8 
address in the comments).

                - Chris

---

From: [email-masked]
Date: Mon, 08 May 2006 15:01:42 +0900 (JST)

I'm afraid that downgrading changes Received header.

There are three possible header downgrading idea.

draft-yoneya-ima-downgrade-01.txt describes 1 and 3.

idea 1. Header by header conversion

  Rewrite all headers to be US-ASCII.

    Received: from ... by ... for <IMA>
      -> Received: from ... by ... for <__RFC2047__(IMA)>

idea 2. Header encapsulation

  define new encapsulated header and replace original header with
  encapsulated header.

    Received: from ... by ... for IMA
      -> IMA-Encapsulated: Received: encapsulate(from ... by ... for IMA)
      or
      -> IMA-Encapsulated: Received: from ... by ... for encapsulate(IMA)

idea 3. MIME encapsulation

  generate new MIME part Message/EAI.
  Received field is preserved in MIME part.

RFC 2821 Section 4.4 Trace Information says:
    An Internet mail program MUST NOT change a Received: line that was
    previously added to the message header.

idea 1:

     Received header's for clause is modified to be equivalent
     US-ASCII form, other clauses are visible, Received header line
     number is preserved.

idea 2:

     Received header information is preserved, Received header line
     number is initialized to be zero in downgraded header.

idea 3:

     Received header information is preserved, Received header line
     number is initialized to be zero in downgraded (outer) header.

All three may break RFC 2821 Section 4.4.

> From: "Charles Lindsey" <[email-masked]>

In my opinion, downgrading is required for not to break existing mail
infrastructure and to send messages to final destination if at all
possible. Whether recipients can read or not is secondary issue. IMA
compliant MUA can decode downgraded message.

> encoded-word includes more of the header than strictly necessary.

This header downgrading is useful and may be added to downgrade ID.

--
Kazunori Fujiwara, JPRS

---

From: Chris Newman <[email-masked]>
Date: Tue, 09 May 2006 12:54:53 -0700

>       -> Received: from ... by ... for <__RFC2047__(IMA)>

This violates the ABNF of the "for" clause in Received.  Thus it's not feasible.

>       -> IMA-Encapsulated: Received: from ... by ... for encapsulate(IMA)

This is going to break things.  Gateways, MUAs and other processing systems are 
permitted to re-order headers, except for trace headers (Received, 
Return-path).  While we can introduce a new trace header, there's no guarantee 
order will be preserved by older MTAs and gateways (which goes against the 
whole point of encapsulation).  As a result, I don't consider this option 
feasible for Received or Return-path.

>   Received field is preserved in MIME part.

This would intrude into the user interface on many systems and thus I consider 
it not feasible.

> All three may break RFC 2821 Section 4.4.

What I'm suggesting is something like this:

Received: from ... by ... for 7bit-addr@7bit-domain (utf8-addr@utf8-domain)

And then grant a very limited exception to 2821 section 4.4 _only_ for 2047 
down-conversion of 8-bit comments, so we get:

Received: from ... by ... for 7bit-addr@7bit-domain (_RFC_2047_)

This preserves the syntax of Received after down-conversion, no ordering 
problems or intrusion into the UI, and the exception to RFC 2821 section 4.4 
should be sufficiently minor and non-destructive to avoid the problems that 
rule attempts to prevent.

Now for cases where there is no 7-bit address, the comment could go after any 
clause (except date) and say (for utf8-addr@utf8-domain).

                - Chris

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 10 May 2006 10:02:20 +0100

> This is going to break things.  Gateways, MUAs and other processing  
> systems are permitted to re-order headers, except for trace headers  
> (Received, Return-path).  While we can introduce a new trace header,  
> there's no guarantee order will be preserved by older MTAs and gateways  
> (which goes against the whole point of encapsulation).  As a result, I  
> don't consider this option feasible for Received or Return-path.

But do agents (transport agents in particular) actually do such reordering  
in practice? Surely all that the usual MTA does is to stick its own  
Received header at the top without even looking at the rest. In that case,  
and 'Received's that get downgraded to 'IMA-Encapsulated: Received...'  
will in fact retain their place in the list, at least during transport.

The worst that can happen is that some agent somewhere will filter all the  
'Received's to the top, in which case the poor human will have to merge  
the encapsulated ones back into the list - that is messy, but not  
impossible and maybe even automizable.

I suspect that whatever we do is not going to be perfect, and we shall  
have to choose the best of a bad bunch.

> Received: from ... by ... for 7bit-addr@7bit-domain  

Yes, that looks an interesting possibility. Please could we have that  
added to the other examples that I have asked for.

My main concern with it is that it works fione for "for", but can we be  
sure that UTF-8 will not arise in other (non-comment) components of the  
Received header?

---

From: [email-masked]
Date: Wed, 10 May 2006 20:56:01 +0900 (JST)

> From: Chris Newman <[email-masked]>

Is this 7bit-addr@7bit-domain alternate-ascii-address which is
specified by Alt-Addr parameter?

(without Alt-Address and with ATOMIC y, alternate-ascii-address is
generated by punycode comversion of utf8@utf8.)

In my understanding,
if "for" clauses has ascii-alternative-address with utf8-address in
comment field, the header shows that UTF8 address was used in that
SMTP session.

> rule attempts to prevent.

It's one candidate method.

--
Kazunori Fujiwara, JPRS

---

From: Chris Newman <[email-masked]>
Date: Thu, 11 May 2006 11:52:20 -0700

> generated by punycode comversion of utf8@utf8.)

Yes.

> comment field, the header shows that UTF8 address was used in that

Yes.

> It's one candidate method.

Thanks.

                - Chris

---

From: Chris Newman <[email-masked]>
Date: Thu, 11 May 2006 12:02:16 -0700

Charles Lindsey wrote on 5/10/06 10:02 +0100:
> in practice?

Our MTA did re-ordering all the time.  When 2822 came out, I believe we changed 
it to not re-order by default, but customers in certain countries are very 
sensitive to the order of the key headers like "date", "subject", "from", "to" 
and want them re-ordered.  Some old clients have problems with headers in the 
wrong order, so re-ordering has to be done to interoperate with them.

We might move trace headers above non-trace headers, as well, but I don't 
recall.  It's certainly a permissible transformation by any MTA that's not pure 
pass-through as it does not change the defined RFC 2822 semantics of the 
headers.

> Surely all that the usual MTA does is to stick its own  Received
> header at the top without even looking at the rest. In that case,  and
> 'Received's that get downgraded to 'IMA-Encapsulated: Received...'  will in
> fact retain their place in the list, at least during transport.

Counting Received headers is one of the mechanisms used for loop detection, so 
encapsulation would break that as well.

> 'Received's to the top, in which case the poor human will have to merge  the
> encapsulated ones back into the list - that is messy, but not  impossible and
> maybe even automizable.

Too much complexity for my tastes.

> I suspect that whatever we do is not going to be perfect, and we shall  have
> to choose the best of a bad bunch.
>
> Yes, that looks an interesting possibility. Please could we have that  added
> to the other examples that I have asked for.
>
> My main concern with it is that it works fione for "for", but can we be  sure
> that UTF-8 will not arise in other (non-comment) components of the  Received
> header?

If we settle on the UTF-8 comment option, that gives an escape valve for things 
like the UTF-8 version of an idn-encoded domain name so it should prevent 
leakage.

                - Chris

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 12 May 2006 11:02:38 +0100

On Thu, 11 May 2006 20:02:16 +0100, Chris Newman <[email-masked]>  
wrote:

> Counting Received headers is one of the mechanisms used for loop  
> detection, so encapsulation would break that as well.
>
Yes, that seems a valid point. But surely it would only make the message  
go round the loop more times, and it would still catch it eventually? Not  
ideal, but still not an infinite loop.

>> added
> If we settle on the UTF-8 comment option, that gives an escape valve for  
> things like the UTF-8 version of an idn-encoded domain name so it should  
> prevent leakage.

All right. Now what about the case of the 'From' and 'By' fields of  
Received? Are these covered? Presumably so, because AIUI there are no  
local parts in them, so an IDNA downgrade is always possible.