Subject: ATOMIC and downgrading (was: Re: [EAI] Comments on draft-ietf-eai-downgrading.00)

From: John C Klensin <[email-masked]>
Date: Tue, 04 Jul 2006 19:06:56 -0400

Sorry for the delay but a few comments on this general situation
(I have omitted significant material, indicating  either that I
agree with you or have nothing to add at this point.)

--On Thursday, 15 June, 2006 21:55 +0100 Charles Lindsey
<[email-masked]> wrote:

>      Downgrading mechanism for Internationalized eMail Address
> (IMA)
>...

>     In this document, an address is "all-ASCII" if every
> character in the
>     address is in the ASCII character repertoire [ASCII]; an
> address is
>     "non-ASCII" if any character is not in the ASCII character
>     repertoire.  The term "all-ASCII" is also applied to other
> protocol
>     elements when the distinction is important, with
> "non-ASCII" or
>     "internationalized" as its opposite.
> 
> s/ASCII/US-ASCII/ in two places.

For convenience of reading and understanding, I'd think this
should be changed only in the citation and reference.  Sloppy
writing aside, ASCII really is the "American Standard Code for
Information Interchange".   With apologies to my Canadian and
Mexican colleagues (CSCII?  MSCII?) and those further south, the
term is unambiguous and "US-ASCII" is hard to read and causes,
rather than reduces, confusion.

> 4.  SMTP Downgrading
>...
  
>     If downgrading is expected, mail sender MUA MUST append
> ALT-ADDR or
>     ATOMIC option to all IMA envelope addresses to denote
> alternative US-
>     ASCII address when sending mail.
> 
> No, I don't think that is right. There is no "MUST" about it.
> If you
> want your message to get through without bouncing, then you
> would be
> *well-advised* to ensure that an ALT-ADDRESS was provided or
> ATOMIC was specified.

Concur, although I think even that language may need further
tuning... see below.

> I found the next few paragraphs rather confusing. It needs to
> be made
> clear exactly what addresses are to be passed on in the MAIL
> FROM and
> RCPT TO in all the possible circumstances. I found what
> Yangwoo Ko wrote
> on June 4th rather helpful in clarifying the situation, and
> from what he
> said I derived the following chart:
> 
> 
>             ALT-ADDRESS available
>                Y            N
>          ---------------------------
>          |            |            |
>         Y|  use ALT-  | do ACE     |
>          |  ADDRESS   | conversion |
>          |            |            |
> ATOMIC  |-------------------------|
>          |            | downgrade  |
>         N|  use ALT-  | not        |
>          |  ADDRESS   | possible   |
>          |            | (bounce)   |
>          ---------------------------
> 
> which shows that ATOMIC is meaningless if an ALT-ADDRESS is
> provided,
> and also that "ATOMIC" would be better called something like
> "DOWNGRADABLE", as Yangwoo Ko suggested.

This is also a strong argument for simply prohibiting the
specification of both ATOMIC and ALT-ADDRESS.  One or the other
or neither, but not both.    I believe that we concluded during
the interim meeting that SMTPext and Framework should be
modified to reflect that view, so this may be an historical
artifact.

> Anyway, I would you include a chart like that, and then
> explain how to
> use the ALT-ADDRESS, and how to do ACE-conversion.

>     Alternative US-ASCII address generation algorithms are
> below:
>     domain-part: Punycode/IDNA [RFC3490]
>     local-part: Punycode[RFC3492] without normalization.
> Prefix MUST be
>        assigned by IANA (which is not "xn--").

These two don't work.  Even without normalization, punycode
would cause case folding, which has traditionally been
prohibited in email.  Adam Costello proposed a way around that
some time ago but, if we are going to use it, we are going to
need to do considerably more documentation and specfication than
saying "Punycode without normalization".

> Well that doesn't seem right, but when I looked into RFCs
> 3490, 3491 and
> 3492 I found such a can of worms that I need to look into it
> more
> deeply, so I shall deal with that in a separate message.

I assume the above describes some of the worms you found.

>     MTA replaces IMA with specified or generated alternative
> US-ASCII
>     address.  Then appends replaced information with
> IMA-Downgraded-From
>     and IMA-Downgraded-To header in mail header (outgoing SMTP
> DATA).
>        IMA-Downgraded-From: <IMA> <US-ASCII>
>        IMA-Downgraded-To: <IMA> <US-ASCII>
> 
> It is not clear to me how this is supposed to work. How am I
> to know
> whether it was the MAIL FROM: or one of the (many) RCPT TO:
> commands
> that was downgraded? Surely that needs to be indicated in
> those headers
> somewhere. OTOH, I am not yet convinced that we actually needs
> those headers at all.

Yes to both.   I also believe that, for _any_ new header, the
document must include an analysis of what the risks are if some
or all of those headers are dropped in transit.

>...

> 5.  SMTP DATA/Header downgrading
> 
> Actually, this section has not much to do with SMTP, since it
> might be
> called upon in all sorts of other parts of various protocols
> (including
> other transports). Yes, I agree that the common case will be
> when
> dealing with the DATA command in SMTP.
> 
> 
>     In this section, four methods for SMTP DATA/Header
> downgrading is
>     proposed.  Working group should select one.
> 
>     o  No header downgrading
>     o  Encapsulating whole SMTP DATA
>     o  Translating each header
>     o  Encapsulating each header
> 
> I think it is pretty clear that the WG is going to select #3,
> with maybe a little bit of #4 in special situations.

It is not yet clear to me, actually.  I hope there is some
specific discussion on this topic in Montreal.

In particular, I am waiting for someone to do the analysis for
what "translating" headers does to protocols and specifications
that rely on digital signatures over some or all headers, such
as DKIM.

>     Trace headers: Received headers which contains IMA MUST be
> target of        downgrading.

> I have no problem with that (for comments anyway), but John
> doesn't seem to like it :-( .

John is still trying to figure out how to get around the
absolute prohibition on modifying earlier trace fields, which is
there for a reason and after much sad experience.  One way is to
decide that any system that does downgrading is actually a
gateway.  That has other implications, of course.

> 5.2.  Downgrading with MIME encapsulation
> 
>     This downgrade method requires new MIME 'Content-Type:'
> which express
>     EAI(Email Address Internationalization).  This document
> assumes
>     'Content-Type: Message/EAI' existence.
> 
> It would need to be a Content-Type which really ensured that
> the whole
> messages was encoded (message/rfc822 would certainly not be
> good
> enough). There is an application/news-transmission already
> defined which
> does exactly the right thing.

How does "application/news-transmission" handle the multiple
encoding prohibition?  Also, forgive my ignorance, but in what
standards-track document is it defined?  (If we are to use it
here, it will have to be a normative reference and, if we are to
have problems in that area, we had best start sorting out the
issues).  I've been assuming that we were going to need
something more like

   multipart/message-eai
      message/converted-eai-headers
      multipart/mixed    or  text/...  or .../...

> However, I do not like this method, since it may cause all
> sorts of
> problems on arrival, and also it splits the Received headers
> into two
> separate places, separating those which were inserted before
> the
> downgrade from those which were inserted after (so it breaks
> RFC 2821 anyway).

yep.  From my point of view, that is yet another reason to avoid
putting any UTF-8 material into Received headers.  If none
exists, then the Received headers can be left alone (at the top
of the external headers) and used to document the downgrading,
while everything else goes into the converted/ encapsulated
headers.

> 6.  Implementation consideration
> 
> 6.1.  MUA
> 
>     MUA MAY encode UTF-8 in Subject header with the same
> encoding of body
>     part while downgrading.
> 
> I am not sure I like that. Has it been discussed already?

No, it has not been discussed, at least while I was paying
attention.

>     4.  If MDA detects that SMTP recipient address is
> downgraded IMA,
>         then MDA MUST decode IMA and perform the same
> processing as if it
>         were IMA.  MDA MAY normalize or canonicalize
> local-part before
>         processing it.
> 
> s/downgraded/ACE-downgraded/ (because if it was ALT-ADDRESS
> downgraded
> you cannot do this - it is essentially the same as case #2
> above). And
> the MUST is probably too strong.

There is not really a guarantee that the ACE encoding is
accurately reversable  (and anything involving canonicalization,
normalization, case-folding, etc., certainly will not be).

    john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 05 Jul 2006 11:46:10 +0100

> Information Interchange".

I am sure I remember reading some RFC which declared that IETF policy was  
to use US-ASCII rather than ASCII in all official documents. But I have  
not managed to locate it (beyond a mention in RFC 1700, which is not the  
one I had in mind). But for sure somebody in Montreal will know.

> or all of those headers are dropped in transit.

It shouldn't happen, but if it does then we should arrange that the worst  
that happens is that some information gets lost (e.g. the history of what  
some pre-downgraded header looked like). Jeff's recent example was fine.

> that rely on digital signatures over some or all headers, such

Undoubtedly some of those protocols are going to break, and there is not  
much we can do about it. OTOH, signing mechanisms CAN be invented which  
will survive downgrading, but they need to employ a pretty aggressive  
canonicalization. I wrote such a proposal once  
(http://www.imc.org/ietf-usefor/drafts/draft-lindsey-usefor-signed-01.txt).
> gateway.  That has other implications, of course.

If the only place where UTF-8 is allowed in within the comments within  
Received headers, then you simply say that they MUST always be  
RFC2047-encoded (even when generated by a UTF8SMTP-compliant agent). Then  
there is never any need to change them en route.

OTOH, I still do not see what harm would come from allowing this simple  
downgrade en route.

> issues).  I've been assuming that we were going to need

It simply wraps the whole 8bit object up in one bundle (without regard to  
its internal structure), and then applies a CTE (Q-P or Base64) to it. So  
it is totally foolproof - not that I am actually recommending it other  
than as a proof-of-concept. It is already registered with IANA (bt Henry  
Spencer), and the upcoming USEPRO (part of the USEFOR effort) will simply  
confirm its existing definition (well, with a couple of irrelevant extra  
options).
>       multipart/mixed    or  text/...  or .../...

No, that is all far too complicated. All that needs to be done is to state  
that message/rfc822 objects can contain Utf8smtp headers (provided they  
include the "Header-Content: UTF8SMTP" or whatever). And provided that  
agents which need to downgrade watch out for this situation and apply the  
downgrade to the message/rfc822 as well (or bounce if they cannot be  
bothered). Actually, if they don't, the message will still get through  
unless it encounters a non-8BITMIME MTA en route, and there are very few  
of those left around these days.

>> downgraded...

I think it essential that we define ACE encoding to be fully reversible.  
Tricky, perhaps, but not impossible.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Nathaniel Borenstein <[email-masked]>
Date: Wed, 5 Jul 2006 09:00:38 -0400

On Jul 5, 2006, at 6:46 AM, Charles Lindsey wrote:

> I am sure I remember reading some RFC which declared that IETF policy 
> was to use US-ASCII rather than ASCII in all official documents. But I 
> have not managed to locate it (beyond a mention in RFC 1700, which is 
> not the one I had in mind). But for sure somebody in Montreal will 
> know.

I believe the reason we used "US-ASCII" in MIME was that there was a 
long-standing (and tedious) dispute about "what is ASCII?" having to do 
with stuff like dollar sign vs currency symbols and some other minutae. 
  We chose a parameter value of "US-ASCII" so that, without taking sides 
in that particular holy war, we would leave no room for doubt about 
which version of ASCII was being referred to.

It is possible that enough people have died by now that one can now 
safely use "ASCII" to refer to, um, the thing most of us think of as 
ASCII.  But I wouldn't bet on it.   -- Nathaniel

---

From: John C Klensin <[email-masked]>
Date: Wed, 05 Jul 2006 09:45:31 -0400

--On Wednesday, 05 July, 2006 09:00 -0400 Nathaniel Borenstein
<[email-masked]> wrote:

>> I am sure I remember reading some RFC which declared that
>> IETF policy  was to use US-ASCII rather than ASCII in all
>> official documents. But I  have not managed to locate it
>> (beyond a mention in RFC 1700, which is  not the one I had in
>> mind). But for sure somebody in Montreal will  know.
> 
> I believe the reason we used "US-ASCII" in MIME was that there
> was a long-standing (and tedious) dispute about "what is
> ASCII?" having to do with stuff like dollar sign vs currency
> symbols and some other minutae.   We chose a parameter value
> of "US-ASCII" so that, without taking sides in that particular
> holy war, we would leave no room for doubt about which version
> of ASCII was being referred to.
> 
> It is possible that enough people have died by now that one
> can now safely use "ASCII" to refer to, um, the thing most of
> us think of as ASCII.  But I wouldn't bet on it.   -- Nathaniel

And that is why, in running text (this is not a parameter which
one could argue needs to be precise and self-referencing), I
suggest using "ASCII", which is clear, and then using
"[US-ASCII]" as the citation anchor.

    john

---

From: Nathaniel Borenstein <[email-masked]>
Date: Wed, 5 Jul 2006 16:17:46 -0400

> "[US-ASCII]" as the citation anchor.

Lest there be any misunderstanding... I agree.  -- Nathaniel