Subject: [EAI] Comments on draft-ietf-eai-downgrading.00

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 15 Jun 2006 21:55:07 +0100

Downgrading mechanism for Internationalized eMail Address (IMA)

s/IMA/EAI (or whatever)/ in lots of places.

Abstract

    Traditional mail system handles only US-ASCII characters in SMTP
    envelope and mail headers.  .....

There are lots of places where you have used the singular where the
plural would be better. E.g. s/system/systems/ and
s/envelope/envelopes/. Generally speaking, if a word is in the singular
it needs to have an article ("the" or "a") in front of it. I would
suggest you have it checked by a native English speaker (or you can
contact me offline). I won't go into all the cases here, but it does
need attending to at some point.

1.  Introduction

    ....  Not only SMTP envelope, but also UTF-8 in mail headers MUST
    be converted to US-ASCII.

Also, "body part headers", which occur in MIME multiparts and
message/rfc822, need to be converted.

2.  Terminology

    The final ("delivery") MTA stores Mail messages in a "message store"
    or resends messages where the receiver has assigned.  In this
    document, this function is called Mail Delivery Agent(MDA).

Traditionally, the IETF has steered clear of the interface between the
final MTA and the MDA, because there is presumed to be no "wire" between
them.  However, they are often completely separate pieces of software,
and a single MTA might send the message to one of several MDAs,
according to the <local-part>. So, even if the MTA advertised IEmail,
some of those MDAs might still be ASCII-only. So the MTA needs to be
aware of which those might be, so that it can still downgrade or bounce
the message (even if it has already accepted in in IEmail form). So
there needs to be some interface between MTAs and MDAs for sorting this
out.

Whether the WG wants to specify such an interface in detail is another
matter - probably not - but it needs to be aware of the problem, and to
have some model in mind of how such a thing would function. Possibly the
overview document is the place to discuss this, but it might also need
some mention here.

    In this document, an address is "all-ASCII" if every character in the
    address is in the ASCII character repertoire [ASCII]; an address is
    "non-ASCII" if any character is not in the ASCII character
    repertoire.  The term "all-ASCII" is also applied to other protocol
    elements when the distinction is important, with "non-ASCII" or
    "internationalized" as its opposite.

s/ASCII/US-ASCII/ in two places.

3.  Downgrade Requirements

3.1.  Timing and conditions of downgrading

    o  Conditions: SMTP client detects that UTF-8 is included in SMTP
       envelope or mail headers.

or in "body part headers". Checking for the presence of the I18Mail
header (for Header-Content header, or whatever we call it) should be
sufficient.

3.2.  Requirements

    2.  Upgrading must be performed at minimized place such as final
        destination like recipient MUA.

I think the proper thing to say is that upgrading MUST NOT take place
until the message reaches someplace where it is legitimate to examine
the <local-part>, and that means the destination MTA at the earliest.
Even then, it SHOULD ONLY be the <local-part> that is upgraded (if
necessary) in order to derermine which MDA to give it to. The MDA MAY
upgrade it (should be under user control), but ideally it should be left
to the MUA after the MDA has finally passed it over.

4.  SMTP Downgrading

I think you need to point out that this section only applies if the
transport method is SMTP (or one of its relations such as LMTP). If some
different transport (e.g. UUCP) is used, then similar changes _may_ be
needed but are not specified here. OTOH, your section 5 on header
downgrading is likely to be needed even if other transports are used.

    If downgrading is expected, mail sender MUA MUST append ALT-ADDR or
    ATOMIC option to all IMA envelope addresses to denote alternative US-
    ASCII address when sending mail.

No, I don't think that is right. There is no "MUST" about it. If you
want your message to get through without bouncing, then you would be
*well-advised* to ensure that an ALT-ADDRESS was provided or ATOMIC was
specified.

I found the next few paragraphs rather confusing. It needs to be made
clear exactly what addresses are to be passed on in the MAIL FROM and
RCPT TO in all the possible circumstances. I found what Yangwoo Ko wrote
on June 4th rather helpful in clarifying the situation, and from what he
said I derived the following chart:

            ALT-ADDRESS available
               Y            N
         ---------------------------
         |            |            |
        Y|  use ALT-  | do ACE     |
         |  ADDRESS   | conversion |
         |            |            |
ATOMIC  |-------------------------|
         |            | downgrade  |
        N|  use ALT-  | not        |
         |  ADDRESS   | possible   |
         |            | (bounce)   |
         ---------------------------

which shows that ATOMIC is meaningless if an ALT-ADDRESS is provided,
and also that "ATOMIC" would be better called something like
"DOWNGRADABLE", as Yangwoo Ko suggested.

Anyway, I would you include a chart like that, and then explain how to
use the ALT-ADDRESS, and how to do ACE-conversion.

    Alternative US-ASCII address generation algorithms are below:
    domain-part: Punycode/IDNA [RFC3490]
    local-part: Punycode[RFC3492] without normalization.  Prefix MUST be
       assigned by IANA (which is not "xn--").

Well that doesn't seem right, but when I looked into RFCs 3490, 3491 and
3492 I found such a can of worms that I need to look into it more
deeply, so I shall deal with that in a separate message.

    MTA replaces IMA with specified or generated alternative US-ASCII
    address.  Then appends replaced information with IMA-Downgraded-From
    and IMA-Downgraded-To header in mail header (outgoing SMTP DATA).
       IMA-Downgraded-From: <IMA> <US-ASCII>
       IMA-Downgraded-To: <IMA> <US-ASCII>

It is not clear to me how this is supposed to work. How am I to know
whether it was the MAIL FROM: or one of the (many) RCPT TO: commands
that was downgraded? Surely that needs to be in" if every character in the
    address is in the ASCII character repertoire [ASCII]; an address is
    "non-ASCII" if any character is not in the ASCII character
    repertoire.  The term "all-ASCII" is also applied to other protocol
    elements when the distinction is important, with "non-ASCII" or
    "internationalized" as its opposite.

s/ASCII/US-ASCII/ in two places.

3.  Downgrade Requirements

3.1.  Timing and conditions of downgrading

    o  Conditions: SMTP client detects that UTF-8 is included in SMTP
       envelope or mail headers.

or in "body part headers". Checking for the presence of the I18Mail
header (for Header-Content header, or whatever we call it) should be
sufficient.

3.2.  Requirements

    2.  Upgrading must be performed at minimized place such as final
        destination like recipient MUA.

I think the proper thing to say is that upgrading MUST NOT take place
until the message reaches someplace where it is legitimate to examine
the <local-part>, and that means the destination MTA at the earliest.
Even then, it SHOULD ONLY be the <local-part> that is upgraded (if
necessary) in order to derermine which MDA to give it to. The MDA MAY
upgrade it (should be under user control), but ideally it should be left
to the MUA after the MDA has finally passed it over.

4.  SMTP Downgrading

I think you need to point out that this section only applies if the
transport method is SMTP (or one of its relations such as LMTP). If some
different transport (e.g. UUCP) is used, then similar changes _may_ be
needed but are not specified here. OTOH, your section 5 on header
downgrading is likely to be needed even if other transports are used.

    If downgrading is expected, mail sender MUA MUST append ALT-ADDR or
    ATOMIC option to all IMA envelope addresses to denote alternative US-
    ASCII address when sending mail.

No, I don't think that is right. There is no "MUST" about it. If you
want your message to get through without bouncing, then you would be
*well-advised* to ensure that an ALT-ADDRESS was provided or ATOMIC was
specified.

I found the next few paragraphs rather confusing. It needs to be made
clear exactly what addresses are to be passed on in the MAIL FROM and
RCPT TO in all the possible circumstances. I found what Yangwoo Ko wrote
on June 4th rather helpful in clarifying the situation, and from what he
said I derived the following chart:

            ALT-ADDRESS available
               Y            N
         ---------------------------
         |            |            |
        Y|  use ALT-  | do ACE     |
         |  ADDRESS   | conversion |
         |            |            |
ATOMIC  |-------------------------|
         |            | downgrade  |
        N|  use ALT-  | not        |
         |  ADDRESS   | possible   |
         |            | (bounce)   |
         ---------------------------

which shows that ATOMIC is meaningless if an ALT-ADDRESS is provided,
and also that "ATOMIC" would be better called something like
"DOWNGRADABLE", as Yangwoo Ko suggested.

Anyway, I would you include a chart like that, and then explain how to
use the ALT-ADDRESS, and how to do ACE-conversion.

    Alternative US-ASCII address generation algorithms are below:
    domain-part: Punycode/IDNA [RFC3490]
    local-part: Punycode[RFC3492] without normalization.  Prefix MUST be
       assigned by IANA (which is not "xn--").

Well that doesn't seem right, but when I looked into RFCs 3490, 3491 and
3492 I found such a can of worms that I need to look into it more
deeply, so I shall deal with that in a separate message.

    MTA replaces IMA with specified or generated alternative US-ASCII
    address.  Then appends replaced information with IMA-Downgraded-From
    and IMA-Downgraded-To header in mail header (outgoing SMTP DATA).
       IMA-Downgraded-From: <IMA> <US-ASCII>
       IMA-Downgraded-To: <IMA> <US-ASCII>

It is not clear to me how this is supposed to work. How am I to know
whether it was the MAIL FROM: or one of the (many) RCPT TO: commands
that was downgraded? Surely that needs to be indicated in those headers
somewhere. OTOH, I am not yet convinced that we actually needs those
headers at all.

    Downgraded envelope to is parsed only in MDA and delivered to final
    mailbox.

Actually, I don't think anyone cares whether some intermediate agent
parses the domain part (in IDNA) of an envelope address. But parsing the
local-part of such an address is a definite No-No until you get to the
final MTA. So maybe s/envelope/local-part/ in there.

5.  SMTP DATA/Header downgrading

Actually, this section has not much to do with SMTP, since it might be
called upon in all sorts of other parts of various protocols (including
other transports). Yes, I agree that the common case will be when
dealing with the DATA command in SMTP.

    In this section, four methods for SMTP DATA/Header downgrading is
    proposed.  Working group should select one.

    o  No header downgrading
    o  Encapsulating whole SMTP DATA
    o  Translating each header
    o  Encapsulating each header

I think it is pretty clear that the WG is going to select #3, with maybe
a little bit of #4 in special situations.

    Trace headers: Received headers which contains IMA MUST be target of
       downgrading.

I have no problem with that (for comments anyway), but John doesn't seem
to like it :-( .

5.2.  Downgrading with MIME encapsulation

    This downgrade method requires new MIME 'Content-Type:' which express
    EAI(Email Address Internationalization).  This document assumes
    'Content-Type: Message/EAI' existence.

It would need to be a Content-Type which really ensured that the whole
messages was encoded (message/rfc822 would certainly not be good
enough). There is an application/news-transmission already defined which
does exactly the right thing.

However, I do not like this method, since it may cause all sorts of
problems on arrival, and also it splits the Received headers into two
separate places, separating those which were inserted before the
downgrade from those which were inserted after (so it breaks RFC 2821
anyway).

5.3.  Header conversion

    Each header has its own downgrading method.  Basically, MIME encoding
    of RFC 2047.  Recipient/Sender addresses and Received headers which
    may contain IMA need special processing.

I think we need to be careful to cover every possible header here,
including all the headers already defined in all sorts of standards,
plus all the ones which are going to be defined in future standards.

I think the first thing it to separate them into 'structured' and
'unstructured' headers. That actually gives you three cases to consider:

1. Known strutured headers.

As so defined in RFC 2822, all the MIME standards, maybe the List-*
standards, and possibly others. So either you need to give a list of all
the RFCs, or else a complete list of headers.

There is a general rule how to downgrade comments and phrases in these
headers (using RFC 2047). After that, some of them will be "UTF-8
forbidden" (the utf8headers document should say which), so no further
downgrade will be needed. For the rest, you can give explicit rules for
dealing with
    addr-specs
    MIME-style parameters (use RFC 2231)
    IRIs
    Received (possibly)
    and so on

2. Known unstructured headers.

Again, as defined in RFC 2822 and other places. The Subject header is
the best known example, but there are others (some, such as Organization
and Summary not defined in mail standards at all, but still widely
used). You should also say that any header whose name begins with "X-"
is to be considerd unstructured.

The downgrade method is, of course, straightforward RFC 2047.

3. Structured state unknown.

These are the real problem. People have been inventing new email headers
at the rate of half-a-dozen a year for as long as anyone can remember,
and there is no reason to suppose it is going to stop.

Anyone inventing such a header in future is welcome to state whether it
is structured, and if so whether utf-8 is to be allowed within it. He
can even proposes an explicit downgrade method for his new header, and
people mighdicated in those headers
somewhere. OTOH, I am not yet convinced that we actually needs those
headers at all.

    Downgraded envelope to is parsed only in MDA and delivered to final
    mailbox.

Actually, I don't think anyone cares whether some intermediate agent
parses the domain part (in IDNA) of an envelope address. But parsing the
local-part of such an address is a definite No-No until you get to the
final MTA. So maybe s/envelope/local-part/ in there.

5.  SMTP DATA/Header downgrading

Actually, this section has not much to do with SMTP, since it might be
called upon in all sorts of other parts of various protocols (including
other transports). Yes, I agree that the common case will be when
dealing with the DATA command in SMTP.

    In this section, four methods for SMTP DATA/Header downgrading is
    proposed.  Working group should select one.

    o  No header downgrading
    o  Encapsulating whole SMTP DATA
    o  Translating each header
    o  Encapsulating each header

I think it is pretty clear that the WG is going to select #3, with maybe
a little bit of #4 in special situations.

    Trace headers: Received headers which contains IMA MUST be target of
       downgrading.

I have no problem with that (for comments anyway), but John doesn't seem
to like it :-( .

5.2.  Downgrading with MIME encapsulation

    This downgrade method requires new MIME 'Content-Type:' which express
    EAI(Email Address Internationalization).  This document assumes
    'Content-Type: Message/EAI' existence.

It would need to be a Content-Type which really ensured that the whole
messages was encoded (message/rfc822 would certainly not be good
enough). There is an application/news-transmission already defined which
does exactly the right thing.

However, I do not like this method, since it may cause all sorts of
problems on arrival, and also it splits the Received headers into two
separate places, separating those which were inserted before the
downgrade from those which were inserted after (so it breaks RFC 2821
anyway).

5.3.  Header conversion

    Each header has its own downgrading method.  Basically, MIME encoding
    of RFC 2047.  Recipient/Sender addresses and Received headers which
    may contain IMA need special processing.

I think we need to be careful to cover every possible header here,
including all the headers already defined in all sorts of standards,
plus all the ones which are going to be defined in future standards.

I think the first thing it to separate them into 'structured' and
'unstructured' headers. That actually gives you three cases to consider:

1. Known strutured headers.

As so defined in RFC 2822, all the MIME standards, maybe the List-*
standards, and possibly others. So either you need to give a list of all
the RFCs, or else a complete list of headers.

There is a general rule how to downgrade comments and phrases in these
headers (using RFC 2047). After that, some of them will be "UTF-8
forbidden" (the utf8headers document should say which), so no further
downgrade will be needed. For the rest, you can give explicit rules for
dealing with
    addr-specs
    MIME-style parameters (use RFC 2231)
    IRIs
    Received (possibly)
    and so on

2. Known unstructured headers.

Again, as defined in RFC 2822 and other places. The Subject header is
the best known example, but there are others (some, such as Organization
and Summary not defined in mail standards at all, but still widely
used). You should also say that any header whose name begins with "X-"
is to be considerd unstructured.

The downgrade method is, of course, straightforward RFC 2047.

3. Structured state unknown.

These are the real problem. People have been inventing new email headers
at the rate of half-a-dozen a year for as long as anyone can remember,
and there is no reason to suppose it is going to stop.

Anyone inventing such a header in future is welcome to state whether it
is structured, and if so whether utf-8 is to be allowed within it. He
can even proposes an explicit downgrade method for his new header, and
people might even implement his method.

The problem is that, once people have implemented our new protocol, they
are not going to rewrite their software every time someone brings out a
new header (maybe with new downgrade rules). This is exactly the problem
that RFC 2047 ran into, because it assumes that, as soon as a new
standard specifies a new header, all the MIME software in the world will
immediately and magically "know" whether this new header is unstructured
or not and use (or not use) RFC 2047 encoding within it accordingly. The
result has been that all sorts of implementations for 2047 encoding of
headers where they should not be doing it, and vice versa. So let us try
to avoid that trap.

I therefore propose a "generic" downgrade method, to be used on all
headers whose state is unknown:

    Downgraded: Header-Name: {RFC encoded form of original header body}

(just like your section 5.4, in fact).

This is always safe; it works whether the header in question is
structured or unstructured; it is easily upgraded; and it will do no
harm whatsoever SO LONG AS that header plays no part in the transport
process (which is likely to be the case - most new headers are for
human consumption, or for use by specific gateways, etc).

The only snag is that it might not always suit the intention of the
author of the new header (who might have preferred you to use his own
specially devised downgrade). Tough! He will just have to define his new
protocol so that it can get around the use of this generic downgrade by
MTAs that were written long before he came on the scene.

As to the specific rules we need for addr-specs in presently known
headers, I think we know how we intend to do it. I would suggest
repeating the table I gave earlier showing how to deal with various
combinations of ALT-ADDRESS and ATOMIC, and after that the details are
very similar to the envelope procedures.

I have mentioned IRIs as a special case, because all the headers which
currently have URIs in them are surely going to be upgraded to use IRIs
before long (indeed, we could include such changes in our utf8headers
document if we so decided). Moreover, there is already a defined way to
downgrade an IRI to a URI, so we may as well use it.

And finally, the downgrade process should include instructions to
remove/modify/whatever the I18MAIL header (or whatever it gets called -
I still prefer the Header-Content header that I mentioned a week or so
back, and which would need to be modified to shown that downgrading had
taken place).

And finally finally there is still the matter of downgrading "body part
headers", as I mentioned earlier.

6.  Implementation consideration

6.1.  MUA

    MUA MAY encode UTF-8 in Subject header with the same encoding of body
    part while downgrading.

I am not sure I like that. Has it been discussed already?

    IMA compliant MUA MUST decode downgraded mail and MUST show IMA on
    display.

s/decode downgraded/upgrade/
But I am not sure this is really a MUST (is any interoperability
broken?). SHOULD by all means.

6.2.  MDA Requirements

    1.  MDA MUST NOT convert downgraded header to UTF-8.

Why not? Indeed, the new imap draft proposes to do just that (under
user's control, of course).

    2.  Record Return-Path header in ACE form.

This might not be possible (the MAIL FROM: may have been downgraded to
an ALT-ADDRESS en route, and all trace of the UTF8 form of it may have
been lost).

    3.  Perform downgrading for each Storage/Back-end-Process.  If and
        only if MDA knows MUA is IMA compliant, then no downgrading is
        performed.

What is this for? I assume this is a compliant MDA. Or did you mean
_up_grading?

    4.  If MDA detects that SMTP recipient address is downgraded IMA,
        then MDA MUST decode IMA and perform the same processing as if it
        were IMA.  MDA MAY normalize or canonicalize local-part before
        processing it.

s/downgraded/ACE-downgraded/ (because if it was ALT-ADDRESS downgraded
you cannot do this - it is essentially the same as case #2 above). And
the MUST is probably too st even implement his method.

The problem is that, once people have implemented our new protocol, they
are not going to rewrite their software every time someone brings out a
new header (maybe with new downgrade rules). This is exactly the problem
that RFC 2047 ran into, because it assumes that, as soon as a new
standard specifies a new header, all the MIME software in the world will
immediately and magically "know" whether this new header is unstructured
or not and use (or not use) RFC 2047 encoding within it accordingly. The
result has been that all sorts of implementations for 2047 encoding of
headers where they should not be doing it, and vice versa. So let us try
to avoid that trap.

I therefore propose a "generic" downgrade method, to be used on all
headers whose state is unknown:

    Downgraded: Header-Name: {RFC encoded form of original header body}

(just like your section 5.4, in fact).

This is always safe; it works whether the header in question is
structured or unstructured; it is easily upgraded; and it will do no
harm whatsoever SO LONG AS that header plays no part in the transport
process (which is likely to be the case - most new headers are for
human consumption, or for use by specific gateways, etc).

The only snag is that it might not always suit the intention of the
author of the new header (who might have preferred you to use his own
specially devised downgrade). Tough! He will just have to define his new
protocol so that it can get around the use of this generic downgrade by
MTAs that were written long before he came on the scene.

As to the specific rules we need for addr-specs in presently known
headers, I think we know how we intend to do it. I would suggest
repeating the table I gave earlier showing how to deal with various
combinations of ALT-ADDRESS and ATOMIC, and after that the details are
very similar to the envelope procedures.

I have mentioned IRIs as a special case, because all the headers which
currently have URIs in them are surely going to be upgraded to use IRIs
before long (indeed, we could include such changes in our utf8headers
document if we so decided). Moreover, there is already a defined way to
downgrade an IRI to a URI, so we may as well use it.

And finally, the downgrade process should include instructions to
remove/modify/whatever the I18MAIL header (or whatever it gets called -
I still prefer the Header-Content header that I mentioned a week or so
back, and which would need to be modified to shown that downgrading had
taken place).

And finally finally there is still the matter of downgrading "body part
headers", as I mentioned earlier.

6.  Implementation consideration

6.1.  MUA

    MUA MAY encode UTF-8 in Subject header with the same encoding of body
    part while downgrading.

I am not sure I like that. Has it been discussed already?

    IMA compliant MUA MUST decode downgraded mail and MUST show IMA on
    display.

s/decode downgraded/upgrade/
But I am not sure this is really a MUST (is any interoperability
broken?). SHOULD by all means.

6.2.  MDA Requirements

    1.  MDA MUST NOT convert downgraded header to UTF-8.

Why not? Indeed, the new imap draft proposes to do just that (under
user's control, of course).

    2.  Record Return-Path header in ACE form.

This might not be possible (the MAIL FROM: may have been downgraded to
an ALT-ADDRESS en route, and all trace of the UTF8 form of it may have
been lost).

    3.  Perform downgrading for each Storage/Back-end-Process.  If and
        only if MDA knows MUA is IMA compliant, then no downgrading is
        performed.

What is this for? I assume this is a compliant MDA. Or did you mean
_up_grading?

    4.  If MDA detects that SMTP recipient address is downgraded IMA,
        then MDA MUST decode IMA and perform the same processing as if it
        were IMA.  MDA MAY normalize or canonicalize local-part before
        processing it.

s/downgraded/ACE-downgraded/ (because if it was ALT-ADDRESS downgraded
you cannot do this - it is essentially the same as case #2 above). And
the MUST is probably too strong.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima