Subject: [IMA] draft-smtpext-00

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 21 Oct 2005 22:01:39 +0100

As with my remarks concerning the headers draft, this is absolutely on the  
right lines, but again I have some issues, and some niggles.

1. The main problem is how an SMTP server will know that it is dealing  
with an I18N Message (I presume it is expected that these servers will  
also be handling vast quantities of the current RFC 2822 mail).

For sure, if it sees an ALT-ADDRESS parameter in a FROM or RCPT command,  
it will know, but some I18N messages will still have ASCII-only addresses.  
Moreover, this message may well have arrived at this server by a complex  
chain of gatewaying and processing by all sorts of intermediate  
transports, and all traces of its earlier history may have been lost.

Clearly, the server can examine the whole of the headers to see if bit-8  
is set in any octet anywhere, but that is tedious, especially since it has  
to repeated at every intermediate site. I suggest it would be far better  
to invent a new header something like
     I18N-Headers: yes
and say in the headers draft that all I18N Messages MUST include it. That  
feature would also be of use to other transport mechanisms, gateways and  
storage agents, all of which may need to apply special processing to these  
messages. (The precise name and syntax of this header does not matter for  
now - there may well be additional information that may need to be put in  
it.)

2. We realise that the Received header is going to be troublesome,  
especially as regards its "for" clause, but it not quite so bad as has  
been suggested as regards its "from" clause, since messages will generally  
arrive from other servers which are also handling old-fashioned ASCII  
messages, and whose own domain names will therefore already be in ASCII.  
So you are not likely to see much UTF-8 in there (and also, we are already  
agreed that EHLO must always be in ASCII - though you do not seem to have  
mentioned that).

3. Your document needs to be much more clear that it is doing two jobs:
     a) Internationalizing the envelope (with downgrading or bouncing as  
needed)
     b) Dealing with I18N within the messages (again, with downgrading).
So you need another section within 2.5 dealing with the DATA command  
because, if it is an I18N Message, that is where the downgrading of the  
headers in the message will have to be done (and maybe of the body too if  
MIME multipart or message types are in there).

So, looking at section 2.2, last paragraph, please make it clear that it  
is about _envelope_ processing (how to construct the FROM and RCPT for the  
next hop). I suggest s/replace/downgrade/. And if you are going to mention  
the IDNA rules, you may as well mention 'Nameprep' explicitly.

The rest is mostly niggles:

4. In 2.3, I sujspect the definition of 'Atom' will turn out to be more  
restrictive that arbitrary UTF-8. In any case, you will need to give ABNF  
in due course.

5. In 2.4, you will need ABNF for ALT-ADDRESS (the job of the Downgrade  
document is to tell you how to derive it, not what it looks like).  
Actually, it will be exactly what RFC 2821 currently permits (it is the  
main FROM/RCPT address that is going to be different).

6. In 2.5, you need to deal with VRFY and EXPN. Although it is not  
fashionable to implement them these days, they could actually be quite  
useful here to give the client a preview of what the downgraded address is  
going to look like before he comits to letting you do it.

7. In 3, current local-parts are not made up of "any ASCII". There are  
some restrictions (though probably not enough :-( ). As regards current  
attempts at I18N using backspaces and so on, how prevalent is this? One of  
the mistakes (IMO) in RFC 2822 was allowing NO-WS-CTL in local-parts and  
other places. It would be nice if we could simply outlaw the use of  
backspace (and the rest).

Incidentally, there is another custom used in German and other languages,  
which is to replace "münchen" by "muenchen". That too could cause  
troubles, but would be harder to forbid.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Sat, 22 Oct 2005 11:12:52 -0400

Charles,

While I'm not the author, let me try to answer a few of these
questions...

--On Friday, 21 October, 2005 22:01 +0100 Charles Lindsey
<[email-masked]> wrote:

> As with my remarks concerning the headers draft, this is
> absolutely on the  right lines, but again I have some issues,
> and some niggles.
> 
> 1. The main problem is how an SMTP server will know that it is
> dealing  with an I18N Message (I presume it is expected that
> these servers will  also be handling vast quantities of the
> current RFC 2822 mail).

Please be careful here, since this whole model hinges on what is
going on in the envelope, starting with the server advertising
IMA capability in the EHLO response.  That advertisement
specifies that it is willing and able to:

	* deal with an envelope address (local-part and/or
	domain) in UTF-8 should one arrive in any forward or
	reverse path. 
	
	* deal with an ALT-ADDRESS parameter attached to the
	forward or reverse path.   The document is not, if I
	recall, specific about whether such a thing is permitted
	if the (primary) address is all-ASCII; we need to
	discuss that and make a rule.
	
	* deal with UTF-8 headers should they arrive.

An "i18n message" is defined by the presence of at least one of
those things.  There is also an underlying theory in all of
this, which perhaps the docs don't make clear.  If you (or
someone else) can suggest text in the next few hours, there is
(barely) time to get a new version of the framework piece posted
before the deadline -- given the paucity of comments suggesting
changes, I had concluded that was unnecessary, but could change
my mind.

That theory is that an EHLO response of "IMA" is essentially an
assertion that the server is i18n-clean, i.e., that it doesn't
need to do anything special to handle these messages.  The only
exception would arise if downgrading were required in a future
step and it was prepared to do that.   It is not an assertion
that the server is prepared to go out and do strange and unusual
internationalization-specific things other than applying IDNA
before going to the DNS for any non-ASCII domain name labels.

For the servers I've poked at in looking at this extension, that
is also the right way to implement this.  Making a server
8bit-clean at this level makes much more sense than detecting
IMAs as somehow special and generating a new code path.
Conversely, if a server is somehow 8bit-hostile, tolerating
these changes is going to take a lot more than detection and a
special case.

> For sure, if it sees an ALT-ADDRESS parameter in a FROM or
> RCPT command,  it will know, but some I18N messages will still
> have ASCII-only addresses.  Moreover, this message may well
> have arrived at this server by a complex  chain of gatewaying
> and processing by all sorts of intermediate  transports, and
> all traces of its earlier history may have been lost.

History or state is not needed.  All the server needs to do is
to announce that it has the relevant capability (otherwise, it
is all a client problem) and then handle whichever of those
three elements it gets (if any) as if they are normal, which
they should be.

> Clearly, the server can examine the whole of the headers to
> see if bit-8  is set in any octet anywhere, but that is
> tedious, especially since it has  to repeated at every
> intermediate site. I suggest it would be far better  to invent
> a new header something like
>      I18N-Headers: yes
> and say in the headers draft that all I18N Messages MUST
> include it. That  feature would also be of use to other
> transport mechanisms, gateways and  storage agents, all of
> which may need to apply special processing to these  messages.
> (The precise name and syntax of this header does not matter
> for  now - there may well be additional information that may
> need to be put in  it.)

I see three problems with this.  First, headers are of no use to
an SMTP server or relay that is trying to follow the "don't mess
with headers or content" rule.  And that rule is still our
target norm, antispam issues notwithstanding.

Second, by the time a server finds that header, it will have
already processed the envelope and might well have already
encountered UTF-8 information, so the information might be
useless.  Conversely, requiring that all i18n messages must
include the header is easy, requiring that all non-i18n messages
NOT include it is a lot harder.  That leads us down a path in
which the right thing for an IMA-capable client to do is simply
to insert that header, always, whether a particular message
needs it or not.  And that takes it very far down the path
toward useless noise.

Finally, remember that gateways, SMTP implementations on
firewalls, and even submission servers, are routinely dropping
headers they don't understand these days.  So there is no
guarantee that the header will arrive and a receiving server
that announces IMA capability must be prepared for i18n data
regardless of whether the header appears, just to protect itself.

> 2. We realise that the Received header is going to be
> troublesome,  especially as regards its "for" clause, but it
> not quite so bad as has  been suggested as regards its "from"
> clause, since messages will generally  arrive from other
> servers which are also handling old-fashioned ASCII  messages,
> and whose own domain names will therefore already be in ASCII.
> So you are not likely to see much UTF-8 in there (and also, we
> are already  agreed that EHLO must always be in ASCII - though
> you do not seem to have  mentioned that).

This may need either more discussion, or clarification in the
document, but I continue to believe that domain names in
Received "from" clauses are going to need to be in punycode if
they are IDNs.  Similarly for EHLO -- I just don't see any
choice there.

>      a) Internationalizing the envelope (with downgrading or
> bouncing as  needed)
> So you need another section within 2.5 dealing with the DATA
> command  because, if it is an I18N Message, that is where the
> downgrading of the  headers in the message will have to be
> done (and maybe of the body too if  MIME multipart or message
> types are in there).

This material was intended to be in the framework document or
the downgrade one.  I think it is, but the document boundaries
may need more sorting in the next iterations.

My own assumption is that, if an IMA-capable server receives a
message and is then required to pass it on to an IMA-incapable
system, it must then (and only them) make a reliable
determination of whether the particular message requires IMA
facilities.  That means an octet-by-octet scan of envelope
address fields and of the headers.  I think the body is an issue
or not depending on whether the next-hop server supports
8BITMIME and should follow its rules, although one of the tasks
that ought to get onto the list or into the downgrade document
is a careful analysis of the impact on all of this on internal
MIME headers associated with multipart or message.

> So, looking at section 2.2, last paragraph, please make it
> clear that it  is about _envelope_ processing (how to
> construct the FROM and RCPT for the  next hop). I suggest
> s/replace/downgrade/. And if you are going to mention  the
> IDNA rules, you may as well mention 'Nameprep' explicitly.

The latter is a matter of theory.  In principle, IDNA is atomic
wrt any application that uses it.  If we need to start talking
specifically about, or dissecting, what Nameprep does, what
punycode does, etc., we are going to need to redesign the IDNA
specification, since some things that we informally think of as
Nameprep are actually done elsewhere, etc.

> The rest is mostly niggles:

I'll leave these to the author to comment on, at least
initially, except for the 7th...

>...
> 7. In 3, current local-parts are not made up of "any ASCII".
> There are  some restrictions (though probably not enough :-(
> ). As regards current  attempts at I18N using backspaces and
> so on, how prevalent is this? One of  the mistakes (IMO) in
> RFC 2822 was allowing NO-WS-CTL in local-parts and  other
> places. It would be nice if we could simply outlaw the use of

Please read 2821 (or 821) carefully.   Some things require
quotes and escapes, but...   As far as "backspaces and so on",
it is one of those architecture-defining cases.  Whether it is
appropriate to eliminate it now is another question, but the
answer, IMO, is probably "no", at least until we have a
widely-deployed alternate solution.

> Incidentally, there is another custom used in German and other
> languages,  which is to replace "münchen" by "muenchen". That
> too could cause  troubles, but would be harder to forbid.

Open that can of worms, and two-meter rattlesnakes come out.  It
is a good example of why I don't believe one can automatically
derive an all-ASCII address or phrase from an i18n one, and an
even better example of why "upgrading" is not on the table, but,
beyond that...

      john

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 25 Oct 2005 22:58:48 +0100 (BST)

John Klensin wrote

questions...

        * deal with an envelope address... in UTF-8 should one arrive...

        * deal with an ALT-ADDRESS ....   The document is not, if I
        recall, specific about whether such a thing is permitted
        if the (primary) address is all-ASCII...

Yes, it specifically allows that. Moreover, it is essential that it
should. Consider the following scenario:

An I18N email is sent to a non-ASCII address (maybe with Alternate),
and with UTF-8 in various headers including, say, To: and Subject: and
passes through servers with IMA capability until it reaches its
destination. So far so good. At the destination server, however, it
gets forwarded (something in the /etc/aliases file, perhaps) to an
ordinary ASCII address. So there is nothing in the forwarded envelope
to show that it is an I18N message, but it still contains UTF-8 in the
To: and Subject: headers (which might still need to be downgraded if
some subsequent agent does not do IMA). I don't think we want to
disallow that perfectly normal behaviour.

those things.  There is also an underlying theory in all of
this, which perhaps the docs don't make clear.  If you (or
someone else) can suggest text in the next few hours, ...

Sorry, I am away from home at the moment with limited email access,
and an unfamiliar email system.

assertion that the server is i18n-clean, i.e., that it doesn't
need to do anything special to handle these messages.  The only
exception would arise if downgrading were required in a future
step and it was prepared to do that.

Sure, but if the next server (or MDA) in line doesn't do IMA, then it
MUST either downgrade or bounce. But if the message is _not_ an I18N
message (we assume that MTAs will be handling a mix of both kinds of
message) then it MUST NOT bounce. Therefore, if the next server does
not announce the IMA capability, the current server MUST know somehow
whether the mesage in hand is an I18N one, and if there is no clue in
the envelope (see above) then it HAS either to scan all the headers
looking for bit-8, or else there HAS to be some information in the
message to indicate so. If the bulk of the messages handled by the
server are ASCII only (which will be true of all servers initially,
and of most servers in N.America permanently), then that is a lot of
extra work for it to do.

Sure. This server will be 8-bit clean. But it is being prepared for
the next server downstream being 8-bit hostile that is going to be the
messy part to implement.

And there is good precedent for having the information in a header in
the message. 10 years ago, when 8BITMIME appeared, there was such a
header - Content-Transfer-Encoding. If the CTE was 7bit or Q-P or
Base64, the server didn't need to bother with any special actions - it
just sent the message on, 8BITMIME or not. But if the CTE was 8bit and
the next server did not advertise 8BITMIME, it was required to
downgrade (and there was not even a bounce option in those days).

>> include it.

an SMTP server or relay that is trying to follow the "don't mess
with headers or content" rule.,,

  And that rule went out when 8BITMIME appeared.

already processed the envelope and might well have already
encountered UTF-8 information,...

and it might not

>  Conversely,... requiring that all non-i18n messages
NOT include it is a lot harder.  That leads us down a path in
which the right thing for an IMA-capable client to do is simply
to insert that header, always,...

Fine, but it then takes the risk of having its messages bounced.

firewalls, and even submission servers, are routinely dropping
headers they don't understand these days.

But IMA capable servers would not be doing that. And the message will
never reach an 'old' server unless it has already been downgraded, at
which point we don't really care if that header gets lost.

they are IDNs

Sure, you have to be prepared for that. I was only pointing out that
servers which also handle non-IMA traffic are unlikely to insert their
own identity in a Received header in UTF-8.

Charles