Subject: [IMA] Identifying i18n messages

From: John C Klensin <[email-masked]>
Date: Thu, 08 Dec 2005 09:32:52 -0500

Hi.

I've been following this discussion and trying to work out the
case analysis.  Dave's comparison to Return-path was quite
useful to me although I drew slightly different inferences from
it (not unusual).

In my first round of thinking about this, I was strongly
influenced by the implementation and deployment experience with
8BITMIME.  With that option, virtually all implementations that
advertise the option do so on the basis of "I can handle this,
transparently, if you send it".  As far as I know, very few
transport or MUA implementations exist that accept 8BITMIME and
then handle it any differently from ordinary 7bit messages.  But
those that might need to do so do have the needed
content-transfer-encoding header to give them a clue.  I also
didn't go back and redo that analysis adequately when the idea
of UTF-8 headers was introduced and, with it, the possibility of
having UTF-8 headers without any envelope symptoms.

I've also been concerned about depending on a header that would
either force header ordering dependencies or that might be
casually deleted in transit, especially by none-too-bright
gateway or firewall implementations of SMTP.

I think, after working through all of the cases I could find,
that we need a two-pronged approach:

(1) If i18n features are being used, the SMTP client must
announce that fact via a parameter.  We need to sort out whether
that parameter is optional if one of the downgrade parameters is
specified or must be specified in any case but, assuming the
latter, this would be something like "I18N" on the MAIL command
(the first command that is guaranteed to be present in the mail
transaction).

(2) We invent a new header that identifies the presence of i18n
information (particularly UTF-8 headers) in the message.  While
we can't require header ordering, it would be good to have it
appear as near the top of the headers as possible.  It would
also be good to be able to (as nearly as possible) guarantee
that it will be there when the message is dropped into a mail
store.  I think this means:

(2a) The header MUST be inserted, along with (and either before
or after) Return-path, by the final delivery MTA.

(2b) The header MAY be inserted by the originating MUA or, on
its behalf, by the submission MTA.

(2c) The header, if present, MUST be removed as part of any
downgrading process that eliminates the UTF-8 header information.

(2d) An MTA that inserts the header MAY check for duplicates (an
instance of the header that is already present) and eliminate
all but one of them.  However, if a receiving MUA encounters
more than one of these headers, it SHOULD simply ignore any
excess ones without comment (I have not convinced myself yet
that this could safely be a "MUST", and that may partially
depend on (3)).

Note that combination guarantees that the header will be present
on delivery even if it is deleted in transit.  That is probably
an important case; I don't know how to get it in any other way.

I haven't come up with a good name for the header, but would
suggest that, given our bad experiences so far with both IMA
(another WG) and IEE (an historical component, and registered
trademark, of IEEE), any names for header fields or parameters
be based on something like "i18n-email" rather than on IMA or
IEE.   I also haven't sorted through the syntax of that header.
I have tried to figure out if we could just use "MIME-Version:
2.0" (since this really requires a different parsing model that
is able to deal with non-ASCII text), but I think I've convinced
myself that is not a good idea -- others should repeat the
exercise themselves, since it would be attractive to not invent
a completely new header requirement.   If the "MIME-Version"
approach were used, insertion by the originating system would,
of course, become a "MUST" but not change anything else.  

Having the header potentially has another benefit, which is that
it would provide a place to put the alternate address
information that Charles wants to see in the headers.  I am
still not convinced that is a good idea or ought to be a
requirement and the syntax might get complicated, but the "can
be inserted by the originating MUA but is guaranteed to be there
on delivery" property that I think this header needs to have is
probably the right one for those addresses.

The above obviously raises as many questions -- about names,
syntax, and edge conditions-- as it answers but (i) does it
sound like a plausible approach that meeting the perceived
needs?  And (ii) if so, can we start sorting out the details?

    john

(3) 

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 09 Dec 2005 11:50:43 -0000

On Thu, 08 Dec 2005 14:32:52 -0000, John C Klensin <[email-masked]> wrote:

> announce that fact via a parameter....

Essentially, that is conveying this information in the envelope, as well  
as in some 'label' header. But it only works in parts of the protocol  
where something like the "envelope" concept exists.
> or after) Return-path, by the final delivery MTA.

That is the latest point to insert it, but ideally it should be there  
already. And you need a definition of "final delivery MTA" that covers all  
the edge cases, such as gateways and list expanders.
> its behalf, by the submission MTA.

I would make that SHOULD, at least.
> downgrading process that eliminates the UTF-8 header information.

Not necessarily.
> that this could safely be a "MUST", and that may partially

Again, duplicates SHOULD be avoided. Depending on what extra information  
we might convey with this header, they might even be harmful.
> IEE.

Given that we have a Content-Type header to describe the format of the  
body of the message, I would suggest this be a "Header-Type". Then it  
would need two keywords - let us call them "OLD" and "NEW" for now (I  
agree the final choice is tricky), where absence of this header implies  
OLD.

Then we might find the need for further parameters (so perhaps it could  
incorporate what Yoneya suggested in his X-IMA-Downgraded: header, and  
other useful stuff). So it might look like

     Header-Type: NEW; language=jp_JP
     Header-Type: OLD; downgraded-by="site-which-downgraded-it"

NOTE. Bruce Lilly will shortly be along to remind us of RFC 2277, which  
REQUIREs us to provide langage information somewhere, so we will have to  
think about it at some stage. OTOH, we do NOT want any "charset=utf-8"  
parameter.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Dave Crocker <[email-masked]>
Date: Sun, 11 Dec 2005 10:12:39 -0800

Folks,

PROBLEMS:

The various threads that have explored different ways to make IEE 
messages -- as distinct from 'legacy' messages -- seem to have uncovered 
the usual problems of trying to run a dual-stack service.  In effect, 
IEE messages are not compatible with the existing message-parsing 
infrastructure, so we are trying to find a way to distinguish them, 
without breaking anything.

The SMTP option approach essentially checks for a safe path between 
originator and recipient.  There is a long-term appeal to this approach, 
given that it provides the cleanest handling of the new object itself. 
Unfortunately this approach has an enormously high barrier to adoption, 
since it requires changing much of the SMTP infrastructure before 
gaining much utility. I am afraid, therefore, that this looks like a 
good *adjunct* procedure, rather than an appealing first step.

Changing the message header, to add a single field -- as a flag -- or 
multiple fields -- as alternate content -- makes the header a mess to 
parse, creates dangers with synchronization across headers, and 
otherwise looks like an attempt to merge two, frankly incompatible formats.

One almost wishes that email had an "envelope" header section, as 
distinct from the "content" header.  (In effect, this serves to create 
an ordering between 2 sets of header fields.)  After all, the reason 
that Content-type works, in describing alternate forms of data, is that 
it is guaranteed to occur before the content it labels.  Hence it does 
not impose a 2-pass parsing model.

TUNNELING:

In looking for an approach that might be simple and sufficient, it 
occurs to me that there is an easy way to achieve a similar effect, that 
is entirely comfortable within the current SMTP *and* Header processing 
environment:

    Content-type: message/iee

The new form is entirely packaged as a body-part, within an old-form 
message.  There is no danger to existing parsers or handlers.  And there 
is explicit labeling of the new form.

Best of all, this is an approach that is quite similar to that done in 
adopting MIME and ESMTP.  That is, they have proved complementary, with 
a low barrier to adoption and quite a bit of benefit for early adopters.

So, the legacy header is used as an envelope header, and the header 
within message/iee is used as the "real" rfc2822bis message header.

The RFC3297 negotiation mechanism ought to nicely support this 
enhancement, too.

d/
-- 

  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

---

From: "Edmon Chung" <[email-masked]>
Date: Sun, 11 Dec 2005 14:43:08 -0500

I think this is not a bad idea.  If I understand correctly, you are suggesting to have an additional part of a multipart for an IEE Header, perhaps it could be:

Content-type: header/iee

That would be similar to or allow something I have chatted with other folks before, which is, to have a completely different header section for Internationalized email.  In fact, to take that further, it could be a foundation for additions to header data in the future.

The header portion can even be structured in xml to allow for extensibiltiy, perhaps:

Content-type: header/xml

Which will allow by default use of UTF8 data.

I do not intend to sidetrack the discussion of IEE here, but I think it might be worth exploring since we already need to address this issue, perhaps it is worth taking a look at this.  The advantage of using an xml scheme is to allow for extension and we can start small, focusing only on IEE required fields.  If there happens to be interest in the future, it is then possible to incorporate much more fields (for which may not be necessary to repeat in the current header).

Edmon

---

From: Dave Crocker <[email-masked]>
Date: Sun, 11 Dec 2005 12:53:31 -0800

Edmon Chung wrote:
> I think this is not a bad idea.  If I understand correctly, you are
> suggesting to have an additional part of a multipart for an IEE
> Header, perhaps it could be:
> 
> Content-type: header/iee

Since there is no "major" type of "header", I am not suggesting creating
one.

I am suggesting a new form of "message" content type, so that
rather than Content-type:Message/RFC822, there is
Content-type:message/IEE.  If would contain one entire message, complete
with RFC2822bis IEE header and the associated body content.

So a message would be:

RFC2822 header

    Mime-header for message/iee

       RFC2822bis IEE header

       MIME body for IEE header

Of course, there is nothing preventing more elaborate uses, in which the 
message/iee portion is merely one body-part out of several.

-- 

   Dave Crocker

---

From: "Edmon Chung" <[email-masked]>
Date: Sun, 11 Dec 2005 20:55:33 -0500

OK. So essentially it would only be a "stub" for IEE information?
Edmon

---

From: John C Klensin <[email-masked]>
Date: Mon, 12 Dec 2005 13:19:39 -0500

Dave,

There are several different issues here, some technical and some
procedural.   I will try to respond to the technical ones in a
separate note, but want to address the procedural one first.

Partially at the suggestion of the ADs, this effort is oriented
toward exploration of one particular approach toward
internationalization of email.  The intent is to explore that
approach through experimental RFCs, implementation, and
evaluation of those implementations.  While, based on doing a
lot of analysis on my own, I personally have my doubts, other
approaches are, of course, possible.  But neither exploring them
nor proving that this approach is the best of all possible ones
are the topic of this effort.  

The clear requirements for the present effort include being able
to use i18n addresses on the network without loss of
functionality.  More specifically, the objective is to permit
those who want to use non-ASCII local parts in addresses to do
so without incurring any long-term penalties vis-a-vis those who
happen to prefer to write their addresses entirely in ASCII
characters.  Message encapsulation may well have a role in
downgrading (indeed, it is my personal hunch that will turn out
to be the most viable -- or least-unviable) of downgrade
approaches), but, for conventional use in conventional email,
forcing a different message structure on i18n email than that
which is used for mail with ASCII-based addresses is not
acceptable --at least within the bounds and focus of the current
effort. 

Within the bounds of the thinking that has motivated this
effort, the reason for UTF-8 headers isn't as an end in
themselves, but because i18n addressing appears to make those
headers the least complex option (an insight for which I need to
thank Paul Hoffman).

In addition to those requirements, there is an assumption that
should probably be explicit if it is not clear already:

	The primary target is effective, long-term,
	communication within communities that use the same
	language in a way that is compatible with the Internet
	email infrastructure.  The other cases are less
	important in practice because, if I send you a message
	whose addresses and content you can't read, can't type,
	and may not even be able to accurately cut and paste, it
	isn't going to do you a tremendous amount of good.
	Because of that, it also won't be a common case.  It is
	obviously desirable that you be able to translate (or
	decode) it if you want to --which is one of many reasons
	why formats, infrastructure, and definitions need to
	remain within the bounds of a single Internet standard--
	but, if that is a tad more difficult that it might be as
	a consequence of optimizing things for users who share
	languages and scripts, that is probably reasonable.
	Conversely, imposing kludges --or different message
	structure organization-- on anyone who chooses to use a
	non-ASCII address for communication with others within
	their language and script community is not reasonable.   

I note that we got this right with MIME: the "content-type:
text/plain; charset=XXX" structure is the same regardless of the
value of XXX -- we did not require an embedded message structure
for anyone using a non-ASCII charset.

One corollary to that assumption is that these changes will
deploy rapidly within the self-defined communities that see
themselves as needing it most.  Note that statement is nearly
tautological: if I'm part of a community that, as a community,
is very anxious to deploy i18n email, then I, and that
community, will see that the needed pieces are in place for our
use.  Downgrading is important for people who are part of the
community from a cultural and linguistic perspective but not
part of it geographically or topologically.   It is less
important (one can debate how much less, but I'm not convinced
that question is interesting) for communications between
communities.

The goal of this effort thus focuses on finding a way to do i18n
email that will serve the community who want to use it well
_and_ avoid causing harm or unexpected surprises to those for
which it is less important or irrelevant.   Just as it does not
aspire to have me learn Chinese or you learn Swahili (much as
either of us might aspire to do so), it puts much less emphasis
on effective communication and regular use across languages,
scripts, and cultural groups.

The more technical note will follow, but please consider it in
the context of the above and, if you want to go to work on a
different approach, please take it somewhere else.

     john

So, at a procedural level

--On Sunday, 11 December, 2005 10:12 -0800 Dave Crocker
<[email-masked]> wrote:

>...
> TUNNELING:
> 
> In looking for an approach that might be simple and
> sufficient, it occurs to me that there is an easy way to
> achieve a similar effect, that is entirely comfortable within
> the current SMTP *and* Header processing environment:
> 
> The new form is entirely packaged as a body-part, within an
> old-form message.  There is no danger to existing parsers or
> handlers.  And there is explicit labeling of the new form.
>...
> So, the legacy header is used as an envelope header, and the
> header within message/iee is used as the "real" rfc2822bis
> message header.