Subject: [IEE] WG Review: Email Address Internationalization (eai)

From: IESG Secretary <[email-masked]>
Date: Tue, 21 Feb 2006 13:05:19 -0500

A new IETF working group has been proposed in the Applications Area.  
The IESG has not made any determination as yet. The following draft charter was
submitted, and is provided for informational purposes only. Please send your
comments to the IESG mailing list ([email-masked]) by March 1st.

+++

Email Address Internationalization (eai)
========================================

Current Status: Proposed Working Group

1.1. Chair(s)
TBD

1.2. Applications Area Director(s)
Ted Hardie <[email-masked]>
Scott Hollenbeck <[email-masked]>

1.3. Applications Area Advisor
To be determined

1.4. Mailing Lists
General Discussion: [email-masked]
To Subscribe: https://www1.ietf.org/mailman/listinfo/ima
Archive: http://www1.ietf.org/mail-archive/web/ima/index.html

1.5. Description of Working Group

Since early in the effort to internationalize domain names, which
resulted in the standards associated with IDNA, it has been
understood that internationalization of email address local parts is
even more important. After all, most prefer variations on their
names for those addresses. At the same time, email address
internationalization poses a series of special problems. Constraints
on the interpretation of local-parts by any system other than the
final delivery one --constraints that go back to before RFC 821 and
that have been vital to the operation of the Internet's email
environment-- make address encoding nearly impossible. The need to
use addresses in both the email envelope and in header fields, and to
do so in ways that are at least compatible, suggests that this is not
a simple and isolated problem.

This working group will address one basic approach to email
internationalization. That approach is based on the use of an SMTP
extension to enable both the use of UTF-8 in envelope address local-
parts and optionally in domain-parts and the use of UTF-8 in mail
headers -- both in address contexts and wherever encoded-words are
permitted today. Its initial target will be a set of experimental
RFCs that specify the details of this approach and provide the basis
for generating and testing interoperable implementations. Its work
will include examining whether "downgrading" -- transforming an
internationalized message to one that is compatible with unextended
SMTP clients and servers and unextended MUAs -- is feasible and
appropriate and, if it is, specifying a way to do so. If it is not,
the WG will evaluate whether the effort is worth taking forward.

Key parts of this effort include extended analyses and, if necessary,
proof of concept in three areas in addition to smooth operation when
all systems and components along a message path have been upgraded to
support the new facilities. They are

o Examination of scenarios for the appearance of these facilities to
users, including ways in which alternate addresses may be
specified if those are needed for downgrading.
o Examination of different locations at which downgrading might be
required and accomplished, differentiating between requirements
and capabilities at the point of origin (at or before the
submission server), those that exist while the message is in
transit, and those that apply after SMTP "final delivery" or in
the logical vicinity or an IMAP or POP server.
o Examination of the "mailing list question", i.e., how a mixture of
traditional and internationalized addresses on a mailing list will
impact message flows, error reports, and delivery notifications in
all plausible combinations of servers and addresses, including
internationalizated and traditional reverse paths.

Once the Experimental RFCs are completed and implemented, the
experience gathered will be evaluated. If the approach is found to
have been successful (using criteria the WG will establish as an
early work item), the WG will be rechartered to update the documents
for processing onto the standards track.

1.6. Deliverables

The following deliverables are foreseen in this charter. The WG
chairs may structure the deliverables into specific documents
or document sets as needed. Adding or removing documents
outside of these deliverables will require a charter update.

o Overview and architecture (Info)
o Interworking scenarios, including the "mailing list question"
(Info)
o SMTP extensions specification (Exp)
o Header format specification (Exp)
o Downgrading specification in SMTP (Exp)
o Downgrading specification in POP servers (Exp)
o Downgrading specification in IMAP servers (Exp)
o Results and evaluation of experiment (Info)

Going forward, it is possible that the SMTP downgrading specification
will go for Informational due to the difficulty of fully specifying
all necessary behaviours.

NOTE IN DRAFT: Additional possible documents suggested:
Advice for MUA implementors (Info)

1.7. Goals and Milestones

(Very tentative so far)

+-------------+--------------------------------------+--------------+
| DONE | Overview/architecture draft first | |
| | draft | |
| Feb 2006 | Interworking scenarios first draft | |
| DONE | SMTP Extensions first draft | |
| DONE | Header format first draft | |
| Mar 2006 | Downgrading in SMTP first draft | |
| Mar 2006 | Downgrading in POP first draft | |
| Mar 2006 | Downgrading in IMAP first draft | |
| Jun 2006 | Overview/architecture draft to IESG | |
| Jun 2006 | Interworking scenarios to IESG | |
| Sep 2006 | SMTP Extensions to IESG | |
| Sep 2006 | Header format to IESG | |
| Sep 2006 | Downgrading in SMTP to IESG | |
| Sep 2006 | Downgrading in POP to IESG | |
| Sep 2006 | Downgrading in IMAP to IESG | |
| Dec 2006 | Results and evaluation first draft | |
| Mar 2007 | Results and evaluation to IESG | |
| Mar 2007 | Group recharter for standards track | |
| | or disband | |
+-------------+--------------------------------------+--------------+

Table 1

1.8. Internet-Drafts

Drafts listed below were posted as initial discussion documents,
significantly preceeding the meeting at IETF 64 (Vancouver, 2005
Oct).

2005.06.27 draft-lee-jet-ima-00.txt
2005.07.18 draft-klensin-emailaddr-i18n-03.txt

Drafts listed below were prepared and posted to inform the discussion
at IETF 64. They replace those above.

o draft-klensin-ima-framework-00.txt
o draft-yao-ima-smtpext-00.txt
o draft-yeh-ima-utf8headers-00.txt
o draft-yoneya-ima-downgrade-00.txt

Drafts produced by the WG

None as yet

1.9. Request For Comments

None as yet

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Martin Duerst <[email-masked]>
Date: Wed, 22 Feb 2006 17:37:59 +0900

At 06:37 06/02/22, Larry Masinter wrote:
 >Should the group to also consider internationalization
 >of "mailto" URIs,
 >
 >         draft-duerst-mailto-bis-01.txt
 >
 >because "email address internationalization" isn't fully
 >specified without doing so.

Hello Larry,

That's a good question. I asked that question previously, at
http://www1.ietf.org/mail-archive/web/ima/current/msg00082.html.
You were copied on that mail, but probably not on all of the
followup.

I think the general opinion (most prominently voiced by
John Klensin) on the ima mailing list was that the WG should
review draft-duerst-mailto-bis-01.txt, but not tie it up in
formalities.
(see http://www1.ietf.org/mail-archive/web/ima/current/msg00087.html).

Did you come to a different conclusion, or are you just asking
to make sure this has been thought about?

Regards,    Martin.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 22 Feb 2006 11:48:07 -0000

> permitted today.

s/both/including/

There may well be other headers, whether in current use or to be invented,  
and even possibly invented as part of this exercise, where the UTF-8  
approach might be introduced. For example, headers containing IRIs as  
oposed to URIs, and the Newsgroups header, in the event that this work is  
later extended to Netnews Tbis is not included in the present charter, but  
is a likely extension in the future. The WG needs to address issues that  
might arise with such future headers (e.g. a general-purpose fallback  
downgrading mechanism).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Dave Crocker <[email-masked]>
Date: Thu, 23 Feb 2006 10:43:43 -0800

Folks,

This is clearly a necessary effort, to enhance Internet Mail.

A few comments/questions about the charter:

Standard, general observation about the first paragraph:  IETF working group 
charters use their first paragraph very widely -- such as when sending notice to 
the ietf-announce list of the wg creation.  So, that it should summarize the 
problem and, at least, the deliverable of the working group.  In other words, 
the paragraph should stand on its own, as a summary of the working group.

> names for those addresses. At the same time, email address

What does "most prefer variations on their names for those addresses" mean?

Perhaps it means something like:  "most users of Internet mail would prefer a 
mailbox name that can be represented in characters other than basic ASCII"?

>  --constraints that go back to before RFC 821 and

Although I realize that there is quite a bit of history to this topic and even 
have some recollection of participating in at least one such conversation, and 
equally realize that the SMTP world (and RFC2822 world) of address do have some 
interesting limitations in the permissible characters for the mailbox string, I 
have not understood that an encoding scheme would be "impossible".

(As a counter-example, note that the BATV work on rfc2821.mailfrom has been 
doing special encoding work and has found a syntax that seems universally 
acceptable.  To be fair, that's not the syntax in the public draft of BATV, but 
a safer scheme HAS been developed.)

Therefore, what is the basis for asserting such an absolute barrier?

The approach of this working group is sufficiently radical/disruptive, so as to 
make it a good idea to have the charter cite or explain why an encoding overlay 
scheme is not viable.

>  Its initial target will be a set of experimental

If an effort has its primary goal be the production of experimental 
specifications, shouldn't that make it an IRTF activity, rather than an IETF 
activity?  This question is not a mere formality.

The entire tone and scope of the described work is quite candid (and I suspect 
accurate) about the effort.

What is described is research, not standards-making.  Based on the nature of the 
work defined here, it is going to be some years before the work is ready for a 
standards effort.

d/

-- 

Dave Crocker
Brandenburg InternetWorking
<http://bbiw.net>

---

From: John C Klensin <[email-masked]>
Date: Wed, 22 Feb 2006 08:42:13 -0500

Charles,

Just a heads-up on part of your comment below.

The WG will certainly work hard to find one, and this is 
personal opinion only, but I don't think there is going to be a 
"general-purpose downgrading mechanism" for addresses.

To have such a mechanism appears to require changing the very 
strict requirement of RFC 2821 and its predecessors that nothing 
other than the final delivery MTA ever try to figure out what a 
local-part means.  Given your netnews experience, you are 
presumably familiar with the difficulties we got into when 
various systems tried to outsmart the presumed explicit routing 
of bang-paths and %-hacks, especially when they were mixed. 
Many of us can also remember difficulties with intermediate 
systems --sometimes even originating systems that tried to 
second-guess the user-- that parsed the local part, applied 
their assumptions about what was going on, and then attempted to 
remap and optimize it, thereby trashing imbedded commands, 
subaddresses, and so on.

In the general case, if one has an i18n local part and wants to 
alter it into a traditional one in transit, the same issues 
apply: the intermediate system has to know how the destination 
host might interpret that address in order to downgrade it.

The current working drafts (versions of ima-framework and 
ima-smtpext that should appear within the next few days) contain 
two suggested mechanisms that should help with downgrading, but 
they are not the general case: requiring either or both causes 
other problems.

The difficulties with downgrading headers are a bit different 
but, because the future headers to which you refer cannot 
generally be parsed by a system that has never heard of them, 
the only fully-general way to downgrade a message body may turn 
out to require extracting critical information and then 
encapsulating the rest, using an extended version of 
message/rfc822 that permits Base64 or Q-P encoding of the 
headers.

If my pessimistic and conservative view is correct, the 
constraints on "general-purpose fallback downgrading" are such 
that a requirement for such downgrading are roughly equivalent 
to "no IETF standard i18n addresses".  Please note that I didn't 
say "no use of i18n addresses": I believe that, should the IETF 
opt-out of the solution, either by declining to charter the WG 
or by imposing enough constraints to make a result impossible, 
the inevitable consequence will be the adoption of a collection 
of local and non-interoperable solutions, not a globally 
interoperable solution that admits that, sometimes, there will 
be non-downgradable messages that will need to be bounced.

This issue is discussed further in another forthcoming I-D named 
draft-klensin-ima-constraints-00 .. I do not expect it to become 
a WG product, incidentally.

regards,
      john

--On Wednesday, February 22, 2006 11:48 AM +0000 Charles Lindsey 
<[email-masked]> wrote:

>> internationalization. That approach is based on the use of an
>> SMTP extension to enable both the use of UTF-8 in envelope
>> address local- parts and optionally in domain-parts and the
>> use of UTF-8 in mail headers -- both in address contexts and
>> wherever encoded-words are permitted today.
>
> s/both/including/
>
> There may well be other headers, whether in current use or to
> be invented,  and even possibly invented as part of this
> exercise, where the UTF-8  approach might be introduced. For
> example, headers containing IRIs as  oposed to URIs, and the
> Newsgroups header, in the event that this work is  later
> extended to Netnews Tbis is not included in the present
> charter, but  is a likely extension in the future. The WG
> needs to address issues that  might arise with such future
> headers (e.g. a general-purpose fallback  downgrading
> mechanism).

---

From: Dave Crocker <[email-masked]>
Date: Thu, 23 Feb 2006 17:13:29 -0800

Ted,

>  Constraints on the interpretation of local-parts by any system other than the
>  final delivery one make address encoding nearly impossible.
...

> I think the idea that the final delivery system may have some unique parsing of
> the local part is pretty strong--so you can identify lots of common syntaxes, but you
> could never be totally sure you got everyone's idiosyncratic behavior covered.

It is certainly known that some characters cause problems.  Given that the 
proposed effort is for experimental, one would expect to see experiments that 
seek to demonstrate that there is no reasonable solution, before doing larger 
violence to the email infrastructure.

Stated differently:  If the concern is that an encoding convention might break 
in some transit situations, we should validate the concern before adopting a 
convention that is expected to break in *all* transit situations.

> Can you publish the updated BATV draft, so we can see how you got around that?

Indeed, that project is already in the queue.

>> The approach of this working group is sufficiently radical/disruptive, so as to make it a good idea to have the charter cite or explain why an encoding overlay scheme is not viable.
> 
> Prior to the BoF in Vancouver, John produced draft-klensin-emailaddr-i18n
> (available at http://bgp.potaroo.net/ietf/idref/draft-klensin-emailaddr-i18n/),
> which went through some of this as background to the BoF discussion.  Are
> there specific aspects of that draft which it would be valuable to include here,
> or would a specific pointer to an updated doc help?

I haven't reviewed the paper in quite awhile.  As I recall, the document is not 
trivially short.  So a citation to the specific text that substantiates the 
claim would be appropriate.

> My personal answer here is that these are experimental specifications because
>  of the weight of standardization and deployment in this area. Making a 
> proposed standard change to header formats at this stage of the game without 
> some real-world experience to say it is worth it seems pretty risky. But that
> doesn't mean this is necessarily IRTF stuff. If someone were to propose IRTF
> work in this area, I would personally strongly urge them to start with a 
> green field and see what vision they could come together on. But that's not
> what we are doing here.

Ted, here is a list of the existing IRTF groups:

     * Anti-Spam Research Group (ASRG)
     * Crypto Forum Research Group
     * Delay-Tolerant Networking Research Group (DTNRG)
     * End-to-End Research Group Charter
     * Host Identity Protocol (HIP)
     * Internet Measurement Research Group
     * IP Mobility Optimizations (Mob Opts) Research Group
     * Network Management Research Group Charter (NMRG)
     * Peer-to-Peer Research Group
     * Routing Research Group Charter
     * Transport Modeling Research Group
     * Internet Congestion Control Research Group

As I understand the goals and activities of most of them, they are attempting to 
enhance existing services, rather than start 'with a green field'.

I know that DTN is an exceptions.  I believe crypto, e2e, hip, mobops, p2p, 
routing, transport and ccrg are not.

So, my question stands.

WHat I'll add is that one could argue that the pressure to produce "standards" 
has been one of the problems with recent email internationalization efforts. 
What should have prompted experimentation with alternatives has instead prompted 
locking into details before their efficacy has been demonstrated.

> I believe we are asking people to engineer a solution within an existing constrained
> environment, rather than research potential solutions without regard to the existing
> constraints.  

Again:  The IRTF generally does not have the goal of researching potential 
solutions without regard to existing constraints.  There I do not see the 
relevance of that distinction here.

d/
-- 

Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Fri, 24 Feb 2006 08:47:06 -0800

Folks,

...
> which went through some of this as background to the BoF discussion.  Are

I am finally remembering more of the discussion at a BOF about this topic.  The 
response to the concerns I expressed, then, landed on a statement that went 
along the lines of "15 years ago, we could not 'just do binary' email, but today 
we can.  In particular, support of UTF is now universal among hosts."

Upon thinking about this perspective, further, it occurred to me that the way to 
resolve the debate over whether to create another encoding overlay -- as we have 
done in the past -- versus make UTF the "native" form instead of ASCII, is to 
completely skip the negative side of the discussion.

In other words, don't have the debate.  Just pursue the positive.

One example of why the previous model of "debate" is problematic is that UTF-8 
is, itself, merely another encoding form. (Raw UTF is not an 8-bit 
representation, so UTF-8 is a means of encoding the raw stuff into bytes.) So, 
debating whether to use UTF-8, which does violence to the existing 
infrastructure, versus creating an encoding form that fits comfortably over the 
existing infrastructure, is a relatively problematic process, as we have been 
seeing for quite awhile.

For some years, the incremental enhancement of Internet mail has carried the 
meta-question of: when will it be more appropriate to change the infrastructure, 
rather than to add another micro-layer on top of it?  I think the current round 
of difficult internationalization effort legitimately suggests consideration 
that "we have reached it now", at least with respect to making a move from 
classic network ascii over to UTF, at least for some data.

So, instead of conducting the debate, the charter should strictly focus on a 
positive line of need, availability and benefit.

Here is some text that attempts to provide that alternative view, in the opening 
paragraphs of the charter:

/////
Given the scale and variety of the modern Internet, the email user experience 
needs to be enhanced, to support addresses and mail header fields comprising 
characters native to that wide variety of user environments.  Basic network 
ASCII does not permit this, but UTF does.  UTF is now widely supported in user 
systems. Therefore, it is Internet mail's reliance upon classic network ASCII 
that has become the bottleneck for the user experience, regarding email handling 
data and other meta-information.  The current effort will develop specifications 
for an SMTP option to permit the direct use of UTF-8 encoded data in RFC2821 
envelope address local-parts, domain-parts and RFC2822 mail header fields. It 
will also explore the ability to downgrade, when attempting to interact with 
legacy email services.  [[ Hmmmm... This also creates a requirement for 
specifying UTF-native RFC2822 header fields, doesn't it?  /d ]

Because the specifications attempt to make a significant change to the 
Internet's mail infrastructure, these specifications will target Experimental
RFC status. The work will include examining whether "downgrading" -- 
transforming a UTF-based message, to one that is compatible with unextended
SMTP clients and servers and unextended MUAs -- is feasible and
appropriate and, if it is, specifying a way to do so. If it is not,
the WG will evaluate whether the effort is worth taking forward.
/////

Charter language notwithstanding, I will, again, note that the question of 
whether downgrading can be reasonably performed, is sufficiently basic as to 
make this an IRTF task, rather than an IETF standards task.

d/

-- 

Dave Crocker

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 24 Feb 2006 12:15:17 -0000

On Wed, 22 Feb 2006 13:42:13 -0000, John C Klensin <[email-masked]> wrote:

> The WG will certainly work hard to find one, and this is personal  
> opinion only, but I don't think there is going to be a "general-purpose  
> downgrading mechanism" for addresses.
>
> To have such a mechanism appears to require changing the very strict  
> requirement of RFC 2821 and its predecessors that nothing other than the  
> final delivery MTA ever try to figure out what a local-part means.   
> Given your netnews experience, you are presumably familiar with the  
> difficulties we got into when various systems tried to outsmart the  
> presumed explicit routing of bang-paths and %-hacks, especially when  
> they were mixed. Many of us can also remember difficulties with  
> intermediate systems --sometimes even originating systems that tried to  
> second-guess the user-- that parsed the local part, applied their  
> assumptions about what was going on, and then attempted to remap and  
> optimize it, thereby trashing imbedded commands, subaddresses, and so on.

Sure. For the case of headers involving <addr-spec>s, the WG will invent  
special purpose downgrade mechanisms Including alternative addresses, and  
whatever else is needed,

Likewise, for headers involving <unstructured>s, <phrase>s, etc, it will  
invent a specific downgrade mechanism (essentially, using RFC 2047).

But for headers not yet invented, it will be hard to introduce a special  
purpose downgrading mechanism, because it will be hard to deploy it in the  
field. Therefore we need a general purpose mechanism. Clearly, such a  
mechanism would be unsuitable for any header that needed to be looked at  
by the transport system, but it would be fine for headers only intended  
for use by the ultimate user agent (headers for human consumption, for a  
start), or for headers that were only intended for specialized gateways.  
One can expect ultimate user agents and specialized gateways to have the  
capability to understand it (e.g. to upgrade it and then act upon it, or  
at least display it).

An example is the Newsgroups header, if Netnews ever adopts this system.  
Currently, if sent in UTF-8, it would already work fine with the existing  
news transport mechanisms (this has already been demonstrated, and there  
exists a Danish test group with a UTF-8 name). But if you wanted to  
gateway such a news article to a mailing list, the gateway would have to  
downgrade it. But, at that point, it ceases to fulfil any purpose for  
directing the transport and is just for human consumption, so it matters  
not if it arrives at systems that don't know how to upgrade it (though any  
user agent that was so capable might well be able to show the reader some  
information that might be of interest to him).
>
> In the general case, if one has an i18n local part and wants to alter it  
> into a traditional one in transit, the same issues apply: the  
> intermediate system has to know how the destination host might interpret  
> that address in order to downgrade it.

Sure. Any general-purpose fallback downgrading mechanism, would be quite  
unsuitable for <local-part>s.

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Feb 2006 15:00:28 -0500

Dave,

My apologies for not addressing some of your comments sooner --
I've been trying to get documents out that explain what is being
suggested here in context, rather than writing lengthy notes to
try to deal with separate questions and perspectives.    I want
to try to respond to what I think are some of your main points
in your recent notes.  If I have missed things that you consider
important, I'd appreciate your patience as I either reread notes
and find them or in identifying them again.

The issues I've identified below are not completely separate
either: a good deal of this falls into the "everything is
connected to everything else" problem whether we like it or not.

Issue 1: Why UTF-8?  At one level, you are correct that UTF-8 is
just another encoding.   But it is also a very special encoding
in the following way:

	* We started out with ASCII.  ASCII in addresses, ASCII
	in headers, ASCII in those protocol elements and
	exchanges for which we did not go to essentially binary
	data structures like ASN.1.  This wasn't US-bias: ISO
	and ITU went down the same path, calling ASCII or
	restricted ASCII by such names at ISO 646 and IA-5 and
	using them in almost every protocol that required some
	sort of control information and didn't use ASN.1 or
	another highly-structured coding.
	
	* For a number of purposes, when early
	internationalization efforts started, the preferred
	character sets were ASCII (aka ISO 646 IRV) based. In
	Europe and North America, the base character set used
	the high order bit of an 8-bit octet, but, if that bit
	was off, the code value was identical to ASCII prefixed
	by a zero bit.   ISO 8859-1 and indeed ISO 8859-N, for
	many values of N, have that property, sometimes called
	"extended ASCII".  In Asia, the same thing was
	accomplished by use of ISO 2022 switching or shifting
	sequences: if there were no shifts, the characters had
	the same codes as ASCII and were interpreted as ASCII.
	In either case, there is a certain upward-compatibility
	involved because, if one writes an ASCII character, one
	uses the ASCII code and an all-ASCII string in the
	alternate coding system looks (at the bit level) just
	like an all-ASCII string with ASCII right-justified in
	an eight-bit byte.   This was not a property of, say,
	EBCDIC or various EBCDIC-based code pages.  If one
	wanted to get back and forth between them the
	ASCII/ISO646BV/IA5 requirements of some protocol, one
	had to do actual code translation.  And one also needed
	to hope the all of the relevant characters appeared both
	character sets in exactly-translate-able form.
	
	* Now we get to UTF-8.  There are several reasons to
	dislike it, but it preserves the "ASCII is ASCII"
	properties of the above.  A collection of RFC822 header
	fields is valid UTF-8, both in the header names and in
	the data fields.  If UTF-8 is used to code the non-ASCII
	characters in Unicode and some of those characters are
	put into the data fields of what would otherwise be
	RFC822 headers the field names remain bit-wise identical
	with the original 822 forms even though the header set
	now requires UTF-8 to encode all of the data.  Even in
	the data fields, ASCII data remains ASCII data: we don't
	need state, or character-code labels, any number of
	other ways around the problem of keeping ASCII
	characters in ASCII coding while introducing other
	characters.  That is a very useful, and important,
	property.  It does not exist with UCS-2 or UCS-4
	encoding.  It would not exist if we somehow forced
	everything into punycode, or base64, or...   We might be
	able to invent other codings with the same properties,
	but the reality is that UTF-8 is fairly widely deployed
	and used and those hypothetical alternatives are not.
	And, for better or worse, there was some IETF consensus
	to favor UTF-8 when RFC 2277 was adopted.

So it is not really just one more coding.  It is ASCII-code-
preserving and hence preserving of all of the definitions and
properties of 2821 and 2822 except where non-ASCII characters
are actually used in a particular address or message.

Issue 2: The "impossibility" of downgrading addresses in
transport in the general case.  You are often credited, I think
correctly, with being the inventor of the %-hack for message
routing without using the explicit routes of RFC 821.  You
remember the difficulties we had when bang-paths and
percent-hacks were mixed and no one could figure out which one
to apply first (or many people could figure it out; they just
didn't agree).  Probably you remember the times at which
organizations with a small number of characters available for
addresses tried to encode all sorts of things into them for
internal routing purposes.  For example, when there was one
institutional gateway host, we might have PHYSJB@host and
CHEMJD@host to designate Joe Blow in the Physics department and
John Doakes in the Chemistry department as effective synonyms
for jb%phys@host or chem!jd@host or worse.  And then there are
"subaddresses" based either on positional arrangements similar
to PHSYSJB or on delimiters which, like "%" or "!" have no
special meaning in the transport of message header
specifications.  

We have survived all of this so far by strict application of the
"no one messes with the local part other than the delivery
server" rule.  And we have a long history of discovering that
violations of that rule, including especially having
intermediate hosts try to "improve" on !a!b!c%d@e by rewriting
it into a!b!c%d%e@a and similar things or to optimize the same
thing by presuming knowledge about the path and rewriting it
into x!c@f get us into huge trouble.  If we internationalize
this, we can't make any guarantees about the delimiters that
will be used for explicit routing, or to delimit subaddresses,
and given the positional approach, even that such characters
will appear at all.  

Can we code this stuff into some sort of ACE, or downgrade to
one, and move it along?  Yes, but we can't do it naively: the
destination system must know what was done to the address and
must also agree to decode the ACE before it has to decide what
to do with the address.  If there is some analogy of a%b as
explicit routing in the local part, the delivery server must be
able to decode the ACE and find the trick characters before,
e.g., it tries to drop the message into the local mail store.
Can that be done?  Of course.  But it implies not only that
everyone is using the same ACE-encoding conventions, but that
all of the delivery servers that can accept mail are doing
decoding and doing that decoding early enough: there is, for
example, no general way to simply drop the ACE into the mail
store and let the MUA do the decoding -- far too late to do most
sorts of mail routing by that time.   Moreover, we have never
tried to standardize the order in which delivery MTAs do their
decoding and delivery steps.  We could start now as part of an
internationalization activity but, as a matter of personal taste
and a preference for not constraining diversity of conforming
implementations when it can be avoided, I'd rather not.   That,
however, is very much a matter of taste -- "impossible"
ultimately means "not without applying a number of other
constraints, or more significant rewriting of the mail system,
including new ports and/or maybe flag days, that we might not
like or be able to accept".  And that brings us to the next
issue...

(3) Why target "experimental", what does that mean, and why not
dump this on the IRTF?     I suggest that there are
research-experiments and engineering-experiments, and that they
are different.  

If the question here was "can we figure out some way to do
internationalized email and demonstrate that", it would, IMO,
clearly be a research issue.  If the question were "what is the
absolutely optimal way to do internationalized email" then that
is a research question too (and one about whose utility in
engineering I think we agree about).  What we have here,
however, is a strong hypothesis about a way to do the job and
what constraints are, and are not, appropriate to go with it.
There have been experimental implementations of some of the
ideas: they show us that, at least for some email
implementations, that they can be implemented and give us an
idea about complexity of implementation.  But we haven't
examined all of the cases we can identify and probably haven't
identified all of the cases.  This is, IMO, precisely the point
at which an IETF WG is useful: the small group discussions about
framework, starting point, and feasibility are done.  There
isn't complete agreement within the design group about how far
some of the strategies can be pushed and we need IETF input
about that and about the case analyses.  There are a number of
tricky issues about protocols, interfaces, and interactions with
the user, and broader IETF input is needed there too.  

But these are engineering problems, not research ones, unless
one pushes anything that involves design decisions and some need
to tinker over into "research". 

The reason to go for experimental within the IETF is a bit
different.  There may be other ways to get address
internationalization.  My belief is that there are not perfect
solutions, so each set will probably come with a different set
of constraints and things that it makes hard to do.  The quest
for the best solution before we start working out the details of
any of them is pointless (and we have already wasted a lot of
time trying to do that).  So the goal is produce a complete set
of specifications for one approach, see how it works, and only
then examine the questions of what other approaches are
feasible, out there, reasonable candidates for alternatives.  If
there are, then there will be a question of what to standardize.
But, in the meantime, it is an explicit plan for this WG to deal
with someone coming along and saying "I have this completely
different idea even though it isn't written up and may or may
not be as good as yours, please stop and consider it" with "go
away, make your own WG to consider it and see if it gets any
traction". 

Understanding that the analogy is not exact, this situation is
both similar to and different from DKIM: on the one hand, a good
deal of design work and testing have been done within a design
team, many of whose members are active in the IETF.  We are
ready to open things up and get IETF input and make changes as
needed within the general model.   But we want to preserve that
model --if only to avoid thrashing-- until we are certain that
it will or will not usefully work.  And experimental, with some
constraints on the range of discussions and outcomes, seems to
us (and the ADs and others who have been consulted) to be the
right way to approach that.

        john

---

From: Eric Brunner-Williams <[email-masked]>
Date: Fri, 24 Feb 2006 15:08:20 -0500

John,

When I looked at the national standards in Asia, I found most had the
property you point out as a desirable property of utf-8, that is, the
code points defined in ASCII are defined as in ASCII.

Eric

---

From: Dave Crocker <[email-masked]>
Date: Fri, 24 Feb 2006 12:33:44 -0800

John, et al,

> Issue 1: Why UTF-8?   
...
> So it is not really just one more coding.  

I had neglected to mention that UTF-8 has the nice property of being upward 
compatible with ASCII.  And that point certainly adds to its appeal.

But I think this is actually secondary to the main point I (finally) cited : 
UTF-8 has become the universal, host-based equivalent, today, of what ASCII was 
30 years ago.

And since the needs of today require using its enhanced capabilities, using it 
directly makes more sense than emulating it or layering it over ASCII.

> transport in the general case.  

Fun history lesson.  And, yes, it underscores the difficulty in doing a 
local-part overlay encoding.

(By the way, as I recall, the research I did in 1978, that finally chose the %, 
produced a list of perhaps 3 letters that were viable candidates.  But, then, I 
had to team with the terminal interface, and we are at least spared that 
indignity, today.  Still, I think there is no question that finding free 
characters to use as syntactic markers is challenging at best.)

> to do with the address.  

I think the issues are more interesting than that:

1. If downgrading can be done well enough to preserve all of the information, 
then the 'native utf-8' mode is merely an efficiency hack, modulo the 
considerable burden of having to translate between the two forms.  That burden 
is sufficient to force one to ask why it is necessary to have parallel encodings.

2. If downgrading cannot be done that well, then what is being proposed is a new 
Internet mail infrastructure, with gateway translation between them.  We have 
some experience doing email gatewaying.  We know how to make it work. Or rather, 
we know how challenging it is to make it at all useful.  Calling such an effort 
"research", when one of the two sides does not already have extensive history, 
is not all that creative.

> what constraints are, and are not, appropriate to go with it.

All that is reasonable.  What I think is that there is nowhere near enough 
operational experience to know that this is constrained enough to call 
"engineering".

Although the approach re-uses much of the existing Internet mail technology, it 
really is re-inventing the service.  That is, I think that the infrastructure's 
reliance on net-ASCII is so deeply ingrained as to make switching to UTF-8 a 
near-term research effort.

As I noted before, I think that much of the dns and email internationalization 
efforts have suffered from trying to be IETF working groups, because the 
community knowledge and focus were not (yes) sufficient for that model of effort.

The same concern applies here.

> framework, starting point, and feasibility are done. 

I apologize for missing this, but which of the existing references documents the 
results of this extensive research work?

d/

-- 

Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Feb 2006 16:39:33 -0500

--On Friday, 24 February, 2006 15:08 -0500 Eric Brunner-Williams
<[email-masked]> wrote:

> John,
> 
> When I looked at the national standards in Asia, I found most
> had the property you point out as a desirable property of
> utf-8, that is, the code points defined in ASCII are defined
> as in ASCII.

Yes.  If I appeared to say anything different, I didn't intend
to.  The intent about my observation about traditional practices
in East Asia is that, while the European internationalization
trend was to go first to 8-bit character sets (of the ISO 8859
family and national and other variants on it), East Asian
practice branched into two other directions: 16-bit character
sets and, especially in Japan, the use of stateful, ISO
2022-based, code table switching to  access a much larger set of
characters than are needed for European languages.   While UTF-8
is clearly a step forward from the ASCII->8859 trend, debate
continues about whether it represents a step forward from the
earlier East Asian character sets or a step in some other
direction.   Part, but only part, of the reasons for that debate
is due to the fact that UTF-8 tends to take up a lot of octets
per character once one gets to the East Asian characters.
Another part, but still only part, is dissatisfaction with "Han
unification" in some parts of the CJK community.  

For the record, I'm taking no position at all about those
debates,  just identifying them because it is unwise to pretend
that they are not part of the picture.

best,
     john

---

From: Dave Crocker <[email-masked]>
Date: Fri, 24 Feb 2006 14:01:37 -0800

John,

I meant to ask about a point you didn't comment on:

I offered some candidate text for the first 2 paragraphs.

Whether the text applies to an ietf group charter or an irtf strikes me as 
secondary.  If the charter text works well, it means there is coherence to the 
project.  Where it is housed is important, but separate.

Any comments on the alternative text?

Any comments from anyone else?

d/

ps.  To spare folks from searching for the previous message:

/////

/////
-- 

Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Sun, 26 Feb 2006 15:21:02 -0500

--On Friday, February 24, 2006 2:01 PM -0800 Dave Crocker 
<[email-masked]> wrote:

> Whether the text applies to an ietf group charter or an irtf
> strikes me as secondary.  If the charter text works well, it
> means there is coherence to the project.  Where it is housed
> is important, but separate.
>

I had hoped that "someone else" would say something, perhaps 
even something that would be more constructive from the 
standpoint of overall principles.  But, since they haven't and 
your suggestion has not received any response...

To me, a WG Charter serves three main purposes.  They are, not 
in order of importance:

    (1) To provide a mechanism by which the community can figure
    out how it wants to invest its resources, whether the
    proposed subject matter is appropriate for the IETF, etc.

    (2) To provide sufficient definition of what is to be done,
    and to structure the work sufficiently, that the community
    can be assured the WG won't go wandering off into the
    proverbial weeds.

    (3) To assure the community that there are sufficient
    resources, both in terms of ideas and people, to have
    reasonable odds of being able to complete the work in a
    timely fashion.

Each of these three exists in context.  For example, the 
community may, justifiably, be much more skeptical of a charter 
proposal to do something completely new to the IETF and coming 
from people with no prior IETF experience than of one that is at 
some opposite end of the spectrum.  And, to me, a charter should 
be as good as necessary to do the job above (and any other jobs 
the community may require-- any better is just a waste of time 
that could be better spent on the technical/engineering work.

I actually get very anxious when we start talking about, e.g., a 
charter as a contract between the WG and the community (although 
I have certainly, and nervously, used that language in the 
past).  If the IETF is operating the way I would prefer that it 
operate, charters can be reasonably informal as long as we have 
ADs who are willing to monitor progress, make decisions, and 
shut down WGs that are either not producing or not pronouncing 
anything the community wants to see.

Now, it seems to me that your proposed alternate text addresses 
the second of these issues.  Personally, although it needs a bit 
of editorial work, I like it at least as much as I like the text 
that is now in the proposed charter.  But I don't think it is 
significantly better.  So, one of my reactions is "would the WG 
be less likely to wander into the weeds with one set or text or 
the other".  The answer I get to that question is "nope".  That 
is a "nope" partially based on what I know of the context I 
referred to above: I am reasonably familiar with the proposed 
leadership of this group, I know that Harald and I have a lot of 
experience with the sorts of weeds that grow around the IETF and 
have both rather sever allergies to them and (at least in my 
case) exaggerated ideas about the value of my time. 
Consequently, regardless of what the charter says, I am 
confident that, should the WG head off toward the weeds and we 
be unable to stop it, we would rather quickly be having "shut 
this down, it is hopeless" conversations with the relevant AD. 
The result is that I can't get excited about fine-tuning charter 
language for that purpose.

That brings me to my other reaction (and why I haven't responded 
to this).   That reaction was one which we have both decried 
when it comes out of WGs or document editors late in the 
standards process in response to an end-stage proposed change by 
the IESG or RFC Editor.  That reaction is, more or less, "well 
this doesn't make things significantly worse, so, if it makes 
you happy, well, whatever".  If your suggested text were somehow 
bad news for the effort, I would have commented.  If I saw it as 
somehow far superior to what had been written earlier, I would 
have commented.  But, as it is, my reaction was "if this amuses 
either the community, or the ADs, or makes someone feel 
significantly more comfortable, then... why not".  And that, it 
didn't seem to me called for much comment until and unless we 
found out who else it would make much more comfortable (or 
uncomfortable).

Two substantive comments about your suggested text...

> Because the specifications attempt to make a significant
> change to the Internet's mail infrastructure,...

I suspect we disagree about how significant this change is to 
the infrastructure.  That may be partially a matter of 
definitions, but I see these changes as much less significant 
than the ones that created the SMTP extension mechanism and MIME 
and less risky than the moves we make when a new MIME multipart/ 
substructure is introduced.  I also see it as a great deal less 
infrastructure-problematic than the handling of some 
specially-formatted stream transmissions types as text/ 
subtypes, for some of the same reasons.  I note that none of 
them were sent to Experimental first (although that really is 
not the reason for Experimental here, as explained elsewhere). 
And I agree that this requires extreme caution and a good deal 
of analysis and examination along the way, regardless of what 
things are called, so any disagreement may not be substantively 
important.

> [[ Hmmmm... This also creates a requirement for specifying
> UTF-native RFC2822 header fields, doesn't it?  /d ]

I hope not.  Part of the reason Internet mail has succeeded so 
far is that we have traditionally treated those header field 
names as protocol elements that can be _translated_ into other 
languages, but not expressed in those languages in transit.  If 
we cross over into producing UTF-8 header field names, I think 
it would be only a short period of time before messages start 
flying around the network with the names of header fields that 
are specified in 2822 written in other languages or scripts. 
And I think that would lead quickly to an interoperability 
nightmare.

But, fortunately, that isn't proposed for a work item for this 
WG and I hope we can keep it that way.

best,
   john

---

From: Dave Crocker <[email-masked]>
Date: Sun, 26 Feb 2006 13:46:32 -0800

John,

> I actually get very anxious when we start talking about, e.g., a charter 
> as a contract between the WG and the community (although I have 
> certainly, and nervously, used that language in the past).  If the IETF 

That precise language, for describing a charter, has been integral to the IETF 
for at least 12 years.

Take a look at Section 2.2 of WG Guidelines and Procedures.

When we wrote that language into the original WG G&P RFC, it used such strong 
language specifically in order to make clear both the constraints and the 
obligations that chartering a working group imposes.

> is operating the way I would prefer that it operate, charters can be 
> reasonably informal as long as we have ADs who are willing to monitor 
> progress, make decisions, and shut down WGs that are either not 
> producing or not pronouncing anything the community wants to see.

Sorry.  I thought we had community consensus that there were problems with 
excessive ADs "flexibility" as well as their being excessively overworked. 
Carefully crafted charters can ameliorate both of these problems.

>  Consequently, 
> regardless of what the charter says, I am confident that, should the WG 
> head off toward the weeds and we be unable to stop it, we would rather 
> quickly be having "shut this down, it is hopeless" conversations with 
> the relevant AD. The result is that I can't get excited about 
> fine-tuning charter language for that purpose.

1. It is rather more than fine-tuning, especially for the large portion of the 
community who does not already have extensive background in this topic

2. Taken to its logical conclusion, your line of comments means that whenever 
the IESG decides it has plenty of trust in the working group chairs, there is no 
need for a charter.  (I suspect that is not what you intended to mean.)

As nearly as I can tell, what you are ignoring is just how difficult and 
problematic this space has already proven to be.

This ought to make folks seek as much structural help as possible.  I can't 
fathom a viable operational model that takes the charter text so lightly.

> I suspect we disagree about how significant this change is to the 
> infrastructure.  That may be partially a matter of definitions, but I 
> see these changes as much less significant than the ones that created 
> the SMTP extension mechanism and MIME and less risky than the moves we 

MIME did not change the infrastructure, John.

And the SMTP Extension mechanism was designed extremely carefully, specifically 
to attend to its potential impact on infrastructure change.  I do not see 
anything like that yet showing up about IMA.

> make when a new MIME multipart/ substructure is introduced.

When IMA breaks, mail will not get delivered.

When a multipart substructure is unknown to a receiving MUA, a portion of the 
message cannot (currently) be read. Frequently, the user has an ability to use 
an out-of band mechanism to interpret that substructure.

So, I think you have the relative negative impacts of the two reversed.

   I also see
> it as a great deal less infrastructure-problematic than the handling of 
> some specially-formatted stream transmissions types as text/ subtypes, 
> for some of the same reasons.  I note that none of them were sent to 

How does a mime type affect deliverability?

> I hope not. 

The current draft text says "and the use of UTF-8 in mail headers".  How can 
this be achieved in any way other than specifying native RFC2822 header fields?

Or, rather, if that is not what is meant by the charter text, then it is another 
example of needing to make the charter text more specific and clear.

> But, fortunately, that isn't proposed for a work item for this WG and I 
> hope we can keep it that way.

If the work is not to be done, it would probably be advisable not to have it 
cited in the charter as something to be done.

d/
-- 

Dave Crocker

---

From: Yangwoo Ko <[email-masked]>
Date: Mon, 27 Feb 2006 17:17:25 +0900

Dave Crocker wrote:

> And the SMTP Extension mechanism was designed extremely carefully, 
> specifically to attend to its potential impact on infrastructure 
> change.  I do not see anything like that yet showing up about IMA.

I totally agree with Dave in the point that current IMA proposal is to 
change Internet's mail infrastructure. Any new protocol requires changes 
in some part of the world. Sometimes it is done at infrastructure, some 
other times at  somewhere else, and some yet another times at both. Any 
one who wants to use or to support IMA will need to update their parts 
(e.g. MUA/MTA/mail store/mail address registration interface/and many 
more).

Dicussion between Dave and John, quoted above, can be understood as how 
significant it will be? I'd rather interpret the "significance" in 
somewhat different aspect.

As we assume, it seems to be relatively easy to do IMA only between 
agreed parties. The real devil lies at the boundary. However good news 
is that, as a design team of this extention, we have a knob that 
controls how significantly we want to change the infrastructure. For 
example, in one extreme, we can just drop any mail that tries to cross 
the boundary. On the other extreme, every part of the Internet 
infrastructure is required to be upgraded somehow to support IMA so 
perfectly that users cannot even recognize that there is such boundary. 
In order to make IMA more usable, we want to deviate from the former 
extreme as much as possible. In order to make IMA be more practical, we 
cannot but deviate from the other extreme.

Even though the current set of proposed documents is not yet clearly 
fixed the position of the knob, we have shown rough range within which 
we can move the knob back and forth. However, I would say that EAI(? 
IMA?) WG should be given a chance to gather wise suggestions on the 
proper position of the knob not only before its charter being fixed but 
also thorough its process toward the closeout. (If now, what is WG for?) 
Or, some wise ideas can come up only after having experimental IMA. In 
that sense, Dave's input will be helpful in finding the final position 
of the knob regardless of its being reflected in the WG charter or not.

Best Regards

---

From: Martin Duerst <[email-masked]>
Date: Mon, 27 Feb 2006 13:03:52 +0900

In the text proposed by Dave, as well as in other mails,
I have seen the abbreviation "UTF" used. "UTF" used alone
doesn't mean anything. Please don't use it alone. Similar
for terms like "UTF-native" or "UTF-based" or "raw UTF".

Please either be specific ("UTF-8", "UTF-8-based",...) or,
if you want to be general, use the acronym UCS (Universal
Character Set or Unicode Character Set). And as for
"raw UTF", or "raw Unicode" or whatever, there just isn't
any such thing except in our heads; if the data is in
electronic form, there always is an encoding, be it UTF-8
or UTF-16BE or UTF-16LE or UTF-32BE,...

Regards,   Martin.

At 07:01 06/02/25, Dave Crocker wrote:

 >Given the scale and variety of the modern Internet, the email user 
experience needs to be enhanced, to support addresses and mail header 
fields comprising characters native to that wide variety of user 
environments.  Basic network ASCII does not permit this, but UTF does.  UTF 
is now widely supported in user systems. Therefore, it is Internet mail's 
reliance upon classic network ASCII that has become the bottleneck for the 
user experience, regarding email handling data and other meta-information. 
The current effort will develop specifications for an SMTP option to permit 
the direct use of UTF-8 encoded data in RFC2821 envelope address 
local-parts, domain-parts and RFC2822 mail header fields. It will also 
explore the ability to downgrade, when attempting to interact with legacy 
email services.  [[ Hmmmm... This also creates a requirement for specifying 
UTF-native RFC2822 header fields, doesn't it?  /d ]

---

From: Martin Duerst <[email-masked]>
Date: Mon, 27 Feb 2006 14:07:20 +0900

At 05:33 06/02/25, Dave Crocker wrote:

 >Although the approach re-uses much of the existing Internet mail 
technology, it really is re-inventing the service.  That is, I think that 
the infrastructure's reliance on net-ASCII is so deeply ingrained as to 
make switching to UTF-8 a near-term research effort.

It may be the first time somebody is saying that using all 8 bits
that TCP/IP provides needs a research effort.

For the record, I agree with John and others that this should
be done in the IETF, not the IRTF. I'd even be happy with standards
track rather than experimental, but I can live with experimental.

 >As I noted before, I think that much of the dns and email 
internationalization efforts have suffered from trying to be IETF working 
groups, because the community knowledge and focus were not (yes) sufficient 
for that model of effort.

The DNS internationalization effort suffered mostly from too much
'vendor' hype and disagreement and from constant interference of
a small number of people with personal pet projects/concerns.
Nothing unusual for a suffering IETF WG.

The other problem was that the charter was much less specific than
this one; even the basic approach (ACE or new record type or
DNS extension or UTF-8 or what else) wasn't specified.

The main really internationalization-relevant point in the DNS
internationalization effort was how to define what characters to
allow or not, and this is both much less of an issue for email
address LHS or free-text header field content than for domain
names, and we already have a reasonably usable and adaptable
framework (stringprep).

The rest of the work is mostly about downgrading/upgrading/
gatewaying and how to signal capabilities and help the user,
which may not be trivial at all (while not being research),
but isn't really internationalization-specific.

Regards,    Martin.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 28 Feb 2006 10:14:15 -0800

> somewhat different aspect.

I think you have describe the question correctly.

I think that a change in the form of email addresses that can cause delivery to 
fail should be considered significant.

It can/will partition the functional infrastructure.

> agreed parties. The real devil lies at the boundary.

Yes.  Exactly correct.

> However good news 

How does a design team control the operational behavior of boundary MTAs?  The 
experience with gatewaying between independent mail services is that it is a 
challenge, at best.

Further, the experience with the use of "private" IP Addresses within networks 
is they leak out into the public Internet.

In the last 25 years, there have been somewhere between 0 and few efforts to 
affect basic deliverability.  (DSN is the possible exception and it's adoption 
history provides quite an education.)

d/

-- 

Dave Crocker