Subject: [EAI] Updated minutes from Vancouver meeting

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 03 Jan 2008 21:10:54 +0100

I have done some slight updates to the minutes from the Vancouver meeting - 
a few spelling errors, a name spelled wrong, and some indenting to make it 
clearer who spoke when.

These are also uploaded to the IETF meeting materials server.

                Harald
_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Martin Duerst <[email-masked]>
Date: Fri, 04 Jan 2008 17:52:54 +0900

I haven't been at the meeting, but here are some comments on the
normalization discussion:

>#1518: There are two statements in EAI draft. In 4.1 says normalization
>will be discussed; in 5 says will not specify normalization. Prudent use
>of normalization will involve restriction.
>
>John Klensin: I'm afraid of starting down a slippery slope. There are
>  many normalizations with different purposes and effects. We are probably
>  better off saying that local parts of addresses are for the receiving
>  server and no one else should dare touch them. 

This is a very clear and firm principle.

>Harald: Some sections of the draft just talk about text all over, not
>  just addresses. Possibilities are
>  1.   delete the section 4.1 note.
>  2.   remove the note at section 5.
>  3.   move provisions to UTF8 header section.
>
>Klensin: We should just drop these provisions.
>Chris Newman: I'm concerned about interoperability. Different
>  normalizations at different recipient domains would be a problem. We
>  need to provide enough pointers that this is unlikely.

Different normalizations at different recipient domains are
already in use: some recipients use case folding, some others
don't. The problem with Unicode normalization is that it's
much more difficult to figure out because there are no
visual differences.

>Klensin: The problem is that pointing at Unicode without being specific
>  can lead to choices of normalization that lose data, are inconsistent,
>  etc. I've been pushing a document going a bit further by defining a Net
>  UTF8 stream more generally for the IETF. But it fails due to
>  controversy.
>Chris: Should we insert an informative reference to Klensin's Net UTF8?
>Klensin: Sure.
>Harald: At least two people want this.
>Chris: We should clarify that this is to be used by recipient domains in
>  interpreting their local part.

The Klensin draft is about what you send over the wire, not what
recipients do. We know we shouldn't change anything in transit,
so the responsible party is the (origin) MUA. Basically, what we
should say is:
- The (origin) MUA normalizes (see draft-klensin-net-utf8, section X)
- The final MTA makes sure it understands such normalized addresses
  (either by just using the normalized form it its "address book",
   or by doing appropriate normalization when looking up a user).

>Harald: Then it sounds like it concerns headers, not general text.
>Klensin: It's more general but the provisions needs a bit of tweaking
>  for which I volunteer to help.

I'd definitely want to see the tweaked text.

>Poll on referencing Klensin draft: 20 in favor, 0 against.
>Klensin: My document has a discussion of issues on normalized versus
>  non-normalized text.

BTW, in Section 3, there are some wording problems:

>   The section above requires that, except in very usual circumstances,
>   all Net-Unicode strings be transmitted in normalized form.

usual -> unusual?

Regards,    Martin.

>*** #1518 resolution: Refer to draft-klensin-net-utf8.
>Placement with the document at the editor's discretion. If removed from
>Section 5, section 5 can be removed.
>
>Pete Coats: Local addresses are usually case mapped. What about things
>  like Turkish?
>Klensin: This sort of question is why no one but the target server
>  manipulates this. Your example is an excellent example of why this
>  should not be discussed in the EAI documents.
>
>Harald: We will confirm these resolutions on the mailing list, update
>  the drafts, and send them to the IESG.

At 05:10 08/01/04, Harald Tveit Alvestrand wrote:
>I have done some slight updates to the minutes from the Vancouver meeting - a few spelling errors, a name spelled wrong, and some indenting to make it clearer who spoke when.
>

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Jan 2008 11:29:22 -0500

--On Friday, 04 January, 2008 17:52 +0900 Martin Duerst
<[email-masked]> wrote:

>...
>> Harald: Some sections of the draft just talk about text all
>> over, not just addresses. Possibilities are
>>  normalizations at different recipient domains would be a
>>  problem. We need to provide enough pointers that this is
>>  unlikely.
> 

Exactly.  And, of course, some normalizations, such as NFKC, do
change things for which there are visual differences and, while
it is rare and UTC certainly wouldn't approve, some rendering
engines compensate for perceived Unicode inadequacies by
rendering a few precomposed characters differently from their
character+ combining character canonical equivalents.

>> Klensin: The problem is that pointing at Unicode without
>> being specific can lead to choices of normalization that lose
>>  data, are inconsistent, etc. I've been pushing a document
>>  going a bit further by defining a Net UTF8 stream more
>>  generally for the IETF. But it fails due to controversy.
>> Chris: Should we insert an informative reference to Klensin's
>> Net UTF8? Klensin: Sure.
>> Chris: We should clarify that this is to be used by recipient
>> domains in interpreting their local part.
> 
> The Klensin draft is about what you send over the wire, not
> what recipients do. We know we shouldn't change anything in
> transit, so the responsible party is the (origin) MUA.
> Basically, what we should say is:
> - The final MTA makes sure it understands such
> normalized addresses   (either by just using the normalized
> form it its "address book",    or by doing appropriate
> normalization when looking up a user).

The second statement effectively requires that the receiving MTA
normalize and permits the sender to expect that it will do so.
While it is unlikely that this would cause any problems with
MUAs or MTAs that carefully observe the robustness principle, we
have ample experience in the email world of people who interpret
that principle as a license rather than a caution.  I think that
puts us on precisely the slippery slope I'm trying to avoid.
Let me therefore make an alternate suggestion that takes us a
small step in the direction of the document with UI and
operational suggestions we've discussed but that may help with
this.

# 	Under most circumstances, the delivery MTA SHOULD be
# 	configured so that local parts of addresses will be
# 	accepted in NFC-normalized form and probably in
# 	NFKC-normalized form.  As with traditional ASCII
# 	addresses and case-sensitivity [XXX], it MAY, of course,
# 	permit alternate forms as aliases and, where the use of
# 	such forms appears likely based on experience, usually
# 	should do so.

Note that this is completely consistent with the suggestion and
style of the first full paragraph after the syntax rules of RFC
2821 Section 4.1.2  ("While the above definition for
Local-part...").  The section number and text are identical in
2821bis, at least so far.  My suggestion would be to incorporate
the above, and reference both 2821 Section 4.1.2 and its 2821bis
equivalent in "[XXX]" and let the RFC Editor sort things out.

Then one might add if one wanted to (and I think there was some
strong feeling for it during the Vancouver discussions)
something like:

# 	Just as message originators should be extremely careful
# 	about oddly-quoted forms in unextended SMTP, the
# 	possibility of Unicode normalization being applied
# 	during in-transit processing independent of the MTA
# 	actions specified here (see [net-utf8]) suggests that
# 	message originators should exert extra caution about
# 	sending unnormalized addresses.

Note lower-case "should".  Some formulation that used a
different word entirely might be better.

This sticks fairly closely to the 2821 formulation and provides
no foundation at all for senders to expect that the delivery MTA
will normalize whatever garbage they send (e.g., it is written
in terms of accepting the normalized form, not in terms of
normalizing whatever comes in -- Martin's text also accomplishes
that, but I think this is more clear).    I note that, while it
has often not been an especially good idea, there is an SMTP
history of using very strange local addresses to reduce the odds
of undesired correspondence (spam included, but by no means
restricted to that).  If someone wants to parallel that with
UTF8SMTP by permitting only an unnormalized form, we can (and
probably should) warn against it as a general practice but
should not, IMO, try to prohibit it.

See also the "footnote" below.

>> Harald: Then it sounds like it concerns headers, not general
>> text. Klensin: It's more general but the provisions needs a
>> bit of tweaking for which I volunteer to help.
> 

Contrary to the discussion in Vancouver, the proposed text above
would be applicable to, and should be applied in, SMTPEXT.   A
reference to it from the headers document would not be a bad
idea.

>> Klensin: My document has a discussion of issues on normalized
>> versus non-normalized text.
> 
>>   The section above requires that, except in very usual
>>   circumstances, all Net-Unicode strings be transmitted in
>>   normalized form.
> 
> usual -> unusual?

Yep.  Fixed in -07.  Thanks.

     john

"footnote". Most of the "should control characters be permitted
in address local parts" discussion that has come up repeatedly
here is bogus.  I apologize to everyone for not catching it
sooner.  There is an explicit prohibition --in text, not in the
ABNF-- in 2821 on the use of any control character in mailbox
names.  For details, watch for a note on the ietf-smtp list (I
will find the discussion impossible to follow if it is going on
in both places).  However, the fact that there is such a
prohibition in 2821 opens us to the possibility of needing a
similar prohibition in SMTPEXT, extending the possible
characters in UTF8SMTP addresses to Unicode-less-controls,
rather than to Unicode generally.  

If we want to make that prohibition, it should probably be based
on Unicode General Category values (see Unicode 5.0, table 4-7
(page 139) and elsewhere) rather than on a range of codepoints.
Candidates for a prohibition include characters with values Cc
(which gets the C0 and C1 controls), Cf, Cs, and Cn, with  Zl
and Zp being other likely candidates.  

Zs is problematic and should probably not be prohibited unless
one wants to write a special rule with a list of characters or
exceptions. It includes the traditional SPACE (U+0020) which
cannot be prohibited without being more restrictive than 2821
but also includes a long series of space and space-like things,
some of which are bad news (but others of which are not).

---

From: Chris Newman <[email-masked]>
Date: Fri, 04 Jan 2008 17:28:51 -0800

When it comes to rules for human interactions, I've always felt a 3-level model 
is helpful: things that are illegal, things that are legal and acceptable 
behavior, and things that are legal and bad etiquette or bad practice.  The 
cost of putting too much in the "illegal" category is reduced freedom.  The 
third category works better for regional variations and allows faster evolution 
of rules (hopefully for the better) over time.

Unicode crosses squarely into the human interaction area and needs the 3-level 
model.  There are regional variations on Unicode normalization, our 
normalization knowledge needs to evolve with experience faster than RFCs are 
revised and there are numerous etiquette issues involved with global mail. 
There are things that are non-compliant in EAI addresses, things that are 
compliant and acceptable and things that need to be compliant but are 
often/always bad practice or bad etiquette.  Our standards need to allow things 
in the third category and recognize they may not interoperate in all cases. 
This is uncomfortable territory for those who value precise specifications and 
interoperability as highly as the IETF traditionally has.

                - Chris

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Jan 2008 20:41:57 -0500

Chris,

I'm not positive because we are using very different
vocabularies and perhaps thinking about this differently, but I
think we are in violent agreement.  And I again note that we
aren't proposing anything really new here: 2821 (and its
predecessors, but 2821 is more explicit) effectively says
"case-sensitive addresses are valid" and "if you use one of
those puppies, bad things will happen and it may not
interoperate".   I think that is exactly your "compliant but
often/always bad practice" case.

So, to make this specific and be sure I understand you, we would
have, e.g.,

* Unnormalized strings are valid, but sufficiently bad practice
that they may not work reliably on a global basis.

* C0 (and presumably C1) controls are prohibited, the first in
2821 and the second by an obvious extension from it.

* Other kinds of punctuation, spaces, etc., are risky practice.
Perhaps they will work, and we should expect SMTP receiver code
to handle them, but creating dependencies on them in mailbox
names that are chosen is usually a bad practice and may lead to
interoperability problems.

Is that reasonably close?
    john

--On Friday, 04 January, 2008 17:28 -0800 Chris Newman
<[email-masked]> wrote:

> When it comes to rules for human interactions, I've always
> felt a 3-level model is helpful: things that are illegal,
> things that are legal and acceptable behavior, and things that
> are legal and bad etiquette or bad practice.  The cost of
> putting too much in the "illegal" category is reduced freedom.
> The third category works better for regional variations and
> allows faster evolution of rules (hopefully for the better)
> over time.
> 
> Unicode crosses squarely into the human interaction area and
> needs the 3-level model.  There are regional variations on
> Unicode normalization, our normalization knowledge needs to
> evolve with experience faster than RFCs are revised and there
> are numerous etiquette issues involved with global mail. There
> are things that are non-compliant in EAI addresses, things
> that are compliant and acceptable and things that need to be
> compliant but are often/always bad practice or bad etiquette.
> Our standards need to allow things in the third category and
> recognize they may not interoperate in all cases. This is
> uncomfortable territory for those who value precise
> specifications and interoperability as highly as the IETF
> traditionally has.
> 
>                 - Chris
>

---

From: Harald Alvestrand <[email-masked]>
Date: Sat, 05 Jan 2008 15:27:58 +0100

> Is that reasonably close?
That corresponds to what I think is reasonable.

One thing that gets people (including me) worried here is that we have
some choices that are analogous to case folding but less widely
understood - NFD/NFC/NFKC being the most important one - where, in the
absence of a recommendation, reasonable people can pick different
choices (as MacOS and Windows did).

Adding a recommendation of "if you don't have a good reason to do
otherwise, do this - it works for most cases" would increase the chances
of interoperability.

But it doesn't fit very well with the "let's write down the rules only"
approach we've so far taken in the EAI specs.

                 Harald

---

From: Martin Duerst <[email-masked]>
Date: Sat, 05 Jan 2008 16:15:39 +0900

>Is that reasonably close?

That looks very good to me.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Sun, 06 Jan 2008 17:11:12 +0900

Hello John,

>normalize and permits the sender to expect that it will do so.

I don't understand this. If I have a receiving MTA, and I set
it up so that all usernames are in NFC, then I'll just rely
on the fact that most senders will input addresses in NFC,
and those who won't will die out.

>#      such forms appears likely based on experience, usually

I'd fully stand behind this, exactly what I had in mind.

>different word entirely might be better.

I'm not sure about this. There is some possibility that
normalization is applied in transit, but in that case,
sending unnormalized addresses wouldn't be a problem,
they'd get normalized. What would be a problem is
having a receipient MTA that only accepts unnormalized,
and the sender knowing that and sending unnormalized,
but they not being able to communicate. So I'd reword
the above to say that because transit MTAs are forbidden
to normalize, and because receiving MTAs most probably
have been set up to receive normalized, there should
be extra caution against sending unnormalized addresses.

>that, but I think this is more clear).

I'm very sure you're better than me to get the terms and
wording right in this area.

>I note that, while it

Well, yes. But there is a tradeoff between making addresses
strange to make sure you don't get reached by strangers and
spammers, and not making your address too strange to make
sure you still get reached by those you want to reach you.
And I think with unnormalized forms, the tradeoffs would
be particularly bad. But we don't have to mention that in
the spec.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Sun, 06 Jan 2008 05:38:50 -0500

--On Saturday, 05 January, 2008 15:27 +0100 Harald Alvestrand
<[email-masked]> wrote:

>> vocabularies and perhaps thinking about this differently, but
>> I think we are in violent agreement.  And I again note that we
>> So, to make this specific and be sure I understand you, we
>> would have, e.g.,
>> 
>> * Unnormalized strings are valid, but sufficiently bad
>> practice that they may not work reliably on a global basis.
>> 
>> * Other kinds of punctuation, spaces, etc., are risky
>> practice. Perhaps they will work, and we should expect SMTP
>> receiver code to handle them, but creating dependencies on
>> them in mailbox names that are chosen is usually a bad
>> practice and may lead to interoperability problems.
>> 
> One thing that gets people (including me) worried here is that
> we have some choices that are analogous to case folding but
> less widely understood - NFD/NFC/NFKC being the most important
> one - where, in the absence of a recommendation, reasonable
> people can pick different choices (as MacOS and Windows did).
> 
> Adding a recommendation of "if you don't have a good reason to
> do otherwise, do this - it works for most cases" would
> increase the chances of interoperability.
> 
> But it doesn't fit very well with the "let's write down the
> rules only" approach we've so far taken in the EAI specs.

Well, 

(i) We have considerable flexibility in Experimental documents
to describe the theory on which decisions are based.  If that
means a fast revision of "framework"*, or that we need to move
the priority of "advice on configuration" up, so be it.  However
I think the community would be better served right now by
patching the information in somehow and then making a document
reorganization the first step in a move onto standard track.

(ii) "if you don't have a good reason to do otherwise, do this -
it works for most cases" is precisely what a SHOULD statement
should (sic) be all about.

     john

* The minutes reflect the fact that I offered to work on a
revision of "framework" to incorporate some scattered material,
notably the appendix in SMTPEXT.    While the text for the I-D
is nearly ready for posting, pulling the text out of SMTPEXT
would have created a mess or normative references to a proposed
revision of an Informational document.  That seemed a little too
convoluted and a little too error-prone if we tried to rush it
through, so the framework-bis project has gone on hold.  We can
change that, but it would need to be important enough to hold
everything else up (by some months unless we can accelerate our
current pace).  Or we can copy everything to 4952bis and then
remove all of the text from SMTPEXT that would not imply a
normative reference.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 07 Jan 2008 11:29:36 -0000

> This is a very clear and firm principle.

Not quite. There are two parties who might fiddle with a local-part
1. The originator of the message, who may choose from various Unicode
normalizations that appear the same to the naked eye.
2. The final recipient MTA who gets to inspect the local-part in detail
and thence figures out who/what to deliver it to.

I agree that nobody between those two has any business changing anything
as it passes through (but it will probably happen nevertheless :-( ).

>    or by doing appropriate normalization when looking up a user).

Exactly. Or, to be more precise, people who advertise their email
addresses in Reply-To headers or in mailto: URLs or in any other way
should do so in the most normal manner that their incoming MTA is able to
understand. That way, MUAs originating mail to that address will use a
well-normalized form without even realising it.

Any incoming MTA that decides to treat visually identical local-parts
(differing only in their normalizations) in different ways should be taken
out and shot at dawn. Our drafts should make this clear.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 7 Jan 2008 11:56:06 -0500

On Fri, Jan 04, 2008 at 08:41:57PM -0500, John C Klensin wrote:

[3 points]

> Is that reasonably close?

Looks about right to me.

A
-- 
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<[email-masked]>                              M2P 2A8
jabber: [email-masked]                 +1 416 646 3304 x4110

---

From: Ted Hardie <[email-masked]>
Date: Mon, 7 Jan 2008 09:19:58 -0800

>out and shot at dawn. Our drafts should make this clear.

I'd like to point out a couple of flies; you can decide whether they are in
our ointment or someone else's.

1) "Visually identical" local-parts can be constructed out of very different
parts of the unicode code space.  Cyrillic will give you a very nice PAT (u0420,
u0410, u0422) that is visually equivalent to PAT (u0050, u0041, u0054). 
"Differing only in their normalizations" might distinguish between those two
(they come from what we recognize as different contexts), but starting
with "visually identical" as the test seems the wrong way round.  We have
to start from something else, it seems to me, and then go on to what
identical means in the context of that something else. 

2) This especially true where there are native readers of a local-part that
might substitute a character they treat as "the same" even when the
two are neither visually identical nor the same unicode code point.
Many Chinese characters fall into this; they are culturally "the same"
so much so that native readers may not be able to remember which
version was associated with a specific string (was that "guo"  fantizi
or jiantizi?)

The way to deal with the problem at an MTA level may well be to use
the address-book mechanism you describe.  If I want PAT (u0050, u0041, u0054)
not to be equivalent to PAT (u0420, u0410, u0422), then the address book
for the first *does not include the second*, visually identical or no.  Similarly,
if I want "guo" in jiantizi and fantizi (u56EF, u570B) to match, I had better put
that in either a global or local address book.  But visual identification is pretty
much off the table, at least as it seems to me.

		regards,
				Ted

---

From: Chris Newman <[email-masked]>
Date: Mon, 07 Jan 2008 09:49:09 -0800

> Is that reasonably close?

Yes.  That's well stated.

                - Chris