Subject: Re: [EAI] localprep and pop3 with saslprep username authetication]

From: Soobok Lee <[email-masked]>
Date: Sat, 9 Dec 2006 00:36:19 +0900

On Wed, Dec 06, 2006 at 01:16:44PM -0000, Charles Lindsey wrote:
> On Mon, 04 Dec 2006 07:02:05 -0000, Harald Alvestrand
> >
> >Do you mean "request" (offer advice) or "require" (make the standard  
> >specify that it has to be that way)?
> 
> Well there is also an intermediate option, which is "recommend" (aka
> SHOULD).
> 
> ....
> 
> At the minimum you would say that domains allocating local-parts to their
> users SHOULD make them NFC-compatible (or else normalize them on receipt).
> 
> You might also say that MUAs when composing addreseses SHOULD make them
> NFC-compatible, but that is likely to be their default behaviour anyway.

I also prefer SHOULD  to MUST or MAY. If we use "MUST", it would force
syntaxes over localparts.

> That also accords with the Unicode tradition of "early normalization". 

Do you have any tech reports (utc) known about this "early normaliation" ?

> For sure, all intermediate MTAs should pass on exactly what they receive. You
> might argue that if an MUA sees an e+ in a Reply-To, it should reply to
> exactly what it sees.

If MUA implementation shares the same email address postprocessing routines for 
replying and composing, their may be no way to "reply to exactly what it 
sees". I can't imagine MUA architects normalized differently for reply-to sender
address and typed/addrbooked recipient addresses. "All or nothing" seems
simpler to me.

> 
> Anf finally, if you are making such a rule, then you can choose between
> NFC, NFKC, saslprep or some other localprep.

Are you suggesting that our draft enumerate those alternatives and recommend
them to choose one among them ?  Or, should we designate single most preferrable
one in our draft ?

> 
> And finally, if it is a SHOULD rather than a MUST, then the script kiddies
> can have fun by using non-normalized local-parts by mutual agreement, so
> that us adults will have difficulty communicating with them.

Yes. it will allow rooms for other normailzation or no normalization.

Soobok

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 11 Dec 2006 12:08:18 -0000

> syntaxes over localparts.

Agreed. The main thing is that the receiving domain should choose  
normalized localparts, and we could usefully say this somewhere. Where?
> Do you have any tech reports (utc) known about this "early normaliation"  

I will let Martin Dürst asnwer that, since he wrote it. Somewhere on the  
Unicode site IIRC.

> them to choose one among them ?  Or, should we designate single most  

If we are going to write it in somewhere, then that is a matter we would  
need to discuss.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 11 Dec 2006 14:45:39 +0100

Gentlemen,

while this discussion can certainly be entertaining, it is hardly on the 
critical path of our working group.

The only piece of our charter it could possibly be relevant for is this one:
> Additional possible documents suggested:
> Advice for MUA implementors (Info)
>
>

And that is something the WG will only get to if it gets the basic 
specifications nailed.
So please - let's concentrate on what IS on our plate now.

                   Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 12 Dec 2006 10:35:38 -0000

>> Advice for MUA implementors (Info)

I am not so sure of that. We could easily say in utf8headers that  
<localpart>s SHOULD be normalized to some level (TBD, but say NFC for  
now). That would be expressed as a RECOMMENDATION directed at those  
responsbible for allocating localparts at the recipient's domain (which is  
the only place, as we have agreed, where any internal structure of  
localparts can be determined).

And the grounds for making that recommendation are that senders'  
keyboards/MUAs/OSs will have difficulty in generating anything other than  
NFC-normalized codes, and so will not interoperate without some such  
restriction. E.e , it is more than mere advice - UTFF8SMTP will not work  
well without it.

---

From: Soobok Lee <[email-masked]>
Date: Tue, 12 Dec 2006 21:50:09 +0900

> <localpart>s SHOULD be normalized to some level (TBD, but say NFC for  

Personally I do not oppose that. 

But, if we say email address SHOULD be NFCed in [uff8header] draft, 
we should do also in [downgrade][utf8smtp], because
[downgrade] use [utf8smtp]-provided recipient email address parameter
to upgrade/downgrade  To: utf8header. If utf8smtp-provided
email addresses are not NFCed, there may be some occasions that
utf8header_NFCed_To:_email_address != utf8smtp-envelope-unNFCed-email-address.

If this ever causes upgrade/downgrade failure, that may be
obviously regarded as *protocol failure*. Again, "All or nothing".

> NFC-normalized codes, and so will not interoperate without some such  

I am finding some DBMS's primary key restriction, filesystems, 
language compilers/libraries that assume both NFCed input and NFCed output 
by default.

Soobok

> Charles?H.?Lindsey?---------At?Home,?doing?my?own?thing------------------------
> Tel:?+44?161?436?6131?                      
> ???Web:?http://www.cs.man.ac.uk/~chl
> Email:?[email-masked]??????Snail:?5?Clerewood?Ave,?CHEADLE,?SK8?3JU,?U.K.
> PGP:?2C15F1A9??????Fingerprint:?73?6D?C2?51?93?A0?01?E7?65?E8?64?7E?14?A4?AB?A5
>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 12 Dec 2006 16:46:28 +0100

> obviously regarded as *protocol failure*. Again, "All or nothing".
I cannot parse this at all.

The envelope MAIL TO email address is completely independent of the 
header To: address. It is the NORMAL CASE that they are different.

(In fact, for all recipients of this message except Soobok, this message 
will have the two different.)

                     Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 12 Dec 2006 19:12:19 -0000

On Tue, 12 Dec 2006 15:46:28 -0000, Harald Alvestrand  
<[email-masked]> wrote:

> Soobok Lee wrote:
>> But, if we say email address SHOULD be NFCed in [uff8header] draft, we  
>> should do also in [downgrade][utf8smtp], because

OK, I get the general gist of it.

Essentially, I suggested that domain-owners/postmasters/whoever gets to  
choose <local-part>s SHOULD choose them already normalized (e.g. to NFC).  
That should cause them to be normalized also in Froms and Reply-Tos in  
mail originating from that domain, and also in mailtos, Vcards and other  
means of publicising email addresses. And that in turn should lead to them  
appearing correctly in To headers, RCPT TO commands, etc (that, plus the  
tendency of hand typed versions to come out NFC-compliant anyway).

I suggested this could be written into utf8headers. I think you and/or  
Soobok are now saying it should then also be written, or otherwise  
mentioned, in smtpext. I am not so sure it needs to be said in downgrade.

It would also be wise for domain-owners/postmasters/whoever to  
re-normalize incoming <local-part>s before attempting to interpret them,  
and that could also usefully be suggested somewhere.
> header To: address. It is the NORMAL CASE that they are different.

ITYM RCPT TO :-) . But even RCPT TO is usually constructed out of what the  
author typed/copied/pasted into the To header in his MUA, so in practice  
it usually comes to the same thing.

---

From: Soobok Lee <[email-masked]>
Date: Wed, 13 Dec 2006 10:14:56 +0900

> mentioned, in smtpext. I am not so sure it needs to be said in downgrade.

If [utm8smtpext] does NFC, [downgrade] don't need to specify it, as you 
pointed out, since [utf8smtpext] feed NFC(NON-ASCII) into [downgrade].
So, I agree with you about that.

there are two cases:
  non-ascii addresses are normalized very "early" (right after entered), 

  non-ascii addresses are not normalized until it enters utf8header/smtpsession.

In the latter case, utf8header/utf8smtpext both need NFC recommendation,
if this WG agree on that. In the former case, we need NFC recommendation
in framework document, rather than [utf8header].

> >header To: address. It is the NORMAL CASE that they are different.

My previous text was intended narrowly to "downgrade/upgrade" context,
so Harald is right generally.

Soobok

---

From: Soobok Lee <[email-masked]>
Date: Wed, 13 Dec 2006 12:03:02 +0900

> so Harald is right generally.

I attach [downgrade] appendix that shows how RCPT_TO addresses are 
used for EAI downgrade. 

If we assume, RCPT TO: NON-ASCII-TO address is not NFCed,

   RCPT TO: <NON-ASCII-TO_noNFC> ALT-ADDRESS <ASCII-TO>

   EAI-Downgraded-To: <NON-ASCII-TO_noNFC {ASCII-TO}> <ASCII-TO>
   To: <NON-ASCII-TO_NFC {ASCII-TO}>

if NON-ASCII-TO_noNFC != NON-ASCII-TO_NFC  in some cases,
 downgrade/upgrade may be not successful, because the search&convert
 jobs over the header section may fail.

In some inplementations where NON-ASCII-TO addresses are normalized 
very "early", we need not care about this problem, because the 
same normalized address will be fed as RCPT_TO parameter.

Soobok

(attached [downgrade] excerpts)

YONEYA & Fujiwara       Expires February 17, 2007              [Page 10]
Internet-Draft                EAI Downgrade                     Aug 2006
   Original EAI message SMTP session
   MAIL From: <NON-ASCII-FROM> ALT-ADDRESS <ASCII-FROM>
   RCPT TO: <NON-ASCII-TO> ALT-ADDRESS <ASCII-TO>
   RCPT TO: <NON-ASCII-CC>
   -------------------------------------------------------------
   UTF8SMTP: UTF8
   Message-Id: MESSAGE_ID
   Mime-Version: 1.0
   Content-Type: text/plain; charset="UTF-8"
   Content-Transfer-Encoding: 8bit
   Subject: UTF8-SUBJECT
   From: <NON-ASCII-FROM {ASCII-FROM}>
   To: <NON-ASCII-TO {ASCII-TO}>
   CC: <NON-ASCII-CC>
   Date: DATE
   MAIL_BODY
                      Figure 1: Original EAI message
   In this example, there are two sessions, one is To:, the other is
   CC:.  The CC: session does not contain ASCII alternative address, it
   is not downgradable and bounced.  The To: session can be downgraded
   to the next example Figure 2.
YONEYA & Fujiwara       Expires February 17, 2007              [Page 11]
Internet-Draft                EAI Downgrade                     Aug 2006
   After SMTP Downgrading
   MAIL From: <ASCII-FROM>
   RCPT TO: <ASCII-TO>
   -------------------------------------------------------------
   EAI-Downgraded-From: <NON-ASCII-FROM {ASCII-FROM}> <ASCII-FROM>
   EAI-Downgraded-To: <NON-ASCII-TO {ASCII-TO}> <ASCII-TO>
   UTF8SMTP: UTF8
   Message-Id: MESSAGE_ID
   Mime-Version: 1.0
   Content-Type: text/plain; charset="UTF-8"
   Content-Transfer-Encoding: 8bit
   Subject: UTF8-SUBJECT
   From: <NON-ASCII-FROM {ASCII-FROM}>
   To: <NON-ASCII-TO {ASCII-TO}>
   CC: <NON-ASCII-CC>
   Date: DATE
   MAIL_BODY
                   Figure 2: SMTP Downgraded EAI message
YONEYA & Fujiwara       Expires February 17, 2007              [Page 12]

Internet-Draft                EAI Downgrade                     Aug 2006

   SMTP Downgrading adds EAI-Downgraded-From: and EAI-Downgraded-To:
   headers.

   EAI-Downgraded-From: <Non-ASCII {DOWNGRADED_FROM}> <DOWNGRADED_FROM>
   EAI-Downgraded-To: <Non-ASCII {DOWNGRADED_TO}> <DOWNGRADED_TO>

                Figure 4: Headers added by SMTP Downgrading

   Result of the header conversion downgrading.

   EAI-Downgraded-From:
       MIME(<Non-ASCII {DOWNGRADED_FROM}) <DOWNGRADED_FROM>
   EAI-Downgraded-To:
       MIME(<Non-ASCII {DOWNGRADED_TO}) <DOWNGRADED_TO>
   UTF8SMTP: Downgraded
   Message-Id: MESSAGE_ID
   Mime-Version: 1.0
   Content-Type: text/plain; charset="UTF-8"
   Content-Transfer-Encoding: 8bit
   Subject: MIME(UTF-8_SUBJECT)
   Downgraded: From: MIME(<NON-ASCII-FROM {ASCII-FROM}>)
   From: <ASCII-FROM>
   Downgraded: To: MIME(<NON-ASCII-TO {ASCII-TO}>)
   To: <ASCII-TO>
   Downgraded: CC: MIME(<NON-ASCII-CC>)
   Date: DATE

   MAIL_BODY

              Figure 5: Header conversion downgraded message

   MIME() stands for [RFC2047] encoding.

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Dec 2006 22:52:55 -0500

--On Wednesday, 13 December, 2006 10:14 +0900 Soobok Lee
<[email-masked]> wrote:

>...
> In the latter case, utf8header/utf8smtpext both need NFC
> recommendation, if this WG agree on that. In the former case,
> we need NFC recommendation in framework document, rather than
> [utf8header].
>...

For reasons that, I hope, have been explained adequately
already, these ideas, however good they may be, cannot be part
of the core protocol.  They simply need to be recommendations to
those who write or configure delivery MTAs and hence don't
belong in "framework" and probably not in "utf8headers" or
"downgrade" either. 

    john

---

From: Soobok Lee <[email-masked]>
Date: Wed, 13 Dec 2006 13:04:05 +0900

> belong in "framework" and probably not in "utf8headers" or

Still I stand on the same position: normalization is a local matter,
as you advised previously. my previous post was under assumption of
sweeping-NFC-recommendation.

Some sites or implementors may prefer NFKC over NFC for their 
local needs, for example, fullwidth katakana ->halfwidth one conversion.

someone may think NFKC is used already in IDNA and  NFKC is also good 
for localparts etc for consistency reason or for easy implementation. 

"Let's leave implementors the choice of normalization forms" may be not
so bad, i think.

Soobok

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 13 Dec 2006 12:03:35 -0000

On Wed, 13 Dec 2006 01:14:56 -0000, Soobok Lee <[email-masked]> wrote:

>> I suggested this could be written into utf8headers. ...

> in framework document, rather than [utf8header].

NO! You are making it all too complicated. If it leaves the originating  
MUA already normalized, then that is a Good Thing. But if, for whatever  
reason, it doesn't (either in the headers or the envelope), then you  
should LEAVE IT ALONE all the way through tramsmission, downgrading, or  
whatever else. Only when it reaches the (apparent) final destination where  
the <localpart> gets looked at should anybody be worrying about it. If  
that site has followed our advice only to allocate normalized (to NFC,  
NFKC or whatever else) localparts, then that would be a good time for it  
to normalize whatever has come in before trying to deliver it further.

One might then discuss whether it then normalizes the headers as well as  
the envelope, and whether that is or is not part of any upgrading that may  
then be done.

BUT, if you start normalizing it en route, then you run the risk of  
breaking any DKIM signatures that may be present, and it is going to be  
troublesome enough to get DKIM signatures to work with UTF8SMTP without  
adding extra complications such as that.

However (in disagreement with John), I do think that we should specify  
"localparts SHOULD be chosen by postmasters (whoever) in an already  
normalized form (to whatever level, but at least NFC, they choose)" and  
write that into one of our documents, probably utf8headers. But that is  
all.

---

From: Soobok Lee <[email-masked]>
Date: Thu, 14 Dec 2006 00:27:44 +0900

On Wed, Dec 13, 2006 at 12:03:35PM -0000, Charles Lindsey wrote:
> whatever else. 

You are right. I had the same idea with you. 
<assume: under the assumption that we agree on sweeping NFC recommendation>
What I really intended was "RCPT_TO parameter SHOULD be
normalized *before* it enters smtp-session. So, SMTP server SHOULD 
assume RCPT_Address is already in normalized form in downgrade/upgrade
conversion and so the server don't need en route normalization/modification." 
</assume>

> Only when it reaches the (apparent) final destination where  

Localparts( LHS of @ ) are mapped into or compared against local mailbox name 
for which  postmaster/adminstrator have the sole right to create 
in the normalized form. 
How to design local mailbox namespace is out-of-protocol issue and had better
be put into advice document for administrator/implementor.
So do mapping/comparison rules between (localpart) ==> (local mailbox name).

For example, while local mailbox name is created in NFC form,
the mapping rule can be looser (NFKC or NFKD), like this:
" SELECT mailbox from mailbox_tbl where mailbox=NFKD(localpart)
or mailbox=NFKC(localpart) ". 
Such rule had better be put into advice document to allow
more freedom to adminstrator/postmasters, i think.

Soobok

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 14 Dec 2006 12:17:25 -0000

> normalization/modification."

Not quite. Change "SHOULD" to "should" and I would agree with you. I.e. it  
may be good practice, but should not be necessary for correct operation.  
The only place where I want to see SHOULD is in regard to the practices of  
domain owners and their postmasters.

---

From: Soobok Lee <[email-masked]>
Date: Thu, 14 Dec 2006 22:19:14 +0900

> domain owners and their postmasters.

I agree with you. This recommendation may be put into the guideline document
section for postmasters.

Soobok

---

From: Martin Duerst <[email-masked]>
Date: Thu, 28 Dec 2006 13:09:16 +0900

>I will let Martin Duerst asnwer that, since he wrote it. Somewhere on the  
>Unicode site IIRC.

No, at W3C: Requirements for String Identity Matching and String Indexing,
http://www.w3.org/TR/WD-charreq, in particular
http://www.w3.org/TR/WD-charreq#3.

It's almost the oldest Working Draft still under development at W3C
(see just above http://www.w3.org/TR/#WD-stop).

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]