Subject: [EAI] mailto

From: Shawn Steele <[email-masked]>
Date: Mon, 1 Nov 2010 18:23:16 +0000

> Second, RFC 6068 (http://tools.ietf.org/html/rfc6068) was just published this month. But I'm happy to work on an update if necessary. I have cc'ed my co-authors to check whether they would also be happy to continue as co-authors.

> In terms of I18N, RFC 6068 is a big progress over RFC 2368. It allows non-ASCII content in any headers (where the header itself allows that) and in the body (via the generic UTF-8-based percent-encoding). This is already widely supported by many (though not all) MUAs.

> Further changes would be appropriate in the Syntax and Examples sections, to make clear that when using EAI, data can be sent as raw UTF-8, without using encoded words.

It seems to me that some updates might be necessary (but it seems reasonable to wait to know a little bit to better understand what they should be).  Particularly I think that users are likely to just use mailto:unicode@unicode without bothering with the % escaping, so it seems like some update it going to be necessary.

-Shawn

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 02 Nov 2010 14:38:59 +0900

On 2010/11/02 3:23, Shawn Steele wrote:

> Particularly I think that users are likely to just use mailto:unicode@unicode without bothering with the % escaping,

The "without bothering with the %-escaping" part is covered where an IRI 
(instead of only an URI in the strict sense) is accepted.

Regards,    Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Tue, 02 Nov 2010 11:36:09 -0400

--On Tuesday, November 02, 2010 14:38 +0900 "\"Martin J.
Dürst\"" <[email-masked]> wrote:

>> Particularly I think that users are likely to just use
>> mailto:unicode@unicode without bothering with the % escaping,
> 
> The "without bothering with the %-escaping" part is covered
> where an IRI (instead of only an URI in the strict sense) is
> accepted.

Martin,

Yes.  But this is exactly where I find several of the parallel
discussions troubling.

(1) EAI thinks an email address is, in Shawn's notation,
unicode@unicode (or possibly unicode@string-with-A-labels).  It
does not permit %-escapes in the domain name and many mail
systems will interpret %-signs in the local part as something
else entirely, e.g., routing information.

(2) For an IRI, mailto:unicode@unicode is perfectly reasonable.
However, the mapping to a URI, unless it is scheme/protocol
dependent, is likely to produce either
   mailto:%-escapes@%-escapes
or
   mailto:%-escapes@string-with-A-labels

Both of which are really bad news if one tries to get from
mailto:String to an email address by dropping the mailto and
leaving String.

I think that means that any sort of i18n MAILTO processor has to
get from those forms back to the non-escaped Unicode-in-UTF-8
strings that EAI expects before passing an internationalized
address off to a mail-sending or processing operation.   If one
reads the second paragraph of Section 5 of RFC 6068 (and the
warning in the third-to-last paragraph of Section 7), one can
claim that is implied there, but it is difficult (at best) to
parse that exact meaning out from the convoluted language that
was used, presumably to avoid a normative reference to the EAI
specs.

Note that this is a key difference from web and web-like
applications, where the applications can be assumed to be able
to deal with decoding the %-escapes themselves.  

My own guess is that, even if we discover that the specification
part of 6068 does not need modification for EAI, we are likely
to want to update the document with some very clear and specific
text, and some lurid examples, about what is expected and what
can go wrong.  In a sense, there is a meta-level Security
Consideration in this that is a bit stronger than the last few
paragraphs of Section 7: any level of carelessness in
implementations is likely to result in very bad behavior
including possible misdirection or loss of mail.

best,
   john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 2 Nov 2010 15:59:34 +0000

I used unicode@unicode instead of utf8@utf8 because the encoding of a document, or the internal representation of the address, may not necessarily be utf8.  (Eg: I don't want someone to stick utf8 bytes in a utf16 document :))

I think my concern is that for mailto: the distinction between an IRI and URI bucket is blurry to the average end user.  So I think that some buckets that are nominally URI's will likely end up with mailto:unicode@unicode in them.  Historically some applications have been generous in the strings they allow, so I expect non-ascii mailtos will probably continue to work.  I don't know if that's worth acknowledging in 6068.  It may be "worse" when people inadvertently cut & past from IRI to URI or URI to IRI.

One approach apps could take is to presume UTF-8 (IRI) if > ASCII is encountered, &/or presume % encoding (URI) if that's encountered.  Though John points out that testing for % isn't completely reliable in the local part.

-Shawn

 
http://blogs.msdn.com/shawnste

________________________________________
From: John C Klensin [[email-masked]]
Sent: Tuesday, November 02, 2010 8:36 AM
To: "Martin J. Dürst"; Shawn Steele
Cc: [email-masked]; [email-masked]; Larry Masinter \([email-masked]\)
Subject: Re: [EAI] mailto

Martin,

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Tue, 02 Nov 2010 12:17:02 -0400

--On Tuesday, November 02, 2010 15:59 +0000 Shawn Steele
<[email-masked]> wrote:

> I used unicode@unicode instead of utf8@utf8 because the
> encoding of a document, or the internal representation of the
> address, may not necessarily be utf8.  (Eg: I don't want
> someone to stick utf8 bytes in a utf16 document :))

Understood.  But we've said pretty strongly that we prefer UTF-8
in EAI addresses on the wire and in text on the wire more
generally.  So, even though I know there will be exceptions
(some deliberate, some screw-ups), I hope that utf16 document is
transmitted in utf8 no matter how it is stored.

> I think my concern is that for mailto: the distinction between
> an IRI and URI bucket is blurry to the average end user.

While mailto: may be worse than some hypothetical average case,
I think that distinction will _always_ be blurry to the "average
end user". One could even argue that it is designed to be
blurry: if it were not, we would have at least made some
modification to the structural syntax that would permit a parser
or not-very-bright end user to easily identify which is which.  

This is one of those situations in which the obvious cases,
especially with the HTTP and HTTPS schemes, will work fine but
the edge cases get nasty.  In some ways and because of the
history and necessity of "no one but the delivery server gets to
interpret a local-part" rule, mailto: is sort of a breeder of
edge cases but I'm not sure it is all that special.

That is ultimately one of the two reasons I've considered the
whole IRI approach the wrong way to go since the beginning, but,
since that position has gotten no traction, I think we need to
try to make it work.   That doesn't stop me from thinking that
doing so will be risky and potentially very hard for UIs that
have to force end lusers into making distinctions that may not
make sense to them.
 
>  So I
> think that some buckets that are nominally URI's will likely
> end up with mailto:unicode@unicode in them.

Yes.  Almost certainly, especially for addresses embedded in
text, signature lines, etc.

>  Historically some
> applications have been generous in the strings they allow, so
> I expect non-ascii mailtos will probably continue to work.  I
> don't know if that's worth acknowledging in 6068.  It may be
> "worse" when people inadvertently cut & past from IRI to URI
> or URI to IRI.

I expect so.  And at least cautionary notes in an update to 6068
are what I had in mind in my earlier note.

> One approach apps could take is to presume UTF-8 (IRI) if >
> ASCII is encountered, &/or presume % encoding (URI) if that's
> encountered.  Though John points out that testing for % isn't
> completely reliable in the local part.

In particular, [email-masked] and [email-masked] are
perfectly valid, 5321- and 5322-conforming, email addresses.
The the first one might be interpreted as a route via "20"
(whatever that meant) and the second as a mailbox name, or vice
versa.  Interpreting either as an IRI or URI containing
%-escapes would be bad news.   Again, I think 6028 actually has
this right, but only for a reader who is exceptionally careful.
For others, the whole situation is a trap for the unwary.

     john

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 04 Nov 2010 20:49:39 +0900

Hello John,

On 2010/11/03 0:36, John C Klensin wrote:
> Dürst\""<[email-masked]>  wrote:
>

Yes. Except that it was ascii@ascii, that has been the same since ages.

> mailto:String to an email address by dropping the mailto and

Yes. But please note that it has never been possible to get from 
"mailto:String" to the email address simply by dropping "mailto:". In 
particular, RFC 2368 (which is more than 12 years old and strictly and 
purely about ASCII-only mail addresses only) says, in Section 2:

                                                 Note that all URL
    reserved characters in "to" must be encoded: in particular,
    parentheses, commas, and the percent sign ("%"), which commonly occur
    in the "mailbox" syntax.

So your troubling "parallel discussions" may be mostly due to some 
people having read the specs, and others not. I seriously recommend you 
to read RFC 2368; you may (but should not) be surprised at how much text 
comes directly from there.

> address off to a mail-sending or processing operation.

Yes. If it simply implements %-decoding, without looking at the byte 
values, it will easily get the UTF-8 strings (Unicode-in-UTF-8 is a 
pleonasm). So EAI will be happy. What may go wrong is where this UTF-8 
string hits an non-EAI mail agent.

If the mailto processor implements some ASCII-only checking on top of 
%-decoding, then there will be some error. But one should assume that 
this check will be eliminated by whoever implements EAI. In case we are 
dealing with a non-EAI mail agent, raising an error of some kind will of 
course be the right thing to do.

> If one
> warning in the third-to-last paragraph of Section 7),

I would also like to point to the many examples in Section 6, for example:

    To indicate the address "[email-masked]" one would use:

    <mailto:[email-masked]>

While the example section is written for people who generate mailto: 
URIs from mail addresses, it shouldn't be too difficult to turn these 
examples around, implicitly reading them e.g.:

    The address indicated by the mailto: URI
    <mailto:[email-masked]> is "[email-masked]".

> one can
> was used,

If you have any proposals to improve the language and make it easier to 
understand, I'll be happy to integrate them.

> presumably to avoid a normative reference to the EAI
> specs.

This presumption is wrong. Escaping and unescaping the components of 
(mailto:) URIs is something that was done since long ago.

The only part where so "convolution" was used to avoid a reference to 
EAI was in Section two, in the item that starts with:

    5.  Percent-encoding of non-ASCII octets in the <local-part> of an
        <addr-spec> is reserved for the internationalization of the
        <local-part>.

> to deal with decoding the %-escapes themselves.

No, there is no such key difference. URIs (and IRIs) can contain 
%-escapes, and any software (be that Web-like or not) taking apart an 
URI and using its components has to know how to deal with these. This is 
the same since ages, and RFC 6068 didn't change anything.

> can go wrong.

I don't mind adding more examples and clarifications. Any text appreciated.

> In a sense, there is a meta-level Security

Regards,    Martin.

---

From: John C Klensin <[email-masked]>
Date: Thu, 04 Nov 2010 14:16:06 -0400

Hi, Martin,

--On Thursday, 04 November, 2010 20:49 +0900 "\"Martin J.
Dürst\"" <[email-masked]> wrote:

>...
>> unicode@unicode (or possibly unicode@string-with-A-labels).
>> It does not permit %-escapes in the domain name and many mail

And there have been email implementations for a decade or two
(even if not "ages") who, in the name of "8-bit-clean" or
interpretations of the robustness principle (with which I'm not
personally sympathetic) have been accepting non-ASCII (and often
non-Unicode) strings and trying to make them work.   And they
have worked in local environments and by prior agreement -- the
discussion in draft-iab-idn-encoding applies here too.

>> (2) For an IRI, mailto:unicode@unicode is perfectly
>> reasonable. However, the mapping to a URI, unless it is
>> scheme/protocol dependent, is likely to produce either
> Yes. But please note that it has never been possible to get
> from "mailto:String" to the email address simply by dropping
> "mailto:". 

You mean "never valid", not "never possible", don't you?  I
could introduce you to several large financial and "e-business"
operations that do it many times a day.  

While I'm often an advocate of "let those who violated the
standard deal with their own messes when we do something that
tightens or enforces the rules" and think it applies here, we
need to understand what we are getting ourselves into with this.

> In particular, RFC 2368 (which is more than 12
> years old and strictly and purely about ASCII-only mail
> addresses only) says, in Section 2:
>...
 
> So your troubling "parallel discussions" may be mostly due to
> some people having read the specs, and others not. I seriously
> recommend you to read RFC 2368; you may (but should not) be
> surprised at how much text comes directly from there.

I have read it and know what it says.  I am not only concerned
about those who have not but about those who deliberately have
ignored its provisions, secure in the belief that we will never
do anything that has bad consequences for them.  See above. 

>> I think that means that any sort of i18n MAILTO processor has
>> to get from those forms back to the non-escaped
>> Unicode-in-UTF-8 strings that EAI expects before passing an
>> internationalized address off to a mail-sending or processing
>> operation.
> 
> Yes. If it simply implements %-decoding, without looking at
> the byte values, it will easily get the UTF-8 strings
> (Unicode-in-UTF-8 is a pleonasm). 

Having run into a file about two weeks ago that had apparently
started life as BIG5 and then had the same transformation that
UTF-8 uses applied to it, and then was shipped with a
content-type of "UTF-8" (according to the guilty party, because
there is no registration for "B5UTF-8" or equivalent) please
forgive me if I sometimes add detail that should not be
necessary.

> So EAI will be happy. What
> may go wrong is where this UTF-8 string hits an non-EAI mail
> agent.

yes

>...
>...
> While the example section is written for people who generate
> mailto: URIs from mail addresses, it shouldn't be too
> difficult to turn these examples around, implicitly reading
> them e.g.:
>...

Right.  But the point was that some of the key examples are not
present for EAI purposes and may  create some false sense of
security, not that the existing examples are incorrect in any
way.

>...
> If you have any proposals to improve the language and make it
> easier to understand, I'll be happy to integrate them.

And I'll be happy to look at it _after_ EAI completes all of the
decision-making that might bear on it.

>...
> No, there is no such key difference. URIs (and IRIs) can
> contain %-escapes, and any software (be that Web-like or not)
> taking apart an URI and using its components has to know how
> to deal with these. This is the same since ages, and RFC 6068
> didn't change anything.

But that was exactly the point.  Web-like applications take URIs
and use their components natively.   Mail applications generally
don't, so we are dealing with the need for a method-specific
decoding intermediary.  And, if the folks who do those are not
familiar with both the MAILTO spec and the mail ones (or read
the MAILTO spec very carefully and conform exactly even without
knowing what they are doing), or if they are in a hurry and
don't care, bad things can happen that are excluded in the
"application deals with URIs directly" cases.

>...

best,
   john

---

From: Shawn Steele <[email-masked]>
Date: Thu, 4 Nov 2010 18:24:26 +0000

> people having read the specs, and others not. 

Part of the problem is that 99.9% of the people that use mailto: haven't read the specs.  Particularly when it's used in plain-text.  Eg: an email message.  In that case it gets seen by an end user.  I'd disagree with John that the right approach is to just let them live in the mess until they comply with the standards.  Instead it might be wise for the standard to acknowledge that in 8 bit environments, there may be information in the 8th bit that people are using.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Thu, 04 Nov 2010 15:03:24 -0400

--On Thursday, 04 November, 2010 18:24 +0000 Shawn Steele
<[email-masked]> wrote:

>> some  people having read the specs, and others not. 
> 
> Part of the problem is that 99.9% of the people that use
> mailto: haven't read the specs.  Particularly when it's used
> in plain-text.  Eg: an email message.  In that case it gets
> seen by an end user.  I'd disagree with John that the right
> approach is to just let them live in the mess until they
> comply with the standards.  Instead it might be wise for the
> standard to acknowledge that in 8 bit environments, there may
> be information in the 8th bit that people are using.

Shawn, no disagreement (and sorry if I wasn't clear).   I think
there are places where it is appropriate to say "let them live
in the mess" and move on.  Probably I think there are more of
those places than you do.  But, in this case, I pretty much
agree with your 99.9% if only because I see things starting with
"mailto:" in signature blocks that are clearly constructed by
people who think they are being clever and elegant and who
assume they know what is in the spec without having a clue where
they would even find it.

I think that is a relevant part of the difference here.  While
there is a formal/systematic definition for mailto: that is
quite well done and that, if read carefully, covers all the
cases, what it defines is a part of user-level syntax.
Expecting end users to understand those subtleties in something
they think they can type is very different from expecting
someone who is implementing SMTP or EAI to have read the spec.
If nothing else, if someone doing those implementations doesn't
read the spec, negative reinforcement will arrive quickly.  For
mailto:, end users and people pretty close to them in skill have
thought they knew what the spec required and permitted, and been
wrong, for, using Martin's term, ages.

    john