Subject: [EAI] Comments on IMA framework

From: Marcos Sanz/Denic <[email-masked]>
Date: Fri, 7 Apr 2006 16:58:58 +0200

All,

these are my comments on draft-klensin-ima-framework-01. I haven't been 
following the last discussions in the list. If some of these issues (or 
any of those in my following mails) have already been dealt with, then 
just give me a pointer.

* Section 1: "to embed them in the 'name phrases' of the relevant 
headers". Is hereby meant the "display-name" of 2822? Or in general all 
fields of type "phrase"? "Name phrase" is just not a concept for me.
* Section 1.2: "written in a subset of a Roman-derived script". I think 
the correct term is "Latin script".
* Section 3, 2nd bullet: "encoded in UTF-8 when the SMTP extension is 
used". That is not 100% correct: the update of the header field syntax 
should be independent of the transport extension.
* Section 5.1: "translating the message into a different language". I 
don't know how that should be an effective measure of downgrade during 
submission to the effects of this spec.
* Section 5.2: s/mechanism preserve/mechanism to preserve/
* Section 8.1: mailto is not to be modified in the IRI spec, but in RFC 
2368
* Section 10: "does not work for email local parts since they are 
case-sensitive". I think it's fair mentioning that most of the 
implementations and users don't make this distinction.
* Section 10: "IMA specifications do not, in general, raise any new 
security issues other than those associated with confusable characters". 
Well, there's the UTF-8 canonicalization exploit, cf Security 
Considerations of RFC 3629.

Best regards,
Marcos

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Fri, 07 Apr 2006 13:45:11 -0400

Marcos,

As has been pointed out earlier in the WG context, these 
documents and specifications cannot contain a tutorial on RFC 
2821 or 2822 (or on the POP and IMAP specs, etc.).  They are 
written on the assumption of reasonable knowledge of those 
specifications and the terminology typically used to discuss 
them.

More below.

--On Friday, April 07, 2006 16:58 +0200 "Marcos Sanz/Denic" 
<[email-masked]> wrote:

> All,
>
> these are my comments on draft-klensin-ima-framework-01. I
> haven't been  following the last discussions in the list. If
> some of these issues (or  any of those in my following mails)
> have already been dealt with, then  just give me a pointer.
>
> * Section 1: "to embed them in the 'name phrases' of the
> relevant  headers". Is hereby meant the "display-name" of
> 2822? Or in general all  fields of type "phrase"? "Name
> phrase" is just not a concept for me.

Yes, the "display name" of RFC 2822.  Many terms have been used 
for this particular phrase over the years.

* Section 1.2: "written
> in a subset of a Roman-derived script". I think  the correct
> term is "Latin script".

No, actually, the correct term is "Roman script" or 
"Roman-derived script".  "Latin" was introduced into what became 
the ISO character-coding community long ago -- back when it 
arguably referred to the actual Latin script plus a few 
extensions -- but is not used in the the linguistic or writing 
system communities except by back-adoption from the (now) 
ISO/IEC JTC1/SC2 usage.

> * Section 3, 2nd bullet: "encoded in UTF-8 when the SMTP
> extension is  used". That is not 100% correct: the update of
> the header field syntax  should be independent of the
> transport extension.

While specifications in the future might do other things, UTF-8 
headers are not permitted except with the protection of the SMTP 
transport option, so, unless I misunderstand your point, the 
statement is correct as written.

> * Section 5.1: "translating the message into a different
> language". I  don't know how that should be an effective
> measure of downgrade during  submission to the effects of this
> spec.

I do not understand this comment.

> * Section 5.2: s/mechanism preserve/mechanism to preserve/

Will try to remember to pick this up on the next iteration

> * Section 8.1: mailto is not to be modified in the IRI spec,
> but in RFC  2368

The advice we have gotten so far is that language in the IRI 
spec will need to be considered, potentially leading to either 
updating either it or replacing 2368.  If you don't consider the 
current wording acceptable, suggest text.

> * Section 10: "does not work for email local parts since they
> are  case-sensitive". I think it's fair mentioning that most
> of the  implementations and users don't make this distinction.

"mentioning" to what end.  Any SMTP implementation that takes 
advantage of that particular knowledge except for final delivery 
processing is in violation of RFC 2821 and, for that matter, of 
821.  If you think it worth mentioning, suggest text.

> * Section 10: "IMA specifications do not, in general, raise
> any new  security issues other than those associated with
> confusable characters".  Well, there's the UTF-8
> canonicalization exploit, cf Security  Considerations of RFC
> 3629.

Working on that in another context -- watch for I-D

best regards,
    john

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Mon, 10 Apr 2006 12:31:05 +0200

John,

> 2821 or 2822

100% agree..

> for this particular phrase over the years.

..and exactly because of that, I would try to stick, whenever possible, to 
the 2822-nomenclature.

> system communities except by back-adoption from the (now) 

AFAICT the most common term at the moment for the script, based on ISO 
usage and Unicode nomenclature, is "Latin script", but that is not 
important as long as we agree on the meaning.

> statement is correct as written.

Admittedly, maybe the point was not clear. In my comments to the 
Headers-document, I have tried to defend the position that the update to 
the internet mail format should be in terms of a syntax extension to 
support new *Unicode* characters, and not *UTF-8* characters. Ideally, the 
word "UTF-8" would not have a single appearance in the Headers-document, 
because it is just a matter of transport. I mean, I could store my mails 
in UTF-16, as long as I use UTF-8 to communicate with an SMTP server.

Now the second bullet in section 3 in the framework document is labelled 
"Email headers in UTF-8" and makes a reference to the Headers-document. 
Ideally, we keep here clearly the separation between envelope an content, 
too, and avoid the UTF-8 use.

> I do not understand this comment.

The section is about downgrading before or during submission by means of 
an off-the-band mechanism. I admit that is something that must be 
mentioned for the sake of completion. However, "translating the message" 
is not very effective, because sending the translated message to exactly 
same address that once didn't work, will not work either. Or am I missing 
your point? Were you talking of somehow "translating the address"?

> current wording acceptable, suggest text.

"The mailto URL scheme [RFC3987] needs to be modified when IMA is 
standardized."

> 821.  If you think it worth mentioning, suggest text.

"..does not work for email local parts since they are case-sensitive. 
Admittedly many implementations and users don't make case distinctions for 
the local part, but any security mechanisms based on that particular 
knowledge except for final delivery 
processing are in violation of RFC 2821".

How do you mean, btw, "except for final delivery processing"?

> Working on that in another context -- watch for I-D

Looking forward to that! Anyway I would not state here "do not raise any 
new security issues", if it is not 100% correct, and would include a 
reference to the Security Considerations section of RFC 3629.

Best regards,
Marcos

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 10 Apr 2006 21:30:58 +0100

> in UTF-16, as long as I use UTF-8 to communicate with an SMTP server.

Yes, but the tradition within the IETF is always to define the format of  
objects as they will appear "on the wire". I.e., if I am going to send you  
(by some transport mechanism) a document conforming to our standard (and  
if we have no prior arrangement otherwise), then it had better have its  
headers in UTF-8 and not in UTF-16. If you choose to convert it for  
storage, and display, then that is up to you.

Likewise, it is well known that Unix systems invariably store line endings  
with a single 'LF' character. Nevertheless, the standards always specify  
CRLF, because that is the form in which it should be passed from one  
application to another (private arrangements excepted).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 10 Apr 2006 21:30:58 +0100

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Tue, 11 Apr 2006 12:06:00 +0200

> > support new *Unicode* characters, and not *UTF-8* characters. Ideally, 
 
> > the

> storage, and display, then that is up to you.

Absolutely. My point is that the definition "on the wire" should appear in 
the STMP extensions document, and not in the header extensions document. 
The header extensions document should be format-neutral, so that if in the 
future e.g. we introduce a "SMTP extensions #2" with transport via UTF-16, 
the Internet Mail Format must not be updated again.

Regards,
Marcos

---

From: Jungshik Shin <[email-masked]>
Date: Tue, 11 Apr 2006 19:52:22 +0900

> the Internet Mail Format must not be updated again.

I bet such an extension will never be introduced given that RFC (2)822
is not "format-neutral". either. My understanding is that the Internet
Mail Format on the wire 'must be' upward compatible with ASCII, which
UTF-16/32 is not.

Jungshik

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Tue, 11 Apr 2006 13:31:13 +0200

> Mail Format on the wire 'must be' upward compatible with ASCII, which

Even if that were true, the new header extensions document should be 
format-on-the-wire neutral.
SoC: Separation of Concerns.

Regards,
Marcos

---

From: John C Klensin <[email-masked]>
Date: Tue, 11 Apr 2006 08:45:32 -0400

--On Tuesday, 11 April, 2006 12:06 +0200 "Marcos Sanz/Denic"
<[email-masked]> wrote:

>...
>> > because it is just a matter of transport. I mean, I could
>> > store my 
> mails
>> objects as they will appear "on the wire". I.e., if I am
>> going to send 
> you 
>> if we have no prior arrangement otherwise), then it had
>> better have its  headers in UTF-8 and not in UTF-16. If you
>> choose to convert it for  storage, and display, then that is
>> up to you.
> 
> Absolutely. My point is that the definition "on the wire"
> should appear in  the STMP extensions document, and not in the
> header extensions document.  The header extensions document
> should be format-neutral, so that if in the  future e.g. we
> introduce a "SMTP extensions #2" with transport via UTF-16, 

If we had a cleaner header-envelope distinction, that might
work.  But the reality is that the transport needs to know the
encoding of the message body.  There are several important
reasons for this, but three stand out:

(1) The MTA must insert trace fields into the top of the message
body (i.e., the headers).  Those trace fields are differentiated
from other headers only by their names, so various things need
to be able to understand those names.

(2) The end of message indicator must be understood by MTAs in
order to figure out when to close out a message and start
reading commands again.  That indicator is strictly defined, in
the transport spec, as _ASCII_ CRLF.CRLF.  UTF-8 works because
of the forward compatibility with ASCII; arbitrary codings do
not.

(3) At an implementation level, rather than a
theoretical-specification one, we are heavily dependent on those
ASCII CRLF sequences to denote logical line termination,
something that impacts the way transport is defined, as well as
the way at least one POP3 command and several extensions have
been defined.

For whatever it is worth, the possibility of multiple header
encodings, especially header encodings that change the
definition of CR and LF and the spelling of header field names
in transport, make the work of designing a gateway to a
different mail environment vastly more difficult as well.

More generally, to state more badly what others have stated
well, much of our experience with Internet protocols is that
they are successful to the extent to which we can keep the
complexity that arises from multiple options to a minimum.
Unless you can show extremely strong benefit for permitting
different message header encodings in transport  --  not merely
the hypothetical possibility of some future change -- I think
this is better specified as UTF-8  as it is now.   If someone
later develops a reason for a different encoding, issuing a new
or updated header specification will be the least of their
worries, since all of the boundary and transition issues will
need to be sorted out and documented.

Similar reasoning argues for specifying the most specific ACE
possible, not the generic term.

     john

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 11 Apr 2006 14:39:42 +0100

> the Internet Mail Format must not be updated again.

No, the header extensions document is in fact an extension to RFC 2822,  
and RFC 2822 in fact defines a format to be used "on the wire" (as that  
term is usually understoof by the IETF). The actual wording un 2822 is:

    This standard specifies a syntax for text messages that are sent
    between computer users, ..

and unless those users have some private agreement, the "wire" between  
them must carry whatever 2822 says.

RFC 2821 (and our proposed extensions to it) is just one of many protocols  
that might be used between those "computer users"; RFC 2822 (and our  
extensions) are meant to aply to them all. Essentially we are saying that  
the only standard form for *any* interchange of messages MUST use UTF-8 in  
the headers (unless the parties have privately agreed to use something  
else).

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Tue, 11 Apr 2006 18:06:06 +0200

John,

> encoding of the message body.

I am not so naive that I am claiming "the transport does not need to know 
about the content". Situations have already been mentioned in the list, as 
you just did, where this is not the case. This is not what I am discussing 
about.

Our particular specification of the transport, the one with the extension 
identifier "IEmail", defines the content is UTF-8 encoded, and can act 
accordingly when inserting new fields, parsing them, rendering them, 
whatever...

But to my eyes, the definition of the content, must be 
transport-independent, so that the same content can be delivered via 
diferent transport-dependent formats. SoC: Separation of Concerns. 

> complexity that arises from multiple options to a minimum.

To keep complexity to a minimum, issue modular definitions and 
specifications, following rules of high internal coherence and low 
external coupling.

> this is better specified as UTF-8  as it is now.

Again, the specification as UTF-8 should remain, but in the transport 
document.
Again, I am not trying to prove the necessity of a new message header 
encoding in transport.

It's about SoC. Are you all for monolithic specifications? Then we should 
quit dealing with six different working group drafts and put it all 
together in one doc. And the lower the number of different sections in 
that doc, the better.

Regards,
MArcos

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Tue, 11 Apr 2006 18:12:34 +0200

> them must carry whatever 2822 says.

Likewise from 2822

   In addition, this standard
   does not specify an encoding of the characters for either transport
   or storage; that is, it does not specify the number of bits used or
   how those bits are specifically transferred over the wire or stored
   on disk.

Regards,
Marcos

---

From: John C Klensin <[email-masked]>
Date: Tue, 11 Apr 2006 12:40:15 -0400

--On Tuesday, 11 April, 2006 18:06 +0200 "Marcos Sanz/Denic"
<[email-masked]> wrote:

>> > should appear in  the STMP extensions document, and not in
>> > the header extensions document.  The header extensions
>> > document should be format-neutral, so that if in the
>> > future e.g. we introduce a "SMTP extensions #2" with
>> > transport via UTF-16,  the Internet Mail Format must not be
>> > updated again.
>> 
> I am not so naive that I am claiming "the transport does not
> need to know  about the content". Situations have already been
> mentioned in the list, as  you just did, where this is not the
> case. This is not what I am discussing  about.
> 
> Our particular specification of the transport, the one with
> the extension  identifier "IEmail", defines the content is
> UTF-8 encoded, and can act  accordingly when inserting new
> fields, parsing them, rendering them,  whatever...
> 
> transport-independent, so that the same content can be
> delivered via  diferent transport-dependent formats. SoC:
> Separation of Concerns. 
> 
> specifications, following rules of high internal coherence and
> low  external coupling.
> 
>> different message header encodings in transport  --  not
>> merely the hypothetical possibility of some future change --
>> I think this is better specified as UTF-8  as it is now.
> 
> Again, the specification as UTF-8 should remain, but in the
> transport  document.
> Again, I am not trying to prove the necessity of a new message
> header  encoding in transport.
> 
> It's about SoC. Are you all for monolithic specifications?
> Then we should  quit dealing with six different working group
> drafts and put it all  together in one doc. And the lower the
> number of different sections in  that doc, the better.

Marcos, you can quote "SoC" as a principle as often as you like.
The IETF experience has been that, in practice, having these
things be exactly specified where people are likely to look (or
explicitly and normatively cross-referenced in those places)
saves a large number of problems.  That doesn't mean "monolithic
specifications"  -- there are reasons for separating many of
these, just as there are reasons for separating MIME into
multiple parts and, indeed, separating [2]821 from [2]822, etc. 

FWIW, these specifications were not separated due to grand
theories about modularization.  They were separated for two
completely pragmatic reasons:

(i) With the specifications separated, the editorial burden can
be spread among several people, rather than being concentrated
and hence larger.

(ii) Our educated guess --but it is no more than a guess-- is
that some of the specifications may be easier to finish than
others and more stable thereafter.  For example, if we can move
the explanations, overview, and theory into the framework
document and everything else out of it, it probably will not
need to change even as the other documents evolve.   At the
other extreme, the downgrade document and the as-yet-unstarted
operational advice one are likely to evolve considerably, and
require revision as a result, as experience accumulates.

If the document mix is wrong, the WG can, of course, change it.

regards,
   john

---

From: Chris Newman <[email-masked]>
Date: Tue, 11 Apr 2006 18:50:44 -0700

> the Internet Mail Format must not be updated again.

I strongly disagree.  Standards exist to promote interoperability and fewer 
options increases the chances of interoperability.  The fact is that the 
multiple variants of UTF-16 are less likely to interoperate in practice than 
the two different variants of TIFF (which have demonstrated interop problems). 
If we're doing our job right, we should specify UTF-8 and only UTF-8 as the 
wire format.

The principle of reducing options to improve interoperability completely trumps 
your "Separation of Concerns" principle in this case.

                - Chris

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Wed, 12 Apr 2006 10:15:37 +0200

Dear all,

My position is for keeping the on-the-wire format in the SMTP extensions 
document and keeping the pure mail syntax definition in the header 
extensions document. Arguments against this have been so far:
* "But the transport must know about the encoding of the content anyway". 
This is somehow a bogus reasoning: the transport definition will bound the 
content to the particular on-the-wire format, and thus implicitly know the 
encoding of the material being transported.
* "RFC 2822 makes an on-the-wire definition already, and thus can we". 
Actually not true, RFC 2822 explicitly does not make any storage or 
on-the-wire encoding definition.
* John's "having these things be exactly specified where people are likely 
to look". A very weak non-technical argument. Documents can be 
cross-referenced and readers will be expected to follow the links.
* Chris' interoperability argument "we should specify UTF-8 and only UTF-8 
as the wire format". I don't necessarily disagree, but you can achieve 
that by cross-requiring the documents.
I am somehow surprised that I am the only proponent of this conceptual 
separation and that, apparently, all participants are actively or 
passively (consenting silence) against it. Next time I will take the blue 
pill :-)

Best regards,
Marcos

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 12 Apr 2006 19:56:51 +0200

Marcos,

I think you are seeing the result of lots of bad history.....

in multiple cases there have been standards people have had to fight 
with where some aspect was not specified (X.500 GeneralText, BMPstring 
and other bizarreness; SQL error messages, the undefined transport 
mappings of CORBA, endian definitions in various places). The result has 
been that the IETF has grown an extremely strong aversion to protocols 
where one can assume, even by implication, that some aspect that is 
essential for communication is an "implementation choice", or may 
"depend on the transport mapping".

We may have  become too fond of overspecifying our implementation 
choices. But it's served us well so far.

In this case, I think we're all agreed that the set of documents 
describes an experiment where the only permitted transport charset for 
headings is UTF-8. No alternates.

So the change you're looking for will not cause any effect on the wire. 
It MAY cause more ease of descibing incompatible experiments at a later 
time; I don't know if that is a Good Thing or not.

More understanding?

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Thu, 13 Apr 2006 08:36:01 +0200

Harald,

> time; I don't know if that is a Good Thing or not.

Thanks for your explanation. Like everybody else, I have read 
specifications in the past which were bizarre and incomplete, thus I have 
*lots* of understanding for allergic reactions against badly written docs.

The change I am proposing is not aiming at leaving implementation choices 
or describing incompatible experiments at a later time, no. It's exactly 
yet another allergic reaction against badly written docs. It's about 
stopping spreading details all over the text at places which are not 
relevant or even misleading (cf the punycode-itis I mentioned in my 
comments to the SMTP extension), it's about putting some structure and 
making ease to read for those people that will have to implement it, it's 
about delivering good quality docs even if we are "only making an 
experiment". I don't have *any* understanding for excuses against that.

I don't want to consume too many resources at this point, though. 
Apparently, not so important.

Regards,
Marcos