Subject: [EAI] text for eai-framework re: 2047+S/MIME

From: Philip Guenther <[email-masked]>
Date: Wed, 11 Oct 2006 02:43:19 -0600

I was asked to provide text for this an eternity ago (i.e., Montreal), so 
here finally is text for inclusion in section 4.2, "Transmission of Email 
Header in UTF-8 Encoding":

 	One particular characteristic of the email format is its
 	persistency: MUA are expected to handle messages that were
 	originally sent decades ago and not just those delivered
 	seconds ago.  As such, MUAs and mail filtering software
 	will need to continue to accept and decode header fields
 	that use the RFC 2047 encoding.  While extensions to both
 	POP3 and IMAP have been proposed to enable automatic
 	EAI-upgrade---including RFC 2047 decoding---of messages by
 	the POP3 or IMAP server, there are MIME content-types for
 	which that cannot be done or where the change would have
 	unacceptable side-effects.

 	For example, message parts that are cryptographicaly
 	signed (Content-Type: multipart/signed [SMIME][OpenPGP])
 	cannot be upgraded without breaking the signature.  Similarly,
 	message parts that are encrypted (Content-Type: multipart/
 	encrypted) may contain, when decrypted, header fields that
 	use the RFC 2047 encoding; such messages cannot be 'fully'
 	upgraded without access to cryptographic keys.

 	[[The possibility of server-side decryption and/or signature
 	verification was briefly discussed in the LEMONADE working
 	group.  The former would permit application of the
 	forward-without-download support to encrypted messages, while
 	the latter would permit verification-without-download.  As
 	currently stands, S/MIME and OpenPGP features are not
 	practically available to a LEMONADE client.]]

...and add references for [SMIME] (RFC 2633) and [OpenPGP] (RFC 3156).

This is unrelated, but while it's on my mind: for the security 
considerations:
 	If the EAI framework creates new forms of 'invalid' or
 	'malformed' message, then a new email attack is created:
 	in an effort to be robust, some or or most agents will
 	accept such message and interpret them as if they were
 	well-formed.  If a filter interprets such a message
 	differently than then final MUA, then it may be possible
 	to create a message which appears acceptable under the
 	filter's interpretation but which should be rejected under
 	the interpretation given it by the final MUA.  Such attacks
 	already exist for the existing encoding layers, e.g.,
 	invalid MIME syntax, invalid HTML markup, and invalid GIF
 	[reference?] encoding.

This exists at all layers, but mentioning TCP segment duplication or IP 
fragment overlaps is probably more distracting than helpful.

Philip Guenther

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 11 Oct 2006 15:38:55 +0100

On Wed, 11 Oct 2006 09:43:19 +0100, Philip Guenther  
<[email-masked]> wrote:

>  	seconds ago.

Indeed, but I don't think there is any requirement for those decades-old  
messages to be transportable through the modern SMTP network, or for them  
to survive filtering software. If you want to resend one of them to  
somebody, then you are more likely to send it as text/plain or  
message/rfc822. But yes, we still expect modern MUAs to display them  
correctly.

>  As such, MUAs and mail filtering software

But I am not so sure which MIME content-types (s/content/media/ BTW) are  
going to cause problems.
>  	upgraded without access to cryptographic keys.

In the first place, I think we have to assume that the signature was done  
_before_ any downgrading (i.e. not by any intermediate MTA). Hence you  
need to be able to restore the original message by upgrading before you  
try to check the signature. And for that you _do_ need to restore the  
original CTE (certainly for multipart/signed - well actually you are  
supposed to use only 7bit, though you might get away with 8bit).

Therefore, it should in principle suffice to upgrade the message before  
checking the signature.

Snags that can arise are when the signed material includes some  
Content-Type or Content-Disposition with UTF8 somewhere in it. That is  
most likely to be in some <parameter>, in which case is it RFC 2231 that  
you have to worry about, not RFC 2047. And we also need to say somewhere  
that Downgraded headers are to be allowed within MIME body-part headers  
(which currently are all supposed to begin with "Content-"). It may also  
be that some idiot tries to include a comment containing UTF-8 within one  
of those Content-* headers.

More realistic problems can arise if the signed object is a  
message/rfc822, in which case the downgrading may indeed have introduced  
some RFC 2047 encoding into the rfc822 headers therein. But RFC 3156 is  
totally silent on how to sign message/rfc888 and multipart objects, which  
is not helpful (they don't have their own CTEs for a start).

An alternative approach is to NOT downgrade message/rfc822 and body-part  
headers arising from multiparts, since it is most unlikely that they will  
fail to be transported intact through MTAs that provide 8BITMIME.

I think we really need to see some examples that might go wrong before we  
can proceed further.

>  	invalid MIME syntax, invalid HTML markup, and invalid GIF

And a good example off that is where a downgraded <addr-spec> is different  
to the version in the Downgraded header (that would be a difference in the  
<alt-address>, of course), and that is a Security Issue to be mentioned  
somewhere.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 Oct 2006 11:10:07 -0400

--On Wednesday, 11 October, 2006 15:38 +0100 Charles Lindsey
<[email-masked]> wrote:

>...
>>  	cannot be upgraded without breaking the signature.
>>  	Similarly, message parts that are encrypted (Content-Type:
>>  	multipart/ encrypted) may contain, when decrypted, header
>>  	fields that use the RFC 2047 encoding; such messages cannot
>>  	be 'fully' upgraded without access to cryptographic keys.
> 
> In the first place, I think we have to assume that the
> signature was done_before_ any downgrading (i.e. not by any
> intermediate MTA). Hence youneed to be able to restore the
> original message by upgrading before youtry to check the
> signature. And for that you _do_ need to restore theoriginal
> CTE (certainly for multipart/signed - well actually you
> aresupposed to use only 7bit, though you might get away with
> 8bit).
> 
> Therefore, it should in principle suffice to upgrade the
> message beforechecking the signature.

But signed messages are sensitive enough to any changes at all
in whatever the signature was calculated over that "sign,
downgrade, upgrade, check signature" is almost guaranteed to
fail (again, assuming signature over what is being upgraded or
downgraded).  

I see only a few ways out of this:

(1) Sign only body parts, one at a time, not headers.  Gee, that
is what S/MIME and PGP usually do.

(2) Encapsulate the signed message, signature and all.  This
will require some very careful de-encapsulation but should, in
principle, work.   As Harald points out, many of us are
convinced that we need a mechanism for encapsulation --for
forwarding in a pure 8bit environment even if not for
downgrading.  As he also points out, it is not on this WG's
critical path, but someone should write it up.  If no one else
gets to it, I'll do so sooner or later.  But I believe that it
will have inevitable normative dependencies on the UTF-8 Headers
document so, until there is rough consensus on that...

(3) Prohibit downgrading of signed messages.   I note that we
have no mechanism for doing that today.  One can prohibit
automatic and general downgrading by leaving off the ALT-ADDRESS
parameter, but cannot now prohibit a number of other possible
strategies, especially those performed by a Submission MTA using
out of band information.  

Suggestions welcome, but I think the discussion is moving toward
the kicking of dead horses.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Oct 2006 15:49:52 +0100

> is what S/MIME and PGP usually do.

Almost, but not quite. Suppose the body of the message is just plain text.  
After signing, it _might_ look like this:

   Header-Type: UTF8SMTP
   From:   } which might or might not have utf-8
   To:     } in them
   Mime-Version: 1.0
   Content-Type: multipart/signed; boundary=bar; micalg=pgp-md5;
     protocol="application/pgp-signature"

   --bar
& Content-Type: text/plain; (utf8-comment) charset=iso-8859-1;  
name="utf8-name"
& Content-Transfer-Encoding: quoted-printable
&
& =A1Hola!
&
& Did you know that talking to yourself is a sign of senility?

   --bar

   Content-Type: application/pgp-signature

   -----BEGIN PGP MESSAGE-----
   Version: 2.6.2
   ......................................
   -----END PGP MESSAGE-----

   --bar--

Just to be awkward, I included some utf-8 in the Content-Type header of  
the text/plain. I am sure no existsting signature program would do such a  
stupid thing, but it is legal. So, it it comes to some non-utf8smtp MTA,  
it has to be downgraded (using RFC 2047 for the <comment> and RFC 2231 for  
the <parameter>,)and the signature is now broken unless a subsequent  
upgrade can mend it.

But why does it need to be downgraded? A. because the MTA might not  
support 8BITMIME either, and you cannot fix utf-8 in those headers by  
providing a CTE.

OK, an exceedingly rare case, but not impossible. We might regard it as an  
accecptable risk, with a warning not to do those awkward things.

Things get more interesting if the body of the message is some  
message/rfc822. Then you get:

   Header-Type: UTF8SMTP
   From:   } which might or might not have utf-8
   To:     } in them
   Mime-Version: 1.0
   Content-Type: multipart/signed; boundary=bar; micalg=pgp-md5;
     protocol="application/pgp-signature"

   --bar
& Content-Type: message/rfc822 (utf8-comment) charset=iso-8859-1;
&   name="utf8-name"
&
& Header-Type: UTF8SMTP
& From:   } which might or might not have utf-8
& To:     } in them
& Subject: uft8-subject
& Content-Transfer-Encoding: quoted-printable
&
& =A1Hola!
&
& Did you know that talking to yourself is a sign of senility?

   --bar

   Content-Type: application/pgp-signature

   -----BEGIN PGP MESSAGE-----
   Version: 2.6.2
   ......................................
   -----END PGP MESSAGE-----

   --bar--

Now, in addition to the two 'awkward' cases in the Content-Type (which  
were totally stupid and useless) we have a Subject header to downgrade  
(with RFC 2047), and that is not stupid at all.

But, again, if it is subsequently upgraded, it should be OK

EXCEPT in the case where it had already been RFC 2047 encoded by the  
original MUA (unnecessary if UTF8SMTP was supported, but not disallowed).  
Now the upgrading agent does not know whether or not it is to be decoded  
before checking the signature.

And THAT is the real case we have to worry about - everything else in  
pgp-signed should work fine.

To make things still more interesting, suppose the thing we are trying to  
sign is some complete multipart/mixed. RFC is totally silent on hos to  
cope with that case, or even whether it is allowed. I someone will explain  
to me what is _supposed_ to happen in that case (even with no utf-8 in  
sight), then I would have a look at it.

> principle, work.

Yes, but that introduces all sorts of other complications, and there  
should be a clear demonstration of its necessity before we build it into  
the main protocol. (No prob;em it it is offered as an additiional feature,  
which the user can call upon manually when all else fails.)

> strategies, especially those performed by a Submission MTA using

I hope that can be avoided at least with pgp-signed etc. Sure there is a  
problem with a mechanism which signs the headers, but there is no such  
mechanism standardized as yet. It should be no problem PROVIDED this is  
taken into account when any such standard is written (but that may be  
difficult to bring about.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 12 Oct 2006 22:50:12 +0200

Charles Lindsey wrote:

> to be awkward, I included some utf-8 in the Content-Type header of
> the text/plain. I am sure no existsting signature program would do
> such a stupid thing, but it is legal

Raw UTF-8 in a MIME part Content-* header field, is that necessary ?
 
>& From:   } which might or might not have utf-8

Wait a moment, message/rfc822 has no charset, and message/rfc822 
is by definition no message/utf8smtp, or do I miss something here ?

Frank

---

From: John C Klensin <[email-masked]>
Date: Thu, 12 Oct 2006 19:17:59 -0400

--On Thursday, 12 October, 2006 22:50 +0200 Frank Ellermann
<[email-masked]> wrote:

> Charles Lindsey wrote:
> 
>> to be awkward, I included some utf-8 in the Content-Type
>> header of the text/plain. I am sure no existsting signature
>> program would do such a stupid thing, but it is legal
> 

It is not necessary.  2822, 4288, and 4829 and their
predecessors do not permit it, at least intentionally, and
nothing in the proposed EAI specs permit it either.  One might
infer that, with i18nemail, a filename that uses non-ASCII
characters should be permitted, but I think it would require a
separate proposal, one carefully enough written to cover
operating systems in which non-ASCII file names are in some
charset and coding other than Unicode expressed in UTF-8.

If the UTF-8 headers spec permits UTF-8 comments in embedded
MIME Content-type and similar headers, we might need to review
that issue for just this reason.  They certainly are not
necessary to support i18nemail address transport and then do
introduce extra complications and edge cases.

>> & Content-Type: message/rfc822 (utf8-comment)
>> charset=iso-8859-1; &   name="utf8-name"
>> &
>> & Header-Type: UTF8SMTP
> Wait a moment, message/rfc822 has no charset, and
> message/rfc822  is by definition no message/utf8smtp, or do I
> miss something here ?

No, you don't, at least as far as I can tell.   And the
message/utf8smtp construct, which I wish we would stop
discussing as if it were real without any sort of I-D on the
table that describes its structure, should not need (or permit)
a charset either, since any non-ASCII characters should
(actually MUST) be in UTF-8 by definition.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Oct 2006 03:44:17 +0200

John C Klensin wrote:

> the message/utf8smtp construct, which I wish we would stop
> a charset either

Yes, message/*  and multipart/* have no charset.  And my naive
definition of message/utf8smtp is "same as message/rfc822, but
adding UTF8-xtra-char as specified in the utf8headers I-D".

Maybe that's far too much, adding UTF8-xtra-char only for the
<dot-atom> of the <local-part> in <addr-spec> is the minimum,
i.e. <utf8-dot-atom> in the utf8headers I-D.

For the RHS I pass, that's your and Martin's territory.

> non-ASCII characters should (actually MUST) be in UTF-8 by 
> definition.

Yes, but 2047 / 2231 encodings still work for <utext> (Subject
etc., anything unstructured) and <ctext> (comments), it's not
strictly required to augment these conctructs by raw UTF-8.

Apparently <text> is only used in <quoted-pair>, and it's not
strictly necessary to allow backslashes before UTF8-xtra-char.

A <quoted-pair> is used in <dcontent>, where <dtext> doesn't
allow UTF8-xtra-char, that could be inconsistent, better leave
<text> as is, no UTF8-xtra-char.

The last case is <qtext> used by <quoted-string>.  For most
fields the complete <quoted-string> can be replaced by a 2047-
encoded word.  For a minimal solution only one <quoted-string>
needs UTF8-xtra-char, the <quoted-string> case of <local-part>.

Allowing UTF8-xtra-char in <qtext> but not <utext> + <ctext>
is pointless, you either get a maximal solution as in the I-D,
or a minimal solution affecting only <local-part>.  

And message/utf8smtp is my name for the resulting MIME message.
IMO it's no message/rfc822 and needs its own message subtype.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 13 Oct 2006 06:54:24 +0300

Frank Ellermann <[email-masked]> writes in gmane.ietf.ima

> is by definition no message/utf8smtp, or do I miss something here ?

Uhh. Message/rfc822 mime-type referes also to any successor  of it.

RFC 2822 did not introduced message/rfc2822 mime-type.

I think that message/rfc822  can be utf8smtp / i18nmail message.

/ Kari Hurtta

RFC 2046 quote:

   It should be noted that, despite the use of the numbers "822", a
   "message/rfc822" entity isn't restricted to material in strict
   conformance to RFC822, nor are the semantics of "message/rfc822"
   objects restricted to the semantics defined in RFC822. More
   specifically, a "message/rfc822" message could well be a News article
   or a MIME message.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Oct 2006 07:11:20 +0200

>    specifically, a "message/rfc822" message could well be a News article

Other quotes (same source)
| The "rfc822" subtype is used when the encapsulated content is itself
| an RFC 822 message.

| A media type of "message/rfc822" indicates that the body contains an
| encapsulated message, with the syntax of an RFC 822 message.

| No encoding other than "7bit", "8bit", or "binary" is permitted for
| the body of a "message/rfc822" entity.  The message header fields are
| always US-ASCII in any case
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [RFC 2822, "internet message format", 2.1]
| A message that is conformant with this standard is comprised of
| characters with values in the range 1 through 127 and interpreted as
| US-ASCII characters [ASCII].

The "isn't restricted to material in strict conformance" note in 2046
offers some wiggle room for message/rfc822, and it claims to encompass
message/news (where an 8bit CTE is okay), but even in netnews a header
is pure US-ASCII:

| The character set for header fields is US-ASCII.  Where the use of
| non-ASCII characters is required, they MUST be encoded using the
| MIME mechanisms defined in [RFC2047] and [RFC2231].

The draft-ietf-usefor-usefor-10 last call ends in a few hours...

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 13 Oct 2006 13:58:20 +0300

> adding UTF8-xtra-char as specified in the utf8headers I-D".
<...>
> IMO it's no message/rfc822 and needs its own message subtype.

Hmm, yes. When it turns out that message/rfc822 can not be expanded,
there is need for message/utf8smtp which serves same purpose as
message/rfc822.

That message/utf8smtp need to be downgraded same way that i18mail
on general. It is not encapsulation.

Downgrading have some details. In practise it can not be allowed
exists outside of i18mail environment (that means that 8BITMIME
is not enough.)

So that needs draft.  That is not critical, but if it is not written,
message/rfc822 will be used for that purpose even when it is
not allowed.

(IMHO using message/rfc822 for that breaks nothing. )

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Oct 2006 14:41:48 +0200

> on general. It is not encapsulation.

What arrives at the downgrade-gateway is something like that:

 To: <utf8-address>
 From: <utf8-address>
 Date: <some date>
 Message-ID: <[email-masked]>
 Subject: <2231-subject>
 x-more-header-fields: yes
 MIME-Version: 1.0
 Contect-Type: text/plain; charset=utf-8
 Content-Transfer-Encoding: 8bit
 
 Body 

Because it arrived via utf8smtp it's by definition message/utf8smtp.
The downgraded 8bit version would be:

 Received: from downgrad-in@example by downgrade-out@example with smtp
  for <downgraded-To>
 To: <downgraded-To>
 From: <downgraded-from>
 Date: <some date>
 Message-ID: <[email-masked]>
 Subject: <2231-subject>
 MIME-Version: 1.0
 Contect-Type: message/utf8smtp
 Content-Transfer-Encoding: 8bit

 Received: from utf8smt-sender by downgrade-in@example with utf8smtp
  for <utf8-address>
 To: <utf8-address>
 From: <utf8-address>
 Date: <some date>
 Message-ID: <[email-masked]>
 Subject: <2231-subject>
 x-more-header-fields: yes
 MIME-Version: 1.0
 Contect-Type: text/plain; charset=utf-8
 Content-Transfer-Encoding: 8bit
 
 Body 

The x-more-header-fields only copied to the downgraded header as far
as they are clean for message/rfc822.  The 2231-subject can support
language tags.  With a raw UTF-8 subject (and Content-language header
field) it's almost identical, just use 2047 to downgrade a raw UTF-8
subject (maybe copy the Content-Language, UTF-8 comments stripped or
encoded).

> that needs draft.

In essence the utf8headers I-D is it.  The 4288 registration would
violate a "should" (lower case) in 2046 5,2.4.  

> That is not critical, but if it is not written, message/rfc822 
> will be used for that purpose even when it is not allowed.

A new subsection 9.1 in "utf8headers" with the RFC 4288 registration
template.  Review by the ietf-types list.  Consensus here to go that
way.  IESG appoval for the I-D.     
 
> (IMHO using message/rfc822 for that breaks nothing. )

Legacy applications expecting that a message/rfc822 has an US-ASCII
header could include virus checkers etc., I've no idea what they do
if they're confronted with raw non-ASCII in a wannabe message/rfc822.

It could break RFC 4155 application/mbox; format=default (I'm not
sure about this, see appendix A).

Frank

---

From: [email-masked]
Date: 13 Oct 2006 16:39:42 +0300

>  Content-Transfer-Encoding: 8bit

Uhh.  Here you are using message/utf8smtp  as encapsulation
( IMHO it will _not_ work  as I stated earlier  -- for
  encapsulation is needed something like application/utf8smtp )

I was meaning something like:

MAIL FROM:<....> BODY=8BITMIME
RCPT TO:<....>
DATA
UTF8SMTP: UTF8
From: xxx    
To: ccc
Subject: Tässä viesti
MIME-Version: 1.0
Content-Type: message/multipart=xx
Content-Transfer-Encoding: 8bit

--xx

See attached message

--xx
Content-Type: message/utf8smtp
Content-Transfer-Encoding: 8bit

UTF8SMTP: UTF8
From: <[email-masked] [email-masked]}>
To: vvvv
Contect-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

xxxxx

--xx--

When this is downgraded result is something like

MAIL FROM:<....> BODY=8BITMIME
RCPT TO:<....>
DATA
UTF8SMTP: Downgraded
From: xxx    
To: ccc
Subject: {mime-encoded word encoded}
MIME-Version: 1.0
Content-Type: message/multipart=xx
Content-Transfer-Encoding: 8bit

--xx

See attached message

--xx
Content-Type: message/rfc822
Content-Transfer-Encoding: 8bit

UTF8SMTP: Downgraded
From: [email-masked]
To: vvvv
Contect-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

xxxxx

--xx--

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 13 Oct 2006 19:28:16 +0100

>> Raw UTF-8 in a MIME part Content-* header field, is that

No it is not, but it is not illegal. All it needs is a warning in the  
framework not to do it if pgp-mime is to be used.
> charset and coding other than Unicode expressed in UTF-8.

No, I think our documents should allow utf-8 in <parameter>s (although the  
only realistic case mentioned so far is filenames). I don't see why the  
operating system comes into it; these filenames, on the wire, will be  
either utf-8 os an RFC 2231 downgrading thereof.

>> message/rfc822  is by definition no message/utf8smtp, or do I

In the UTF8SMTP universe, the charset in message/rfc822 objects will be  
extended to utf-8. But they will then need a Header-Type: UTF8SMTP  
(whatever), both in the message/rfc822 _and_ in the main message headers,  
and the downgrade document must tell you to downgrade them.
> (actually MUST) be in UTF-8 by definition.

Yes, I agree. If we need to design an encapsulation mechanism for complete  
messages, then it will be called application/utf8smtp rather than  
message/utf8smtp, and its semantics will be exact;y the same as  
application/news-transmission - i.e. you encapsulate the whole object,  
ignoring its structurem and using a CTE of Q-P or Base84. That ensures it  
can be reconstructed exactly.

But I would rather not design such a mechanism at all, if we can avoid it  
for now.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 13 Oct 2006 19:43:56 +0100

> | The "rfc822" subtype is used when the encapsulated content is itself

Which everybody understands as now extended to RFC 2822, and which will be  
extended to UTF8SMTP, and which is also in process of being extended to  
news articles.

> | the body of a "message/rfc822" entity.

Whatever is the reason for that? And what if you want to send a  
message/rfc822 in some charset other than ASCII, and your MTA doesn't do  
8BITMIME? Is this restriction observed in practice?

>  The message header fields are

Which will no longer be true in the UTF8SMTP universe, so downgrading or  
bouncing will be required.

---

From: Kari Hurtta <[email-masked]>
Date: 13 Oct 2006 22:57:42 +0300

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> message/rfc822 in some charset other than ASCII, and your MTA doesn't
> do  8BITMIME? Is this restriction observed in practice?

Practice seems that when MIME message is processed recursively,
Content-Transfer-Encoding: -header is ignored on all other types
than leaf types.

This causes that following kind protocol violations often works:

        Content-type: multipart/mixed; boundary=XXX
        Content-Transfer-Enconding: quoted-printable

        --XXX
        Content-type: text/plain; charset=iso-8859-1
        Content-Transfer-Enconding: quoted-printable

        Some quoted-printable encoded text

I'm seen these quite often. Same parsers which handles multipart
that way, will also look message/rfc822 that way.  After all 
message/rfc822 is essentially processed recursively as is multipart/* 
-- differences between these two classes of types are quite little.

These are just my observations.   Number of MIME parser codes
which I have familiar is quite small.   Looking Content-Transfer-Encoding:
only on leaf types is quite common on them.

Because only "7bit", "8bit", or "binary" are allowed
on non-leaf types, parsers takes advantage of that fact.

It is not restricted to using
     Content-Transfer-Enconding: quoted-printable
for message/rfc822 but I claim that this 
Content-Transfer-Enconding -header will be ignored, 
when message is viewed  :-)

/ Kari Hurtta

---

From: Philip Guenther <[email-masked]>
Date: Fri, 13 Oct 2006 15:29:22 -0600

...
> Whatever is the reason for that?

RFC 2045, page 17, final paragraph:
    NOTE ON ENCODING RESTRICTIONS:  Though the prohibition against using
    content-transfer-encodings on composite body data may seem overly
    restrictive, it is necessary to prevent nested encodings, in which
    data are passed through an encoding algorithm multiple times, and
    must be decoded multiple times in order to be properly viewed.
    Nested encodings add considerable complexity to user agents:  Aside
    from the obvious efficiency problems with such multiple encodings,
    they can obscure the basic structure of a message.  In particular,
    they can imply that several decoding operations are necessary simply
    to find out what types of bodies a message contains.  Banning nested
    encodings may complicate the job of certain mail gateways, but this
    seems less of a problem than the effect of nested encodings on user
    agents.

> And what if you want to send a message/rfc822 in some charset other than 
> ASCII, and your MTA doesn't do 8BITMIME?

The body of the message/rfc822 can have its own content-transfer-encoding, 
which is set by that header field in the header of the message/rfc822 part 
itself (as opposed to the MIME header that wraps the message/rfc822 
itself).  For an example, please see RFC 2049, Appendix A ("A Complex 
Multipart Example").

> Is this restriction observed in practice?

Yes.  For example, the IMAP servers I know of won't do any CTE decoding of 
the header of message/rfc822 part when performing searches and would not 
treat a header field with the name "Content-T=79pe" as giving the 
content-type of the nested body part.

Philip Guenther

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 14 Oct 2006 09:07:54 +0200

[email-masked] wrote:

> Here you are using message/utf8smtp as encapsulation

Yes, the proposed "UTF8SMTP" header field is IMO not good enough
to manage this.  It labels the message _within_ its header, a
bit like the meta-content-type element for HTML documents.  Or
like the "guess which charset I'm in" XML-game.

With message/TBD (something different from message/rfc822) it's
possible to label it also outside of the message (no guesswork).

It's mostly relevant for MIME parts, but it could be also used
for files (with filesystems supporting MIME types, or mapped to
an extension, *.eml for rfc822, *.uml for utf8smtp, or similar).

> IMHO it will _not_ work  as I stated earlier  -- for
> encapsulation is needed something like application/utf8smtp

A message/TBD 8bit won't work with a 7bit-MTA at the next hop,
but it's okay for an 8BITMIME-MTA.  For 8bit to 7bit RFC 1652
apparently says that this is left as an exercise for gateway
operators.

If you have 8TO7 (8bit to 7bit) and U2D8 (utf8smtp downgraded
with encapsulated message/TBD 8bit), then 8TO7( U2D8( x )) does
the trick.  On the far end of this time travel thru 7bit-land
a simple gateway could undo 8TO7 resulting in a message/TBD
again, and an upgrade gateway could undo both 8TO7 and U2D8.

With some traps and pitfalls like trace header fields added
on the downgraded route.  A radical approach would be to remove
them:  How the downgrade and the upgrade gateways communicate,
via avian carrier, Fido, xmpp, mms, news, multihop 7bit SMTP,
or what else, is their private business.

From the POV of the "utf8smtp universe" anything else is just
"other-land", nobody cares what it is as long as it's no black
hole.  That's the responsibility of the downgrade gateway, it
is free to reject the mail if it doesn't like this.  Same idea
as chapter 3.2 in RFC 821 (real SPF fans know this by heart :-)

> I was meaning something like:
[...truncated....]
> UTF8SMTP: UTF8
[...]
> From: <[email-masked] [email-masked]}>
[...]

Okay, that's the input.  Why so complex with a message/utf8smtp
within "multipart/mixed; boundary=xx" ?  I guess you didn't want
"message/multipart=xx", and with my interpretation your input is
in esssence the same as:

~~~ cut ~~~

xxxxx
~~~ end ~~~

> When this is downgraded result is something like
[...truncated...] 
> UTF8SMTP: Downgraded
[...]
> Content-Transfer-Encoding: 8bit

You've apparently lost all info necessary to upgrade it again.
With CTE 8bit it's clear that you're talking with an 8BITMIME
MTA, and for that why not simply send this:

~~~ cut ~~~
Subject: =?latin-1?Q?T=E4ss=E4_viesti?=

UTF8SMTP: UTF8

xxxxx
~~~ end ~~~

For my variant the "UTF8SMTP: UTF8" header field in a top-level
header is merely a "please downgrade if you must / can" indicator.
Similarly "UTF8SMTP: downgraded" is "please upgrade if you can",
the actual process is a no-brainer over an 8BITMIME route.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 14 Oct 2006 11:10:27 +0300

> apparently says that this is left as an exercise for gateway

Yes, but in practise there is problems.  Sendmail will strip
message/TBD to 7-bit if next host do not announce 8BITMIME.
( I have already mentioned that. )

That hapens with sendmail 8.7 (1995/09/16) or any newer version.

(Yes.  I'm perhaps somewhat quilty about that. (*) )

/ Kari Hurtta

(*) Well, I suggested that mail should be bounced, but it is not 
    implemented. However, I may have some difficultions to found
    my over 10 year old mail archives... So on some details my
    memory may fail.

---

From: John C Klensin <[email-masked]>
Date: Sat, 14 Oct 2006 04:13:44 -0400

--On Friday, 13 October, 2006 19:28 +0100 Charles Lindsey
<[email-masked]> wrote:

>... 
> In the UTF8SMTP universe, the charset in message/rfc822
> objects will beextended to utf-8. But they will then need a
> Header-Type: UTF8SMTP(whatever), both in the message/rfc822
> _and_ in the main message headers,and the downgrade document
> must tell you to downgrade them.

I don't see any possible way to retroactively change the
definition of message/rfc822 "in the UTF8SMTP universe" or
otherwise.  To be able to do so without causing strange behavior
and problems in existing MUAs requires that one assume that
headers never get lost, that portions of messages never leak
from one environment to another, or otherwise.

>> table that describes its structure, should not need (or
>> permit) a charset either, since any non-ASCII characters
>> should (actually MUST) be in UTF-8 by definition.
> 
> Yes, I agree. If we need to design an encapsulation mechanism
> for completemessages, then it will be called
> application/utf8smtp rather thanmessage/utf8smtp, and its
> semantics will be exact;y the same
> asapplication/news-transmission - i.e. you encapsulate the
> whole object,ignoring its structurem and using a CTE of Q-P or
> Base84. That ensures itcan be reconstructed exactly.

Yes, but, as has been pointed out several times already, you
can't do that without either breaking the nested encoding rule
or needing to decode and re-code encoded body parts.  The latter
is likely to be inconsistent with "reconstructed exactly".

This is _not_ a proposal and several details would need to be
worked out to turn it into one (again, for those who believe an
encapsulation model is necessary, the I-D posting deadline is
0900 US ET on Monday), but I think that, if one wants to
encapsulate this stuff, one needs to have something very much
like

    Content-type: multipart/i18nmessage boundary="--xyz"
    
    --xyz
    Content-type: text/utf8smtp-header 
         charset=UTF-8 (required, no other options)
    Content-transfer-encoding: Base64 
          (probably, although "8bit" with 8BITMIME might work)

    [[headers, and only the headers, of the original message]]
    --xyz
    Content-type: (content-type of original message body 
        which, could, in principle, be a multipart)

    [[original body, encoded if needed according to usual 
      rules of, e.g., message/rfc822]
    --xyz--

Now that can be reconstructed exactly, modulo knowing how many
blank lines separate the original header from the body and
assuming that the body ends in an unambiguous CRLF.   Any
signing algorithm that is sensitive to the first, and probably
the second, is probably broken in practice anyway.   But that
is, I think, it -- shortcuts are just not going to work well.

> But I would rather not design such a mechanism at all, if we
> can avoid itfor now.

Agreed.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 14 Oct 2006 14:45:36 +0200

John C Klensin wrote:

> I think that, if one wants to encapsulate this stuff, one needs
> to have something very much like

> assuming that the body ends in an unambiguous CRLF.

Similar to message/TBD, but working in one step from utf8smtp
down to 7bit.  So that idea (multipart/TBB + text/header-TBD) is
probably better.  DKIM offers two ways of "canonicalization", I
missed that part of the discussion, but I hope it's okay.  And
the DKIM draft proposes a IANA registry for canonicalization
algorithms.  We can check if it's good enough in the last call.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 14 Oct 2006 17:07:30 +0200

Kari Hurtta wrote:
 
> Sendmail will strip message/TBD to 7-bit if next host do not
> announce 8BITMIME.

Odd, does it try that stunt with any 8bit content ?  1652 says
"it must cause no loss of information; MIME transport encodings
must be employed as needed to insure this is the case".  Losing
one of eight bits is FUBAR.

> I suggested that mail should be bounced, but it is not
> implemented. 

We can't freeze this WG until sendmail is upgraded to support 
RFC 1652 published more than 12 years ago.  Maybe all poor old 
pre-1652 MTAs (are forced to) duck behind non-sendmail MXs.

Are you really saying that sendmail announces 8BITMIME support,
and then forwards 8bit mails to non-8BITMIME hops by stripping
this bit ?

[2821 2.4] An originating SMTP client which has not successfully
| negotiated an appropriate extension with a particular server
| MUST NOT transmit messages with information in the high-order
| bit of octets.

That certainly doesn't mean "just strip the 8th bit on the side
of the client if the server doesn't offer 8BITMIME".  

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 14 Oct 2006 18:58:04 +0300

> pre-1652 MTAs (are forced to) duck behind non-sendmail MXs.

I just warn :-)

> and then forwards 8bit mails to non-8BITMIME hops by stripping

Only for types where quoted-printable or base64 is not allowed
and type can not handled recursively.   

That means message/* other that message/rfc822

( 8bit is also stripped from multipart preamble, I think.
  But that 8bit on multipart preamble is more or less undefined
  area. )

quoted-printable or base64 are applied to leaf mime types,
where these encodings are allowed.

/ Kari Hurtta

KNOWNBUGS file from sendmail distribution:

* 8->7 bit MIME conversion

  When sendmail is doing 8->7 bit MIME conversions, and the message
  contains certain MIME body types that cannot be converted to 7-bit,
  sendmail will strip the message to 7-bit.

doc/op/op.me from sendmail distribution:

      $=s  contains the set of subtypes of message that  can
           be  treated  recursively.  By default it contains
           only "rfc822".  Other "message/*" types cannot be
           8->7  bit encoded.  If a message containing eight
           bit data is sent to a seven bit  host,  and  that
           message  cannot  be  encoded  into seven bits, it
           will be stripped to 7 bits.

So at least documentation say so.

8BITMIME downgrade happens on sendmail/mime.c but it
is difficult to give meaningfull quote.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 15 Oct 2006 00:01:36 +0200

> That means message/* other that message/rfc822

That sounds less dangerous, ignoring message/news and a few
others like message/http for the moment...  

> 8bit is also stripped from multipart preamble, I think.

...and that too, maybe theoretically interesting for DKIM.

2046 says *text as defined in 822, modulo CTL details that's
US-ASCII, if senders try something else and the illegal bits
are stripped it's their problem.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Oct 2006 11:01:32 +0100

> and type can not handled recursively.

Which might just about make it legal. I have been looiing carefully at RFC  
1652, and I think I could argue convincingly either way. But it does  
include the words:

    Once a server SMTP supporting the 8bit-MIMEtransport service
    extension accepts a content body containing octets with the high-
    order (8th) bit set, the server SMTP must deliver or relay the
    content in such a way as to preserve all bits in each octet.

It is certainly most illiberal, and an example of extra code inserted into  
an implementation with the express intent of making it less liberal,  
without providing the slightest benefit elsewhere.

Jon Postel should be turning in his grave.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Oct 2006 11:40:03 +0100

> from one environment to another, or otherwise.

And there I disagree with you entirely. It is esential that we allow  
utf8smtp messages to be transmitted in a message/rfc822 (within the  
utf8smtp universe, that is). It is just too hard to persuade the world to  
implement new message types.

Currently, the concept of a correctly constructed object compliant with  
RFC 2822 and the MIME standards is well understood, as is the concept of a  
message/rfc822 object.

This WG is in the business of defing various extensions to such objeccts.  
We are defining extensions to RFC 2822, and we can define extensions to  
other standards as needed. Thus we define a new class of objects known as  
"utf8smtp messages" and define how they may be transported.

If we define an "extended message/rfc822 object", then clearly we must say  
that it may only be contained within an "utf8smtp message", in which case  
if you have occasion to downgrade the one, you must at the same time  
downgrade the other (or bounce the lot if that is too much trouble).

> is likely to be inconsistent with "reconstructed exactly".

I think there is a possible exception to the "nested encoded rule" when  
the inside encoding is absolutely nothing to do with the email protocols.  
That was the case with the media type news/transmission, whose prime  
purpose was to prevent any mail agent from getting inside it and tinkering  
with any internal structure that just happened to look like a mail  
structure. The object encapsulated inside was a news article, and if it  
chose to use some CTE within itself for whatever reason, then it was for  
the Netnews protocols to take it apart or put it together again.

I think the same argument could infact be applied to any application type  
- to the mail protocols they are just streams of octets to be preserved  
intact (by applying CTEs as necessary) and then served up to the intended  
application at the far end.

I agree, however, that it might be taking that principle a little bit too  
far to invent an application/utf8smtp whose purpose was to preserve the  
internal structure of a utf8smtp email message against the machinations of  
present day transports.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 17 Oct 2006 14:51:36 +0100

On Fri, 13 Oct 2006 22:29:22 +0100, Philip Guenther  
<[email-masked]> wrote:

>     restrictive, it is necessary to prevent nested encodings,...

Yes, I agree that nested encodings are to be avoided, certainly in these  
circumstances.

My problem was that I had understood (wrongly as it seems) that ANY CTE  
was forbidden on composite media types, on the grounds that Q-P and Bas64  
were clearly inappropriate, and the other three would produce no semantic  
effect, even if you wrote them (because you could only write ASCII in  
those places anyway - but of course we are about to change that).
>
>> And what if you want to send a message/rfc822 in some charset other  
>> than ASCII, and your MTA doesn't do 8BITMIME?
>
> The body of the message/rfc822 can have its own  
> content-transfer-encoding, which is set by that header field in the  
> header of the message/rfc822 part itself (as opposed to the MIME header  
> that wraps the message/rfc822 itself).

Yes, I knew that, but I had interpreted the words

>>> | the body of a "message/rfc822" entity.

as forbidding Q-P and Base64 even in the CTE field of the encapsulated  
email message itself. Which is clearly stupid.

But on re-reading those words, and on reading your paragraph above, I  
still do not see how I am supposed to read your meaining into what RFC  
2046 actually says.

---

From: Philip Guenther <[email-masked]>
Date: Tue, 17 Oct 2006 08:28:17 -0600

...
>> itself).
> as forbidding Q-P and Base64 even in the CTE field of the encapsulated email 
> message itself. Which is clearly stupid.
>
> But on re-reading those words, and on reading your paragraph above, I still 
> do not see how I am supposed to read your meaining into what RFC 2046 
> actually says.

The phrase <<the body of a "message/rfc822" entity>> must be read in the 
context of the definitions in section 2 of RFC 2045.  Looking there we 
see:
2.4.  Entity

    The term "entity", refers specifically to the MIME-defined header
    fields and contents of either a message or one of the parts in the
    body of a multipart entity.  The specification of such entities is
    the essence of MIME.  Since the contents of an entity are often
    called the "body", it makes sense to speak about the body of an
    entity.  <...>

So, in that phase, "the body" refers to the entire embedded message/rfc822 
message and not just the embedded message's body.  Okay, so we don't have 
to do any decoding to obtain access to the embedded message as a whole. 
However, to interpret that embedded message we have to whip out our handy 
rfc822 message parser, which may quite naturally notice fields in that 
message that themselves mark this as a MIME message whose body has some 
sort of transfer encoding, etc.

(I think this is a good example of why examples are important.  This 
confused the heck out of me when I first read RFC 1521...until I saw the 
examples, at which point the words in my mind stopped tumbling and started 
dancing.)

Philip Guenther