Subject: Re: [EAI] MIME prohibition stupid vs. brilliant

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 01 May 2007 11:38:14 +0100

On Mon, 30 Apr 2007 15:17:41 +0100, Harald Alvestrand  
<[email-masked]> wrote:

> Frank Ellermann wrote:
>>
>> Now I'm confused.  It's no stupid prohibition.  It's based on three
>> assumptions:  (1) Messages consist of an ASCII header and a body.
>> (2) The body can be structured and/or encoded as specified in the
>> header of the message.  (3) Eveybody knows how a message/rfc822 is
>> structured, and how it specifies the encoding of its body, because
>> the message/rfc822 structure is what MIME and RFC 2045 are about.
>>
> The thing that is (possibly) stupid is extending this restriction across  
> every future MIME type that starts with the letters "message/". See RFC  
> 2046 section 5.2.2 for the rather specific restrictions on encoding for  
> message/partial.

There are lots of good things in MIME, but the message/* types are not one  
of them. They are so badly defined in RFC 204[56] that any attempt to add  
news ones, except of the most trivial kind, will inevitably lead to  
interoperability problems with the presently installed base.

Contrast this with multipart/* types where the constraints on future types  
are well documented, allowing new such types to have been introduced on  
several occasions without problem.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 01 May 2007 20:17:21 +0200

>> the message/rfc822 structure is what MIME and RFC 2045 are about.

 [quotes rearranged]
> The content of a message/partial doesn't consist of an ASCII header
> and a body, so I lost you at (1).

We're talking about different things.  When I wrote "messages" I meant
a complete message like a news article or an SMTP mail object, not the
MIME type message/*.  A message/partial is no complete mail, like an
image/jpeg or text/pdf is no complete mail:  They can only occur in
the body of a complete mail or in the body of a complete MIME part.
And for a complete mail (or a complete MIME part) there's a header.

This header can be empty in some special cases related to multipart/*,
but "empty" is no contradiction to ASCII.

> The thing that is (possibly) stupid is extending this restriction
> across every future MIME type that starts with the letters "message/".
> See RFC 2046 section 5.2.2 for the rather specific restrictions on
> encoding for message/partial.

It uses the term "individual message" for what I called "message" or
"complete message".  When I'm talking about media types I'd (try to)
say "message/*" or similar.  The terminology is interchangable for
message/rfc822, message/news, or message/utf8, not other message/*.

The message/partial stuff is a bit odd, but it also follows the
spirit of "make it as obvious for MIME unaware UAs as possible" with
its 7bit requirement.

There are subtle issues for EAI in 5.2.2, the spec. claims that the
final result of the reassembly is a "complete MIME entity".  That
can be of course a complete EAI message using UTF-8 in its header.

No "downgrading magic" can prevent this, after reassembly users may
find something in their inbox that's a message/utf8, if they like it
or not.

We are doomed if we stick to the fiction of a "UTF8SMTP universe".
Our message/utf8 can leak into legacy message/rfc822 environments,
we can't control it everywhere.  But we can document this case, the
reassembly of a "complete MIME entity" isn't guaranteed to be a
traditional message/rfc822.  One thing with MIME is stupid, its
security considerations are lousy, we have to roll our own.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 01 May 2007 20:34:27 +0200

> can be of course a complete EAI message using UTF-8 in its header.

No, that's crap, it can't be a message/utf8 because message/partial
is always 7bit, including the first part with the original header of
the <q cite="2046"> "inner" message </q>.

It gives us another problem, it's impossible to use message/partial
for fragments of a message/utf8.  If somebody wishes to fragment a
message/utf8 they have to downgrade it first.

At least it avoids the ugly case of an unexpected message/utf8 (after
reassembly) showing up in the inbox of a traditional user.  We need
a section somewhere stating that fragmentation is by definition (in
2046) impossible for a message/utf8.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 02 May 2007 12:12:58 +0100

> message/utf8 they have to downgrade it first.

Yes, I think that is correct. Message/partial is a can of worms and I can  
no obvious way in which it could be extended to work with EAI. I could  
happily live without it.
> 2046) impossible for a message/utf8.

Agreed. It needs to be mentioned somewhere, probably in utf8headers and  
maybe in smtpext.

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 02 May 2007 16:19:29 +0200

> but "empty" is no contradiction to ASCII.
OK, I was confused about your message because you had 1) and 2) talking 
about a "message", and 3) talking about "message/rfc822", and I thought 
you meant "message/*" in 1) and 2).

But you have still lost me, because I don't understand how you draw a 
conclusion from your 3 bullets that say anything about the stupidity or 
not of the restrictions on "message/*".

Harald

---

From: Kari Hurtta <[email-masked]>
Date: 06 May 2007 12:10:13 +0300

Harald Tveit Alvestrand <[email-masked]> writes in gmane.ietf.ima:

> > And the dicussion completely failed to consider the third possibility
> > of using message/rfc822, downgraded as necessary before it is allowed
> > to be seen in the non-UTF8SMTP universe (but presumably upgraded again
> > for display by recipients so capable).
> Yes, it failed to consider this possibility. Rather, there was nobody in
> the room who had taken leave of his senses enough to seriously want to
> advance such a suggestion.
> 
> Your suggestion has NOT seen any support on the list. And for good reason.

Current drafts supports that:   ( draft-ietf-eai-utf8headers-05.txt )

   4.2.  Changes on MIME headers

   <...>

   In all those header fields, Observe that such Content-Type and other
   header fields may be found both amongst the top-level fields of a
   message and also within multiparts; and also that a complete message
   conforming to this document may now appear as a message/rfc822 (in
   both cases, subject to downgrade when that is necessary)

ie. "a complete message conforming to this document may now appear as 
a message/rfc822"

That is exactly same what Charles Lindsey was suggesting. 

When working group was decided that there is no need labeling
for UTF8SMTP messages, that quite clearly follow that UTF8SMTP
messages does not need new label (ie. message/utf-8 type) when
they are attached to another message.

When whole message is not need labeling on UTF8SMTP case (ie no ESMTP
paramater), there is no need also different kind labeling if
message is attached to another message.

That makes "message/rfc822" just mean "mail message". And consistent
that "message/rfc822" is just that type what occurs inside of DATA
on smtp.

Actually "message/rfc822" is not mean to be only mail message,
but superset of it (rfc 2046):

   It should be noted that, despite the use of the numbers "822", a
   "message/rfc822" entity isn't restricted to material in strict
   conformance to RFC822, nor are the semantics of "message/rfc822"
   objects restricted to the semantics defined in RFC822. More
   specifically, a "message/rfc822" message could well be a News article
   or a MIME message.

If there is reason why "message/rfc822" can not be used to
mean any mail message on UTF8SMTP universe, same problems are
also on top level.   

For example if there is some problems for IMAP on UTF8SMTP universe,
when UTF8SMTP message is attaches as "message/rfc822", same problems
also aply to top level message. 

If someone is sending UTF8SMTP message and mail bounces, normal 
situation is that sender is UTF8SMTP capable and therefore original 
UTF8SMTP message can be returned to sender as "message/rfc822" without 
that any downgrading occurs.  This should not be optimized for
rare cases where sender can send  UTF8SMTP messages but can not receive
them for some reason.

Logically from this also follows, that "text/rfc822-headers; charset=utf-8" 
is used when mail header section is returned on bounced message on
UTF8SMTP universe. ('charset' parameter is used, because text/* types
require it. )  But that is side issue.  

/ Kari Hurtta

---

From: "YAO Jiankang" <[email-masked]>
Date: Wed, 9 May 2007 17:18:01 +0800

----- Original Message ----- 
From: "Kari Hurtta" <[email-masked]>
To: <[email-masked]>
Cc: "Kari Hurtta" <[email-masked]>
Sent: Sunday, May 06, 2007 5:10 PM
Subject: [EAI] Not a good reason? (Re: Minutes from Prague (fwd))

> Current drafts supports that:   ( draft-ietf-eai-utf8headers-05.txt )

I have discussed with the utf8headers editors who confirmed to me that he forgott to update this point to reflect the last EAI meeting discussion result in Prague.

the correct statment is 

"a complete message
   conforming to this document may now appear as a message/utf8smtp (in
   both cases, subject to downgrade when that is necessary). "

the editor will update this in next version.

YAO Jiankang

---

From: "abel" <[email-masked]>
Date: Wed, 9 May 2007 17:28:27 +0800

Yes, I am sorry about that I forgot to change that to 'message/utf8smtp'.
Sorry to Kari.

> I have discussed with the utf8headers editors who confirmed to me that he
forgott to update this point to reflect the last EAI meeting discussion
result in Prague.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 11 May 2007 16:59:24 +0100

On Wed, 09 May 2007 10:18:01 +0100, YAO Jiankang <[email-masked]> wrote:

> I have discussed with the utf8headers editors who confirmed to me that  
> he forgott to update this point to reflect the last EAI meeting  
> discussion result in Prague.
>
> the correct statment is
>
> "a complete message

No, you can't say that until there exists a correct definition of
message/utf8smtp to refer to.

And since it is the case that any message/utf8smtp that is invented will
not interoperate with the existing network, because many existig (and
compliant) MTAs will fail to downgrade it when sending it to non-8BITMIME
agents, I do not see how any such beast could be defined.

The only ways of sending an encapsulated UTF8SMTP message that will
actually interoperate with the existing network are:

1. To send it as an application/utf8-message, or

2. To downgrade it (if needed), and then send it as a message/rfc822

In both those cases, a further downgrade to Non-8BITMIME will work
correctly.

---

From: Chris Newman <[email-masked]>
Date: Fri, 11 May 2007 15:28:21 -0700

> agents, I do not see how any such beast could be defined.

I question the claim that an MTA which passes material outside the US-ASCII 
octet range to a non-8BITMIME server is standards complaint.

----excerpt from RFC 1652:
   If a server SMTP does not support the 8-bit MIME transport extension
   (either by not responding with code 250 to the EHLO command, or by
   not including the EHLO keyword value 8BITMIME in its response), then
   the client SMTP must not, under any circumstances, attempt to
   transfer a content which contains characters outside the US-ASCII
   octet range (hex 00-7F).
----excerpt from RFC 1652

                - Chris

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 14 May 2007 10:36:46 +0100

On Fri, 11 May 2007 23:28:21 +0100, Chris Newman <[email-masked]>  
wrote:

> I question the claim that an MTA which passes material outside the  
> US-ASCII octet range to a non-8BITMIME server is standards complaint.

Indeed. Correct behaviour for an MTA that finds it has 8bit data that it  
cannot downgrade to send to an non-8BITMIME system is to bounce the  
message back to the sender. However, it is known that some MTAs do, in  
fact, attempt to send the 8bit stuff as-is in that situation, and some  
truncate the 8th bit, which is worse IMO.

But, given that the 'correct' behaviour is to bounce, in this case this  
means bouncing a DSN back to the Reporting MTA, which is not really in  
anybody's interest. Therefore, it would be quite wrong for us to define  
the EAI DSN extension in such a way that bouncing of the DNS was  
inevitable in such situations - especially as there are two other ways in  
which it could be defined (using application/message-utf8smtp or  
message/rfc822) which  entirely avoid the problem.

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 18 May 2007 07:40:13 +0200

Charles Lindsey wrote:
> On Fri, 11 May 2007 23:28:21 +0100, Chris Newman 
> <[email-masked]> wrote:
>
> Indeed. Correct behaviour for an MTA that finds it has 8bit data that 
> it cannot downgrade to send to an non-8BITMIME system is to bounce the 
> But, given that the 'correct' behaviour is to bounce, in this case 
> this means bouncing a DSN back to the Reporting MTA, which is not 
> really in anybody's interest. Therefore, it would be quite wrong for 
> us to define the EAI DSN extension in such a way that bouncing of the 
> DNS was inevitable in such situations - especially as there are two 
> other ways in which it could be defined (using 
> application/message-utf8smtp or message/rfc822) which  entirely avoid 
> the problem. 
The correct behaviour when sending a DSN is to use a MAIL FROM of <>.

The correct behaviour when bouncing a message with MAIL FROM <> is to 
discard the message.

Next question?

---

From: "Kari Hurtta" <[email-masked]>
Date: 18 May 2007 12:27:09 +0300

Harald Alvestrand <[email-masked]> writes in gmane.ietf.ima:

> > Indeed. Correct behaviour for an MTA that finds it has 8bit data
> > that it cannot downgrade to send to an non-8BITMIME system is to
> > bounce the message back to the sender. However, it is known that
> > some MTAs do, in fact, attempt to send the 8bit stuff as-is in that
> > situation, and some truncate the 8th bit, which is worse IMO.
> >
> > us to define the EAI DSN extension in such a way that bouncing of
> > the DNS was inevitable in such situations - especially as there are
> > two other ways in which it could be defined (using
> > application/message-utf8smtp or message/rfc822) which  entirely
> > avoid the problem.

Yes.   And therefore DSN on lost, if it reaches 8BITMIME to
ASCII gateway.

Of course it is unlikely that it reaches 8BITMIME to
ASCII gateway when original message, which caused DSN, was 
UTF8SMTP message.  To avoid these border cases, UTF8SMTP to 
8BITMIME gateway is better downgrade this DSN. 

Specially many MTAs exists which do not support 8BITMIME. They
can configured announce 8BITMIME, but then they just-send-8bit.
More about that on later on this message. Therefore that border
case where next host do not announce 8BITMIME is quite possible
and common.

Equal important case is when UTF8SMTP message is attached to 
some message and message is sent to ASCII user. 

Possible downgrade alternatives for UTF8SMTP to 8BITMIME gateway is 
discussed later on this message.   Frank Ellerman is saying that that 
downgrade must be done on 8BITMIME to ASCII gateway. But there is problem:
   When mail is leaved UTF8SMTP universe, there is no any guarantee
   that next 8BITMIME to ASCII gateway knows anything about
   these message/utf8smtp and other new message/* types. Therefore 
   correct place handle this is on  UTF8SMTP to 8BITMIME gateway.

  +---------------------------------------------------------+
  | ASCII universe                                          |
  |                                                         |
  |   +--------------------------------------------------+  |
  |   | 8BITMIME universe                                |  |
  |   |                                                  |  |
  |   |   +-------------------------------------------+  |  |
  |   |   | UTF8SMTP universe                         |  |  |
  |   |   |                                           |  |  |
  |   |   |                                           |  |  |
  |   |   |                                           |  |  |
  |   |   |                                           |  |  |
  |   |   +-------------------------------------------+  |  |
  |   |                                                  |  |
  |   +--------------------------------------------------+  |
  |                                                         |
  +---------------------------------------------------------+

Seems that currently that (subset of) WG wants that message/utf8smtp 
is sent as is from UTF8SMTP universe to 8BITMIME universe; that is 
with content-transfer-encoding 8bit.

That itself does not violate RFC 2045 "EXPRESSLY FORBIDDEN"
rule.  However if that is specified that way, it is saying:

   * 8BITMIME universe to ASCII universe gateways must
     violate standards, otherwise that does not work.

I do not think that this is wise message to be sent.
Of course it is possible that actually all implementations
violate standards, but that is hard to prove.

I oppose that attached 8BITMIME messages are sent as 
message/utf8smtp to 8BITMIME universe.

UTF8SMTP universe is that area which knows these new types.
UTF8SMTP extension for SMTP is defined so that experient can 
be compatible with existing software.

When message/utf8smtp type with 8-bit content-transfer-encoding 
is sent from  8BITMIME universe to ASCII universe, there is 
following possibilities for gateway:

      (1) "bounce" message instead (or discard on case of DSN) 
           -- that is case where can be said that this 
              experiment does  not work.

      (2) convert 8-bit to quoted-printable or base64 --
          on that case it is violating standards (A2)

      (3) send 8-bit as is -- on that case it is
          violating standards (A3)

      (4) downgrade message/utf8smtp. This is corresponding
          that RFC 2045 says that encoding must be done
          at the innermost level.  However most of 8BITMIME
          to ASCII universe gateways knows nothing about
          message/utf8smtp. If it knows about message/utf8smtp,
          it is from UTF8SMTP to ASCII universe gateway instead.

      (5) strip 8-bit to 7-bit  -- on that case it is violating 
          standards (A5)       

A2: RFC 2045 "EXPRESSLY FORBIDDEN" rule:

   Certain Content-Transfer-Encoding values may only be used on certain
   media types.  In particular, it is EXPRESSLY FORBIDDEN to use any
   encodings other than "7bit", "8bit", or "binary" with any composite
   media type, i.e. one that recursively includes other Content-Type
   fields.  Currently the only composite media types are "multipart" and
   "message".  All encodings that are desired for bodies of type
   multipart or message must be done at the innermost level, by encoding
   the actual body that needs to be encoded.

A3: RFC 1652 "under any circumstances" rule:

A5: RFC 1652 "preserve data" rule:

   Once a server SMTP supporting the 8bit-MIMEtransport service
   extension accepts a content body containing octets with the high-
   order (8th) bit set, the server SMTP must deliver or relay the
   content in such a way as to preserve all bits in each octet.

There is several possible alternatives which UTF8SMTP to 8BITMIME
gatewaty may use when downgrading message which includes message/utf8smtp
mime type as on subpart.

      (1) Type message/utf8smtp can be replaced with message/rfc822 and
          content downgraded same way than top level UTF8SMTP message.

      (2a) Type message/utf8smtp can be replaced with application/message-utf8smtp
           and content is keep unaltered.

      (2b) Type message/utf8smtp can be replaced with 
           applicate/message; type=" message/utf8smtp" and content is keep 
           unaltered.

      (3)  Type message/utf8smtp can be replaced with 
           multipart/utf8-encapsulated and content is generated
           as on draft-hurtta-eai-encapsulation have specified
           (chapter "5.  Encapsulation").

      (4)  Type message/utf8smtp can be replaced with 
           multipart/alternative and content is generated as
           following:

           (4.1) First body part is type message/rfc822 and
                 content of that body part is downgraded from
                 original message/utf8smtp  same way than top level 
                 UTF8SMTP message.

           (4.1) Second body part is type application/message-utf8smtp
                 and content is form original  message/utf8smtp
                 unaltered.

           This alternative duplicates space usage, but is quite
           easy to handle for exiting MUAs.

On past discussions following MTAs are at least found which
do not support 8BITMIME (but can be configured to announce
8BITMIME):

     Communigate Pro:
          http://www.stalker.com/CommuniGatePro/SMTP.html

          Advertise 8BITMIME

          If your Server does not report the 8BITMIME capability, some mailers 
          and servers will MIME-encode all non-ASCII messages that they send to 
          your Server . This server-side encoding can cause troubles for many 
          old mail clients. To avoid these troubles, your Server should report 
          the 8BITMIME capability.

          Note:The CommuniGate Pro SMTP module never converts non-ASCII messages 
          into the MIME form itself, and (according to RFC1652) it should not 
          advertise the 8BITMIME capability. But the modern Internet is completely 
          8-bit transparent and clean, so it is safe to enable the Advertise 
          8BITMIME option, preventing other servers from doing unneeded 8bit-to-MIME 
          message conversion.

    Exim:
          http://www.exim.org/exim-html-current/doc/html/spec_html/ch14.html#SECTalomo

          accept_8bitmime	Use: main	Type: boolean	Default: false

          This option causes Exim to send 8BITMIME in its response to an SMTP EHLO 
          command, and to accept the BODY= parameter on MAIL commands. However, 
          though Exim is 8-bit clean, it is not a protocol converter, and it takes no 
          steps to do anything special with messages received by this route. 
          Consequently, this option is turned off by default. 

    Qmail:
          http://cr.yp.to/smtp/8bitmime.html

http://en.wikipedia.org/wiki/8BITMIME lists also following MTAs which do not announce
8BITMIME:

    * Microsoft Exchange Internet Mail Service (through version 5.5)
    * Netscape Messaging Server 4.15

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 18 May 2007 14:09:27 +0100

>> inevitable in such situations - especially as there are two other ways  
>> in which it could be defined (using application/message-utf8smtp or  

Indeed, but that is actually slightly worse than the bouncing scenario,  
because it will be less likely that the problem will get noticed.

If we are defining a protocol for reporting delivery of messages, then it  
seems highy desirable that such reports should be returned through a  
reliable channel. So why are we proposing to define a reporting channel  
which actually has unreliability designed into it?

---

From: John C Klensin <[email-masked]>
Date: Fri, 18 May 2007 10:08:49 -0400

--On Friday, 18 May, 2007 14:09 +0100 Charles Lindsey
<[email-masked]> wrote:

>> The correct behaviour when bouncing a message with MAIL FROM
>> <> is to   discard the message.
> 
> Indeed, but that is actually slightly worse than the bouncing
> scenario, because it will be less likely that the problem will
> get noticed.
> 
> If we are defining a protocol for reporting delivery of
> messages, then it seems highy desirable that such reports
> should be returned through a reliable channel. So why are we
> proposing to define a reporting channel which actually has
> unreliability designed into it?

Because otherwise it is really easy to get mail loops.

More to the point, this has been established practice for more
than a quarter century.  Trying to reopen it at this point
violates both the general principle that EIA doesn't try to
change things that involve the same issue whether i18email is
used or not as well as the simple improbability of the installed
base agreeing to change this.

     john

---

From: Kari Hurtta <[email-masked]>
Date: 19 May 2007 09:05:09 +0300

> base agreeing to change this.

Let's look scenario again. I prove that removing UTF8SMTP from
scenario DSN is not lost.  Therefore this is UTF8SMTP issue.

That is where there is " unreliability designed into it".

Ascii user X sends mail to address which bounces (recipient mailbox
is full or recipient address is mistyped)
   
MUA which is used to send message is UTF8SMTP capable.
Because UTF8SMTP is negotiated proprerty, there is no 
"use UTF8SMTP" switch on MUA. That just confuses users.

Subject of message is   Terveisiä täältä

All addresses used are ASCII 

1)  Case: MSA is UTF8SMTP capable
    ( message/utf8smtp used on scenario)
     
     ( Sender X )
                       normal mail (A)
     [ MUA      ]      -------------> [ MSA ]                 
                        UTF8SMTP        |    UTF8SMTP
                                        |
                                        v
                                      [ MTA ]             
                                        |   UTF8SMTP
                                        v
                                      [ MTA ]         
     DSN with  message/utf8smtp subpart  |   DSN ( B)
               with 8bit content         |   UTF8SMTP
                                         v
                                      [ MTA ]
     DSN with  message/utf8smtp subpart  |   DSN (C)
               with  8bit content        |   8BITMIME
                                         |
                                         v
                                      [ MTA ]
                   DSN is dropped     [     ]    
                   (D)                [     ]   
                                         .  no 8BITMIME
                                         .
                                         .
                                      [ MTA ]
                                      [ mailstore ]
                                      ( Sender X )

(A)  Message is UTF8SMTP message because subject is
     not encoded.
      
(B)  Mail bounces. On generated DSN there is
     subpart with
          Content-type: message/utf8smtp
          Content-Transfer-Encoding: 8bit 

(C)  UTF8SMTP to 8BITMIME gateway was not 
     downgraded message/utf8smtp   subpart. It is still
          Content-type: message/utf8smtp
          Content-Transfer-Encoding: 8bit 
     This allowed by MIME.

(D)  8BITMIME to 7-bit gateway can not convert DSN                       
     with that message/utf8smtp subpart 
     because of RFC 2045 (*) forbids:
          Content-type: message/utf8smtp
          Content-Transfer-Encoding: quoted-printable 

(*)   RFC 2045 "EXPRESSLY FORBIDDEN" rule:

2) Case: MSA is not UTF8SMTP capable 

     ( Sender X )
                       normal mail (E)
     [ MUA      ]      -------------> [ MSA ]                 
                        8BITMIME        |    8BITMIME
                                        |
                                        v
                                      [ MTA ]             
                                        |    8BITMIME
                                        v
                                      [ MTA ]         
     DSN with  message/rfc822 subpart    |   DSN ( F)
                                         |   8BITMIME
                                         v
                                      [ MTA ]
     DSN with  message/rfc822 subpart    |   DSN 
                                         |   8BITMIME
                                         |
                                         v
                                      [ MTA ]
                   DSN is not dropped [     ]    
                   (G)                [     ]   
                                         |  DSN
                                         |  no 8BITMIME
                                         v
                                      [ MTA ]
                                      [ mailstore ]
                                      ( Sender X )

(E) Because MSA does not annouce UTF8SMTP
    MUA encodes subject Terveisiä täältä

    Therefore mail is just normal 8BITMIME message

(F)  Mail bounces. On generated DSN there is
     subpart with
          Content-type: message/rfc822
          Content-Transfer-Encoding: 8bit 

(G) 8BITMIME to 7-bit gateway can convert DSN                       
     with that message/rfc822 subpart to:
          Content-type: message/rfc822
          Content-Transfer-Encoding: 7bit

Now Sender X is informed that recipient mailbox
is full or recipient address is mistyped.

On case 1 it was because  MSA  was UTf8SMTP capable
and message/utf8smtp was used outside of 
"UTF8SMTP universe".

On next case spacification is rewritten and
application/message-utf8smtp  is used instead.

3)  Case: MSA is UTF8SMTP capable
    ( application/message-utf8smtp used on scenario)

     ( Sender X )
                       normal mail (H)
     [ MUA      ]      -------------> [ MSA ]                 
                        UTF8SMTP        |    UTF8SMTP
                                        |
                                        v
                                      [ MTA ]             
                                        |   UTF8SMTP
                                        v
                                      [ MTA ]        
 DSN with  application/message-utf8smtp  |   DSN ( I)
    subpart    with 8bit content         |   UTF8SMTP
                                         v
                                      [ MTA ]
 DSN with  application/message-utf8smtp  |   DSN 
     subpart   with  8bit content        |   8BITMIME
                                         |
                                         v
                                      [ MTA ]
                   DSN is not dropped [     ]    
                   (J)                [     ]   
                                         |  DSN
                                         |  no 8BITMIME
                                         v
                                      [ MTA ]
                                      [ mailstore ]
                                      ( Sender X )

(H) Message is UTF8SMTP message because subject is
    not encoded.
    
(I) Mail bounces. On generated DSN there is
     subpart with
          Content-type: application/message-utf8smtp
          Content-Transfer-Encoding: 8bit   

(J) 8BITMIME to 7-bit gateway can convert DSN                       
     with that application/message-utf8smtp subpart to:
          Content-type: application/message-utf8smtp
          Content-Transfer-Encoding: quoted-printable

Now Sender X is informed that recipient mailbox
is full or recipient address is mistyped.

On next case spacification is rewritten
and message/utf8smtp is used firts and
then downgraded to message/rfc822

4) Case: MSA is UTF8SMTP capable
   ( message/utf8smtp used and downgraded on scenario)

     ( Sender X )
                       normal mail (K)
     [ MUA      ]      -------------> [ MSA ]                 
                        UTF8SMTP        |    UTF8SMTP
                                        |
                                        v
                                      [ MTA ]             
                                        |   UTF8SMTP
                                        v
                                      [ MTA ]         
     DSN with  message/utf8smtp subpart  |   DSN ( L)
               with 8bit content         |   UTF8SMTP
                                         v
                                      [ MTA ]
     DSN with  message/rfc822 subpart    |   DSN (M)
                                         |   8BITMIME
                                        v
                                      [ MTA ]
                   DSN is not dropped [     ]    
                   (N)                [     ]   
                                         |  DSN
                                         |  no 8BITMIME
                                         v
                                      [ MTA ]
                                      [ mailstore ]
                                      ( Sender X )

(K) Message is UTF8SMTP message because subject is
    not encoded.

(L) Mail bounces. On generated DSN there is
     subpart with
          Content-type: message/utf8smtp
          Content-Transfer-Encoding: 8bit 

(M) UTF8SMTP to 8BITMIME gateway is downgrading
    message/utf8smtp subpart to: 
          Content-type: message/rfc822
          Content-Transfer-Encoding: 8bit 

(N) 8BITMIME to 7-bit gateway can convert DSN                       
     with that message/rfc822 subpart to:
          Content-type: message/rfc822
          Content-Transfer-Encoding: 7bit

Now Sender X is informed that recipient mailbox
is full or recipient address is mistyped.

( If UTF8SMTP addresses are used on original
  message, then conversion message/utf8smtp
  to message/rfc822 can fail and DSN is dropped,
  but on this scenario all addresses are ASCII. )

Thank you for your attention.

/ Kari Hurtta

PS.  With subject Terveisiä täältä
     someone can think that ASCII user 
     is using Internet cafe
     on his vacation trip.
     Even when this is case ASCII
     user can still access his 7-bit
     mailbox.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 21 May 2007 07:17:38 +0200

>All addresses used are ASCII

Here you have made the interesting assumption that the subject of the 
message will be represented in raw UTF-8 when the message is available for 
DSN creation.

This is contrary to the WG decision of "no upgrade within the network".

               Harald

---

From: Kari Hurtta <[email-masked]>
Date: Mon, 21 May 2007 09:51:12 +0300 (EEST)

>>> Harald Tveit Alvestrand
[ Charset UTF-8 unsupported, converting... ]
> This is contrary to the WG decision of "no upgrade within the network".

There is NO upgrade happened !!!
There is no any contrary with the 
WG decision of "no upgrade within the network".

MSA and mail client supports UTF8SMTP.

Read carefully, please.

Message submission path is full UTF8SMTP.

On first picture:

     ( Sender X )
                       normal mail (A)

Notice that key point between picture 1) and Picture 2)
is that on picture 1 MSA announces UTF8SMTP
and on picture 2 MSA do not announce UTF8SMTP to MUA.

On second picture:

     ( Sender X )
                       normal mail (E)

On both case MUA is UTF8SMTP capable.

It is MUA which decides is message sent as UTF8SMTP
or with MIME encoded words on Subject.

It is message store and return path where there
is no UTF8SMTP and 8BITMIME either available.

> > "use UTF8SMTP" switch on MUA. That just confuses users.

On my message with subject
    UTF8SMTP / 8BITMIME / ASCII -universe amd message/utf8smtp
      (Re: Not a good reason? (Re: Minutes from Prague (fwd)))      

I explain why it is likely scenario that there is no 8BITMIME.

/ Kari Hurtta

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 21 May 2007 13:44:11 +0200

You are right - I did not read carefully enough.

You have identified a message loss based on 3 assumptions:

- Forward path (including sender) is UTF8SMTP capable
- Reverse path is NOT UTF8SMTP capable
- The RFC 2045 requirement is upheld.

My source of confusion was because you used the term "ASCII user", which is 
defined in -framework as:

 An "ASCII user" (i) exclusively uses email addresses that contain
 ASCII characters only, and (ii) cannot generate recipient
 addresses that contain non-ASCII characters.

I missed the fact that you said the sender was UTF8SMTP capable.

That said, I still have trouble taking the last 2 of your 3 assumptions as 
given.

                     Harald

--On 21. mai 2007 09:51 +0300 Kari Hurtta <[email-masked]> wrote:

>> message will be represented in raw UTF-8 when the message is available
>> for  DSN creation.
>>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 14:15:14 +0100

> base agreeing to change this.

Will you PLEASE STOP introducing a Strawman, accusing me of advocating
that Strawman, and then shooting that Strawman down in flames. Of course
it is out of the question to change the current practice that a DSN is
sent with Return-Path:<>, and hence will be dropped silently if it fails
to be delivered. But since I have NEVER suggested such a change, what was
your point?

Please can we get back to the issue at hand (which apparently bears the
number #1485), and please can we address it on its technical merits.

I think the following summarizes the problem:

We are concerned with a UTF8SMTP message which contains 8bit (UTF-8) in
some of its headers, and likely contains 8bit in its body as well. For
whatever reason, a DSN including the full original message has to be
returned along the Return-Path.

The awkward case is where the route taken by the DSN an MTA that does not
support UTF8SMTP and a further MTA that does not support 8BITMIME.

There are three proposed methods to encapsulate the original message in
the DSN:

A. message/utf8smtp
B. application/message-utf8smtp
C. message/rfc822

Method C assumes an 'experimental extension' to RFC 204[56], together with
a matching downgrade mechanism to be defined in our 'downgrade' draft.

One may envisage the following types of MTA which may or may not currently
exist, classified according to their behaviour when receiving an 8BITMIME
message to be forwarded to a non-8BITMIME MTA:

1. 100% standards-compliant
          A. message/utf8smtp will be dropped, silently if Return-Path is <>
          B. application/message-utf8smtp will be converted to Q-P or Base64
          C. message/rfc822 (already downgraded to 8BITMIME) will have its
body
             converted to Q-P or Base64.

2. "EXPRESSLY OVERRIDDEN"
These are what Chris Newman, as author of out DSN draft, and the attendees
at Prague seem to have had in mind by suggesting that the "stupid"
"EXPRESSLY FORBIDDEN" in RFC2045 should be ignored, and MTAs "educated" to
treat unknown message types sensibly.
          A. message/utf8smtp will be converted to Q-P or Base64
          B. application/message-utf8smtp will be converted to Q-P or Base64
          C. message/rfc822 (already downgraded to 8BITMIME) will have its
body
             converted to Q-P or Base64.

3. "Just send 8bit" for unknown message types
          A. message/utf8smtp will be sent as-is, including 8bit in headers
             and bodies
          B. application/message-utf8smtp will be converted to Q-P or Base64
          C. message/rfc822 (already downgraded to 8BITMIME) will have its
body
             converted to Q-P or Base64.

4. "Just send 8bit" always
          A. message/utf8smtp will be sent as-is, including 8bit in headers
             and bodies
          B. application/message-utf8smtp will be sent as-is, including 8bit
             in headers and bodies
          C. message/rfc822 (already downgraded to 8BITMIME) will be sent
as-is,
             including 8bit in bodies.

5. Truncate everything to 7bit
          A. message/utf8smtp - both headers and bodies garbled
          B. applicatio/message-utf8smtp - both headers and bodies garbled
          C. message/rfc822 (already downgraded to 8BITMIME)  - bodies
garbled

The currently deployed base of MTAs, with which UTF8SMTP will have to
interoperate as best it can, doubtless includes examples of all five
types, and maybe others. It has been reported that:

     Earlier versions of exim did #5.
     Current exim and gmail do #4, and some people (DJB) openly advocate
this practice.
     Sendmail does #3.
     I am not aware of ANY that currently do either #1 or #2, but would
welcome reports of any that do. But what is clear is that a significant
proportion of the installed base exhibits behaviours #3 and #4.

The possible outcomes when a DSN is processed by the various MTAs are:

     DSN delivered intact
     8bit headers delivered, and hopefully displayed
     bodies garbled
     headers and bodies garbled
     Dropped silently, nothing delivered

So we can now prepare a chart of the outcomes in the various scenarios:

\ Encapsulation       A                    B                    C
   \
MTA type
         #1          dropped               intact               intact
         #2          intact                intact               intact
         #3          8bit h&b              intact               intact
         #4          8bit h&b              8bit h&b             8bit b
         #5          garbled h&b           garbled h&b          garbled b

It should be clear from that that Option A (message/utf8smtp) would be the
worst choice to make.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 14:21:05 +0100

> This is contrary to the WG decision of "no upgrade within the network".

I think you will find that, if you
    s/Ascii user X/UTF8SMTP user who uses ASCII-only addresses/, you will  
find that Kari has presented a valid scenario which needs proper technical  
consideration.

I would expect that such users will commonly be found amongst  
Scandinavians who are fortunate enough to possess names which can be  
expresses without the extra Scandinavian characters.

---

From: John C Klensin <[email-masked]>
Date: Mon, 21 May 2007 09:53:34 -0400

--On Monday, May 21, 2007 1:44 PM +0200 Harald Tveit Alvestrand 
<[email-masked]> wrote:

> My source of confusion was because you used the term "ASCII
> user", which is defined in -framework as:
>
> That said, I still have trouble taking the last 2 of your 3
> assumptions as given.

Let me try to say that a little bit more strongly.

It has been a property of the Internet's email system, at least 
since we switched over from FTP, that one can configure oneself 
into silly states in which mail won't go through and 
notifications will not get through either.  Even reverse-paths 
with explicit, forward-generated, source-routes were not 
sufficient to prevent those cases.    Some examples of this:

    (i) Even in the all-ASCII (no EAI facilities) world, it is
    possible to configure the MX records for a mail receiver so
    that one of the intermediate MX hosts will refuse to accept
    mail for the destination host or user.  Cases of this will
    continue to increase if "intermediate MX server must know
    the current mailbox list at the destination site" plans
    develop and race conditions occur.

    (ii) Any moves toward "reject or drop" (rather than "reject
    or bounce") increase the number of messages that will
    silently disappear.  No matter what precautions are taken,
    unless very fundamental changes are made to the SMTP model,
    cases remain in which a relay will either need to pass mail
    along for which neither the sender nor receiver can be
    adequately validated.   The most obvious case arises from
    "mailbox temporarily full" conditions, which a relay cannot
    know about no matter how much information it has about
    mailbox name validity.  Again, this is an issue even in the
    all-ASCII world.

    (iii) In the EAI-specific environment, configuring MX
    records so that a destination server that is EAI-capable and
    supports i18n mailboxes has intermediate servers that are
    not EAI-capable is, in the long term, dumb.  The WG has
    agreed that, as a transitional matter, we need to allow for
    those cases and that is what downgrading is all about, but
    relays do not get into the mail path by accident.

    (iv) Sending mail that contains UTF-8 content to a
    submission server that is not EAI-capable is also dumb (and
    ignores protocol requirements that the server announce the
    capability to receive and process such messages).   I would
    hope that such a submission server would simply reject such
    messages.

    (v) I suggest that sending mail that bears non-ASCII
    addresses (or other UTF-8 header content) from a host that
    cannot receive them -- or that is supported by relays that
    cannot receive them -- is significantly more dumb than
    (iii).  In addition, we have always understood that delivery
    of MDNs and similar messages is more fragile than delivery
    of forward-pointing messages: the formal handoff of
    responsibility and requirement that NDNs be generated if
    needed for forward-pointing messages does not exist for
    these reverse ones and, as Harald has reminded us, one must
    not even attempt generate an NDN for a message that arrives
    with a null return path.

For me, the situation here isn't that people can configure 
themselves into conditions in which a EAI NDN or the equivalent 
cannot be competently delivered.   Nor do I believe that weak, 
non-conforming, implementations will not get themselves into 
trouble and risk trashing any mail that passes through them.  I 
take both of those as given.    Instead, the question is how 
much more complex we should make the protocol to ward off these 
cases or reduce the risk of them causing problems.  My personal 
opinion is "very little".  YMMD.

Incidentally, for the specific case you listed in your note, 
unless I still don't understand it (and I have read your notes 
several times and very carefully),

> and on picture 2 MSA do not announce UTF8SMTP to MUA.

In case 2, the MUA MUST NOT send _any_ UTF-8 header material to 
the submission server.  If it does, it is so far outside the 
specification that we are not obligated to specify, or even 
worry about, what happens next.

And, if the MUA sends non-ASCII header information in encoded 
words, rather than UTF-8, and UTF-8 headers later appear 
anywhere in the system prior to final delivery, then we have 
"upgrade in the network".

What am I missing?

      john

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 17:44:18 +0100

>      Current exim and gmail do #4, and some people (DJB) openly advocate

s/gmail/qmail/

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 22 May 2007 16:07:55 +0100

On Mon, 21 May 2007 14:53:34 +0100, John C Klensin <[email-masked]> wrote:

>     (iii).

I would not put it as dumb as that. If you are on vacation, and have  
access only to an ASCII MUA cum submission agent, you might nevertheless  
try to send mail to a utf-8 address, or send it From: your own utf-8  
address. If it works well enough to get out and be relayed to a  
UTF8SMTP-capable agent somewhere, then it will probably go the whole way,  
and Good Luck to you. It if didn't, then don't expect the bounce to get  
back to you, but it was probably still worth the attempt.

>  In addition, we have always understood that delivery

However, a DSN consists of three parts, of which the first is textual and  
says something like "We were unable to deliver your message ...". Even if  
the third part, containing the full original message as message/utf8smtp  
or whatever else we decide, has got garbled beyond recognition, the DSN is  
still of some value to you, and certainly to silently drop the whole DSN  
just because that third part could not be delivered intact is simply  
counter-productive.
>
>
> For me, the situation here isn't that people can configure themselves  
> into conditions in which a EAI NDN or the equivalent cannot be  
> competently delivered.   Nor do I believe that weak, non-conforming,  
> implementations will not get themselves into trouble and risk trashing  
> any mail that passes through them.  I take both of those as given.     
> Instead, the question is how much more complex we should make the  
> protocol to ward off these cases or reduce the risk of them causing  
> problems.  My personal opinion is "very little".  YMMD.

However, the cost of specifying application/message-utf8smtp rather than  
message/utf8smtp in our DSN draft is zero, and even the cost of specifying  
a (downgraded if necessary) message/rfc822 is not large given that we  
already propose to allow (and downgrade) MIME body-part headers such as  
Content-Description. Once you have written the code to do that, extending  
that code to do message/rfc822 is quite simple.

> In case 2, the MUA MUST NOT send _any_ UTF-8 header material to the  
> submission server.  If it does, it is so far outside the specification  
> that we are not obligated to specify, or even worry about, what happens  
> next.

I think you will find that Kari's Case 2 sent the Subject RFC2047-encoded,  
for just the reaon you gave, and consequently its DSN was returned to the  
sender without problem. It was his Case 1 that got the DSN silently  
dropped.

---

From: Kari Hurtta <[email-masked]>
Date: 23 May 2007 08:27:42 +0300

What you missed ?

You missed that on Picture 2 MUA is sending Subject with using
MIME encoded words on Subject: header field.  And there is no
"upgrade in the network" anywhere.

In all pictures MUA is assumed to be same. 
Changed items are:
   Picture 1 / 2   :   MSA  UTF8SMTP capable or not   
   Picture 3 and 4 :   Specification changed 
( Picture 1 was where DSN on lost. On all other cases DSN
  is not lost. )

In all cases Subject what user types is Terveisiä täältä

MUA selects is Subject (and message) send with UTF8SMTP
or is it encoded with mime encoded words based on fact
that is MSA announcing UTF8SMTP on it's EHLO reponse or not.

There is no GUI on MUA where this is selected -- that is just
protocol level negation.

( It is not reasonable for MUA try look when sending message
  that is message store for return path UTF8SMTP capable or not. 
  I say that format decision is based only from SMTP negation 
  with MSA. )

-- start quote -------------------------------------------------
2) Case: MSA is not UTF8SMTP capable 

     ( Sender X )
                       normal mail (E)

On case 1 it was because  MSA  was UTf8SMTP capable
and message/utf8smtp was used outside of 
"UTF8SMTP universe".
-- end quote --------------------------------------------------

I suggest that readers look pictures again and not make
conclusions from Harald  Alvestrand's or John Klensin's
messages. On both cases significant details was missed.

John C Klensin <[email-masked]> writes:

> It has been a property of the Internet's email system, at least since
> we switched over from FTP, that one can configure oneself into silly
> states in which mail won't go through and notifications will not get
> through either.  Even reverse-paths with explicit, forward-generated,
> source-routes were not sufficient to prevent those cases.    Some
> examples of this:
> 

> Incidentally, for the specific case you listed in your note, unless I
> still don't understand it (and I have read your notes several times
> and very carefully),
> 
> that we are not obligated to specify, or even worry about, what
> happens next.
> 
> And, if the MUA sends non-ASCII header information in encoded words,
> rather than UTF-8, and UTF-8 headers later appear anywhere in the
> system prior to final delivery, then we have "upgrade in the network".
> 
> What am I missing?
> 
>       john