Subject: [EAI] ISSUE: smtpext-08 2.2 erroneous MUSTard

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 24 Sep 2007 12:18:51 +0200

| If the UTF8SMTP SMTP extension is not offered by the Server, the SMTP
| client MUST NOT transmit an internationalized address and MUST NOT
| transmit a mail message which contains internationalized mail headers
| [EAI-utf8header] at any level within its MIME structure.

It MUST NOT transmit a message with an I18N header.  As soon as that's
clear the object is by definition a message/rfc822, and its internal
MIME structure isn't the business of relaying MTAs or this I-D.

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 24 Sep 2007 13:22:45 +0200

IMO belonging to the same issue, 2.6 states:

| Since there is no ESMTP parameter which tells whether the
| message is UTF8SMTP message, SMTP server needs to parse
| all message header fields and MIME header fields in the
| message body to discover which messages are UTF8SMTP.

Editorial: "a UTF8SMTP message" and "an SMTP server".
Technical: It needs only to parse _the_ message header.

The internal MIME structure of the body is irrelevant for
relaying MTAs.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 24 Sep 2007 14:04:19 +0100

> MIME structure isn't the business of relaying MTAs or this I-D.

No, that is not enough. Suppose the message has all-ascii addresses  
throughout, and everything in the top-level header is pure ASCII, but  
suppose that the body is a multipart and one of the multiparts has a  
Content-Description written in Chinese using raw UTF-8.

Then that message is a global/UTF8SMTP message and must not be transmitted  
outside the UTF8SMTP world without downgrading.

The existing wording makes this perfectly clear (and makes it clear that a  
scan through the full body is always necessary to check for such cases).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 24 Sep 2007 23:28:04 +0200

> Content-Description written in Chinese using raw UTF-8.

The wonders of 8BITMIME or even binary mail bodies aren't relevant
for the UTF8SMTP ESMTP extension, they're discussed elsewhere.

Besides MIME 1.0 doesn't allow raw UTF-8 in Content-* header fields,
so who knows what that is you're talking about, maybe it's MIME 2.0.

Relaying MTAs should look for trouble in the body only on a "need to
know" basis.  They don't need to know MIME 2.0 or say yenc to support
internationalized addresses.

> Then that message is a global/UTF8SMTP message and must not be
> transmitted outside the UTF8SMTP world without downgrading.

If this global/UTF8SMTP message format is incompatible with known
8BITMIME or binary bodies it has to indicate this in its header,
maybe using MIME-Version: <UTF-8 for Arabic number 2>.

But that has nothing to do with internationalized addresses and the
goal of the EAI experiment.  Unless the desired outcome is "FAIL",
and that would be sneaky.  The EAI idea is interesting enough to
deserve a fair chance, it must not be burdened with MIME 2.0.

Besides I don't believe for a second that message/global could be
limited to the "UTF8SMTP world" (i.e. to participants of the EAI
experiment), anybody can use it as part of a message/rfc822, or
FWIW as part of message/news, everywhere.

For crying out loud, the smtpext-08 "security considerations" are
*empty*.  They don't mention an "UTF8SMTP world" or "sandbox" or
other fantastic stories not limited to MIME 2.0.
 
 Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 25 Sep 2007 09:45:09 +0200

> MIME structure isn't the business of relaying MTAs or this I-D.

Technical disagreement.

If I forward a Message/Global, without a C-T-E, and using UTF-8 characters 
in the headers, the message containing the forwarded message is a 
Message/Global, not a message/rfc822, no matter what I put into the headers 
of the outermost message.

If there is a C-T-E on the message/global, the message is (as far as I can 
see, and if the current resolution on extending MIME is accepted) a 
message/rfc822.

But the issue is real, and will be tracked; I'm still not sure whether it 
is the same issue number as Charles' protest against the WG decision on 
relaxing the restriction in MIME or not.

             Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 25 Sep 2007 12:46:08 +0100

> so who knows what that is you're talking about, maybe it's MIME 2.0.

We had this discussion way back when we got rid of the Header-Type header.  
I disagreed with that, but the consensus was against me.

So the situation now is that our documents extend what is allowed in MIME  
1.0, but only for use within the UTF8SMTP universe. So when you want to  
pass stuff out of that universe you have to detect that these extensions  
have been used and downgrade (or reject) accordingly. That is what we all  
agreed some months ago, and actually it is not that hard to detect it  
since current MTAs already do similar tests when passing stuff into the  
Non-8BITMIME world.

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 25 Sep 2007 14:52:01 +0200

Harald Tveit Alvestrand wrote:
 
> If I forward a Message/Global, without a C-T-E, and using UTF-8
> characters in the headers, the message containing the forwarded
> message is a Message/Global, not a message/rfc822, no matter what
> I put into the headers of the outermost message.

I'd say your object is a message/rfc822 containing a message/global
part, and it needs CTE: 8bit, e.g.

| From: [email-masked]
| To: [email-masked]
| Date: Tue, 25 Sep 2007 09:45:09 +0200
| Subject: Loo what he said
| MIME-Version: 1.0
| Content-Type: message/global
| Content-Transfer-Encoding: 8bit
|
| From: [email-masked]
| To: [email-masked]
| Date: Mon, 24 Sep 2007 12:34:56 -0000
| Subject: äöü (pls consider this as UTF-8 example)
|
| Body of forwarded message/global (ASCII example)

It's the same idea if this message/global is a part of a 
multipart/message, just add the boundaries.

Without the CTE: 8bit in the "outer" header you'd send binary 
garbage, i.e. raw 8bit within a 7bit CTE.  

> If there is a C-T-E on the message/global, the message is
> (as far as I can see, and if the current resolution on 
> extending MIME is accepted) a message/rfc822.

Are you really talking about "forwarding" as sending the
original message/global as part of a containing message ?

The CTE of the parts determines what you can use as CTE of
the "outer" container, if all parts are 7bit, QP, or B64,
then the container can be 7bit.  If at least one part is
8bit the container needs to be 8bit, unless you can replace
all "inner" 8bit by say QP or B64.  

I'd say that a message/* can't be B64 or QP, but that's
another issue.  Let's assume a message/global can be sent
with CTE B64 or QP in the sending container.  Then the
content (header plus body) is pure ASCII, and the "outer"
container is a perfect message/rfc822 ready to go to a
7bit MTA.  In my example above replace "8bit" by "QP",
and fix the example Subject to get QP-encoded umlauts.

> I'm still not sure whether it is the same issue number
> as Charles' protest against the WG decision on relaxing
> the restriction in MIME or not.

IMO it's not the "EXPRESSLY FORBIDDEN" issue:  We could
replace message/global 8bit by application/message 8bit
in my example, and I'd still say that the complete object
is a message/rfc822 containing "something".

 Frank

---

From: SM <[email-masked]>
Date: Tue, 25 Sep 2007 06:51:24 -0700

At 00:45 25-09-2007, Harald Tveit Alvestrand wrote:
Technical disagreement.

>message is a Message/Global, not a message/rfc822, no matter what I 
>put into the headers of the outermost message.
>
>If there is a C-T-E on the message/global, the message is (as far as 
>I can see, and if the current resolution on extending MIME is 
>accepted) a message/rfc822.

No, it is still not a message/rfc822 except for the purposes of this 
experiment.  Any extension or relaxation of MIME requirements apply 
to EAI only.  Other transport mechanisms may treat a message/rfc822 
as defined in Section 6.4 of RFC 2045.

Regards,
-sm

---

From: SM <[email-masked]>
Date: Tue, 25 Sep 2007 07:31:39 -0700

>since current MTAs already do similar tests when passing stuff into the

What we have is MIME 2.0 without calling it that.  The changes aren't 
restricted to 8-bit content only.  The MIME changes will be passed to 
the non-UTF8SMTP universe and may affects parsers.

Regards,
-sm

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 25 Sep 2007 17:37:22 +0200

Charles Lindsey wrote:
 
> We had this discussion way back when we got rid of the Header-Type
> header. I disagreed with that, but the consensus was against me.

Yes, I recall that decision.  IIRC your idea was *NOT* to emulate a
kind of "MIME-Version: 2.0", it was something else.  Maybe I missed
that it's in essence the same idea as "MIME-Version: 2.0", affecting
not only the "outer" header, but also all "inner" MIME Content-*
headers.

If the latter is the case we'd still disagree about the purpose of
this stage of the EAI experiment.  I dare not screw with MIME 1.0
unless Ned, Keith, or the IAB say that this could be fun ;-)

> So the situation now is that our documents extend what is allowed
> in MIME 1.0, but only for use within the UTF8SMTP universe.

IMO there's no such thing as an "UTF8SMTP sandbox".  As soon as the
message/global type exists, it can be put into multipart/message
or similar MIME containers everywhere, and these containers can be
transported with any protocol that can transport MIME 1.0 messages.

Http, nntp, and 8bit smtp can handle a Content-Type: message/global
in their header introducing a body with a raw message/global.  Only
7bit smtp can't do this, because it can't transport raw 8bit data
in the body.

I'm too lazy to look, did http register its own message/http MIME
type, with charset Latin-1 for its header field bodies ?  I guess
that exists for Web-by-mail access, and it should be no big problem
for 8bit smtp.     

Let's say I've got a "message/http" containing an ASCII text/html
document in the body.  And let's say the "message/http" header
contains some raw Latin-1 characters.  Now I want to forward this
object within a message/rfc822 over 7bit smtp.

How does this work, does message/http permit CTE B64, in spite of
the "expressly forbidden" rule ?  Or is there an application/http
type for this purpose ?  This can't be a "mission impossible", or
can it ?

One thing is sure, the complete object transported by smtp is then
still a message/rfc822, no matter if it contains message/global,
message/http, or application/foo parts.  And FWIW it's also still
MIME 1.0 without experimental updates of RFC 2045.

 Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 26 Sep 2007 17:36:12 +0100

> the non-UTF8SMTP universe and may affects parsers.

Not if the people who implement our extensions stick rigorously to what we
have said.

We had this discussion when we abolished the Header-Type header. The
argument was that we could not trust people to insert Header-Type: UTF-8
in all the cases where it was strictly needed, and henceforth the ONLY way
to determine whether a given message was a global/UTF8SMTP message would
be to scan through the whole message looking for trouble. In defence of
this, it was pointed out that MTAs routinely have to do this anyway for
other reasons.

That was the decision we took, and the consequence was that we need to
specify very carefully what is to be done. Which we have, except for some
uncertainty with the message/global -  but there is a separate ISSUE
regarding that, so do not let us confuse the two ISSUEs. For this
discussion, just assume that message/global works "by magic" (or will
somehow be fixed to do so), and leave the details of that to that other
ISSUE.

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 27 Sep 2007 15:03:11 +0200

Merged with comment on 2.2, issue #1505.

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 27 Sep 2007 15:04:15 +0200

I believe the WG has previously had consensus that the entire message 
had to be scanned to determine if the UTF8SMTP extension was required, 
and that this was OK.

However, it's worth re-checking that consensus.

Issue #1505 assigned.