Subject: Re: [EAI] Proposal for decision, approach for downgrading in UTF8SMTP

From: John C Klensin <[email-masked]>
Date: Mon, 09 Oct 2006 16:03:05 -0400

--On Monday, October 09, 2006 10:04 +0300 Kari Hurtta 
<[email-masked]> wrote:

>> > Is there differences between these two strategies
>> > (encapsulation strategy   and a conversion strategy) on
>> > what situations these strategies are  possible when
>> > downgrading is required?
>> I'm sorry, but I can't parse this question. Can you try again?
>
> Lets assume that dowgrade is required for message
>
> Case
> 	1) Select encapsulation strategy
>
> 	   When this is succeed and when bounce is required ?
>
> Case
>
>         2) Select conversion strategy
>
>            When this is succeed and when bounce is required ?

Let me take a shot at reviewing this...

I hope we are in agreement by now that offering the possibility 
of both as alternatives in the specification would be a bad 
idea.
One must be required (on either a "do it or bounce" basis, 
similar to 8BITMIME) and the other, at most, permitted as 
optional.

Beyond that, each requires a fair amount of work relative to 
bouncing.  Some implementations will do the work, with whatever 
option is selected.  Others won't.  It is not possible to 
estimate accurately how many would do either.  My own guess is 
that, whatever choice we make, it won't have a very significant 
impact on the number of implementations that will choose to 
support downgrading.  That is just a guess, but I don't think a 
decision can be made on projections of "(1) is more likely to be 
fully supported than (2)" or "(2) is more likely to be fully 
supported than (1)".

So there is a "bounce" case for either option that results from 
"no downgrade implementation".  Now,  the questions seem to 
hinge on what you mean by "success".  My criterion for "success" 
is "be deliverable with little or no information loss and have a 
good chance of figuring out how to reply, even if that requires 
out of band communications and information".   Harald adds 
"being able to reply automatically".    By Harald's criterion, 
encapsulation without _also_ doing most of the work of 
conversion is a non-starter.  By mine, one might still consider 
encapsulation, but is would not be an easy sell.

Does that answer your question?

     john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Kari Hurtta <[email-masked]>
Date: 10 Oct 2006 08:12:07 +0300

John C Klensin <[email-masked]> writes in gmane.ietf.ima:

> I hope we are in agreement by now that offering the possibility of
> both as alternatives in the specification would be a bad idea.
> One must be required (on either a "do it or bounce" basis, similar to
> 8BITMIME) and the other, at most, permitted as optional.

> So there is a "bounce" case for either option that results from "no
> downgrade implementation".  Now,  the questions seem to hinge on what
> you mean by "success".  My criterion for "success" is "be deliverable
> with little or no information loss and have a good chance of figuring
> out how to reply, even if that requires out of band communications and
> information".   Harald adds "being able to reply automatically".    By
> Harald's criterion, encapsulation without _also_ doing most of the
> work of conversion is a non-starter.  By mine, one might still
> consider encapsulation, but is would not be an easy sell.
> 

I was thinking requiring "conversion strategy", but I was looking cases
where "encapsulation strategy" may succeed (when "conversion strategy"
may require bounce.)

If these cases ("conversion strategy" bounces, "encapsulation strategy" 
succees), are many, then "encapsulation strategy" should have permitted as
optional.

( Although I suspect that most of implementions will generate mime
  encoding words (RFC 2047) incorrectly.  Correctly parsing mail 
  headers seems hard problem. So that RFC 2047 encoding is applied
  syntaxtically correct places. 

  Generally MTAs do not have parses which handles all syntax. Just 
  some parser for address headers (not for other headers.)

  That using RFC 2047 (syntaxtically correct places) was required on 
  "conversion strategy".
)

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 10 Oct 2006 09:27:50 +0300

>   That using RFC 2047 (syntaxtically correct places) was required on 

Encapsulation is required for some headers.
That is needed for future compatibility.

Otherwise MTAs is required to know syntax of all headers
(including these standards which are not yet written.)

There must be encapsulation (as opposite of "conversion strategy")
for headers which MTA do not know.

I suggest that conversion is limited to RFC 2822 headers and syntax.

All other headers (and headers with obsolete syntax on  RFC 2822)
are encapsulated.  (If on result have missing required headers,
then there is bounce option.)

Or perhaps it can be considered to limited more:

  -  Only convert
     Subject:
     From:
     To:
     Cc:
     Reply-To:

     and correspond Resend-* headers.

  - All other headers are encapsulated (if downgrading is required)

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Oct 2006 03:45:32 -0400

Kari,

Before I try to comment on this, could you be absolutely clear
about what you mean by "encapsulating headers"?  I know, more or
less, how to encapsulate a body part.  I can probably devise
ways to encapsulate an entire header set.  But it seems to me
that you are asking for/ requiring encapsulation of individual
headers.

    john

--On Tuesday, 10 October, 2006 09:27 +0300 Kari Hurtta
<[email-masked]> wrote:

>> ( Although I suspect that most of implementions will generate
>> mime encoding words (RFC 2047) incorrectly.  Correctly
>>   parsing mail  headers seems hard problem. So that RFC 2047
>>   encoding is applied syntaxtically correct places. 
>> 
>>   Generally MTAs do not have parses which handles all syntax.
>>   Just  some parser for address headers (not for other
>>   headers.)
>> 
>>   That using RFC 2047 (syntaxtically correct places) was
>>   required on  "conversion strategy".
>> )
> 
> There must be encapsulation (as opposite of "conversion
> strategy") for headers which MTA do not know.
> 
> All other headers (and headers with obsolete syntax on  RFC
> 2822) are encapsulated.  (If on result have missing required
> headers, then there is bounce option.)
>

---

From: Kari Hurtta <[email-masked]>
Date: Tue, 10 Oct 2006 11:38:08 +0300 (EEST)

>>> John C Klensin

I mean invidual headers, because all is not possible to downgraded
that way that they are usefull.

There is two possibilities.

Either put these headers to new body part, or put them
to some header

  Encapsulated:  {header-name}: {RFC 2047 data}

where {RFC 2047 data} is treated as unstructured.

( That was something also on Charles Lindsey's
  "Comments on draft-ietf-eai-downgrade-02". )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: Tue, 10 Oct 2006 13:04:20 +0300 (EEST)

>>> Kari Hurtta

Or perhaps correct terminology is:  invidual header fields

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 10 Oct 2006 15:11:03 +0100

> (including these standards which are not yet written.)

That is a flaw in the design of RFC 2047 (and one that Keith steadfastly  
refuses to recognize). But it is not our job to fix that (though in the  
long run UTF8smtp will render the use of RFC 2047 unnecessary).
> for headers which MTA do not know.

Yes you are correct there, but the downgrade document already provides  
that, in the form of the Downgraded header. What it doesn't do is to make  
it clear that mechanism MUST be used for all headers for which there is no  
special provision.

It _does_ make special provision for most of the headers which are  
necessary for propagation, though it could usefully add a few more  
(notably the List-* headers).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Kari Hurtta <[email-masked]>
Date: 10 Oct 2006 19:03:24 +0300

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> That is a flaw in the design of RFC 2047 (and one that Keith
> steadfastly  refuses to recognize). But it is not our job to fix that
> (though in the  long run UTF8smtp will render the use of RFC 2047
> unnecessary).

I do not suggest that we try fix RFC 2047

> that, in the form of the Downgraded header. What it doesn't do is to
> make  it clear that mechanism MUST be used for all headers for which
> there is no  special provision.

It (draft-ietf-eai-downgrade-02.txt) says
            +  If the header is an address header which is described in
            Section 5.1.1,
            -  Preserve the header in 'Downgraded' header.

If I read correctly, that provides Downgraded -header ONLY for address header.
(I think that you mentioned that already.)

For other headers it says:

         +  The other header case, encode the header by [RFC2047] with
            UTF-8 tag.

That of course can not be done, when syntax of header is not known.

I say that it must provide Downgraded -header for other headers
(when syntax is not known).

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 10 Oct 2006 20:09:52 +0300

> (I think that you mentioned that already.)
  ^ this sentence should refer other header case below.    -----------------+
                                                                            |
> For other headers it says:                                                |
>                                                                           |
>          +  The other header case, encode the header by [RFC2047] with    |
>             UTF-8 tag.                                                    |
>                                                                           |
> That of course can not be done, when syntax of header is not known.    <--+
> 

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 10 Oct 2006 19:17:03 +0200

Kari Hurtta wrote:

  [quoting Charles]
>> steadfastly  refuses to recognize).

I'm not aware of any flaws in RFC 2047 apart from its minimalistic
security considerations, what's that about ?

> (draft-ietf-eai-downgrade-02.txt) says
>       +  If the header is an address header which is described
>          in Section 5.1.1,

> If I read correctly, that provides Downgraded -header ONLY for
[...]

>       +  The other header case, encode the header by [RFC2047]
>          with UTF-8 tag.

> That of course can not be done, when syntax of header is not known.

If the syntax isn't known the default is <unstructured>, and the
gateway (UTF8SMTP to SMTP or vice versa is a gateway) could handle
it like say a Subject: header field with UTF-8.

Maybe UTF8SMTP could register header fields (3864 registry) where
supporting the header field syntax is critical for SMTP-gateways.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 10 Oct 2006 19:59:49 +0100

> That of course can not be done, when syntax of header is not known.

Exactly.
> (when syntax is not known).

And there I agree with you entirely. I have been asking for this to be  
included in the downgrade document for a long time.

---

From: Kari Hurtta <[email-masked]>
Date: 10 Oct 2006 22:10:04 +0300

> it like say a Subject: header field with UTF-8.

I disagree on here.

If MTA does not know syntax, MUA may still know syntax.

And syntax what MUA is using may give completely different parsing
rules.

If MUA think that header is structured, it must _first_ parse
structure and after that do RFC 2047 decoding on
    comment  ('ctext')
    phrase   (*word)

/ Kari Hurtta

Quote from RFC 2047:

6.2. Display of 'encoded-word's

   Any 'encoded-word's so recognized are decoded, and if possible, the
   resulting unencoded text is displayed in the original character set.

   NOTE: Decoding and display of encoded-words occurs *after* a
   structured field body is parsed into tokens.  It is therefore
   possible to hide 'special' characters in encoded-words which, when
   displayed, will be indistinguishable from 'special' characters in the
   surrounding text.  For this and other reasons, it is NOT generally
   possible to translate a message header containing 'encoded-word's to
   an unencoded form which can be parsed by an RFC 822 mail reader.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 10 Oct 2006 23:09:10 +0200

> If MTA does not know syntax, MUA may still know syntax.

The gateway could note it as Downgraded-unknown-<name>: =?UTF-8?B?etc?=
where <name> is the original header field name, and Downgraded-unknown-
indicates "garbage in garbage out" processing.

Or maybe the original field <name> should appear behind the colon, e.g.
Downgraded-unknown: <name>: =?UTF-8?Q?etc?=

The next downgrade draft could profit from an example how that's all
supposed to work.

> structure and after that do RFC 2047 decoding

ACK, using the same header field name won't fly.  A more radical idea
could be to transform the complete message/utf8smtp as is into the body
(= MIME message/utf8smtp part) of a forwarded mail, and only keep the
downgraded "vital" header fields (anything in 2822 + the Content header
fields for a message/utf8smtp).

In some situations like a mailing list on the downgraded route it won't
be possible to reliably upgrade it later.  Maybe the MUA should figure
this out.  Most MUAs can handle message/rfc822 today, they will learn
how to handle message/utf8smtp (it's almost identical).

Frank

---

From: "Kari Hurtta" <[email-masked]>
Date: 11 Oct 2006 06:50:52 +0300

> fields for a message/utf8smtp).

In here there is no disagreement (except that message/* subparts do not work)

Putting unknown header fields _only_ to Downgraded: -header fields or
to MIME part are both using "encapsulation strategy".

Using same header field name and trying to use RFC 2047 without syntax is
using "conversion strategy" (which do not work on here).
 
( I suggest text/utf8smtp part instead of message/utf8smtp part.
  Reason why message/* parts do not work is mentioned on my article
  with title: "downgrade must able to fail"

  Another reason is that RFC 2045 forbids other than  
  Content-Transfer-Encoding "7bit", "8bit", and "binary" on message/*
  subparts. That means that 8BITMIME downgrading of message/utf8smtp
  is not possible.)

/ Kari Hurtta

RFC 2045 quote:

   Certain Content-Transfer-Encoding values may only be used on certain
   media types.  In particular, it is EXPRESSLY FORBIDDEN to use any
   encodings other than "7bit", "8bit", or "binary" with any composite
   media type, i.e. one that recursively includes other Content-Type
   fields.  Currently the only composite media types are "multipart" and
   "message".  All encodings that are desired for bodies of type
   multipart or message must be done at the innermost level, by encoding
   the actual body that needs to be encoded.

---

From: Kari Hurtta <[email-masked]>
Date: 11 Oct 2006 07:15:57 +0300

>   subparts. That means that 8BITMIME downgrading of message/utf8smtp

That MIME subparts idea is quite complex. It needs little more than
just one MIME subparts.

Example:

=====================================
{received-header-fields}        | original header
{ascii-header-fields}           |  
{utf-8 header-fields}           |
MIME-Version: 1.0               |
{content-header-fields}         |

{body}
=====================================

becomes =>

=====================================
{received-header-fields}           |
{ascii-header-fields}              | converted header
{converted-header-fields}          |
MIME-Version: 1.0                  |
Content-Transfer-Encoding: 8bit    |  or "binary" if original was "binary"
Content-Type: message/utf8message; |
  boundary="{new-boundary}"        |

--{new-boundary}                   | Header for new body part
Content-Transfer-Encoding: 8bit    |
Content-Type: text/utf8smtp;       |
  charset=utf-8                    |

{original-utf-8 header-fields}

--{new-boundary}                   | Content header fields from
{content-header-fields}            | original header 

{original body}

--{new-boundary}--
=====================================

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 11 Oct 2006 07:19:29 +0300

> That MIME subparts idea is quite complex. It needs little more than

   this is mistake. "message/utf8message" should be
   "multipart/utf8message"

/ Kari Hurtta

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 11 Oct 2006 00:10:40 -0700

Kari,

please be specific - I can't tell what you are thinking about.

Please use the language from draft-ietf-eai-scenarios, and describe a 
specific scenario in which you think that "encapsulation strategy" may 
succeed, while "conversion strategy" requires a bounce, and explain why.

Note: Encapsulation by the sender will be allowed as soon as someone 
delivers on the promise to write up a draft on what the MIME part for an 
encapsulation should look like - but this isn't in the critical path of 
this effort....

                               Harald

---

From: Kari Hurtta <[email-masked]>
Date: Wed, 11 Oct 2006 10:34:09 +0300 (EEST)

>>> Harald Alvestrand

At least now it look that "conversion strategy" requires knowledge 
about syntax of header fields.

Problem seems to be that converting MTA is required to know all
possible standards which define syntax of mail header fields.

Of course that does not prevent conversion of RFC 2822 header fields
or any other standard header fields, when it is given list of standards,
which syntax must be supported on conversion.

Encapsulation  can be done without without knowledge. Encapsulation 
(of unknown header field) can be either to new header field or to new 
MIME part.

There may be some other cases also. These was what I was asking.
I do not know what are these cases. I was asking about that.

( Also conversion may be impossible if there is allowed UTF-8
  on some places, to where conversion is not possible. For
  example mime-encoded-words or some other currently existing
  standard does not define ascii compatible encoding. If
  new encoding is defined for missing parts, it is equivalent
  of encapsulation. )

I try look draft-ietf-eai-scenarios.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 11 Oct 2006 10:38:22 +0200

>    this is mistake. "message/utf8message" should be

Yes, you want a boundary.

Okay, RFC 2045 says EXPRESSLY FORBIDDEN about base64 for message/*.
But it does allow 8bit as needed by a hypothetical message/utf8smtp.
With that you'd get...

===============================
converted-header-fields
MIME-Version: 1.0                
Content-Transfer-Encoding: 8bit
Content-Type: message/utf8smtp

original-utf-8 header-fields

original body
===============================

...and we're in trouble if we hit a 7bit MTA.  If all else fails 
there's application/utf8smtp and base64.  After a short look into
RFC 1495 + 1652:  Harald and John can do gateway-giggle-tests. :-)

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 11 Oct 2006 12:48:19 +0300

> But it does allow 8bit as needed by a hypothetical message/utf8smtp.

I was essentially suggesting multipart/utf8message for top level,
and put utf-header to inside of text/utf8smtp or application/utf8smtp.
I think that text/* is more friendly.

It can be used multipart/mixed instead of  ultipart/utf8message,
but I think that multipart/utf8message is usefull for upgrading,
and MUAs which will implement that standard.

But really challenge is handling of Content-* headers, which
includes utf-8 data.  That encapulation not really fly on here
(result is messy when done recursively for mime parts and
content-type and content-transfer-encoding need different 
handling).

Conversion is doable for content-type and content-disposition.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 11 Oct 2006 10:54:48 +0100

> security considerations, what's that about ?

It requires that every agent (especially agents generating encoded-words)  
knows which headers are structured and which are unstructured. So if  
somebody invents a new header and gets it approved by the IETF as a  
Proposed Standard, then all implementations that encode or decode RFC 2047  
stuff magically become non-compliant overnight.

Hence the rule, that Kari quoted, that any protocol action brought about  
in any MUA by any header must be applied _before_ the decoding is done  
(and the decoding is only to be done for human intelligibility).

Hence if we prescribe the use of the Downgraded header (as Kari and I  
propose), and if that header (in its original UTF8smtp form) is intended  
to have some protocol effect, then the 'upgrading' operation must be  
performed _before_ protocol actions are taken (i.e. before it gets to the  
MUA, or as the very first action taken by the MUA). Yes, this needs  
pointing out somewhere, but fortunately most new headers that get invented  
don't lead to protocol actions.

> If the syntax isn't known the default is <unstructured>,

No it isn't, and it cannot be. For the sending agent may then treat is as  
unstructured and the receiving agent as structured, and vice versa  
(becuase one has been upgraded to the latest standard and the other has  
not).

But this is a situation that present implementatiosn have to work around  
somehow, and the result is that few of them are strictly compliant with  
RFC 2047.

---

From: Kari Hurtta <[email-masked]>
Date: 11 Oct 2006 14:52:50 +0300

> content-type and content-transfer-encoding need different 

If there is 
          {utf-8-content-header-fields}

          {bodypart}

that seems that then it is needed to use application/utf8smtp
and replace {utf-8-content-header-fields} with application/utf8smtp.
That makes it

          Content-Type: application/utf8smtp 
          Content-Transfer-Encoding: 8bit       |  or "binary" if original was "binary"

          {utf-8-content-header-fields}

          {bodypart}

That essentially collapses mime structure starting from
{utf-8-content-headers} to one MIME subpart.

Otherwise structure 
      multipart/utf8message
          text/utf8smtp
         {original-type}

may be used.

So if originally there

      {utf-8-header-fields}
      Content-Type: multipart/mixed; boundary=....
      Content-Transfer-Encoding: 8bit 

      --....
      Content-Type: text/plain

      something
      --....
      {utf-8-content-header-fields}

      {bodypart}
      --....--

that makes it

      {converted-header-fields}
      Content-Type: multipart/utf8message=boundary=xxxx
      Content-Transfer-Encoding: 8bit

      --xxxx
      Content-Type: text/utf8smtp; charset=utf-8
      Content-Transfer-Encoding: 8bit

      {utf-8-header-fields}

       --xxxx
      Content-Type: multipart/mixed; boundary=....
      Content-Transfer-Encoding: 8bit 

      --....
      Content-Type: text/plain

      something
      --....
      Content-Type: application/utf8smtp 
      Content-Transfer-Encoding: 8bit          

      {utf-8-content-header-fields}

      {bodypart}
      --....--

      --xxxx--

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 11 Oct 2006 18:07:27 +0300

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> It requires that every agent (especially agents generating
> encoded-words)  knows which headers are structured and which are
> unstructured. So if  somebody invents a new header and gets it
> approved by the IETF as a  Proposed Standard, then all implementations
> that encode or decode RFC 2047  stuff magically become non-compliant
> overnight.

Well, if MUA does not implement some standard, it either do not generate 
headers of that standard.   So I do not think that this happends on MUAs
which are generating encoded-words.             (or what I are missed?)

Yes, technically that (non-compliant) happens on receiving MUAs.

> Hence the rule, that Kari quoted, that any protocol action brought
> about  in any MUA by any header must be applied _before_ the decoding
> is done  (and the decoding is only to be done for human
> intelligibility).

Yes.
 
> propose), and if that header (in its original UTF8smtp form) is
> intended  to have some protocol effect, then the 'upgrading' operation
> must be  performed _before_ protocol actions are taken (i.e. before it
> gets to the  MUA, or as the very first action taken by the MUA). Yes,
> this needs  pointing out somewhere, but fortunately most new headers
> that get invented  don't lead to protocol actions.

Yes. That happens with Downgraded header or with any new MIME subpart,
if that is used for encapsulation. 

Encapsulation requires 'upgrading' operation before protocol actions are taken.

There is no that problem with conversion, but unfortunately conversion to
existing (and widely implemented) protocols is not always possible.

> No it isn't, and it cannot be. For the sending agent may then treat is
> as  unstructured and the receiving agent as structured, and vice versa
> (becuase one has been upgraded to the latest standard and the other
> has  not).

Yes.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 11 Oct 2006 18:20:16 +0300

Harald Alvestrand <[email-masked]> writes in gmane.ietf.ima:

> delivers on the promise to write up a draft on what the MIME part for
> an encapsulation should look like - but this isn't in the critical
> path of this effort....
> 
>                                Harald
> 

 A, an i18mail user, sends to X, an ascii user.

 Message contains protocol elements for which converting gateway is not
 aware.

 Encapsulation does not require knowing these protocol elements.

 Converting gateway need to know these protocol elements, otherwise it
 is not able convert them.   If there is no permission to drop these
 elements, it is required to bounce message.

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 Oct 2006 13:01:52 -0400

--On Wednesday, 11 October, 2006 18:20 +0300 Kari Hurtta
<[email-masked]> wrote:

>  Message contains protocol elements for which converting
> gateway is not  aware.
> 
>  Converting gateway need to know these protocol elements,
> otherwise it  is not able convert them.   If there is no
> permission to drop these  elements, it is required to bounce
> message.

Kari,

There is no question that there is a problem in this area.
Oddly, it isn't an i18nemail problem, although i18n may make it
more obvious.

The problem derives from the fact that one can invent new
headers at will _and_ that there is no way to tell a header
whose interpretation is expected to be mandatory for
interpretation of the message and one whose interpretation is
strictly optional.  We have dealt with that by pretending that
_all_ unrecognized headers are optional, but we do so knowing
that assumption is neither completely reasonable nor completely
safe.  

For example, assume that a plain-ASCII (821/822 conformant) user
sends a message to A, a plain-ASCII (2821/2822 conformant) user.
Somewhere between the two there is a protective relay or gateway
-- a spam filter, an applications-level firewall trying to keep
malware out of a LAN, a gateway into some other transport
environment, or something else.  Now that intermediate system is
faced with the following sorts of headers:

	(i) Those that it understands by virtue of their being
	required by 2822
	
	(ii) Those that it thinks it understands.  Note that it
	could be wrong about this, because there is no required
	/standard definition of the content of any header field
	other than those in 2822 and updating standards and,
	more important, no way to know the properties of any
	header developed subsequent to the implementation base
	of the intermediary.
	
	(iii) Those that it does not understand but that might,
	in the mind of the sender, be required for understanding
	or properly processing the message.  In other words, the
	sender would prefer that the message be bounced if the
	receiver (or an intermediate system) cannot properly
	interpret that field as intended.
	
	(iv) Those that it does not understand but that are, in
	the mind of the sender, simply additional information
	associated about the message.  In other words, the
	sender would like the message delivered regardless of
	whether this information can be delivered or not.

There is no possible way for the intermediate system to
distinguish between (iii) and (iv).  There is actually no way to
distinguish (ii) from either of them either, but the belief that
one understands the field might be worth the risk.

Now, what is the intermediate to do?   If it has a security
motivation and fears malware triggers or inappropriate
information disclosure, it should probably either bounce the
message or discard all headers that are not in the first
category or maybe the first two.  "Discard" is more common than
"bounce" today, but neither is unheard of.   Unfortunately, if a
signature has been computed across the full set of headers and
some are discarded, verification of the signature by the
receiver will fail.

Again, none of this requires any i18n extensions.  It is
consequently unreasonable to expect this WG to solve the
problems in the general case -- we have lived with them for
years and will probably continue to do so, with or without EAI.

Now, what does this mean for downgrading?  I suggest it means:

(1) It is at least as plausible, if not more so, to discard any
unknown headers than it is to convert them.

(2) One should have no, or very limited, expectations that
signed headers will survive downgrading (or downgrading and
later upgrading) unless the signature mechanism is defined as
occurring only over known headers that are well-defined wrt both
syntax and semantics.

(3) If one had a signature over the entire message body (not one
body part at a time), any attempt to encapsulate the headers and
incorporate them as a new body part (or multipart/ part) will
probably destroy that signature unless it is very carefully
defined and interpreted in a way that is sensitive to the
downgrade process.

(4) Certainly the insertion of new headers (in addition to the
ones sent) will destroy a full-header signature.  They, I
believe obviously, should do so because otherwise there are
wonderful opportunities for precisely the attacks that signed
headers are typically intended to protect against.  On the other
hand, upgrading back to a signed message using additional
("Downgraded-...") headers for which responsibility is not
guaranteed by a trusted party creates even greater risks of
attacks unless one has a hash that is completely resistant to
any possible alternate text matching the same signature.  Recent
developments suggest that the only way to get that involves
massive good luck, not long-term technology.

(5) If the combination of a signed full header set and a
requirement for downgrading are encountered, there is a strong
case to be made for bouncing, rather than downgrading the
message and delivering the downgraded approximation.

If that reasoning is even close to correct, then the documents
(probably "downgrade", but possibly "framework") should contain
an appropriate explanation and warning, but this topic is
otherwise a rathole.

Do you disagree?  If so, do you have a proposal for how to do
the downgrading in a way that will be robust against both
filtering and signature techniques, both those that are now
available and those that might be invented in the future?   Such
a proposal would be welcome for the current, 821/2821/822/2822
mail environment, quite independent of any i18n work.   I await
an I-D.

     john

---

From: Kari Hurtta <[email-masked]>
Date: 11 Oct 2006 21:19:27 +0300

> Oddly, it isn't an i18nemail problem, although i18n may make it

Well, are you saying that we should not try help i18nemail
downgrade ?

<...> 
> 	interpret that field as intended.

If it is allowed to try help i18nemail downgrade process,
then draft-ietf-eai-utf8headers can define 

   UTF8SMTP-required-headers

header field, which list these header fields which sender
want preserve and would prefer that the message be bounced
if downgrade destrys these header fields.

Note that this can not be

   UTF8SMTP-optional-headers

because someone other that sender may add header fields
during transport.

> unknown headers than it is to convert them.

Yes. However its is possible to try help i18nemail downgrade process.

> defined and interpreted in a way that is sensitive to the

This prevent encapsulation of headers to new bydy part.

This does NOT prevent encapsulation of whole mail to
application/utf8smtp -part.

But problem is that it may not be possible to detect,
when this encapsulation of whole mail is required.

One option is draft-ietf-eai-utf8headers to define

   UTF8SMTP-full-encapsulation

header field, which triggers this.

For example there may be

   UTF8SMTP-full-encapsulation: required
   
when sender done signature over the entire message body. 

Essentially 

   UTF8SMTP-full-encapsulation: required

says that sender wants gatewaying operation and sender
knows that recipient is able to upgrade i18nemail.

> massive good luck, not long-term technology.

Also insertion of new headers happens to all mails. So signature
mechanism are somewhat selective what they sign.

> message and delivering the downgraded approximation.

Yes.   Full encapsulation however is possible, although
not desired.

If contining my suggestion, there may be

   UTF8SMTP-full-encapsulation: bounce

header field.

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 Oct 2006 16:07:39 -0400

--On Wednesday, 11 October, 2006 21:19 +0300 Kari Hurtta
<[email-masked]> wrote:

>> Oddly, it isn't an i18nemail problem, although i18n may make
>> it more obvious.
> 

Nope.  See below.

> if downgrade destrys these header fields.

This is a very interesting idea, IMO.  We would have to think
through what should be listed and how, which would require a
very specific proposal, but the idea is interesting.  See below.

> because someone other that sender may add header fields

Yes, obviously.
 
>> (1) It is at least as plausible, if not more so, to discard
>> any unknown headers than it is to convert them.
> 

Perhaps.  Or even "probably".

>> occurring only over known headers that are well-defined wrt
>> both syntax and semantics.
>> 
>> (3) If one had a signature over the entire message body (not
>> one body part at a time), any attempt to encapsulate the
>> headers and incorporate them as a new body part (or
>> multipart/ part) will probably destroy that signature unless
>> it is very carefully defined and interpreted in a way that is
>> sensitive to the downgrade process.
> 
 
> application/utf8smtp -part.

But I have yet to see a serious/complete proposal that provides
whole-message encapsulation without a potential violation of the
multiple encoding rule.  Remember that the fact that I've got
UTF8SMTP headers, and even 8BITMIME transport specified, does
not prevent having a body part that is in Base64 or Q-P encoding.

There are some other issues about using application/ for this,
but maybe they aren't important in comparison.

> header field, which triggers this.

I _think_ it just defers the problem.  One of the things that
yields complex MIME structures is message forwarding, with or
without message/??? encapsulation.   Someone trying to forward
an i18n message to a non-i18n site presumably has to do the same
analysis (or make the same determination) that an in-transit
downgrading gateway would.  That strikes me as hard and unlikely
to be gotten right.

> knows that recipient is able to upgrade i18nemail.

Yes.  But, in the general case, the sender doesn't know and
can't find out in a reliable way, at least in-band.  I'd guess
many potential recipients wouldn't know either, especially if
this were an operation applied on the way to the message store,
rather than in the MUA or POP/IMAP server.   Just finding out a
downgrade (alt-address) address is _lots_ easier and more
reliable.

>> headers are typically intended to protect against.  On the
>> other hand, upgrading back to a signed message using
>> additional ("Downgraded-...") headers for which
>> responsibility is not guaranteed by a trusted party creates
>> even greater risks of attacks unless one has a hash that is
>> completely resistant to any possible alternate text matching
>> the same signature.  Recent developments suggest that the
>> only way to get that involves massive good luck, not
>> long-term technology.
> 
> Also insertion of new headers happens to all mails. So
> signature mechanism are somewhat selective what they sign.

Yep.  And that is why traditional uses of PGP and S/MIME don't
sign headers at all.  Whether existing signature mechanisms that
do sign headers are selective enough is, I believe, an open
question.

>    UTF8SMTP-full-encapsulation: bounce

As you have probably gathered, I am a big fan of simplicity
whenever possible and have not seen many complex things
deploying, interoperating, and working on the Internet.  The
more complexity, and the more subtle interpretation of options,
we discover that effective downgrading requires, the less I
become a fan of downgrading.  I think I'm in a small minority so
far in believing that we should be considering the impact of a
UTF8SMTP model that never downgrades in transit, but I think the
number may grow as these complexities and restrictions
accumulate.

If we are going to have downgrading, and we are going to expect
sufficient upgrading to permit signatures over the headers to
function across the transformations, I think your suggestions
may point the way to a workable solution ... or at least one
that is more likely to be workable than anything else we have
seen suggested.  But I can't evaluate whether or not it is
workable from email -- I need an I-D that I can sit down with
and compare to 2821 and 2822, to the various signing standards
and methods, etc.

Any chance that you can get something written and posted before
the cutoff?

     john

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 12 Oct 2006 01:39:29 +0200

Charles Lindsey wrote:

 [2047]
> Proposed Standard, then all implementations that encode or decode
> RFC 2047 stuff magically become non-compliant overnight.

The point of this rule is IMO that in-transit MTAs are free to ignore
MIME, they still "see" what they might need to know, the (2)822 syntax
of vital header fields.

To some degree even the MUA of the receiver is free to ignore MIME, some
header fields might look odd, some body parts might be shown as base64
garbage, but it is a syntactically valid message/rfc822, all essential
operations like resend or reply still work for MIME-ignorants.

I hope that you (this WG) get the same degree of backwards compatibility
for message/utf8smtp.  If you end up with "bounce" whenever a legacy MTA
is on the route you're in trouble.

> No it isn't, and it cannot be.

That MUSTard is in RFC 2822 3.6.8 (optional header fields), for optional
header fields MIME-ignorants don't need to know the structure (if any).

If the "optional header fields" in 2822 are the same as the "extension-
fields" in 822, then they are covered by RFC 2047 5 (1).  You can encode
them as is, their structure (if any) isn't vital for MIME-ignorants.

Kari pointed out that that's confusing for agents knowing the structure.

But maybe that can never happen as far as utf8smtp is concerned:  Either
header fields are upgraded somehow again, then 2047-encodings introduced
by downgrading are decoded.  Or there's no opportunity to ugrade the
header fields, then the MTA or MUA trying to do something with these
header fields cannot handle utf8smtp, and an additionally lost structure
just confirms that it cannot handle the header field, downgraded or not.

That's when we started to discuss plan B (= at least change the header
field name before you encode the unknown header field body), and plan C,
convert the message/utf8smtp to a message/rfc822 containing the original
message/utf8smtp as is.  That would BTW preserve DKIM signatures, DKIM
is designed to work on a complete message/rfc822, it could also handle
an unmodified message/utf8smtp part whereever it is.

Kari's next objection was that multipart/xyz with a text/xyz (for the
original utf8smtp header) plus a second part for the original body is
more friendly for UAs.  That's tricky, RFC 2049 (IIRC) says that unknown
message subtypes should be handled as application/octet-stream.  That's
not what you want for years until all MUAs can handle message/utf8smtp.

OTOH the multipart + text idea might be more complex for gateways, and
John apparently thinks that downgrade will never work if it's not KISS.
Or maybe it's just my interpretation because I'm a KISS (and MIME) fan.

I've seen the effect of message/news produced by my UA with another UA
treating it as application/octet-stream, it's ugly (open with notepad
and similar stunts).

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 12 Oct 2006 08:26:02 +0300

> Kari pointed out that that's confusing for agents knowing the structure.

That says:
  
  (1) An 'encoded-word' may replace a 'text' token (as defined by RFC 822)
      in any Subject or Comments header field, any extension message
      header field, or any MIME body part field for which the field body
      is defined as '*text'.  An 'encoded-word' may also appear in any
      user-defined ("X-") message or body part header field.

This gives 'default is <unstructured>' for X- headers, but IMHO that is 
unclear for other header fields.  Anyway that default can not be used (as
I noted.)

> just confirms that it cannot handle the header field, downgraded or not.

( I think that I'm lost. )
 
> an unmodified message/utf8smtp part whereever it is.

And on someplace on suggestion I changed message/utf8smtp to
application/utf8smtp if idea was to use encapsulation. That
provides double encoding what on message/* is not allowed.

( Was someone trying to register application/mime on past? )
 
> not what you want for years until all MUAs can handle message/utf8smtp.

text/xyz and putting unknown header fields to new header field (ie. change
the header field name before encoding) are alternative methods to
encapsulate individual header fields.    

But perhaps just putting them to new header field is simpler.  
I really not know which one alternative works better.

It was  more like analyze what is required to put unknown header fields 
to new body part. That new body part idea was mentioned on several
articles. But perhaps I just misunderstand what was idea.

Using text/* just gives 'use text/plain for unknown subparts' -rule.

> Or maybe it's just my interpretation because I'm a KISS (and MIME) fan.

What gateways you are thinking? That may fawor "new header field" 
-alternative.

> treating it as application/octet-stream, it's ugly (open with notepad

I think that message/* do not fly.

> Frank

I think that there is now some aggreement what is required mail header 
(as whole) handling. 

I think John Klensin was also suggesting that unknown mail header fields
should be dropped instead (if that is route, then there is my suggestion
about "UTF8SMTP-required-headers" header field. )

Then there is mime header part's of mail. 

And signed mail issue, which John Klensin mentioned.
( There I suggested "UTF8SMTP-full-encapsulation: required" 
  and application/utf8smtp for encapsulation. That is little
  radical. )

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: Thu, 12 Oct 2006 09:43:01 +0300 (EEST)

> but maybe they aren't important in comparison.

Effectively application/utf8smtp WILL violate the
multiple encoding rule.  However, it no violate letter 
of that rule (on MIME standard.)

Someone was trying to get application/mime on some other
working group for same purpose on past?

I think that if that is forbidden, only what can do is
provide signal 'bounce if conversion is required'.
(That is

 in my previous article, but it also can be ESMTP parameter
 to MAIL -command. 
)

/ Kari Hurtta

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 12 Oct 2006 01:31:49 -0700

I wonder if my memory is fooling me....

A rather surprising result of some analysis we did in the work on the 
MIXER specification was that in order for an implementation to be 
robust, it had to treat all unknown header fields as *structured* header 
fields, not unstructured ones.

There is a difference between structured and unstructured in folding; 
structured header field bodie (RFC 2822 section 2.2.2) can only be split 
at existing/allowed whitespace, while unstructured field bodies can, in 
theory, be split anywhere, since spaces are legal anywhere in their content.

So if you assume that a field is unstructured, you may split it 
somewhere it causes damage; if you assume that it's structured, you'll 
do no harm even when it's unstructured.

Surprising, but consistent.

                              Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Oct 2006 11:05:58 +0100

> which are generating encoded-words.             (or what I are missed?)

No, that is not correct. A decent NUA (which excludes quite a lot of them)  
will allow the sender to add any headers that he fancies. Let us suppose  
that the sender knows about this new Foobar: header (and that he  
understands the correct way to use it). So he writes in the new header  
(and it happens to contain some utf-8). And then he expects the existing  
MUA to send it, and he expects the far end (which he happens to know has  
been upgraded) to recognise it and take the appropriate protocol action.

So his MUA has to decide whether to assume the unknown header is  
unstructured (it then just uses RFC 2047) or not (it then replaces it with  
a
     Downgraded: Foobar: <2047 encoding of utf-8 stuff> )

if it guesses wrong, the far end may not understand it (though the 2nd  
alternative is probably the best guess, if guess it must).

It seems the nice thing about the Downgraded: header is that it will work  
for both structured and unstructured headers, provided only that the far  
end is utf8smtp-compliant, and knows that is must upgrade any Downgraded  
headers before it does anything else (or its friendly neighbourhood POP3  
or IMAP does it for him).

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Oct 2006 11:10:02 +0100

On Thu, 12 Oct 2006 00:39:29 +0100, Frank Ellermann  
<[email-masked]> wrote:

>> encoded-words)

Sure, but we are concerned with the end-points, which cannot ignore MIME.
> operations like resend or reply still work for MIME-ignorants.

Yes, if push comes to shove, the recipient MUA can display 2047 stuff (and  
even Downgraded: headers) exactly as received, and let the reader figure  
out what to do with it. Messy and inconvenient, but not illegal, and no  
information has been lost.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Oct 2006 11:12:58 +0100

>  Encapsulation does not require knowing these protocol elements.

And neither does use of the Downgraded: header. So that is no argument for  
building complete message encapsulation into the protocol. Whatever one  
can achieve, so can the other.

---

From: Kari Hurtta <[email-masked]>
Date: Thu, 12 Oct 2006 13:27:07 +0300 (EEST)

>>> Charles Lindsey

Downgraded: header    is encapsulation, but there is cases where that
mini encapsulation do not handle, I think.   

These cases however may require something like application/utf8smtp and seems 
that it is not acceptable. 

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Oct 2006 11:42:47 +0100

> faced with the following sorts of headers:

But I don't wee why such a filter should be trying to introduce utf8smtp  
stuff (and hence a necessity to downgrade) where there was none before).  
So the situation (and problem, if any) you describe already exists on the  
present network, and I am not aware that present systems have to worry  
about it.

You need to discuss an example where at leats one of the endpoints is  
trying to use utf8smtp.
> 	interpret that field as intended.

I think the number of readers taking that attitude will be relatively  
small - a plausible example with some existing but rarely used header  
might help - maybe one of those headers which requests that an  
acknowledgement receipt be ruturned.
> "bounce" today, but neither is unheard of.

I would regard discarding of any header by intermediaries as a gross  
violation of the protocols, and if you tell me it is commenly done, then I  
am dismayed.

Anyway, suppose we want a mechanism for the sender to be able to say "if  
you cannot transmit the Foobar header, then please bounce rather than  
munge/discard it.

Ok, that may be a need, and it is a good example of a feature that might  
require one o those <parameter>s to the Header-Type header (or whatever we  
finally call it) which Harald was asking about.

So the sender writes:

    Header-Type: UTF8SMTP; headers-not-to-be-discarded="Foobar,Barfoo"

or something like that (and maybe another parameter for headers you  
actively wish to be bounced).

>   Unfortunately, if a

That is a matter of correct design of the header-signature protocol. For  
sure, any such signature scheme should include, with the signature, a list  
of the headers included within it. E.g. PGPVERIFY does that, but PGPMOOSE  
doesn't. Designing such a scheme is hard work to get it right, but I am  
sure it is doable (and I have some experience of trying to do it).

> ones sent) will destroy a full-header signature.

No, that certainly should not be so (and is not so with either of the  
schemes I mentioned above).

>  They, I
> headers are typically intended to protect against.

The headers you should sign are exactly those necessary to prevent the  
scam you are trying to stop - no more, no less.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Oct 2006 15:52:12 +0100

> mini encapsulation do not handle, I think.

Then please show us such a case, and we can look at it.

---

From: Kari Hurtta <[email-masked]>
Date: 12 Oct 2006 20:40:00 +0300

<...>
<...>
> Then there is mime header part's of mail. 

Hmm. It occurs to me that encapsulation to Downgraded: -header field is
equivalent of removing header field if recipient is not i18nmail-aware.
So maybe there also on need of "UTF8SMTP-required-headers" header field
(or equivalent) also on that case?

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 12 Oct 2006 20:25:48 +0300

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> On Thu, 12 Oct 2006 11:27:07 +0100, Kari Hurtta

NOTE: One soluntion is to forbid data on i18nmail if  Downgraded: -header
      field can not handle it. But let assume that data is allowed.

Case #1:

Almost anything what counts signature over (some) header fields or 
whole header blocks. 

If Downgrade header field does not hide signature, then i18mail unaware MUA
and also any MTA before upgrading agent may note that signature fails.

There is S/MIME, domainkeys and so on .... Some of these are likely
to be checked on border MTAs.   Some of these failures are likely cause 
mail to be rejected or moved to spam folder.

Even when signature is on one header field, it is unlikely that it includes
utf-8 characters and therefore is is not moved to Downgrade: -header
(and that way hiden.)

Encapsulation to application/utf8smtp prevent signature verification 
before it is undone, because signature is not seen when message (or message
part) is downgraded.

And there is problems even when mail is upgraded.

Succeed of signature requires that upgrade is able exactly return 
correct header fields with same order and even whitespaces. Also
extra headers must not be added.

   - If syntax of some header field is known, it is converted.
     Therefore there was no reason for downgrading component 
     to add Downgraded: -header field. 

     In general from downgraded header it is not possible
     to return orignal header field exactly (whitespace differences
     and other differences which do not alter semantic.)
     Conversion tries just preserve semantic. That is not
     enough to return original header so that signature 
     verification counts same signature.

   - Even when downgrading  agent is added Downgraded: -header 
     field with converted header. Upgrading process must known
     to remove converted header and pick data only from
     Downgraded: -header field.

   - And there may be some headers, which are not downgraded,
     but was originally using RFC 2231 or RFC 2047.  Upgrading 
     process  may still convert them to UTF-8. That also changes 
     signature.

Case #2:

I'm not sure about that case. This depends on what protocol elements 
is UTF-8 allowed on Content-Type -header.

It is posisble to add Downgraded: header -field, but
actual problem is that Content-Type must always to able
to be converted also.   If there is UTF-8 on protocol elements
of Content-Type, for which there is no conversion available
(ie. RFC 2231 or RFC 2047 do not ably), it may be that
that converted Content-Type header field can not be formed.

that does not prevent putting mime header block and mime subpart
inside of application/utf8smtp.

Of course there is problem when put some message or mime subpart
inside of application/utf8smtp. I addressed that only partially
on one my other article.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 12 Oct 2006 20:44:07 +0300

Case #1:

Case #2:

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: Thu, 12 Oct 2006 21:27:13 +0300 (EEST)

( I think that my answer get lost, so I try recreate it. 
  I'm sorry if this is duplicate. )

> Then please show us such a case, and we can look at it.

NOTE: There is always(?) option to to forbid data which  Downgraded: header field
      can not handle. Let's assume that is is not forbidded.

Case #1:

      Almost any case where signature is calculated over header fields or
      over whole header block. There is S/MIME, domainkeys and so on ...

      If recipeint MUA is not i18nmail aware it will calculate signature
      over downgraded header. That will cause failure of signature.

      Also signature will fail if some MTA calculates it before mail
      is upgraded. It is likely that some border MTAs calculates signatures.
      This may cause mail to be rejected or moved to spam folder.

      Encapsulation with application/utf8smtp do not fail, because
      signature is not seen before encapsulation is undone.

      Even when signature is on header field, it normally do not
      include utf-8 characters, so downgarding routine is not reason 
      move it to Downgraded: -header field and that way hide signature.

      There is also problems after mail is upgraded:

           - If downgarding routine knows synta of header filed,
             it have no reason to add Downgraded: header field.
             Instead header field is just converted.  It is
             not possibile recover original header field from 
             converted header field so that signature calculation
             succeed. Conversion tries preserve semantic of
             data, it does not preserve other aspects of data
             which affect signature (such as whitespace differences
             for example.)
             
           - Even when  downgarding routine  adds 
             Downgraded: header field with converted header field,
             upgrading routine must the known remove converted 
             header field and pick data only from Downgraded:
             -header field.

           - There may be some header field which are not converted
             at all. However upgrading routine may able to
             upgrade them to use UTF-8. That also may cause signature
             to fail.

Case #2:

I'm not sure about that. That depends what protocol elements are
allowed to include UTF-8 on Content-type header field.

It is possible to put Content-type header field to Downgraded:
header field. But problem is that there must be also
converted Content-Type header field. If there is UTF-8 data
on elements where conversion is not available ( for example
RFC 2231 or 2047 do not apply), then it maybe is not 
possible to generate converted Content-Type header field.

However it is possible to put whole mime header block and
that mime part inside of application/utf8smtp encapsulation.

Of course there is problem that when to use application/utf8smtp
for encapsulation.  That I have only partially addressed
on some other my mail or article.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 12 Oct 2006 23:53:23 +0200

> fields, not unstructured ones.

What does that mean if you don't know the structure ?  Some lexical
analysis as defined in 822, comments, white space, etc. ?

With gateways one hard part is to get the terminology right, if things
appear to be similar on both sides.  I'll never forget the day when I
finally understood that a Usenet Message-ID is the essence, and a Path
only secondary - exactly the opposite of Fido.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 13 Oct 2006 00:22:15 +0200

> ( I think that I'm lost. )

Sorry, I try it again.  One important scenario is S -> D -> U -> R,
sender S, downgrading gateway D, upgrading gateway U, receiver R.

What D and U exactly do, how they encode and decode header fields, 
is irrelevant for R as long as it gets anything as sent by S.  

For S it's more interesting if the downgraded message is bounced by
a hop between D and U, but S knows what it sent, it's not critical.

Another scenario is S -> D -> L, where L is a "legacy" system, any
agent that doesn't support utf8smtp.  

Let's say L supports a structured header field Y, a Jabber-ID or an
Archived-At or similar.  And let's assume that D does NOT know this
structure.  Then we have three cases:

S didn't use Y                     => no issue
S used Y as specified (ASCII+2231) => no issue, D won't touch it
S used the Y structure with UTF-8  => D doesn't know it and encodes
or encapsulates it as good as it can.  If that procedure "destroys"
(from L's POV) the structure of Y it's again no issue, because L 
cannot handle the UTF-8 header field Y as sent by S.  What D does
is irrelevant from L's POV.

L is only interested in the "known" downgraded header fields (From,
To, CC, etc.), that's covered in the downgrade draft.  It's not the
same situation as for R in the first scenario, where R of course
wants to get the original header field Y as sent by S.

In the 1st scenario it's also okay if D doesn't know the structure
of Y and encodes it beyond recognition.  Only U needs to know how
to decode it, structured or unstructured.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 13 Oct 2006 07:05:08 +0300

> is irrelevant from L's POV.

 S -> D -> L   header field Y is maybe some protocol action what
               sender may want. 

               sender should have realized that it must not use
               utf-8 on that header field Y.   However sender prefers
               that it get's bounce on that case so that error
               is fixed.

> Frank

Thank you. Now I see what you mean.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 13 Oct 2006 19:14:51 +0100

> Almost anything what counts signature over (some) header fields or

Sure. Any scheme that includes header fields withiun a signature is going  
to have problems. But there are no such standardized schemes yet in  
existence. If, when they come, there are badly designed, they are going to  
fail badly on the present network, let alone on a network with UTF-8. But  
if they are sensibly designed, so that they are proof agains the mungings  
commonly encountered (and that means a good canonicalization algorithm),  
they should be OK.

But we cannpt discuss such schemes unless we have specific examples top  
look at.

So All I am claiming for now is that there need be no serious problems  
with pgp-mime and the like, since they do not sign headers (except for the  
particular cases I discussed with John, which we can probably get by with  
a little care.

> extra headers must not be added.

On the contrary, those are just the sort of things that a well-desinged  
header signature field should be designed to work with, It can be done  
(because I once did it).

> is UTF-8 allowed on Content-Type -header.

Comments (which will be rare, but not forbidden) and <parameter>s  
(downgraded by RFC 2231).
> to be converted also.

Yes, Content-Type should be one of the cases that the dwngrade document  
knows how to do. It may well involve providing a Downgraded header  
_as_well_, buts as it fom with From: To: etc.

    If there is UTF-8 on protocol elements
> (ie. RFC 2231 or RFC 2047 do not ably),

There aren't any.

---

From: Kari Hurtta <[email-masked]>
Date: 21 Oct 2006 22:20:02 +0300

> But it does allow 8bit as needed by a hypothetical message/utf8smtp.

Then there is RFC 2046 which writes:

| 5.2.4.  Other Message Subtypes
| 
|    MIME implementations must in general treat unrecognized subtypes of
|    "message" as being equivalent to "application/octet-stream".
| 
|    Future subtypes of "message" intended for use with email should be
|    restricted to "7bit" encoding. A type other than "message" should be
|    used if restriction to "7bit" is not possible.

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Sat, 21 Oct 2006 15:39:25 -0400

--On Saturday, 21 October, 2006 22:20 +0300 Kari Hurtta
<[email-masked]> wrote:

>> Okay, RFC 2045 says EXPRESSLY FORBIDDEN about base64 for
>> message/*. But it does allow 8bit as needed by a hypothetical
>> message/utf8smtp. With that you'd get...
>> 
> |    MIME implementations must in general treat unrecognized
> subtypes of |    "message" as being equivalent to
> "application/octet-stream". | 

Note that this means that, if I am a legacy MUA or MTA, and I
receive a message/utf8smtp body part and, being a legacy
implementation, don't understand it, then I must treat it as
completely opaque, with no chance of displaying it or trying to
decode it.

> |    Future subtypes of "message" intended for use with email
> should be |    restricted to "7bit" encoding. A type other
> than "message" should be |    used if restriction to "7bit" is
> not possible.

Are we creeping toward either an amendment to that text in 2046
or a new top-level type for, e.g., "message8/"  ?

    john

---

From: Kari Hurtta <[email-masked]>
Date: 21 Oct 2006 23:24:32 +0300

<...>
> or a new top-level type for, e.g., "message8/"  ?

( summary: I do not know answer. )

Uhh.  Any unknown (as not know how to parse) composite types have same 
problems.

message/TBD   and TBD/*      are equal bad,

If TBD/* is considered to be composite.

But these really do not matter    if    EAI documents allow these only on 
UTF8SMTP   environment.    

Then can say "not for mail"  == "not for EAI-incompliant mail"

If type is restricted of  UTF8SMTP  environment then
       any of        message/rfc822  with UTF-8 header fields
                     message/utf8smtp
                     message8/mail

will work.

Any of these are on problems if they are allowed to see by
EAI-incompliant  MTA or MUA   (or if they leaked).

Any of these needs some kind   downgrading when leaving UTF8SMTP.

Unkwnown discrete types are OK on MIME.

( For new top level types there is MTAs and MUAs also problem
  that they do not  know is they composite or discrete.  I
  suspect that 'discrete' is assumed by default.  This means
  that double encoding can not avoided. )

Hmm.   Perhaps message8/*   works best, because it will most
likely to be treated as discrete (and therefore base64 or 
quoted-printable encoded for example on 8BITMIME downgrading. )
This of course requires that double encoding rules must be
lifted for message8/*  

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 23 Oct 2006 15:52:15 +0100

> |    used if restriction to "7bit" is not possible.
And yet RFC 2046 also writes, in 5.2.1:

    |    No encoding other than "7bit", "8bit", or "binary" is permitted for
    |    the body of a "message/rfc822 entity.

which seems to contradict what it says in 5.2.4 (we also discussed the
meaning of that sentence in another thread a few days ago).

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 23 Oct 2006 15:55:07 +0100

>> or a new top-level type for, e.g., "message8/"  ?

I think we are creeping toward an ammendment to RFC 2046.

> But these really do not matter    if    EAI documents allow these only on

Yes, they must be kept away from all old-style implementations.

> Any of these needs some kind   downgrading when leaving UTF8SMTP.

Exactly, or bouncing.

---

From: Kari Hurtta <[email-masked]>
Date: 23 Oct 2006 18:13:14 +0300

> meaning of that sentence in another thread a few days ago).

I read that 5.2.4 is instruction what kind subtypes of message should be
registered. In other words on what limitations should be on that particular
new subtype registeration.

And anyway message/rfc822   is not future subtype of "message".

Message/rfc822 is one of subtypes what is specified on MIME standard itself.

Instruction on 5.2.4 is probably because of 8BITMIME downgrade problems.
if particular new subtype of message includes some other data than "7bit",
8BITMIME  requires to bounce message on downgrading ...

As I mentioned earlier, new composite types (either new top level types or 
new subtypes of message) are pain on MIME. That is where is breaks.

If type is "discrete", there is no problems...  

/ Kari Hurtta