Subject: [EAI] knowing the encoding for something before it is encountered

From: John C Klensin <[email-masked]>
Date: Wed, 01 Nov 2006 19:23:53 -0500

Hi.

Several ideas have been floating around that seem to violate, or
come close to violating, a design principle for email and some
other systems.  Maybe it is worth trying to articulate that
design principle.

The principle is that we insist on know the content structure
(to the extent possible) and the encoding before you get the
content or values.  In particular, one should not have to parse
something and then back up when one figures out how material
already encountered is to be interpreted, nor should one need to
do heuristics to figure out what something is.  

Traditional MIME conforms exactly to that rule: one knows that
the headers are in ASCII, the main header can contain a
content-type and content-transfer-encoding that specify the
structure and encoding of the body, but all of that header
material is in ASCII so there are no encoding issues with it.
There are no headers that say "go back and reinterpret all of
the other headers in some other way".

In addition, the internal MIME headers are entirely in ASCII.
That has proven to be a good thing because it is common practice
to slice body parts out of messages (in MUAs and the like, after
delivery) and treat them in separate ways.  Needing some remote
information (e.g., in the main headers) to find out how internal
MIME headers are encoded makes that task considerably more
difficult and implies that one must build up and carry much more
message structure to handle them.

As we move into the UTF8SMTP world, we get onto slippery slopes
with this.  We can do UTF-8 in headers because the envelope
tells us that they may be coming (and because ASCII characters
are proper UTF-8). But, once the envelope is stripped away, we
are proposing to rely on a header that changes the encoding of
other headers, which would be pretty close to a first.  We are
also looking at proposals to change the coding of parts of both
the main content-type header and of internal MIME headers based
on that header field.  That isn't necessarily bad, but it is a
little scary, at least to me.  And I think we should all at
least try to understand the issues.

    john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Kari Hurtta <[email-masked]>
Date: 02 Nov 2006 08:17:35 +0200

John C Klensin <[email-masked]> writes in gmane.ietf.ima:

<...>
> least try to understand the issues.

Yes.  I also asked that why UTF8SMTP do not need parameter
on SMTP when 8BITMIME uses BODY=UTF8SMTP  parameter.

So if outside indicator is need then it means

    1) HDR=UTF8   parameter is needed for MAIL -command

    2) message/rfc822  need to be replaced with somewhat else,
       ie.

       message/utf8   or application/utf8-message

       ( or with that encapsulation was on my draft )

   3)  text/rfc822-headers  need be replaced with something else
       ie.

       text/utf8-header         (that was from my draft)

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 02 Nov 2006 12:13:40 -0000

>     1) HDR=UTF8   parameter is needed for MAIL -command

Yes, I can see some merit in that. With that in place, then all MTAs would  
be aware that the message might require more careful scrutiny for places  
that might need to be downgraded. MDAs (whatever they are) would know,  
because the MTAs that delivered to them would tell them. Sending MUAs  
would know because they constructed it that way.

So it only leaves gateways that might have to scan the whole message  
looking for
    Header-Type: UTF8
and gateways nearly always get involved in messy processing such as  
scanning things twice. Actually, it should only be necessary to scan the  
top-level header-fields for that Header-Type header, and only look further  
if it was present.

>     2) message/rfc822  need to be replaced with somewhat else,

Or else they need to occur only in an environment which is protected as in  
(1) (with messiness in gateways as usual),

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Kari Hurtta <[email-masked]>
Date: 02 Nov 2006 16:30:38 +0200

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> Yes, I can see some merit in that. With that in place, then all MTAs
> would  be aware that the message might require more careful scrutiny
> for places  that might need to be downgraded. MDAs (whatever they are)
> would know,  because the MTAs that delivered to them would tell
> them. Sending MUAs  would know because they constructed it that way.
> 
> scanning things twice. Actually, it should only be necessary to scan
> the  top-level header-fields for that Header-Type header, and only
> look further  if it was present.
> 
> >
> Or else they need to occur only in an environment which is protected
> as in  (1) (with messiness in gateways as usual),

On someway that can also follow 1)  ie.

   Content-Type:  message/rfc822; hdr=UTF8

if UTF8  on top level do not imply that all attached messages also are
UTF8SMTP messages.

I think that it (UTF8  on top level) does not necessary imply that.
So indication may be still needed.

Anyway attached UTF8SMTP means that top level is also UTF8.

If top level is not UTF8, then that, what I suggested on my encapsulation
draft, is needed.   (or  application/utf8-message, but I think that 
this is not wanted.)

After all that  UTF8 flag  makes difference, 

  Is message using some unknown 8-bit headers (very many system uses raw 8-bit ISO-8859-*
        charcters on subject without any labeling)

versus

  Is UTF8 expected

/ Kari Hurtta