Subject: Re: [IMA] DRAFT minutes from IEE BOF

From: Martin Duerst <[email-masked]>
Date: Fri, 02 Dec 2005 18:48:23 +0900

Harald Tveit Alvestrand wrote:
 > 5. Downgrading, draft-yoneya-ima-downgrade - K. Fujiwarea

Fujiwarea -> Fujiwara.

Regards,    Martin. 

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Martin Duerst <[email-masked]>
Date: Fri, 02 Dec 2005 19:05:00 +0900

Dear IEE BOF participants,

As some of you may know, I'm working on an update of the
'mailto:' URI RFC to bring it more in line with newer
practice, in particular IRIs. The latest draft can be
found at:
http://www.ietf.org/internet-drafts/draft-duerst-mailto-bis-01.txt
There are still a few open issues/comments that
need to be addressed.

I have tried to make this draft forward-compatible with
any future solution to the internationalization of email
addresses (in particular, the LHS). Given that the current
draft is using UTF-8 for compatibility with IRIs, and
given that the proposed work in the charter is going into
the direction of UTF-8, this should work out fine.
But it may make sense for you to have a look at the draft,
both for technical issues as well as with respect to the
proposed charter.

With respect to the charter, I can see various solutions:
1) There is no need to coordinate
2) The draft can go ahead as an individual submission, but
    there should be some coordination, maybe put down in
    the charter.
3) The draft can be an individual submission, but should
    be delayed until the solution being worked on is clearer.
4) The draft should become part of chartered WG.

My preference is somewhere around 2).

Regards,    Martin.

---

From: John C Klensin <[email-masked]>
Date: Fri, 02 Dec 2005 09:05:42 -0500

Martin,

I've taken the liberty of copying Ted and Scott, since I'm not
sure they both are on the mailing list.  

Let me give you my take on this...

We need to get a charter approved by the IESG.  Until we do,
this group is just the outgrowth of a design team and can't
really say anything definitive.  I've been preoccupied since
IETF with trying to get some things out vis-a-vis the ICANN
meeting, but should be able to sit with the minutes in one hand
and the charter in the other over this weekend and then work
with Harald to get a revised draft posted early next week.

I think we can reasonably make something like "check the general
design of the framework and SMTP extensions document and verify
that their syntax assumptions and those of the IRI draft are
consistent" the WG's first work item, with a very short
timeframe.    While I assume that there will be some tuning of
the document, I hope we can lock down framework and syntax
quickly.  The notion that should be possible was reinforced by
the BOF: all of the controversy seems to be about headers,
indications of UTF-8 presence in the headers and how it (or
they) are handled, downgrading and support for it, and a
collection of fine details.

I think that falls somewhere between your (2) and your (3).

That said, it would clearly be up to the IESG, but I have
trouble seeing how, once this work is started, you will be able
to publish a standards-track IRI mailto document without a
normative reference to it :-(

     john

p.s. UTF-8 was chosen for the LHS (and, optionally, the RHS) for
compatibility with RFC 2821 given UTF-8's upward compatibility
with ASCII and in conformance with RFC 2277. "Compatibility with
IRIs" was, at best, a secondary consideration.

---

From: "YAO Jiankang" <[email-masked]>
Date: Sat, 3 Dec 2005 12:05:30 +0800

----- Original Message ----- 
From: "Dave Crocker" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, December 01, 2005 4:48 AM
Subject: [IMA] Re: Header for stored messages

> 
> 
> > In theory, the content of messages in transit is not examined at
> > all, so it makes little difference what is in the headers except
> > at the endpoints.  
> 
> Parts of transit are packaging and unpackaging for posting and delivery.
> 
> These are examples of needing self-labeling.
> 
> 
> > I'm also reminded of the famous "MIME-Version: 1.0" header
> > field, which was supposed to denote a new way of identifying
> > headers (particularly the presence and interpretation of
> > "Content-type:") as well as a formatted body.  Now, it was
> > arguably necessary because "Content-type:" was already in use
> 
> Right.  Self-labeling.
> 
> We can quibble about the how, but the main point is that a message with a
> free-text body has a different labeling than a message with a MIME body, so
> that a processor can know which model to apply.
> 
> 
> > More generally, is this a header that should be required to be
> > pushed into the message on final delivery or as part of a
> > gateway process, not unlike "Return-path:".  And, if not, what
> > is the value of having it in transit?
> 
> Yeah, I suppose we could quibble whether to have it only in the SMTP
> commands during transit, but in the message itself otherwise, but my main
> concern is robustness in the absence of context.
> 

Yes, in order to get the robustness in the absence of context,  it may be a issue we should consider that whether or not we need a mechanism or some indicator to distinguish or recognize  wether the message is a "stored" message be of new format(i.e. IEE compliant) or old one  (i.e. RFC 822 compliant).

Yao Jiankang

> d/
> -- 
> 
> Dave Crocker
> Brandenburg InternetWorking
> <http://bbiw.net>
>

---

From: "YAO Jiankang" <[email-masked]>
Date: Sat, 3 Dec 2005 12:39:56 +0800

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: <[email-masked]>
Cc: "Jamie Zawinski" <[email-masked]>; "Larry Masinter" <[email-masked]>
Sent: Friday, December 02, 2005 6:05 PM
Subject: [IMA] URIs/IRIs

> the direction of UTF-8, this should work out fine.

The solution should be the direction of UTF-8.

> 4) The draft should become part of chartered WG.

What your concern is may also be a concern of Aread directors or IESG.
If your draft is going to RFC, to make your document compatible with the future possible IEE standard,
it is better that the part relating to IEE in your document is referenced to the future IEE standard.
if your draft is just a draft, since our general direction is same  (utf-8), you may do something between the solution 2 and solution 3.

thanks.
Yao Jiankang
CNNIC

---

From: Martin Duerst <[email-masked]>
Date: Mon, 05 Dec 2005 17:37:42 +0900

>sure they both are on the mailing list.

Thanks.

 >with Harald to get a revised draft posted early next week.

Of course I know this group isn't yet chartered, but that's
exactly why I wrote this mail.

 >that their syntax assumptions and those of the IRI draft are

Maybe just a detail:
Please don't confuse the IRI spec (RFC 3987, Proposed Standard)
and draft-duerst-mailto-bis-01, a draft for an update of the
mailto: scheme.

 >normative reference to it :-(

Again, first the details:
draft-duerst-mailto-bis-01 is a mailto URI document, not a
mailto IRI document. But it is written so that it works
well with IRIs.

So what you are saying is that we'd better finish up the
mailto URI update before the new WG is chartered, or we have
to wait for quite a while :-).

 >IRIs" was, at best, a secondary consideration.

That's fine. The mailto: URI needs to use (%-encoding based on)
UTF-8 for compatibility with IRIs. But there is no requirement
that URIs in a certain scheme and the underlying protocol use
the same encoding; the classical example is the imap: URI scheme
uses UTF-8 for compatibility with IRIs, but the IMAP protocol
uses a modified version of UTF-7 for compatibility with very
old file systems.

Of course, using UTF-8 in both places makes things easier.
And in both places, the reason for using UTF-8 is the same:
the upwards compatibity with ASCII. So this is more than
just a coincidence.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Mon, 05 Dec 2005 09:10:33 -0500

> the upwards compatibity with ASCII. So this is more than

That was the key point, and really the only point, I was trying
to make.

I think some of us should just look at the thing, soon, and get
comments to you, rather than tying things up in formalities.

    john

---

From: "Jeff Yeh" <[email-masked]>
Date: Tue, 6 Dec 2005 10:59:11 +0800

It seems to me that to have a labeling mechanism is better.
And since the sender would like to sent out the message in UTF8,  it make
most sense to require sender to add the label. 
The problem is that are we going to
1. introduce new header like "IEE-Version: 1.0"
2. or use "MIME-Version: 2.0"
3. or more meaningful "MIME-Content-Type"

I preferred 3>1>2.
However, if we do any of these 3 option, we have to take more action when
downgrading occurs.
Because the header are not UTF8 anymore.

Jeff Yeh
TWNIC 
<http://www.twnic.net.tw/>

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Dave Crocker
> Sent: Thursday, December 01, 2005 4:49 AM
> To: John C Klensin
> Cc: [email-masked]
> > In theory, the content of messages in transit is not 
> examined at all, 
> > so it makes little difference what is in the headers except at the 
> > endpoints.
> 
> > I'm also reminded of the famous "MIME-Version: 1.0" header field, 
> > which was supposed to denote a new way of identifying headers 
> > (particularly the presence and interpretation of
> > "Content-type:") as well as a formatted body.  Now, it was arguably 
> > necessary because "Content-type:" was already in use
> 
> Right.  Self-labeling.
> 
> We can quibble about the how, but the main point is that a 
> message with a free-text body has a different labeling than a 
> message with a MIME body, so that a processor can know which 
> model to apply.
> 
> 
> > More generally, is this a header that should be required to 
> be pushed 
> > into the message on final delivery or as part of a gateway process, 
> > not unlike "Return-path:".  And, if not, what is the value 
> of having 
> > it in transit?
> 
> Yeah, I suppose we could quibble whether to have it only in 
> the SMTP commands during transit, but in the message itself 
> otherwise, but my main concern is robustness in the absence 
> of context.
> 
> d/
> -- 
> 
> Dave Crocker

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 05 Dec 2005 18:01:03 -0000

> comments to you, rather than tying things up in formalities.

The most important thing to decide, but one not likely to be clear for  
some time, arises from the proposal to allow an alternatate delivery  
address in the RCPT TO: command. Nobody has really indicated how these two  
addresses get generated. But it would seem desirable to be able to  
indicate in a mailto: URI "this is my address if you are sending mail to  
my I18N address, but if your system cannot manage that, then please send  
to this alterative (but not to both)".

Now there are many notations in which that might be expressed, but we also  
need to consider what would appear in the To: and Reply-To: headers of a  
message, because the same problem arises there too.

For example, if you want a gold-plated solution, you might invent a new  
format for <addr-spec>, such as

     <[email-masked][email-masked]>

(Harald can suggest more plausible Norwegian examples).

OTOH, that might be considered too big an upgrade of RFC 2822 for us to be  
making, though it would lead to an obvious corresponding upgrade for the  
mailto: URI.

NOTE: "ålice" above is written in ISO-8859-1, because that is what my  
mailer does, but clearly the corresponding UTF-8 was intended.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 06 Dec 2005 14:08:28 +0100

> that a processor can know which model to apply.

This is actually an example of how our standards for self-identifying stuff 
often turn out not to work as well as we'd like them to, but standards for 
container-identifying-content works fairly well.

Viz. - the MIME-Version is an example of self-labelling, and has not worked 
well. The content-type is an example of a label outside the labelled 
object, as is the protocol identifier in IP packets, the port number (an 
example of a "weak" label, btw - it's often misleading) and so on.

The HTTP version number (GET /URL/goes/here HTTP/1.1) turned out to be in 
the wrong place when we wanted to modify the way URLs were encoded - so we 
added the "Host: " header hack instead, and the "URL=*" hack for the case 
of shttp (another good idea lost in history...)

The version number in the IP header, on the other hand, seems to have 
worked reasonably well - the container label says "some kind of IP", and 
one of the first few bytes of the header says which version it is.

My worry is the same as was voiced about the SMTP MTA: All legal "legacy" 
SMTP messages will be legal IMA messages, and there will be implementations 
that need to do no more to support IMA than to stop checking for the 8th 
bit of data - apart from the gateway scenario.

So.... what is the scenario where the processor needs to apply a different 
processing to an IMA-labelled message, AND can safely assume that all 
non-IMA-labelled messages are "legacy" conformant?

                          Harald

---

From: Dave Crocker <[email-masked]>
Date: Tue, 06 Dec 2005 08:36:32 -0800

sigh.

> Viz. - the MIME-Version is an example of self-labelling, and has not 
> worked well. 

It hasn't?

If you mean it does not simply, accurately and reliable distinguish between
MIME-structured email body content, and free-form lines of ASCII text
content, this is the first I have heard of its failing.

You need to substantiate the claim, since simply, accurately and reliably 
distinguishing between two forms of data is the reason I gave for needing a 
context-independent label.

(Reference to the problem of having an unchangeable version number is quite
irrelevant to the current discussion.)

>     The content-type is an example of a label outside the 
> labelled object,

MIME was offered as an example of distinguishing a characteristic of an
email message body.  The first Content-type is within the email message
header, so it is NOT outside the labeled object.

>     as is the protocol identifier in IP packets, the port 
> number (an example of a "weak" label, btw - it's often misleading) and 
> so on.

Since Content-type and these additional examples have no goal of being used
outside of their immediate context -- and since the point I made was
specifically about identification outside of the transit context -- these
examples are irrelevant to the discussion.

> My worry is the same as was voiced about the SMTP MTA: All legal 
> "legacy" SMTP messages will be legal IMA messages, and there will be 
> implementations that need to do no more to support IMA than to stop 
> checking for the 8th bit of data - apart from the gateway scenario.
> 
> So.... what is the scenario where the processor needs to apply a 
> different processing to an IMA-labelled message, AND can safely assume 
> that all non-IMA-labelled messages are "legacy" conformant?

A nicely useful question.

A shame it was obfuscated by the earlier portions of the message.

The example I'll suggest is a message standing alone in a file, after receipt.

We already have an example of having to correct this problem: the
Return-Path header field.  It corrects the error of having that information
only be in the transit context (SMTP envelope) and not in the stand-alone
email object.

d/
-- 

Dave Crocker

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 06 Dec 2005 17:59:49 +0100

> content, this is the first I have heard of its failing.

People who write UAs for a living will have to speak to whether they 
actually check it.
It was supposed to give the rule for parsing the content-type header, but 
can occur before or after the content-type header, so if they actually use 
it for that, it means that they have to delay content-type header 
processing until after the whole header is parsed. Is that how people's 
code works?

> distinguishing between two forms of data is the reason I gave for needing
> a context-independent label.
>

But the content-type labels the content, not the header. And it's outside 
the content.

> The example I'll suggest is a message standing alone in a file, after

And what scenario do you have where a processor needs to apply a different 
processing to an IMA-labelled message, standing alone in a file, to what 
processing it needs to apply to a non-IMA-labelled message, AND can safely 
assume that all non-IMA-labelled messages, standing alone in a file, are 
"legacy" conformant?

> only be in the transit context (SMTP envelope) and not in the stand-alone

In that case, I think we have a lot of procmail-type scripts that attest to 
the fact that the information is useful.

Note also that in the proposals this week, the transit context (SMTP 
envelope) has been proposed to NOT contain information about whether a 
specific message is "IMA" or not.

                                Harald

(Personal note: I am in two minds about whether or not a label in the 
headers is an useful thing. My gut feeling is that it might be, but I 
haven't seen the clincher argument for it yet - nor have I seen the 
clincher argument that it's a significant cost and complexity that needs to 
be avoided.)

---

From: Dave Crocker <[email-masked]>
Date: Tue, 06 Dec 2005 09:50:40 -0800

> But the content-type labels the content, not the header. And it's 
> outside the content.

But it is not outside the message.

<irony>
Your point is not magically made helpful by your re-defining the question.
</irony>

Self-labeling of message content was the stated task.  Hence the issue is 
whether the label needs to be within the message, not whether it needs to be 
within a particular portion of the message.

For your example to be useful, you would need to show that the type of the 
data is correctly identified when that data are outside the message, yet 
still be identified correctly. To that end, the translation from 
content-type to the field extension of a specific piece of mail body content 
-- stored on its own and without the message -- would be a representative 
case.  In fact that is done with reasonable success, though certainly not 
always correct.

> And what scenario do you have where a processor needs to apply a 
> different processing to an IMA-labelled message, standing alone in a 
> file, to what processing it needs to apply to a non-IMA-labelled 
> message, AND can safely assume that all non-IMA-labelled messages, 
> standing alone in a file, are "legacy" conformant?

Perhaps you have not noticed that it is quite common to have Internet Mail 
stored in files in a form that is identical -- or, at least, very close -- 
to the form used in transit?

I have used quite a number of MUAs in recent years and they have all 
retained the original form.

> In that case, I think we have a lot of procmail-type scripts that attest 
> to the fact that the information is useful.

Harald, you have again missed my point, though I could have sworn it was 
made very clearly:  Return-path is a retrofit.  It was added later, 
specifically because having that information be only in envelope was not 
sufficient.  The envelope is lost after transit, unless its information is 
added to the message.  It needs to be associated with the message data 
directly.  In other words, Harald, Return-path is an example of needing to 
have the information available outside the context of the transit.

As for procmail, the fact that some software uses the envelope information 
does not have any relevance at all to this discussion.  No one said the 
envelope information was not useful.  So your example does not demonstrate 
that the information is not needed after transit.  (Really, Harald, I 
strongly suggest you get a copy of "Crimes Against Logic" and read it 
carefully.)

d/

-- 

Dave Crocker

Dave Crocker

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 06 Dec 2005 20:39:00 +0100

Dave,

I have obtained my copy of "crimes against logic" already (arrived today), 
and look forward to reading it.

And now - could you please answer my question about processing?

> Your point is not magically made helpful by your re-defining the question.

and irony is not made effective by ignoring the division of a message into 
a header and a content.... these are two pieces, the content-type is in one 
of them, and the content that it labels is in the other.

> whether the label needs to be within the message, not whether it needs to
> be within a particular portion of the message.
>
> For your example to be useful, you would need to show that the type of
> the data is correctly identified when that data are outside the message,
> yet still be identified correctly. To that end, the translation from
> content-type to the field extension of a specific piece of mail body
> content -- stored on its own and without the message -- would be a
> representative case.  In fact that is done with reasonable success,
> though certainly not always correct.

Here, I'm entirely lost about what you intended to say, and its relevance 
to the previous parts of the thread, so I'll not attempt a response.

> Perhaps you have not noticed that it is quite common to have Internet
> Mail stored in files in a form that is identical -- or, at least, very
> close -- to the form used in transit?
>

I have not questioned that at all. I've asked you (and I ask you again): 
What scenario do you have where a processor (that is, something that 
processes a message) needs to apply a different processing to an 
IMA-labelled message than to a non-IMA-labelled message?

This is about *processing* - that is, doing something about a message.
Having a message stored in a file is not processing.

>>> only be in the transit context (SMTP envelope) and not in the
>>> stand-alone email object.
>>
> sufficient.  The envelope is lost after transit, unless its information
> is added to the message.  It needs to be associated with the message data
> directly.  In other words, Harald, Return-path is an example of needing
> to have the information available outside the context of the transit.

I could have sworn that I was agreeing with you about Return-Path - both 
the fact that it's useful and the fact that it's a retrofit. Obviously I 
did not specify clearly enough that I agreed with you on those points.

> As for procmail, the fact that some software uses the envelope
> information does not have any relevance at all to this discussion.  No
> one said the envelope information was not useful.  So your example does
> not demonstrate that the information is not needed after transit.
> (Really, Harald, I strongly suggest you get a copy of "Crimes Against
> Logic" and read it carefully.)

I have, and I will.

And can you now please answer my previous question about processing?

                        Harald

---

From: Dave Crocker <[email-masked]>
Date: Tue, 06 Dec 2005 12:44:53 -0800

Harald,

> and irony is not made effective by ignoring the division of a message 
> into a header and a content.... these are two pieces, the content-type 
> is in one of them, and the content that it labels is in the other.

No, but focus of the current discussion definitely IS made effective.

As I said, you are trying to answer a question that was not asked.  The
question that was asked was about a "message", not about "part of a message
when it is separate from the message".

Hence, the distinction between header and body is not relevant.

>> Self-labeling of message content was the stated task.

My original text was "a message with a free-text body has a different
labeling than a message with a MIME body,"

So my original text was about a message, not about content that is not
currently part of the message.

That's not the same thing as you keep trying to focus on, Harald. You keep
wanting to consider identifying the content when it is outside of the
message.  That's not the issue that I originally raised.

> Here, I'm entirely lost about what you intended to say, and its 
> relevance to the previous parts of the thread, so I'll not attempt a 
> response.

Is the message in its transit context or is it separate from that context?
If the IMA label is only in the envelope, then it is only available during
transit.  It is not available after transit.

Further, my focus has been on a a message -- not content that is separate
from a message -- when the message is outside the context of transfer.

By contrast, the distinction you keep focusing on is an example of having
message content that is separate from (the context of) the message.
(Otherwise, your claim that the content-type label is "outside" does not
make sense.)

> IMA-labelled message than to a non-IMA-labelled message?

When does a processor need to know that the data are 7-bit ascii vs. unicode?

I would think that any processor that is going to render the data would need
to know that.  Are you claiming that the difference is self-identifying?

Especially if it is 7-bit net-ascii, rather than pure 7-bit ascii, I think
it unlikely that it is automatically self-identifying and think it likely
that failing to distinguish between them would not be so wonderful.  I will,
of course, defer to the unicode experts.

> Having a message stored in a file is not processing.

I was under the impression that data, without the ability to process it, was
not very useful.  Therefore discussing data in a file, without worrying
about its being processed, would not occur to me.

d/
-- 

Dave Crocker

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 06 Dec 2005 23:14:55 +0100

> People who write UAs for a living will have to speak to whether they

Gnus doesn't require the header field by default.  Gnus did for some
period of time, but we discovered in 2004 that some MUA/MTAs strip the
header, making it appear to our users as if Gnus is MIME-unaware.

One report suggested that Lotus Domino clients (version 5.0.6, if it
means anything to anyone) was one such buggy client.

To improve interoperability, Gnus will now guess whether the message
is MIME encoded based on the presence of other MIME headers.  This has
proven to be more reliable than to require the MIME-Version header
before applying any MIME treatment.

> It was supposed to give the rule for parsing the content-type header,
> but can occur before or after the content-type header, so if they
> actually use it for that, it means that they have to delay
> content-type header processing until after the whole header is
> parsed. Is that how people's code works?

That is not my experience.  All applications I've worked with store
the entire e-mail header in memory before looking at specific fields.

Hope this helps,
Simon

---

From: Martin Duerst <[email-masked]>
Date: Wed, 07 Dec 2005 15:53:19 +0900

[I'm likewise of two minds, or maybe just of no mind for the moment.]

As I have joined this list recently, this may be an old argument,
but what about thinking from the other end:

What are the scenarios where it is necessary to have such a header?
Technically, it would always suffice to check headers for UTF-8,
I assume.

Are there cases where that might go wrong, but a header would be
correct? What are they.

Are there cases where the overhead of checking each header for UTF-8
is significantly higher than checking for the specific header?

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 07 Dec 2005 18:50:20 +0900

>to know that.  Are you claiming that the difference is self-identifying?

The difference between 7-bit ascii and UTF-8 is definitely
self-identifying, unless you want to make a distinction between
data intended to be 7-bit ascii only and data that is UTF-8
but by chance doesn't contain any characters above U+007F,
and therefore looks like US-ASCII. But I don't think that
such a distinction would make any sense.

 >Especially if it is 7-bit net-ascii,

This is the first time I have seen the term 'net-ascii', although I
consider myself somewhat familiar with character encoding on the net :-).
Can you clarify?

 >rather than pure 7-bit ascii, I think

UTF-8 is extremely highly self-identifying. For some detailled
discussion, see e.g.
http://www.ifi.unizh.ch/groups/mml/people/mduerst/papers/PDF/IUC11-UTF-8.pdf
(rather old now). See in particular p.13 and following.

However, if we are talking about a couple characters e.g. in an
email address, AND we are talking about distinction from arbitrary
other encodings (e.g. EUC-xx, Shift_JIS, and so on), rather than
US-ASCII, then there is indeed a certain probability of false
positives.

Regards,    Martin.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 07 Dec 2005 09:27:37 +0100

--On 6. desember 2005 12:44 -0800 Dave Crocker <[email-masked]> wrote:

> to know that.

Thanks for coming up with a specific example!

> Are you claiming that the difference is self-identifying?

In fact UTF-8 is 100% self-identifying if the only two alternatives are 
UTF-8 and 7-bit ASCII stored in 8-bit bytes in the usual way - you simply 
check for whether anything has the high bit set.

For the current definitions, any processor capable of displaying an IMA 
message would also correctly display a traditional email message, since 
displaying the ASCII code positions is required to display UTF-8 correctly.

If there turns out to be any differences between an IMA object and a 
traditional mail object apart from the character set allowed in pieces of 
the header, it's an open question whether they are self-identifying - at 
the moment, I don't think any such differences have been proposed.

> of course, defer to the unicode experts.

query: what difference do you see between "7-bit net-ascii" and "pure 7-bit 
ascii" - ie which of them isn't correctly characterized by 
"charset=us-ascii"?

---

From: Dave Crocker <[email-masked]>
Date: Wed, 07 Dec 2005 07:59:26 -0800

>> I would think that any processor that is going to render the data would
>> need to know that.
> 

tsk, tsk, harald.

> UTF-8 and 7-bit ASCII stored in 8-bit bytes in the usual way - you 
> simply check for whether anything has the high bit set.

Ahh, good.  Then we do not need any of this protocol overhead people keep 
insisting on, for distinguishing between them.  We'll just let processing 
engines wander over the data and figure it out.

I guess we don't have to worry about UTF-8 that happens to contain only data 
in the netascii set.

Designing heuristics into basic protocol mechanism ought to become quite 
fashionable.

> query: what difference do you see between "7-bit net-ascii" and "pure 
> 7-bit ascii" - ie which of them isn't correctly characterized by 
> "charset=us-ascii"?

Take a look at the Telnet specification.

d/
-- 

Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Wed, 07 Dec 2005 08:05:19 -0800

> but by chance doesn't contain any characters above U+007F,

I do.  The idea of using heuristics for identitying data form strikes me as
poor protocol design.

> consider myself somewhat familiar with character encoding on the net :-).

Shortened form for "Network Virtual ASCII" as defined in Telnet (RFC 854).
When Internet protocols use 7-bit ASCII it is pretty much always netascii.

d/
-- 

Dave Crocker

---

From: Martin Duerst <[email-masked]>
Date: Thu, 08 Dec 2005 12:24:52 +0900

At 00:52 05/12/08, Dave Crocker wrote:
 >I do.

Could you explain why you want to do it?
Actually, I can at least see one good reason for not wanting
to do it: A message where all headers are 7-bit US-ASCII can
be sent with non-IMA SMTP, without any problems. But once it
is labeled as IMA, that won't happen. This is unnecessarily
reducing interoperability.

 >The idea of using heuristics for identitying data form strikes me as poor 
protocol design.

Is using a heuristic with a 100% success rate poor protocol
design? Would we even want to call that a heuristic?

 >Shortened form for "Network Virtual ASCII" as defined in Telnet (RFC 854).

Thanks. I have looked at RFC 854.

 >When Internet protocols use 7-bit ASCII it is pretty much always netascii.

I have found nothing in RFC 854 that is in any way relevant to the
current discussion, but just to be sure:

That RFC mentions the following code ranges:
- 95 USASCII graphics (codes 32 through 126): No prolem at all
- 33 USASCII control codes (0 through 31 and 127):
   Of these, NUL, LF, CR, BEL, BS, HT, VT, and FF are used.
   I don't think email headers use all of these, but none of them
   are a problem for the current discussion.
- 128 uncovered codes (128 through 255): These are the values that
   UTF-8 uses to represent non-ASCII characters, so here it gets
   interesting. Codes 240-255 are indeed used in the telnet protocol
   for escape sequences that transmit commands. Are you saying that these
   are part of what you call 7-bit net-ascii (I wouldn't call that 7-bit
   anymore in that case)? Are you saying that these escape sequences
   actually turn up in decent email headers?

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 08 Dec 2005 12:34:57 +0900

Well, if we use a specific header, processing engines will have to
find and parse that header. Which means that they have to wander
over the header data and figure it out.
Depending on the details, one or the other 'wandering over the data'
may be faster, but there shuldn't be a huge difference.

 >I guess we don't have to worry about UTF-8 that happens to contain only 
data in the netascii set.

Yes indeed. Unless for some reason, which I hope you can tell us,
you would think it is a feature that you can make sure that a plain
old mail (with all headers ASCII-only) always gets treated as an
IMA message.

Regards,    Martin.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 06 Dec 2005 17:58:09 -0000

> that all non-IMA-labelled messages are "legacy" conformant?

Where it might need to be downgraded/bounced before further transport
      N.B. the proposed SMTP enhancements do *not* give enough information
      to decide this.

In IMAP (almost certainly)

In Gateways (in both directions)

When stored in a file (as Dave has suggested) and it needs to be
re-inserted into the mail system, for whatever reason.

N.B. It is obviously possible to scan thrpough the headers looking for
bit-8, instead of having a label, but that is extra work - it all depends
how often it will need to happen.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 07 Dec 2005 19:14:53 -0000

> I do.  The idea of using heuristics for identitying data form strikes me  

I don't. If a pure ASCII message gets labelled as "IEE", no protocols will  
be broken, but eyebrows may be raised, time wasted in some agents, and  
even the message may be bounced or otherwise treated differently if some  
agent believes the label rather than the actual content.

But I agree that using heuristics to identify format is poor design. And  
even though checking for the lack of any 8th bit is not a heuristic it is  
still a waste of time if it has to be done often on the same message.

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Dec 2005 09:41:18 -0500

--On Tuesday, 06 December, 2005 23:14 +0100 Simon Josefsson
<[email-masked]> wrote:

>> It was supposed to give the rule for parsing the content-type
>> header, but can occur before or after the content-type
>> header, so if they actually use it for that, it means that
>> they have to delay content-type header processing until after
>> the whole header is parsed. Is that how people's code works?
> 
> That is not my experience.  All applications I've worked with
> store the entire e-mail header in memory before looking at
> specific fields.

But, FWIW, there certainly have been implementations that work
the other way.  Remember that we have email implementations
around that work in very constrained embedded environments in
which "read everything in at once" is a bad implementation
strategy -- especially in the presence of some of the very long
and drawn-out headers that have come into existence as the
result of various antispam and authentication strategies.

    john