Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard

From: Ian Jackson <[email-masked]>
Date: Tue, 10 Aug 2004 12:19:22 +0100

The IESG writes ("Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard "):
> The IESG has received a request from an individual submitter to consider the 
> following document:
> 
> - 'The APPLICATION/MBOX Media-Type '
>    <draft-hall-mime-app-mbox-02.txt> as a Proposed Standard
> 
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action.  Please send any comments to the
> [email-masked] or [email-masked] mailing lists by 2004-09-06.

I have the following comments:

 * This specification is incomplete.  There are unresolved issues
   regarding the semantics of the format.

 * Since mbox files are text files (assuming that any binary messages
   in the mailbox are themselves encoded) and can be read sensibly
   with the naked eye, the content type should be text/* not
   application/*.  This will also remove ambiguity surrounding line
   endings.

 * Since an mbox is actually an aggregate type - a way of encoding a
   set of RFC822 messages - transfer encodings other than 7bit and
   8bit should be discouraged.  The spec should probably deprecate
   them in most cases.

 * The Proposed Standard should either include or refer to a specific
   mbox format.  The fact that there are variant implementations
   doesn't mean that the Proposed Standard should hesitate to declare
   those broken (at least, broken when a file is sent as text/mbox).
   Those variant implementations are not wholly interoperable anyway,
   and in order to write software which deals correctly with text/mbox
   it will be necessary for the spec to say what the format is
   supposed to be !

 * The format specified should be that described in Rahu Dhesi's
   posting to comp.mail.misc in 1996, <[email-masked]>.

 * If an mbox file contains messages with unencoded binary data, the
   file is difficult to sensibly process on a machine with non-UN*X
   line-endings, because of the bare CRs in the binary data.  (Bare
   LFs are fine and look just like line endings, with From_-escaping
   and all.)  As far as I can tell there is then no non-lossy
   representation of the file which allows sensible local processing
   by non-mbox-specific tools.  This issue should be resolved (or at
   least acknowledged).

Ian.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 10 Aug 2004 11:14:21 -0400

Thanks for the comments.

If we start from the premise that messages in mbox files are RFC2822
objects, then your conclusions and arguments are appropriate and correct.
However, that premise is demonstrably false, and most of the conclusions
which follow that premise are also false.

Let's be clear about this: mbox files ARE NOT collections of 2822 objects,
but instead are app-specific databases of app-specific message objects. As
such, the right way to handle them is to treat them as opaque databases,
just as if they ~Eudora databases, or ~Outlook databases, or anything
else, and to treat them accordingly.

As a simple example of this principle, mbox files often contain relative
email addresses with no domain qualifier (especially common with users who
exchange messages with other local users, but this isn't limited to that
context). OTOH, section 3.4.1 of RFC2822 unambiguously states that email
addresses (those that use addr-spec) must be qualified. So, starting with
this message from the E2E-Interest archives:

ftp://ftp.isi.edu/end2end/end2end-interest-1985-1987.mail

| Date: Wed, 3 Jun 87 13:18:28 PDT
| From: Bob Braden <braden>
| Message-Id: <[email-masked]>
| To: end2end-interest
| Cc: postel

What should an import agent do when it finds that, and wants to import it
into an ~IMAP folder, or wants to ~remail the message? should it append
the local domain name? should it append a default domain name? should it
reject the message because it ~doesn't conform to 2822 and because no
assumptions are safe? Clearly, the message is not conformant to 2822, and
the data has to be analyzed in the context of the application which
created it, rather than the application-neutral format that we all wish
was being used. From that position, out-of-band negotiation over the
formatting of the database is the only thing that will work.

Addresses are a minor example, and one that a lot of folks would gladly
brush off as irrelevant, but there are a lot more examples, some of which
are much more significant. Since the messages are not 2822 compliant
objects, there is no gurantee (or even any reasonable assumption) that
binary objects have been previously encoded into a safe format. Messages
can easily contain untagged 8-bit characters, bare CR and/or LF, and can
be thousand characters in length, all of which violates other assumptions
about 2822 formatting. 998-character line-lengths would destroy the data.
Automatic EOL conversion (as implied by text/* media-type definitions)
would destroy the data. And so forth.

What if two local users are exchanging big5-encoded messages, but neither
mailer has tagged the messages with the appropriate MIME tags (not
uncommon, and not a requirement, since these are not 2822 compliant, nor
even MIME-compliant). What assumptions should an importer make? The only
safe thing here is out-of-band negotiation ("these are big5").

It would be nice if mbox files only contained 2822 objects. But they
don't. They don't even have to conform to any one set of assumptions, and
can contain messages in different charsets (all of which are untagged), or
which have different relative domains (all of which are unspecified), or
any number of other variances. As such, they have to be treated as opaque
databases, not as collections of well-formed 2822 objects.

I also note that the digest media-type is already specified and is the
appropritate interchange format if you actually do have a collection of
well-formed 2822 objects. But if you have an mbox file, you have to
exchange it as an opaque database, and you have to delineate any internal
assumptions through out-of-band negotiations. The mbox media-type is for
use with tagging and identifying the data being exchanged ("here is an
opaque database of unspecified message objects") only.

Detailed responses to points follow:

>    regarding the semantics of the format.

This is not intended to serve as an authoritative reference to the mbox
database format, but is only intended to provide an identifier for the
database-type when it is transferred. Out-of-band negotiation is necessary
in all cases anyway, and I don't really think it's appropriate for the
IETF to define an application-specific database format anyway.

I'd like to see one, and I'd like to see whatever *NIX consortium is
responsible for such things get together and define one.

>    application/*.  This will also remove ambiguity surrounding line

Automatic EOL conversion destroys message objects. There is no assumption
that messages are conformant with 2822, nor can there be any such assumption.

>    8bit should be discouraged.  The spec should probably deprecate

I don't see why. It may be efficient to append an app/mbox database to a
[2822-compliant] email message, and to encode is as QP, or it may be
necessary to encode it as base64 if it contains long lines of raw 8bit data.

>    it will be necessary for the spec to say what the format is

Well, this would require that every local application be changed to suit
the needs of the transfer format, which is not reasonable. Instead the
goal here is to define a transfer identifier which says that the data
probably looks like XYZ.

>    posting to comp.mail.misc in 1996, <[email-masked]>.

The message with that message-ID does not define a format.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Tim Bray <[email-masked]>
Date: Tue, 10 Aug 2004 09:53:58 -0700

>    application/*.  This will also remove ambiguity surrounding line

I'm not very close to this problem, but I'd like to point out that 
using text/anything gets you into i18n issues... if the server has 
trouble getting the charset= right and doesn't provide it, then the 
rules say the client MUST assume US-ASCII, which in a high proportion 
of cases is going to be wrong; and by the way the client is forbidden 
to use any other heuristics to figure out what the encoding might be.  
text/* also blesses intermediate transcoders which can create all sorts 
of problems.

Maybe these issues don't apply to mbox data, in which case feel free to 
ignore me -Tim
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Philip Guenther <[email-masked]>
Date: Tue, 10 Aug 2004 19:59:21 -0700

Eric A. Hall <[email-masked]> writes:
...
>opaque database of unspecified message objects") only.
...
>IETF to define an application-specific database format anyway.

If there are no defined semantics for the content of an application/mbox
part, how does the type differ from application/octect-stream?  In both
cases you have to look to out-of-band info to interpret the data.
Indeed, there appear to be no requirements at all on the content.  If
successful uses of this content-type effectively requires private
arrangement, why does it need a standard content-type instead of each
such exchange using a content-type tailored to the circumstance and
taken from the "vnd.", "prs.", or "x." trees.  How does use of
application/mbox over application/octet-stream or some other
content-type improve interoperability?

...
[regarding creating a spec for a mailbox file format]
>responsible for such things get together and define one.

At that point, would application/mbox be updated to refer to said spec,
rendering non-compliant some chunk of the previous uses, or would a new
content-type be specified?

Philip Guenther

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 11 Aug 2004 10:16:31 -0400

On 8/10/2004 10:59 PM, Philip Guenther wrote:

> part, how does the type differ from application/octect-stream?

It provides an identifier for the content, so that transfer agents can
perform specific tasks against the data (such as importing or searching a
remote mailstore, or handing the data to an agent that knows what to do
with it). The agent still needs to deal with content-specific issues like
determining the EOL markers, applying default domains to relative
addresses, and so forth. That's a pretty common separation of powers;
application/postscript doesn't relieve the system from needing a
postscript interpreter, and we leave things like ~version tags for the
content agent to worry about instead of the transfer agent.

> content-type be specified?

Given that the current proposal specifies minimal formatting (essentially
being limited to the likely presence of some kind of From_ line), I'd
think that a reasonably authoritative spec could be referenced in an
update to this proposal. It would depend in large part on the depth and
comprehensiveness of the specification, I'd imagine.

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 11 Aug 2004 08:44:43 -0600 (MDT)

> From: "Eric A. Hall" <[email-masked]>
>

I cannot understand that except as saying that this document explicitly
and intentionally does not provide enough information for the recipient
of a message defined by the document to decode the message.  I understand
that statement as saying that the data is essentially opaque.

Isn't the point of any RFC on the standards track to promote
interoperability?  What good is an RFC that says "consult as yet
unwritten specifications from undetermined sources to handle the data
standardized by this RFC"?   Isn't the first sanity test of a standard
whether one can determine if an implementation is compliant?  As far
as I can see, Eric Hall is saying that compliance of senders and
receivers of such messages could not be determined until some.

Aren't there already enough opague application data MIME types?

Vernon Schryver    [email-masked]