Subject: [ietf-types] Comments on draft-melnikov-mime-default-charset-00.txt

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 14 Jun 2011 17:18:17 +0200

Hi,

  On http://www.ietf.org/id/draft-melnikov-mime-default-charset-00.txt

The first paragraph is horrible. It fails to properly separate the RFC
2046 context from the RFC 2616 context, and instead makes encoding de-
tection a property of the media type independently of context. That is
not how media types are used, it is common for instance to use them in
contexts where a character encoding is not used at all, for instance,
in a XHTML document with

  <style type='text/css'>
    ...
  </style>

there is no value used for the absent charset parameter, it's a notion
that does not make sense in the context. Similarily, I might make a
protocol similar to HTTP where all text/* is transported UTF-16 encoded
which would result in a similar situation. Even if that protocol had a
Content-Type header and allowed media type parameters, the charset para-
meter would have no influence on the bytes-to-text mapping.

There is also confusion about what RFC 2046 specifies. The default of
US-ASCII is for the text/plain type. While there are suggestions that it
"kinda" applies to other sub-types, for instance, like when handling an
unrecognized subtype, it should be handled like "plain", the language in
RFC 2046 leaves room for much interpretation and running code often does
not follow a particular interpretation like the one in the draft.

I think this should all be something more like "This memo changes RFC
2046 by defining and clarifying rules how to determine the character
encoding of resources labeled with a text subtype." and then say little
about the various sources of disputes and confusion.

In section 3 the notion that you always get to know the encoding, as is
implied by, for instance, "i.e. the charset information is transport
inside the payload", does not reflect current practise. Look, for text/
plain, and for many other subtypes, if there is no charset parameter,
then you don't know what encoding it is in. Maybe there are internal
markers, maybe there are not. Some formats do not even care, it's common
for "source code" for instance to just care about some special bytes and
ignore or pass verbatim the rest; some allow to mix encodings.

It seems to me the rules should rather be something like:

  1. If there is a higher-level protocol with rules, those rules win.

  2. If there is a charset parameter but it's malformed: out of scope.

  3. If there is a properly formed charset parameter, that wins. That
     wins even if the encoding is unsupported and there are other hints
     inside the document. That wins for all text subtypes, always.

  4. Otherwise it's type-specific and if the type specification does
     not define how you determine the encoding in this case, you don't
     know what the encoding is. For text/plain in particular, you do
     not know, it's not US-ASCII, you don't know.

I don't see why you would not define it this way, it sets expectations
reasonably well while also being relatively simple. There is no utility
for instance in having some text types that do not use the charset pa-
rameter at this level. You may very well have a format that just says it
is always UTF-8, but when you add charset=iso-8859-1 then some will take
the charset parameter as authoriative, possibly due to text/plain de-
faulting, so you get confusion and security vulnerabilities.

What should happen in a case like that is that the receiving software
recognizes the presence of a charset parameter with an encoding that is
not expected, like in the example, as indicating corruption and maybe a
deliberate attack against the system and respond as appropriate. It's a
bad idea to simply ignore the charset parameter and essentially hope the
other components of the system handle the situation safely even as they
are updated and replaced over time.

Leaving the US-ASCII default as-is seems rather silly even, there is so
much non-ASCII text/plain content without a proper label out there, it's
without informational value to say the default is US-ASCII beyond saying
not including a charset parameter is probably a bad idea (but plenty of
people do it anyway).

regards,
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 15 Jun 2011 00:00:25 +0200

> US-ASCII is for the text/plain type. While there are suggestions that it

No, it *really* applies to text/*.

See 
<http://greenbytes.de/tech/webdav/rfc2046.html#rfc.section.4.1.2.p.18> 
-- that's in the section 4.1.2 (Charset Parameter), not in 4.1.3 (Plain 
Subtype).

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 15 Jun 2011 00:13:58 +0200

>-- that's in the section 4.1.2 (Charset Parameter), not in 4.1.3 (Plain 

It's easy to get caught up on archaeological details like this, here I
might argue that this is for the Content-Type header in "Internet mail",
but as I noted, I do not think it's valuable to overanalyze what some
specification may have meant to specify 15 years ago. As I understand
it, the idea of the draft is come up with something coherent to replace
what's in there. I would prefer to keep the draft and the discussion as
close as possible to "This whole charset, content-type, mail vs http,
stuff sucks, but we are fixing this now saying X".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 15 Jun 2011 00:15:24 +0200

> stuff sucks, but we are fixing this now saying X".

I'm not disagreeing.

On the other hand, it's necessary that we agree on what needs to be 
fixed; and this certainly is something that does not to be fixed...

Best regards, Julian

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 15 Jun 2011 23:27:47 +0200

On 14 June 2011 17:18, Bjoern Hoehrmann wrote:

> it is common for instance to use them in contexts where a
> character encoding is not used at all, for instance, in a
> XHTML document with

> there is no value used for the absent charset parameter,
> it's a notion that does not make sense in the context.

Sorry, you lost me here, inline CSS as in your example uses
the document charset, doesn't it?  You can certainly have
non-ASCII in CSS comments or in, say, CSS "content" values.

For external CSS files the charset can be specified inline
with, e.g., @charset "us-ascii"; (begin of file).  Please
correct me if I'm wrong, my CSS knowledge is very limited.

I'd be surprised if the charset for inline CSS could be
different from the document charset -- if that's what you
are talking about it's IMO obvious, but maybe the new I-D
should mention the issue of "an immutable per document
default for inline text/css or text/javascript".

> a protocol similar to HTTP where all text/* is transported
> UTF-16 encoded which would result in a similar situation.

Now I'm confused, the HTTP default is ISO-8859-1, and that
is precisely what has to be fixed a.s.a.p.  As far as I'm
concerned US-ASCII also for HTTP is a solution; the HTML5
proposal "interpret ISO-8859-1 as windows-1252" is an odd
hack, but better than nothing; and more radical "let's just
say UTF-8 and be done with it" ideas didn't make it on the
HTTPbis list three years ago.

Are you talking about a hypothetical protocol with its own
UTF-16 default for text/*?  I'm not aware of any protocol
violating MIME for its purposes, only HTTP has this issue,
and the draft apparently tries to fix this.

> The default of US-ASCII is for the text/plain type.

No, already answered by Julian.  As you said RFC 2046 is
not a new RFC, and it should be updated to use the new
ABNF STD when the message format and SMTP RFCs are finally
full Internet standards (insert "errata exist" blurb here).

But just because the MIME RFC is old doesn't mean that it
is wrong, a compliment I would not extend to HTTP.  Maybe
the draft should get an i18n section as mandated by the
IETF charset policy, and explain why UTF-8 is *not* the
default for historical reasons (and practical reasons
wrt to old mail software not "yet" supporting 8BITMIME).

> "This memo changes RFC 2046 by defining and clarifying
>  rules how to determine the character encoding of
>  resources labeled with a text subtype."

Sounds good.  Hopefully inline definitions maintained by
the authors of resources can overrule whatever server
admins configure, unlike server admins the authors should
know which charset they use.

> for text/plain, and for many other subtypes, if there is
> no charset parameter, then you don't know what encoding
> it is in.

Of course you know this, because MIME specifies US-ASCII.

It could end up as UNKNOWN-8BIT (registered by IANA) if
the US-ASCII theory fails miserably, but that is another
story.  The text/plain case is a special case, because
there is no standard way to note the document charset
within the document -- or rather, I'm not aware of any
standard apart from Unicode "BOM" signatures, or the old
and forgotten ISO charset escape-sequences.

Other text/* types I'm aware of (text/xml, text/html,
text/css) permit this to some degree -- but there was
something special with text/xml, I forgot the details;
was it perhaps "text/xml MUST be US-ASCII"?

-Frank

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 15 Jun 2011 23:44:19 +0200

>default for inline text/css or text/javascript".

Your understanding above is correct. My point was that the context where
the media type label and the content occur in is important. RFC 2616 did
not "change the default", it spelled out rules for HTTP messages, which
is a different context than what RFC 2046 was about.

>know which charset they use.

(I am not sure what you are saying here. Right now the model is that
higher level information overrides lower-level information, so the
server administrator wins over the content maintainer. That's a bad
model as information about encodings, media types, is not stored in
typical file systems, and cannot be conveyed through some protocols,
so explicit out of band configuration is required, but that's what we
have got at the moment. That's not likely to change for old protocols.)

>Of course you know this, because MIME specifies US-ASCII.

I am saying that the draft maybe should specify "don't know" as that
reflects reality fairly well. I do understand that people think, and
quite rightly so, that you "need" to know the encoding, but that does
not mean the protocols must specify an answer even if that does not
correspond to reality.

>was it perhaps "text/xml MUST be US-ASCII"?

RFC 3023 specifies that text/xml "is" US-ASCII even though RFC 2616
says it "is" ISO-8859-1, that is probably what you mean here.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 16 Jun 2011 00:40:32 +0200

On 15 June 2011 23:44, Bjoern Hoehrmann wrote:

> My point was that the context where the media type label
> and the content occur in is important. RFC 2616 did
> not "change the default", it spelled out rules for HTTP
> messages, which is a different context than what
> RFC 2046 was about.

IBTD, RFC 2616 has a reference to RFC 2046, and I think
that should be a "normative reference" in a modern RFC.

And unlike the old "HTML i18n" RFC the HTTP RFC already
knew BCP 18 (listed in the references).  In other words,
it ignored MIME *and* the IETF charset policy for its
own purposes (= compatibilty with existing software at
the time of its publication).

[...]
> Right now the model is that higher level information
> overrides lower-level information, so the server
> administrator wins over the content maintainer. That's
> a bad model as information about encodings, media
> types, is not stored in typical file systems, and
> cannot be conveyed through some protocols, so explicit
> out of band configuration is required, but that's what
> we have got at the moment. That's not likely to change
> for old protocols.

Yes, that's what I meant.  The most specific info should
win, not the most general (server) info.  If exactly the
same resource is also available by FTP it could be valid
for FTP, and invalid for HTTP.  That is not only "bad",
it is plain wrong:  Hopefully HTTPbis fixes this, or at
least documents the fact of its own "unusabilty" for many
users unable to configure Web servers of a third party,
i.e., their Web hoster.

> I am saying that the draft maybe should specify "don't
> know" as that reflects reality fairly well.

Well, the MIME default is a last resort for "dunno", one
step before you are forced to sniff and guess.  Charset
sniffing does not work well for UTF-8 vs. ASCII-based
SBCS, if non-ASCII bytes occur too late in text/plain.

> I do understand that people think, and quite rightly
> so, that you "need" to know the encoding, but that does
> not mean the protocols must specify an answer even if
> that does not correspond to reality.

And that is the problem that has to be fixed for HTTPbis.
If the solution is "don't talk about it, use a normative
reference to MIME, and decree that the old ISO-8859-1
default turned out to be a bad idea" I'd like it better
than what we have now in RFC 2616.

Sadly this business also affects AUTH DIGEST, and it was
one of several reasons why SASL DIGEST was obsoleted.

The HTTP DIGEST RFC inherited the obscure HTTP default
charset.  From there this beast spread out, AUTH DIGEST
(ignoring the now potentially obsolete SASL DIGEST) is
used in other protocols, I vaguely recall "SIPPING".

> RFC 3023 specifies that text/xml "is" US-ASCII even
> though RFC 2616 says it "is" ISO-8859-1, that is
> probably what you mean here.

Yes, when a text/xml resource clearly states that it is
UTF-8 it shouldn't be rejected as invalid by validators
only because a server or browser transmitted the file
as text/plain.

This mess was caused by not strictly following MIME in
other RFCs.  Nothing is wrong with MIME -- other RFCs
trying to outsmart MIME failed miserably, and caused
unnecessary pains for many authors trying to create
"valid" documents.

-Frank

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jun 2011 15:23:08 +0200

> tection a property of the media type independently of context. That is

I think mentioning both here makes sense as RFC 2616 overrides the 
default set in 2046 (when used over HTTP).

> contexts where a character encoding is not used at all, for instance,

That's a good point. The same applies to <script> in (X)HTML, and AFAIR 
<atom:content>.

We should describe this use case somewhere, and point out that in cases 
like these, the charset parameter doesn't make sense.

Does this belong into *this* document, though?

Best regards, Julian