Subject: Re: Unknown text/* subtypes

From: Martin Duerst <[email-masked]>
Date: Wed, 26 Dec 2007 10:22:47 +0900

At 03:17 07/12/19, Frank Ellermann wrote:
>Julian Reschke wrote:
>
>> It would be nice if somebody could provide some insight why this ever 
>> made it into HTTP. Was that just an attempt to allow text/html encoded 
>> in latin1 to be served without charset parameter?

Yes, in some ways. The Web was started at CERN in Geneva, and at
that time, iso-8859-1 seemed like a forward-looking choice allowing
to cover not only the US, but also (most of) Western Europe.
The first versions of HTTP (HTTP 0.9 or before) didn't have any
version indication, didn't allow a charset parameter, and also
didn't have any request or response headers. Responses were just
HTML, nothing else. For a short summary,
please see http://www.w3.org/Protocols/HTTP/AsImplemented.html.

HTTP 0.9 was later generalized into HTTP 1.0 and HTTP 1.1 as we
know it. For quite some time, there were a lot From [email-masked] Tue Dec 25 20:33:21 2007
Date: Wed, 26 Dec 2007 10:22:47 +0900
To: Frank Ellermann <[email-masked]>,
        [email-masked]
From: Martin Duerst <[email-masked]>
Cc: [email-masked]
Subject: Re: Unknown text/* subtypes
Archived-At: <http://www.w3.org/mid/6.0.0.20.2.20071226100422.0955a080@localhost>
Resent-From: [email-masked]
Sender: [email-masked]
Resent-Sender: [email-masked]
Resent-Message-Id: <[email-masked]>
Resent-Date: Wed, 26 Dec 2007 01:31:13 +0000

At 03:17 07/12/19, Frank Ellermann wrote:
>Julian Reschke wrote:
>
>> It would be nice if somebody could provide some insight why this ever 
>> made it into HTTP. Was that just an attempt to allow text/html encoded 
>> in latin1 to be served without charset parameter?

Yes, in some ways. The Web was started at CERN in Geneva, and at
that time, iso-8859-1 seemed like a forward-looking choice allowing
to cover not only the US, but also (most of) Western Europe.
The first versions of HTTP (HTTP 0.9 or before) didn't have any
version indication, didn't allow a charset parameter, and also
didn't have any request or response headers. Responses were just
HTML, nothing else. For a short summary,
please see http://www.w3.org/Protocols/HTTP/AsImplemented.html.

HTTP 0.9 was later generalized into HTTP 1.0 and HTTP 1.1 as we
know it. For quite some time, there were a lot of clients out
there that badly choked on charset parameters.

So it wasn't that the default was an attempt to save some bytes
for Latin-1, but that it was in some way necessary to be backwards-
compatible with very early versions not documented as RFCs.

Such backwards compatibility is no longer necessary, fortunately.
The situation currently on the Web is different. The actual
'default' used by browsers isn't simply iso-8859-1, it's whatever
the user has set as his/her preferred encoding, or whatever the
setting of the specific language version is. This means that in
essence, there is NO default. The HTTP spec clearly should be
fixed to say so.

>Some parts of this puzzle:  RFC 2070 introduced an "ideally anything
>is Unicode" concept, later adopted by HTML 4+, XHTML 1+, and XML 1+.
>AFAIK HTML 3.2 and maybe also HTML 3 still didn't have this feature.
>
>As far as RFC numbers mean something 2070 was published "after" 2068,
>both say January 1997, and "the law" 2277 was clearly a year later.

At least 2070 and 2277 were in the works for a really long time.
That may also apply to 2068.

>RFC 2068 (HTTP/1.1) was the successor of 1945 (HTTP/1.0, May 1996),
>2070 (HTML i18n) was the successor of 1866 (HTML 2, November 1995).
>
>Tim Berners-Lee, one co-author of RFC 1866 and 1945, wrote in 1866:
>
>| NOTE - To support non-western writing systems, a larger character
>| repertoire will be specified in a future version of HTML. The
>| document character set will be [ISO-10646], or some subset that
>| agrees with [ISO-10646]; in particular, all numeric character
>| references must use code positions assigned by [ISO-10646].

That was put in because the HTML WG at that time already more
or less understood (after a lot of discussions) that the direction
to go was Unicode/ISO 10646. A lot of the work on HTML 2.0 and
HTML i18n (RFC 2070) and some other pieces was going on somewhat
in parallel.

>Speculation, in May 1996 it made sense that HTTP/1.0 can transport
>HTML 2 "as is", default Latin-1, and it took Harald and Martin some
>months to fix this in RFC 2070 and 2277, too late for RFC 2068, and
>RFC 2616 simply inherited "default Latin-1" wholesale. 

It wasn't just Harald and me. It was a lot more people, in particular
all the coauthors of RFC 2070 and 2277.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Martin Duerst <[email-masked]>
Date: Wed, 26 Dec 2007 10:29:29 +0900

At 00:45 07/12/19, Frank Ellermann wrote:
>Julian Reschke wrote:
> 
>> there's also RFC2616
>
>Yes, that's an ugly legacy exception...  
> 
>> <http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i20>
>
>...maybe 2616bis can drop this oddity in favour of a simple
>"unknown text is ASCII" rule.

The new version of the HTTP spec, 2616bis, should definitely
drop the iso-8859-1 default, but NOT in favor of "unknown text is ASCII".
It should just say that there is no default.
There is a big difference between these two, especially
for document formats that contain internal 'charset' information.
A default of US-ASCII makes document-internal 'charset' information
useless (because the external information wins). No default means
that the recipient will look at the internal information.

>HTTP oddities shouldn't affect
>MIME registrations, there's no string "2616" in BCP13.

One reason for the problems with text/xml was that the
original MIME default of US-ASCII was enforced. This made
it impossible to serve XML documents with internal 'charset'
information only as text/xml.

Regards,    Martin.

 1.22 ESMTP Server]
	id <S2A211B> for <[email-masked]> from <[email-masked]>;
	Wed, 26 Dec 2007 10:26:39 +0900
Date: Wed, 26 Dec 2007 10:29:29 +0900
To: Frank Ellermann <[email-masked]>,
        [email-masked]
From: Martin Duerst <[email-masked]>
Cc: [email-masked]
Subject: Re: Unknown text/* subtypes (was: Request for review of Turtle   (an RDFserialization) media type: text/turtle)
Archived-At: <http://www.w3.org/mid/6.0.0.20.2.20071226102314.083e2170@localhost>
Resent-From: [email-masked]
Sender: [email-masked]
Resent-Sender: [email-masked]
Resent-Message-Id: <[email-masked]>
Resent-Date: Wed, 26 Dec 2007 01:31:09 +0000

At 00:45 07/12/19, Frank Ellermann wrote:
>Julian Reschke wrote:
> 
>> there's also RFC2616
>
>Yes, that's an ugly legacy exception...  
> 
>> <http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i20>
>
>...maybe 2616bis can drop this oddity in favour of a simple
>"unknown text is ASCII" rule.

The new version of the HTTP spec, 2616bis, should definitely
drop the iso-8859-1 default, but NOT in favor of "unknown text is ASCII".
It should just say that there is no default.
There is a big difference between these two, especially
for document formats that contain internal 'charset' information.
A default of US-ASCII makes document-internal 'charset' information
useless (because the external information wins). No default means
that the recipient will look at the internal information.

>HTTP oddities shouldn't affect
>MIME registrations, there's no string "2616" in BCP13.

One reason for the problems with text/xml was that the
original MIME default of US-ASCII was enforced. This made
it impossible to serve XML documents with internal 'charset'
information only as text/xml.

Regards,    Martin.

---

From: Eric Prud'hommeaux <[email-masked]>
Date: Wed, 26 Dec 2007 10:10:41 -0500

* Martin Dürst <[email-masked]> [2007-12-26 10:29+0900]
> that the recipient will look at the internal information.

I'm fuzzy on the logic here. It seems one of the big values of text/
tree is that you know the browser will render it on the screen if it
doesn't know anything about the subtype. How then will it know where
to look for internal charset info? (By the same token, how then will
it know what the default charset is for the mystery media type?)

Don't get me wrong, I'm ok with the browser making a guess and the
user overriding it. In fact, I'd like to see the first guess be utf-8
if the stream appears to be valid utf-8. And if it's really shift-JIS
or utf-16, you'll get some funny looking chars (hopefully the browser
won't be sensitive to misinterpreting something as "^drm -r /").

Are we on the cusp of a new HTTP1.1 draft? May I register media types
that pre-empt that this a little in favor of doing the right thing?

> #-#-#  Martin J. Dürst, Assoc. Professor, Aoyama Gakuin University

-- 
-eric

office: +1.617.258.5741 NE43-344, MIT, Cambridge, MA 02144 USA
mobile: +1.617.599.3509

([email-masked])
Feel free to forward this message to any list for any purpose other than
email address distribution.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Fri, 28 Dec 2007 09:51:59 +0100

On Fri, 28 Dec 2007 03:26:46 +0100, Frank Ellermann  
<[email-masked]> wrote:
> The odd text/xml case is different, there's a MUST somewhere
> in the text/xml spec.  But nobody treats text/html as "default
> Latin-1" ignoring the internal declaration.

FWIW, I don't know of any software (apart from maybe the Universal Feed  
Parser) that doesn't treat text/xml as it treats application/xml.

-- 
Anne van Kesteren
<http://annevankesteren.nl/>
<http://www.opera.com/>

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 28 Dec 2007 13:05:54 +0100

Anne van Kesteren wrote:
 
> I don't know of any software (apart from maybe the Universal Feed  

The W3C validator does this, and when I whined Olivier quoted the
RFC about text/xml, with a link to your text/xml blog entry... :-)
-- 
http://article.gmane.org/gmane.org.w3c.validator/9908/match=text+xml