Subject: HTML5 vs content type sniffing

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jan 2008 12:08:23 +0100

Hi,

during the last HTML WG telecon 
(<http://www.w3.org/mid/20080124183402.GH18827%2540sideshowbarker>), I 
was asked to collect input from the IETF HTTPbis working group with 
respect to content type sniffing 
(<http://www.w3.org/html/wg/tracker/actions/44>).

So, please have a look at

   <http://www.hixie.ch/tests/adhoc/http/content-type/sniffing/>

and supply feedback on the expected results ("should be treated as" column).

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 25 Jan 2008 12:34:41 +0100

Julian Reschke wrote:
 
> and supply feedback on the expected results

I don't understand test case 11, I'd expect the same outcome as
for cases 9 and 10.

For test case 15 I don't get why it differs from test case 8.
Similar 16 vs. 9, and 17 vs. 10.  What are "invalid text/plain
characters" with respect to Latin-1, something about C0 or C1 ?

Treating certain case sensitive (sic!) variations of text/plain
or text/plain;charset=iso-8859-1 as erroneous depending on the
actual content is something that should never show up in an RFC,
it is only a hack (apparently designed for a popular Web server).

News from the UTF-7 front, there's a volunteering area director:
<http://permalink.gmane.org/gmane.ietf.apps-discuss/953>

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jan 2008 13:00:47 +0100

> I don't understand test case 11, I'd expect the same outcome as

That's because HTML5 tries to restrict content sniffing to precisely 
three content type header values, see 
<http://www.w3.org/TR/2008/WD-html5-20080122/#content-type-sniffing>.

> For test case 15 I don't get why it differs from test case 8.

Same reason.

> characters" with respect to Latin-1, something about C0 or C1 ?

Not sure. I just checked 016, and it doesn't contain any characters 
beyond 127.

> it is only a hack (apparently designed for a popular Web server).

I personally agree with that.

> <http://permalink.gmane.org/gmane.ietf.apps-discuss/953>

Sounds good.

BR, Julian

---

From: Robert Siemer <[email-masked]>
Date: Fri, 25 Jan 2008 23:22:15 +0100

> <http://www.w3.org/TR/2008/WD-html5-20080122/#content-type-sniffing>.

That's the best: html5 rules over what "text/plain" really means.

Apart from that, I would like to ask you all:

a) Which Content-Type makes pretty clear that the entity should be shown 
   "as-is" in a specified encoding?
   (Meaning: show anything as something character like, exception: make 
   line breaks (where? In case of doubt, make one))
   [my "as-is" is what browsers do when showing "text without 
   interpretation"]

b) I request that any browser offers a possibility to show whatever it 
   receives "as-is" (as described in a)), the same way a browser can be 
   made to save whatever it receives in a file.
   Having a browser like that: what is the difference in
   "text/plain" being interpreted as "text/plain" or 
   "application/octet-stream"?
   [see html5 content sniffing to know why I care]
   The only difference I see is wheter I get the "save as" dialog 
   automatically or not...

c) Knowing that it is easy to make a browser treat something as 
   "application/octet-stream", but that it is not easy to make it handle 
   content as "text/plain" (or "as-is"): Is it desirable to make a 
   browser treat "text/plain" labled stuff as 
   "application/octet-stream"?

Robert

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 26 Jan 2008 12:18:50 +0100

> That's the best: html5 rules over what "text/plain" really means.

While they're at it...  overrule CHARMOD, replace IANA by a Wiki
page, use an undocumented URL scheme, silently deprecate XHTML,
claim that breaking backwards compatibility everywhere is about
IE bugs, etc.  Please let's say that HTML5 is off topic here :-(

---

From: Ian Hickson <[email-masked]>
Date: Mon, 28 Jan 2008 04:54:16 +0000 (UTC)

On Sat, 26 Jan 2008, Frank Ellermann wrote:
> While they're at it...  overrule CHARMOD, replace IANA by a Wiki page, 
> use an undocumented URL scheme, silently deprecate XHTML, claim that 
> breaking backwards compatibility everywhere is about IE bugs, etc.  

In all fairness, we're not deprecating XHTML. Indeed we're not deprecating 
anything; in HTML5 either something is allowed or not, there's no 
"deprecated" limbo where a feature is allowed but planned for obsoletion.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 29 Jan 2008 11:53:49 +0100

>> page,

 From http://ln.hixie.ch/?start=1144794177&count=1
"Unfortunately, we're now at a stage where browsers are continuously  
having to reverse-engineer each other to determine why they are  
handling content differently. A browser can't afford to render any  
less content than a browser with more market share, because otherwise  
users won't switch, and the new browser will not be adopted."

By that argument, browsers with largest market share would profit  
from changing their content sniffing rules continously. Hmm, somthing  
is missing here...

I think a better description would be that browsers continously try  
to adapt to new/faulty server configurations. This is a process that  
will go on as long as the internet evolves. And it is not under  
control of browser makers. HTML5 special rules for "+xml" content  
types had no need 5 years ago and who knows what will be needed next?  
Tackling this process from the "we define what browser currently do  
as standard" seems to be a strange approach.

In the case of "text/plain" apache httpd is still (2.2.6) shipping  
with DefaultType set to it, ignoring the rules set up by RFC 2616  
(which seem to be unchanged in httpbis as far as i can see). So, if  
the apache defaults are changed, will whatwg have to change the  
sniffing "standard"? most likely.

//Stefan

--
<green/>bytes GmbH, Hafenweg 16, D-48155 Münster, Germany
Amtsgericht Münster: HRB5782

---

From: Robert Siemer <[email-masked]>
Date: Tue, 29 Jan 2008 13:24:28 +0000

> sniffing "standard"? most likely.

Where does RFC2616 set up rules on how Apache has to serve
its documents? A section mandating on how to deduce the Content-Type of
arbitrary files would be new to me.

Robert

---

From: Robert Siemer <[email-masked]>
Date: Tue, 29 Jan 2008 13:23:25 +0000

> sniffing "standard"? most likely.

The whole "text/plain" vs. "application/octet-stream" sniffing as 
mentioned in html5 is brain dead.

The algorithm shown there has no stable results e.g. regarding content 
served as "text/plain", because it works with the first N bytes 
available. - The browser always has the excuse of not having any bytes 
of the entity (--> result is "text/plain"). Using a larger N, the larger 
the likelyness of recognizing e.g. random data as 
"application/octet-stream".

But whole sniffing is useless, because the html5 authors do not say 
what do to with the one or the other. - The real difference in todays 
browsers is

a) show it
b) download it

Something the content author wants to dictate, something that is 
absolutly independent of the byte representation. Something that the 
proposed algorithms is not able to detect: semantics.

Robert

---

From: Robert Siemer <[email-masked]>
Date: Tue, 29 Jan 2008 14:26:33 +0100

Robert

---

From: Robert Siemer <[email-masked]>
Date: Tue, 29 Jan 2008 14:27:23 +0100

a) show it
b) download it

Robert

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 29 Jan 2008 14:44:00 +0100

> arbitrary files would be new to me.

http://www.greenbytes.de/tech/webdav/rfc2616.html#entity
7.2.1 Type
[...]
Any HTTP/1.1 message containing an entity-body SHOULD include a  
Content-Type header field defining the media type of that body. If  
and only if the media type is not given by a Content-Type field, the  
recipient MAY attempt to guess the media type via inspection of its  
content and/or the name extension(s) of the URI used to identify the  
resource. If the media type remains unknown, the recipient SHOULD  
treat it as type "application/octet-stream".

This defines the way how entities where the content-type is not known  
are handled. I hear that apache has changed its implementation in the  
dev trunk already. So, httpd will by default not send a content-type  
for "unkown" resources.

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 29 Jan 2008 15:08:43 +0100

Stefan Eissing wrote:

>| 7.2.1 Type
[...]
>| If and only if the media type is not given by a Content-Type field,
>| the recipient MAY attempt to guess the media type via inspection
>| of its content and/or the name extension(s) of the URI used to
>| identify the resource. If the media type remains unknown, the
>| recipient SHOULD treat it as type "application/octet-stream".

Attempting to outsmart MIME, apparently.  For MIME it is straight
forward, no Content-Type is a shorthand for text/plain US-ASCII,
and if that is obviously wrong recipients will manage to treat it
as appication/octet-stream or x-unknown/x-garbage without SHOULD.

 Frank

---

From: Ian Hickson <[email-masked]>
Date: Wed, 30 Jan 2008 22:43:07 +0000 (UTC)

On Tue, 29 Jan 2008, Stefan Eissing wrote:
> "Unfortunately, we're now at a stage where browsers are continuously having to
> reverse-engineer each other to determine why they are handling content
> differently. A browser can't afford to render any less content than a browser
> with more market share, because otherwise users won't switch, and the new
> browser will not be adopted."
> 
> By that argument, browsers with largest market share would profit from 
> changing their content sniffing rules continously. Hmm, somthing is 
> missing here...

The browser the largest market share is already deployed; its behaviour 
can't be changed. The next version of the browser with the largest market 
share has no market share. That's one of the reasons Microsoft developed 
their new <meta> tag that requires pages to opt-in to particular sets of 
bugs -- they want to guarentee that when they release IE8, it can render 
exactly everything that IE7 can render. If IE8 had different behaviour 
than IE7, then people wouldn't adopt it.

Now, Microsoft are obviously taking this to an extreme that I think is 
beyond necessary, but hopefully you understand why it makes no sense to 
say that "browsers with largest market share would profit from changing 
their content sniffing rules continously".