Subject: Re: HTML5 vs content type sniffing

From: Julian Reschke <[email-masked]>
Date: Sat, 02 Feb 2008 12:42:03 +0100

Ian Hickson wrote:
> ...
> The browser the largest market share is already deployed; its behaviour 
> can't be changed. The next version of the browser with the largest market 
> share has no market share. That's one of the reasons Microsoft developed 
> their new <meta> tag that requires pages to opt-in to particular sets of 
> bugs -- they want to guarentee that when they release IE8, it can render 
> exactly everything that IE7 can render. If IE8 had different behaviour 
> than IE7, then people wouldn't adopt it.
> ...

Following that reasoning, people wouldn't adopt IE7 instead of IE6. But 
they do. May be not as many and as happily as Microsoft would like, but 
they do.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 02 Feb 2008 12:46:20 +0100

Thanks for the feedback so far. Below is an attempt to summarize what's 
already been said, and also some more feedback from myself:

Related to the test cases themselves:

1) Content-Encoding vs sniffing

The tests at 
<http://www.hixie.ch/tests/adhoc/http/content-type/sniffing/> are 
somewhat broken; case 8 through 10 are supposed to trigger content 
sniffing (as per HTML5, 
<http://www.w3.org/TR/2008/WD-html5-20080122/#content-type-sniffing>), 
but don't, as the server sends the response with Content-Encoding: gzip 
(see <http://lists.w3.org/Archives/Public/public-html/2008Jan/0235.html>).

FF2 and FF3 beta currently do not implement sniffing in this case, 
matching what the spec says. Others apparently do. The fact that FF does 
not could be taken as an argument that it's not needed to "not break 
existing content".

2) Character sets vs sniffing

The spec currently requires sniffing for "text/plain; 
charset=iso-8859-1" and "text/plain; charset=ISO-8859-1", assuming that 
those servers that do send an incorrect default content type always send 
it with a very specific character set name. It appears that some servers 
sometimes ship with other defaults, thus more character sets would need 
to be considered 
(<http://lists.w3.org/Archives/Public/public-html/2008Jan/0239.html>). 
Where do you draw the line?

3) "illegal characters"

Some test cases, such as 16, claim the contents contains "invalid 
text/plain characters". At least case 16 doesn't. 
(<http://lists.w3.org/Archives/Public/ietf-http-wg/2008JanMar/0122.html>)

4) other type of sniffing

HTML5 defines other types of sniffing (such as unknown -> PDF) that 
aren't covered by these tests, and haven't been discussed within this 
thread.

Related to the topic of content sniffing in general:

5) content-type default

It seems in general Apache httpd is blamed for having caused the 
original problem (content being served with wrong default content-type 
instead of no content-type at all). In the meantime, httpd supports a 
default type of "none" 
(<http://lists.w3.org/Archives/Public/public-html/2008Jan/0258.html>), 
so at least the right steps have been made to get rid of the problem in 
the future.

6) conflict with Webarch and TAG finding

The current text in HTML5 contradicts WebArch 
(<http://www.w3.org/TR/webarch/#error-handling>) and the TAG finding 
"mime respect", in particular "avoid silent recovery" 
(<http://www.w3.org/2001/tag/doc/mime-respect.html#silent-recovery>).

There seems to be broad agreement that it's good to document what widely 
deployed user agents actually do with respect to content sniffing. 
However, there was *no* agreement that it's HTML5's task to make that a 
"MUST" level requirement 
(<http://lists.w3.org/Archives/Public/public-html/2008Jan/0214.html>).

Also, if it's still the goal to reduce the amount of content where 
content sniffing takes place, then it would be useful to make it easier 
for an author to actually find out that content sniffing took place. 
Thus, user agents that do content sniffing SHOULD offer a way to (1) 
turn if off and/or (2) notify the user when the UA decided to override 
the specified content type 
(<http://lists.w3.org/Archives/Public/public-html/2008Jan/0260.html>).

More feedback appreciated,

Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 02 Feb 2008 13:08:38 +0100

> (<http://lists.w3.org/Archives/Public/public-html/2008Jan/0260.html>).

Surprise: IE7 has that option - see 
<http://blogs.msdn.com/ie/archive/2005/02/01/364581.aspx#364853> (and 
yes, it's also available through the UI).

BR, Julian

---

From: Ian Hickson <[email-masked]>
Date: Sat, 2 Feb 2008 20:15:36 +0000 (UTC)

On Sat, 2 Feb 2008, Julian Reschke wrote:
> > exactly everything that IE7 can render. If IE8 had different behaviour than
> > IE7, then people wouldn't adopt it.
> > ...
> 

So few people have adopted IE7 instead of IE6 (as opposed to other 
browsers instead of IE6, or even more commonly, just staying with IE6), 
and at such a slow rate, that Microsoft have decided to both make IE7 a 
required security update _and_ make IE8 ship with IE7's renderer, and use 
that renderer by default. And that's with just the minor mistakes in IE7; 
if browsers did what was proposed earlier in this thread, one can easily 
predict the situation would be much worse (from Microsoft's perspective).

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Julian Reschke <[email-masked]>
Date: Sat, 02 Feb 2008 21:52:14 +0100

> browsers instead of IE6, or even more commonly, just staying with IE6), 

Few? <http://en.wikipedia.org/wiki/Internet_Explorer> claims that in 
January 2008, market shares were 42.93% for IE7 and 32.30% for IE6.

I understand that it's hard to get reliable statistics, and I'd 
personally would *love* to see market shares for other browsers being 
higher.

But how can you say "few people have adopted IE7 instead of IE6" when 
the market share is that high? Do you have drastically different numbers?

BR, Julian

---

From: Ian Hickson <[email-masked]>
Date: Sat, 2 Feb 2008 21:03:38 +0000 (UTC)

> the market share is that high? Do you have drastically different 

To make a fair assessment of upgrade adoption rate, you have to discount 
IE7 installs due to bundling with new computers, and you have to compare 
the rate of adoption against other browsers. For example, Safari 2 to 
Safari 3, or Firefox 1.5 to 2.0. Unfortunately, as you say, it is hard to 
get reliable public statistics.

However, that the second most widely used browser today is one that 
shipped over half a decade ago, despite it being obsolete in its own 
product line _and_ having been marked for auto-upgrade to the latest 
version by its manufacturer, is a pretty clear indication of what I am 
talking about.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 02 Feb 2008 22:18:12 +0100

> IE7 installs due to bundling with new computers, and you have to compare 

Why? I'm not sure why it would be relevant why somebody uses a specific 
browser. What counts is that he or she does. If I replace a Win2000 
machine with IE6 by a XP machine with IE7, and now use IE7 instead of 
IE6, how is that not "adoption"?

(My new desktop machine came with IE7 bundled, but that will not affect 
its market share, because my default browser is FF3.)

> get reliable public statistics.

I'm not disputing that the adoption rate *inside* the IE browser family 
isn't that great. There may be many other reasons for it, such as IE7 
not being available for pre-XP systems.

But even with IE7 not being available (or usable) on older Windows 
machines, it still has the highest market share today (according to at 
least one statistic).

> shipped over half a decade ago, despite it being obsolete in its own 

Well, it stopped shipping only two years ago.

> version by its manufacturer, is a pretty clear indication of what I am 

You said "...few people have adopted IE7 instead of IE6...", and I don't 
see how that follows either from the statistics, or from what you have said.

BR, Julian

---

From: Ian Hickson <[email-masked]>
Date: Sat, 2 Feb 2008 21:52:39 +0000 (UTC)

> see how that follows either from the statistics, or from what you have 

I'm not sure how to explain it more than I have already. The details don't 
affect my original point, though, so I'll just leave it at that.

---

From: David Morris <[email-masked]>
Date: Sun, 3 Feb 2008 20:47:07 -0800 (PST)

> they do. May be not as many and as happily as Microsoft would like, but

Hard to avoid for the average user if they receive a new computer and
discover that even though they have carefully chosen XP Pro as the OS, the
vendor has upgraded their Office version and IE version.

I also encounter mandatory upgrades to critical business software which
demand an newer version of OS or IE than I would choose on my own.

Informed consent is marginal in this space.

Dave Morris

---

From: Henrik Nordström <[email-masked]>
Date: Tue, 05 Feb 2008 15:45:22 +0100

> Where do you draw the line?

I follow the small crowd who likes what the HTTP rfc says. If the server
says something then it's explicit, and sniffing should only be allowed
when there is no explicit information to go on.

I.e. there is perfectly valid reasons for a server to say that a file is
of type text/plain instead of text/html. Having an user agent guess that
the content should be displayed as HTML only because it looks like it is
HTML is plain wrong. It's equally valid for a server to say it's
ISO-8859-1 even if it looks like the content may be UTF-8 as there may
be ISO-8859-1 code sequences in there or perhaps the purpose is simply
to let the user know how odd it renders when rendered in the wrong
characterset.

Having user agents actively work around server misconfigurations is just
wrong. All this does is delaying getting the actual problem fixed, and
moving the burden of getting the problem fixed from the server / webbapp
maintainer who caused the problem to the user agent vendors.

Once you start going down the road of routinely secondguessing the
intentions of the server or webapp then you enter a never ending road,
making sure that these problems will stay forever and never get fixed.

So to summarise my preferences:

Content-Type guessing MAY be performed ONLY and ONLY IF there is no
Content-Type specified. (already a MUST level criteria in the RFC)

charset parameter guessing MAY be performed ONLY and ONLY IF there is no
charset parameter specified. (currently a MUST NOT in the RFC. charset
guessing is currently never allowed)

Related to this I also support removing the strict default ISO-8859-1
charset from HTTP text/* types, downgrading it to just a mere suggestion
that if there is no charset information available then a good guess for
the text/* types is ISO-8859-1 for historical reasons.

> aren't covered by these tests, and haven't been discussed within this 

Already in the definition of Type. Not much to discuss.

  "If
   and only if the media type is not given by a Content-Type field, the
   recipient MAY attempt to guess the media type via inspection of its
   content and/or the name extension(s) of the URI used to identify the
   resource."

Regards
Henrik

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 5 Feb 2008 17:31:58 +0100

Henrik Nordström wrote:

> I also support removing the strict default ISO-8859-1 charset
> from HTTP text/* types, downgrading it to just a mere suggestion
> that if there is no charset information available then a good
> guess for the text/* types is ISO-8859-1 for historical reasons.

Apparently all agreed that the "strict default Latin-1" should go.
One way to to handle this situation is to do whatever MIME and 
the specification of the Content-Type (if given) offer, and for
historical reasons "Latin-1" can be a good guess.

In theory.  But unsurprisingly the HTML5 draft tells us that it
is in practice often not good enough.  Often "wannabe Latin-1"
turns out to be "windows-1252".  If you want to suggest a "best
guess" in the HTTP spec. for historical reasons please mention
windows-1252, it is an important difference for some documents:

Latin-1 C1 controls may be not permitted, and windows-1252 0x80
is the only (*) backwards compatible way to say €.  

While that's IMO irrelevant for HTTP, if you decide to talk about
it anyway let's get it right:  Latin-1 is a "historical" charset,
windows-1252 is the "real" legacy.

 Frank

*: Agents knowing Latin-9 etc. would also know Unicode or &euro;