Subject: Re: NEW ISSUE: content sniffing

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 01:18:01 -0700

On Tue, Mar 31, 2009 at 7:44 PM, Roy T. Fielding <[email-masked]> wrote:
> It is impossible to determine a media-type (how a recipient should
> process a given representation) by sniffing the content (the data format).

Regardless, many popular user agents override the server-provided MIME
type after examining the content of HTTP responses.

> When a media type
> is not present (or is detectably incorrect), only the implementation
> doing the processing can determine an appropriate guess because that
> guess is almost always determined by the context in which the
> reference was made (not by the content).

Content sniffing algorithms in browsers largely ignore the context in
which the HTTP response is being used.  Specifically, algorithm for
computing the effective MIME type is a function of the HTTP response
alone (both in practice and in draft-abarth-mime-sniff).

> Since the context is
> deliberately not sent on the wire, there is absolutely no way that
> accurate sniffing can be defined by HTTP.

Thankfully, we do not require "accurate" sniffing.  We simply require
an algorithm for determining a MIME type that is compatible with
existing Web content.

> We aren't talking about
> a protocol decision regarding communication; we are talking about
> an operating default that is specific to the purpose of a given
> client and will likely be different for each one.

I disagree that we're interested in something specific to a given
client.  We're interested in an algorithm for determining the MIME
type of an HTTP response that works with existing Web content.

For example, suppose you're implementing an image editing program,
let's call it Imageshop.  You'd like users of Imageshop to be able to
open images specified by URL.  A user asks to edit
http://example.com/fancy-image.  Imageshop issues an HTTP request for
that URL and receive the following response:

Content-Type: image/jpeg

GIF89a...

Because popular user agents have historically interpreted such HTTP
responses as image/gif, it is quite likely that the server intends
this response to be treated as image/gif (and not as image/jpeg).

If the Imageshop developers follow the existing HTTP spec, they will
receive complaints from their users and be forced to reverse engineer
a content sniffing algorithm that is compatible with existing Web
content.  If, instead, we specify a sniffing algorithm, Imageshop will
interoperate with existing Web content as it's users expect.

> In any case, there is no algorithm for sniffing that is anywhere
> near the same level of standardization as HTTP.

You're right that a number of popular user agents use different
sniffing algorithms.  I'm hoping to converge these implementations on
a single sniffing algorithm.  Having the HTTP spec recommend a
specific algorithm will aid this process.

> The one that HTML5 is working on would barely qualify as Experimental.

I'm not sure what qualifies as "experimental," but the algorithm in
draft-abarth-mime-sniff is quite similar to the algorithms that ship
in Firefox and Chrome.  We have a great deal of data from the Google
search index and from opt-in user metrics with which to evaluate its
compatibility with existing Web content.

> If the folks
> promoting such software can successfully deploy it across all HTTP
> clients, then it should be referenced.

Surely this is too high a bar.  Why bother writing standards if we
require all implementations to interoperate perfectly before putting
pen to paper?

> Until then, it remains an
> unproven and, IMO, mistaken idea which is far more likely to
> be overcome by events than become a standard way to handle HTTP.

I don't think that content sniffing will magically disappear if we
just ignore it long enough.  Instead, we should shed some light on
this dark corner of reality.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Wed, 01 Apr 2009 10:46:42 +0200

> type after examining the content of HTTP responses.

But only for a certain set of MIME types, which are known to be 
frequently incorrect, right?

> alone (both in practice and in draft-abarth-mime-sniff).

My understanding was that the context *is* relevant (stylesheet? image?).

> ...
> ...

As long as it is clear that it's still an option not to sniff at all, 
and as long as the goal remains to minimize the amount of sniffing going 
on...

That said, why is it important that the HTTP spec references this document?

BR, Julian

---

From: Geoffrey Sneddon <[email-masked]>
Date: Wed, 1 Apr 2009 14:42:28 +0100

On 31 Mar 2009, at 21:37, David Morris wrote:

> On Tue, 31 Mar 2009, Adam Barth wrote:
>
>> On Tue, Mar 31, 2009 at 12:51 PM, Mark Baker <[email-masked]> wrote:
>>> On Tue, Mar 31, 2009 at 3:37 PM, Adam Barth <[email-masked]>  
>>> wrote:
>>>> When different user agents use different sniffing algorithms,  
>>>> content
>>>> authors pay a large cost, both in terms of compatibility and in  
>>>> terms
>>>> of security.  For user agents that wish to perform sniffing, I  
>>>> think
>>>> we'd be doing the Web a service by specifying which algorithm they
>>>> should use.
>>>
>>> I agree, which is why I suggested a link from 2616bis to the
>>> algorithm.  Do you feel that to be insufficient?  If so, why?
>>
>> I don't have a strong opinion about which document should contain the
>> algorithm, but I think we're better off making the algorithm  
>> normative
>> (for those agents that wish to sniff) rather than informative.  That
>> will help prevent developers of sniffing user agents from  
>> implementing
>> divergent sniffing algorithms.
>
> I disagree ... encoding what is essentially a heuristic algorithm  
> which will need to change as content types morph into standard  
> status is the
> wrong thing to do. Certainly in the HTTP standard.

But the Content-Type sniffing algorithms used in browsers have pretty  
much stagnated. They rarely change, and when they do, it is normally  
to get closer to interoperability. I don't think they change enough  
for it to be an issue. Ideally we probably want the algorithms to get  
frozen where they are now, where they are required for compatibility,  
and never sniff more types.

--
Geoffrey Sneddon
<http://gsnedders.com/>

---

From: Mark Baker <[email-masked]>
Date: Wed, 1 Apr 2009 10:55:08 -0400

On Tue, Mar 31, 2009 at 9:35 PM, Adam Barth <[email-masked]> wrote:
> On Tue, Mar 31, 2009 at 6:26 PM, Barry Leiba <[email-masked]> wrote:
>> I'd be in favour of something like, "The use of different sniffing
>> algorithms in different implementations creates [these sorts of
>> problems].  Because of that, settling on common sniffing mechanisms is
>> important.  To that end, implementations that sniff SHOULD
>> [...etc...]."  The explanation of why it's important may sway some
>> implementors, and making it a SHOULD recognizes that some will go
>> their own way in any case.
>
> This sounds reasonable to me.

For the record, I agree with the points made by Roy, and don't believe
normative text of any kind is appropriate here at this point in time.

I could live with something like this in HTTPbis;

"Some HTTP clients intentionally ignore the authoritative media type
of some messages and instead 'sniff' the representation to determine
the effective type.  For Web browsers in particular, work is ongoing
to document a sniffing algorithm[non-normative ref] that reflects
common practice amoung popular implementations."

Mark.

---

From: Bil Corry <[email-masked]>
Date: Wed, 01 Apr 2009 10:05:17 -0500

> frequently incorrect, right?

I don't know if it has changed, but Internet Explorer will sniff when no mime-type is provided, or when the mime type is something IE knows about:

	http://blogs.msdn.com/ie/archive/2005/02/01/364581.aspx

Of course, IE8 added a flag that allows sites to turn off content sniffing, so that would be the exception.

> My understanding was that the context *is* relevant (stylesheet? image?).

IE will sniff images and render them as HTML if it believes they're actually HTML:

	http://www.heise-online.co.uk/security/Risky-MIME-sniffing-in-Internet-Explorer--/features/112589

> and as long as the goal remains to minimize the amount of sniffing going

Microsoft tried turning off mime sniffing for plain/text and it broke a bunch of sites:

	http://blogs.msdn.com/ie/archive/2005/02/01/364581.aspx

Undoubtedly, that was the impetus for adding the flag to turn off content sniffing.  I think it's safe to assume content sniffing in IE won't be going away any time soon.

> That said, why is it important that the HTTP spec references this document?

Adam Barth's comments from earlier in the thread:

-----
When different user agents use different sniffing algorithms, content
authors pay a large cost, both in terms of compatibility and in terms
of security.  For user agents that wish to perform sniffing, I think
we'd be doing the Web a service by specifying which algorithm they
should use.
-----

- Bil

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 09:01:39 -0700

On Wed, Apr 1, 2009 at 1:46 AM, Julian Reschke <[email-masked]> wrote:
> But only for a certain set of MIME types, which are known to be frequently
> incorrect, right?

Yes.  It appears to be sufficiently compatible to limit sniffing to a
handful of MIME types.  These are listed in draft-abarth-mime-sniff.

> My understanding was that the context *is* relevant (stylesheet? image?).

In browsers, there are two contexts that do not use the
context-independent sniffer:

1) The <script> tag largely ignores MIME types.  I don't really think
of this as MIME sniffing.  It's more that the <script> tag doesn't
care about MIME types.

2) A style sheet loaded via <link rel="stylesheet" href="..."> in
quirks mode also ignores MIME types.  In standards mode, however, this
context uses the same context-independent MIME sniffer as everything
else.

Historically, the <img> tag did have a more aggressive image-specific
sniffer, but this does not appear to be required for Web compatibility
and has been removed in major implementations.

> As long as it is clear that it's still an option not to sniff at all, and as
> long as the goal remains to minimize the amount of sniffing going on...

Yes.  I believe we are in agreement on these points.

> That said, why is it important that the HTTP spec references this document?

The current HTTP spec contains a requirement that contradicts this
widespread practice.  It seems we should update this part of the spec
to reflect reality (and, perhaps, nudge reality into a slightly better
equilibrium).

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 09:09:45 -0700

On Wed, Apr 1, 2009 at 8:05 AM, Bil Corry <[email-masked]> wrote:
> I don't know if it has changed, but Internet Explorer will sniff when no mime-type is
> provided, or when the mime type is something IE knows about:

Internet Explorer has the most aggressive content sniffing algorithm.
Evidence is that we can be far less aggressive and still be
sufficiently compatible with existing Web content.

> IE will sniff images and render them as HTML if it believes they're actually HTML:

This is no longer the case in Internet Explorer 8 (due to security concerns).

> Microsoft tried turning off mime sniffing for plain/text and it broke a bunch of sites:

The handling of text/plain is the most contentious point in the
content sniffing algorithm.  I suspect text/plain will be the last
MIME type to be handled uniformly across user agents.

> Undoubtedly, that was the impetus for adding the flag to turn off content sniffing.  I think
> it's safe to assume content sniffing in IE won't be going away any time soon.

Indeed.  Every major browser (not just IE) employs content sniffing.
Content sniffing appears to be required to correctly handle
approximately 1% of HTTP responses.  Turning off content sniffing
entirely breaks major Web sites.

Adam

---

From: Mark Baker <[email-masked]>
Date: Wed, 1 Apr 2009 12:10:56 -0400

> to reflect reality (and, perhaps, nudge reality into a slightly better

Which requirement is that?

Mark.

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 09:20:04 -0700

>From http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.2.1

"If and only if the media type is not given by a Content-Type field,
the recipient MAY attempt to guess the media type via inspection of
its content and/or the name extension(s) of the URI used to identify
the resource."

For example, user agents also need to sniff HTTP responses with a
Content-Type of */* in order to be compatible with existing Web
content.

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 09:39:14 -0700

> Content-Type of */* in order to be compatible with existing Web

Apologies if the above statement is unclear.  I don't mean "*/*" to
represent every MIME type but rather that the Content-Type header
contains exactly those three bytes.

Adam

---

From: Mark Baker <[email-masked]>
Date: Wed, 1 Apr 2009 12:46:08 -0400

> its content and/or the name extension(s) of the URI used to identify

Thanks.  I would be happy to remove that text, not to encourage
sniffing, but because it has nothing to do with the HTTP protocol.

Mark.

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 10:09:30 -0700

> sniffing, but because it has nothing to do with the HTTP protocol.

That about the sentence below?

"Content-Type specifies the media type of the underlying data."

Are you suggesting we remove Section 7.2.1 entirely and replacing it
with another document that specifies how to determine the data type of
an HTTP response?  We could use draft-abarth-mime-sniff for this
purpose (suitably altered to allow for the possibility of not
sniffing).

Adam

---

From: Mark Baker <[email-masked]>
Date: Wed, 1 Apr 2009 13:43:18 -0400

> "Content-Type specifies the media type of the underlying data."

The definition of the Content-Type header in 14.17 says all that needs
saying about the protocol IMO.

>
> Are you suggesting we remove Section 7.2.1 entirely

No, just that paragraph.

Mark.

---

From: David Morris <[email-masked]>
Date: Wed, 1 Apr 2009 11:13:51 -0700 (PDT)

> entirely breaks major Web sites.

If wonder how long it would take for those major sites to get their act 
together if all major browsers agreed to disable sniffing as of some 
not too distant future date?

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 02 Apr 2009 10:19:49 +1300

some servers sent certain mime types as a default if the actual mime 
type was unknown or there was no mapping for the file extension.

I don't know how common this is any more, but it certainly created a 
need for sniffing, since the "authoritative" mime type was a bad guess.

In fact the concept of what constitutes an authoritative mime type is a 
bit tenuous.  if it's a web server guess, then what makes it any better 
than a client's guess.  frequently it will be worse.

maybe we need a weakness indicator for the content type field, e.g

Content-Type: W/"application/octet-stream"

or

Content-Type: application/octet-stream ; certainty = random-guess

meaning "we think it might be application/octet-stream"

IOW, maybe an indication of whether to sniff or not shouldn't be another 
header, but a parameter of the Content-Type header.

Then one could indicate the source/strength of the mime type, e.g.

* explicitly set by management interface
* explicitly set by CGI / script
* guessed by file extension
* guessed by sniffing at server
* guessed by sniffing at intermediary
* pulled out of a hat

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 14:55:31 -0700

On Wed, Apr 1, 2009 at 11:13 AM, David Morris <[email-masked]> wrote:
> together if all major browsers agreed to disable sniffing as of some not too
> distant future date?

It's unlikely that all the browser vendors would agree to this because
the one browser vendor that defects from this pack would garner more
market share by being compatible with more Web sites.  Even if we
could get all the browser vendors to overcome this prisoner's dilemma,
users would simply continue to use older browsers that actually work.

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 14:58:59 -0700

> saying about the protocol IMO.

This section does not contain enough information to implement HTTP in
a way that interoperates with existing Web content.  For example, the
developers of Imageshop must resort to reverse engineering.

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 15:00:40 -0700

> maybe we need a weakness indicator for the content type field, e.g

This doesn't solve the problem of interoperating with existing Web content.

Adam

---

From: Mark Baker <[email-masked]>
Date: Wed, 1 Apr 2009 19:45:54 -0400

On Wed, Apr 1, 2009 at 5:58 PM, Adam Barth <[email-masked]> wrote:
> a way that interoperates with existing Web content.

Right.  The information you're talking about is irrelevant to most
non-browser HTTP agents and so doesn't belong in the HTTP
specification.

Mark.

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 16:48:02 -0700

> non-browser HTTP agents and so doesn't belong in the HTTP

It's relevant to any HTTP user agent that wishes to interoperate with
existing Web content.  For example, Imageshop (described earlier in
this thread) is not a browser but is interested in knowing when its
users expect an HTTP response to be treated as an image.

Adam

---

From: Mark Baker <[email-masked]>
Date: Wed, 1 Apr 2009 20:08:23 -0400

> users expect an HTTP response to be treated as an image.

Yes, of course.  I'm trying to emphasize to you that the Web is a lot
larger than just those clients and servers exchanging HTML,
Javascript, and JPGs.  If that weren't the case, I would agree with
you that HTTP should be updated.

Mark.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 1 Apr 2009 17:43:39 -0700

> users expect an HTTP response to be treated as an image.

No, it isn't relevant at all.  If the purpose of the user agent is
to find errors in metadata, then your prescribed sniffing requirements
would obviously be false.  A testing client (like MOMspider) is one
such user agent.

What you are talking about is error recovery from the perspective
of a particular type of casual user, not the protocol.
The protocol is stating the communication from the sender.  If the
communication is false, the protocol is being violated and that
violation is shown by failing to meet one of the protocol requirements.
The fact that different user agents deal with protocol violations in
different ways is a good thing.

The whole philosophy that protocol specs, as opposed to browser
implementation specs, must describe the error-handling quirks
of browsers is bankrupt.  HTTP is not defined by browser reactions
to HTTP messages.  The correct way to interoperate with broken Web
content is to display a very large error message that explains why
it is broken.  The fact that *some* HTTP user agents, notably the
big browser vendors, choose not to display those errors in a usable
way is a design choice for those user agents.  It does not in any way
change the semantics of the HTTP messages and the meaning of metadata
within those messages.

The common browser behavior does not need to be standardized in HTTP
because it is not common to the vast majority of user agent
implementations, which far outnumber the general purpose browsers.
It cannot be standardized in a way that would be safe for safety- 
critical
environments such as health care, where failure to display the errors
could very well result in serious injury or death.  And I have no
doubt that MSIE will change its sniffing behavior shortly after a
few lawsuits demonstrate their stupidity.

....Roy

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Apr 2009 23:57:07 -0700

On Wed, Apr 1, 2009 at 5:43 PM, Roy T. Fielding <[email-masked]> wrote:
> No, it isn't relevant at all.

That's an extreme point of view.  The implementors of Imageshop
certainly want to know how to determine the MIME type of HTTP
responses they receive from the Web.

> of a particular type of casual user, not the protocol.

Without specifying how to determine the MIME type of HTTP responses,
implementor of user agents interested interoperating with existing Web
content will be forced to reverse engineer each other.  Undoubtedly,
they will make mistakes, and we'll be in an even worse position than
we are today.

> violation is shown by failing to meet one of the protocol requirements.

I agree that the spec should require that servers send the correct
MIME type in the Content-Type header.  No one is disputing this.

> different ways is a good thing.

In this case, a diversity of sniffing algorithms is harmful both to
interoperatiblity and security.  We'd be much better off if all the
user agents that require sniffing used the same algorithm.

> of browsers is bankrupt.

I'm not proposing the spec describe the error handling quirks of
browsers.  I'm propose that the spec contain enough detail that
implementors of future user agents (if they are so inclined) can
determine the MIME type of HTTP responses from the Web.

> The correct way to interoperate with broken Web
> it is broken.

Sadly, such user agents will not be as popular as those that just work.

> The fact that *some* HTTP user agents, notably the
> way is a design choice for those user agents.

Be that as it may, as an implementor of a new user agent that would
like to interoperate with the Web, I would like to know how to
determine the MIME type of existing Web content.

> implementations, which far outnumber the general purpose browsers.

I am not interested in imposing browser behavior on other user agents.
 In fact, I think the spec should recommend against sniffing.

> It cannot be standardized in a way that would be safe for safety-critical
> could very well result in serious injury or death.

These user agents are free to follow the recommended course of action
and not perform sniffing.  If they do require sniffing, then we're
much better off if they follow a standardized algorithm than if they
incorrectly reverse engineer other user agents.  Being explicit has
fewer failure modes that burying our heads in the sand.

> And I have no

You're entitled to that opinion, but I don't see content sniffing
going away anytime soon.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 2 Apr 2009 09:19:17 +0200 (CEST)

On Wed, 1 Apr 2009, Adam Barth wrote:

>> non-browser HTTP agents and so doesn't belong in the HTTP specification.
>
> existing Web content.  For example, Imageshop (described earlier in this 
> thread) is not a browser but is interested in knowing when its users expect 
> an HTTP response to be treated as an image.

I think this is an example of mixed layerings that leads to no good.

HTTP is a transport layer and a HTTP transport does not need to perform any 
guessing of contents at all. It provides data and meta-data. Fine.

Applications that get stuff over HTTP may want to guess the content and ignore 
the Content-Type. Sure, but that's not HTTP.

I consider that a little like an unzip program checking that the .zip file 
looks like an actual zip before it operates on it. If the zip program 
downloads the file over HTTP it doesn't mean that we need to teach the HTTP 
spec on how the zip client may want to guess if the content truly is a zip or 
not.

-- 

  / daniel.haxx.se

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 02 Apr 2009 10:10:09 +0200

On Thu, 02 Apr 2009 09:19:17 +0200, Daniel Stenberg <[email-masked]> wrote:
> Applications that get stuff over HTTP may want to guess the content and  
> ignore the Content-Type. Sure, but that's not HTTP.

So if what Web browsers do is fine, where would be the place to specify  
our sniffing behavior to get interoperable behavior? A lot of other people  
have been saying it is in violation of HTTP, hence us trying to coordinate  
here.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 2 Apr 2009 14:32:42 -0700

> responses they receive from the Web.

Maybe the implementors of Imageshop will read the thread and
understand that the media type of a message is not the same
thing as the data format of a message.  The media type (what was
called MIME type ages ago) is a processing instruction supplied
by the sender.  The media type cannot be discerned by looking at
the bits.  The data format can sometimes be discerned by looking
at the bits, which is a reasonable fallback behavior depending on
the context in which the request was made.

It is impossible to sniff for a media type because any given
data format matches at least two or more media types.  Making
a preference of one over another is something a particular kind
of user agent can do in a particular context in which it has
been given a particular configuration to do so.  None of those
variables have anything to do with HTTP.  HTTP is responsible
for communicating the sender's intentions.

> they will make mistakes, and we'll be in an even worse position than

Then fix the content metadata.  No other solution will work, period.

> user agents that require sniffing used the same algorithm.

We would be better off if none of them sniffed.  That is the most
interoperable solution.

> determine the MIME type of HTTP responses from the Web.

It already does contain everything that can be truly said about
determining the media type.  The only thing it doesn't define is
what the recipient should do when it detects an error or when
no type is supplied, and the reason for that is because the behavior
is different for every single type of recipient.  The only reason
that the HTML5 folks can pretend to answer that question is because
they currently ignore the needs of all recipients other than the
big general-purpose browsers.  IETF concerns >> WHATWG concerns.

> Sadly, such user agents will not be as popular as those that just  

That is a matter of opinion.  I have seen no evidence to suggest
that MSIE bugs actually helped it in competing with other browsers.
I have seen plenty of evidence that MSIE is upgraded or uninstalled
on an institutional basis when its bugs create a liability.  The
same will hold true for other browsers.

> determine the MIME type of existing Web content.

Read the Content-Type header field and behave accordingly. If it is
obviously in error, then work around that error while informing
the user.

> You're entitled to that opinion, but I don't see content sniffing

Time will tell.  I only document the technical solutions that
actually work.

....Roy

---

From: Adam Barth <[email-masked]>
Date: Thu, 2 Apr 2009 16:22:14 -0700

> the context in which the request was made.

You're ignoring the reality of existing Web content.  To interoperate
with existing Web content, a user agent must consider both the
Content-Type headers and the content when determining the media type
contained in a response.  To claim otherwise is fantasy.

> data format matches at least two or more media types.

This is true in general.  However, existing Web content assumes that
user agents will override their specified media types in certain
cases.  For example, suppose a user agent receives the following HTTP
response from the Web.

Content-Type: */*

GIF89a....

If this user agent wishes to interoperate with this server, the user
agent should use the media type image/gif when processing this
request.

> None of those

Currently, the HTTP spec ignores reality and forbids user agents from
interoperating with existing Web content.

> Then fix the content metadata.  No other solution will work, period.

I agree that servers should fix their metadata.  However, not all
servers will.  If a user agent wishes to interoperate with these
servers (as many do), then we should be helpful and explain how to do
so in a reliable way.

> We would be better off if none of them sniffed.  That is the most

Forbidding sniffing prevents user agents from interoperabling with
existing Web content.

> determining the media type.

The spec could provide a sniffing algorithm that allows user agents to
interoperate with existing Web content.

> The only thing it doesn't define is
> is different for every single type of recipient.

The type of recipient isn't really relevant.  What's relevant is how
existing servers expect their existing content to be interpreted.  A
server on the Web that specifies a Content-Type of "*/*" and a payload
that begins "GIF89a" expects this response to be treated as image/gif.

> The only reason
> big general-purpose browsers.

Imageshop is not a "big general-purpose browser."

> IETF concerns >> WHATWG concerns.

I don't think it's helpful to frame this discussion in terms of
identity politics.

> that MSIE bugs actually helped it in competing with other browsers.

If you asked them, I'm sure implementors of non-MSIE user agents would
tell you that sniffing is required to avoid losing user.  For example,
I recently received a bug report from a user who was unable to buy
something at BestBuy.com because Chrome did not sniff aggressively
enough in some corner case.  Had we not fixed this issue, that user
would have simply switched to another browser that worked.

> same will hold true for other browsers.

Compatibility is widely documented to be one of the most important
factor (if not THE most important factor) in determining whether a
user will adopt a new browser.

> obviously in error, then work around that error while informing

In order to interoperate correctly, I need to know HOW to work around
the error.  If we don't specify an algorithm that works, I'll have to
reverse engineer other implementations.  This will lead to further
compatibility and security problems.

> Time will tell.  I only document the technical solutions that

The algorithm described in draft-abarth-mime-sniff is a technical
solution that actually works.  If a user agent implements that
algorithm, the user agent can determine the media type of HTTP
responses and interoperate with existing Web content.

Adam

---

From: Shane McCarron <[email-masked]>
Date: Thu, 02 Apr 2009 19:25:38 -0500

Adam,

> contained in a response.  To claim otherwise is fantasy.
Roy was not "ignoring the reality of existing Web content".  He is 
saying the same thing every other expert from this community has said - 
that the error handling mechanism you are proposing to codify at the 
*protocol* layer is not a protocol issue.  If you and others like you 
want to carefully define how your applications deal with situations 
where the underlying layers have provided you with mis-information, that 
is completely up to you.  However, it is inappropriate to foist those 
"solutions" onto a community that is clearly saying "this is not an 
issue for our layer"!  Moreover, it would be inappropriate to attempt to 
define your solution in such a way that encourages the continued 
transmission of that mis-information.

The protocol provides mechanisms for servers to declare the type of a 
payload.  If some servers lie about it, that's their mistake.  We should 
not be trying to insist that every endpoint deal with errors from the 
broadcaster.  That way lies madness.  Look at it this way - if there 
were billions of devices that could magically capture music out of the 
air (call them radios), and hundreds or even thousands of sources for 
that music, and tens of those sources were sending the music out in such 
a way that the devices couldn't capture it... what would happen?  Would 
all the devices get changed?  Absolutely not!  The sources would fix 
themselves or they would disappear.  Either way, problem solved.

I am sure that the solution you are proposing works for the small subset 
of the data and limited collection of data processors that you are 
considering.  In the greater collection of data that is the entire 
Internet, and the greater collection of data processors that are all 
agents that use HTTP, the solution just has no general place.  At least 
that's my opinion.  I could be wrong.

-- 
Shane P. McCarron                          Phone: +1 763 786-8160 x120
Managing Director                            Fax: +1 763 786-8180
ApTest Minnesota                            Inet: [email-masked]

---

From: Ian Hickson <[email-masked]>
Date: Fri, 3 Apr 2009 01:46:55 +0000 (UTC)

On Thu, 2 Apr 2009, Shane McCarron wrote:
> *protocol* layer is not a protocol issue.

With all due respect, not "every other expert from this community" agrees 
with Roy. I don't, nor do many others.

Personally I think that whatever spec defines Content-Type should define 
how Content-Type works in reality. I would have no problem with HTTP 
itself not defining the Content-Type header at all. What I personally 
object to is the HTTP spec defining Content-Type in a way which is 
incompatible with billions of deployed resources.

> If you and others like you want to carefully define how your 
> applications deal with situations where the underlying layers have 
> provided you with mis-information, that is completely up to you.

Could we please not make this an "us vs them" discussion?

> However, it is inappropriate to foist those "solutions" onto a community 
> that is clearly saying "this is not an issue for our layer"!

By the same logic, it would be inappropriate to foist the solution 
currently in the HTTP specification onto a community that is clearly 
saying "this IS an issue for this layer".

> Moreover, it would be inappropriate to attempt to define your solution 
> in such a way that encourages the continued transmission of that 
> mis-information.

The current specification text, as it has stood since the dawn of the Web, 
has failed to discouraged the continued transmission of this 
mis-information. If we are trying to remove specification text that 
encourages the transmission of such mis-information, we should change the 
spec!

> themselves or they would disappear.  Either way, problem solved.

If there were billions of devices called radios, and of thousands of 
sources, tens of those sources were sending the music out in such a way 
that the devices couldn't capture it, but one of those sources was the 
incredibly popular BBC, then new devices called radios would immediately 
appear that DID handle those few incorrect sources. This is elementary 
capitalism.

> that's my opinion.  I could be wrong.

I believe you are.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 03 Apr 2009 15:13:04 +1300

I agree with Roy and Shane

the issue of next-level content is not an issue for the HTTP (transport) 
layer, it's an issue for the next layer up.

Take a look at the network stack.

Start with an ethernet network packet , there is a frame type header in 
there which tells the stack which next-level handler to pass the packet 
to. 

assuming the frame type is IP, then the packet is passed to the IP 
protocol hander.

In the IP header there is a protocol header as well, that specifies the 
next-level protocol handler.  The IP protocol handler looks at this. 
Supposing this is TCP, the packet is then passed up to the TCP protocol 
handler.

At no stage does the ethernet handler VALIDATE that the next level 
header is actually an IP header just because the frame type is IP.  This 
is left to the IP handler
At no stage does the IP handler VALIDATE that the next level header is 
actually a TCP header just because the protocol ID is 6.

So why should HTTP validate that the next level identifier 
(Content-Type) is correct for the entity body?

There are zillions of content types.  Why should HTTP care about them?  
It's an impossible task.

Now in a browser there is a next-level processor above HTTP. It's the 
thing that actually gets the resource.  It's free to do whatever it 
likes with that content.  However transport of the content is irrelevant.

HTTP is only a very small part of the function of a browser.  It's 
purely used to transfer resources around.  You could build a browser on 
top of pretty much any transfer protocol.  Blurring the lines between 
the transport, and what is transported is a really bad idea.  Apply that 
idea to other scenarios and you see how bad it really is.  Imagine if 
SMTP had to be concerned about validating email attachments.  Imagine if 
FTP had to be concerned about validating the content of files.  Imagine 
if the postal system had to be concerned about validating the content of 
letters.  You could argue that in these 3 cases they actually tried to a 
certain extent.  But you'd have a hard time claiming any success at it.

By all means develop framework and guidelines for how to process (e.g. 
sniff) transported data, but it's a layer above HTTP issue.  Referring 
to it in HTTP would be like referring to TCP in the IP spec, and once 
you start that, where do you stop? ICMP? IGMP? UDP?  SIP? GRE? L2TP?  
Every time you created a new higher-level protocol you'd need to update 
the specs for each lower one it could be layered on.

For implementors, they need to be able to easily find what they need to 
do.  It's like there needs to be some higher-level resource which 
contains a reference to all the things you should consider when creating 
an HTTP agent.  If there were such a resource, it would contain a 
reference to the transport spec (HTTP) as well as other considerations, 
and could be expanded to refer to issues such as sniffing or any other 
issue that confronted a browser developer.

just my 3.5c

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Thu, 02 Apr 2009 21:42:28 -0500

> contained in a response.  To claim otherwise is fantasy.

No, it's a statement of fact.

Roy has (correctly) pointed out to you, multiple times, that your perceived
flaw of "broken web sites", are the direct result of inaccurate Content-Type
headers being sniffed by user agents and represented inaccurately.

The inaccurate headers are a direct -result- of content sniffing by the
handful of user agents which tolerate inappropriate server behavior.
You perceive this to a user agent issue.  It is a flaw in the servers.

There is no need to modify the spec (and I agree the reference to content
sniffing can be dropped altogether) to describe what the UA's have foisted
on the world.

As soon as all browsers conform to spec, authors/administrators will correct
their errors, because those errors will be obvious to them.

Ponder for a moment; 10 years ago, 1% - 3% of content could not be properly
rendered with the Content-Type headers provided.  Today, has that number
grown?  No, it's probably shrunk.

Look Adam, as soon as you describe how, by spec, I can offer an html file
as an example in a text/plain repesentation (for illustration of tags)
in spite of inappropriate behavior by IE and the host of others, I'll
respect your position.  But your rants are getting irritating.  If you
demand that IE's autodeterministic features are of value, you've lost my
attention already, because the world would not be polluted by utf7 xss
vulnerabilities if not for such obnoxious, spec-incompliant behavior.

FWIW I AM a fan of content sniffing; of local filesystem data with no other
semantic clues as to its contents.  HTTP is designed to eliminate this
requirement, but browser user agents must retain it for file:// content.

---

From: Adam Barth <[email-masked]>
Date: Thu, 2 Apr 2009 21:51:11 -0700

On Thu, Apr 2, 2009 at 7:42 PM, William A. Rowe, Jr.
> No, it's a statement of fact.

The fact is content sniffing is required to correctly process
approximately 1% of HTTP responses on the Web.  The HTTP spec forbids
this practice.  I cannot implement a user agent that both
interoperates with the Web and follows the HTTP specification.

> You perceive this to a user agent issue.  It is a flaw in the servers.

Cause and effect are irrelevant.  My job as a user agent implementor
is to deal with the reality  with which I am presented not to
fantasize about other realities that might have been.

> sniffing can be dropped altogether) to describe what the UA's have foisted

There appear to be three roads forward:

1) Remove all references to Content-Type from the HTTP specification
and separately specify how to interpret the header.

2) Recommend against content sniffing but acknowledge that some user
agents might perform content sniffing to interoperate with legacy HTTP
servers.

3) Option (2) plus a recommended concrete sniffing algorithm that
works with existing Web content.

Leaving the spec unmodified renders the spec dead letter.

> their errors, because those errors will be obvious to them.

Economics will prevent this from occurring.

> grown?  No, it's probably shrunk.

This is pure speculation.

> respect your position.

In fact, this works precisely as you desire according to
draft-abarth-mime-sniff!  Simply specify "text/plain" as the
Content-Type.

> But your rants are getting irritating.

I have attempted to provide technical arguments and not rants.

> If you

I'm not suggesting adopting IE sniffing algorithm.  That algorithm
makes a poor security / compatibility trade-off, as you point out.
draft-abarth-mime-sniff makes a security / compatibility trade-off
that has been informed by careful threat modeling and extensive
empirical evaluation.  I encourage you to read
http://www.adambarth.com/papers/2009/barth-caballero-song.pdf for
further details.

Adam

---

From: Adam Barth <[email-masked]>
Date: Thu, 2 Apr 2009 21:56:06 -0700

> layer, it's an issue for the next layer up.

It sounds like you recommend option (1) from my previous email.  If
it's not an HTTP issue, why does the HTTP spec forbid me from
implementing a user agent that interoperates with the existing Web?

> Now in a browser there is a next-level processor above HTTP. It's the thing
> that actually gets the resource.  It's free to do whatever it likes with
> that content.

Except that the HTTP spec forbids me from doing the very thing I need to do!

> sniff) transported data, but it's a layer above HTTP issue.  Referring to it
> in HTTP would be like referring to TCP in the IP spec, and once you start
> that, where do you stop?

Following this train of thought to its logical conclusion, we ought to
remove Content-Type fro the spec entirely.

> For implementors, they need to be able to easily find what they need to do.
>  It's like there needs to be some higher-level resource which contains a
> reference to all the things you should consider when creating an HTTP agent.
>  If there were such a resource, it would contain a reference to the
> transport spec (HTTP) as well as other considerations, and could be expanded
> to refer to issues such as sniffing or any other issue that confronted a
> browser developer.

Again, this appears to recommend option (1), in which a separate
document specifies how to process the Content-Type header.

Adam

---

From: Ian Hickson <[email-masked]>
Date: Fri, 3 Apr 2009 05:36:33 +0000 (UTC)

> (Content-Type) is correct for the entity body?

Because there is a practical difference between the Ethernet/IP and the 
IP/TCP boundaries, and the HTTP/Content boundary. In the latter case, 
there are millions if not billions of cases where authors who have never 
even heard of HTTP are misconfiguring their servers, leaving user software 
in the unfortunate position of having to balance the satisfaction of their 
users (which would require some heuristic processing of the claimed type) 
and compliance with the specifications.

Generally, money will win in this kind of balance.

In the Ethernet/IP and the IP/TCP cases, the software is written by people 
who have at least heard of the relevant specifications, and may even have 
read them. There are also FAR fewer implementations of those levels of the 
stack than there are misconfigured installations of HTTP servers.

> There are zillions of content types.  Why should HTTP care about them?  

We're not saying HTTP should care about them. We're saying that one of 
these two options should be picked:

Either:

a) HTTP should not have any requirements for how to process Content-Type 
headers, and should just leave Content-Type to another spec, or:

b) HTTP should include the proposed limited Content-Sniffing algorithm, 
which would allow us to get software tools to converge on a single set of 
heuristics and thus reduce the security risk.

Note that we are _not_ asking for a uniform way of detecting all content 
types. In the vast majority of those "zillions" of cases, the algorithm 
Adam's I-D specifies will require UAs to honour the type.

> Now in a browser there is a next-level processor above HTTP.

This is not in any way limited to browsers. It applies to any tool that 
uses HTTP for user-visible content processing.

> It's the thing that actually gets the resource.  It's free to do 
> whatever it likes with that content.

No, HTTP says of the Content-Type header "its value indicates what 
additional content codings have been applied to the entity-body, and thus 
what decoding mechanisms must be applied in order to obtain the media-type 
referenced by the Content-Type header field".

If this is intended to allow user agents to ignore the Content-Type 
header, or apply heuristics, then a clearer statement of this would be an 
acceptable solution (proposal "a" above).

---

From: Julian Reschke <[email-masked]>
Date: Fri, 03 Apr 2009 07:51:21 +0200

> referenced by the Content-Type header field".

No it doesn't. It says this about Content-*Encoding* 
(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.7.2.1.p.3>).

BR, Julian

---

From: Ian Hickson <[email-masked]>
Date: Fri, 3 Apr 2009 07:55:44 +0000 (UTC)

On Fri, 3 Apr 2009, Julian Reschke wrote:
> Ian Hickson wrote:
> > ...
> > > It's the thing that actually gets the resource.  It's free to do whatever
> > > it likes with that content.
> > 

My apologies, I meant to quote the following: "If and only if the media 
type is not given by a Content-Type field, the recipient MAY attempt to 
guess the media type via inspection of its content and/or the name 
extension(s) of the URI used to identify the resource."