Subject: NEW ISSUE: content sniffing

From: Mark Nottingham <[email-masked]>
Date: Tue, 31 Mar 2009 10:58:38 +1100

[ As discussed in SF ]

Browsers now routinely 'sniff' content to determine its type,  
sometimes in conflict with the media type conveyed in the Content-Type  
header. The reasons most often cited for this practice are the  
misconfiguration of servers, the inability of content authors to  
configure servers (either for technical reasons or lack of education),  
and generally the incentives placed upon browser vendors to "just work."

p3-payload section 3.2.1. currently says:
> If and only if the media type is not given by a Content-Type field,  
> the recipient MAY attempt to guess the media type via inspection of  
> its content and/or the name extension(s) of the URI used to identify  
> the resource. If the media type remains unknown, the recipient  
> SHOULD treat it as type "application/octet-stream".
which effectively disallows this practice, despite widespread use that  
(apparently) isn't stopping any time soon.

The language should be updated to reflect this reality, without unduly  
encouraging the use of sniffing except where necessary. Ideally, it  
will be done in such a way that:

   * Does not require sniffing for all uses of HTTP (i.e., a  
particular implementation and/or user can "opt in" to the use of a  
sniffing algorithm), since this is most commonly a problem for the  
browser case, and
   * Specifically allows a user and/or content provider to opt out of  
the use of sniffing in a particular interaction, and
   * Promotes interoperability (i.e., if two implementations sniff,  
they will do so in the same way).

See:
   http://tools.ietf.org/html/draft-abarth-mime-sniff
for a proposal sourced from HTML5.

Besides the issue of sniffing itself, there's also an open question of  
whether the sniffing algorithm would remain in a separate document  
(i.e., our work would only be to relax requirements to allow it), or  
whether it would be in-document.

In either case, we'd have to take a serious look at security  
considerations, and also look at impact on intermediaries, etc.

Note that this is not about sniffing encoding directly (see issue  
#20), although the resolution may be related.

--
Mark Nottingham     http://www.mnot.net/

---

From: Adam Barth <[email-masked]>
Date: Tue, 31 Mar 2009 08:35:40 -0700

If you'd like to learn more about the security and compatibility
considerations that lead to draft-abarth-mime-sniff, you might enjoy
reading:

http://www.adambarth.com/papers/2009/barth-caballero-song.pdf

Let me know how I can be of assistance on this issue.

Adam

On Mon, Mar 30, 2009 at 4:58 PM, Mark Nottingham <[email-masked]> wrote:
> [ As discussed in SF ]
>
> Browsers now routinely 'sniff' content to determine its type, sometimes in
> conflict with the media type conveyed in the Content-Type header. The
> reasons most often cited for this practice are the misconfiguration of
> servers, the inability of content authors to configure servers (either for
> technical reasons or lack of education), and generally the incentives placed
> upon browser vendors to "just work."
>
>> If and only if the media type is not given by a Content-Type field, the
>> recipient MAY attempt to guess the media type via inspection of its content
>> and/or the name extension(s) of the URI used to identify the resource. If
>> the media type remains unknown, the recipient SHOULD treat it as type
>> "application/octet-stream".
>
> encouraging the use of sniffing except where necessary. Ideally, it will be
> done in such a way that:
>
>  * Does not require sniffing for all uses of HTTP (i.e., a particular
> implementation and/or user can "opt in" to the use of a sniffing algorithm),
> since this is most commonly a problem for the browser case, and
>  * Specifically allows a user and/or content provider to opt out of the use
> of sniffing in a particular interaction, and
>  * Promotes interoperability (i.e., if two implementations sniff, they will
> do so in the same way).
>
> See:
> whether the sniffing algorithm would remain in a separate document (i.e.,
> our work would only be to relax requirements to allow it), or whether it
> would be in-document.
>
> In either case, we'd have to take a serious look at security considerations,
> and also look at impact on intermediaries, etc.
>
> Note that this is not about sniffing encoding directly (see issue #20),
> although the resolution may be related.
>
>
> --

---

From: Mark Baker <[email-masked]>
Date: Tue, 31 Mar 2009 14:13:47 +0000

> (apparently) isn't stopping any time soon.

That paragraph doesn't seem to me to apply to the typical sniffing
scenario discussed by draft-abarth-mime-sniff, as it deals only with
unspecified media types rather than their intentional overriding.

Obviously that kind of sniffing is widespread amoungst one class of
agent, and I'm all for documenting how it works, but IMO that has no
place in the protocol specification, save perhaps for an informative
note that it happens and a pointer to the algorithm.

Mark.

---

From: Adam Barth <[email-masked]>
Date: Tue, 31 Mar 2009 12:37:14 -0700

> note that it happens and a pointer to the algorithm.

When different user agents use different sniffing algorithms, content
authors pay a large cost, both in terms of compatibility and in terms
of security.  For user agents that wish to perform sniffing, I think
we'd be doing the Web a service by specifying which algorithm they
should use.

Adam

---

From: Mark Baker <[email-masked]>
Date: Tue, 31 Mar 2009 15:51:52 -0400

> we'd be doing the Web a service by specifying which algorithm they

I agree, which is why I suggested a link from 2616bis to the
algorithm.  Do you feel that to be insufficient?  If so, why?

Mark.

---

From: Adam Barth <[email-masked]>
Date: Tue, 31 Mar 2009 13:02:26 -0700

> algorithm.  Do you feel that to be insufficient?  If so, why?

I don't have a strong opinion about which document should contain the
algorithm, but I think we're better off making the algorithm normative
(for those agents that wish to sniff) rather than informative.  That
will help prevent developers of sniffing user agents from implementing
divergent sniffing algorithms.

Adam

---

From: David Morris <[email-masked]>
Date: Tue, 31 Mar 2009 13:37:50 -0700 (PDT)

> divergent sniffing algorithms.

I disagree ... encoding what is essentially a heuristic algorithm which 
will need to change as content types morph into standard status is the
wrong thing to do. Certainly in the HTTP standard.

I recall some months ago a 'proposal' for some kind of flag which 
essentially said believe what I say or reject my content .. sniffing not
allowed. Something like that makes sense.

Sniffing, if documented/standardized, needs to be a different document.
Sniffing started because of incorrect marking of response content. Getting 
engineers who needed to sniff in the first place to limit themselves to a
common algorithm seems unlikely. To even follow a highly static process in
a dynamic place like the web makes no sense to me.

Dave Morris

---

From: Adam Barth <[email-masked]>
Date: Tue, 31 Mar 2009 13:55:07 -0700

On Tue, Mar 31, 2009 at 1:37 PM, David Morris <[email-masked]> wrote:
> I disagree ... encoding what is essentially a heuristic algorithm which will
> need to change as content types morph into standard status is the

In practice, sniffing algorithms tend to calcify because authors
create content that works in one or more popular user agents.  Browser
vendors change their sniffing algorithms only after careful
consideration.

> allowed. Something like that makes sense.

Yes.  Internet Explorer 8 will refrain from sniffing if an HTTP
response contains a certain header.  I can provide more details about
how this works if you like.

> Getting
> common algorithm seems unlikely.

I agree that converging existing user agents on a common sniffing
algorithm will take time.  We've made some amount of progress on this,
but its a long road.

> To even follow a highly static process in

This statement appears to argue against all Web standards.

Adam

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 01 Apr 2009 10:06:36 +1300

Are there any plans to address the root cause of the issue?

E.g. content authors not having any mechanism to specify entity 
attributes, only entity content.

Combining the entity (MIME) headers with the transport (HTTP) headers 
has created this problem.  How about splitting them out again?

It would of course require an iteration of HTTP.

But surely until the root cause is fixed, we can only apply band-aids to 
the problem.

David Morris wrote:
> I disagree ... encoding what is essentially a heuristic algorithm 
> which will need to change as content types morph into standard status 
> is the
> Sniffing started because of incorrect marking of response content. 
> Getting engineers who needed to sniff in the first place to limit 
> themselves to a

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Julian Reschke <[email-masked]>
Date: Tue, 31 Mar 2009 23:08:12 +0200

> attributes, only entity content.

Depends on how the content is authored. In case it happens over HTTP, 
many servers already respect the Content-Type header sent with the 
content (in the PUT request).

 > ...

BR, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 01 Apr 2009 10:23:21 +1300

Sounds like a good argument for some apache mod.... e.g if the file has 
a certain extension, look for some <ATTRIB> </ATTRIB> section at the 
start of the file to contain entity attributes which are then stripped 
out of the content, but poked into the headers when returned by the server.

Then anyone can set whatever attributes they need to by putting it into 
their content, and the server doesn't need to maintain a separate 
database of that infomation, or try to deduce it itself (maybe by sniffing).

Do servers sniff to try and fill in the Content-Type field?  Most I 
think have a fairly simplistic static mapping of file extension to 
Content-Type.  Many types of content already have a signature in them 
which can be used to determine type. e.g jpegs, gifs etc.

Regards

Adrien

---

From: Adam Barth <[email-masked]>
Date: Tue, 31 Mar 2009 14:26:31 -0700

On Tue, Mar 31, 2009 at 2:23 PM, Adrien de Croy <[email-masked]> wrote:
> Do servers sniff to try and fill in the Content-Type field?

Yes.  We found this is quite common when we examined open-source Web
applications that accept user uploads.  For example, Wikipedia does
this.

> Most I think have a fairly simplistic static mapping of file extension to Content-Type.

This is how Apache works.

> Many types of content already have a signature in them which can be used to
> determine type. e.g jpegs, gifs etc.

Wikipedia uses this technique.  Mismatches between a site's sniffing
algorithm and the user agent's sniffing algorithm often lead to
exploitable vulnerabilities.  See Section 2.5 of
http://www.adambarth.com/papers/2009/barth-caballero-song.pdf for two
concrete examples of how this happens.

Adam

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 01 Apr 2009 10:54:01 +1300

So then surely the last word on what type of content something is, 
should be the actual content itself?

Things like Content-Type headers can be

* wrong (bad sniffing or mapping in server)
* missing
* tampered with

Relying on Content-Type therefore has associated risks.

So if any sniffing is to be done, surely it should only be the client?  
In which case why don't clients just ignore the Content-Type header 
always and always try and determine the type themselves.  Some seem to 
do this already.

Adrien

---

From: Adam Barth <[email-masked]>
Date: Tue, 31 Mar 2009 15:02:27 -0700

On Tue, Mar 31, 2009 at 2:54 PM, Adrien de Croy <[email-masked]> wrote:
> So then surely the last word on what type of content something is, should be
> the actual content itself?

Such an algorithm would maximize compatibility but cost security.

Suppose we had an oracle that told us the "true" MIME type for a given
HTTP response.  The Content-Type header would still be an important
security feature.  For example, consider a server that replies with
the following:

Content-Type: image/gif

<html><body>I am an HTML document</body></html>

If a user agent treats this response as text/html (supposing the
oracle agrees with our intuition that this response is, in fact,
HTML), then the user agent has likely opened the server up to a
cross-site scripting attack.  Instead, the user agent should treat
this response as an image.

> So if any sniffing is to be done, surely it should only be the client?  In
> which case why don't clients just ignore the Content-Type header always and
> always try and determine the type themselves.  Some seem to do this already.

None of the major browsers do this anymore because of these security issues.

Adam

---

From: Mark Baker <[email-masked]>
Date: Tue, 31 Mar 2009 20:25:14 -0400

On Tue, Mar 31, 2009 at 4:02 PM, Adam Barth <[email-masked]> wrote:
> algorithm,

Good 8-)

> but I think we're better off making the algorithm normative
> (for those agents that wish to sniff) rather than informative.

I don't know what it means for an algorithm to be normative.  Can you
elaborate, perhaps by offering the text you'd ideally like to see in
the HTTPbis spec (assuming the algorithm will be in a separate spec)?

Mark.

---

From: Barry Leiba <[email-masked]>
Date: Tue, 31 Mar 2009 21:26:56 -0400

> the HTTPbis spec (assuming the algorithm will be in a separate spec)?

I suspect he's going for something like (abbreviated version here),
"Implementations MAY [or SHOULD NOT] sniff.  If an Implementation does
sniff, it MUST do it this way...."

We should be careful about putting in normative text that we know a
good portion of implementors will ignore.  It's pretty much an
invitation to ignore other bits as well.

I'd be in favour of something like, "The use of different sniffing
algorithms in different implementations creates [these sorts of
problems].  Because of that, settling on common sniffing mechanisms is
important.  To that end, implementations that sniff SHOULD
[...etc...]."  The explanation of why it's important may sway some
implementors, and making it a SHOULD recognizes that some will go
their own way in any case.

Barry

---

From: Adam Barth <[email-masked]>
Date: Tue, 31 Mar 2009 18:35:14 -0700

> their own way in any case.

This sounds reasonable to me.

Adam

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 31 Mar 2009 19:44:57 -0700

It is impossible to determine a media-type (how a recipient should
process a given representation) by sniffing the content (the data  
format).
Many media-types overlap the same data formats.  Many media types are
only distinguished from other types by what the user agent should do
when processing them, not by the bits on the wire.  When a media type
is not present (or is detectably incorrect), only the implementation
doing the processing can determine an appropriate guess because that
guess is almost always determined by the context in which the
reference was made (not by the content).  Since the context is
deliberately not sent on the wire, there is absolutely no way that
accurate sniffing can be defined by HTTP.  We aren't talking about
a protocol decision regarding communication; we are talking about
an operating default that is specific to the purpose of a given
client and will likely be different for each one.

In any case, there is no algorithm for sniffing that is anywhere
near the same level of standardization as HTTP.  The one that HTML5
is working on would barely qualify as Experimental.  If the folks
promoting such software can successfully deploy it across all HTTP
clients, then it should be referenced.  Until then, it remains an
unproven and, IMO, mistaken idea which is far more likely to
be overcome by events than become a standard way to handle HTTP.

....Roy