Subject: Re: NEW ISSUE: content sniffing

From: =JeffH <[email-masked]>
Date: Tue, 07 Apr 2009 15:11:47 -0700

> Ian Hickson <[email-masked]> wrote:
 >
 > We're not saying HTTP should care about them. We're saying that one of
 > these two options should be picked:
 >
 > Either:
 >
 > a) HTTP should not have any requirements for how to process Content-Type
 > headers, and should just leave Content-Type to another spec, or:
 >
 > b) HTTP should include the proposed limited Content-Sniffing algorithm,
 > which would allow us to get software tools to converge on a single set of
 > heuristics and thus reduce the security risk.

upon reflection WRT the above, I would pick option (a) -- i.e. RFC2616 (s7.2.1) 
arguably says "too much" concerning the Content-Type header field. My 
overriding rationale being that there is a (growing) plethora of "protocols" 
layered upon HTTP, and RFC2616 was written back when a "web browser cum user" 
(my poor phrase) was arguably dominant.

Here's the relevant last para of section 7.2.1..

     7.2.1 Type

     <snip/>

     Any HTTP/1.1 message containing an entity-body SHOULD include a
     Content-Type header field defining the media type of that body. If
     and only if the media type is not given by a Content-Type field, the
     recipient MAY attempt to guess the media type via inspection of its
     content and/or the name extension(s) of the URI used to identify the
     resource. If the media type remains unknown, the recipient SHOULD
     treat it as type "application/octet-stream".

Specifically, I'm thinking that the last paragraph really should read
something more like..

     Any HTTP/1.1 message containing an entity-body SHOULD include a
     Content-Type header field defining the media type of that body.
     Processing of the entity-body SHOULD be based on the Content-Type
     header field value and MAY be based as well on other header field
     values.  Definition of such further processing is out-of-scope of
     this specification. If the media type is not given by a
     Content-Type header field the recipient's behavior is not defined
     by this specification.

..though I'm not very sure about the last sentence (wrt absence of
Content-Type header field).

Might this -- i.e. the last para of RFC2616 s7.2.1 presently saying "too much" 
-- be viewed as a "bug" in RFC2616?

Overall this thread of course begs the question of where/how to publish 
draft-abarth-mime-sniff and/or other specs defining behavior(s) of (certain 
classes of) HTTP-based apps and their implementations. E.g. how do we signal to 
browser vendors et al the set of HTTP-oriented specs (for example) they ought 
to adhere to (this question of course applies to the specs of a wide array of 
protocols and formats).

thanks,

=JeffH

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Apr 2009 09:23:39 +1000

Personally, I can get on board with this approach.

The only thing that I think may need to be added (and I think this was  
discussed in SF) is advice on allowing origins and users the ability  
to opt out of sniffing on a per-response basis. Putting that advice in  
HTTPbis is probably best, although I could see arguments for putting  
it in the sniffing algorithm.

WRT defining the 'stack' of specs that a browser should adhere to --  
this has come up in side conversations, but never addressed fully. I  
know Lisa is quite interested in defining "profiles" for use of HTTP,  
so that other IETF specs can specify them and not have to worry about  
all of the details of what should or should not be done.

Likewise, a profile for browsers sounds like it would be useful.

If HTTPbis does its job well, the new HTTP specs should be much more  
suitable for profiling than 2616, both because of the explicit split  
into many components, and because we're chartered to more carefully  
define extensibility.

I can see this group starting to work on a HTTP profile for IETF specs  
at some point (arguably soon, if there's interest, so that we can get  
feedback from building the profile into HTTPbis). I'm not as sure that  
doing a browser profile in this group makes as much since, because the  
profile they're interested  in has a lot of other technologies in it  
as well...

AIUI HTML5 was originally that profile, but since recently it's been  
fragmenting into multiple specs, it may be that a separate browser  
profile is needed. CC'ing Sam and Chris to get their thoughts on  
whether this is something they'd like to take on, and what level of  
liaison might be necessary.

On 08/04/2009, at 8:11 AM, =JeffH wrote:

> upon reflection WRT the above, I would pick option (a) -- i.e.  
> RFC2616 (s7.2.1) arguably says "too much" concerning the Content- 
> Type header field. My overriding rationale being that there is a  
> (growing) plethora of "protocols" layered upon HTTP, and RFC2616 was  
> written back when a "web browser cum user" (my poor phrase) was  
> arguably dominant.
>
> Might this -- i.e. the last para of RFC2616 s7.2.1 presently saying  
> "too much" -- be viewed as a "bug" in RFC2616?
>
>
> Overall this thread of course begs the question of where/how to  
> publish draft-abarth-mime-sniff and/or other specs defining  
> behavior(s) of (certain classes of) HTTP-based apps and their  
> implementations. E.g. how do we signal to browser vendors et al the  
> set of HTTP-oriented specs (for example) they ought to adhere to  
> (this question of course applies to the specs of a wide array of  
> protocols and formats).
>
> thanks,
>
> =JeffH
>
>
>
>
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Adam Barth <[email-masked]>
Date: Tue, 7 Apr 2009 16:47:54 -0700

On Tue, Apr 7, 2009 at 4:23 PM, Mark Nottingham <[email-masked]> wrote:
> discussed in SF) is advice on allowing origins and users the ability to opt
> out of sniffing on a per-response basis. Putting that advice in HTTPbis is
> probably best, although I could see arguments for putting it in the sniffing
> algorithm.

As you might be aware, IE8 and Chrome let servers opt out of content
sniffing using a HTTP header.  I've done some preliminarily
measurement experiments on the use of this header:

1) The nosniff directive is included in about 8% of all HTTP responses
received by Chrome.  (I believe that virtually all of google.com uses
this directive, for example.)

2) Of the HTTP responses that include the nosniff directive,
approximately 1.5% of them lack a Content-Type header.  (Compare this
to the ~1% of such responses in the general population.)

The nosniff directive has caused some small amount of incompatibility
because servers both specify the directive and require sniffing for
proper operation.  These sites appear relatively responsible to
evangelism and tend to correct their use of the Content-Type header
rather than abandon the nosniff directive.

Eric Lawrence might have some additional implementation experience to
share on this topic.

Adam

---

From: Mark Baker <[email-masked]>
Date: Tue, 7 Apr 2009 20:25:46 -0400

On Tue, Apr 7, 2009 at 6:11 PM, =JeffH <[email-masked]> wrote:
>    values.

That text isn't specifying a protocol, it's specifying how
implementations should behave.  We need the protocol specification to
specify what messages *mean* when the Content-Type header is there and
when it isn't, and we already have that text.

My previous suggestion is as far as I would feel comfortable going
along these lines;

http://lists.w3.org/Archives/Public/ietf-http-wg/2009AprJun/0018.html

Mark.

---

From: Ian Hickson <[email-masked]>
Date: Wed, 8 Apr 2009 04:58:57 +0000 (UTC)

On Tue, 7 Apr 2009, Adam Barth wrote:
> sniffing using a HTTP header.

To be precise, they allow servers to opt out of content sniffing in 
certain specific cases. It doesn't affect, for instance, how the 
Content-Type header is treated for images (e.g. an image/png image sent as 
image/gif is still treated as a PNG, even with this header set, if I'm not 
mistaken; and Content-Type headers are entirely ignored in certain 
contexts like <embed> elements with explicit type="" attributes or 
@font-face declarations in CSS).

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Adam Barth <[email-masked]>
Date: Tue, 7 Apr 2009 22:44:10 -0700

On Tue, Apr 7, 2009 at 9:58 PM, Ian Hickson <[email-masked]> wrote:
> mistaken;

IE8 has a more awesome implementation than Chrome.  In IE8, these
images won't render, but in Chrome they will.  Eventually, I'd like to
make Chrome's implementation match IE8, but it's not that high a
priority.  The main security win comes from not sniffing when loading
a frame.

> and Content-Type headers are entirely ignored in certain

I haven't tested these cases.

Adam

---

From: Ian Hickson <[email-masked]>
Date: Wed, 8 Apr 2009 05:47:45 +0000 (UTC)

On Tue, 7 Apr 2009, Adam Barth wrote:
> > Content-Type header is treated for images (e.g. an image/png image 
> > sent as image/gif is still treated as a PNG, even with this header 
> > set, if I'm not mistaken;
> 
> IE8 has a more awesome implementation than Chrome.  In IE8, these images 
> won't render

Even in <img> elements?

---

From: Adam Barth <[email-masked]>
Date: Tue, 7 Apr 2009 22:48:24 -0700

>> IE8 has a more awesome implementation than Chrome.  In IE8, these images

I think so.  /me goes and makes a test case.

Adam

---

From: Adam Barth <[email-masked]>
Date: Tue, 7 Apr 2009 22:54:46 -0700

> I think so.  /me goes and makes a test case.

Yep.  Well, I tested a GIF with a Content-Type header of image/png.
Sorry I don't have the test case available at a public URL.  I use
netcat for these kinds of tests to make sure I get the network bytes
rights.

I think the <script> tag still accepts any type, but we got a request
from someone on the security team to make that strict when the nosniff
directive is enabled.  It's unclear what we'll end up doing in that
case.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Apr 2009 16:00:48 +1000

It seems like Mark's proposal is the minimum required to declare  
victory, from an HTTP standpoint at least.

Remove this text from p3 section 3.2.1:
> "If and only if the media type is not given by a Content-Type field,  
> the recipient MAY attempt to guess the media type via inspection of  
> its content and/or the name extension(s) of the URI used to identify  
> the resource."
We'd still need security considerations text.

Regarding a mechanism to opt in or out of sniffing -- while this is a  
necessary discussion, it would be a new mechanism, as well as new  
conformance requirement (almost certainly making existing  
implementations non-conformant), so it's out of scope for us. That  
said, drafts can still be reviewed on-list, it just isn't a WG product.

Thoughts?

---

From: Adam Barth <[email-masked]>
Date: Tue, 7 Apr 2009 23:21:39 -0700

On Tue, Apr 7, 2009 at 11:00 PM, Mark Nottingham <[email-masked]> wrote:
> It seems like Mark's proposal is the minimum required to declare victory,
> from an HTTP standpoint at least.
>
>> "If and only if the media type is not given by a Content-Type field, the
>> recipient MAY attempt to guess the media type via inspection of its content
>> and/or the name extension(s) of the URI used to identify the resource."

I'm not an expert at spec reading, but the spec would still say:

"When an entity-body is included with a message, the data type of that
body is determined via the header fields Content-Type and
Content-Encoding."

This seems false since the data type might be determined after taking
other information into account.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Apr 2009 16:26:51 +1000

I think the disconnect here is that HTTP folks are assuming that this  
statement is made within the scope of HTTP; i.e., someone using HTTP  
will take that value and figure out what to do with it.

---

From: Adam Barth <[email-masked]>
Date: Tue, 7 Apr 2009 23:31:46 -0700

Maybe we should say something like:

"When an entity-body is included with a message, the data type of that
body SHOULD be determined via the header fields Content-Type and
Content-Encoding."

That seems to clarify the level of conformance required.

Adam

On Tue, Apr 7, 2009 at 11:26 PM, Mark Nottingham <[email-masked]> wrote:
> statement is made within the scope of HTTP; i.e., someone using HTTP will
> take that value and figure out what to do with it.
>
>
>>>> content

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Apr 2009 16:49:07 +1000

I don't think a SHOULD is necessary here; 'via' says that they are  
part of the process, not necessarily the whole process. SHOULD is a  
pretty poor way of clarifying conformance, after all :)

I could see s/via/using/ if you think it would help...

---

From: Adam Barth <[email-masked]>
Date: Tue, 7 Apr 2009 23:55:23 -0700

Now we're beyond my expertise.  :)

I think the point is to recommend using these headers but let
conforming implementations use additional information if they need to.
 Maybe something like:

"Implementations MAY use additional information (such as request
context, response headers, or the entity body itself) in determining
the data type, but [doing so can have security consequences; see
Section X.Y.Z]."

Adam

On Tue, Apr 7, 2009 at 11:49 PM, Mark Nottingham <[email-masked]> wrote:
> I don't think a SHOULD is necessary here; 'via' says that they are part of
> the process, not necessarily the whole process. SHOULD is a pretty poor way
> of clarifying conformance, after all :)
>
>>>>>> the

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Apr 2009 17:06:41 +1000

Yep. The point I'm making is that because there isn't a RFC2119  
requirement there currently, conforming implementations are fine;  
adding a new one just adds complexity...

On 08/04/2009, at 4:55 PM, Adam Barth wrote:

>> part of

---

From: Adam Barth <[email-masked]>
Date: Wed, 8 Apr 2009 00:09:15 -0700

On Wed, Apr 8, 2009 at 12:06 AM, Mark Nottingham <[email-masked]> wrote:
> Yep. The point I'm making is that because there isn't a RFC2119 requirement
> there currently, conforming implementations are fine; adding a new one just
> adds complexity...

Ok.  Thanks for the crash course in reading these specs.  :)

Adam

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 08 Apr 2009 02:42:12 +0000

> http://lists.w3.org/Archives/Public/ietf-http-wg/2009AprJun/0018.html

+1. There's a lot of text in RFC 2616 that are better off removed because
they don't actually help define the protocol; often these parts of the spec
just create confusion. This is a prime example.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 08 Apr 2009 15:30:39 +0200

> other information into account.

First of all, we're only discussing Content-Type, *not* Content-Encoding 
right?

That being said, in the spirit of defining the meaning of the message, 
not it's processing, how about:

"When an entity-body is included with a message, the data type of that
body is declared using the header fields Content-Type and Content-Encoding."

?

BR, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 8 Apr 2009 08:31:22 -0700

On Wed, Apr 8, 2009 at 6:30 AM, Julian Reschke <[email-masked]> wrote:
> That being said, in the spirit of defining the meaning of the message, not
> it's processing, how about:
>

That sounds good to my untrained ear.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 9 Apr 2009 08:40:50 +1000

Works for me.

On 08/04/2009, at 11:30 PM, Julian Reschke wrote:

> First of all, we're only discussing Content-Type, *not* Content- 
> Encoding right?
>
> That being said, in the spirit of defining the meaning of the  
> message, not it's processing, how about:
>

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 09 Apr 2009 12:04:01 +1200

Julian Reschke wrote:
>
> First of all, we're only discussing Content-Type, *not* 
> Content-Encoding right?
>

to me that implies that Content-Encoding is always required, whereas in 
fact it's only required if there is an encoding also applied to the content.

I'd rather leave C-E out of it, or if referring to it, make it clear 
it's only required when there is an encoding.

Also it's not clear what "data type" means.  In the above text it 
implies some aggregation of "MIME" type plus encoding.  I think this 
could be confusing.  The concepts of type and encoding are discrete and 
I think should remain so.

What about something like:

"When an entity-body is included with a message, attributes of that 
entity-body are declared in Content-* headers.

Content-Length SHOULD be used to declare the length in bytes of the 
content where known and where Transfer-Encoding: chunked is not used.
Content-Type SHOULD be  used to declare the type of the content.
Content-Encoding MUST be used to declare any encoding where that may 
have been applied to the content.

..."

with references to relevant sections on those headers.

?

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 9 Apr 2009 10:04:16 +1000

On 09/04/2009, at 10:04 AM, Adrien de Croy wrote:

> to me that implies that Content-Encoding is always required, whereas  
> in fact it's only required if there is an encoding also applied to  
> the content.
>

It might be good to step back and look at the context:

> When an entity-body is included with a message, the data type of that
> body is determined via the header fields Content-Type and Content-
> Encoding. These define a two-layer, ordered encoding model:
>
>     entity-body := Content-Encoding( Content-Type( data ) )
>
> Content-Type specifies the media type of the underlying data.
> Content-Encoding may be used to indicate any additional content
> codings applied to the data, usually for the purpose of data
> compression, that are a property of the requested resource. There is
> no default encoding.

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 09 Apr 2009 12:09:18 +1200

> It might be good to step back and look at the context:
fair call -sorry!

---

From: Yves Lafon <[email-masked]>
Date: Thu, 9 Apr 2009 03:35:51 -0400 (EDT)

On Thu, 9 Apr 2009, Adrien de Croy wrote:

> Content-Length SHOULD be used to declare the length in bytes of the content 
> where known and where Transfer-Encoding: chunked is not used.
> Content-Encoding MUST be used to declare any encoding where that may have 
> been applied to the content.

Why making a difference betwee CT and CE?
If the content is gzipped, it can be both CT application/gzip or CT 
whatever, CE: gzip. Why should be force CE to be filled if CT is not?

And Content-Length is a weird beast as it applies to both the 
entity-length and transfer-length.
Btw, your last sentence "Content-Encoding MUST be used to declare any 
encoding..." seems to apply more to Transfer Encoding rather than CE, as 
in that case you really want the encoding explicit to present the right 
content once all the connection-level processing is done..

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Julian Reschke <[email-masked]>
Date: Thu, 09 Apr 2009 14:46:13 +0200

Mark Nottingham wrote:
> ...

So 3.2.1 would become:

-- snip --
3.2.1 Type

When an entity-body is included with a message, the data type of that 
body is declared by the header fields Content-Type and Content-Encoding. 
These define a two-layer, ordered encoding model:

Content-Type specifies the media type of the underlying data. 
Content-Encoding may be used to indicate any additional content codings 
applied to the data, usually for the purpose of data compression, that 
are a property of the requested resource. There is no default encoding.

Any HTTP/1.1 message containing an entity-body SHOULD include a 
Content-Type header field defining the media type of that body. If the 
media type remains unknown, the recipient SHOULD treat it as type 
"application/octet-stream".
-- snip --

Note that by removing the second sentence from the last paragraph, we 
now have a dangling "...if remains unknown...". So should this sentence 
go as well? (note that declaring application/octet-stream is really 
different from not declaring the type at all, IMHO).

> We'd still need security considerations text. 

So what would these be if we do not even mention sniffing?

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 9 Apr 2009 09:56:24 -0500

> "application/octet-stream".

My suggestion is:

1. Remove 3.2.1 completely.
2. Remove all references to 3.2.1 (especially the one in section 5.9).
3. Add the statement "A message containing an entity-body SHOULD include a
Content-Type header field defining the media type of that body" to section
5.9.

Since the "MUST" requirement for Content-Encoding is in 5.5, it makes sense
to have the "SHOULD" requirement for Content-Type in 5.9. And, the
specification should avoid duplication as much as possible.

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 09 Apr 2009 17:08:56 +0200

> 1. Remove 3.2.1 completely.

I think this goes to far, as it explains the interaction between 
Content-Encoding and Content-Type.

> specification should avoid duplication as much as possible.

That makes sense.

Related issue: Section 5.5 currently is phrased as if Content-Encoding 
only works in presence of Content-Type:

"The entity-header field "Content-Encoding" is used as a modifier to the 
media-type. When present, its value indicates what additional content 
codings have been applied to the entity-body, and thus what decoding 
mechanisms must be applied in order to obtain the media-type referenced 
by the Content-Type header field."

It seems to me we should rephrase that as well.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 9 Apr 2009 10:20:11 -0500

> Content-Encoding and Content-Type.

It is already explained well enough in 5.5, particularly in the section you
quoted below. The explanation in 3.2.1 also has the same problem you mention
below; "Content-Encoding(Content-Type(data))" doesn't make sense when there
is no Content-Type, especially if there is no default content-type.
 
> It seems to me we should rephrase that as well.

Additionally, the whole description of Content-Encoding is written
specifically for HTTP responses (e.g. "request-target", "...the response
MUST include...".

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 09 Apr 2009 17:31:05 +0200

> is no Content-Type, especially if there is no default content-type.

Disagreed; I think the concept itself needs to be explained as part of 
Section 3, and not be hidden in the definition of a specific header.

> specifically for HTTP responses (e.g. "request-target", "...the response

Indeed.

BR, Julian

PS: Mark, should we fix all of this as part of issue 155, or do we want 
to track this separately?

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 11 Apr 2009 10:42:55 -0500

Julian Reschke wrote:
> > It is already explained well enough in 5.5, particularly in the
> > section you quoted below. The explanation in 3.2.1 also has
> > the same problem you mention below;
> > "Content-Encoding(Content-Type(data))" doesn't make sense when
> > there is no Content-Type, especially if there is no default
> > content-type.
> 

Why do you think so? Removing 3.2.1 makes the specification shorter and more
precise, thus easier to understand. Almost every sentence in 3.2.1 is wrong
in some way (or at least confusingly imprecise) and correcting them all will
make 3.2.1 even more redundant with 5.5 and 5.9. There are many other
instances in RFC 2616 where the introductory/tutorial text creates confusion
with the normative (SHOULD/MUST/MAY) parts of the specification. The best
strategy for resolving these conflicts is to reduce the tutorial text to its
absolute minimum. That will lead to the clearest specification and it should
considerably reduce the amount of time it takes to finish HTTPbis.

Somebody can write a HTTP primer or an annotated version of the spec for
implementers that need more exposition. I think there is a need for such a
document anyway, because many people care about how web browsers and other
kinds of UAs diverge from or subset the specification. Such a document would
be an excellent place to describe content sniffing and its pros and cons,
for example.

- Brian

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 12 Apr 2009 20:05:33 +0100

> be an excellent place to describe content sniffing and its pros and cons,

And because many people read the existing spec and implement it wrong
either by accident or because they're not sure which parts are really
important, and they're not going to do it all.  Really, they do.

-- Jamie