Subject: comments on draft-abarth-mime-sniff-03

From: Larry Masinter <[email-masked]>
Date: Wed, 20 Jan 2010 14:41:43 -0800

Although this document was discussed a while back
in httpbis, it seems to address non HTTP sniffing
as well. 

I'll send notes to http-wg, html-wg, www-tag
to move discussion here (as advised).

Several W3C specs are advancing using this document 
as a normative reference (including HTML), so getting 
this reviewed should be timely.

The document should note the venue for discussion.

I admit I haven't chased down the archives on
the previous discussion of this, and maybe I've
gotten some things wrong, but the current draft
is already 4 months old. (Compared to other drafts
the authors are updating regularly.) 

=============================
All or nothing:

  Whenever possible, user agents should avoid
  employing a content sniffing algorithm.

The normative advice seems to apply to roles
other than "user agents"; whether or not
there is a user isn't really relevant.

Secondly, it's not clear the scope of the
"should avoid" or whether this is really
intended to be normative.  I'll assume
that it is.

   However, if a user agent does employ a
   content sniffing algorithm, the user agent
   should use the algorithm in this document exactly

The algorithm isn't exact (more below).

And think this is the wrong normative advice.
Implementations SHOULD NOT sniff labeled content,
and MUST NOT sniff any way beyond the ways 
explicitly allowed here, and when they do sniff,
should opt into sniffing on a situation by situation,
type by type basis.

The application of sniffing should be reserved 
for situations where the implementor has sufficient
reason to believe that there is sufficient existing 
deployed content that would not function as expected 
if sniffing were not implemented, and these situations
should be encouraged to be as fine a granularity
as the implementor needs.

This normative advice should not require
updating as the deployed content evolves.

=============================
TERMINOLOGY "resource"

This document seems to have the same use of "resource"
to talk about what is fetched and not just the source
from which it is fetched, as discussed in HTML-WG
at length:

http://www.w3.org/html/wg/tracker/issues/81

For example

   For HTTP resources, only the last Content-Type HTTP header,
   if any, contributes any type information; the official type 
   of the resource is then the value of that header, 
   interpreted as described by the HTTP specifications.  

But HTTP specification cited says is:

   The "Content-Type" entity-header field indicates the
   media type of the entity-body. 

not applying "media type" to the resource but as metadata
of the response.

For example, instead of using "resource" inconsistently,
this sentence might say:

  For HTTP resources ... the official media type of
  the representation is obtained from the value ...

since of course, the same HTTP resource might return
different representations depending on accept headers.

==========================================================
Malformed content-type information vs. missing content-type 

I think the cases where content-type information supplied is
malformed should be treated differently than the cases where
content-type information is missing. In particular, it
should be possible and allowed to raise an error
if content-type information is malformed, even when
sniffing is performed on well-formed, but incorrect, 
content-type. That is, the "opt-in" behavior could allow
an implementation to sniff when no content-type, but
not sniff when there is a malformed content-type.

==============================================================
Contextual application

There are many different contexts of use of MIME labeled
material in the web, and the opt-in behavior for sniffing
should be allowed to be contextualized. For example, one
might opt in to sniffing for <img src=...> but opt out of
sniffing a script or rel link and opt in when doing a GET
on the main body of a web page.

================================================================
multiple content-type headers are malformed:

> For HTTP resources, only the last Content-Type HTTP header,
> if any, contributes any type information

=============================================================
A message with more than one content-type header
should be treated as malformed.

   If the Content-Type HTTP header is present but
   the value of the last such header cannot be 
   interpreted as described by the HTTP 
   specifications (e.g. because its value doesn't 
   contain a U+002F SOLIDUS ('/') character), then
   the resource has no type information (even 
   if there are multiple Content-Type HTTP headers and
   one of the other ones is syntactically correct).

=============================================
The "algorithm for extracting an encoding ...."
 
   Note: The above algorithm is a willful violation of the HTTP
   specification.  [RFC2616]

The word "encoding" is confusing, since HTTP uses
content-encoding and this is actually talking 
about what HTTP calls "charset".

The HTML algorithm for 'sniffing' a charset when
one is explicitly supplied but is changed by HTML --
that isn't part of this draft? It is HTML only?
It seems to be part of the same family of
"sniffing" and should be reviewed with the
same scrutiny, moving it either into this
document or to a parallel one. (Perhaps someone
can raise this as a bug in the HTML spec.)

The nature of the "willful violation" 
(I.e., how it is different) and the 
justification for the "willful violation" 
should be included. I can't fathom any
justification for it.

========================
file extensions:

  Note: It is essential that file extensions
  are not used for determining the media type
   for resources fetched over HTTP because
  file extensions can often by supplied by
   malicious parties.

 "Often" is dubious. How can file extensions be
supplied more often than  content-type headers?
What is the security threat? 

  For resources fetched over most other protocols, e.g. 
  FTP, there is no type information.

"most other protocols" is imprecise.
Does this apply to imap? data:? There seems
to be some attempt to define this for RSS?

Is it the protocol or the scheme that 
determines the method, or both?

I also don't think this matches implementations
for FTP, aren't file extension used frequently
to determine content-type?

=========================
Waiting (section 5)

   1.  The user agent MAY wait for 512 or more bytes
   of the resource to be available.
   2.  Let /stream length/ be the smaller of either
   512 or the number of bytes already available.

*This* document doesn't define "available" or what
it means to "wait" or really have much to do with
what most of this document is about. I think the
choice here is that type sniffing apparently
can work on whatever prefix the agent chooses
to give it, from anywhere from 0 to 512 bytes.
If it chooses to send 0 bytes, this is equivalent
for turning off MIME sniffing completely 
(since there is nothing to sniff).

I'd think that the behavior of "how to sniff"
should start out with what the inputs are
(the first N bytes of some data from a response).

Note that the n bytes are not the bytes
of the actual traffic over the wire, but the
bytes resulting from undoing any
content-encoded.

============
Adding new types:

" User agents may support additional types if desired,
   by implicitly adding to the above table."

This makes no sense and is a disaster for
interoperability. 

The only reason why we have badly deployed content
is that user agents "sniffed" new types. If the
deployed infrastructure doesn't deploy new
kinds of sniffing, then people won't distribute
content.  This is a harmful escalating path and
encouraging implementations to add to the table
is disruptive and harmful. Don't do it.

---

From: Adam Barth <[email-masked]>
Date: Wed, 20 Jan 2010 15:14:37 -0800

Thanks Larry.  These are great comments.  I'll incorporate them when I
next update the draft.  To answer a couple of your questions:

> should be treated as malformed.

What does it mean to treat the response as malformed?  I've seen
examples of servers that blissfully send more than one Content-Type
header.  This document just describes how to process the responses
without making a judge about whether the server is acting properly or
not.

> The "algorithm for extracting an encoding ...."
[...]
> should be included. I can't fathom any

Charset sniffing is required to avoid ugly replacement characters from
being shown to the user.  Sad, but true.

> supplied more often than  content-type headers?

For example, the attacker can chose the file extension in most PHP
installations because foo.php happily processes:

http://example.com/foo.php/bar.qux

> What is the security threat?

The security threat is that if you treat an HTML file extension as
evidence the server wants the response to be treated as text/html you
will introduce XSS vulnerabilities into some large number of sites
running PHP (among others).

> (the first N bytes of some data from a response).

This business about waiting for 512 bytes has to do with a poor
interaction between buffering for sniffing and Comet
<http://en.wikipedia.org/wiki/Comet_(programming)>.  Basically, if you
wait forever for the 512 bytes you need to sniff completely, then you
break things like chat in Gmail.  For example, Gmail chat used to not
work in Safari for this reason.  However, always using the first chunk
of data off the network to sniff means you'll get unpredictable
results based on how exactly the response was chunked.  Hence the
advice to wait for 512 bytes but not a requirement to wait forever.

Adam

---

From: Larry Masinter <[email-masked]>
Date: Fri, 22 Jan 2010 17:34:46 -0800

> without making a judge about whether the server is acting properly or

Implementations should be allowed to not do sniffing when
the content-type is malformed, even when they do sniffing
when the content-type is missing, or specific kinds of
sniffing when the content-type is supplied but just wrong.

Malformed content is an indication of more serious site
misconfiguration than the typical excuse that the default
Apache installation used to label any file extension it
didn't understand as "text/plain" rather than
"application/octet string"

> being shown to the user.  Sad, but true.

I'm sorry, but the paragraphs preceding the discussion of
"willful violation" seem to be talking about how to determine
the value of the charset= MIME parameter, not about the 
interpretation of the results. As I pointed out, this 
document doesn't include the algorithm for charset
sniffing, which still seems to be in the HTML4 document.
If you mean by your disclaimer of a "willful violation" that
you were doing charset sniffing, then moving that text from
the HTML specification into this one (or, as I had also
thought reasonable) a parallel one, would seem to be more
reasonable. The reason for including an algorithm for parsing
the MIME parameters that differs from the ordinary one requires
more justification than you've supplied.

> http://example.com/foo.php/bar.qux

I suggest including this example in the security considerations section
of the document.

> running PHP (among others).

Yes, I think this belongs in the "security considerations" section
of the document.

> advice to wait for 512 bytes but not a requirement to wait forever.

I'm just pointing out that the consequence is that sniffing is indeterminate,
and that even if you have an implementation that claims that it sniffs,
that it might not if the server responds slowly and your wait times out.
This might even be an attack vector? (I'm not sure how, though).

I don't think you've made clear the advantage of sniffing at all until
you've received the entire message body, much less the first 512 bytes.

Your example of gmail chat isn't convincing... I suppose the browser
doesn't know that it's talking to a presumably sniffing-not-necessary
site (Google), but the first 512 bytes of the response might not arrive
in a timely fashion?

Maybe you could explain this use case in more detail.

Larry

---

From: Adam Barth <[email-masked]>
Date: Fri, 22 Jan 2010 19:25:47 -0800

At a higher level, what do folks think about re-writing the draft in a
more informative style instead of a normative style?  A bunch of
Larry's points boil down to the strengths of the normative
requirements and the scope of the affected user agents.  I certainly
have no wish to ram sniffing down anyone's throats.  I'd rather
provide this document as a reference for folks who feel compelled to
do content sniffing but who don't want to invest the year and a half
of research that my colleagues and I invested to arrive at this
algorithm.

Some comments below.

> "application/octet string"

To ground this discussion, here's a real-world example from Google
News.  Google News syndicates images from a bunch of news sources.
For a while, instead of providing the content type of the images, the
servers were providing the "magic numbers" that are found at the
beginning of the images.  Once we discovered the issue, we convinced
the team to fix the issue.

> sniffing, which still seems to be in the HTML4 document.

I presume you mean HTML5.

> more justification than you've supplied.

This text originated from the HTML5 specification.  It's entirely
possible it doesn't make sense without the surrounding context.  I'll
investigate.

> I suggest including this example in the security considerations section

Will do.

> Yes, I think this belongs in the "security considerations" section

Will do.

> This might even be an attack vector? (I'm not sure how, though).

In reality, the sniffing algorithm is highly predictable.  Although
it's possible for the HTML heuristic to use all 512 bytes, it's
extremely rare.  In fact, in 97% of cases when the heuristic fires,
the HTML tag begins at the first byte of the entity-body.  It's also
possible for the text-or-binary algorithm to consume all 512 bytes,
but in most binary files, the first "non-text" character occurs well
before the 512th byte.

This is not an attack vector, that I'm aware of, because the default
types you could force by truncating the sniffing buffer are "safe"
(e.g., text/plain or application/octet-stream).

> you've received the entire message body, much less the first 512 bytes.

There's no reason to wait for the entire message body.  The algorithm
considers only the first 512 bytes.  Also, waiting for the whole
entity body would be disastrous to performance.

In practice, waiting for the first 512 bytes is very cheap, so you
might as well do it to make the algorithm 100% predictable.  In some
rare cases, it's expensive to wait, so the epsilon extra
predictability isn't worth the cost, e.g., breaking Gmail chat.

> Your example of gmail chat isn't convincing...

Isn't convincing to whom?  To you?  It was convincing enough to the
Safari team that they changed their sniffing algorithm in this regard
to what's described in the document.

> I suppose the browser

In this case, the server was responding with a sniffable Content-Type
and the first 512 bytes would never arrive.  It sounds like you don't
understand how Comet works.  I'd encourage you to read the citation I
provided before giving your opinion about things you don't understand:

http://en.wikipedia.org/wiki/Comet_(programming)

> Maybe you could explain this use case in more detail.

The use case is as follows:

1) The server responds with a sniffable Content-Type and a chunked encoding.
2) The server sends the first chunk (say of five bytes) and then blocks.
3) The client blocks waiting for the server to send more content.
4) ... time passes ...
5) The server sends the second chunk (say another five bytes) and then blocks.

Now, what the web site wants is that the first five bytes are
delivered to the XMLHttpRequest API soon after they arrive from the
server.  Then, the web site wants the second five bytes to be
delivered to the XMLHttpRequest API soon after they arrive from the
server.  This techniques, known as Comet, allows the web site to
simulate a server data "push" and is why you're able to receive a
Gmail chat message soon after your contact sends it.  However, if the
user agent blocks the response waiting for 512 bytes, those bytes will
never arrive and you'll never get your chat messages.

Of course, there are ways for the server to work around this issue by
spamming the Comet channel with 512 bytes of junk at the beginning or
specifying a non-sniffable media type.  However, the reality is that
most user agents won't wait forever for the 512 bytes needed for
sniffing.  Instead, they'll get board of waiting and deliver the bytes
to the XMLHttpRequest API.  Hence, server operators don't employ these
workarounds, and if you're the lone user agent that doesn't behave
this way (e.g., Safari), then Gmail chat doesn't work in your browser
and your users are sad.  Sad users stop using your browser and use
another browser.

All that is a long-winded way of saying that these things are in the
draft for a reason.  The sniffing algorithm is highly constrained by
reality.  Looking for philosophical purity in content sniffing is like
trying to find a Newtonian explanation of Brownian motion.

Adam

---

From: Ian Hickson <[email-masked]>
Date: Sat, 23 Jan 2010 06:32:18 +0000 (UTC)

On Fri, 22 Jan 2010, Larry Masinter wrote:
> >>
> >> A message with more than one content-type header should be treated as 
> >> malformed.
> > 
> Implementations should be allowed to not do sniffing when the 
> content-type is malformed, even when they do sniffing when the 
> content-type is missing, or specific kinds of sniffing when the 
> content-type is supplied but just wrong.

I strongly disagree. Getting reliably interoperable behaviour in all user 
agents trumps theoretical concerns about what is right or wrong, IMHO.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Ian Hickson <[email-masked]>
Date: Sat, 23 Jan 2010 06:41:09 +0000 (UTC)

On Fri, 22 Jan 2010, Adam Barth wrote:
> more informative style instead of a normative style?  A bunch of Larry's 
> points boil down to the strengths of the normative requirements and the 
> scope of the affected user agents.  I certainly have no wish to ram 
> sniffing down anyone's throats.  I'd rather provide this document as a 
> reference for folks who feel compelled to do content sniffing but who 
> don't want to invest the year and a half of research that my colleagues 
> and I invested to arrive at this algorithm.

I think all user agents should be required to behave the same, so that 
users and content creators can interchange them at will, leading to a 
healthier competitive landscape. To this end, I think the spec should be 
normative with no optional behaviour.

I'm ok with allowing the sniffing to be skipped altogether, since that is 
equivalent to not implementing the spec. However, I'm not especially happy 
about allowing that, and I certainly do not think we should water down the 
requirements any more than that, _especially_ not on purely theoretical 
grounds.

> trying to find a Newtonian explanation of Brownian motion.

Indeed.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 25 Jan 2010 14:16:43 +0100

Adam Barth wrote:
 > ...

Sounds good to me.

In general, defining certain behaviors normatively is a good thing, but 
it doesn't need to be all-or-nothing. Also, throwing around MUSTs 
frequently doesn't help in practice; what's more important is to 
*convince* the reader that doing something is the right thing. More inline.

For instance, as Larry mentioned, (1) missing Content-Type, (2) 
mislabeled Content-Type and (3) invalid Content-Type (multiple headers) 
are distinct cases with different underlying reasons, so requiring the 
same behavior may not always be the best approach. So having different 
sections define this, and allowing people to pick the right pieces might 
be a good idea here.

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Tue, 26 Jan 2010 20:45:14 +0000

On Mon, Jan 25, 2010 at 1:16 PM, Julian Reschke <[email-masked]> wrote:
> In general, defining certain behaviors normatively is a good thing, but it
> doesn't need to be all-or-nothing. Also, throwing around MUSTs frequently
> doesn't help in practice; what's more important is to *convince* the reader
> that doing something is the right thing. More inline.

Believe me, I'm doing a lot to convince the implementers directly.  In
fact, I'm going as far as sending them patches to align their behavior
with the spec.

> For instance, as Larry mentioned, (1) missing Content-Type, (2) mislabeled
> Content-Type and (3) invalid Content-Type (multiple headers) are distinct
> cases with different underlying reasons, so requiring the same behavior may
> not always be the best approach. So having different sections define this,
> and allowing people to pick the right pieces might be a good idea here.

As I wrote to Larry, having more permutations is antithetical to our
security and compatibility goals.

Adam

---

From: Adam Barth <[email-masked]>
Date: Tue, 26 Jan 2010 20:46:52 +0000

>> I suggest including this example in the security considerations section

Actually, I've just included the example inline in the text because
the document doesn't have a separate security considerations section.
In some sense, the entire document is one giant security
considerations section.

>> Yes, I think this belongs in the "security considerations" section

This is explained in the introduction.

Adam

---

From: Adam Barth <[email-masked]>
Date: Tue, 26 Jan 2010 20:42:18 +0000

> there is a user isn't really relevant.

What other roles did you have in mind?  I think the primary consumer
of this specification with be web browsers, which commonly consider
themselves to be user agents.

> intended to be normative.  I'll assume

I've made this clearer by using SHOULD in all caps.

> The algorithm isn't exact (more below).

I've removed the word exactly,

> should opt into sniffing on a situation by situation,

I disagree.  Life would be more predictable if there were fewer
sniffing options, not more.  The more predictable life is, the more
secure it is.

> as the implementor needs.

I agree that sniffing should be reserved for those cases where the
implementor believes it is necessary.  However, once the implementor
has decided its necessary, they should go "whole hog" and implement
the algorithm defined in the spec.  If everyone got to pick and choose
what heuristics to use, then we'd be in much the same mess we're in
now.

> updating as the deployed content evolves.

I don't think it does.

I've removed this word from the draft.

This would make the algorithm less predictable.  The goal is to make
life more predictable.

This would make the algorithm less predictable.  The goal is to make
life more predictable.  Either you want to sniff or you don't.  Cherry
picking what to sniff leads to even more complexity.

That's not the way sniffing algorithms work.  We might wish they
worked some other way, but that's life.

I've removed this section.  I suspect it should have remained in HTML5
instead of being moved into this document.

I've removed the word "often" in favor of "in some cases" which is
factually indisputable.

> What is the security threat?

I've explained this in a previous reply.

> to be some attempt to define this for RSS?

I've replaced the word "most" with "some" which is factually indisputable.

> determines the method, or both?

That's a question for folks more pedantic than me.  I have no idea,
but I suspect it's the protocol.

> to determine content-type?

The FTP protocol does not transmit the type information to the user agent.

I've already replied to these comments.

Sadly, it admits the reality of what will happen.  For example, no one
knows what will happen with audio and video content on the web now
that HTML supports these media types natively.  To pretend like we can
control user agents here is fantasy.  I've made the requirements here
tighter.

> is disruptive and harmful. Don't do it.

I've changed "if desired" to "if necessary".  If user agents feel that
sniffing video formats is necessary, it doesn't matter what we write
in the spec.  They'll still do it.

Adam