Subject: Re: Content Sniffing impact on HTTPbis - #155

From: Adam Barth <[email-masked]>
Date: Tue, 26 Jan 2010 20:18:08 +0000

On Fri, Jun 5, 2009 at 9:50 PM, Bjoern Hoehrmann <[email-masked]> wrote:
> * Adam Barth wrote:
>>For better or for worse, this is the way browsers work.  I believe
>>this is related to historical behaviors of Apache and other HTTP
>>servers.  Removing these rules cause binary spew to fill up the
>>content area.
>
> What the draft should say why you have these and not others, in
> particular, not canonically equivalent values. Otherwise one might
> naively implement it like
>
>  if (response.content_type == 'text/plain')
>       if (response.charset == 'ISO-8859-1')
>       or (response.charset == 'iso-8859-1')
>       or (response.charset == 'utf-8')
>            ...
>
> Which would result in different behavior. Clearly pointing out why
> you have these and no others would make implementers think twice.

The spec text is clear that you should apply those steps only when the
octet sequence matches exactly what's in the table.  I've emphasized
that further in the new revision.

In general, I don't think it makes sense to explain the reasoning
behind every detail of the algorithm in the spec.  Implementations
should just do what the spec says.  The introduction alludes to the
scope of information considered in creating the spec.  If we explained
the reasoning behind every detail, the spec would easily be over 100
pages long.

> This is my point with respect to the other sections aswell. I'll
> just go through some examples:
>
>>[If /official type/ is an image type supported by the user agent]
>>
>>Would you prefer this step applied to all media types that begin with
>>"image/"?  That might let us remove step 6 as well.
>
> I just see that if I make a PNG image and deliver it as image/
> vnd.adobe.photoshop, then browsers with PSD support will treat it
> as PNG, and browsers without PSD support will treat it as PSD. I
> find that obscure and would like the reasoning documented in the
> draft.

That's correct.  The reason has to do with the historical behavior of
user agent, especially Internet Explorer and Firefox.  I don't think
it will add clarify to the specification to include the reasoning for
this detail.

>>> why step 3 only applies when you have at least three
>>> bytes and then only compares two bytes,
>>
>>We could change this to be slightly tighter, but it's a bit pedantic.
>
> When I read the text I suspect there is an error in the specifi-
> cation, and would then implement what I think my application ought
> to do; but your goal is that people implement it exactly as written.

If you're going to second-guess the requirements in the document, then
there's not much I can do to help you.  In this case, the impact of
altering this detail in miniscule.

>>> why the UTF-32 BOM is not being detected,
>>
>>We measured and determined that it was not needed for compatibility.
>>In general, we tried to minimize the amount of sniffing.
>
> As I read the draft, UTF-32LE encoded text/plain documents will be
> sniffed as text/plain because they have a UTF-16LE BOM; UTF-32BE
> encoded text/plain documents will be sniffed as application/octet-
> stream. This is inconsistent and confusing (there is suddenly some
> doubt whether you treat the document as UTF-16 or UTF-32, and while
> browsers might not support UTF-32, other applications will).

Looking for consistency in the sniffing algorithm is a fools errand.
The behavior described in the spec is the way life is.

>>> in section 6 the special handling of image/svg+xml;
>>
>>This is to above mistakenly changing the type of an image/svg+xml
>>resource that happens to begin with a magic number, e.g., BM.  Again,
>>this is a place where we are able to minimize the amount of sniffing.
>
> As I read the draft, the "official type" cannot be the SVG type at
> that point.

You are correct that the official-type cannot be image/svg+xml if you
enter the algorithm from the main entry point.  However, the
specification is designed to have it's various subsections invoked
independently by other specs.  In this case, the correction for
image/svg+xml in the Images section is correct.

> Again, the point is that people less familiar with the subject, and
> perhaps specifications in general, have access to this information;
> posting it here does not help them.

Indeed.  The subject of content sniffing is quite complicated.  The
draft says as much.

Adam

---

From: Adam Barth <[email-masked]>
Date: Tue, 26 Jan 2010 20:22:42 +0000

On Fri, Jun 12, 2009 at 10:03 PM, Ian Hickson <[email-masked]> wrote:
> Adam: assuming you have a conformance section which invokes RFC2119 and
> includes a sentence such as "Requirements phrased in the imperative as
> part of algorithms (such as "strip any leading space characters" or
> "return false and abort these steps") are to be interpreted with the
> meaning of the key word ("must", "should", "may", etc) used in introducing
> the algorithm.", and assuming you keep the "must"s in the invokations of
> the algorithms, I agree that it makes sense to remove the "must"s from the
> steps.

Done (using the text you gave me for the cookie spec more recently).

>> Separately, as an editorial comment, as listed directly above, I'd like
>> to see a big s/resource/resource representation/g (or just
>> s/resource/representation/g as the resource is what is identified by the
>> URI, not the bag-o-bits returned in an HTTP response.  I have some other
>> editorial comments too, but those will have to wait until I have time to
>> write them down.
>
> A resource is a bag of bits. I would object to this change.

I've removed all mention of resource.  The algorithm now operates
directly on the octets, however they are obtained.

Adam

---

From: Adam Barth <[email-masked]>
Date: Tue, 26 Jan 2010 20:24:19 +0000

On Sun, Jun 14, 2009 at 1:18 AM, Adrien de Croy <[email-masked]> wrote:
> OK..This does raise the question of what does a server do.
>
> I don't know of any servers (in my relatively limited knowledge of servers)
> that doesn't just use the file extension to choose content type when
> serving.
> If the served type is supposed to be the authoritative reference, you get
> problems when a server has a smaller database of extension-to-type than the
> client.
>
> Otherwise you'd need to re-engineer servers so that content providers can
> manually specify content type when providing content.  That then means
> authors need to know about registered types etc.
>
> So I can't see this changing any time soon, no matter what we write into
> specs.

Servers are free to select the Content-Type header they send in any
way they choose.  The requirements w.r.t. not using the file extension
over HTTP are for user agents.

Adam

---

From: Adam Barth <[email-masked]>
Date: Tue, 26 Jan 2010 20:53:24 +0000

On Thu, Jun 18, 2009 at 6:32 PM, Henrik Nordstrom
<[email-masked]> wrote:
> tor 2009-06-18 klockan 09:13 -0700 skrev Adam Barth:
>
>> The intent is after transfer and content encodings are removed.
>> What's the correct way to refer to those bytes?
>
> There is no unique defined HTTP term for this, so "entity-body with
> transfer and content encodings removed".

I've side-stepped this issue by having the algorithm operate directly
on sequences of octets.  Whatever specs decide to require use of the
algorithm will need to figure out where to get the octets from.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 28 Jan 2010 14:36:17 +1100

> Done (using the text you gave me for the cookie spec more recently).

Adam, it might be worth having a quiet word with a few people at the IETF about whether or not this style of specification is going to cause problems down the road. Lisa and Alexey would be a good start, and Julian would surely have an informed opinion.

> directly on the octets, however they are obtained.

Very diplomatic :)

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 28 Jan 2010 15:04:26 +0900

On 2010/01/28 12:36, Mark Nottingham wrote:
>> On Fri, Jun 12, 2009 at 10:03 PM, Ian Hickson<[email-masked]>  wrote:

Even without the explicit sentence ("Requirements phrased in the 
imperative..."), something like the above definitely has been done in 
the past. The example I'm most familiar with (for obvious reasons) is
"Applications MUST map IRIs to URIs by using the following two steps."
(see http://tools.ietf.org/html/rfc3987#section-3.1), but I'm sure you 
can find quite a few more in various RFCs.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]