Subject: Content Sniffing impact on HTTPbis - #155

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Jun 2009 01:29:48 +1000

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/155>

The text in question is in p3 section 3.2.1:
> If and only if the media type is not given by a Content-Type field,  
> the recipient MAY attempt to guess the media type via inspection of  
> its content and/or the name extension(s) of the URI used to identify  
> the resource. If the media type remains unknown, the recipient  
> SHOULD treat it as type "application/octet-stream".
>
Two possible approaches AFAICT;

1) remove the text "If and only if the media type is not given by a  
Content-Type field", leaving the specification of the sniffing  
algorithm to a separate document (possibly with some further  
constraints to discourage sniffing unless it's controlled, but this  
would be necessarily vague), or

2) specify the algorithm to use if sniffing is done by referring to  
draft-abarth-mime-sniff directly.

Thoughts? Other suggestions?

--
Mark Nottingham     http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 01 Jun 2009 21:20:23 +0200

Mark Nottingham wrote:
>> the resource. If the media type remains unknown, the recipient SHOULD 
>> treat it as type "application/octet-stream".
>>
> Content-Type field", leaving the specification of the sniffing algorithm 
> to a separate document (possibly with some further constraints to 
> discourage sniffing unless it's controlled, but this would be 
> necessarily vague), or
> 

2) is problematic, as a normative downref to draft-abarth-mime-sniff 
would be a downref; I don't think this is acceptable even if it's for an 
optional feature.

1) sounds good to me.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 1 Jun 2009 14:22:41 -0500

> > SHOULD treat it as type "application/octet-stream".

"If and only if...MAY..." is not the same as "MUST NOT ... if not". That is,
the above statement doesn't forbid an implementation from doing content
sniffing since it isn't a MUST NOT requirement.

> would be necessarily vague), or

+1. How applications interpret content is not really relevant as far as the
protocol is concerned. The HTTP specification should make it clear what
Content-Type represents and let applications decide whether (and how) to
make use of it.

- Brian

---

From: Mark Baker <[email-masked]>
Date: Mon, 1 Jun 2009 15:30:35 -0400

On Mon, Jun 1, 2009 at 11:29 AM, Mark Nottingham <[email-masked]> wrote:
>> If and only if the media type is not given by a Content-Type field, the
>> recipient MAY attempt to guess the media type via inspection of its content
>> and/or the name extension(s) of the URI used to identify the resource. If
>> the media type remains unknown, the recipient SHOULD treat it as type
>> "application/octet-stream".
>>
> Content-Type field", leaving the specification of the sniffing algorithm to
> a separate document (possibly with some further constraints to discourage
> sniffing unless it's controlled, but this would be necessarily vague), or

Yes, I proposed this before;

http://lists.w3.org/Archives/Public/ietf-http-wg/2009AprJun/0018.html

> draft-abarth-mime-sniff directly.

I don't think that has any place in the HTTP spec, especially as even
the revised version continues to use prescriptive language that
supercedes key parts of the HTTP protocol.

Mark.

---

From: Adam Barth <[email-masked]>
Date: Mon, 1 Jun 2009 13:58:10 -0700

> supercedes key parts of the HTTP protocol.

I'd like to incorporate your feedback into the draft, but the above is
too vague for me to act upon.  Are there specific statements you'd
like me to revise?

Adam

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 01 Jun 2009 23:44:05 +0200

>too vague for me to act upon.  Are there specific statements you'd

A good start would be including detailed rationale for the parts that
contradict other specifications or are otherwise obscure. I would make
suggestions, but I have a hard time following the document at all; to
pick one problem, section two defines an algorithm, but I could not
find how and where that algorithm is supposed to be relevant.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 02 Jun 2009 04:23:59 +0200

><http://trac.tools.ietf.org/wg/httpbis/trac/ticket/155>

I believe you misunderstand the organization of the specification. What
RFC 2616 says is that the media type of the entity body of a message is
defined by the Content-Type header; if there is no Content-Type header,
then the media type is defined by the implementation; if the implemen-
tation fails to define a media type, assume application/octet-stream.

If you want to say that the media type of the entity body is defined by
the implementation and not the Content-Type header, then you have to
change the definition of the Content-Type header. The phrase you pro-
pose to remove is there to avoid conflicting definitions of what the
media type of the entity body is.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Jun 2009 13:06:48 +1000

Are you referring to this:

> The entity-header field "Content-Type" indicates the media type of  
> the entity-body sent to the recipient or, in the case of the HEAD  
> method, the media type that would have been sent had the request  
> been a GET.

in p3 5.9?

If so, I think you're reading too much into 'indicates', and IIRC this  
discussion has been had on list before. The question is not what  
defines the media type, it's what an application can or cannot do with  
that information once the indication is available.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 02 Jun 2009 14:00:35 +0200

>that information once the indication is available.

I believe RFC 2616 uses the word "indicate" to mean "define", "specify";
the part you quoted for example is immediately preceded by "an entity-
body SHOULD include a Content-Type header field defining the media type
of that body." That would not be possible if the Content-Type header, if
present, did not define the media type.

It seems to me that all the original text said is that the HTTP layer
must report the media type as specified in the Content-Type header to
the next layer. What that layer does with the information is out of
scope of the HTTP specification. In that I agree with you on what the
issue really is.

A viable alternative to your proposals might, then, be to replace the
whole paragraph in question by (removing any "sniffing" discussion):

   Any HTTP/1.1 message containing an entity-body SHOULD include a
   Content-Type header field defining the media type of that body.
   How the media type of the entity-body affects the behavior of
   higher-level applications is out of scope of this specification.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Baker <[email-masked]>
Date: Tue, 2 Jun 2009 08:41:15 -0400

> too vague for me to act upon.  Are there specific statements you'd

Here they are;

"The /sniffed type/ of a resource MUST be found as follows:"

to;

"The /sniffed type/ of a resource is found as follows:"

This should be removed;

"File extensions MUST NOT be used for determining resource types for
resources fetched over HTTP."

As it's clear from the algorithm that file extensions are not
referenced, and it's HTTP's job to say what meaning is or isn't given
to them (which, of course, is none).

Also, this introductory claim should be removed;

"However, if the user agent does emply[sic] a
content sniffing algorithm, it is imperative that the algorithm in
this document be followed exactly"

It should suffice that the draft explains the drawbacks of not
following the algorithm because a) the algorithm is still evolving
(e.g. text/plain), b) if an implementor has a good reason to implement
a different algorithm, they will, and c) the document should describe
the algorithm, and any conformance requirements about its use should
be left to the referrer (e.g. if the HTTP spec refers to it).

Just for completeness' sake (removing the MUSTs), this;

"the user agent MUST stop this
algorithm, and assume that the /sniffed type/ of the resource is
"text/html"."

could be changed to "the algorithm stops and the /sniffed type/ of the
resource representation is "text/html"

Separately, as an editorial comment, as listed directly above, I'd
like to see a big s/resource/resource representation/g (or just
s/resource/representation/g as the resource is what is identified by
the URI, not the bag-o-bits returned in an HTTP response.  I have some
other editorial comments too, but those will have to wait until I have
time to write them down.

Mark.

---

From: Adam Barth <[email-masked]>
Date: Tue, 2 Jun 2009 10:04:58 -0700

Thanks the below is very helpful.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 3 Jun 2009 11:30:24 +1000

Wors for me, although I don't know that the last sentence is really  
necessary. Are you explicitly removing application/octet-stream as a  
default if no other type is found or derived?

Others?

---

From: Mark Baker <[email-masked]>
Date: Tue, 2 Jun 2009 21:40:12 -0400

On Tue, Jun 2, 2009 at 9:30 PM, Mark Nottingham <[email-masked]> wrote:
> necessary. Are you explicitly removing application/octet-stream as a default
> if no other type is found or derived?
>
> Others?

I don't know what that last sentence means, in particular
"higher-level applications".

Mark.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Jun 2009 16:01:41 +1000

Revised proposal:

> SHOULD treat it as type "application/octet-stream".
with

"""
If the Content-Type field is not present in a message with a body, the  
recipient SHOULD assume that the message was sent with a Content-Type  
of "application/octet-stream".

Note that neither the interpretation of the data type of a message nor  
the behaviours caused by it are not defined by this specification;  
this potentially includes examination of the content to override the  
indicated type ("sniffing").
"""

---

From: Mark Baker <[email-masked]>
Date: Thu, 4 Jun 2009 10:18:45 -0400

+1

On Thu, Jun 4, 2009 at 2:01 AM, Mark Nottingham <[email-masked]> wrote:
> recipient SHOULD assume that the message was sent with a Content-Type of
> Note that neither the interpretation of the data type of a message nor the
> behaviours caused by it are not defined by this specification; this
> potentially includes examination of the content to override the indicated
> type ("sniffing").
> """

Mark.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Jun 2009 22:41:59 +0200

Mark Nottingham wrote:
> the behaviours caused by it are not defined by this specification; this 
> potentially includes examination of the content to override the 
> ...

The second part sounds good.

The first part, now that sniffing isn't mentioned there anymore, makes 
"application/octet-stream" a default at SHOULD level.

The definition of "application/octet-stream" in 
<http://greenbytes.de/tech/webdav/rfc2046.html#rfc.section.4.5.1> has:

"The recommended action for an implementation that receives an 
"application/octet-stream" entity is to simply offer to put the data in 
a file, with any Content-Transfer-Encoding undone, or perhaps to use it 
as input to a user-specified process."

It's not really the intent to make *that* the default, right?

BR, Julian

---

From: Adam Barth <[email-masked]>
Date: Thu, 4 Jun 2009 13:52:55 -0700

2009/6/3 Mark Nottingham <[email-masked]>:
> """
> If the Content-Type field is not present in a message with a body, the recipient SHOULD assume that the message was sent with a Content-Type of "application/octet-stream".
>
> Note that neither the interpretation of the data type of a message nor the behaviours caused by it are not defined by this specification; this potentially includes examination of the content to override the indicated type ("sniffing").
> """

That seems fine.

Thanks,
Adam

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Thu, 04 Jun 2009 16:43:30 -0500

>> indicated type ("sniffing").

The first part 'should' read 'MUST', as Julian mentions below, the choice
is in interpretation, not the value of the Content-Type header;

> "application/octet-stream" a default at SHOULD level.

Right, this should be default at MUST level; the behavior is entirely
incumbent upon the user agent's choice of action, just as it is for any
arbitrary content-type.

> It's not really the intent to make *that* the default, right?

It has been; clearly documented since '96.  This *surprises* you why?

"recommended action" is orthogonal to the UA's choice of action, right?
Clearly, this is not the "MUST", but the existing "SHOULD", to be
superseded by future recommendations for said content type.

The definition clearly provides the latitude to handle octet-stream by
the users' choice of actions.  Oh, wait, bother... existing user agents
give the end user no choice.  In any case I'm certain your draft will
remedy this.

---

From: Adam Barth <[email-masked]>
Date: Thu, 4 Jun 2009 15:40:27 -0700

On Thu, Jun 4, 2009 at 2:43 PM, William A. Rowe, Jr.
> is in interpretation, not the value of the Content-Type header;

This isn't workable.  The content sniffing algorithm needs to
distinguish between an absent Content-Type header and a Content-Type
header with the value "application/octet-stream".  Making this a MUST
requirement forces the algorithm to treat them the same.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 5 Jun 2009 10:10:01 +1000

Given that, is the proposal acceptable as worded?

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 01:16:13 -0700

On Mon, Jun 1, 2009 at 2:44 PM, Bjoern Hoehrmann <[email-masked]> wrote:
> contradict other specifications or are otherwise obscure.

For which parts would you like a more detailed rationale?  It's hard
for me to guess which parts you think are obscure.

> I would make
> suggestions, but I have a hard time following the document at all;

How can I make the document clearer?

> to

The document defines algorithms for extracting information from the
Content-Type.  That algorithm, in particular, extracts the charset
attribute from the Content-Type header.  The algorithm is intended to
be reference by other specifications, such as HTML 5, which need to
determine the charset attribute of the Content-Type header is a manner
compatible with existing web content.

Is there some way I could change the I-D to make this more clear?

I'm not trying to be obtuse.  Your feedback is just not very actionable.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Fri, 05 Jun 2009 10:26:03 +0200

>> requirement forces the algorithm to treat them the same.

No, I think even "SHOULD" is too strong.

We want servers to provide Content-Type headers. But we also don't want 
them to send *incorrect* headers. That is, if the server doesn't know, 
or can't guess with any reliability, the best thing to do is *not* to 
send the Content-Type header.

This, btw, is now at least *possible* with Apache httpd.

If we make the absence of Content-Type equivalent to sending 
Content-Type: application/octet-stream, *and* recipients treat 
application/octet-stream is recommended by 
<http://greenbytes.de/tech/webdav/rfc2046.html#rfc.section.4.5.1>, then 
sending no Content-Type becomes less plausible.

So, I think one way to fix this is to drop that sentence, making the 
paragraph just:

"Note that neither the interpretation of the data type of a message nor 
the behaviours caused by it are not defined by this specification; this 
potentially includes examination of the content to override the 
indicated type ("sniffing")."

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 5 Jun 2009 10:29:49 +0200

On Jun 4, 2009, at 8:01 AM, Mark Nottingham wrote:

>> If and only if the media type is not given by a Content-Type  
>> field, the recipient MAY attempt to guess the media type via  
>> inspection of its content and/or the name extension(s) of the URI  
>> used to identify the resource. If the media type remains unknown,  
>> the recipient SHOULD treat it as type "application/octet-stream".
> with
>
> """
> If the Content-Type field is not present in a message with a body,  
> the recipient SHOULD assume that the message was sent with a  
> Content-Type of "application/octet-stream".
>
> Note that neither the interpretation of the data type of a message  
> nor the behaviours caused by it are not defined by this  
> specification; this potentially includes examination of the content  
> to override the indicated type ("sniffing").
> """

I think that conflicts with my analysis in the mime-respect TAG finding.
I would prefer that no Content-Type means that the server doesn't know
the media type, thereby allowing the recipient to guess.

....Roy

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 01:30:17 -0700

> "The /sniffed type/ of a resource is found as follows:"

Fixed.

> to them (which, of course, is none).

I've changed this to a note explaining why using file extension to
determine the media type of HTTP resources is a bad idea.

> be left to the referrer (e.g. if the HTTP spec refers to it).

I've softened this a bit and made it just a lead in to the explanation
of why using a divergent algorithm is a bad idea.  If you see a better
way to bridge the first sentence (UAs shouldn't sniff) to the example
(using a different algorithm causes problems), please let me know.

> resource representation is "text/html"

Fixed.

> other editorial comments too, but those will have to wait until I have

Ugg.  That's super wordy.  Is there really not a better way to refer
to the thing contained in an HTTP response?  Would entity-body be
appropriate?

Adam

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 01:34:23 -0700

> the media type, thereby allowing the recipient to guess.

It seems the point of this thread is to allow (but discourage) the
recipient to guess in all cases.  Perhaps you mean that we ought not
to discourage guessing when the Content-Type header is absent.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 5 Jun 2009 18:50:26 +1000

Well, the reasoning behind saying something about application/octet- 
stream is that senders may rely on 2616's defaulting to that media  
type -- thereby preserving backwards-compatibility.

If we think that there's an interop problem here and it would serve  
better to not define a default, I'm fine with that.

It may be worth putting a note in to the effect that there used to be  
a default, though.

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Fri, 05 Jun 2009 09:10:27 -0500

> sending no Content-Type becomes less plausible.

The question is, what is the default metacontent transmitted when there is
no other information?  As the majority of content is unpadded, organized
into eight bit octets with zero padding, etc, this is not a bad assumption.
Although I suspect that a new value of application/unknown-stream would be
appropriate.

Server misrepresentation of Content-Type will cease, once browsers stop
misrepresenting the content type.  Until web authors and administrators
(including mass vhosters) become aware that they have misrepresented the
data they are serving, they will continue to generate the 3% (IIRC) of
mislabeled content.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 05 Jun 2009 18:14:46 +0200

>for me to guess which parts you think are obscure.

I've already mentioned the encoding extraction algorithm, but to add
some others: in draft-abarth-mime-sniff-01 section 3 step 3's special
handling of very particular sequences, the handling of unregistered
and malformed values in step 5, the special handling of XML types in
step 6, the relevance of the implementation supporting particular types
in step 7.

In section 4 why implementations may decide to pick any number of bytes
between 0 and 512, why step 3 only applies when you have at least three
bytes and then only compares two bytes, why the UTF-32 BOM is not being
detected, why step four has those bytes and not others; in section 6 the
special handling of image/svg+xml; in section 7 why the UTF-16 BOM is
ignored.

>compatible with existing web content.

I see no justification for having a special algorithm for the charset
parameter; you extract the parameter just like any other. I also don't
know of any implementation that processes the header value like that;
if you have

  text/plain;whatever="charset=iso-8859-2";charset=iso-8859-3

Then the result of your algorithm is iso-8859-2", while the correct be-
havior yields iso-8859-3, which is also what IE6, FF 3.x, Opera 9, and
various non-browser applications use. The same goes for a simpler:

  text/plain;whatever="charset";charset=iso-8859-3

Where your algorithm returns nothing, and implementations implement the
correct behavior, which yields iso-8859-3. There also appears to be no
need to process escape sequences within quoted strings incorrectly, for
instance Opera 9 seems to implement that properly, so does my own code.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Hickson <[email-masked]>
Date: Fri, 5 Jun 2009 19:05:49 +0000 (UTC)

On Fri, 5 Jun 2009, Bjoern Hoehrmann wrote:
> know of any implementation that processes the header value like that; if 
> you have
> 

My testing at the time was written disagrees with the results of your 
testing. I believe this was primarily intended for charset extraction for 
<script> nodes, if that matters. However, if your results can be confirmed 
then that would certainly be good news.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 12:12:50 -0700

On Fri, Jun 5, 2009 at 7:10 AM, William A. Rowe, Jr.
> data they are serving, they will continue to generate the 3% (IIRC) of

This is unlikely to ever occur given the market dynamics of browsers.

Adam

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Fri, 05 Jun 2009 14:28:15 -0500

> This is unlikely to ever occur given the market dynamics of browsers.

Which is what makes the entire discussion so entirely laughable.

I concur with Roy and Julian, leave Content-Type null with an undefined
content type.  while user agents persist in nonsense (such as decoding
UTF-7 when presented an explicit charset), this is out of our server-side
and authors' hands, and no spec is going to correct their misbehavior or
unsafe practices.

For some tiny minority who care, the very absence of Content-Type conveys
new, useful metadata that should not break 2616 definitions.

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 12:40:13 -0700

On Fri, Jun 5, 2009 at 12:28 PM, William A. Rowe, Jr.
> new, useful metadata that should not break 2616 definitions.

I don't share your pessimistic view, but that resolution is workable
from my perspective.

Thanks,
Adam

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 12:55:56 -0700

On Fri, Jun 5, 2009 at 9:14 AM, Bjoern Hoehrmann <[email-masked]> wrote:
> handling of very particular sequences,

For better or for worse, this is the way browsers work.  I believe
this is related to historical behaviors of Apache and other HTTP
servers.  Removing these rules cause binary spew to fill up the
content area.

> the handling of unregistered and malformed values in step 5,

HTTP responses commonly contain these values and depend on user agents
sniffing the actual media type.  Removing these values causes
compatibility problems.

> the special handling of XML types in step 6,

I believe this is to avoid step 7 applying to SVG images.

> the relevance of the implementation supporting particular types
> in step 7.

Would you prefer this step applied to all media types that begin with
"image/"?  That might let us remove step 6 as well.

> between 0 and 512,

This is to avoid breaking sites that use comet
<http://en.wikipedia.org/wiki/Comet_(programming)>.

> why step 3 only applies when you have at least three
> bytes and then only compares two bytes,

We could change this to be slightly tighter, but it's a bit pedantic.

> why the UTF-32 BOM is not being detected,

We measured and determined that it was not needed for compatibility.
In general, we tried to minimize the amount of sniffing.

> why step four has those bytes and not others;

That's just how browsers work.  This is a point at which there is
broad interoperability already.  The costs of convincing
implemenations to change this table outweigh the benefits.

> in section 6 the special handling of image/svg+xml;

This is to above mistakenly changing the type of an image/svg+xml
resource that happens to begin with a magic number, e.g., BM.  Again,
this is a place where we are able to minimize the amount of sniffing.

> in section 7 why the UTF-16 BOM is ignored.

The algorithm doesn't work for UTF-16 anyway.  What would be the point
of skipping over the UTF-16 BOM?  Again, this is a place we've
minimized the amount of sniffing.

> parameter;

I've added a TODO to investigate whether this algorithm is still needed.

Adam

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 05 Jun 2009 23:50:35 +0200

>servers.  Removing these rules cause binary spew to fill up the

What the draft should say why you have these and not others, in
particular, not canonically equivalent values. Otherwise one might
naively implement it like

  if (response.content_type == 'text/plain') 
       if (response.charset == 'ISO-8859-1')
       or (response.charset == 'iso-8859-1')
       or (response.charset == 'utf-8')
            ...

Which would result in different behavior. Clearly pointing out why
you have these and no others would make implementers think twice.
This is my point with respect to the other sections aswell. I'll
just go through some examples:

>[If /official type/ is an image type supported by the user agent]
>

I just see that if I make a PNG image and deliver it as image/
vnd.adobe.photoshop, then browsers with PSD support will treat it
as PNG, and browsers without PSD support will treat it as PSD. I
find that obscure and would like the reasoning documented in the
draft.

>We could change this to be slightly tighter, but it's a bit pedantic.

When I read the text I suspect there is an error in the specifi-
cation, and would then implement what I think my application ought
to do; but your goal is that people implement it exactly as written.

>In general, we tried to minimize the amount of sniffing.

As I read the draft, UTF-32LE encoded text/plain documents will be
sniffed as text/plain because they have a UTF-16LE BOM; UTF-32BE
encoded text/plain documents will be sniffed as application/octet-
stream. This is inconsistent and confusing (there is suddenly some
doubt whether you treat the document as UTF-16 or UTF-32, and while
browsers might not support UTF-32, other applications will).

>this is a place where we are able to minimize the amount of sniffing.

As I read the draft, the "official type" cannot be the SVG type at
that point.

Again, the point is that people less familiar with the subject, and
perhaps specifications in general, have access to this information;
posting it here does not help them.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Hickson <[email-masked]>
Date: Fri, 5 Jun 2009 21:56:07 +0000 (UTC)

On Fri, 5 Jun 2009, Bjoern Hoehrmann wrote:
> >>
> >> why step 3 only applies when you have at least three bytes and then 
> >> only compares two bytes,
> >
> When I read the text I suspect there is an error in the specifi- cation, 
> and would then implement what I think my application ought to do; but 
> your goal is that people implement it exactly as written.

The original reason for this was that I did not want to sniff as a 
particular type a file that only contained a BOM, since it is more likely 
that this is an error and that the file is really some other encoding.

> >We measured and determined that it was not needed for compatibility. In 
> >general, we tried to minimize the amount of sniffing.
> 
> sniffed as text/plain because they have a UTF-16LE BOM; UTF-32BE encoded 
> text/plain documents will be sniffed as application/octet- stream. This 
> is inconsistent and confusing (there is suddenly some doubt whether you 
> treat the document as UTF-16 or UTF-32, and while browsers might not 
> support UTF-32, other applications will).

We're explicitly not supporting UTF-32. For more details see HTML5.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 06 Jun 2009 00:28:30 +0200

>that this is an error and that the file is really some other encoding.

I do not really share that assessment, but as it is written in the
draft, you end up "sniffing" text/plain either way.

>We're explicitly not supporting UTF-32. For more details see HTML5.

I fail to see the relevance. The draft is unclear and misleading with
respect to the handling of UTF-32 encoded text/plain documents. There
is nothing that "HTML5" could say as a remedy, at least not until the
draft references "HTML5" to that end.

The draft has a similar problem with the iso-8859-1 cases in 3.3: if
such documents start with what appears to be a BOM, then the BOM is
the reason for "sniffing" them as text/plain, casting doubt whether
you then also treat them as in some UTF encoding or not.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 6 Jun 2009 13:05:29 +1000

Sounds like we're moving towards consensus (because no one is  
particularly happy).

That would make the proposal to replace p3 3.2.1:

> When an entity-body is included with a message, the data type of that
> body is determined via the header fields Content-Type and Content-
> Encoding.  These define a two-layer, ordered encoding model:
>
>   entity-body := Content-Encoding( Content-Type( data ) )
>
> Content-Type specifies the media type of the underlying data.
> Content-Encoding may be used to indicate any additional content
> codings applied to the data, usually for the purpose of data
> compression, that are a property of the requested resource.  There is
> no default encoding.
>
> Content-Type header field defining the media type of that body.  If
> and only if the media type is not given by a Content-Type field, the
> recipient MAY attempt to guess the media type via inspection of its
> content and/or the name extension(s) of the URI used to identify the
> resource.  If the media type remains unknown, the recipient SHOULD

with:

"""
When an entity-body is included with a message, the data type of that
body is determined via the header fields Content-Type and Content-
Encoding.  These define a two-layer, ordered encoding model:

   entity-body := Content-Encoding( Content-Type( data ) )

Content-Type specifies the media type of the underlying data. Any
HTTP/1.1 message containing an entity-body SHOULD include a
Content-Type header field defining the media type of that body. If
the Content-Type header field is not present, it indicates that the
sender does not know the media type of the data; recipients MAY
either assume that it is "application/octet-stream" or examine the
content to determine its type.

Content-Encoding may be used to indicate any additional content
codings applied to the data, usually for the purpose of data
compression, that are a property of the requested resource.  There is
no default encoding.

Note that neither the interpretation of the data type of a message nor
the behaviours caused by it are not defined by HTTP; this
potentially includes examination of the content to override any
indicated type ("sniffing").
"""

Question -- the wording above explicitly allows sniffing of both  
content-type and content-encoding; do we want to allow C-E?

I'm not sure what "that are a property of the requested resource"  
means in the context of content-encoding; it seems to be a variation  
of the "requested variant" problem. Perhaps we can deal with that  
then...

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 23:43:54 -0700

> content-type and content-encoding; do we want to allow C-E?

I seem to recall that here are a bunch of servers that send
Content-Type: application/gzip when they mean Content-Encoding: gzip
(or is it vice-versa?).  I can look up the code if you'd like more
information.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 6 Jun 2009 17:10:15 +1000

Yes, thank you.

On 06/06/2009, at 4:53 PM, David Morris wrote:

>
>
> On Sat, 6 Jun 2009, Mark Nottingham wrote:
>
>
>> nor
> I think the 'not' in the 2nd line doesn't belong ... a double negative
> seems to exist. Or remove 'neither' and change 'nor' to 'and'.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 6 Jun 2009 09:16:24 +0200

> (or is it vice-versa?).  I can look up the code if you'd like more

No, they send app/gzip as the content-type when they do not want
the recipient to remove the encoding on-the-fly.  That is a
typical config for software distribution sites.

....Roy

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Sat, 06 Jun 2009 02:31:32 -0500

> Sounds like we're moving towards consensus (because no one is

/bows

/genuflects

you got it.  So rare that anyone does in that scenario :)

---

From: Julian Reschke <[email-masked]>
Date: Sat, 06 Jun 2009 11:58:27 +0200

> sender does not know the media type of the data; recipients MAY

Q: maybe we should relax the "SHOULD" a bit more; I think it should be 
totally clear that if a server doesn't know the type, and can't guess it 
with some confidence, it SHOULD NOT include it.

How about:

"Any HTTP/1.1 message containing an entity-body SHOULD include a 
Content-Type header field defining the media type of that body, unless 
that information is unknown."

> content-type and content-encoding; do we want to allow C-E?

Does it? It's not obvious to me. If we don't want that (and I think this 
is the case), we may want to swap the last two paragraphs.

> I'm not sure what "that are a property of the requested resource" means 
> in the context of content-encoding; it seems to be a variation of the 
> "requested variant" problem. Perhaps we can deal with that then...

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 06 Jun 2009 12:06:16 +0200

> Sounds like we're moving towards consensus (because no one is 

I have attached a proposed patch to ticket 155, containing Mark's 
proposal (plus the removal of the surplus "not", plus an explicit 
reference to the spec defining application/octet-stream): see 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/155/155.diff>.

The whole paragraph would then read:

-- snip --

3.2.1.  Type

    Content-Type specifies the media type of the underlying data.  Any
    HTTP/1.1 message containing an entity-body SHOULD include a Content-
    Type header field defining the media type of that body.  If the
    Content-Type header field is not present, it indicates that the
    sender does not know the media type of the data; recipients MAY
    either assume that it is "application/octet-stream" ([RFC2046],
    Section 4.5.1) or examine the content to determine its type.

    Note that neither the interpretation of the data type of a message
    nor the behaviors caused by it are defined by HTTP; this potentially
    includes examination of the content to override any indicated type
    ("sniffing").

-- snip --

Note that this does NOT contain any changes related to the feedback I 
sent in my previous mail.

BR, Julian (looking forward to get this issue from the table soon!)

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 6 Jun 2009 20:10:16 +1000

On 06/06/2009, at 7:58 PM, Julian Reschke wrote:

> Q: maybe we should relax the "SHOULD" a bit more; I think it should  
> be totally clear that if a server doesn't know the type, and can't  
> guess it with some confidence, it SHOULD NOT include it.
>
> How about:
>
> Content-Type header field defining the media type of that body,  
> unless that information is unknown."

I read that as effectively equivalent to the current text, but sure;  
it is more explicit about when the SHOULD can be violated.

> Does it? It's not obvious to me. If we don't want that (and I think  
> this is the case), we may want to swap the last two paragraphs.

That's one solution, yes, but I'm not yet sure we have a problem.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 06 Jun 2009 12:23:36 +0200

Mark Nottingham wrote:
> I read that as effectively equivalent to the current text, but sure; it 
> is more explicit about when the SHOULD can be violated.
> ...

I'm pretty sensitive to this kind of SHOULD requirements, because of all 
the crap I've seen servers putting into WebDAV metadata, because the 
implementors thought they had to (like providing content language info 
defaulting to the server's locale, or time stamps defaulting to Jan 01, 
1970).

BR, Julian

---

From: Adam Barth <[email-masked]>
Date: Sat, 6 Jun 2009 12:17:19 -0700

> typical config for software distribution sites.

So, I looked it up.  Apparently Apache (maybe an old version?) sends
*both* Content-Encoding: gzip and Content-Type: application/gzip for
files that end in ".gz".  To work around this bug, some popular user
agents ignore the Content-Encoding in this case.

I'm not suggesting we spec this behavior.  The point is more that
reality sometimes forces user agents to doctor the Content-Encoding.

Adam

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 6 Jun 2009 22:22:39 +0200

> agents ignore the Content-Encoding in this case.

I don't know of any version of Apache that would do that, but if you
see a specific version in the wild let me know (third-party distros  
often
introduce bugs that we don't know about).  There are literally dozens of
ways to set that metadata in Apache, only a few of which involve file
extensions, so it is certainly possible to configure Apache to send  
both.

> reality sometimes forces user agents to doctor the Content-Encoding.

It is also possible to gzip a gzipped document, so reality should be
asking the browser to treat it as such.  In any case, the normal
behavior is to leave the encoding as is unless the media type is
one of those rendered by the browser, so it is unlikely we'd see
a difference in behavior even if the config is weird.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 7 Jun 2009 11:01:51 +1000

OK. Please regenerate the diff with this change. It doesn't sound like  
people feel we need to close off the possibility of sniffing encoding,  
so I think the rest can stay the same...

On 06/06/2009, at 8:23 PM, Julian Reschke wrote:

>> I read that as effectively equivalent to the current text, but  
>> sure; it is more explicit about when the SHOULD can be violated.
>> ...
>
> I'm pretty sensitive to this kind of SHOULD requirements, because of  
> all the crap I've seen servers putting into WebDAV metadata, because  
> the implementors thought they had to (like providing content  
> language info defaulting to the server's locale, or time stamps  
> defaulting to Jan 01, 1970).
>
> BR, Julian
>

---

From: David Morris <[email-masked]>
Date: Sat, 6 Jun 2009 18:34:18 -0700 (PDT)

On Sun, 7 Jun 2009, Mark Nottingham wrote:

> OK. Please regenerate the diff with this change. It doesn't sound like people 
> feel we need to close off the possibility of sniffing encoding, so I think 
> the rest can stay the same...

I don't see the point in outlawing common behavior whether that would be 
sniffing content or anything else.

Dave Morris

---

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Jun 2009 10:58:30 +0200

> so I think the rest can stay the same...

Will do later today!

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Jun 2009 16:01:06 +0200

> so I think the rest can stay the same...

OK, new proposed patch: 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/155/155.2.diff>

Full text of 3.2.1:

--snip --

3.2.1.  Type

    Content-Type specifies the media type of the underlying data.  Any
    HTTP/1.1 message containing an entity-body SHOULD include a Content-
    Type header field defining the media type of that body, unless that
    information is unknown.  If the Content-Type header field is not
    present, it indicates that the sender does not know the media type of
    the data; recipients MAY either assume that it is "application/
    octet-stream" ([RFC2046], Section 4.5.1) or examine the content to
    determine its type.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 8 Jun 2009 08:26:57 +1000

Looks good to me. Any other objections?

On 08/06/2009, at 12:01 AM, Julian Reschke wrote:

> Mark Nottingham wrote:
>> OK. Please regenerate the diff with this change. It doesn't sound  
>> like people feel we need to close off the possibility of sniffing  
>> encoding, so I think the rest can stay the same...
>> ...
>
> OK, new proposed patch: <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/155/155.2.diff 
> >
>
> Full text of 3.2.1:
>
> --snip --
>
> 3.2.1.  Type
>

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Jun 2009 07:50:32 +0200

> Looks good to me. Any other objections?

I have applied the patch with 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/592>.

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 12 Jun 2009 14:26:22 +1000

Closing the issue.

On 08/06/2009, at 3:50 PM, Julian Reschke wrote:

> I have applied the patch with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/592 
> >.
>
> BR, Julian
>

---

From: Ian Hickson <[email-masked]>
Date: Fri, 12 Jun 2009 22:03:12 +0000 (UTC)

> "The /sniffed type/ of a resource is found as follows:"

I don't really agree with this (how do you conform to the spec if it 
doesn't have any conformance criteria?) but in case this happens, I've 
referenced the algorithm with a "must" from HTML5.

> referenced, and it's HTTP's job to say what meaning is or isn't given to 
> them (which, of course, is none).

I don't mind making this requirement non-normative (since as you say it's 
implicit), but I do think we should explicitly state that file extensions 
don't and mustn't have an effect, since it is so common to use them for 
this exact purpose in clients.

> be left to the referrer (e.g. if the HTTP spec refers to it).

I don't really understand what you mean here.

> resource representation is "text/html"

Adam: assuming you have a conformance section which invokes RFC2119 and 
includes a sentence such as "Requirements phrased in the imperative as 
part of algorithms (such as "strip any leading space characters" or 
"return false and abort these steps") are to be interpreted with the 
meaning of the key word ("must", "should", "may", etc) used in introducing 
the algorithm.", and assuming you keep the "must"s in the invokations of 
the algorithms, I agree that it makes sense to remove the "must"s from the 
steps.

> Separately, as an editorial comment, as listed directly above, I'd like 
> to see a big s/resource/resource representation/g (or just 
> s/resource/representation/g as the resource is what is identified by the 
> URI, not the bag-o-bits returned in an HTTP response.  I have some other 
> editorial comments too, but those will have to wait until I have time to 
> write them down.

A resource is a bag of bits. I would object to this change.

---

From: David Morris <[email-masked]>
Date: Fri, 12 Jun 2009 16:05:45 -0700 (PDT)

> this exact purpose in clients.

I find it absurd to disallow use of file extensions given that on most 
OSes, there is no other mechanism to annotate content type. And they are 
a common way web servers choose content/type values.

Legislating our result into irrelevance is the likely outcome of dictating 
against common practice without a better commonly available alternative.

David Morris

---

From: Adam Barth <[email-masked]>
Date: Fri, 12 Jun 2009 16:38:02 -0700

On Fri, Jun 12, 2009 at 4:05 PM, David Morris<[email-masked]> wrote:
> I find it absurd to disallow use of file extensions given that on most OSes,
> there is no other mechanism to annotate content type. And they are a common
> way web servers choose content/type values.

For better or worse, we can't use file extensions as part of the
content sniffing algorithm because it's insecure.  In many attack
scenarios, the attacker chooses the file extension.

> against common practice without a better commonly available alternative.

Neither Firefox nor Chrome uses the file extension in their sniffing
algorithm.  Safari uses the file extension only in one corner case.  I
don't think we legislating the algorithm to irrelevance with this
requirement.

Adam

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 13 Jun 2009 01:48:35 +0200

> sniffing since it isn't a MUST NOT requirement.

Correct, it's just a SHOULD NOT, which is the level it should be.

The text as-is tries to say that receiving agents SHOULD NOT attempt to
guess the media type of the message if there is an Content-Type
indication. This means that from the protocol perspective it's
recommended the media type indication in the protocol is what is used
for determining the media type of the enclosed entity. This does not
forbid agents from using other means for finding the media type as there
is no MUST level requirements, but clearly sets the tone on what the
HTTP protocol considers as authoritative media type indication for the
entity contained within the HTTP message.

Imho the correct resolution of #155 is to remove just the text "and only
if", softening the tone just slightly. Removing the whole condition on
when guessing/sniffing is allowed opens it up too much, downgrading the
whole text almost to the level of pure ignorance, giving a general MAY
level rule that sniffing/guessing of the media type is allowed under all
conditions which is not what the HTTP protocol intends.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 13 Jun 2009 02:06:16 +0200

>     octet-stream" ([RFC2046], Section 4.5.1) or examine the content to

Just a minor change to better reflect reality earlier HTTP specs:

instead of "examine the content to" simply say "heuristically" without
going into what means is used for determining the unknown type as it may
include other factors as well beyond the actual content transmitter
(i.e. URI details, context etc)

Regards
Henrik

---

From: Adam Barth <[email-masked]>
Date: Fri, 12 Jun 2009 18:08:55 -0700

On Fri, Jun 12, 2009 at 4:48 PM, Henrik
Nordstrom<[email-masked]> wrote:
> Removing the whole condition on
> whole text almost to the level of pure ignorance,

This statement makes me wonder if you've read the whole thread on this
topic.  Several folks pointed out that it's not HTTP's job to
determine which media type is used by the application.  I think
ignorance is the desired effect.

Adam

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 13 Jun 2009 14:57:58 +1200

> scenarios, the attacker chooses the file extension.
I presume therefore you can't use the content-type or file content 
either, since these are also potentially provided by an attacker?

I'm not sure what a Windows client that wants to launch an external 
application will do if it can't use the file extension, unless there's 
some database in Windows that maps to executables on some index other 
than file extension?

How does Chrome handle this?

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adam Barth <[email-masked]>
Date: Fri, 12 Jun 2009 22:31:14 -0700

On Fri, Jun 12, 2009 at 7:57 PM, Adrien de Croy<[email-masked]> wrote:
> I presume therefore you can't use the content-type or file content either,
> since these are also potentially provided by an attacker?

The content sniffing algorithm is a careful balancing act of
compatibility and security concerns.  For an in-depth discussion of
the security rationale behind the current algorithm, please see:
http://www.adambarth.com/papers/2009/barth-caballero-song.pdf

> application will do if it can't use the file extension, unless there's some
> database in Windows that maps to executables on some index other than file
> extension?

Thankfully this algorithm is not intended for use by a Windows client
in determining which external application to launch.

> How does Chrome handle this?

Chrome determines which external application to launch by the file
extension, not by the media type.  However, that's somewhat irrelevant
to this discussion.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Sat, 13 Jun 2009 10:47:31 +0200

> A resource is a bag of bits. I would object to this change.

Mark is correct. Please use terminology consistent with other IETF specs.

BR, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 13 Jun 2009 17:56:49 +0100

> a common way web servers choose content/type values.

Does the sniffing document not apply to browsers looking at content on
a local disk (therefore with no Content-Type), or does this mean it
recommends sniffing the content without looking at the filename on the
local disk?

I'm pretty sure Firefox and the like look at the file extension when
looking at content found on local disk.  But surely it does sniffing
at well, on local disk files?

Does the sniffing document not apply at all in that case, or is there
a different sniffing algorithm used which remains undocumented?

-- Jamie

---

From: Adam Barth <[email-masked]>
Date: Sat, 13 Jun 2009 10:16:29 -0700

> recommends sniffing the content without looking at the filename on the

I haven't investigated this question in detail, but I suspect the
answer will vary by browser.  There is very little interoperability
between browsers when interacting with the file system.

> at well, on local disk files?

Do you have evidence for this belief?  It should be fairly easy to
determine by looking at the source code.

> a different sniffing algorithm used which remains undocumented?

There is only one sniffing algorithm.  The question is only whether
its applied in this case.  More precisely, the question is whether the
"file" protocol handler assigns a media type using OS-specific
functionality before handing the response off to the next layer, where
content sniffing is performed on various media types (e.g., the empty
media type).

Adam

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 13 Jun 2009 19:15:10 +0100

> determine by looking at the source code.

It's easy to determine by simply trying it.

I've just created a small file with this content (not indented):

    <html><head></head><body>
    Hello, I am <b>HTML</b>
    </body></html>

If it's called test.html, it will display as HTML.
If it's called test.txt, it will display as plain text.
==> If it's called test.foo, it will display as HTML.
==> If it's called just test (no extension), it will display as HTML.

But if we change the file slightly, putting a single character x in
front like this:

    x<html><head></head><body>
    Hello, I am <b>HTML</b>
    </body></html>

If it's called test.html, it will display as HTML.
If it's called test.txt, it will display as plain text.
==> If it's called test.foo, it will display as plain text.
==> If it's called just test (no extension), it will display as plain text.

Therefore Firefox (3.0.10) does sniff a local file to determine how to
display it, and the sniffing algorithm (or whether to apply it) _does_
depend on the file extension.

> content sniffing is performed on various media types (e.g., the empty

It's clear from trying it that Firefox applies a sniffing algorithm to
local files, and either it is influenced by the file's extension, or
decided whether to apply the algorithm at all depending on the
extension.

I don't know what it does with FTP, but I wouldn't be surprised if
it's the same as local files.

Now, let's get back to HTTP.  I've done the same test as above with
HTTP in the same Firefox.

If the Content-Type is text/plain or text/html, then Firefox honours
the Content-Type, independent of whether the content has "x" at the
start in these two test files.

If the Content-Type is application/octet-stream, then Firefox does
different things depending on the URL's file extension.  If it ends
with .html, Firefox shows an error dialog(!), otherwise it offers to open
the file in an application of your choice.

If the Content-Type is blank, because I couldn't persuade Apache to
omit it completely, then Firefox behaviour depends on the URL's file
extension.

    <html><head></head><body>
    Hello, I am <b>HTML</b>
    </body></html>

If it's called http://.../test.html, it will display as HTML.
If it's called http://.../test.txt, it will display as plain text.
If it's called http://.../test.foo, it will display as plain text.
If it's called http://.../test, it will display as plain text.

    x<html><head></head><body>
    Hello, I am <b>HTML</b>
    </body></html>

If it's called http://.../test.html, it will display as plain text.
If it's called http://.../test.txt, it will display as plain text.
If it's called http://.../test.foo, it will display as plain text.
If it's called http://.../test, it will display as plain text.

As you see, Firefox applies a similar sniffing test in these examples
to decide whether to treat the resource as HTML or plain text, and it
does use the URL's file extension in making it's decision.

However, it doesn't use quite the same algorithm as for local files,
as you can see from the .html and .foo extension differences.

In the bigger picture, my point is that sniffing is used in practice,
in a major browser, for local files as well as HTTP (and FTP but not
shown here), and the decision about _whether_ to use it (at least)
does depend on the file extension for HTTP as well as for local files.

It would be good to document and standardise when the sniffing
algorithm is applied, dependent on file/URL extensions, for the same
reason that it is good to document and standardise what the sniffing
algorithm is.

I don't know from these tests if the sniffing is simply switch on/off
depending on file extensions or if it is influenced in a more
fine-grained way.

-- Jamie

---

From: Adam Barth <[email-masked]>
Date: Sat, 13 Jun 2009 11:36:12 -0700

> depend on the file extension.

Thanks for running these tests.  This is very helpful.  I don't agree
with you conclusion, however.  Here's an alternate explanation:

1) The "file" protocol handler uses OS-specific mechanisms to
determine the media type of the file.  (In this case, the OS uses the
extension).

2) After receiving the response from the file protocol handler, the
sniffing algorithm decides whether or not to sniff based on the media
type.  (In this case, the sniffing algorithm runs only if the media
type is unknown.)

3) Your first example triggers the HTML heuristic in the sniffing
algorithm.  Your second example does not and falls back to
text-or-binary.

I believe this theory accounts for all the behavior you report above.
Under this explanation, we see that the sniffing algorithm does not
use the file extension.

> decided whether to apply the algorithm at all depending on the

I believe the second account is clearer way to understand what's going on.

> it's the same as local files.

We should test to be sure.

> start in these two test files.

This is consistent with draft-abarth-mime-sniff.

> the file in an application of your choice.

I'm not able to reproduce this error dialog.  Can you test again?

> omit it completely, then Firefox behaviour depends on the URL's file

I recommend against using Apache to conduct these tests.  You can
achieve much more accurate results using netcat because you can
controller precisely which bytes you send to the client.

> If it's called http://.../test, it will display as plain text.

I'm not able to replicate these results.  When I omit the Content-Type
header (using netcat), these bytes render as HTML regardless of file
extension.

> If it's called http://.../test, it will display as plain text.

I can replicate these results.

> as you can see from the .html and .foo extension differences.

I suspect there's something strange going on with your Apache testing
harness.  I'd recommend testing again with netcat.

> does depend on the file extension for HTTP as well as for local files.

I disagree.  The decision about whether to use it depends on the media
type reported by the protocol handler.  How the protocol handler
determines the media type is up to it.  For HTTP, it looks at the
Content-Type header.  For files, Firefox apparently consults the OS.

> reason that it is good to document and standardise what the sniffing

If you'd like browsers to interoperate on local HTML files, then you
have much bigger problems then precisely when to apply the sniffing
algorithm.  That's a task for another day.  In any case, no changes to
the sniffing algorithm spec will be required.  We'll only have to spec
how the file protocol handler determines the media type.

> depending on file extensions or if it is influenced in a more

We'd likely gain additional insight from reading the code.

Adam

---

From: Ian Hickson <[email-masked]>
Date: Sat, 13 Jun 2009 19:43:00 +0000 (UTC)

On Sat, 13 Jun 2009, Julian Reschke wrote:
> Ian Hickson wrote:
> > ...
> > > Separately, as an editorial comment, as listed directly above, I'd like to
> > > see a big s/resource/resource representation/g (or just

In this particular case, I really would encourage the use of terms people 
understand. In any case, the term "resource" is correct when refering to a 
file, stream, or other "bag of bits". The terminology used by HTTP in this 
instance is inconsistent with wider usage (and remarkably confusing).

---

From: Julian Reschke <[email-masked]>
Date: Sat, 13 Jun 2009 21:51:49 +0200

> instance is inconsistent with wider usage (and remarkably confusing).

No, that usage isn't exclusive to HTTP. URI identify resources, but the 
retrieval returns you a representation of the resource, not the 
resource. For instance, that's true for "file:" as well (the contents 
may vary over time).

BR, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 14 Jun 2009 09:57:36 +1200

> determine the media type of the file.  (In this case, the OS uses the
Do you have any information about such APIs in windows?  I'd be keen to 
look into them further.  I thought I knew win32 API pretty well, and I'm 
unaware of any function to return the content-type for a file.

Regards

Adrien

---

From: Adam Barth <[email-masked]>
Date: Sat, 13 Jun 2009 14:58:41 -0700

On Sat, Jun 13, 2009 at 2:57 PM, Adrien de Croy<[email-masked]> wrote:
> Do you have any information about such APIs in windows?  I'd be keen to look
> into them further.  I thought I knew win32 API pretty well, and I'm unaware
> of any function to return the content-type for a file.

Looks like they're just setting in the registry:

http://src.chromium.org/viewvc/chrome/trunk/src/net/base/platform_mime_util_win.cc

Adam

---

From: Ian Hickson <[email-masked]>
Date: Sat, 13 Jun 2009 23:35:26 +0000 (UTC)

On Sat, 13 Jun 2009, Julian Reschke wrote:
> > 
> > In this particular case, I really would encourage the use of terms 
> > people understand. In any case, the term "resource" is correct when 
> > refering to a file, stream, or other "bag of bits". The terminology 
> > used by HTTP in this instance is inconsistent with wider usage (and 
> > remarkably confusing).
> 

Right, the resource varies over time. The URI identifies a different 
resource over time. Distinguishing the "representation of the resource" 
and the resource itself is a theoretical distinction that really isn't of 
much use in practice, and continuing to insist on the use of such 
terminology does nothing but alienate people who read specifications for 
the first time.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 14 Jun 2009 09:40:58 +1000

IME this distinction is critical and not making it causes all sorts of  
problems. Are you seriously suggesting that you want to re-defined  
"resource" across the scope of the Web -- both in the W3C and IETF?

---

From: Ian Hickson <[email-masked]>
Date: Sat, 13 Jun 2009 23:48:30 +0000 (UTC)

On Sun, 14 Jun 2009, Mark Nottingham wrote:
> problems.

Like what?

> Are you seriously suggesting that you want to re-defined "resource" 
> across the scope of the Web -- both in the W3C and IETF?

I'm claiming that the term "resource" already means "bag of bits" and that 
it is only within the context of the URI specs and the HTTP specs that 
anyone claims otherwise, and that these claims are based on a distinction 
that is purely theoretical and doesn't actually affect deployed content, 
or users, except for confusing them. (Can you get any Web designer to 
correctly explain the difference between the terms resource identifier, 
resource, and resource representation as you use them?)

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 14 Jun 2009 13:01:19 +1200

that code shows Chrome using the file extension to look up a content 
type from the windows registry, so it's clearly showing Chrome using the 
file extension to determine content type.  The fact that the DB of 
extension to type is provided by the OS, doesn't alter the fact that 
Chrome is using it.

Adrien

---

From: Adam Barth <[email-masked]>
Date: Sat, 13 Jun 2009 18:03:06 -0700

On Sat, Jun 13, 2009 at 6:01 PM, Adrien de Croy<[email-masked]> wrote:
> that code shows Chrome using the file extension to look up a content type
> from the windows registry, so it's clearly showing Chrome using the file
> extension to determine content type.  The fact that the DB of extension to
> type is provided by the OS, doesn't alter the fact that Chrome is using it.

You'd have to examine the callers of that function to see what it's
used for.  It might not actually be called at all.  I think it's used
in some security checks for downloaded files.  It's certainly not used
by the sniffing algorithm.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 14 Jun 2009 11:04:59 +1000

>> IME this distinction is critical and not making it causes all sorts  

Confusion, of course. Changing the terminology of the Web fifteen  
years after it was established will only add to that.

> or users, except for confusing them.

You're wrong, and in terms of the Web, HTTP and URI are two out of  
three -- you're outnumbered as well.

> (Can you get any Web designer to

Do you think they'll actually be reading these documents?

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 14 Jun 2009 13:18:03 +1200

OK..This does raise the question of what does a server do.

I don't know of any servers (in my relatively limited knowledge of 
servers) that doesn't just use the file extension to choose content type 
when serving. 

If the served type is supposed to be the authoritative reference, you 
get problems when a server has a smaller database of extension-to-type 
than the client.

Otherwise you'd need to re-engineer servers so that content providers 
can manually specify content type when providing content.  That then 
means authors need to know about registered types etc.

So I can't see this changing any time soon, no matter what we write into 
specs.

Adrien

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 01:32:23 +0000 (UTC)

On Sun, 14 Jun 2009, Adrien de Croy wrote:
> when serving. If the served type is supposed to be the authoritative 
> reference, you get problems when a server has a smaller database of 
> extension-to-type than the client.

I believe that everything you're writing above is consistent with what 
Adam's draft says, namely:

# For resources fetched from the file system, user agents should use
# platform-specific conventions, e.g. operating system file extension/
# type mappings.
#
# File extensions MUST NOT be used for determining resource types for
# resources fetched over HTTP.

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 01:45:55 +0000 (UTC)

On Sun, 14 Jun 2009, Mark Nottingham wrote:
> > > IME this distinction is critical and not making it causes all sorts 
> > > of problems.
> > 
> > Like what?
> 
> Confusion, of course. Changing the terminology of the Web fifteen years 
> after it was established will only add to that.

I contend that this isn't a change except for a tiny number of people 
(probably all of whom are on the HTTP WG mailing list). Most people have 
no idea what a "resource representation" is or why it would be different 
than a resource.

> > I'm claiming that the term "resource" already means "bag of bits" and 
> > that it is only within the context of the URI specs and the HTTP specs 
> > that anyone claims otherwise, and that these claims are based on a 
> > distinction that is purely theoretical and doesn't actually affect 
> > deployed content, or users, except for confusing them.
> 
> You're wrong, and in terms of the Web, HTTP and URI are two out of three 
> -- you're outnumbered as well.

The HTTP and URI specs are confusing. This is the kind of problem we 
should be working to stop, not making worse.

> > (Can you get any Web designer to correctly explain the difference 
> > between the terms resource identifier, resource, and resource 
> > representation as you use them?)
> 

They certainly won't if we make them impenetrable! We should be working to 
make our documents more accessible to a wider audience. Making them use 
terms that are counter to what everyone actually understands is not a 
good way to welcome more people to the standards development process. It 
merely continues the stereotype that we are all ivory tower academics.

Can you get any Web browser developer to correctly explain the difference 
between the terms resource identifier, resource, and resource 
representation as you use them? I hope you agree that we'd at least expect 
Web browser developers to read our specs, even if you don't think we 
should be aiming for Web designers in general to read them.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 14 Jun 2009 11:55:05 +1000

> merely continues the stereotype that we are all ivory tower academics.

The first task of a specification is to clearly, unambiguously state  
how to implement. If it can appeal to a broader audience, that's  
great, but that's not a primary goal; you can rely on user guides,  
tutorials and the like for that (and indeed these documents are where  
the bulk of users will get their knowledge, not the spec).

A good specification will also outlive both its authors and the  
context it was written in. Using imprecise and conflicting terminology  
gets in the way.

> should be aiming for Web designers in general to read them.

If they're reading the specs and paying attention, define the term.  
Easy.

Changing terminology because lots of people don't understand the  
concepts of the Web won't improve their understanding of the Web; they  
have to apply themselves for that to happen.

At any rate, we're obviously not going to convince each other about  
this. Go ahead and submit this use of the term in a spec -- either to  
the IETF or the W3C -- and let's see what happens.

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 14 Jun 2009 14:05:42 +1200

If there were no distinction between "resource" and the bag of bits 
returned, how would one go about multi language content, where the URIs 
are the same, but different bags of bits are returned for the same URI 
requested based on Accept-Language?

Adrien

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 04:06:43 +0000 (UTC)

> requested based on Accept-Language?

Different resources are returned every day for the same URI. For example, 
news.google.com differs from minute to minute. Two different people 
looking at the same Google search results page get different results 
depending on their search history. Why is that a problem? When you access 
a server and give it a particular URI, you get a resource back. No need to 
have a nebulous theoretical concept of "resource" that doesn't actually 
exist; it isn't needed to explain what's going on, and most people -- even 
most people who work with Web technologies daily -- don't think that way.

(BTW, can you point me to a site that actually does that with Accept- 
Languge? Or any HTTP con neg for that matter. I've never been able to find 
any site that does con neg other than demos, experiments, and user agent 
tests. I'd love to actually test how browsers do this in real-world 
situations rather than in controlled experiments.)

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 04:09:44 +0000 (UTC)

On Sun, 14 Jun 2009, Mark Nottingham wrote:
> 
> The first task of a specification is to clearly, unambiguously state how 
> to implement.

*Exactly*. And there is no need to refer to a theoretical separation 
between "resources" and "resource representations" to do that.

> have to apply themselves for that to happen.

No. Their understanding of the Web is fine. The problem is that the Web 
doesn't actually match what you want it to be. There is no actual 
difference between the "resource" and the "resource representation". The 
"resource" as you describe it _doesn't exist_. It's a theoretical 
construct that is completely unnecessary to describe how the Web works.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 14 Jun 2009 14:15:05 +1000

Of course resources are abstract, Ian.

You may have no need to refer to anything but the response (or entity,  
or representation; all might be appropriate), but others do need to  
refer to the thing that the URI identifies, and the thing that  
generates the response.

This argument seems to be covering the same ground quite quickly. As I  
said, if you want to press the point, go ahead and use the term this  
way in a submission; going around in circles on this list doesn't do  
any good.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 14 Jun 2009 08:46:02 +0200

> servers) that doesn't just use the file extension to choose content type 

Many HTTP servers that are also WebDAV servers will store the mime type 
as separate meta data, and will also allow clients to specify it when 
doing a PUT.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 14 Jun 2009 08:54:55 +0200

> most people who work with Web technologies daily -- don't think that way.

*You* may not need that distinction. However, when writing a spec, you 
aren't doing it for yourself, but for an audience of readers.

Other spec writers and standards bodies *clearly* see this differently. 
Changing terminology at this point is only going to cause confusion.

So, as Mark said, you can ignore the advice, and then will have to 
change things anyway once you attempt to publish as RFC.

> situations rather than in controlled experiments.)

I have written servers that use Accept-Language when the authenticated 
user's locale was unknown. But I currently can't point to any public 
instance you might be able to test with.

Other than Accept-Language, there is an incredible amount of conneg 
going on; consider "Accept-Encoding:", "Cookie:", and "User-Agent:" -- 
yes, all of these cause content negotiation once the server decides to 
vary the representations it sends.

BR, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 14 Jun 2009 09:14:20 +0200

>situations rather than in controlled experiments.)

I normally point people to <http://www.debian.org/intro/cn>; searching
old Usenet articles to see for how long I've been doing that I found
<news:[email-masked]>, which reports, in April 2000, that
<http://www.google.com/> recently started delivering different content
based on the Accept-Language header. I've checked, and it still does.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 07:36:56 +0000 (UTC)

Thanks to everyone for the various examples of pages that do content 
negotiation.

On Sun, 14 Jun 2009, Julian Reschke wrote:
> > 
> > No need to have a nebulous theoretical concept of "resource" that 
> > doesn't actually exist; it isn't needed to explain what's going on, 
> > and most people -- even most people who work with Web technologies 
> > daily -- don't think that way.
> 

It is precisely this audience of readers who are most likely to benefit 
from specs using concrete, understandable, and familiar terms like 
"resource" rather than theoretical, abstract, and confusing terms like 
"resource representation". There really is no reason to refer to an 
abstract concept here. There is an identifier, there is a server, there is 
an actual resource (by the dictionary definition, not the "newspeak" 
definition used in the HTTP spec). Why introduce another term? What else 
is there to refer to?

When you tell your command-line tool to fetch http://google.com/, it 
returns an actual sequence of bytes form a server. It doesn't obtain a 
representation of something, it just gets a thing. There's no nebulous 
"resource" here, just a concrete set of bytes (which will vary based on 
many things, such as cookies, IP, headers, etc).

> Changing terminology at this point is only going to cause confusion.

It is the HTTP and URI specifications that are, IMHO, continuing to cause 
confusion by insisting on unfamiliar terminology.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 14 Jun 2009 09:39:17 +0200

> based on the Accept-Language header. I've checked, and it still does.

Oh well :-)

I even didn't bother to check www.google.com, as I assumed that Ian did.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 14 Jun 2009 09:51:53 +0200

> definition used in the HTTP spec). Why introduce another term? What else 

Under your "definition", the resource is selected based on *both* the 
URI and the context of the GET request (request headers, point in time). 
It's *possible* to explain things that way, but it is totally confusing 
because the *URI* is supposed to be the identifier, by definition. So it 
seems to me you're creating more confusion, not less.

> returns an actual sequence of bytes form a server. It doesn't obtain a 

Yes. Plus metadata.

> many things, such as cookies, IP, headers, etc).

Yes, it does get you a sequence of octets (entity body) plus metadata. 
"We" call it "representation of the resource", you call it "resource".

> confusion by insisting on unfamiliar terminology.

Unfamiliar to whom?

Certainly not the W3C, see <http://www.w3.org/TR/webarch/#dereference-uri>.

But again; this discussion is pointless in that the inconsistency of 
terminology will be raised in Last Call, and then will have to change 
anyway.

BR, Julian

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 08:23:42 +0000 (UTC)

> seems to me you're creating more confusion, not less.

That's why "URL" is a far better term than "URI". Most of these 
terminology problems were fine until we started taking perfectly good Web 
resource addresses and started treating them like they meant something 
more. URIs in practice don't identify anything; that is a theoretical 
model that simply doesn't match common practice.

Indeed, the spaces where URIs have been used as identifiers have shown why 
having dereferencable identifiers is a bad idea -- for instance they are 
unwieldy, leading to people using techniques like prefixes and prefix 
declarations to avoid the problem; and they invite dereferencing even when 
that isn't necessary, leading to high load for popular identifiers [1].

[1] http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic

> > It is the HTTP and URI specifications that are, IMHO, continuing to 
> > cause confusion by insisting on unfamiliar terminology.
> 
> Unfamiliar to whom?

The vast majority of engineers, including those implementing technologies 
such as HTTP and URIs. By using such confusing abstract terminology, these 
specs contribute to the less than perfect interoperability that has 
plagued the Web for the past two decades.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 14 Jun 2009 10:49:45 +0200

> model that simply doesn't match common practice.

HTTP URIs are URLs, and URLs simply are URIs that also double as 
locators (see RFC 3986). I don't see how this changes the definition of 
being an identifier at all.

Whether HTTP URLs as identifiers for things like RDF properties or XML 
namespaces is a good idea is an entirely different discussion. Mixing 
those here isn't helpful.

> [1] http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic

Again, what does this have to do with the terminology discussion we have?

> plagued the Web for the past two decades.

Let's agree to disagree on this. But it would be interesting if you 
could back up that claim. In particular, how does the terminology choice 
you are disagreeing with cause interoperability problems?

BR, Julian

---

From: Werner Baumann <[email-masked]>
Date: Sun, 14 Jun 2009 11:26:20 +0200

Am Sun, 14 Jun 2009 07:36:56 +0000 (UTC)

> Thanks to everyone for the various examples of pages that do content 
Now its going to get cynical, so I must join.

Please enter "telnet www.debian.org 80" at your command line and after
the connection is established enter either

GET / HTTP/1.1
Host: www.debian.org
Accept-Language: de

or

GET / HTTP/1.1
Host: www.debian.org
Accept-Language: en

The bags of bits will look quite different. Indeed they represent the
same content in different languages.

I must admit this example is a bit unfair on my side, because Debian
lacks everything that is important in WWW today (no advertising, no
data mining, no cross-side-whatsoever, no active content trying to
conquer your computer, just information).

Werner

P.S.: As far as I can remember there has been a lengthy discussion
about this terminolgy about halve a year ago. It ended in removing the
unnecessary term "variant".

---

From: Werner Baumann <[email-masked]>
Date: Sun, 14 Jun 2009 11:45:34 +0200

Am Sun, 14 Jun 2009 08:46:02 +0200
schrieb Julian Reschke <[email-masked]>:

> Many HTTP servers that are also WebDAV servers will store the mime
> type as separate meta data, and will also allow clients to specify it
> when doing a PUT.
> 
Do you know servers that really get this right?
I am asking because I once tried to implement setting MIME-type in
davfs2 (using PROPPATCH). I did my first tests with
Apache/mod_dav/mod_dav_fs and IIS and then immediately stopped work on
this. What I got is:
PROPFIND returned the MIME-type as set by davfs2
GET and HEAD returned another MIME-type, the one derived from the file
extension.

Werner

---

From: Julian Reschke <[email-masked]>
Date: Sun, 14 Jun 2009 11:52:54 +0200

> Do you know servers that really get this right?

At least one (SAP KM), but I'm quite sure this is also true for others, 
such as Apache Jackrabbit or Xythos WFS.

> GET and HEAD returned another MIME-type, the one derived from the file

Well, that's a bug. No matter what it returns, it should be consistent. 
It may be a good idea to open a bug wrt to Apache/mod_dav then.

BR, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 14 Jun 2009 12:24:31 +0100

> way in a submission; going around in circles on this list doesn't do  

I sort of agree with everyone on this.

A 'representation' often isn't a representation; it's sometimes the
result of a calculation.  You could think of 1.41421356 as a
'representation' of the resource at (hypothetical)
http://example.com/calculator?sqrt(2), but the common language sense
of 'representation' doesn't fit that.  It's just a result.

But it's not a resource being returned either.

The calculator at (again hypotheical) http://example.com/calculator is
a resource in the common language sense, without the query part.  Yet
it might never be permitted to fetch that URL.

You can't use the fact there's a query part to distinguish, because of
http://example.com/calculator/sqrt_2, where the calculator accepts
it's parameters as the remainder of the URL.

So the terminology doesn't model the real web particularly well either
way.  So let's just use definitions and admit they don't fit common
language usage particularly well.

Personally I was quite happy with 'entity', which I always thought of
as what is found in a request or response, and that all the stuff
about resources and representations was outside the scope of HTTP,
because it's implicitly describing server internals which HTTP should
not do, even abstractly.

So far as I can tell in a hand-waving *practical* sense, 'resource' is
another name for URI/URL (because "the thing at the URI/URL" is pure
abstraction and often doesn't exist other than as an idea), the
URI/URL distinction isn't proving helpful any more (better to use URNs
or a non-HTTP method if you want to say you don't want dereferencing),
and 'representation' means practically the same as 'entity' or 'body'.

I appreciate there's a theoretical model behind resource and
representations, but it doesn't fit what really happens particularly
well in a lot of HTTP usages.

-- Jamie

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 18:41:15 +0000 (UTC)

> Now its going to get cynical, so I must join.

Sorry if that sounded cynical; I did not mean it that way. I truly am 
thankful for the examples, they will be very useful for me.

---

From: Ian Hickson <[email-masked]>
Date: Sun, 14 Jun 2009 19:21:56 +0000 (UTC)

> being an identifier at all.

I'm not arguing that they aren't identifiers, I'm arguing that when you 
dereference them you get an actual concrete resource, and that saying 
that you get a resource representation is pointless and confusing hair- 
splitting which doesn't actually help people understand the specs when 
they implement them, since the thoretical "resource" construct never 
actually needs to be dealt with in practice.

> those here isn't helpful.

I couldn't agree more.

> > Indeed, the spaces where URIs have been used as identifiers have shown 
> > why having dereferencable identifiers is a bad idea -- for instance 
> > they are unwieldy, leading to people using techniques like prefixes 
> > and prefix declarations to avoid the problem; and they invite 
> > dereferencing even when that isn't necessary, leading to high load for 
> > popular identifiers [1].
> > 

By persuing a terminology that encourages people to think of URIs as 
something more than just a way to obtain a bag of bits, we have encouraged 
the kind of usage seen in RDF and Namespaces in XML.

If we just stuck to concrete terms and didn't go off on a theoretical 
tangent distinguishing "resource" and "resource representation", people 
would not be tempted to use that theoretical difference for actual 
purposes.

> > > > It is the HTTP and URI specifications that are, IMHO, continuing 
> > > > to cause confusion by insisting on unfamiliar terminology.
> > >
> > > Unfamiliar to whom?
> > 
> > The vast majority of engineers, including those implementing 
> > technologies such as HTTP and URIs. By using such confusing abstract 
> > terminology, these specs contribute to the less than perfect 
> > interoperability that has plagued the Web for the past two decades.
> 

When implementors don't understand the terms used in a specification, they 
start assuming what it means. When faced with two terms that for all 
practical purposes mean the same thing as here, they start inventing 
differences to justify the spec text in their mind. I've seen this over 
and over again.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 14 Jun 2009 13:06:13 -0700

> actually needs to be dealt with in practice.

I'd like to see how you would describe a resource that accepts POSTed
information without ever returning a "bag of bits", how you are going
to describe a resource whose only purpose is to redirect to the
"site of the day", or how you would implement a gateway to my
friend's air conditioning unit.  I could enumerate thousands of
examples of how limited your view of the Web really is, but then
I already have done that many times.  Read the archives.

....Roy

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 15 Jun 2009 15:06:54 +0200

> instance is inconsistent with wider usage (and remarkably confusing).

HTTP resources vary over multiple axis, and may have a large number of
different representations each with different bags of bits but
supposedly mostly the same semantic meaning to the end user.

  - Time, as already mentioned (versions)
  - Language (English & Swedish, depending on preference by the user)
  - Encoding (i.e. identity or gzip depending on capabilities of the
agent)
  - Cookies (i.e. anonymous or authenticated variants of the same
resource)
  - and even media type (i.e. gzip / jpeg / png depending on
capabilities of the agent)
  - or other server defined properties

resource is used as the verb defining the resource in general,
irregardless of it's specific representation, and is identified by an
URI.

Each representation MAY also have a unique URI identifying just that
representation as a resource, but is kind of besides the point.

HTTP intends that these representations should have pretty much the same
semantic meaning to the user, but can naturally not enforce it. It's
clearly stated however that the representations DO NOT share the same
bag of bits..

We have already had this resource/entity/whatever discussion many times,
and settled for resource in the general term and representation (of a
resource) for a specific bag of bits. Lets not open that discussion
again.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 15 Jun 2009 15:15:50 +0200

> model that simply doesn't match common practice.

Are you sure you don't mix up URI with URN here?

Regards
Henrik

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 16 Jun 2009 16:45:09 +0900

> Thanks to everyone for the various examples of pages that do content

Here are two more:
http://httpd.apache.org/docs/2.2/ (other versions, too)

http://www.w3.org/International/O-charset (and various other articles)

> There is an identifier, there is a server, there is

I just checked the online dictionary I'm usually relying on:
http://www.merriam-webster.com/dictionary/resource

I don't see "bag of bits" or anything close to that anywhere on that 
page. If you can point to a dictionary that includes "bag of bits" (or 
anything close to that), I'd appreciate.

In traditional computer science, the word 'resource' is most widely used 
in the field of Operating Systems (OS), where it denotes the stuff that 
the OS manages, such as memory, hard disks, processor time, printers, 
and so on.

Hope this helps,    Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 16 Jun 2009 15:45:01 +0100

> I already have done that many times.  Read the archives.

Ok, but what about a resource that accepts POSTed information _and_
information in the URL?  For example a hypothetical POSTable
http://example.com/post-news/author-is/Jamie/subject-is/Good%20morning

Then the URL doesn't point to the conceptual resource any more.  Part
of the URL does (http://example.com/post-news/); the rest of the URL
does not identify a resource, but is input to it.

Conceptually it's a mistake to equate "the thing that a specific URL
refers to" with the useful concept of a "resource that we may interact
with", because the most useful mapping often doesn't work that way.

You can say that the thing identified by the above long URL is a
resource, but it's a bit of a stretch and doesn't fit what it really does.

So if it's a conceptual mistake, why is it useful in specifications?

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 16 Jun 2009 15:52:22 +0100

Henrik Nordstrom wrote:
> semantic meaning to the user,

The is clearly not true in the case of some forms and web applications,
where every step through the form/application is served on the same
URL, with state managed through a combination of POSTed hidden form
fields and/or server-side state addressed through cookies or a URL token.

It's clearly the same resource they are dealing with ("the
form/appliction"), I won't argue against that, but the representations
served up have quite different semantic meanings to the user.

An entire complex site can be constructed that way.

> but can naturally not enforce it.

Does that mean such applications are not complying with
specifications, even though they are technically possible?

-- Jamie

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 16 Jun 2009 09:47:52 -0700

> does not identify a resource, but is input to it.

No, (http://example.com/post-news/) is not *the* resource.
It may be a resource, but it is not the resource to which
the URI above identifies, and therefore your assumption about
the meaning of "resource" in Web architecture is wrong.

> So if it's a conceptual mistake, why is it useful in specifications?

Because it isn't a conceptual mistake.  Your mistake is assuming
that "resource" has something to do with the implementation behind
the server interface.  Ian's mistake is assuming that what people
refer to is the bag of bits they get back, as opposed to the function
being exploited by the request method on the server-provided resource
at the time of the request (e.g., if the link is to today's weather
report then the resource is today's weather report, not some former
weather report's bag of bits).

In any case, the word "resource" is far more commonly used the way
I use it than the way that Ian described.  As indicated, it comes
direct from the English language.  Ian's opinion is simply wrong
and you can't "simplify" a specification by defining the basic
architectural elements incorrectly.

....Roy

---

From: Adam Barth <[email-masked]>
Date: Tue, 16 Jun 2009 10:03:31 -0700

Thank you all for this discussion.  It appears that the path of least
resistance is for me to change the terminology in the next draft,
which I will do.

Adam

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 16 Jun 2009 19:09:16 +0100

> the meaning of "resource" in Web architecture is wrong.

No, because I'm not describing the meaning of "resource" in Web
architecture here, I'm describing the sensible use of the English
concept of a resource.

Web architecture says the resource accessed through

is different from the resource accessed through

http://example.com/post-news/author-is/Roy/subject-is/Bzzt

Which doesn't fit very well the English concept of "the resource"
which is being accessed when you actually do those things.

(If those URLs didn't just post news, say if you could read filtered
news through them too, _then_ the concept would fit better.  It has
nothing to do with implementation; it is about conceptual models).

So you have to *define* the Web architecture meaning of resource, and
that will always be a little dissonant with what the conjures up for
people who aren't familiar with Web architecture.

> Your mistake is assuming that "resource" has something to do with
> the implementation behind the server interface.

No, I'm not talking about the implementation at all.  I'm talking
about concepts, models and common language meanings.

> Ian's mistake is assuming that what people

I agree with that.

> direct from the English language.

Your usage comes from the meaning as used in Web architecture, and
seems to my mind dissonant with the meaning used in English in these
cases.

I'm quite familiar with the Web architecture meaning.  I just don't
think it fits very well the conceptual models that are actually being
used on some web sites, and I think that is the root of Ian's idea
that it confuses some developers.  Though I don't agree with Ian's
suggestion.  Obviously the Web architecture meaning fits some sites'
conceptual models quite well.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 16 Jun 2009 20:08:31 +0200

> fields and/or server-side state addressed through cookies or a URL token.

For POST resources the assumption is that the function of the resource
is pretty much the same regardless of which "representation" was
negotiated given similar request payload. But it's hard to talk about
"representation" in the same manner for POST resources..

> served up have quite different semantic meanings to the user.

The "resource" in case of POST is the server agent accepting the POST
request and producing a response representation based on the input and
server state.

> An entire complex site can be constructed that way.

Sure. And some do it that way... one of the banks I use for example..
(very annoying interface, but that's another story).

> specifications, even though they are technically possible?

Do you mean applications providing entirely different representations of
the resource in response to negotiation, or the POST case discussed
above?

Both comply with the spec, even if the first is not how negotiation of
the representation is supposed to be used, but nothing breaks from it
except for confused users and their confidence in the site/service when
content differ completely when using different browsers or browser
versions...

Regards
Henrik

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 17 Jun 2009 00:39:39 +0200

* Mark Baker wrote:
>the URI, not the bag-o-bits returned in an HTTP response.

When people pack their bag, they rarely arrange the items in a se-
quential order. A bag is typically a synonym for a multiset, which
is usually not ordered either. So this is not a bag of bits.

The current draft uses the term "resource" in several senses. Where
it is meant to refer to a bit stream, neither resource nor represen-
tation is appropriate; neither term makes it clear which bit stream
it referred to, e.g., whether content encodings are removed or not.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 16 Jun 2009 16:05:06 -0700

> architecture here, I'm describing the sensible use of the English

Which definition would that be?

> http://example.com/post-news/author-is/Roy/subject-is/Bzzt

Hard to tell from your example.  The resource is either a form
containing that subject pre-filled (e.g., like a GET on any mailto
URI) or accessing that URI actually does post that news (which is
still a resource and might even be standards-compliant with HTTP
if it does not allow GET).  The latter is not a very useful resource,
but it is still a resource because I can use it to Bzzt many times.

> which is being accessed when you actually do those things.

Again, which English concept of "the resource"?  There are five
definitions in Webster and more in OED.  Or do you have some other
authoritative definition in mind, such as one found in a peer-reviewed
journal or academic work?

In Webster, the 1a definition (a source of supply or support) and
1e (a source of information or expertise) are both about *doing*
something.  That's what a resource is -- something that you can go
back to in the *future* as a source to supply something, whether
that something is inbound in nature (e.g., storage space) or
outbound in nature (e.g., a home page).

Web architecture says (by definition) that a URI is a resource
identifier, and thus anything identifiable with a URI must fit
within our model of "resource".  It is not possible to define a
model that we say "describes the Web", like the REST model that is
the basis of URI and HTTP standards, and at the same time say that
we are only interested in describing things that look like a
response to a file-like request.  The standard protocols must
support the entire scope of the Web.  The terminology must be
sufficiently expressive to encompass the entire scope without
preventing that which already appears on the Web.

> people who aren't familiar with Web architecture.

I did define that meaning and it is *also* consistent with the English
term.  Honestly, I think you need to do a little more research on the
subject before you can seriously suggest that the Web architecture
definition in use for the past 14 or so years, peer-reviewed five
times in the IETF, twice in IEEE, and by my own dissertation committee
is somehow less accurate or useful in standards than ... what?

> about concepts, models and common language meanings.

That's what I am talking about.  I can't solve the case where your
opinion of "common" does not match my opinion of "common" other than
to point to established peer-reviewed writing, which is the very
thing you are arguing against.

> seems to my mind dissonant with the meaning used in English in these

No, it doesn't.  My research is based on the entire Web and how
it has been/is/will be used everywhere.  That research fed into
most of the Web architecture and my dissertation, which has been
reviewed by a ridiculously large number of academics (both inside
and outside computer sciences) and industry professionals.  If I
had screwed up the definition of resource or somehow brutalized
the "common" use of the term, then there would have be a long trail
of comments and articles telling the whole world how I got it wrong.

Maybe someone will do that this year, and after being adequately
peer reviewed and found to be more expressive or understandable
than the current terminology set, then the associated standards
will be rephrased in that new terminology.  However, that sure
as heck isn't going to come out of an email thread based on
individual opinions about what is "common".  Common is always
a function of context. If your context is a management school,
then your definition of "resource" is a person employed or
contracted to do work.

In any case, the purpose of our specifications is to *define* Web
architecture, so it is kind of pointless suggesting that our audience
would not be familiar with the terms.

> conceptual models quite well.

I have yet to see a site that it doesn't fit well.  The fact that
a typical browser page consists of representations of many different
resources rendered according to a particular media type processing
algorithm does not change the fact that each of those resources
can also be accessed and manipulated individually.  Thus, we can
and do say that "google reader" is a resource even though the
content it displays is unique to each user's cookie set and
request time, and that does not take away from the individual
elements being composed dynamically from other resource
representations via AJAX.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 17 Jun 2009 13:35:06 +1200

I think the original issue raised by Ian was related to the concept of
a "resource" in how it's defined in HTTP being an "abstract" thing, and
inferring that being abstract is a bad thing.

However if you consider all the types of thing that can be referred to
by a URI, then you quickly see it is indeed an abstract concept.  Given
that we use the name URI, where R stands for "resource", therefore the
resource must be the thing identified by a URI.  I don't see HTTP
attempting to define the resource any further than this, and yes it is
necessarily abstract, and that provides it scope to be anything you can
think of.  So being abstract isn't a bad thing - it's the reason for
HTTP's power.

As for the data you get back from a request, "resource representation"
probably IMO reads a bit too much into it.  It may not necessarily
represent any resource.  To me it's simply a result of the request.
It's just data.  It may (e.g. a GIF file) be the same binary data as
the "resource" requested, or may not (e.g the result of script output,
where the resource could be deemed to be the PHP file, or e.g. the coke
machine itself vs the bytes coming back telling you how many cans are
left).  You could consider on some level that the result of a request
gives you something you can use, which could be deemed a resource in
the broader sense.  But it's not *the* resource identified by the URI.
And since there must be a distinction between the thing identified by a
URI, and the bytes (if any) you get back from a request, and since the
label "resource" is already taken, we need another name for the result
data.  We simply can't use the word "resource" there without creating
confusion.

The concept of "resource representation" becomes perhaps more easily
understood where there may be more than one result of a request which
may be deemed on some meta-level to be equivalent (as defined by the
content author) but which is different in terms of the bytes returned,
such as different language versions of the same page.  It contains the
same information, but in a different language so the site author deems
it to be the same on some meta-level.

However, I'm happy with the concept of what "resource" and "resource
representation" refer to.

My 2c

Regards

Adrien

Ian Hickson wrote:

I couldn't agree more.

Again, what does this have to do with the terminology discussion we
have?

Unfamiliar to whom?

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 18 Jun 2009 12:00:22 +0200

> it referred to, e.g., whether content encodings are removed or not.

Can you point out an example where this is ambiguous regarding encoding?

In HTTP content-encoding is never added/removed as part of normal
operations. Doing so creates a new resource variant, same as selecting a
different language or media type.

Agents naturally must remove content-encoding in their presentation
layer, and non-shared caches MAY to optimize reuse but if they do it's
their responsibility to properly handle the difference.

Regards
Henrik

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 18 Jun 2009 12:13:50 +0200

>their responsibility to properly handle the difference.

I was referring to draft-abarth-mime-sniff-01, not any of the httpbis
drafts. In the former there is for example "The user agent MAY wait for
512 or more bytes of the resource to be available." That could be 512
bytes before removing chunked transfer encoding, all transfer encodings,
the content encoding, etc. depending on which byte stream one believes
"resource" (or "representation" per Mark's suggestion) refers to.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 18 Jun 2009 12:43:10 +0200

tor 2009-06-18 klockan 12:13 +0200 skrev Bjoern Hoehrmann:

> drafts.

Phew.. you got me a bit worried there for a while ;-)

Regards
Henrik

---

From: Adam Barth <[email-masked]>
Date: Thu, 18 Jun 2009 09:13:46 -0700

> "resource" (or "representation" per Mark's suggestion) refers to.

The intent is after transfer and content encodings are removed.
What's the correct way to refer to those bytes?

Adam

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 18 Jun 2009 20:32:50 +0200

> What's the correct way to refer to those bytes?

There is no unique defined HTTP term for this, so "entity-body with
transfer and content encodings removed".

Regards
Henrik

---

From: Adam Barth <[email-masked]>
Date: Thu, 18 Jun 2009 14:29:06 -0700

> transfer and content encodings removed".

Thanks.

Adam