Subject: Use of extension-header in entity-header

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 20 May 2009 20:04:52 +0200

Hi,

  In RFC 2616 and the current drafts the extension-header is used only
in the entity-header production, suggesting all headers not defined by
it are entity headers, which I don't think is intended. Perhaps it
should be referenced in Request and Response as alternative to entity-
header instead?

regards,
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 20 May 2009 14:53:27 -0500

Bjoern Hoehrmann wrote:
> it are entity headers, which I don't think is intended.

Part 1 and Part 2 both explicitly states that all unrecognized header fields
are to be treated as entity headers. However, Part 6 says "Unrecognized
header fields SHOULD be ignored by the recipient and MUST be forwarded by
transparent proxies." I think Part 6 is right the other statements regarding
unrecognized header as entity header fields should removed. I'm not sure how
this interacts with the rules for Content-* header fields though.

> Perhaps it should be referenced in Request and Response as alternative to
> entity- header instead?

+1

But, even with that done, there is a serious problem with extension-header
in the grammar. If the value of a HTTPbis-defined header field is malformed
in a message, then according to the grammar, the malformed header field is
to be treated as an extension header. For example:

    HTTP/1.1 200 OK
    Content-Length: hello

According to the grammar, we are supposed to parse this as a
syntactically-valid response with a "Content-Length" extension header.
Combined with the statement above, we should ignore this unrecognized
Content-Length extension-header field; that is, we shouldn't report an
error. This also means that a single-valued header field that is duplicated
should be ignored as an unrecognized header field. It also means that any
general-, request-, or response- headers in a chunked message trailer must
be treated as (usually unrecognized) extension-headers and ignored. I think
those are all pretty surprising and unintended consequences. Almost all
implementations parse headers using the message-header production, and then
parse and interpret them based on their names, which ends up with much
different results for fields malformed header field values.

- Brian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 20 May 2009 22:47:41 +0200

* Brian Smith wrote:
>unrecognized header as entity header fields should removed.

How implementations treat unrecognized headers is a separate concern; my
issue is with other specifications adding new headers that aren't entity
headers. They would require an extension point that's not currently en-
coded in the grammar.

>to be treated as an extension header.

That seems okay, as implementations would be unable to tell whether it
is indeed malformed, or just conforms to some definition that postdates
the implementation, in other words, some kind of extension.

>error.

That is not the implication. An implementation is always free to note
that it has encountered content that it does not fully support, or re-
fuse to process a message for reasons of security if it detects what
seems like a malformed header.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 20 May 2009 17:10:54 -0500

Bjoern Hoehrmann wrote:
> How implementations treat unrecognized headers is a separate concern;
> my issue is with other specifications adding new headers that aren't
> entity headers. They would require an extension point that's not
> currently encoded in the grammar.

It is encoded in the grammar. The names of the grammar productions do not
define the semantics. The extension-header rule could (should) be factored
out into Request, Response, and Trailer, but the resultant grammar will be
equivalent to the original.

> >But, even with that done, there is a serious problem with extension-
> >header in the grammar. If the value of a HTTPbis-defined header field
> >is malformed in a message, then according to the grammar, the
> >malformed header field is to be treated as an extension header.
> 

Specifications cannot change the syntax of the header fields defined in HTTP
1.1 in an incompatible way. If they change the syntax they need to use a
different HTTP version number.

> seems like a malformed header.

A conditionally-compliant implementation can, but an
unconditionally-compliant implementation can't, since it would be ignoring a
SHOULD-level requirement.

I think it makes a lot of sense to restrict the names of extension-headers
to be disjoint from at least the set of general-, request-, and response-
headers defined in the specification. Then an unconditionally-compliant
implementation could behave the way you describe.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 21 May 2009 17:14:18 +0200

Bjoern Hoehrmann wrote:
> ...

I think this is on purpose; Roy has pointed out frequently that HTTP/1.1 
doesn't have an extension point for this.

Now, should we / can we fix this? Dunno.

In practice, new request/response headers are defined all the time, and 
people minting them just need to deal with the fact that conforming 
HTTP/1.1 implementations may treat them as entity headers.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 21 May 2009 17:23:04 +0200

> this interacts with the rules for Content-* header fields though.

s/Part 6/Part 3/.

> to be treated as an extension header. For example:

As far as I can tell, this is a left-over from RFC2616 that we're 
planning to address later on. The ABNF as currently written references 
all the micro-ABNFs for the individual headers (well, those defined in 
RFC2616).

What it should do is just define the generic parsing of header lines 
(what is a legal line? If it is legal, what is the header name and what 
is the field value?), and then refer to each "value" ABNF for the 
individual headers (that's why, in draft-05, we already split the header 
ABNFs into "Fieldname" and "Fieldname-v").

BR, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 21 May 2009 17:38:52 +0200

>equivalent to the original.

I disagree.

>implementation could behave the way you describe.

They can do so already, because the requirement is for unrecognized
headers. If an implementation generates an error because it suspects
a header value is malformed, then it does recognize the header. And
merely noting the issue would happen outside the realm of the proto-
col; the specification does not regulate what may or may not be put
into a web server log file, for example.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 21 May 2009 13:00:13 -0500

Julian Reschke wrote:
> individual headers (that's why, in draft-05, we already split the
> header ABNFs into "Fieldname" and "Fieldname-v").

That makes a lot of sense. It means there won't be a complete ABNF grammar
for HTTP messages, but it does mean that the specification will match what
implementations actually do.

(See more comments below.)

Julian Reschke wrote:
> Bjoern Hoehrmann wrote:
> > In RFC 2616 and the current drafts the extension-header is used
> > only in the entity-header production, suggesting all headers not
> > defined by it are entity headers, which I don't think is intended.
> > Perhaps it should be referenced in Request and Response as
> > alternative to entity-header instead?
> 
> I think this is on purpose; Roy has pointed out frequently that
> HTTP/1.1 doesn't have an extension point for this.
> 

If the change you described above is made, then the grammar won't have
separate general-/response-/request-/entity-/extension- productions, so it
won't matter.

- Brian