Subject: Unknown and misplaced headers as entity headers

From: "Brian Smith" <[email-masked]>
Date: Tue, 26 Feb 2008 17:49:24 -0800

The specification repeatedly says that "unrecognized header fields are treated as
entity-header fields": [Part 1, section 4.5], [Part 2, section 4], [Part 2, section 6], [Part 3, section 4.1]. However, it also says that unrecognized header fields MAY be treated as response and/or request headers. 

The BNF for extension-header is "extension-header = message-header".

Here are several issues:

* In practice, applications cannot usefully classify an unrecognized header, and they definitely do not treat them as entity headers. Declaring them to be entity headers adds confusion with no benefit.

* The BNF for extension-header matches all other header types. That means that when a client includes a response-header in a request, or a server includes a request-header in a response, the header is to be treated as an entity header. That doesn't make any sense. 

PROPOSAL:

* Remove the statements that say that unrecognized header fields are treated as entity-header fields.

* Move the extension-header definition to Part 1, and replace the ABNF for Request, Response, and trailer-part:

  Request       = Request-Line            ; Section 5.1
                  *(( general-header      ; Section 4.5
                    | request-header      ; [Part2], Section 4
                    | entity-header       ; [Part3], Section 4.1
                    | extension-header) CRLF)
                  CRLF
                  [ message-body ]        ; Section 4.3

  Response      = Status-Line             ; Section 6.1
                *(( general-header        ; Section 4.5
                  | response-header       ; [Part2], Section 6
                  | entity-header         ; [Part3], Section 4.1
			| extension-header) CRLF)  
                 CRLF
                 [ message-body ]         ; Section 4.3

  trailer-part   = *((entity-header
                     |extension-header CRLF))

As I mentioned before, I would also like to see some way of using the Accept-* headers in a "415 Unsupported Media Type" response, so that the server can indicate which media types, languages, encodings, etc. it would be willing to accept. After reviewing the charter, it seems like this would be out of scope for httpbis. If so, httpbis should at least leave the door open for it (that is, do not preclude the use of request-headers as response-headers).

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Feb 2008 12:06:16 +0100

> entity-header fields": [Part 1, section 4.5], [Part 2, section 4], [Part 2, section 6], [Part 3, section 4.1]. However, it also says that unrecognized header fields MAY be treated as response and/o…

Yes. As in, for example, 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-02.html#rfc.section.4.p.3>:

"...Unrecognized header fields are treated as entity-header fields.".

> * In practice, applications cannot usefully classify an unrecognized header, and they definitely do not treat them as entity headers. Declaring them to be entity headers adds confusion with no bene…

I agree that we have a conflict between what RFC2616 says, and what many 
(most?) servers do. However I'm not sure there's a simple way to fix this.

> * The BNF for extension-header matches all other header types. That means that when a client includes a response-header in a request, or a server includes a request-header in a response, the header…

The BNF for extension header, given in 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-02.html#rfc.section.4.2>:

   message-header = field-name ":" [ field-value ]
   field-name     = token
   field-value    = *( field-content | LWS )
   field-content  = <field content>
                    ; the OCTETs making up the field-value
                    ; and consisting of either *TEXT or combinations
                    ; of token, separators, and quoted-string

just defines the allowable syntax. We probably could state in prose that 
the value must be distinct from all other header names defined in the 
spec(s). Would that help?

> * Remove the statements that say that unrecognized header fields are treated as entity-header fields.

I'd like to see this opened as a separate issue.

>                   [ message-body ]        ; Section 4.3

I'm not sure how this is an improvement.

Right now we are stating that a header in a requets is either a 
general-header, a request-header, or an entity-header. That distinction 
is lost if we allow an additional "unclassified" type.

If what you're looking for is the BNF to allow new, for instance, 
general header fields, we should extend the BNF for general-header.

> ...

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 27 Feb 2008 05:17:54 -0800

Julian Reschke wrote:
> > * In practice, applications cannot usefully classify an 
> > unrecognized header, and they definitely do not treat them as 
> > entity headers. Declaring them to be entity headers adds 
> > confusion with no benefit.
> 
> I agree that we have a conflict between what RFC2616 says, 
> and what many (most?) servers do. However I'm not sure there's
> a simple way to fix this.
> 
> > * The BNF for extension-header matches all other header 
> types. That means that when a client includes a 
> response-header in a request, or a server includes a 
> request-header in a response, the header is to be treated as 
> an entity header. That doesn't make any sense. 
> 
> The BNF for extension header [...]
> just defines the allowable syntax. We probably could state in 
> prose that the value must be distinct from all other header 
> names defined in the spec(s). Would that help?

As much as possible, the ABNF completely define the syntax, so that a parser can be generated from the ABNF. Also, there is no reason for response-header names and request-header names to be disjoint, and trying to force them to be disjoint creates its own real problems.

> Right now we are stating that a header in a requets is either 
> a general-header, a request-header, or an entity-header. That 
> distinction is lost if we allow an additional "unclassified" type.

What problem does this cause? When an HTTP application encounters an unknown header, that header would still be a request-header, response-header, entity-header, or general-header. The only difference would be that the specification doesn't try to dictate how the application classifies it. 

> If what you're looking for is the BNF to allow new, for 
> instance, general header fields, we should extend the BNF for 
> general-header.

If you make extension-header an alternative of general-header then the BNF would match every header as a general-header. Otherwise, I would have suggested to make extension-header an alternative for every header production.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Feb 2008 14:28:33 +0100

> As much as possible, the ABNF completely define the syntax, so that a parser can be generated from the ABNF. Also, there is no reason for response-header names and request-header names to be disjoi…

Well, that's probably not possible to do in ABNF. How would you write a 
rule that excludes specific names?

I didn't want to imply that request and response headers need to be 
disjoint. But I think that the sets of generic headers, request+response 
headers, and entity header should be disjoint.

> What problem does this cause? When an HTTP application encounters an unknown header, that header would still be a request-header, response-header, entity-header, or general-header. The only differe…

But it's the prose, not the BNF rule, doing that.

> If you make extension-header an alternative of general-header then the BNF would match every header as a general-header. Otherwise, I would have suggested to make extension-header an alternative fo…

Well, yes. If we say that in specific cases new general headers *can* be 
introduced, the BNF rule should reflect that.

And as there is no way to decide the nature of an unknown header, the 
BNF is not going to help here.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 27 Feb 2008 05:48:31 -0800

Julian Reschke wrote:
> Brian Smith wrote:
> > As much as possible, the ABNF completely define the syntax, 
> so that a parser can be generated from the ABNF. Also, there 
> is no reason for response-header names and request-header 
> names to be disjoint, and trying to force them to be disjoint 
> creates its own real problems.
> 
> Well, that's probably not possible to do in ABNF. How would 
> you write a rule that excludes specific names?

RFC 3986 uses a "first-match-wins" disambiguation rule. If the input matches more than one alternative, then it is matched by the first matching alternative. So, in the BNF I gave, using a similar rule, a header in a request would only match extension-header if it did not match general-header, request-header, or entity-header.

> What problem does this cause? When an HTTP application 
> encounters an unknown header, that header would still be a 
> request-header, response-header, entity-header, or 
> general-header. The only difference would be that the 
> specification doesn't try to dictate how the application 
> classifies it. 
> 

Right, but you don't have to know the classification of an extension-header to parse it.

> > If you make extension-header an alternative of 
> general-header then the BNF would match every header as a 
> general-header. Otherwise, I would have suggested to make 
> extension-header an alternative for every header production.
> 
> Well, yes. If we say that in specific cases new general 
> headers *can* be introduced, the BNF rule should reflect that.
>
> And as there is no way to decide the nature of an unknown 
> header, the BNF is not going to help here.

What I meant was, using a "first-match-wins" rule, every header, including request/response/entity-headers, would get parsed as a general header.

If the ABNF is not required to be unambiguous, then I would put extension-header in each of the general-header, request-header, response-header, and entity-header productions, to better match the semantics, since the semantics are ambiguous.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Feb 2008 15:06:21 +0100

> RFC 3986 uses a "first-match-wins" disambiguation rule. If the input matches more than one alternative, then it is matched by the first matching alternative. So, in the BNF I gave, using a similar…

Interesting, thanks for mentioning this.

However, I'm not sure we want to apply that trick here. How would it 
affect code generated based on the ABNF?

> Right, but you don't have to know the classification of an extension-header to parse it.

Well, you don't need anything of this to parse. Just a general rule 
specifying the syntax for headers.

> What I meant was, using a "first-match-wins" rule, every header, including request/response/entity-headers, would get parsed as a general header.

Yes. We wouldn't want that.

> If the ABNF is not required to be unambiguous, then I would put extension-header in each of the general-header, request-header, response-header, and entity-header productions, to better match the s…

I guess the question here is: given the constraints of both ABNF and the 
header extensibility, what exactly is the goal of the ABNF in the spec?

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 27 Feb 2008 10:59:34 -0800

We are going to rewrite the first part anyway to use a generic parsing
algorithm that does not distinguish header field types.  In other words,
don't worry about this yet.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 28 Feb 2008 11:48:03 +1100

This is <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/104>.

Cheers,

On 27/02/2008, at 12:49 PM, Brian Smith wrote:

> entity-header fields": [Part 1, section 4.5], [Part 2, section 4],  
> [Part 2, section 6], [Part 3, section 4.1]. However, it also says  
> that unrecognized header fields MAY be treated as response and/or  
> request headers.
>
> * In practice, applications cannot usefully classify an unrecognized  
> header, and they definitely do not treat them as entity headers.  
> Declaring them to be entity headers adds confusion with no benefit.
>
> * The BNF for extension-header matches all other header types. That  
> means that when a client includes a response-header in a request, or  
> a server includes a request-header in a response, the header is to  
> be treated as an entity header. That doesn't make any sense.
>
> PROPOSAL:
>
> * Move the extension-header definition to Part 1, and replace the  
> ABNF for Request, Response, and trailer-part:
>

--
Mark Nottingham     http://www.mnot.net/