Subject: Header field-name token and leading spaces

From: Karl Dubost <[email-masked]>
Date: Sun, 3 Mar 2013 14:03:56 -0500

Hi,

This is a bit long but it came because we were trying to fix a bug into python library for HTTP headers and production rules.

request.add_header('foo', 'bar')
→ "foo:bar"
request.add_header(' foo', 'bar')
→ " foo:bar"
request.add_header('foo ', 'bar')
→ "foo :bar"

What I gathered from the spec:

In 3.2.  Header Fields, 
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#section-3.2 

the header production rules are defined as:

     header-field   = field-name ":" OWS field-value BWS
     field-name     = token
     field-value    = *( field-content / obs-fold )
     field-content  = *( HTAB / SP / VCHAR / obs-text )
     obs-fold       = CRLF ( SP / HTAB )
                    ; obsolete line folding
                    ; see Section 3.2.4

   The field-name token labels the corresponding field-value as having
   the semantics defined by that header field.

So far so good, but we do not know what are the production rules for "field-name     = token". It might come later. Let's read a bit more. 

In 3.2.3, Whitespace, there are production rules for OWS, BWS, and RWS:
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#section-3.2.3

This defines at least the rules for 

     header-field   = field-name ":" OWS field-value BWS

which says basically.

--------------------------------------------------
OK    "Foo: bar"       (1 space or more)
OK    "Foo:	bar"   (1 tab or more)
OK    "Foo:bar"        (no space)
--------------------------------------------------
AVOID "Foo: bar "      (1 trailing space or more)
AVOID "Foo: bar "      (1 trailing tab or more)
--------------------------------------------------

AVOID means:

* senders SHOULD NOT generate it in messages.
* recipients MUST accept such bad optional whitespace and remove it
  before interpreting the field value or forwarding the message
  downstream.

ok cool. Let's go on.

In 3.2.4 Field Parsing
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#section-3.2.4

   No whitespace is allowed between the header field-name and colon.  

So In production rules, we can't do:

-------------------------------------------------------
BAD  "Foo :bar"    (1 or more space/tab before the ":")
-------------------------------------------------------

   In
   the past, differences in the handling of such whitespace have led to
   security vulnerabilities in request routing and response handling.  A
   server MUST reject any received request message that contains
   whitespace between a header field-name and colon with a response code
   of 400 (Bad Request).

OK. This is clear too. Tested on W3C Server, 

→ curl -I -H "foo :bar" --trace-ascii - http://www.w3.org/

W3C server sent back 

    HTTP/1.0 400 Bad request

Though not all servers do that:

→ curl -I -H "foo :bar" --trace-ascii - http://www.ietf.org/
HTTP/1.1 200 OK

There is a rule also for proxies:

  A proxy MUST remove any such whitespace from a
   response message before forwarding the message downstream.

-------------------------------------------------------
"foo :bar" → "foo:bar"
-------------------------------------------------------

MY QUESTION (finally) :) 

Nothing is said about 
-------------------------------------------------------
" foo:bar"   (1 or more space/tab before the fied-name)
-------------------------------------------------------

In appendix C, the ABNF defines token for:
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#appendix-C

The section of the spec saying

     field-name     = token

with

   token = 1*tchar

and tchar as 

   tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
    "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA

So the production rules forbid a leading space, but nothing is said about parsing this leading space. 

* Should it say something? 
* If yes, what? 
* If not, why?

-- 
Karl Dubost
http://www.la-grange.net/karl/

---

From: Julian Reschke <[email-masked]>
Date: Sun, 03 Mar 2013 20:11:36 +0100

On 2013-03-03 20:03, Karl Dubost wrote:
> ...
> ...

It's not a legal field name, thus not a legal start of a header field line.

> * Should it say something?

It's forbidden by the grammar, so it's invalid.

Best regards, Julian

---

From: Jeff Pinner <[email-masked]>
Date: Sun, 3 Mar 2013 11:17:06 -0800

The production rules forbid a leading space because you can't
disambiguate it from (obsolete) line folding. Nothing is said about
parsing it because as long as the leading space isn't on the first
header sent, the parser assumes its part of the previous header value.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 03 Mar 2013 19:21:25 +0000

In message <[email-masked]>, Karl Dubost wr

>Nothing is said about 

Actually, something is said:  It is a continuation line for
the previous header.

Unless, of course, it happens to be the second line, in which
case it is just plain illegal.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Karl Dubost <[email-masked]>
Date: Sun, 3 Mar 2013 17:57:01 -0500

> Actually, something is said:  It is a continuation line for

Ah indeed.

ok. cool. So it should be forbidden in production. It is not the intent of the author with:

Cool Thanks!