Subject: Status of issue #30 (Implied LWS)

From: Mark Nottingham <[email-masked]>
Date: Thu, 13 Nov 2008 16:45:08 -0800

Based on list discussion and that in Dublin, the plan of record for  
closing issue #30 <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/ 
30> is:

   * Where it occurs, make implicit LWS in the BNF explicit
   * When doing so, discriminate between LWS that is allowed and "bad"  
or discouraged
   * Explicitly disallow LWS between header field-names and ":"

To this end, the editors have created a number of new BNF productions  
and associated text:

--->8---
All linear white space (LWS) in header field-values has the same  
semantics as SP. A recipient may replace any such linear white space  
with a single SP before interpreting the field value or forwarding the  
message downstream.
Historically, HTTP/1.1 header field values allow linear white space  
folding across multiple lines. However, this specification deprecates  
its use; senders must not produce messages that include LWS folding  
(i.e., use the obs-fold rule), except within the message/http media  
type (Section 9.3.1). Receivers should still parse folded linear white  
space.

This specification uses three rules to denote the use of linear white  
space; BWS ("Bad" White Space), OWS (Optional White Space), and RWS  
(Required White Space).

"Bad" white space is allowed by the BNF, but senders should not  
produce it in messages. Receivers must accept it in incoming messages.

Required white space is used when at least one linear white space  
character is required to separate field tokens. In all such cases, a  
single SP character should be used.

OWS            = *( [ obs-fold ] WSP )    ; "optional" white space
RWS            = 1*( [ obs-fold ] WSP )   ; "required" white space
BWS            = OWS                      ; "bad" white space
obs-fold       = CRLF
--->8---

The editors plan to have a draft incorporating these changes (as well  
as other ABNF-related changes) soon.

--
Mark Nottingham     http://www.mnot.net/

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 14 Nov 2008 02:03:09 +0000

> BWS            = OWS                      ; "bad" white space

That syntax permits header lines containing only whitespace, like this:

    Field:<SP>value<CRLF>
    <SP><CRLF>
    <SP>more value<CRLF>

Is this intentional?  (RFC2616 allows it.)

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 13 Nov 2008 18:06:03 -0800

Yes; we looked at disallowing it, but implementations that support  
folding do already support whitespace-only lines. We don't want to  
make them non-conforming. Also, it made the ABNF really, really ugly.  
Really.

We're considering discouraging producing all-whitespace continuation  
lines in prose. Thoughts?

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 14 Nov 2008 02:15:31 +0000

> lines in prose. Thoughts?

This seems very sensible.

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 14 Nov 2008 02:59:33 +0000

> lines in prose. Thoughts?

All continuation lines are discouraged; maybe that could be worded a
little more strongly.  With the word "discouraged" :-)

-- Jamie

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 13 Nov 2008 23:02:54 -0600

> little more strongly.  With the word "discouraged" :-)

Mark already said "[S]enders must not produce messages that include LWS
folding[.]" It would be confusing to make LWS folding a MUST NOT and then
make a special case of it "discouraged." 

- Brian

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 14 Nov 2008 13:34:11 +0000

> make a special case of it "discouraged."

Ok, make sure to capitalise the MUST NOT and I'm happy :-)

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 14 Nov 2008 23:02:38 +0100

On tor, 2008-11-13 at 18:06 -0800, Mark Nottingham wrote:
> folding do already support whitespace-only lines.

Some. Many fail, misreading it as end-of-headers...

> We don't want to  
> make them non-conforming.

It only needs to be disallowed in producing, not parsing. In fact
parsers SHOULD know how to deal with it I think.

> lines in prose. Thoughts?

A MUST NOT be sent in prose is the right approach I think.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 14 Nov 2008 22:27:58 +0000

> Some. Many fail, misreading it as end-of-headers...

Last time I looked, I think Mozilla was in that category.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 14 Nov 2008 23:33:37 +0100

> Last time I looked, I think Mozilla was in that category.

Still?

There was a security whitepaper on this some years ago which made a lot
of people jump.. (or actually two with about a year inbetween, one
looking at responses, one at requests)

Regards
Henrik

---

From: Amit Klein <[email-masked]>
Date: Sat, 15 Nov 2008 00:54:19 +0200

> looking at responses, one at requests)

Yes, that was me ;-)

2004 - HTTP Response Splitting: 
http://packetstormsecurity.org/papers/general/whitepaper_httpresponse.pdf
2005 - HTTP Request Smuggling: 
http://www.cgisecurity.com/lib/HTTP-Request-Smuggling.pdf

---

From: Amit Klein <[email-masked]>
Date: Sat, 15 Nov 2008 01:31:52 +0200

Amit Klein wrote:
>>>>> Yes; we looked at disallowing it, but implementations that 
>>>>> support  folding do already support whitespace-only lines.
>>>>>         

The "HTTP Request Smuggling" paper is actually the relevant one. I think 
I made a note about this earlier.

-Amit

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 15 Nov 2008 14:01:27 -0800

Yes, all requirements language in the proposal is normative (i.e.,  
uppercase it). Sorry for the confusion.