Subject: line folding - ABNF vs prose

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Jul 2011 15:49:35 +0200

<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-15.html#rfc.section.3.2.p.8> 
says:

"Historically, HTTP header field values could be extended over multiple 
lines by preceding each extra line with at least one space or horizontal 
tab octet (line folding). This specification deprecates such line 
folding except within the message/http media type (Section 10.3.1). 
HTTP/1.1 senders MUST NOT produce messages that include line folding 
(i.e., that contain any field-content that matches the obs-fold rule) 
unless the message is intended for packaging within the message/http 
media type. HTTP/1.1 recipients SHOULD accept line folding and replace 
any embedded obs-fold whitespace with a single SP prior to interpreting 
the field value or forwarding the message downstream."

I think we have mismatch between ABNF and prose here. Consider

"HTTP/1.1 senders MUST NOT produce messages that include line folding 
(i.e., that contain any field-content that matches the obs-fold rule)...."

..but..

   OWS            = *( [ obs-fold ] WSP )
                  ; "optional" whitespace
   obs-fold       = CRLF

...which is ok-ish, because the only way to get obs-fold into the field 
is to have trailing WSP.

But...

"HTTP/1.1 recipients SHOULD accept line folding and replace any embedded 
obs-fold whitespace with a single SP..."

indicates that a sequence of

   obs-fold WSP

should be replaced by

   SP WSP

Wasn't the intention to replace the sequence by a single SP?

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 27 Jul 2011 16:07:44 +0200

Hi Julian,

> Wasn't the intention to replace the sequence by a single SP?

Replacing with a single SP is harder than replacing with any number of
spaces. For instance, haproxy replaces the CRLF with a comma followed
by a space so that the line is folded without moving a memory block,
and it works quite well.

Maybe the intention was to replace with a single SP but the result is
still valid with multiple spaces, so maybe we should explicitly allow
them ? Anyway good catch on this impact on the ABNF !

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Jul 2011 16:19:16 +0200

On 2011-07-27 16:07, Willy Tarreau wrote:
> ...

That might be ok betweek words (tokens/quoted-strings), but might be a 
problem inside quoted-string...

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 27 Jul 2011 16:59:28 +0200

> problem inside quoted-string...

But I thought we were only allowed to fold lines for headers which are
defined as comma-separated lists, and to add a comma. So by definition
we should not do that on a header that is not a list, and since a header
which is not a list must not appear multiple times, this should not be
a problem.

Am I wrong anywhere ?

Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 27 Jul 2011 08:20:58 -0700

> which is not a list must not appear multiple times, this should not be

No, (un)folding long field values has nothing to do with combining multiple
header fields of the same name into a single value.

In any case, I don't think we should worry about extra spaces breaking a
quoted string that has been folded on whitespace.  HTTP does not have
semantically meaningful long quoted strings and normal clients do not
perform any folding (only test or evil clients).  What is important is
that recipients do not treat a folded line as an empty line indicating
the end of headers.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 27 Jul 2011 17:42:02 +0200

> that recipients do not treat a folded line as an empty line indicating

... or don't consider that a line beginning with a space followed by some
text and a colon is a header whose name starts with spaces. I've already
seen this :

  GET / HTTP/1.1
  Host: localhost
  Cookie: foo1="a:b", foo2="c:d",
    foo3="e:f", foo4="g:h"
  Connection: close

And the server considered that the string <  foo3=e"> was the name of a
header, with <f"> the first value, and <foo4="g:h"> the second value.
Unfortunately I don't remember which server it was.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 28 Jul 2011 00:02:45 +0200

> header fields of the same name into a single value.

Sorry Roy, I said totally stupid things here, I was mixing two things,
multiple header procesing and multiple lines. No of course I'm not adding
the comma here. I'm just replacing all LWS with SP, so basically we get at
least 2 SP in the result (in case of pure LF+SP) or 3 SP for the common
case (CRLF+SP).

Sorry for the noise.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 28 Jul 2011 11:35:44 +1200

> least 2 SP in the result (in case of pure LF+SP) or 3 SP for the 

 Thank you, that discussion explains this old bug which we have seen on 
 some networks:

 HTTP/1.1 200 Okay
 Connection: Keep-Alive
 Content-Type: multipart/byteranges; boundary=":gn1, Tl__lXvUYr73ia-"
 Date: Wed, 31 Jul 2010 16:12:17 GMT

 ---:gn1Tl__lXvUYr73ia-
 blah

 The agents involved there have now been fixed not to wrap at 50 octets. 
 But still, boundary may be very long and there are other cases like this 
 out there. Particularly amongst long quoted-string custom headers.
   http://bugs.squid-cache.org/show_bug.cgi?id=3061

 AYJ