Subject: Re: Whitespace before responses

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Feb 2012 11:53:36 +1100

Current text:
"""
   In the interest of robustness, servers SHOULD ignore at least one
   empty line received where a Request-Line is expected.  In other
   words, if the server is reading the protocol stream at the beginning
   of a message and receives a CRLF first, it SHOULD ignore the CRLF.
"""

Proposal:

"""
   In the interest of robustness, servers SHOULD ignore at least one
   empty line received where a Request-Line is expected.  In other
   words, if the server is reading the protocol stream at the beginning
   of a message and receives a CRLF first, it SHOULD ignore the CRLF.

   Likewise, clients SHOULD ignore at least one empty line received
   where a Status-Line is expected. 

   Note that this relaxation does not apply to other characters; ignoring
   arbitrary non-whitespace characters before a message enables
   cross-protocol attacks.
"""

On 08/07/2011, at 6:08 AM, Eric Lawrence wrote:

> If I recall correctly, some browsers apply different rules for leading whitespace/junk based on whether it's a top-level document or not. 
> 
> IE (including IE9) are very liberal about ignoring unexpected junk preceding a HTTP response. For instance, IE will render bolded "Hello World" text when receiving this:
> 
> 
> 	This is not HTTP
> 
> 	
> 	Is ItHTTP/1.1 200 OK
> 	Cache-Control: max-age=0
> 	Content-Type: text/html
> 	Connection: close
> 
> 	<b>Hello world.</b>
> 
> 
> 
> In developing Fiddler, I ran across a few sites delivering HTTP/0.9 responses (e.g. without headers).  Here's an example: http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.htm&r=0&p=1&f=S&l=50&Query=IN%2FEric+Lawrence%0D%0A&d=PTXT which sends no headers, though it does lead the response with \n\n. It seems to render as a HTML document just fine in current versions of all major browsers.
> 
> -Eric 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Nottingham
> Sent: Tuesday, July 05, 2011 4:57 PM
> To: Bjoern Hoehrmann
> Cc: httpbis Group
> Subject: Re: Whitespace before responses
> 
> On 04/07/2011, at 6:56 AM, Bjoern Hoehrmann wrote:
> 
>> * Mark Nottingham wrote:
>>> <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-14#section-3.1> says:
>>> 
>>> In the interest of robustness, servers SHOULD ignore at least one  
>>> empty line received where a Request-Line is expected.  In other  
>>> words, if the server is reading the protocol stream at the beginning  
>>> of a message and receives a CRLF first, it SHOULD ignore the CRLF.
>>> 
>>> Should a similar approach be taken when clients parse responses?
>>> 
>>> See also:
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=668168
>> 
>> Well, we absolutely need to come to an agreement where messages start 
>> and where they end. I would rather ask which behavior we are more 
>> like- ly to agree on. Apart from that it's obviously best if nothing 
>> has to be skipped, as skipping requires more work than not skipping.
> 
>> From a quick test, Firefox and Chrome will skip at least two blank lines (without any whitespace on them); Opera will offer to download it as a file (!). Safari and Curl consider the whitespace and headers as the body (i.e., displaying it), as per HTTP/0.9. I don't have access to IE at the moment, but AIUI it'll operate as FF and Chrome.
> 
> Interestingly, Safari sniffs the response; if it looks like HTML, it will display it as such. I guess HTTP/0.9 by definition requires sniffing.
> 
> Cheers,
> 
> --
> Mark Nottingham   http://www.mnot.net/
> 
> 
> 
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 7 Feb 2012 17:22:35 -0800

>   arbitrary non-whitespace characters before a message enables

No, there is no need nor desire for such a relaxation.  The first rule is
to allow for backwards-compatible behavior with clients that send CRLF at
the end of a request without including it in the request message body count.
This new addition has no corresponding need.  IE is just handling a
message error, which is entirely dependent on the type of client being used.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Feb 2012 12:25:34 +1100

> message error, which is entirely dependent on the type of client being used.

Yeah. I'm on the fence about this one; on the one hand, it's not a hard interop requirement, but on the other, pretty much every client does it, AFAICT.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 8 Feb 2012 08:03:53 +0100

On Wed, Feb 08, 2012 at 12:25:34PM +1100, Mark Nottingham wrote:
> Yeah. I'm on the fence about this one; on the one hand, it's not a hard
> interop requirement, but on the other, pretty much every client does it,
> AFAICT.

And probably that if they do it, it's because some old buggy servers used to
send this CRLF at the end of a response. Eg: a CGI script doing an "echo" after
"cat $file". I don't know how hard it would be to collect statistics on such
bad practices. We'd need to find a commonly deployed client which does not do
it and which confirms there's no issue when not accepting a CRLF in a response.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 09 Feb 2012 17:33:57 +1300

> it and which confirms there's no issue when not accepting a CRLF in a response.

Willy, the rest of the thread is about *request* not response. Squid has 
not accepted response whitespace prefixes in a very long time and bug 
reports about that are very rare.

Squid does skip whitespace at the beginning of the request and in my 
recent experience it seems to be an active problem with scripts and 
certain poular PDF readers.

AYJ

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 9 Feb 2012 08:01:25 +0100

Hi Amos,

On Thu, Feb 09, 2012 at 05:33:57PM +1300, Amos Jeffries wrote:
> Willy, the rest of the thread is about *request* not response.

No, look at the subject :-) The initial text covered only requests
and Mark asked if we should cover responses too.

> Squid has 

Excellent ! In my opinion, Squid certainly qualifies as a reference for
something widely used. So if you don't have the issue, I think it means
the issue does not really exist, so we don't need to skip any CRLF before
a response.

> certain poular PDF readers.

Same here. I've recently received a trace of a request emitted by a browser
announcing webkit in its UA, I believe it was Safari, where a POST request was
sent, then a few tens of milliseconds later, a CRLF was sent in a lone packet
while it was not part of the message-body. So I can confirm this is observed
from time to time. I'll see if I can find the capture so that we get the exact
UA, maybe it was old and has been fixed since.

Regards,
Willy

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 10 Feb 2012 01:47:23 +1300

> and Mark asked if we should cover responses too.

:-( /egg

> the issue does not really exist, so we don't need to skip any CRLF before

I suppose its also worth noting that the particular response code in 
Squid is also violating the RFC by disallowing SP characters before the 
"HTTP/1." string. With same lack of problems. It would be nice to drop 
that requirement on the response line out of the spec if it is still there.

> UA, maybe it was old and has been fixed since.

Shockwave Flash has one report here 
http://bugs.squid-cache.org/show_bug.cgi?id=2829. Not sure if that was 
Flash itself or a badly coded app.

I only encountered PDF in help forum discussions, so no idea which is 
was exactly.

AYJ

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 9 Feb 2012 13:54:30 +0100

> that requirement on the response line out of the spec if it is still there.

I have not seen this requirement in 2616. Haproxy doesn't allow this either.

Regards,
Willy

---

From: Patrick McManus <[email-masked]>
Date: Thu, 09 Feb 2012 08:55:10 -0500

I definitely see whitespaces before responses now and again and tolerate
them as best as I can. Messing with that in a bis standardization update
seems to be an over reach. I find it hard to believe a document change
would change any deployed behavior (as it introduces breakage without
upside in a legacy protocol) - so the spec would just be less accurate
which is pretty much the opposite of the bis charter goal.

this embarrassing nit is, however, a good example of why http/2 needs to
be binary framed. Too much energy is expended on the commodity activity
of parsing headers the current way.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 9 Feb 2012 15:01:29 +0100

On Thu, Feb 09, 2012 at 08:55:10AM -0500, Patrick McManus wrote:
> seems to be an over reach.

OK, thanks for the feedback.

> I find it hard to believe a document change

It's not that much a document change since it's already not allowed. Either
we acknowledge that a number of tools don't accept them and don't have any
issue and the text can remain as is, or we acknowledge that some implementers
had to relax the rule so we'd need to clarify this in the spec. Right now
you're saying that you had to implement something that is not covered by the
spec, and *this* is embarrassing. I must say I did this in haproxy without
realizing that the spec only allowed the leading CRLF in the request message.

> of parsing headers the current way.

While binary framing can improve this, it is also much more complicate to
debug and test by hand. However strict fixed-length elements could be a
good improvement.

Regards,
Willy

---

From: Zhong Yu <[email-masked]>
Date: Thu, 9 Feb 2012 08:53:18 -0600

On Wed, Feb 8, 2012 at 10:33 PM, Amos Jeffries <[email-masked]> wrote:
> Squid does skip whitespace at the beginning of the request and in my recent
> experience it seems to be an active problem with scripts and certain poular
> PDF readers.

Do you mean arbitrary whitespaces before the request method, not just one CRLF?

Zhong

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 10 Feb 2012 04:06:55 +1300

On 10/02/2012 3:53 a.m., Zhong Yu wrote:
> On Wed, Feb 8, 2012 at 10:33 PM, Amos Jeffries<[email-masked]>  wrote:

series of CRLF before the first line or series of SP before the method. 
That is all right now.

AYJ