Subject: Bad browser behaviour?

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 20 Mar 2012 02:20:42 +0000

Hi all
  
we're seeing some (IMO) undesirable behaviour for all tested current 
browsers (we tested FF, Chrome, IE and Opera).
  
It relates to abortive closes on chunked transfers.  In this case, I'm 
talking about a server close prior to the final 0 chunk.
  
All the browsers we tested ignore this and display the content with no 
warning whatsoever.
  
For our proxy to treat it as an abortive close is therefore a problem 
in our customers' eyes.
  
So what's the deal?  Should we allow this behaviour in the spec?  Or 
should browser vendors be encouraged to break the page / download?
  
Isn't it a potential security issue?

Adrien

---

From: Adam Barth <[email-masked]>
Date: Mon, 19 Mar 2012 22:02:34 -0700

It's probably impossible for browsers to do anything else given that
browsers incrementally render chunk-transfered content.  For example,
if the network were to hang at that point (rather than drop), they'd
do the same thing.

Adam

On Mon, Mar 19, 2012 at 7:20 PM, Adrien W. de Croy <[email-masked]> wrote:
> For our proxy to treat it as an abortive close is therefore a problem in our
> customers' eyes.
>
> So what's the deal?  Should we allow this behaviour in the spec?  Or should
> browser vendors be encouraged to break the page / download?
>

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 20 Mar 2012 05:16:04 +0000

what about pop a warning, or clear the page?

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 20 Mar 2012 18:30:22 +1300

> what about pop a warning, or clear the page?

or a range request to fetch the missing data and continue to render?

AYJ

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 20 Mar 2012 18:29:41 +1300

> Isn't it a potential security issue?

Why is it a problem? Abortive close does not necessarily mean the sky is 
falling. It just means incomplete transfer. Happens all the time.

Chunked transfer just offers the possibility of automatic recovery by 
the client UA by informing that it *was* incomplete/aborted instead of 
complete/terminated. The proxy should be relaying that information along 
with an early abort on the client connection.

AYJ

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 20 Mar 2012 05:33:39 +0000

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 20/03/2012 6:30:22 p.m.
Subject: Re: Bad browser behaviour?
>or a range request to fetch the missing data and continue to render? 
  
in our case, wouldn't work, but in some it could I guess.  In our case 
there's a broken server that doesn't send 0 chunks, but just closes.
  
Adrien

---

From: Adam Barth <[email-masked]>
Date: Mon, 19 Mar 2012 22:34:44 -0700

I suspect these things would just annoy users with unreliable network
connections.  Wouldn't you be frustrated if a web page you were
looking at suddenly went blank just because your WiFi cut out?

Adam

On Mon, Mar 19, 2012 at 10:16 PM, Adrien W. de Croy <[email-masked]> wrote:
>>> should

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 20 Mar 2012 05:36:48 +0000

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 20/03/2012 6:29:41 p.m.
Subject: Re: Bad browser behaviour?
>>All the browsers we tested ignore this and display the content with 
>>no warning whatsoever. 
>Why is it a problem? Abortive close does not necessarily mean the sky 
>is falling. It just means incomplete transfer. Happens all the time. 
several problem scenarios.
  
1. intermediary with AV scanning which can only do an abortive close if 
it detects a virus since it already drip-fed a portion of the resource 
to the client to stop it timing out.  Client can receive an infected 
file and execute it without any warning.
  
2. intermediary spooling content for scanning (without drip-feeding any 
to client) aborts without sending anything to client.

We could fix 2. by making the intermediary just pass whatever it had on 
an abortive close from the server, but if it was chunking, should it 
send a 0 chunk or just close?
  
>complete/terminated. The proxy should be relaying that information 
>along with an early abort on the client connection. 
many chunking scenarios the server doesn't have a length, which is why 
it was chunked in the first place, so range won't work.
  
Adrien

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 20 Mar 2012 05:37:26 +0000

OK, so what we're saying is that the 0 chunk is basically redundant.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 20 Mar 2012 05:39:58 +0000

------ Original Message ------
From: "Adam Barth" <[email-masked]>
To: "Adrien W. de Croy" <[email-masked]>
Cc: "HTTP Working Group" <[email-masked]>
Sent: 20/03/2012 6:34:44 p.m.
Subject: Re: Re[2]: Bad browser behaviour?
>looking at suddenly went blank just because your WiFi cut out?
  
do you receive a TCP RST from the peer when your Wifi cuts out?

Adrien

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 20 Mar 2012 16:40:45 +1100

It might be worth differentiating between the download use case and normal browsing. 

Also, in both cases, it would be good to highlight incomplete transfers in the console.

Note that all of this really isn't part of HTTP, per se, but rather how browsers choose to use it.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Adam Barth <[email-masked]>
Date: Mon, 19 Mar 2012 22:53:55 -0700

>  OK, so what we're saying is that the 0 chunk is basically redundant.

What if you want to send another response?  Don't you need to
terminate the first response somehow?

> It might be worth differentiating between the download use case and normal browsing.

Yeah, there it makes sense not to truncate the response since there
isn't any incremental processing of the response.

Adam

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 20 Mar 2012 21:19:03 +1300

On 20/03/2012 6:53 p.m., Adam Barth wrote:
> On Mon, Mar 19, 2012 at 10:37 PM, Adrien W. de Croy<[email-masked]>  wrote:

ok, I'll re-phrase.  The 0 chunk becomes expendable if the server closes.

this isn't much different to the case where there's no content length 
and the server closes without chunking.

IMO we should use the information we get though.

Other sorts of connection closure (e.g. timeout, no route to dest etc) 
may indicate another sort of handling is appropriate.

Adrien

>
> On Mon, Mar 19, 2012 at 10:40 PM, Mark Nottingham<[email-masked]>  wrote:

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 21 Mar 2012 02:50:10 +1300

On 20/03/2012 9:19 p.m., Adrien de Croy wrote:
>> On Mon, Mar 19, 2012 at 10:37 PM, Adrien W. de Croy wrote:

Only to the browser and only iff the connected server or proxy wanted 
connection:close anyway.
All the middleware will be trying to use connections more efficiently 
and use pipelines. That all depends on 0-chunk termination to keep 
connections open and requests flowing.

> and the server closes without chunking.

Exactly the same case AFAICT. The fact the server is actually sending 
all the response bytes is a coincidence.

AYJ

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 21 Mar 2012 02:55:02 +1300

> do you receive a TCP RST from the peer when your Wifi cuts out?

Possibly ye. On a TCP timeout after wifi outage as the wifi devices 
stack dumps connections.

AYJ

---

From: "Robert Brewer" <[email-masked]>
Date: Tue, 20 Mar 2012 08:35:05 -0700

> IMO we should use the information we get though.

This is a serious problem at my current company, where we regularly
stream large CSV output of survey results to clients. If something goes
wrong on the server after the response headers have been written but
before the complete dataset has been written out, then it has been
considered proper to simply end the connection on the server side (what
else could it do? too late to change the response status).
Unfortunately, most browsers don't warn the user that they have received
a partial dataset, and it's only perhaps days later that someone notices
their analyses are way off the expected results. Sometimes they don't
notice at all, which can become a serious issue (think: clinical studies
informing public policy; won't somebody please think of the children?).
IMO therefore, all clients should be encouraged to fail loudly if the
connection is closed by the server before a 0 chunk is reached, since
there is no other mechanism for a server to communicate that there has
been an error (at least not a generic one; in our case of a CSV file,
and assuming the error occurs in the server's application layer and can
be trapped, we could try to emit a plain text warning or garbage bytes;
however, this is not a good strategy for all media types, nor does it
address errors in lower-level code).

Robert Brewer
[email-masked]

---

From: Eric Lawrence <[email-masked]>
Date: Tue, 20 Mar 2012 17:22:15 +0000

Most browsers don't warn on Content-Length underruns, and Chunked Encoding typically has the same behavior.

We have seen some real-world scenarios where a server using Chunked never sends the trailing 0CRLFCRLF and simply closes the connection. There are two scenarios of interest when considering the premature closure-- 1> where the last block was of the promised size (e.g. server promised n bytes and sent n) and 2> where the last block was shorter than the promised size when the connection was closed. The latter is obviously somewhat more "suspicious" than the former.

-----Original Message-----
From: Robert Brewer [mailto:[email-masked]] 
Sent: Tuesday, March 20, 2012 8:35 AM
To: Adrien de Croy; Adam Barth
Cc: HTTP Working Group
Subject: RE: Bad browser behaviour?

> IMO we should use the information we get though.

This is a serious problem at my current company, where we regularly stream large CSV output of survey results to clients. If something goes wrong on the server after the response headers have been written but before the complete dataset has been written out, then it has been considered proper to simply end the connection on the server side (what else could it do? too late to change the response status).
Unfortunately, most browsers don't warn the user that they have received a partial dataset, and it's only perhaps days later that someone notices their analyses are way off the expected results. Sometimes they don't notice at all, which can become a serious issue (think: clinical studies informing public policy; won't somebody please think of the children?).
IMO therefore, all clients should be encouraged to fail loudly if the connection is closed by the server before a 0 chunk is reached, since there is no other mechanism for a server to communicate that there has been an error (at least not a generic one; in our case of a CSV file, and assuming the error occurs in the server's application layer and can be trapped, we could try to emit a plain text warning or garbage bytes; however, this is not a good strategy for all media types, nor does it address errors in lower-level code).

Robert Brewer
[email-masked]

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 20 Mar 2012 20:16:32 +0100

Hi,

On Tue, Mar 20, 2012 at 05:22:15PM +0000, Eric Lawrence wrote:
> We have seen some real-world scenarios where a server using Chunked never
> sends the trailing 0CRLFCRLF and simply closes the connection. There are two
> scenarios of interest when considering the premature closure-- 1> where the
> last block was of the promised size (e.g. server promised n bytes and sent n)
> and 2> where the last block was shorter than the promised size when the
> connection was closed. The latter is obviously somewhat more "suspicious"
> than the former.

Indeed, I was wondering how the client would react if before closing, the
AV gateway would advertise a large chunk and then close. Alternatively, if
the client supports Trailers, it would be possible to emit only garbage to
fill the last chunk or the content-length and then emit a trailer indicating
that the content has failed validation. But for this, clients have to be
adapted.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 20 Mar 2012 20:30:32 +0100

On 2012-03-20 20:16, Willy Tarreau wrote:
> ...
> ...

We should add this use case (indicating a server error while streaming 
content) to the HTTP/2.0 wish list.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 20 Mar 2012 13:39:16 -0700

> informing public policy; won't somebody please think of the children?).

Yes, this is in fact a well-known bug in some browsers.  That is why the
HTTPbis spec has the following REQUIREMENT in p1 sec 3.4:

   A user agent MUST NOT render an incomplete response message body
   as if it were complete (i.e., some indication must be given to the
   user that an error occurred).

We don't require a specific UI, but it should be obvious to developers
why silent truncation of a page can be life-threatening in many contexts
where browser-based applications are in use today.  It is not difficult
to add the equivalent of a broken-image icon (or an off-color line with
a tooltip, or spoken phrase, or a fail whale, or whatever might be an
appropriate error stream for a command-line tool) at the end
of a truncated page or embedded element.

Cheers,

Roy T. Fielding                     <http://roy.gbiv.com/>
Principal Scientist, Adobe Systems  <http://adobe.com/enterprise>