Subject: Question on HTTP 408

From: William Chan (陈智昌) <[email-masked]>
Date: Mon, 2 Jun 2014 17:16:30 -0700

Hello folks,

Here's the relevant text from p2 (
http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.5.7
):

6.5.7 <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.5.7>.
 408 Request Timeout

   The 408 (Request Timeout) status code indicates that the server did
   not receive a complete request message within the time that it was
   prepared to wait.  A server SHOULD send the close connection option
   (Section 6.1 of [Part1
<http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#ref-Part1>])
in the response, since 408 implies that the
   server has decided to close the connection rather than continue
   waiting.  If the client has an outstanding request in transit, the
   client MAY repeat that request on a new connection.

Server operators using HAProxy have filed a bug report with Chromium due to
our lack of support for 408 (
https://code.google.com/p/chromium/issues/detail?id=377581). We're going to
fix that. But HAProxy uses 408s in an intriguing way that violates some of
our architectural assumptions. Namely, HAProxy wants to shut down an idle
connection, it sends a 408 irrespective of whether or not there was a HTTP
request.

Currently, Chromium's connection management code treats this as a server
bug, since when we bind a HTTP transaction to an idle connection, we expect
that the connection does not have any buffered read data (since the
previous transaction, if any, should have completed successfully). However,
there's a race condition here, since the server may have terminated that
idle connection with a 408 HTTP response followed by a FIN.

There's fundamentally a race here, which is crappy and we fix in HTTP/2
with GOAWAY on the connection. My question here is, is it kosher for
HAProxy to be doing this? In retrospect, I feel like it's a pragmatic
decision for them to do so. That said, it feels incredibly ugly for an
HTTP/1.X server to generate responses without a request. It's not
completely clear to me what the intent is in the spec here, because it says
that the 408 means the server did not receive a complete request. If it
only receives a partial request, then that makes sense it could send a
timeout. But if there's no request whatsoever, it's not immediately obvious
to me that the spec says that that is allowed.

I'm curious if any other implementers have thoughts here. Would love to
hear them.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 3 Jun 2014 00:51:19 -0400

On Jun 2, 2014, at 8:16 PM, William Chan (陈智昌) <[email-masked]> wrote:
> 
> But HAProxy uses 408s in an intriguing way that violates some of our architectural assumptions. Namely, HAProxy wants to shut down an idle connection, it sends a 408 irrespective of whether or not there was a HTTP request.

That's a bug (and a complete waste of bytes). In most cases, clients are not expected to read when there is no outstanding request, so that 408 will just sit around the buffers. Even if the client happens to send a request in flight such that it would be ready to read, it is more likely to get a TCP RST than see the 408.

Most clients will assume that a closed connection after idle is due to a timeout. 408 is really for use as an error status in log files, more than anything else, or for communicating an upstream timeout.

...Roy

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 3 Jun 2014 07:36:50 +0200

Hi Roy,

On Tue, Jun 03, 2014 at 12:51:19AM -0400, Roy T. Fielding wrote:
> On Jun 2, 2014, at 8:16 PM, William Chan (?????????) <[email-masked]> wrote:
> > 
> That's a bug (and a complete waste of bytes). In most cases, clients are not
> expected to read when there is no outstanding request, so that 408 will just
> sit around the buffers. Even if the client happens to send a request in
> flight such that it would be ready to read, it is more likely to get a TCP
> RST than see the 408.

Warning, it's not an idle connection, it's when waiting for the *first*
request of that connection. After the first one, when a persistent
connection is idle, it refrains from emitting it because the client is
supposed to retry.

> Most clients will assume that a closed connection after idle is due to a
> timeout.

But when a client sees a closed connection, it has no way to know if the
request was processed or not before the connection was closed. For example,
if the server crashes (classical cause of silent close), the exact same
behaviour will happen. And the client is not allowed to retry a non-
idempotent request then. So here the 408 is the only way to tell the
client that the request was not handled at all and is safe to retry.

> 408 is really for use as an error status in log files, more than
> anything else, or for communicating an upstream timeout.

I don't really agree, the text of the 408 is clearly meant for clients
and even says what they should do in this situation. And it happens
quite a lot over lossy networks such as 3G or WiFi where the time to
send a large request can easily reach multiple seconds under poor
radio condition, then informing the client it can safely retry seems
essential.

Don't get me wrong, I have nothing against leaving the 408 error empty
by default, but I really fear that we'd make the situation worse for
some clients which strictly comply with the spec. Hence this discussion
that William initiated so that we clarify what should be done in this
situation.

Willy

---

From: Michael Sweet <[email-masked]>
Date: Tue, 03 Jun 2014 07:54:18 -0400

William,

This sounds like a pretty obvious bug - a HTTP server only responds when it has received something.  The usual keep-alive timeout of connections is silent (server just closes the connection, with a preceding TLS shutdown for HTTPS).

On Jun 2, 2014, at 8:16 PM, William Chan (陈智昌) <[email-masked]> wrote:

> Hello folks,
> 
> Here's the relevant text from p2 (http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.5.7):
> 
> 
> 
> 6.5.7.  408 Request Timeout
> 
> 
>    (Section 6.1 of [Part1]) in the response, since 408 implies that the
> Server operators using HAProxy have filed a bug report with Chromium due to our lack of support for 408 (https://code.google.com/p/chromium/issues/detail?id=377581). We're going to fix that. But HAProxy uses 408s in an intriguing way that violates some of our architectural assumptions. Namely, HAProxy wants to shut down an idle connection, it sends a 408 irrespective of whether or not there was a HTTP request.
> 
> Currently, Chromium's connection management code treats this as a server bug, since when we bind a HTTP transaction to an idle connection, we expect that the connection does not have any buffered read data (since the previous transaction, if any, should have completed successfully). However, there's a race condition here, since the server may have terminated that idle connection with a 408 HTTP response followed by a FIN.
> 
> There's fundamentally a race here, which is crappy and we fix in HTTP/2 with GOAWAY on the connection. My question here is, is it kosher for HAProxy to be doing this? In retrospect, I feel like it's a pragmatic decision for them to do so. That said, it feels incredibly ugly for an HTTP/1.X server to generate responses without a request. It's not completely clear to me what the intent is in the spec here, because it says that the 408 means the server did not receive a complete request. If it only receives a partial request, then that makes sense it could send a timeout. But if there's no request whatsoever, it's not immediately obvious to me that the spec says that that is allowed.
> 
> I'm curious if any other implementers have thoughts here. Would love to hear them.

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Patrick McManus <[email-masked]>
Date: Tue, 3 Jun 2014 08:20:46 -0400

this 408-reply without a request is odd but common. HA proxy isn't the only
server that does it - though I can't recall off the top of my head which
other one does it. Firefox has had code to cope for quite a while. For
better or for worse its definitely part of the http/1 cruft implementors
need to deal with that http/2 shouldn't be burdened by.

On Tue, Jun 3, 2014 at 7:54 AM, Michael Sweet <[email-masked]> wrote:

> William,
>
> This sounds like a pretty obvious bug - a HTTP server only responds when
> it has received something.  The usual keep-alive timeout of connections is
> silent (server just closes the connection, with a preceding TLS shutdown
> for HTTPS).
>
>
> 6.5.7 <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.5.7>.  408 Request Timeout
>
>    (Section 6.1 of [Part1 <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#ref-Part1>]) in the response, since 408 implies that the
> Server operators using HAProxy have filed a bug report with Chromium due
> to our lack of support for 408 (
> https://code.google.com/p/chromium/issues/detail?id=377581). We're going
> to fix that. But HAProxy uses 408s in an intriguing way that violates some
> of our architectural assumptions. Namely, HAProxy wants to shut down an
> idle connection, it sends a 408 irrespective of whether or not there was a
> HTTP request.
>

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 06:42:43 +0200

Hi Michael,

On Tue, Jun 03, 2014 at 07:54:18AM -0400, Michael Sweet wrote:
> William,
> 
> This sounds like a pretty obvious bug - a HTTP server only responds when it
> has received something.  The usual keep-alive timeout of connections is
> silent (server just closes the connection, with a preceding TLS shutdown for
> HTTPS).

Please could you check my other response to Roy ? In short I do instead
think that responding 408 is the only way to gracefully shut down *and*
inform the client it can safely retry and I really think it was designed
for this exact purpose.

Regards,
Willy

---

From: Michael Sweet <[email-masked]>
Date: Wed, 04 Jun 2014 07:52:27 -0400

Willy,

> inform the client it can safely retry and I really think it was designed

I don't think it was designed for being returned when no request had started coming in, but rather when a partial request was received and then timed out.  There is nothing in any of the HTTP RFCs that talks about returning a response line, headers, and message body to the client without first receiving something from the client.

What CUPS does (and this was largely based on asking the question, "What does Apache HTTP Server do?" many years ago) is to silently close the connection after the configured timeout, which is large enough (300 seconds for CUPS and Apache) to account for any reasonable traffic delays (on the Earth, at least).

I can understand if you don't want to have as large of a timeout for a proxy that is shared by thousands of clients, but you can definitely make this configurable and default to something reasonable (30-60 seconds).  If the network connectivity is so bad that the request line and headers don't start flowing within that timeframe, then there is probably no point in retrying automatically anyways.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 14:49:10 +0200

Hi Michael,

On Wed, Jun 04, 2014 at 07:52:27AM -0400, Michael Sweet wrote:
> I don't think it was designed for being returned when no request had started
> coming in, but rather when a partial request was received and then timed out.

It's hard to navigate through the W3C archives to find the original discussion
for this status code introduced in HTTP/1.1. Some minutes talk about request
timeout :

  http://lists.w3.org/Archives/Public/ietf-http-wg-old/1994SepDec/0257.html

On the WG, 408 was already mentionned in a few exchanges here :

  http://lists.w3.org/Archives/Public/ietf-http-wg-old/1995MayAug/0340.html
  http://lists.w3.org/Archives/Public/ietf-http-wg-old/1995SepDec/0158.html

The first mention of it I could find in a published draft was in draft-00
dated 1995/11/22 :

    http://www.w3.org/Protocols/HTTP/1.1/draft-ietf-http-v11-spec-00.txt

And the text says :

  "
   408 Request Timeout

   The client did not produce a request within the time that the 
   server was prepared to wait. The client may repeat the request 
   without modifications at any later time.
  "

At least I'm not seeing any mention about starting to send anything, there
is a client and it did not produce a request, that's how I've always read
it. Also, please note this specific point in -p2 :

Note the last sentence : "*If* the client has an outstanding request ...".
For me that clearly means that the client might very well receive a 408
while it did not have an outstanding request, which implies that no byte
was sent over the wire yet.

> There is nothing in any of the HTTP RFCs that talks about returning a
> response line, headers, and message body to the client without first
> receiving something from the client.

Of course, but the same is true for the reverse proposition, nothing
says it's not possible. And in fact, 400 bad request is a perfect
example. If the client sends unparsable crap (so nothing matching
even the beginning of a valid request), a 400 bad req will be
returned. We're talking about protocol errors here leading to a
connection being closed as a result, I'm not seeing anything wrong
with that.

> What CUPS does (and this was largely based on asking the question, "What does
> Apache HTTP Server do?" many years ago) is to silently close the connection
> after the configured timeout, which is large enough (300 seconds for CUPS and
> Apache) to account for any reasonable traffic delays (on the Earth, at
> least).

But as I said in the other thread, the spec prevents the client from retrying
a non-idempotent request if the server does this, while 408 explicitly allows
it to. So I think that the intent behind the 408 was pretty clearly to inform
the client that no damage was made and that it had a chance to retry. On silent
close, you must not resend a POST for example, since it might very well have
already been processed :

   http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26#section-6.3.1
   6.3.1. Retrying Requests
   Connections can be closed at any time, with or without intention.
   Implementations ought to anticipate the need to recover from
   asynchronous close events.

   When an inbound connection is closed prematurely, a client MAY open a
   new connection and automatically retransmit an aborted sequence of
   requests if all of those requests have idempotent methods (Section
   4.2.2 of [Part2]).  A proxy MUST NOT automatically retry non-
   idempotent requests.

   A user agent MUST NOT automatically retry a request with a non-
   idempotent method unless it has some means to know that the request
   semantics are actually idempotent, regardless of the method, or some
   means to detect that the original request was never applied.

> I can understand if you don't want to have as large of a timeout for a proxy
> that is shared by thousands of clients, but you can definitely make this
> configurable and default to something reasonable (30-60 seconds).  If the
> network connectivity is so bad that the request line and headers don't start
> flowing within that timeframe, then there is probably no point in retrying
> automatically anyways.

I'm seeing some sites running with smaller timeouts to limit the risk of DoS.
When you know that you can accept 300k connections per second, with a 5s
timeout you already have 1.5 million connections. That's not unreasonable
to use aggressive timeouts in such conditions (especially for sites which
run older pre-forked servers which cannot accept such large numbers).

And you can clearly have a few seconds pause on 3G during a handover. I've
observed multi-second delays which cumulate with TCP retransmits to result
in something in the 10s range. The handover is an interesting case because
the network condition is temporarily bad and is good again once you get the
response, so there's a real incentive to retry instead of telling the user
"I'm sorry, I tried to validate your order but something has cut in the
middle so I don't know if you'll get it or not, please check your cart".

In the case of haproxy, sending the 408 only if something was started on
the connection is a trivial change, and I even delayed its final release
by a few days so that we have the time to discuss this use case and decide
what the proper cut-off mode is. But I think it would be a wrong change
that degrades the client's ability to seamlessly recover. For me the most
likely explanation for a close is "the server crashed after receiving the
request" (FIN) or "before receiving the request" (RST). 408 makes it pretty
clear that it's neither.

What would be interesting in fact is to know how various clients act on
silent close an on 408, with idempotent and non-idempotent requests. And
based on that we should likely clarify the text on what should be done.
Pat said that Firefox handles 408 fairly well (I found a bug report for
it being fixed in 2004). I don't know however how a POST over a silently
closed connection is handled there. Maybe non-browser clients like Curl
would help us determine the best behaviour as well. Daniel ?

Regards,
Willy

---

From: Sébastien Barnoud <[email-masked]>
Date: Wed, 4 Jun 2014 15:11:34 +0200

Hi,

I have several years ago used customized Curl and GSoap to automaticaly retry requests on such a condition. 
As this is used for money transfer, it is very important to distinguish retryable errors (connect timeout, ...) from others errors (request timeout, ...).
On a close, no retry is possible except if the server replies with this 408.
So i agree with you, and i know that some application uses this feature. 

> Le 4 juin 2014 à 14:49, Willy Tarreau <[email-masked]> a écrit :
> 
> Hi Michael,
>

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 05 Jun 2014 01:29:51 +1200

On 4/06/2014 11:52 p.m., Michael Sweet wrote:
> Willy,
> 
> On Jun 4, 2014, at 12:42 AM, Willy Tarreau wrote:
>> Hi Michael,
>> 
>>> This sounds like a pretty obvious bug - a HTTP server only
>>> responds when it has received something.  The usual keep-alive
>>> timeout of connections is silent (server just closes the
>>> connection, with a preceding TLS shutdown for HTTPS).
>> 
>> Please could you check my other response to Roy ? In short I do
>> instead think that responding 408 is the only way to gracefully
>> shut down *and* inform the client it can safely retry and I really
>> think it was designed for this exact purpose.
> 
> I don't think it was designed for being returned when no request had
> started coming in, but rather when a partial request was received and
> then timed out.  There is nothing in any of the HTTP RFCs that talks
> about returning a response line, headers, and message body to the
> client without first receiving something from the client.

In counterpoint I don't think HTTP/1 was designed for the model of
opening TCP connections then leaving them unused for up to 60 minutes
with no request.

* In the IPv6 network environment we have ongoing issues with PMTUd
brokenness from admin A wrongly disabling all ICMP traffic and admin B
some networks away sending traffic over a tunnel or hop with MTU
restructions. What this looks like to HTTP intermediaries on the
receiving end (CDN reverse-proxies etc) is a connection where no request
was ever sent.

* There are similar well-known issues with ECN and Window Scaling that
result in the same behaviour. And any number of other future or weird
TCP level issues may come up at any time preventing a mid-flight request.

* Bad/mistaken end-user configuration resulting in any server-initiated
protocol such as FTP or SMTP being incorrectly directed at the HTTP
proxy result in hung connections. A client sitting waiting for server
HELO wastes resources and is not easily debugged.

* Then there is RTT race conditions on idle connections of any type
(used or not). The request may be half written and in-flight when the
server end closes on a long timeout. What then is the client expected to
do? if the request was pipelined a bad choice of reaction could
seriously break the client applications expectations.

If the intermediary waits for a few minutes then sends a small 408
followed by immediate termination there is some chance that the client
will be able to recover from a lost request which was sent but never
received. At the very least it adds visible clues to the troubleshooting
process fingering the problem.

RFC2616 requires that recipients treat such unexpected bytes as garbage
and mandated connection close. Using a valid HTTP/1.1 408 instead of
actual garbage/RST/FIN allows the recipient to treat it as response to
any sent request and fixes the pipeline race issue.

PS. Squid is not (yet) doing this 408 behaviour as HAProxy but it has
been on the todo list for some time for all the above reasons.

Amos

---

From: Roland Zink <[email-masked]>
Date: Wed, 04 Jun 2014 15:35:47 +0200

This seems somewhat unclear. In our original server implementation the 
408 was returned after an idle connection was closed. However there was 
a bug report that some client got this wrong so this was changed to only 
return 408 when at least one byte was received and otherwise to just 
close the connection.

 From the last sentence in the cited text from p2 I would say to return 
a 408 on an idle connection is allowed even when no byte was received. 
If you bind to such a connection as a client, send your request and then 
read the 408 you are free to open a new connection and repeat the 
request. The same would happen if there is a real race condition.

Roland

On 03.06.2014 02:16, William Chan (???) wrote:
> Hello folks,
>
> Here's the relevant text from p2 
> (http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.5.7):
>
>
>         6.5.7
>         <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.5.7>.
>         408 Request Timeout
>
>
>
>
> Server operators using HAProxy have filed a bug report with Chromium 
> due to our lack of support for 408 
> (https://code.google.com/p/chromium/issues/detail?id=377581). We're 
> going to fix that. But HAProxy uses 408s in an intriguing way that 
> violates some of our architectural assumptions. Namely, HAProxy wants 
> to shut down an idle connection, it sends a 408 irrespective of 
> whether or not there was a HTTP request.
>
> Currently, Chromium's connection management code treats this as a 
> server bug, since when we bind a HTTP transaction to an idle 
> connection, we expect that the connection does not have any buffered 
> read data (since the previous transaction, if any, should have 
> completed successfully). However, there's a race condition here, since 
> the server may have terminated that idle connection with a 408 HTTP 
> response followed by a FIN.
>
> There's fundamentally a race here, which is crappy and we fix in 
> HTTP/2 with GOAWAY on the connection. My question here is, is it 
> kosher for HAProxy to be doing this? In retrospect, I feel like it's a 
> pragmatic decision for them to do so. That said, it feels incredibly 
> ugly for an HTTP/1.X server to generate responses without a request. 
> It's not completely clear to me what the intent is in the spec here, 
> because it says that the 408 means the server did not receive a 
> complete request. If it only receives a partial request, then that 
> makes sense it could send a timeout. But if there's no request 
> whatsoever, it's not immediately obvious to me that the spec says that 
> that is allowed.
>

---

From: Zhong Yu <[email-masked]>
Date: Wed, 4 Jun 2014 10:22:48 -0500

> was sent over the wire yet.

How does the client receive the unprompted 408 reliably? Should the
client be reading the inbound all the time?

Zhong Yu

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 17:28:25 +0200

> client be reading the inbound all the time?

When it sends a request, it receives the unprompted 408 which was sitting
in the buffers, there's no desynchronization here. As Amos stated it, if
the client sees a 408 without sending anything, it's garbage and it trashes
it. If the client sends a request and sees a 408, it's the response to his
request. It does not matter whether the 408 was built before the client
started to send or after.

Willy

---

From: Zhong Yu <[email-masked]>
Date: Wed, 4 Jun 2014 11:15:30 -0500

> in the buffers, there's no desynchronization here. As Amos stated it, if

I'm not sure how it works. Presumably the client has received server
FIN before it starts to write the request. This likely results in an
RST, which destroys the client's inbound buffer as well.

If the client is monitoring its inbound all the time for ahead-of-time
response, we don't need 408 either - the server FIN serves the same
purpose.

Zhong Yu

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 4 Jun 2014 20:14:47 +0200 (CEST)

On Wed, 4 Jun 2014, Sébastien Barnoud wrote:

> I have several years ago used customized Curl and GSoap to automaticaly 
> retry requests on such a condition. As this is used for money transfer, it 
> is very important to distinguish retryable errors (connect timeout, ...) 
> from others errors (request timeout, ...). On a close, no retry is possible 
> except if the server replies with this 408. So i agree with you, and i know 
> that some application uses this feature.

But then, curl doesn't read from the connection until after it has sent its 
request (or part of request) so the "problematic" case with getting a response 
before even sending a request will never be noticed.

-- 

  / daniel.haxx.se

---

From: Sébastien Barnoud <[email-masked]>
Date: Wed, 4 Jun 2014 20:43:38 +0200

But i have made a custom version if curl and GSoap to handle that. 

> Le 4 juin 2014 à 20:14, Daniel Stenberg <[email-masked]> a écrit :
> 
>> I have several years ago used customized Curl and GSoap to automaticaly retry requests on such a condition. As this is used for money transfer, it is very important to distinguish retryable errors (connect timeout, ...) from others errors (request timeout, ...). On a close, no retry is possible except if the server replies with this 408. So i agree with you, and i know that some application uses this feature.
> 
> But then, curl doesn't read from the connection until after it has sent its request (or part of request) so the "problematic" case with getting a response before even sending a request will never be noticed.
> 
> -- 
> 
> / daniel.haxx.se

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 20:46:58 +0200

On Wed, Jun 04, 2014 at 11:15:30AM -0500, Zhong Yu wrote:
> FIN before it starts to write the request.

It needs to consume the response to be aware of the FIN.

> This likely results in an

No, the RST is emitted if the server has totally closed (ie: the socket
buffers are destroyed and the socket orphan has been killed).

> response, we don't need 408 either - the server FIN serves the same

No, the FIN alone tells you that anything happened, first thing being the
server crashed while processing your request, which is why the spec prevents
the client from retrying a non-idempotent request.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 20:55:34 +0200

On Wed, Jun 04, 2014 at 08:14:47PM +0200, Daniel Stenberg wrote:
> >from others errors (request timeout, ...). On a close, no retry is 
> >possible except if the server replies with this 408. So i agree with you, 
> >and i know that some application uses this feature.
> 
> request (or part of request) so the "problematic" case with getting a 
> response before even sending a request will never be noticed.

No problem, it will be noticed *after* the request is sent when the
client waits for the response since it's present in the network buffers.

Look, first case is when network loses packets :

  client                        SYN                       server
  connect()      ------------------------------------>
                              SYN-ACK
                 <------------------------------------
                                ACK
                 ------------------------------------>  accept() : timer starts.
                            POST /login
  send(req)      -----------------X  packet lost due to MTU or radio
                            POST /login
                 -----------------X  packet lost due to MTU or radio
                            POST /login
                 -----------------X  packet lost due to MTU or radio

                            HTTP/1.1 408 + FIN
  recv(resp)     <-----------------------------------
                            POST /login
                 -----------------X  packet lost due to MTU or radio
                            RST
  close()        ------------------------------------> 

So as you see, client receives the 408 as a response to the request.

Second case is when there's a preconnect :

  client                        SYN                       server
  connect()      ------------------------------------>
                              SYN-ACK
                 <------------------------------------
                                ACK
                 ------------------------------------>  accept() : timer starts.
  wait...
                            HTTP/1.1 408 + FIN
                 <-----------------------------------

                            POST /login
  send(req)      ----------------------------------->
  recv(resp)
                            RST
  close()        ------------------------------------> 

Here the 408 is received because it was pending in the network buffers,
which is why people were seeing it in the case of the pre-connect.

Regards,
Willy

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 4 Jun 2014 12:50:59 -0700

Thanks everyone for chiming in. At this point, I think I should make some
clarifying points and inject / re-emphasize some personal opinion here. You
can probably ignore this email unless you're interested in all the details.

Clarifications:
1) There is fundamentally a race between the client sending a request and
the server closing its end of the connection, due to the network latency.
2) If the client does not read from the socket before/during sending a
request, then the time gap for the "race" may include not only the network
latency, but also the time until the client transmits the request and
starts reading from the socket.
3) When the client receives FIN or a socket error when reading from the
socket, it usually cannot tell the root cause (server timing out the
connection, server application crash, etc).
4) Clients have different heuristics for retrying on such errors. Here's
Chromium's (
https://code.google.com/p/chromium/codesearch#chromium/src/net/http/http_network_transaction.cc&q=http_network_transaction.cc&sq=package:chromium&l=1479
):

bool HttpNetworkTransaction::ShouldResendRequest() const {  bool
connection_is_proven = stream_->IsConnectionReused();  bool
has_received_headers = GetResponseHeaders() != NULL;  // NOTE: we
resend a request only if we reused a keep-alive connection.  // This
automatically prevents an infinite resend loop because we'll run  //
out of the cached keep-alive connections eventually.  if
(connection_is_proven && !has_received_headers)    return true;
return false;}

As one can see, we opt not to retry on an unused connection
(HttpStream::IsConnectionReused() is actually misnamed, it also includes
sockets that were ever idle, like our preconnected sockets). This is
because the HTTP request may have caused a server error (like the server
application to crash) and it would not be nice to re-send that request.
However, for idle HTTP connections, there's a reasonable likelihood due to
the idleness that we hit one of those races, so we will retry the
transaction on a new connection. I guess I should show the situation that
leads us to call this function...which is on relevant socket errors (
https://code.google.com/p/chromium/codesearch#chromium/src/net/http/http_network_transaction.cc&q=http_network_transaction.cc&sq=package:chromium&l=1411
):

int HttpNetworkTransaction::HandleIOError(int error) {  // Because the
peer may request renegotiation with client authentication at  // any
time, check and handle client authentication errors.
HandleClientAuthError(error);  switch (error) {    // If we try to
reuse a connection that the server is in the process of    // closing,
we may end up successfully writing out our request (or a    // portion
of our request) only to find a connection error when we try to    //
read from (or finish writing to) the socket.    case
ERR_CONNECTION_RESET:    case ERR_CONNECTION_CLOSED:    case
ERR_CONNECTION_ABORTED:    // There can be a race between the socket
pool checking checking whether a    // socket is still connected,
receiving the FIN, and sending/reading data    // on a reused socket.
If we receive the FIN between the connectedness    // check and
writing/reading from the socket, we may first learn the socket    //
is disconnected when we get a ERR_SOCKET_NOT_CONNECTED.  This will
most    // likely happen when trying to retrieve its IP address.    //
See http://crbug.com/105824 for more details.    case
ERR_SOCKET_NOT_CONNECTED:    // If a socket is closed on its initial
request, HttpStreamParser returns    // ERR_EMPTY_RESPONSE. This may
still be close/reuse race if the socket was    // preconnected but
failed to be used before the server timed it out.    case
ERR_EMPTY_RESPONSE:      if (ShouldResendRequest()) {
net_log_.AddEventWithNetErrorCode(
NetLog::TYPE_HTTP_TRANSACTION_RESTART_AFTER_ERROR, error);
ResetConnectionAndRequestForResend();        error = OK;      }
break;

5) HTTP is typically request/response, so Chromium makes the assumption
that it is a server bug if there is data in the socket before we try to use
it, and will discard the socket and try to find another to use. However,
recently we discovered that for our preconnected sockets and SPDY / HTTP/2,
the server always wrote the SETTINGS frame, therefore we weren't leveraging
preconnect properly for SPDY / HTTP/2. So we *recently* decided to allow
sockets with readable data if the socket hadn't been used yet (
https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/client_socket_pool_base.cc&q=client_socket_pool_base.cc&sq=package:chromium&l=651
):

bool ClientSocketPoolBaseHelper
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/client_socket_pool_base.h&cl=GROK&ct=xref_jump_to_def&l=156&gsn=ClientSocketPoolBaseHelper>::IdleSocket
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/client_socket_pool_base.h&cl=GROK&ct=xref_jump_to_def&l=339&gsn=IdleSocket>::IsUsable
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/client_socket_pool_base.cc&ct=xref_usages&gs=cpp:net::internal::class-ClientSocketPoolBaseHelper::class-IdleSocket::IsUsable()-const@chromium/../../net/socket/client_socket_pool_base.cc%257Cdef&l=651&gsn=IsUsable>()
const {  if (socket
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/client_socket_pool_base.h&cl=GROK&ct=xref_jump_to_def&l=357&gsn=socket>->WasEverUsed
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/stream_socket.h&cl=GROK&ct=xref_jump_to_def&l=76&gsn=WasEverUsed>())
   return socket
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/client_socket_pool_base.h&cl=GROK&ct=xref_jump_to_def&l=357&gsn=socket>->IsConnectedAndIdle
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/stream_socket.h&cl=GROK&ct=xref_jump_to_def&l=54&gsn=IsConnectedAndIdle>();
 return socket <https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/client_socket_pool_base.h&cl=GROK&ct=xref_jump_to_def&l=357&gsn=socket>->IsConnected
<https://code.google.com/p/chromium/codesearch#chromium/src/net/socket/stream_socket.h&cl=GROK&ct=xref_jump_to_def&l=49&gsn=IsConnected>();}

Previously, we always checked IsConnectedAndIdle(), but now we just check
IsConnected() for unused sockets. One could imagine tightening this check
so that we only allowed readable data if the socket was detected to be a
SPDY / HTTP/2 connection.

6) Armed with the previous knowledge, we can see what happens when Chromium
did *not* support 408s, and when 408 responses are sent by the server
*before* any request is sent by the client. With the recent change to allow
using unused sockets that have readable data, we passed on these sockets
(with 408 response data) to the HTTP transaction code. Since Chromium did
not support 408s, this led to 408s being treated as transaction errors, not
socket IO errors, so we did not retry these transactions. Due to the recent
change as described in (5), this meant that users saw more errors when
browsing websites with servers that sent these 408 responses on idle
connection timeouts. Chromium (still unreleased in Google Chrome release
channels) is supporting 408s now (
https://src.chromium.org/viewvc/chrome?revision=274760&view=revision).

Personal opinion:
1) Sending unprompted 408 responses when gracefully terminating a
connection clearly does provide a signal for clients to explicitly know
it's safe to retry. Explicit knowledge of retry safety is a good thing.
2) Sending unprompted HTTP responses is...very weird. This is non-obvious
to most HTTP implementers. Judging from the previous responses so far on
this thread, I think many people agree with this analysis.

That said, I personally think (1) is more important than (2) here. Yes, (2)
is ugly. But (1) makes life better for users, so deal with it? In my
opinion, the spec should explicitly acknowledge this case for 408s. I don't
really care to argue what the original intent of 408 was, I care about
coming to a consensus on how we should treat it and making it clear in the
spec.

Cheers.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 22:01:41 +0200

Hey William,

> coming to a consensus on how we should treat it and making it clear in the

Thanks for the in-depth analysis of the reasons explaining why there was
a recent increase in this report, especially when Chrome+pre-connect+haproxy
were combined.

I agree with the points you made. I'd go further, do you think we could
improve the client's ability to safely retry if a 408 was also emitted
on persistent connections *after* a request/response was already served ?
Haproxy explicitly disables sending of 408 here and I've not seen a server
emitting a 408 in this condition either. I've read that Firefox is already
fine with this, and apparently now Chromium is OK with it. But we need to
know if we're improving the client's ability to safely reuse the connection
or if there is no chance to gain anything (eg: due to the vast majority of
deployed servers not doing it anyway).

I agree we need to find a consensus and better document these corner cases.
Most commonly deployed clients, servers and intermediaries are on this list,
so it should not be too hard to know what to change where :-)

Best regards,
Willy

---

From: Zhong Yu <[email-masked]>
Date: Wed, 4 Jun 2014 15:45:55 -0500

> buffers are destroyed and the socket orphan has been killed).

The server intends to kill the idle connection and free the resources,
correct? Then the client will get RST when it tries to write to the
connection; if the client tries to read from the connection
afterwards, it'll likely fail as well, depending on the TCP stack.

> No, the FIN alone tells you that anything happened, first thing being the

The FIN is received before the client writes the request. It's safe to
infer that the request is not being processed the server. A reasonable
server will not close its outbound, then read and process another
request.

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 4 Jun 2014 22:46:05 +0200 (CEST)

On Wed, 4 Jun 2014, Willy Tarreau wrote:

> I'd go further, do you think we could improve the client's ability to safely 
> retry if a 408 was also emitted on persistent connections *after* a 
> request/response was already served ?

You mean as a second response? That would effectively kill the connection for 
curl at least (without even reading the response) since it'll consider data on 
a connection it wants to re-use as a sign of badness and just close it.

> so it should not be too hard to know what to change where :-)

It is also important to remember that many of us have very old code bases 
still alive and in use out there so what we change now can easily break things 
we wrote a decade ago that still is being used...

-- 

  / daniel.haxx.se

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 4 Jun 2014 22:49:27 +0200 (CEST)

On Wed, 4 Jun 2014, Willy Tarreau wrote:

> No problem, it will be noticed *after* the request is sent when the client 
> waits for the response since it's present in the network buffers.

Yes, sorry for being muddy but that is what I meant.

Also, since curl doesn't do pre-connects or "speculative" connects in any ways 
that would risk to add the time between the TCP connect and the first bytes of 
the HTTP request to get sent I would have guessed that the amount of 408s that 
had been sent or received in that interval to be fairly small.

-- 

  / daniel.haxx.se

---

From: Michael Sweet <[email-masked]>
Date: Wed, 04 Jun 2014 17:00:52 -0400

Willy,

I can go along with a 408 (and only a 408) response being sent by a server right before a connection is torn down due to inactivity.  Document this as "the server MAY send a 408 response prior to closing a connection and a client SHOULD use the presence of a 408 response as an indication that the request should be retried" or something along those lines, with the discussion below distilled to the critical bits.  We should also be clear that this problem is most pronounced when using small inactivity timeouts.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 23:55:22 +0200

On Wed, Jun 04, 2014 at 03:45:55PM -0500, Zhong Yu wrote:
> connection;

No, it will first get the response which is already present in its
buffers, the RST will happen at best one RTT after the request is sent.
That's exactly why so many people were recently seeing the 408 in their
browser.

> if the client tries to read from the connection

Once the RST reaches the client, it will fail. But before that the
pre-queued response will be delivered. And I don't see any reason
why a client would send, decide to wait an arbitrary delay, then
receive. The client sends the request and immediately polls for
receive, then it immediately gets the queued response.

> server will not close its outbound, then read and process another

The problem is not when the FIN reaches the client early. The problem is
that in many circumstances, this FIN will be sent in parallel to the late
request, and will then reach the client *after* the request was sent, while
it was sent *before* the request reached the server. That's the race Amos
was speaking about, and which is even documented in p1-6.5 :

   http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26

   For example, a client might have started to send a new request
   at the same time that the server has decided to close the "idle"
   connection.  From the server's point of view, the connection is being
   closed while it was idle, but from the client's point of view, a
   request is in progress.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jun 2014 23:58:47 +0200

On Wed, Jun 04, 2014 at 10:46:05PM +0200, Daniel Stenberg wrote:
> >I'd go further, do you think we could improve the client's ability to 
> >safely retry if a 408 was also emitted on persistent connections *after* a 
> You mean as a second response? That would effectively kill the connection 
> for curl at least (without even reading the response) since it'll consider 
> data on a connection it wants to re-use as a sign of badness and just close 
> it.

Yes that was the idea.

> >I agree we need to find a consensus and better document these corner 
> >cases. Most commonly deployed clients, servers and intermediaries are on 
> >this list, so it should not be too hard to know what to change where :-)
> 
> still alive and in use out there so what we change now can easily break 
> things we wrote a decade ago that still is being used...

Sure, I'm well aware of this and that's precisely because I'm not fond of
breaking what works that I've been hesitating to change the way 408 is
currently emitted by haproxy. It's been this way for 12 years I think,
so who knows how many issues will surface if we start not sending 408
anymore. At least I guess from Sebastien's report that some of his
deployments will definitely break.

But if we can all together find the safest compatibility use case between
most or even all products, we could hope to smoothly improve our respective
next versions.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Jun 2014 00:04:00 +0200

Hi Michael,

On Wed, Jun 04, 2014 at 05:00:52PM -0400, Michael Sweet wrote:
> Willy,
> 
> I can go along with a 408 (and only a 408) response being sent by a server
> right before a connection is torn down due to inactivity.  Document this as
> "the server MAY send a 408 response prior to closing a connection and a
> client SHOULD use the presence of a 408 response as an indication that the
> request should be retried" or something along those lines, with the
> discussion below distilled to the critical bits.

Concerning the second part of the sentence, I think that the existing text
already covers it (though it uses MAY instead of SHOULD as we shouldn't
dictate to the client how it needs to handle the retry I guess) :

   "If the client has an outstanding request in transit, the
    client MAY repeat that request on a new connection."

> We should also be clear
> that this problem is most pronounced when using small inactivity timeouts.

I think that it might be worth mentionning it, yes, so that clients decide
whether or not they want to implement it depending on their target use case.
Eg: maybe your cups client does not have any benefit in retrying in such a
case (I have not idea, just an example).

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Jun 2014 00:05:26 +0200

On Wed, Jun 04, 2014 at 10:49:27PM +0200, Daniel Stenberg wrote:
> >>But then, curl doesn't read from the connection until after it has sent 
> >>its request (or part of request) so the "problematic" case with getting a 
> Also, since curl doesn't do pre-connects or "speculative" connects in any 
> ways that would risk to add the time between the TCP connect and the first 
> bytes of the HTTP request to get sent I would have guessed that the amount 
> of 408s that had been sent or received in that interval to be fairly small.

I think so as well :-)

Willy

---

From: Zhong Yu <[email-masked]>
Date: Wed, 4 Jun 2014 18:24:19 -0500

> That's exactly why so many people were recently seeing the 408 in their

Got it. However, RST is still very likely for requests with bigger
payloads. If a client wants to reliably receive unprompted 408s, it
better proactively poll the inbound. In that case, I argue that FIN is
as good as 408, because the client agent knows that the server closed
the connection before the client sends (or is about to send) the
request, therefore it's safe to retry the request.

We could argue that unprompted 408 is beneficial for most requests (of
moderate size). However, if browsers need to be patched to take
advantage of 408, they are better off to spend the effort to handle
the more general case - proactively poll the inbound, discard the
connection immediately if anything is received without a request
(mostly FIN).

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 05 Jun 2014 16:08:07 +1200

> most or even all products, we could hope to smoothly improve our respective

I'm pretty sure the experience Chrome had is a good example of how such
old/naive software will react. A switch of some FIN/RST from whatever
error handling to act as they would to a 400 reply, and other FIN/RST as
they would garbage+close. Disambiguating and allowing better
troubleshooting in the process.

If the old client is not handling unexpected 408 properly as alternative
to FIN/RST. Then its already not operating as per spec. The requirement
for garbag+FIN on unexpected *bytes* has nothing specifically to do with
those bytes being an early 408.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Jun 2014 07:32:33 +0200

On Wed, Jun 04, 2014 at 06:24:19PM -0500, Zhong Yu wrote:
> payloads.

Yes, if the request is larger than 2 MSS (or more precisely doesn't
fit into the initcwnd), it will need at least one RTT and the RST will
be visible before it is sent.

> If a client wants to reliably receive unprompted 408s, it

It's safe if the FIN or RST arrives before the empty line marking the end
of headers is sent. If it was sent and some payload was being aborted, it
is not necessarily safe since it's very common for application servers
not to consume the whole request payload but just what they need, before
processing the request. However they must consume whatever the client
sends even after they send the response (cf p1-6.3). The problematic
use case I'm seeing is when an application server uses an HTTP auth
mechanism (typically Basic auth) accompanying a large request (eg:
you need to log in to upload a file or post a message in a webmail).
If the server detects a wrong password, it will increment the account's
bad attempts counter and return a 401 and possibly shutdown. Here the
client receiving the 401+FIN must not consider that it's safe to retry,
even if it arrives while it's sending the payload. And if the server
does not correctly drain the request and finally closes (very common
unfortunately), the client will get an RST at some point, so here again
the RST while sending the payload is not safe for automatic retry.

> connection immediately if anything is received without a request

I completely agree that if any change needs to be applied, the first
one is to check the inbound first, eventhough I can understand that
it's easier to say than to apply to any existing code which was
architected based on a different sequencing. It will cover most use
cases and will only leave open the race where the server decides to
close at the same time the client decides to send. That said, when
you look at Matt's patches to make Chromium aware of the 408, I don't
think it would represent a huge effort to adjust clients to correctly
consider it.

Regards,
Willy

---

From: Zhong Yu <[email-masked]>
Date: Thu, 5 Jun 2014 10:13:50 -0500

> it's easier to say than to apply to any existing code which was

agreed.

> think it would represent a huge effort to adjust clients to correctly

ok. It seems harmless that servers send "psychic" responses ahead of
requests. Besides 408, other codes may also qualify, like 503.