Subject: NEW ISSUE(S): Retrying Requests

From: Mark Nottingham <[email-masked]>
Date: Wed, 5 Mar 2008 15:27:13 +1100

> 14.37 Retry-After
> The Retry-After response-header field can be used with a 503  
> (Service Unavailable) response to indicate how long the service is  
> expected to be unavailable to the requesting client. This field MAY  
> also be used with any 3xx (Redirection) response to indicate the  
> minimum time the user-agent is asked wait before issuing the  
> redirected request. The value of this field can be either an HTTP- 
> date or an integer number of seconds (in decimal) after the time of  
> the response.
>    Retry-After = "Retry-After" ":" ( HTTP-date | delta-seconds )
> Two examples of its use are
>    Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
>    Retry-After: 120
> In the latter example, the delay is 2 minutes.

Issues;

1) 14.37 mentions 3xx methods and 503 as valid placements; in addition  
the definition of 413 refers to Retry-After. The text of 14.37 should  
be updated to include 413.
2) 10.3 (3xx codes) doesn't mention Retry-After; it should be added  
there, or 3xx should be removed from 14.37 (I suspect the latter,  
since AFAIK this isn't implemented).
3) What is the relationship between retry-after and the Cache-Control:  
max-age on those responses? (for the 503 case, I wonder if CC: max-age  
is a better practice than Retry-After)

> 8.1.4 Practical Considerations [of persistent connections]
>
> Non-idempotent methods or sequences MUST NOT be automatically  
> retried, although user agents MAY offer a human operator the choice  
> of retrying the request(s). Confirmation by user-agent software with  
> semantic understanding of the application MAY substitute for user  
> confirmation. The automatic retry SHOULD NOT be repeated if the  
> second sequence of requests fails.

Issues;

4) This is defined in the context of persistent connections that have  
closed, which is fine, except that retry behaviour for idempotent  
requests isn't defined anywhere else; the reader has to deduct that  
it's possible by reading between the lines here. Since automatic  
retries are implemented in a number of places (client and  
intermediary), I think it should be at least mentioned in a way that  
isn't just specific to connections closing, and as a MAY, not just a  
MUST NOT on non-idempotent requests. I.e., I think request retries (as  
opposed to connection retries) deserves its own section.
5) The text in 8.2.4 (below) needs to be taken into account in this  
section (e.g., by qualifying the MUST NOT and SHOULD NOT with a  
reference to it).

> 8.2.4 Client Behavior if Server Prematurely Closes Connection
> If an HTTP/1.1 client sends a request which includes a request body,  
> but which does not include an Expect request-header field with the  
> "100-continue" expectation, and if the client is not directly  
> connected to an HTTP/1.1 origin server, and if the client sees the  
> connection close before receiving any status from the server, the  
> client SHOULD retry the request. If the client does retry this  
> request, it MAY use the following "binary exponential backoff"  
> algorithm to be assured of obtaining a reliable response:
>
> 1. Initiate a new connection to the server
> 2. Transmit the request-headers
> 3. Initialize a variable R to the estimated round-trip time to the  
> server (e.g., based on the time it took to establish the  
> connection), or to a constant value of 5 seconds if the round- trip  
> time is not available.
> 4. Compute T = R * (2**N), where N is the number of previous retries  
> of this request.
> 5. Wait either for an error response from the server, or for T  
> seconds (whichever comes first)
> 6. If no error response is received, after T seconds transmit the  
> body of the request.
> 7. If client sees that the connection is closed prematurely, repeat  
> from step 1 until the request is accepted, an error response is  
> received, or the user becomes impatient and terminates the retry  
> process.

Cheers,

--
Mark Nottingham       [email-masked]

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 5 Mar 2008 05:36:45 -0800

Mark Nottingham wrote:
> This is defined in the context of persistent connections 
> that have closed, which is fine, except that retry
> behaviour for idempotent requests isn't defined anywhere
> else; the reader has to deduct that it's possible by
> reading between the lines here. Since automatic  
> intermediary), I think it should be at least mentioned in
> a way that isn't just specific to connections closing,
> and as a MAY, not just a MUST NOT on non-idempotent
> requests. I.e., I think request retries (as  

I agree that is a good idea. It also affects the Pipelining section, and
anywhere else that mentions idempotency. With this new section, maybe
the term idempotency can be totally removed from the specification? I
agree with Roy [1] that the use of the term in RFC 2616 has caused more
trouble than benefit.

[1] http://www.imc.org/atom-protocol/mail-archive/msg10759.html

- Brian

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 6 Mar 2008 00:33:45 +0000

> [1] http://www.imc.org/atom-protocol/mail-archive/msg10759.html

I'm still puzzled as to when a client should reuse a persistent
connection for requests that (it knows) shouldn't be retried.

Since all servers close a persistent connection an unspecified time
after the first request, and that's perfectly healthy (all servers
must do it), how long since it saw activity should a client wait
before considering the connection unsafe for those type of requests?

E.g. if a server closes idle connections after 60 seconds, and a
client transmits a request 58 seconds after the previous response,
there's a good chance the client will get a connection error.

If the error is due to perfectly healthy idle connection shutdown by
the server, the client should retry automatically no matter what type
of request (ideally).  That is expected in ordinary circumstances; no
real errors have occurred.

But if it's due to a fault (such as the server crashed), although it
might retry, sometimes it shouldn't be automatic, and definitely not
for some kinds of request.  Ideally, it would be reported as a fault.

This works fine, as long as the client has a cut-off time somewhat
lower than the server's idle shutdown time.

So to make this reliable, is there a recommended cut-off time for
clients and an idle shutdown time for servers?  Or is this left to
implementors to look at each others code, and pick a value that
matches what they think the other end does?

-- Jamie

---

From: Robert Siemer <[email-masked]>
Date: Thu, 6 Mar 2008 20:47:17 +0100

On Thu, Mar 06, 2008 at 12:33:45AM +0000, Jamie Lokier wrote:

> must do it), ...

Why must all servers do that?

Robert

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 6 Mar 2008 20:22:13 +0000

> Why must all servers do that?

Two reasons:

   1. Because idle TCP/IP sockets get into a stuck state, if the other
      end disappears off the net.  Especially on internet facing
      servers, these tend to accumulate without bound (older servers
      had to be rebooted from time to time because of this).

      Servers will accumulate these until they run out of ports or
      memory, if they don't have some way to drop old, idle connections.

   1. To defend against too many clients keeping connections open for
      an arbitrarily long time, whether maliciously or too popular.

-- Jamie

---

From: Robert Siemer <[email-masked]>
Date: Fri, 7 Mar 2008 00:29:39 +0100

>       had to be rebooted from time to time because of this).

Newer servers have TCP keepalive.

>       memory, if they don't have some way to drop old, idle connections.

The server runs out of port 80? What ports are you talking about?

Apart from that you just switched from "persistent connections" to "old, 
idle connections". TCP connections don't grow old. Servers don't need to 
close connections in use - be it after the 100th or 100000th request or 
be it after a "long" time.

>       an arbitrarily long time, whether maliciously or too popular.

You give your second reason the same number (1.), because it is actually 
the same? ;-)   Apart from memory consumption is there anything else to 
defend?

Robert

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 07 Mar 2008 12:52:56 +1300

> Newer servers have TCP keepalive.
Last time I looked at keepalive specs for TCP the sorts of timeouts 
before keepalives were sent were in the order of hours.

IOW not particularly useful, and definitely not useful for a highly 
loaded HTTP server.

> The server runs out of port 80? What ports are you talking about?
* connection handles then. 
* kernel resources,
* Memory
* firewall hash entries
* authentication tokens
* etc etc etc

There are lots and lots of things you really don't want hanging around 
forever for idle clients.  Until we all have ready access to an OS that 
has truly unlimited numbers of these things with no performance 
penalties as the number grows, we'll want to clean up.  As long as we 
use physical matter for RAM, it's a limited resource.  Demand can always 
exceed supply capability in this case.

> be it after a "long" time.

As someone who has seen what happens if you don't clean up... it's 
definitely necessary.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 7 Mar 2008 02:29:28 +0000

> Newer servers have TCP keepalive.

Yes.  The server has to choose a TCP keepalive timeout.  For some
reason I had a brain fart and thought it was the same problem (of
matching timeouts) in another guise, but no, you're right.  The server
does have to choose a timeout, but it's only to detect connectivity
failure, and doesn't have to match any client application behaviour.

> The server runs out of port 80? What ports are you talking about?

Sorry, I meant sockets.  Some systems run out of TCP sockets (fixed
table in the kernel), and many unix systems have a limited number of
file descriptors that the server application can keep open.  Even if
the limit is 10000 sockets, it's easily reached if you keep
connections open "forever".

> be it after a "long" time.

By old, I meant one that was idle for a long time, needing the
application to close it.  As you rightly say, TCP itself will maintain
a connection forever if there is no traffic.

> the same? ;-)   Apart from memory consumption is there anything else to 

I'm seeing your point.  Still, even a fancy event-driven server on
Solaris or Linux will hit limits from time to time (not memory, but
resources like file descriptors).

It does depend on having a newer TCP stack, though.  TCP keepalive
isn't available in some of the old ones, and many older systems cannot
handle large numbers of sockets.

Last time I looked at Apache's code, it had a timeout on idle
connections.  I presume this is mainly to avoid having thousands of
idle sockets around, and because it was more of a problem in the past,
with older systems, and possibly lacking TCP keepalive.

The implication from what you're saying is that modern systems aren't
bothered, so it's fine to leave connections open for a "long time"
whatever that may mean, and with TCP keepalive.

I'm not sure how happy I am to rely on that as a strategy.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 7 Mar 2008 02:33:56 +0000

> before keepalives were sent were in the order of hours.

Perhaps he meant _newer_ servers, for which the keepalive intervals
are adjustable.

> IOW not particularly useful, and definitely not useful for a highly 

Quite.
> forever for idle clients.

Heh.  That was one reason HTTP was _invented_: to avoid the problem of
tying up state for FTP connections.  HTTP was celebrated as "stateless".

It's interesting that people now regard FTP servers as "lightweight"
in comparison to HTTP.  Total turnaround.

I have had people beg me to install an FTP server so they don't have
the overhead of a modern HTTP server!

> As someone who has seen what happens if you don't clean up... it's 

Ok.  And does anyone have any advice, for long a server _needs_ to
keep a persistent connection open?

As noted, if it closes too early, compared with deployed clients, it
can lead to unnecessary errors or ambiguous retryings.

So what sort of timeout is needed to work with the major clients?

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 07 Mar 2008 16:27:46 +1300

actually I don't know if it's really something HTTP spec needs to care 
about.

I'd speculate that there are very few web servers under any serious load 
that aren't behind a firewall. 

I think you'll be struggling to find a firewall that won't time out idle 
TCP connections.

A typical firewall timeout value is anything from 120s to 300s. But 
commonly adjustable, and admins with highly loaded systems would 
probably be tempted to reduce the timeout.

Again as a server, I wouldn't rely on the firewall timing out 
connections either but would definitely implement my own timeouts. As a 
client I wouldn't rely on connections being maintained if I wasn't using 
them.

I believe a server should be free to disconnect whenever it likes for 
whatever reason it likes (one of my gripes with SMTP spec is it 
prohibits the server from hanging up on the client even if the client 
abuses it).

Adrien