Subject: p1 7.2.4: retrying requests

From: Mark Nottingham <[email-masked]>
Date: Sat, 4 Jun 2011 09:06:07 +1000

<http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-14#section-7.2.4>

   If an HTTP/1.1 client sends a request which includes a request body,
   but which does not include an Expect header field with the "100-
   continue" expectation, and if the client is not directly connected to
   an HTTP/1.1 origin server, and if the client sees the connection
   close before receiving a status line from the server, the client
   SHOULD retry the request. 

So, this basically requires retrying POSTs and other unsafe request methods, under certain conditions (using a proxy OR connected to a 1.0 origin server, AND conn closed before response status).

However, 7.1.4 <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-14#section-7.1.4> says:

   This means that clients, servers, and proxies MUST be able to recover
   from asynchronous close events.  Client software SHOULD reopen the
   transport connection and retransmit the aborted sequence of requests
   without user interaction so long as the request sequence is
   idempotent (see Section 7.1.2 of [Part2]).  Non-idempotent request
   methods or sequences MUST NOT be automatically retried, although user
   agents MAY offer a human operator the choice of retrying the
   request(s).  Confirmation by user-agent software with semantic
   understanding of the application MAY substitute for user
   confirmation.  The automatic retry SHOULD NOT be repeated if the
   second sequence of requests fails.

where non-idempotent request methods MUST NOT be automatically retried.

These seem to conflict.  A generous reading would be that 7.2 is a special case of 7.1.4, but if that's the case, it should be explicit (e.g., by placing them in the same section, or explicitly referring to each other in context).

Personally -- I'm uncomfortable with 7.2, because a) it's pretty complex with all of those conditions, and b) POST is non-idempotent, and the request may have already been processed on the server. 

I.e., AFAIK we don't have a requirement that the server MUST send a status code at the moment that the resource's state is affected, and the connection may have been closed by an intermediary.

In my experience, implementations do retry GETs, but not POSTs, and they often use a fresh connection for POSTs (et al) to avoid just this situation. 

So, I'm inclined to just drop 7.2.4, but maybe I don't know the whole story. Thoughts?

--
Mark Nottingham   http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 4 Jun 2011 07:32:11 +0200

Hi Mark,

> So, I'm inclined to just drop 7.2.4, but maybe I don't know the whole story. Thoughts?

I agree with you. However, this point still leaves open a small hole where
a difficult case still exists : some gateways are using connection pools to
servers and aggregate incoming requests over those existing connections. In
theory those connections never die (in theory...). If a connection dies during
an idempotent request, it's easy to retry it. POSTs are also sent over those
connections, at the risk of loosing them and having to retry them. In my
opinion, adding an "Expect: 100-Continue" header to those requests is enough
to ensure they are sent over a valid connection. But the issue remains when
there is an empty body with the POST, because if we set the Expect header
with the empty body, we'll cause a deadlock (or the server might notice it
and proceed anyway).

So for empty POST requests, we still have no means of testing the connection
before reusing it. Or maybe by using chunked encoding and sending the 0<CRLF>
after 100 is received, provided the server accepts chunked encoded requests ?

In fact, connection pooling becomes so much common nowadays that I think we
should ensure any implementation gets all corner cases right than just saying
they can't replay a POST over a broken connection, because they'll do stupid
or dangerous things to get it working anyway (and if you knew the number of
people I encounter who are amazed that a POST must not be blindly replayed...).

Cheers,
Willy

---

From: Dzonatas Sol <[email-masked]>
Date: Sat, 04 Jun 2011 06:19:26 -0700

> an idempotent request, it's easy to retry it. POSTs are also sent over those

Hi, I noticed the stickiness over the use of the word "retry" when there 
is already the availability of "recovery" mode, especially in mesh 
networks were some prefer to simply apply error correction instead of 
retry or recovery. Whatever flow, our optimizations agree on law of 
conservation.

> to ensure they are sent over a valid connection. But the issue remains when

That depends if that state exists in an inner connection or outer 
connection. In&out can be virtualized in many ways, so no specific 
hardware can deem what to expect besides diagnostic mode (which we can 
always say this mode does not agree with law of conservation). I suggest:

 > Expect: 100-Continue / media-type

If the media type does not agree then another connection for retries or 
recovery. That clarifies the "Expect:" header as some signal for verbose 
mode (of particular media states) or as the pseudo-trace method (chunked 
mode).

> people I encounter who are amazed that a POST must not be blindly replayed...).

I know of cases where every request is immediately logged, the 
connection closed, and later the response is sent after processed and 
another connection is opened. Sounds like whatever frustration is of 
cause above, the HTTP Status 202 Accepted is not being issued 
appropriately, expected, or allowed. Basically, if the origin server 
already knows the response cannot be given by the end of the request 
(processed or not) then 202 Accepted should occur, which prevents retries.

-- 
--- https://twitter.com/Dzonatas_Sol ---
Web Development, Software Engineering, Virtual Reality, Consultant

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 5 Jun 2011 11:06:51 +1000

On 04/06/2011, at 3:32 PM, Willy Tarreau wrote:

> If a connection dies during

I don't think we can generalise your pattern using Expect/continue, because clients can't always be sure that the next hop is 1.1.

However, we could make it clear that it's OK for clients to retry non-idempotent requests when they have some sort of agreement (in-band, e.g., a protocol extension like POE, or out-of-band).

Would that be workable for you?

Regards,

---

From: Dzonatas Sol <[email-masked]>
Date: Sun, 05 Jun 2011 09:00:49 -0700

> Would that be workable for you?

These cases may be easier to think about in comparison if we think of 
click-wrap (licenses) to cover the chunked-mode with Expect/continue. 
Instead of some cookie that is stored in the headers if someone has seen 
the click-wrap or not, the Expect/continue could be used instead. 
Browsers can take action and prompt "Abort, Retry or Fail?" or would 
need implementation of "remember this choice" checkbox for like conditions.

In 7.2.4, "the client SHOULD retry the request" may indicate the 
connection closed due to timeout of the click-wrap condition. When the 
user responds to the prompt, the request "SHOULD" retry with appropriate 
remembered action on the condition.

That can help in cases where licensed media is aggressively searched for 
more content (and action links) before the condition is satisfied, or 
determine if licensed media is new to some device despite whatever the 
user response.

That is some sort of agreement, yet I also wondered if these conditions 
helps prevent or detect double-click and know redundant actions. For 
example, request on key-down, continue on key-up, and complete response 
with action.

"Drop 7.2.4...?" Yesterday, I thought over the HTML5 canvas again for 
rasterized images and where the UI is on that canvas, the window 
instance is on the proxy (inner-browser), and the client-browser only 
manages tabs (outer-browser). I wouldn't want to discourage that 
ability, as I know of businesses that rather keep their keep their 
analog assets private and just render digital versions of them for 
licensed proxies.

The common conflict exists where that is prevented with monolithic 
design, all-in-one.

--

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 7 Jun 2011 07:31:27 +0200

Hi Mark,

sorry for the delay.

> I don't think we can generalise your pattern using Expect/continue, because clients can't always be sure that the next hop is 1.1.

Indeed, I agree on this point, though I was assuming that products which
already aggregate connections are on the server side and definitely expect
the next hop to be 1.1. And I have no trouble fixing this as granted as it
can be a prerequisite for the feature to be enable and it can be checked in
previous responses.

> Would that be workable for you?

I'm not sure what those sort of agreement means, and in fact I'm not even
sure that we have solid ways to cover all situations. Still, some products
already do that (possibly at some risks, I don't know). I regularly get
strong requests from users to implement the same features in haproxy, and
while I agree it makes sense to implement those, my reading of the spec
does not make me 100% comfortable with it, because I think that at some
times we have to ignore some important rules.

My real concern is to ensure that such an implementation does not lie to
the user, and for instance that if such a gateway is installed in front
of a server and it decides to automatically retry a request over a new
connection, the visitor will not receive two products in his mail box
and will not have is account debitted twice.

The only "reliable" solution I have against this is for the gateway to
break the client connection if the server connection dies, and let the
client retry. While this can work for the 2nd and subsequent requests
from the client, if it does this for the first request of the client's
connection, the client will display an error and will not retry. So as
you see, a gray area remains (at least for me) here.

Best regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 Jun 2011 16:08:52 +1000

> times we have to ignore some important rules.

Well, a gateway / reverse proxy (like haproxy) is implicitly a device that has an agreement with the server; the protocol it speaks can be HTTP/1.1, but because it's a gateway, it can be something else too. We need to make this more apparent generally, I think.

> and will not have is account debitted twice.

Right. If it's vanilla HTTP, and the request is a POST, it can't retry it. If it's not HTTP (or HTTP++), it has some other kind of agreement covering this situation.

> you see, a gray area remains (at least for me) here.

Why is it different for the first request?

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 7 Jun 2011 08:39:53 +0200

> Well, a gateway / reverse proxy (like haproxy) is implicitly a device that has an agreement with the server; the protocol it speaks can be HTTP/1.1, but because it's a gateway, it can be something…

OK, I agree on this point.

> Right. If it's vanilla HTTP, and the request is a POST, it can't retry it.

That was my understanding too, hence the Expect tricks I suggested.

> If it's not HTTP (or HTTP++), it has some other kind of agreement covering this situation.

Let's only focus on HTTP since it's what we're interested in here, and where
the difficulty appears.

> Why is it different for the first request?

Because RFC2616 expects a client to retry a request over a keep-alive
connection that has just died, and my observations is that various browsers
get it right. However, when it's the first request over this connection,
they know that the closed connection is not a keep-alive timeout and it
indicates a server error. As such, all browsers I have tested immediately
report a connection error if the connection is broken during the first
request.

For this reason it makes it hard to multiplex incoming requests over
existing connections, because in theory a POST will always need a fresh
new connection to the server, or will have to silently retry (forbidden)
or to make use of some Expect tricks as I described in the first mail.
Maybe there are other cleaner ways but I haven't found them right now.

And the fact that I see this problem as complex while most people I talk
to just shake their shoulders saying things starting with "bah you just
have to ..." makes me think that it's not necessarily obvious to get the
various corner cases from a quick reading of the spec (or that I'm really
dumb, that's possible too).

Cheers,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 Jun 2011 17:02:45 +1000

On 07/06/2011, at 4:39 PM, Willy Tarreau wrote:

> get it right.

Are you referring to the requirement in p1 7.1.4, p1 7.2.4, or something else here?

> However, when it's the first request over this connection,

I don't see a requirement to do that anywhere in the spec; if anything, 7.1.4 SHOULD-requires clients to retry in this circumstance, unless the request is non-idempotent.

Are you reading those requirements as only applying to the second or later request on a connection because 7.1.4 falls under a section called "Persistent Connections"? If so, I don't think that's an intentional constraint on those requirements, and I'd be interested to hear what others think.

> Maybe there are other cleaner ways but I haven't found them right now.

Agreed. 

> dumb, that's possible too).

OK. Can you suggest things to add to or take away from the spec as it sits?

Thanks,

---

From: Dzonatas Sol <[email-masked]>
Date: Tue, 07 Jun 2011 08:59:37 -0700

> OK. Can you suggest things to add to or take away from the spec as it 

Since "multiplexing" was brought up, then the Expect: limits that 
ability to just one status code. There would need to be something like 
an outer and inner Expect: with different code to denote the continue on 
current stream or something to wait for the token of another stream.

Consider this code (what clouds want, a.k.a. "otoy") with HTML5 <canvas>:

"""
var Canvas = require('canvas'), canvas = new Canvas(300, 300), ctx = 
canvas.getContext('2d'), fs = require('fs');
var out = fs.createWriteStream('example.png'), stream = 
canvas.createPNGStream();
stream.on('data', function(chunk) {  out.write(chunk); });
"""

That would allow chunked mode to act like frames. It's nice to have that 
mode supported for it's raw-ness. I think it may be more viable to 
script an X11 framebuffer in that chunked-stream rather than multiplex 
outside of that, otherwise mime-parts, XMPP, and now WebRTC already 
exist. The above code, however, exploits how javascript "expects" to 
control the connection state.

--

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 8 Jun 2011 20:05:37 +0200

Hi Mark,

> Are you referring to the requirement in p1 7.1.4, p1 7.2.4, or something else here?

In RFC2616 it was 8.1.4, but yes it's basically the same.

> Are you reading those requirements as only applying to the second or later request on a connection because 7.1.4 falls under a section called "Persistent Connections"? If so, I don't think that's a…

I'm indeed reading it that way and the fact that browser's behaviours I've
observed seems to be similar makes me think I was not the only one. It's
very possible too that since handling retries is not the easiest thing to
do, most implementations only started to care about that with persistent
connections, and only implemented it in this case after reading section
8.1 of 2616 (persistent conns).

> OK. Can you suggest things to add to or take away from the spec as it sits?

I'm really embarrassed about how to proceed from now on. As a gateway
implementer, I expected to be able to impolitely shut the client-side
connection upon server-side errors and let the client decide whether
to retry or not. After all, unexpected server-side closures should not
be *that* common. But given that existing browsers will not retry, even
if we clarify the spec to ensure that new versions get this right in the
future, intermediaries will still have to support existing products.

So maybe we could move the "retry" text out of the "persistent" section,
but we'll still have to ensure there is a working solution for today. If
it's not that much complicated, we could suggest it as a hint, just as
we have in 7.2.4 about the exponential backoff. After all, the more hints
we give for complex situations, the more traps will be avoided and the
less ugly workarounds will be deployed. For instance, I've seen people
who were completely at ease with rewriting requests to replace POST with
GET to workaround nasty issues (!), I'd like to ensure nobody gets this
idea in order to force an intermediary to blindly retry non-idempotent
request when it prefers to close and rely on the client to retry !

I have not checked how various products handle the situation right now,
but I'm already afraid of what we could encounter :-/

I'm willing to propose something if it can be useful, but I'm not sure I'm
able to cover all corner cases, so I will need several iterations with
suggestions from more experienced people if we want to get something that
can be put in the draft.

Cheers,
Willy

---

From: Dzonatas Sol <[email-masked]>
Date: Wed, 08 Jun 2011 13:08:34 -0700

> but I'm already afraid of what we could encounter :-/

Yes, especially when people design queries in ways just for maximal 
page-hits under ambiguity of reasons for 1:1 query per hit ratios. With 
that weight, the evolution of HTTP streams (by persistence) does not 
seem effectual (at least in IPv4).

Maybe solutions specifically for UDP in an HTTP stream is not so 
obvious. OAuth MAC appears like the big toe dip to little more than test 
that temperature of the water. If successful then people may diagnose 
such cascade of streams easier, feel the dynamics, and optionally allow 
the length of such UDP (over OAuth) be maximally determined by 
"Content-Length:". Note that I assumed the possibility of IPv4 UDP 
traffic over IPv6 HTTP streams in this path and gateways can assume all 
IPv4 UDP traffic is paravirtual on IPv6 only networks.

On that wonder, maybe "Pragma:" or "Cache-Control:" helps determine 
other factors for HTTP retries, analytic ratios, structured UDP types, 
and globally quantized HTTP headers (into schema).

Welcome IPv6.

--

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 24 Jun 2011 16:25:10 +1000

> future, intermediaries will still have to support existing products.

I think we should figure out whether we need consistent behaviour regarding retries between products, or it's just something that we allow implementations to do, based on the known semantics of some request methods.

I tend to view it in the latter way; given that you can never guarantee that clients won't see failure, retries are just an optimisation, not something we need full interop on. 

This includes user agents as well as intermediaries, of course. In fact, I know of some UAs who definitely don't retry, and don't want to (because of the added complexity it, as well as the view that they're a "low-level" HTTP implementation).

> less ugly workarounds will be deployed.

I think we're on the same page.

> For instance, I've seen people

My current thinking is to life the retry-specific text out of 7.1.4 into a new section, possibly adding advice (along the lines you mention). 

I'm inclined to propose removing section 7.2.4 altogether, because it's very specific to a certain kind of request (one with a body where the connection drops to a proxy or HTTP/1.0 server, when there isn't an Expect: 100-continue header), and doing so can result in the server seeing a non-idempotent request being repeated multiple times, which is bad.

I realise that's a fairly big change. Is anyone aware of any actual implementation of 7.2.4?

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 25 Jun 2011 08:18:46 +0200

> I tend to view it in the latter way; given that you can never guarantee that clients won't see failure, retries are just an optimisation, not something we need full interop on. 

Retries on persistent connections should remain mandatory, as there is
no way to know that the connection is closing while sending a request.

> This includes user agents as well as intermediaries, of course. In fact, I know of some UAs who definitely don't retry, and don't want to (because of the added complexity it, as well as the view th…

There is still a minor difference between UAs and intermediaries : the UA
holds all the request if it wants to retry, while the intermediary will
always have size limitations preventing it from resending some reqeusts
(especially the ones with a body).

> I realise that's a fairly big change. Is anyone aware of any actual implementation of 7.2.4?

It is possible that some of them exist, it's been noted as 8.2.4 in RFC2616
for the last 12 years :-/

But I agree with you that it covers such a specific case that it could be
fold into a more general case.

Also I suddenly have an idea : why shouldn't we recommend that an
intermediary which must retry but can't would send a 5xx status
explicitly asking the client to retry ? It could be a lot cleaner
than closing and expecting the client to retry on a failed first
request, and even if the client does not handle this specific 5xx
status, at least it would get an indication about what went wrong.
Maybe it could be a 503 + Retry-After.

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 26 Jun 2011 16:49:06 +1000

> no way to know that the connection is closing while sending a request.

They're SHOULD NOTs now, and I suspect that many implementations don't do it.

> for the last 12 years :-/

Yes, but if it's problematic for interop we can change it. 

What do other folks think about this? Personally, I'm concerned about clients retrying POSTs; I think this will surprise many folks server-side, and can cause some problems. Semantically, POST (for example) is very clearly non-idempotent, and retrying under these circumstances opens a race condition.

> Maybe it could be a 503 + Retry-After.

I think 504 is more appropriate for a proxy, and the ability to retry is implicit in its semantics (I don't know of any software that automatically retries based upon a Retry-After header). 

Effectively, this is what happens already (after all, you have to give up retrying *sometime*), but I don't see any harm in spelling that out.

I've created <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/297> to track this.

Cheers,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 26 Jun 2011 08:24:50 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

[re: retrying POST/PUT etc]

If we loosened the language of "Expect: 100-continue", that could
be used to solve the problem:  There is no race in retrying if you
do not receive the "100 Continue" from the server.

Presently a "MUST NOT" prevents using "Expect: 100-continue" for non-body
methods, that would have to be relaxed.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 26 Jun 2011 17:53:01 +0200

Hi Mark,

> They're SHOULD NOTs now, and I suspect that many implementations don't do it.

When I performed some tests, both firefox and MSIE 6 were able to retry the
second request on a persistent connection if that connection was closed
before a response would be sent. However I must admit that my tests were
only for simple GET requests, there was no body.

> Yes, but if it's problematic for interop we can change it. 

True.

> What do other folks think about this? Personally, I'm concerned about clients retrying POSTs; I think this will surprise many folks server-side, and can cause some problems. Semantically, POST (for…

I agree that retrying a POST is *always* problematic, because the only one
who knows whether it's safe is the application. That said, it's not a problem
until all the body has been sent. But in general, the most sensible retries
will have small bodies (eg: delete first mail in a webmail, or validate a
payment).

Still we must offer provisions for UAs to guess if it's safe to emit a non
idempotent request over an existing connection. Expect is fine when the
server is 1.1. It becomes trickier if the body is empty. Maybe we could
reintroduce the Keep-Alive header with clear semantics ?

> I think 504 is more appropriate for a proxy, and the ability to retry is
> implicit in its semantics

504 is commonly encountered when the server fails to process the request
in time because of external dependency (webservice, database, etc...).
In my experience, it's very common to encounter 504 in complex
infrastructures, but the request are fully received and eventually
processed. It's just that a gateway between the client and the server has
enforced a timeout which is shorter than what it took the server to process
the request.

502 would be cleaner since it corresponds to a failure to send the whole
request to the next step. I proposed 503 because I saw that it supports
the Retry-After header, but as you pointed, it does not seem that UAs are
implementing it anyway.

> (I don't know of any software that automatically retries based upon a Retry-After header). 

My quick tests seem to agree with you :-/

> I've created <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/297> to track this.

Thanks. I'll keep trying to figure out solutions which require minimal effort
for each participant in a communication.

Cheers,
Willy