Subject: Fwd: New Version Notification for draft-nottingham-httpbis-retry-01.txt

From: Mark Nottingham <[email-masked]>
Date: Wed, 1 Feb 2017 19:26:29 +1100

FYI; fairly minor update. Would love to hear what people think about the various suggested paths forward.

Cheers,

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-nottingham-httpbis-retry-01.txt
> Date: 1 February 2017 at 7:25:43 pm AEDT
> To: "Mark Nottingham" <[email-masked]>
> 
> 
> A new version of I-D, draft-nottingham-httpbis-retry-01.txt
> has been successfully submitted by Mark Nottingham and posted to the
> IETF repository.
> 
> Name:		draft-nottingham-httpbis-retry
> Revision:	01
> Title:		Retrying HTTP Requests
> Document date:	2017-02-01
> Group:		Individual Submission
> Pages:		18
> URL:            https://www.ietf.org/internet-drafts/draft-nottingham-httpbis-retry-01.txt
> Status:         https://datatracker.ietf.org/doc/draft-nottingham-httpbis-retry/
> Htmlized:       https://tools.ietf.org/html/draft-nottingham-httpbis-retry-01
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-nottingham-httpbis-retry-01
> 
> Abstract:
>   HTTP allows requests to be automatically retried under certain
>   circumstances.  This draft explores how this is implemented,
>   requirements for similar functionality from other parts of the stack,
>   and potential future improvements.
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Mike Bishop <[email-masked]>
Date: Wed, 1 Feb 2017 18:50:29 +0000

Happy to see this updated.

Ironic note:  I currently own two different HTTP stacks with divergent behaviors here.  In the past week, we have received two bug reports about this topic.

  *   One of the stacks doesn't retry failed POST requests; this was reported as a bug, because a failed request should be retried if it didn't reach the server
  *   One of the stacks retries failed POST requests in some circumstances; this was reported as a bug, because the RFC says POSTs shouldn't be retried automatically

From: Mark Nottingham [mailto:[email-masked]]
Sent: Wednesday, February 1, 2017 12:26 AM
To: HTTP Working Group <[email-masked]>
Subject: Fwd: New Version Notification for draft-nottingham-httpbis-retry-01.txt

Cheers,

Begin forwarded message:

From: [email-masked]<mailto:[email-masked]>
Subject: New Version Notification for draft-nottingham-httpbis-retry-01.txt
Date: 1 February 2017 at 7:25:43 pm AEDT
To: "Mark Nottingham" <[email-masked]<mailto:[email-masked]>>

Abstract:

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

---

From: Tom Bergan <[email-masked]>
Date: Wed, 1 Feb 2017 12:41:18 -0800

> Applications sometimes want requests to be retried by
> infrastructure, but can't easily express them in a non-idempotent
> request (such as GET).

nit: did you mean "in an idempotent request (such as GET)"?

> A client SHOULD NOT automatically retry a failed automatic retry.

Why does RFC 7230 say this? I am aware of HTTP clients that completely
ignore this suggestion, and I can't offhand think of a reason why this is a
good rule-of-thumb to follow.

On Wed, Feb 1, 2017 at 12:26 AM, Mark Nottingham <[email-masked]> wrote:

> *From: [email-masked]
> *Subject: **New Version Notification for
> draft-nottingham-httpbis-retry-01.txt*
> *Date: *1 February 2017 at 7:25:43 pm AEDT
> *To: *"Mark Nottingham" <[email-masked]>
>
>

---

From: Matt Menke <[email-masked]>
Date: Wed, 1 Feb 2017 16:30:25 -0500

A bit more elaboration on Chrome's behavior:

I didn't see it mentioned in the draft, but the reason we retry is that any
time we reuse a socket, there's a race - servers timeout sockets after some
period, and we don't know what that period is.  So we could be retrying at
the same time the server is closing the socket on us.

We'll retry on a "stale" socket (one that was connected and then sat in a
socket pool, with or without being used first).  So if we fully preconnect
to a server before there's a connection request, we consider the socket
stale,if we've used a socket before, it's stale, or if we created a socket
to service a request, but the request was cancelled, and then the socket
connected and was returned to the socket pool, it's stale.

We only retry if we've received no response from a server, or if we get a
failure while still sending the request (i.e., if we're in the middle of
sending the request on error, we'll retry, even if there's data on the
socket from the server, since we don't bother to check for that).

We also only retry on certain errors (reset, connection closed, connection
aborted (Which is a weird error from the OS of some sort), H2 ping failure,
H2 server refusing the stream, QUIC handshake error.  This probably doesn't
include all the errors that we could get racily while reusing a socket (TCP
ping timeout, for instance?).

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Feb 2017 12:20:30 +1100

> On 2 Feb 2017, at 7:41 am, Tom Bergan <[email-masked]> wrote:
> 

Thanks, fixed in source.

> Why does RFC 7230 say this? I am aware of HTTP clients that completely ignore this suggestion, and I can't offhand think of a reason why this is a good rule-of-thumb to follow.

Good question. The immediate answer is that RFC2616 said it, and RFC2068 said it before that (and apparently introduced the requirement). 

If we end up revising the text regarding retries, that's something we should consider updating too.

Cheers,

---

From: Tom Bergan <[email-masked]>
Date: Wed, 1 Feb 2017 17:26:38 -0800

On Wed, Feb 1, 2017 at 5:20 PM, Mark Nottingham <[email-masked]> wrote:

> Good question. The immediate answer is that RFC2616 said it, and RFC2068
> said it before that (and apparently introduced the requirement).
>

Maybe it there was a concern about accidental DoS? Infinite retries are
probably a bad idea without exponential backoff.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Feb 2017 12:28:11 +1100

> On 2 Feb 2017, at 12:26 pm, Tom Bergan <[email-masked]> wrote:
> 
> Maybe it there was a concern about accidental DoS? Infinite retries are probably a bad idea without exponential backoff. 

I suspect so. This algorithm doesn't seem to match reality, as you point out, and so we'd need to either specify a different one, or just loosen it to a more general requirement.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 2 Feb 2017 09:14:30 +0100

Hi Mark,

> FYI; fairly minor update. Would love to hear what people think about the various suggested paths forward.

Quite an interesting document. The questions left open are even more
difficult for an intermediary because as you've mentionned, methods are
not sufficient to guess application's idempotency and everyone seems to
expect that transient network issues should be hidden from the end user.

I'm even wondering if applications should pass some information to the
client to indicate a guaranteed idempotency that could be indicated back
along the chain by a header. We could thus imagine to replay such POSTs
when the application uses some anti-replay request IDs. Regarding GET,
there's probably no point asking applications to pass such information
because changing them to do this is as difficult as expecting them to
use POST instead, so we should probably continue to consider that GETs
are idempotent.

I found something which can be slightly improved in 4.1 regarding the
detectable conditions for a retry. "Connection closes" and TCP RST
could be cut into to very distinct categories :
  - those which happen without ACKing the request
  - those which happen after ACKing the request

In the first case it's mostly a race on a idle timeout causing the server
(or an intermediary) to close at the same time the client sends the request.
It is safe to retry because it is guaranteed at the TCP layer that the data
were not consumed.

In the second case you don't know. The data may simply have been delivered
to TCP socket buffers (and ACKed) at the moment the application closed the
listening socket (eg during an application reload). Or the request may have
been delivered to the application, causing it to crash, explaining the close
or reset.

I'd argue that the first ones SHOULD be retried (with reasonable efforts)
while the second ones SHOULD NOT unless the request's idempotency has been
figured out.

One difficulty is that current TCP stacks don't make it easy to find if
data was ACKed and can be dropped. In fact the problem is not the moment
the reset/close event is detected, but the moment before. For an
intermediary it's not possible to buffer infinite amounts of data, so
it's needed to drop transmitted data to make buffer room for more data
(eg for a POST request). And data being ACKed doesn't translate into an
event that can suddenly cause a paused connection to be woken up
(SNDLOWAT 1) usually is ignored. For a client (browser) the issue is
the same except that all elements having lead to the request should
probably still be present to build a completely new request from scratch
so the network buffering issue should not exist.

So in the end we do have some reliable transport-level information to
detect the conditions for a safe retry of non-idempotent requests but
these information are not easy to pick and exploit. It's a particular
issue for TCP Fast Open implementation as a client (I have implemented
TCP Fast Open to servers in haproxy but not merged it yet since I have
not sorted this out yet).

Cheers,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 3 Feb 2017 17:15:29 +1100

Thanks for that. If you can, a pull request on the doc would be wonderful; you can capture the detail better than I.

See:
  https://github.com/mnot/I-D/blob/gh-pages/httpbis-retry/draft.md

Cheers,

> On 2 Feb 2017, at 8:30 am, Matt Menke <[email-masked]> wrote:
> 
> I didn't see it mentioned in the draft, but the reason we retry is that any time we reuse a socket, there's a race - servers timeout sockets after some period, and we don't know what that period is.  So we could be retrying at the same time the server is closing the socket on us.
> 
> We'll retry on a "stale" socket (one that was connected and then sat in a socket pool, with or without being used first).  So if we fully preconnect to a server before there's a connection request, we consider the socket stale,if we've used a socket before, it's stale, or if we created a socket to service a request, but the request was cancelled, and then the socket connected and was returned to the socket pool, it's stale.
> 
> We only retry if we've received no response from a server, or if we get a failure while still sending the request (i.e., if we're in the middle of sending the request on error, we'll retry, even if there's data on the socket from the server, since we don't bother to check for that).
> 
> We also only retry on certain errors (reset, connection closed, connection aborted (Which is a weird error from the OS of some sort), H2 ping failure, H2 server refusing the stream, QUIC handshake error.  This probably doesn't include all the errors that we could get racily while reusing a socket (TCP ping timeout, for instance?).
> 
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 3 Feb 2017 22:03:40 -0800

> On Feb 1, 2017, at 12:41 PM, Tom Bergan <[email-masked]> wrote:
> 

This is only referring to retries due to a dropped connection. The reason is because a
second connection drop is (in almost all cases) due to the request itself, as opposed to
something transient on the network path.  [BTW, this doesn't refer to requests yet to be
sent in a request queue or pipeline -- just the retried request in flight for which no response
is received prior to FIN/RST (or equivalent).]

There might be a good reason to go ahead and retry with an exponential back-off,
but I don't know what that would be in general. I know lots of clients do stupid
things because they are afraid of communicating server errors to their user.

....Roy

---

From: Tom Bergan <[email-masked]>
Date: Sat, 4 Feb 2017 11:24:43 -0800

> is received prior to FIN/RST (or equivalent).]

There seem to be many underlying assumptions. I'm glad that Mark is making
things more explicit. For example, "a second connection drop is (in almost
all cases) due to the request itself" likely does not hold in flaky
cellular networks, such as getting on/off a subway. This is why Chrome will
automatically retry a failed page load when it detects a change in network
connectivity. Also think of the "parking lot problem" where you transition
between WiFi in the office and cellular in the parking lot.

I'm not sure your claim is strictly true even assuming good network
connectivity. A data point: I help run a very large proxy service. When we
experimented with automatic retries, we saw a ~40% success rate on the
first retry, ~20% on the second retry, and ~5% on the third retry. We only
retried GETs and HEADs where the connection was dropped before we received
a single byte of response. That ~20% success rate is arguably high enough
to make a second retry worthwhile (depending on your POV).

That said, I'm not trying to convince you that HTTP clients should send
multiple retries. Rather, I'm surprised that the HTTP spec takes a position
on this question in the first place. Retry logic seems like a fundamentally
application-specific property, beyond really basic things like "requests
with idempotent methods must be safe to retry."

There might be a good reason to go ahead and retry with an exponential
> back-off,
> but I don't know what that would be in general.

It is common to use exponential backoff with RPC-over-HTTP (one example
<https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md>).
Speaking of RPC-over-HTTP, in Section 2.1 of Mark's document, perhaps we
could generalize "User Retries" to "Application Retries"? As written,
Section 2.1 is kind of specific to browsers and it's not clear how
non-browser applications fit in. RPC could be one such application. Also
think of programs like daemons that make HTTP requests (often
RPC-over-HTTP) without any direct prompting from a user.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 6 Feb 2017 14:04:30 -0700

> various suggested paths forward.

FWIW, Squid mind-boggling algorithm for retries is partially summarized
at
http://wiki.squid-cache.org/SquidFaq/InnerWorkings?highlight=%28reforward%29#When_does_Squid_re-forward_a_client_request.3F

Your draft already mentions a single Squid decision point, but the
actual logic is a lot more complex than the draft currently implies.
Some of that complexity is Squid's fault, as the source code comment you
quoted illustrates, but a lot of it is genuine.

Recommending a unified (but necessarily parameterized) approach to
retries would be useful for future implementors, but I suspect that
doing so properly would take too much time while oversimplifying the
situation would not help much. Just cataloging various retry factors to
consider may be very helpful on its own, even if you then suggest
nothing more than "keep these factors in mind when implementing retries".

Thank you,

Alex.

>> Begin forwarded message:
>>
>> *From: [email-masked] <mailto:[email-masked]>
>> *To: *"Mark Nottingham" <[email-masked] <mailto:[email-masked]>>
>>
>>
>> Name:draft-nottingham-httpbis-retry
>> Revision:01
>> Title:Retrying HTTP Requests
>> Document date:2017-02-01
>> Group:Individual Submission
>> Pages:18
>> URL:
>>            https://www.ietf.org/internet-drafts/draft-nottingham-httpbis-retry-01.txt
>> Status:
>>         https://datatracker.ietf.org/doc/draft-nottingham-httpbis-retry/
>> Htmlized:
>>       https://tools.ietf.org/html/draft-nottingham-httpbis-retry-01
>> Diff:
>>           https://www.ietf.org/rfcdiff?url2=draft-nottingham-httpbis-retry-01
>>
>> Abstract:

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 Feb 2017 14:35:32 +1100

> On 7 Feb 2017, at 8:04 am, Alex Rousskov <[email-masked]> wrote:
> 

Ah, I should have remembered that one; thanks. I've added a link.

> nothing more than "keep these factors in mind when implementing retries".

That's a reasonable outcome. Some folks might want more, e.g., a recommended algorithm that itself isn't mandatory for all implementations (yet). Whether we go that far (and when) is still an open question, I think.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 7 Feb 2017 08:31:11 +0100

> That's a reasonable outcome. Some folks might want more, e.g., a recommended algorithm that itself isn't mandatory for all implementations (yet). Whether we go that far (and when) is still an open…

I like a lot the way it's presented in Squid's doc above. It's indeed
much better to enumerate a number of conditions that should prevent a
retry than to suggest when it's possible. As seen above, that makes
the list much easier to understand (shorter and simpler conditions)
and easier to adapt to other products.

HAProxy currently doesn't implement retry if at least one byte was sent.
The main reason is that once forwarded, these bytes are lost, but I
contemplated the possibility to pin and rewind the buffer to deal with
failures on persistent shared connections (by default we don't share them).
I've come to a number of conditions (much less) and some are common with
Squid's, so it's encouraging to read this :-)

I tend to think that just like we audited many implementations when
trying to redefine the state of deployed stacks for RFC723x, by
enumerating what various implementations do combined with their successes
or failures, we may end up with MAY and SHOULD that can be considered as
safe guidelines for future implementations, just because existing code
has had to adapt to what these current implementations do as well.

Cheers,
Willy

---

From: Wenbo Zhu <[email-masked]>
Date: Fri, 10 Feb 2017 19:16:57 -0800

> "with some sites even requiring browsers to retry POST requests in order
to properly interoperate"

Do we know the exact reason (and scale) behind such a behavior?

---

From: Mike Bishop <[email-masked]>
Date: Sat, 11 Feb 2017 04:23:33 +0000

Some assorted reasons show up in the discussion at https://www.youtube.com/watch?v=uQKZIiecqDE&list=PLC86T-6ZTP5i0Bp03yAK5oWrxMMwBjIq7&index=109 starting around 1:15:30.  Scale, I’m not sure; Patrick had the direct experience and might be able to tell more.

From: Wenbo Zhu [mailto:[email-masked]]
Sent: Friday, February 10, 2017 7:17 PM
To: Mark Nottingham <[email-masked]>
Cc: HTTP Working Group <[email-masked]>; Roy T. Fielding <[email-masked]>
Subject: Re: New Version Notification for draft-nottingham-httpbis-retry-01.txt

> "with some sites even requiring browsers to retry POST requests in order to properly interoperate"

On Fri, Feb 3, 2017 at 10:03 PM, Roy T. Fielding <[email-masked]<mailto:[email-masked]>> wrote:
> On Feb 1, 2017, at 12:41 PM, Tom Bergan <[email-masked]<mailto:[email-masked]>> wrote:
>

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 11 Feb 2017 08:19:46 +0100

On Sat, Feb 11, 2017 at 04:23:33AM +0000, Mike Bishop wrote:
> Some assorted reasons show up in the discussion at
> https://www.youtube.com/watch?v=uQKZIiecqDE&list=PLC86T-6ZTP5i0Bp03yAK5oWrxMMwBjIq7&index=109
> starting around 1:15:30.  Scale, I'm not sure; Patrick had the direct
> experience and might be able to tell more.

Thanks for the link, Mike, that was quite an interesting talk! For haproxy
I planned to have an action to let the admin indicate that if certain
criteria match, then a request is idempotent. For example, a POST request
containing a transactionId argument, etc... I'm realizing that stuff like
this could be proposed to be generalized instead of having to be made
specific to each and every component. In short what is needed (from my
perspective) is :
  - a way for the application to signal the client that a request will
    be idempotent (could be either by adding a specific field to HTML
    forms, or passing a header field in the HTTP response saying that
    the exact same URI is safe for POSTs, etc).

  - a way for the client to notify the chain back to the origin that
    it's safe to replay the current request based on the info learned
    from the application. That could simply be a header field or a
    value added to a header field such as :
        content-type: application/x-www-form-urlencoded; safe

Just my 2 cents,
Willy