Subject: Retry safety of HTTP requests

From: Subodh Iyengar <[email-masked]>
Date: Tue, 22 Mar 2016 05:55:33 +0000

HTTP/1.1 and the current HTTP/2 spec don't define the concept of retry safety of requests, i.e. when are requests are safe to retry and what are the limits on how retries can be performed by the lower layers.

I believe this is a new item and I'm trying to gauge people's interest in this topic, and whether such an idea has been explored before in the httpwg.

I think defining retry safety will become an important problem in the near future. I can see several use cases if we have a proper definition of retry safety in HTTP:

1) Usually an application like a piece of javascript will create a HTTP request and send it to a browser or mobile app which executes it. The browser or a mobile app might want to retry the request. In unreliable networks, retrying requests is important for reliability. The application could decide that it would create a non idempotent request like a POST, and have an application layer mechanism to enforce idempotence. It could then tell the browser that the request was retry safe because it was enforcing idempotence, and let the browser perform optimizations to retry it depending on network conditions, including caching the request offline, retrying when the network comes back, sending the request across 2 connections on different interfaces, etc.

2) Some transport protocols which treat retry safe requests differently from non retry safe requests. For example in TLS 1.3, idempotent requests may be sent in a 0-RTT flight, which reduces the latency for the request. An application might desire a non idempotent request which is retry safe be sent in this 0-RTT flight.

3) A non-application server such as a load balancer might also desire to know whether or not the client expressed that this request is retry safe so that it may retry this request if the backend fails over offering more reliability to clients.

Currently, the only semantics for an application to express retry safety is via idempotency (the request method). The status quo shows that even this is not expressive enough, since even though GETs are not technically idempotent, some browsers will retry them anyway.

Having an document which would define a new property like retry-safety would help apps take advantage of reliability features of browsers for requests other than GETs and also prepare HTTP to be able to use TLS1.3 effectively.

In our Facebook mobile apps we annotate all the requests that our app makes as retry safe or not and that helps us decide how retry policies should behave for each request. It's been very useful for request reliability.

Would the HTTP-wg be interested in a document describing retry-safety and its limitations?

Cheers,
Subodh Iyengar

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 22 Mar 2016 20:32:38 +1300

On 22/03/2016 6:55 p.m., Subodh Iyengar wrote:
> HTTP/1.1 and the current HTTP/2 spec don't define the concept of
> retry safety of requests, i.e. when are requests are safe to retry
> and what are the limits on how retries can be performed by the lower
> layers.

RFC 7230 section 6.3.1 titled "Retrying Requests"
<http://tools.ietf.org/html/rfc7230#section-6.3.1>

The criteria for *when* are outlined.

The *how* is not possible to be defined at the HTTP layer IMO. There are
just so many possible ways. Up to and including things like RFC 1149
Avian Carrier protocol as fallback.

So the "how" is application and indeed message specific and should be
defined at those upper layers. HTTP already provides a number of
mechanisms such as; GET, HEAD methods and/or ETag, If-*, Range headers,
etc. which can be used by the application layer to check the state
before its chosen method of retry (if wanted).

> 
> I believe this is a new item and I'm trying to gauge people's
> interest in this topic, and whether such an idea has been explored
> before in the httpwg.
> 
> I think defining retry safety will become an important problem in the
> near future. I can see several use cases if we have a proper
> definition of retry safety in HTTP:
> 
> 1) Usually an application like a piece of javascript will create a
> HTTP request and send it to a browser or mobile app which executes
> it. The browser or a mobile app might want to retry the request. In
> unreliable networks, retrying requests is important for reliability.
> The application could decide that it would create a non idempotent
> request like a POST, and have an application layer mechanism to
> enforce idempotence.

Application layer mechanisms are outside the scope of HTTP.

HTTP is not even the only protocol such messages can be sent over. There
is HTTP (1.x, 2), WebSockets, SPDY, QUICK, COAP(S),  and others all in
use various places.

> It could then tell the browser that the request
> was retry safe because it was enforcing idempotence, and let the
> browser perform optimizations to retry it depending on network
> conditions, including caching the request offline, retrying when the
> network comes back, sending the request across 2 connections on
> different interfaces, etc.

It tells the HTTP layer by using the correct RESTful method for the
semantic operation that message seeks to perform:
- a fetch (GET)
- a store (PUT)
- a removal (DELETE)
- a submission (POST)
- a state query (HEAD)
- a service capabilities query (OPTIONS, TRACE)

All HTTP software understands those semantics regardless of what other
extensions it may or may not support. Or how old it is.

> 
> 2) Some transport protocols which treat retry safe requests
> differently from non retry safe requests. For example in TLS 1.3,
> idempotent requests may be sent in a 0-RTT flight, which reduces the
> latency for the request. An application might desire a non idempotent
> request which is retry safe be sent in this 0-RTT flight.

Do not confuse transport layer frame/message with transfer layer
frame/message.

The semantics of the transfer layer (HTTP) message and even its framing
boundaries are opaque to the transport layer (TLS).

If the TLS/1.3 layer is going to perform packet retry itself in the
presence of 0-RTT flight data, then it needs to ensure that opaque data
is only ever delivered to a recipient once regardless of how many times
its actually sent on-wire. To be a "reliable transport" it needs to
guarantee that property.

I find it hard to believe that the TLS WG would allow TLS/1.3 to become
an unreliable transport layer.

> 
> 3) A non-application server such as a load balancer might also desire
> to know whether or not the client expressed that this request is
> retry safe so that it may retry this request if the backend fails
> over offering more reliability to clients.
> 
> Currently, the only semantics for an application to express retry
> safety is via idempotency (the request method). The status quo shows
> that even this is not expressive enough, since even though GETs are
> not technically idempotent, some browsers will retry them anyway.

What? thats false.

RFC 7231 section 4.2.2:
 "PUT, DELETE, and safe request methods are idempotent".

RFC 7231 section 4.2.1:
  "GET, HEAD, OPTIONS, and TRACE methods are defined to be safe".

thusly "idempotent" == PUT, DELETE, GET, HEAD, OPTIONS, TRACE

plus any other extension method defined as being idempotent or safe.

Note that certain response status codes also indicate explicitly that
nothing reached the origin. So their request is now retriable for
exactly +1 try despite its natural idempotency.

HTTP/2 extended that with GOAWAY to mark a range of pipelined request
streams.

IIRC we have a WG consensus that applications using methods counter to
their documented idempotency are not HTTP compliant. Such software is
explicitly left to face the consequences of its non-compliance when the
HTTP layer "unexpectedly" treats the message according to correct semantics.
Your application(s) treating GET as non-idempotent is a case for that.

> 
> Having an document which would define a new property like
> retry-safety would help apps take advantage of reliability features
> of browsers for requests other than GETs and also prepare HTTP to be
> able to use TLS1.3 effectively.

TLS is just a transport for HTTP. There is no reason (nor should there
be) for any difference in semantics of the HTTP messages than over
TLS/1.2 or TCP plain-text transports.

> 
> In our Facebook mobile apps we annotate all the requests that our app
> makes as retry safe or not and that helps us decide how retry
> policies should behave for each request. It's been very useful for
> request reliability.

Why not use the REST semantics of HTTP itself as the signal?

> Would the HTTP-wg be interested in a document describing retry-safety

I'm certainly interested to find out why HTTP semantics are suddenly so
insufficient that its needed. IME Facebook was turning into one of the
better HTTP participants before it abandoned for HTTPS-world.

Amos

---

From: Subodh Iyengar <[email-masked]>
Date: Tue, 22 Mar 2016 08:27:21 +0000

Thanks for the comments Amos. 

> Application layer mechanisms are outside the scope of HTTP.
I was presenting an example use case of retry safety by an application. The mechanism of indicating this is definitely outside of the scope of HTTP.

> its actually sent on-wire. To be a "reliable transport" it needs to

I should have added more context here.
The property of retry safety for TLS is not for reliability here, but for security of 0-RTT data.
In TLS1.3, the 0-RTT payload is not protected against replay attacks.
Only idempotent or retry safe data should be sent in the 0-RTT payload. This would enable
even requests which do not have idempotence or safe semantics to be used in 0-RTT data if
the application takes care of the idempotency. The exact mechanism for the application taking care
of this are outside the scope of HTTP.

> Do not confuse transport layer frame/message with transfer layer
frame/message.
I'm not. The idea here was to provide a mechanism to indicate to HTTP that this request is retry safe, and then the HTTP layer can propagate this information to the TLS client, hence it is free to send its data in the 0-RTT payload.

> What? thats false.

I was mistaken and you're correct on this. A lot of applications though practically do send non-idempotent GET requests. I can understand why the WG considers them non-compliant. However these considerations do not take into account restrictions of app developers for example security features like CORS, and I can imagine many applications that might use the methods like POST, GET. and HEAD to get around CORS restrictions to build legitimate features. 

> - a service capabilities query (OPTIONS, TRACE)

Even if we used the correct semantics for each operation, an application could want POST semantics yet want to allow the request to be retried.

> TLS/1.2 or TCP plain-text transports.

For the above mentioned reasons, the semantics of the HTTP messages are important for TLS1.3.

I agree *how* is difficult for HTTP to define, however HTTP can for example impose limits on retries, for example, you shouldn't retry a message whose response you've even partially delivered to the client. HTTP might also be able to define certain best practices for browsers that perform retries, and applications that ask for retries, and provide mechanisms for browsers to indicate to servers how many retries have been performed so far similar to :scheme and :path.

Subodh
________________________________________
From: Amos Jeffries [[email-masked]]
Sent: Tuesday, March 22, 2016 12:32 AM
To: [email-masked]
Subject: Re: Retry safety of HTTP requests

RFC 7230 section 6.3.1 titled "Retrying Requests"
<https://urldefense.proofpoint.com/v2/url?u=http-3A__tools.ietf.org_html_rfc7230-23section-2D6.3.1&d=CwIC-g&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=dqzc-LMd_ldrS1bDvcdcNQqv8Bqf6kkvaOyNBWSMeqg&s=OC5WwQavs-zI-QCuNCeV99nz6sTtsOcG23ffZlJH8yg&e= >

What? thats false.

RFC 7231 section 4.2.2:

RFC 7231 section 4.2.1:

Amos

---

From: Subodh Iyengar <[email-masked]>
Date: Tue, 22 Mar 2016 08:44:40 +0000

> Why not use the REST semantics of HTTP itself as the signal?
We have several POST mutations from the app, however the realistic scenario is that a lot of these requests will fail in bad networks, most times without a HTTP/2 GOAWAY. In that case what should we do? We have 2 choices, give up and assume that the server might have seen the data and leave the user with a bad experience, or build an application layer idempotency. The retry safety property allows our HTTP library to know that certain requests even mutating ones have idempotent properties and it is free to retry the requests even if the underlying request fails. 

> better HTTP participants before it abandoned for HTTPS-world.

I'm not sure I understand what you mean. I'm reaching out well intentioned way and asking for feedback.

Subodh
________________________________________
From: Subodh Iyengar [[email-masked]]
Sent: Tuesday, March 22, 2016 1:27 AM
To: Amos Jeffries; [email-masked]
Subject: RE: Retry safety of HTTP requests

frame/message.

Subodh

What? thats false.

RFC 7231 section 4.2.2:

RFC 7231 section 4.2.1:

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 22 Mar 2016 10:27:12 +0100

> before its chosen method of retry (if wanted).

Also for POST requests, Expect: 100-continue provides an easy way to
probe the connection and to ensure idempotence in case of failure.

Willy

---

From: Cory Benfield <[email-masked]>
Date: Tue, 22 Mar 2016 10:14:09 +0000

> On 22 Mar 2016, at 08:44, Subodh Iyengar <[email-masked]> wrote:
> 

I’m a little unclear on this.

How it retry-safety established from the perspective of the client? Is it application specific? If it is, there’s no need for this WG to get involved: your HTTP libraries can simply have a flag that marks some requests as retry-able (and indeed, the libraries I work on all provide this functionality).

However, I seem to understand your request as wanting retry-safety to be communicated over the HTTP channel itself. The only way I can see that happening is if a server communicates that fact: some kind of header, frame, or flag that says “this request can safely be retried”. That, of course, only works if the client actually receives the response.

Given that that doesn’t address your actual problem (when no response is received), I can only assume you’d be hoping for some form of caching where a client learns from the server what requests are safe to retry and then remembers that for the future. Such a model could absolutely work, but it’s not clear to me that there’s much gained over the standard retry logic based on idempotency and safety that HTTP already defines.

If that really is what you’re considering pursuing, I highly recommend using FB’s large scale and ability to grab analytics to implement this change and then instrument it. Have your server return these headers, and have your clients cache and act appropriately, then check how that affects user experience. You could even A/B test it.

With data in hand that proves the value of such a change, you’d have a lot more weight to make the argument that *some* signaling should be done at the HTTP layer.

Cory

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 22 Mar 2016 23:38:32 +1300

> We have several POST mutations from the app, however the realistic
> scenario is that a lot of these requests will fail in bad networks,
> most times without a HTTP/2 GOAWAY. In that case what should we do?
> We have 2 choices, give up and assume that the server might have seen
> the data and leave the user with a bad experience, or build an
> application layer idempotency.

Ah. This is where I would expect the application to use a unique ID in
the client POST and the server to ignore repeated IDs.

That is simple, relatively efficient and very much an application-layer
thing not related to the protocol itself.

> The retry safety property allows our
> HTTP library to know that certain requests even mutating ones have
> idempotent properties and it is free to retry the requests even if
> the underlying request fails.

I get the idea that this "library" is part of the client or server
endpoints. If that is correct, then from the HTTP protocol viewpoint it
is classified as part of the application-layer. Which in turn means that
the signal is only going between the library and some other layer within
the application itself.

In other words: The signal is not relevant to the protocol, just very
relevant to how that library portion of the application uses the protocol.

Which is very reasonable. But maybe not worthwhile extending the protocol.

> 
>> I'm certainly interested to find out why HTTP semantics are
>> suddenly so insufficient that its needed. IME Facebook was turning
>> into one of the better HTTP participants before it abandoned for
>> HTTPS-world.
> 

I meant only that I'm curious where is the new hole that you can see in
HTTP. As an old protocol with a wide amount of use-case testing it is a
bit of a surprise to find something so fundamental as retriability
falling short without already being a known problem for years.

You have somewhat clarified that with your responses. Thanks.

Yes, IMO this WG is the right place to discuss the topic. At least until
the powers that be says things have gone too far off-topic.

Amos

---

From: Patrick McManus <[email-masked]>
Date: Tue, 22 Mar 2016 10:47:59 -0400

> Would the HTTP-wg be interested in a document describing retry-safety and

It's not a passion of mine to work on, mostly because I think a lot of this
stuff will inherently be kicked to application space, but if you have an
idea for a draft that accurately describes what is going on and what needs
to go on I would certainly think that has potential as a valuable source
for interoperability. What are you thinking?

One example is pretty simple - POST on a persistent h1 connection. every
browser I'm aware of when POST-ing data to a previously used connection
will replay that POST if they get a RST or FIN before receiving any data
bytes. That's wrong from an idempotency pov of course, but dealing with the
reality of the race condition of the peer having closed the connection is
more important. Sometimes browsers will try not to do some forms of POST on
reused persistent connections for this reason, but that can't really be
generalized to higher transaction rate things like XHR.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Mar 2016 11:33:46 +1100

Hi Subodh,

My .02 - It sounds like what you're asking for is a "I really mean this is safe" flag for GET. 

Historically, this kind of mechanism doesn't work out very well, because if people misunderstand the original semantics (in this case, the safety and idempotency of GET), they're just as likely to misunderstand/misuse the "I really mean it" flag. Then, you need a "I really, REALLY mean it" flag, and so on.

Furthermore, motivating this with the use case that people are trying to work around security mechanisms like CORS isn't likely to work; it's be better to fix any problems in CORS than create workarounds in other specs*.

Explicitly allowing retries of POST (or POST-like methods) is something that might get more traction. Recently we've seen a proposal for a safe, idempotent SEARCH method; would that meet your needs? <https://tools.ietf.org/html/draft-snell-search-method>.

Another possibility would be a pattern of use that assured that non-idempotent requests weren't able to be retried; e.g., <https://github.com/mnot/I-D/blob/gh-pages/Abandoned/http-poe/draft-nottingham-http-poe-00.txt> (I abandoned this draft because it's not really a new protocol element, it's just a pattern of use.).

Cheers,

* CORS is now maintained as part of Fetch <https://fetch.spec.whatwg.org>, and Anne is very receptive to proposed improvements; I think a lot of people recognise its shortcomings.

P.S. Our "snow bunny" has been banned from the list; apologies for the noise.

> On 22 Mar 2016, at 7:27 PM, Subodh Iyengar <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Mar 2016 12:00:02 +1100

> Another possibility would be a pattern of use that assured that non-idempotent requests weren't able to be retried; e.g., <https://github.com/mnot/I-D/blob/gh-pages/Abandoned/http-poe/draft-notting…

That's one approach to the problem.  It's also possible to only use
POST for actions that don't have any real consequences.  For instance,
creation of a resource can be absurdly cheap.  If costly actions are
bound to idempotent methods, then you have a far more robust
application.

TLS 1.3 with 0-RTT is making some people nervous.  One of the things
that we're going to have to grapple with is how much of the
responsibility for replay we (in this working group) are able to take
on.

Now, as Amos outlined, there are many ways in which we can determine
that a retry is acceptable *according to the protocol as specified*.
The question of what is acceptable *according to the protocol as it is
used*.  There will be a subset of the first set that we want to
identify as applicable to 0-RTT [45], and maybe we need to pay a
little attention to the gap between theory and practice.  I suspect
that is where Subodh is coming from here.

[45] Obviously, anything that relies on a response from a server, like
421 or GOAWAY, isn't going to be a useful way to identify something as
being OK for use with 0-RTT.  For me, the big question is between safe
and idempotent (PUT especially).

---

From: Subodh Iyengar <[email-masked]>
Date: Wed, 23 Mar 2016 01:03:41 +0000

@Mark
Thanks for your comments. I'm more interested to have a "I really mean this is safe" for POST-like methods and dealing with methods that aren't already safe.

The mention of CORS was not to be used as a motivation for this but to highlight real world problems that people have. The more appropriate motivation is that "I'm in a bad network, I have to retry this POST, is it safe to do that?". It's unfortunate that got mixed up in the email ordering.

Patrick brought up a point that this could be handled in the application layer itself, and Amos brought up good points that this could be handled by the browser itself, and HTTP doesn't need to know about this. These are totally legitimate, and that's a layer to handle it. 

However, I think this could be useful to deal with in HTTP because people would end up accessing the same resources over mobile apps or browsers and it's probably useful to have a common meaning, especially if this could be used to express things like security properties of the request to the transport for TLS1.3.

@Patrick I was thinking of writing a draft and sending it to the HTTP workshop that's coming up. Maybe that's a good venue to discuss this.

Subodh
________________________________________
From: Mark Nottingham [[email-masked]]
Sent: Tuesday, March 22, 2016 5:33 PM
To: Subodh Iyengar
Cc: [email-masked]
Subject: Re: Retry safety of HTTP requests

Hi Subodh,

Explicitly allowing retries of POST (or POST-like methods) is something that might get more traction. Recently we've seen a proposal for a safe, idempotent SEARCH method; would that meet your needs? <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dsnell-2Dsearch-2Dmethod&d=CwIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=xVN9l9_sP6syJBnc6W9vKrug9wRBOrFgkmMAs-TbteE&s=iWWfZHGVJvuoN_CUfirnYHIs5eLjiJ3VRBBfYgC2j4I&e= >.

Cheers,

* CORS is now maintained as part of Fetch <https://urldefense.proofpoint.com/v2/url?u=https-3A__fetch.spec.whatwg.org&d=CwIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=xVN9l9_sP6syJBnc6W9vKrug9wRBOrFgkmMAs-TbteE&s=T0ixsfqeSSHggCjL88RL8WpxmPPAHB0IioI2X8yAFts&e= >, and Anne is very receptive to proposed improvements; I think a lot of people recognise its shortcomings.

--
Mark Nottingham   https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mnot.net_&d=CwIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=xVN9l9_sP6syJBnc6W9vKrug9wRBOrFgkmMAs-TbteE&s=mt5I39QkpkVJ75Kv2PIImSivFtEsbIwH9hHPI2zXY_M&e=

---

From: Mike Bishop <[email-masked]>
Date: Wed, 23 Mar 2016 02:21:54 +0000

Idempotency is useful against short-time replay, like just resending until you get a response.  However, 0-RTT would permit replay seconds, minutes, or more later, no?

If we’re talking about a pattern of DELETE, PUT, GET, the fact that every separate action is idempotent doesn’t save us from a replay of the DELETE after the PUT.  I’d probably start with “safe only”, possibly expanded to idempotent-with-preconditions.

Sent from my Windows 10 phone

From: Martin Thomson<mailto:[email-masked]>
Sent: Tuesday, March 22, 2016 6:07 PM
To: Mark Nottingham<mailto:[email-masked]>
Cc: Subodh Iyengar<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: Retry safety of HTTP requests

---

From: Subodh Iyengar <[email-masked]>
Date: Wed, 23 Mar 2016 02:33:56 +0000

@Mike Bishop there are some proposals for 0-RTT to include the client timestamp in the client nonce to limit the retryability of 0-RTT which are still being discussed on the TLS mailing lists. This is still an open question.

> If we’re talking about a pattern of DELETE, PUT, GET, the fact that every separate action is idempotent doesn’t save us from a replay of the DELETE after the PUT

That's an excellent point, and probably something the application can only determine to be safe. Ideally if an application determines an action to be safe (with a new flag) then it should be safe to retry the same request 5 months from now, although browsers should do a best effort not to do that and TLS 1.3 should also limit the time of 0-RTT to something reasonable.

Subodh

________________________________
From: Mike Bishop [[email-masked]]
Sent: Tuesday, March 22, 2016 7:21 PM
To: Martin Thomson; Mark Nottingham
Cc: Subodh Iyengar; [email-masked]
Subject: RE: Retry safety of HTTP requests

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Mar 2016 14:12:49 +1100

On 23 March 2016 at 13:21, Mike Bishop <[email-masked]> wrote:
> Idempotency is useful against short-time replay, like just resending until
> you get a response.  However, 0-RTT would permit replay seconds, minutes, or
> more later, no?

As Subodh notes, there is a suggestion (one that hasn't been
implemented) of including timestamps in the 0-RTT.  That would limit
replay to a very narrow window.  That window would only widen to allow
for a errors in estimating the round trip time, and any amount that
two clocks might drift relative to each other.  I would hope that this
would be seconds, probably something like 2s.

The open question part (at least to my mind) includes: who writes the
PR; and which endpoint estimates the round trip time.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Mar 2016 16:12:58 +1100

> On 23 Mar 2016, at 1:33 PM, Subodh Iyengar <[email-masked]> wrote:
> 

Yes. 6.3.1 says: "For example, a user agent that knows (through design or configuration) that a POST request to a given resource is safe can repeat that request automatically."

The challenge is how to give the UA that knowledge; it sounds more like a Fetch <https://fetch.spec.whatwg.org> flag (and maybe corresponding API in XHR, HTML forms, etc.) than something here.

It's even harder for a sequence of requests; that really is application-specific.

Cheers,

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Mar 2016 21:25:42 +1100

> It's even harder for a sequence of requests; that really is application-specific.

There, the onus is clearly on the application.  If request B depends
on request A, then a response to A needs to be seen before request B
can be sent.

HTTP/1.1 pipelining allows for a client to offload that to a server
for non-idempotent requests.  In theory.  In practice, pipelining was
fairly inconsistently implemented, particularly the bits around
serialization of requests.  I know of server implementations that
would process in parallel regardless of method; and client
implementations would happily pipeline non-idempotent requests.

HTTP/2 doesn't have the same "feature".  I think that's a good thing.

---

From: David Morris <[email-masked]>
Date: Wed, 23 Mar 2016 10:10:47 -0700 (PDT)

As the protocol architects, we've basically punted this concern to the end
user w/o providing anything in the protocol(s) to support the end user.

It is probably impossible to avoid the possiblity that today's generation
of complex http based applications will sometimes hang at some point in
the processing of a user initiated 'transaction'.

This is handled by a browser message to the effect that retrying a request
may be risky and expecting the user to decide whether it is safe. With
applications that initiate field level data store updates w/o an explicit
user 'command', there is no way for a user to judge the safety of a
repeated request, in particular when they often don't even understand the
boundary between 'requests'.

I see (and frequently experience) the problem at the user level and sort
of understand the underlying technical problems but I don't have a
solution.

I suspect that the solution will require some change to HTTP and perhaps a
layer above HTTP. Perhaps an option for something like 3 phase DB commits
but between the useragent and application. Essentially UX tranactions to
allow roll-back/retry w/o pestering the end user to make impossible
decisions. While a very careful well done application can make it work,
protocol support could make a big difference to the vast majority of
smaller applications.

Dave Morris

---

From: Phil Lello <[email-masked]>
Date: Wed, 23 Mar 2016 18:12:21 +0000

I think it's a bad idea to try to solve this in HTTP. It was originally
designed to be stateless, and extending it towards transactional messaging
is a road fraught with peril.

I consider it highly unlikely that enough websites will implement this
consistently for there to be a realistic hope of a consistent user
experience - and if that is eventually achieved, the transition will leave
users frustrated.

I think if a solution is worked on, it should be on a separate WG focused
on recommended application architecture  for software using HTTP as a
transport.

Phil Lello

Dave Morris

---

From: Subodh Iyengar <[email-masked]>
Date: Wed, 23 Mar 2016 18:17:42 +0000

Another point that my colleague Daniel Sommermann brought up is that the agent that retires the application request might not be colocated on the same machine. As in the original email thread, considering the example of a load balancer that wants to retry a request to a backend service if one of the backend HTTP service goes down. Here it makes sense to have an HTTP layer flag for retries, and a browser layer flag might not work. 

> may be risky and expecting the user to decide whether it is safe.
I agree, it's really hard for users to decide. For example if my payment for $X fails, will the browser refuse to make the request ever again or blame the user for having warned it that retrying was potentially dangerous. 

@Mark I think SEARCH might help this problem in quite a few cases, where applications could convert several of their POST queries into SEARCHes however it probably does not solve this particular user issue. I'm still checking out the Fetch spec, thanks for the pointer.

Subodh
________________________________________
From: David Morris [[email-masked]]
Sent: Wednesday, March 23, 2016 10:10 AM
To: [email-masked]
Subject: Re: Retry safety of HTTP requests

Dave Morris

---

From: Wenbo Zhu <[email-masked]>
Date: Wed, 23 Mar 2016 11:19:43 -0700

> protocol support could make a big difference to the vast majority of
Exactly-once delivery (RPC) over unreliable network is unsolvable ... so I
wouldn't bet on anything from HTTP to support safe-retry of POSTs.

---

From: Erik Nygren <[email-masked]>
Date: Wed, 23 Mar 2016 14:37:59 -0400

This post on attacks against POST retries in HTTPS is also worth reading
for those who haven't seen it:

         http://blog.valverde.me/2015/12/07/bad-life-advice/#.VvLe6rMpDmE

(I was a little surprised to see that the behavior of browsers had shifted
over the years to transparently retry POSTs over broken connections by
default.)

On Tue, Mar 22, 2016 at 1:55 AM, Subodh Iyengar <[email-masked]> wrote:

>
> 2) Some transport protocols which treat retry safe requests differently
> from non retry safe requests. For example in TLS 1.3, idempotent requests
> may be sent in a 0-RTT flight, which reduces the latency for the request.
> An application might desire a non idempotent request which is retry safe be
> sent in this 0-RTT flight.
>

Given the discussions on the TLS WG around TLS 1.3 0RTT mode,
it seems likely that a requirement of using TLS 1.3 0RTT in any application
protocol will be a document describing the binding for when and how it is
safe for that
application to use replayable 0RTT data, rather than leaving this to chance.
>From that perspective, I think Subodh's proposal here could be a good start
towards something that could include a safety story for HTTP
over replayable TLS 1.3 0RTT early data.  Especially with TLS 1.3 is going
to the extreme of specifying a different application interface for sending
attacker-replayable early data, HTTP can't just blindly use that new
interface without clear ground rules without introducing new
vulnerabilities.

Having clients start to implement HTTP over TLS 1.3 0RTT early data
without a clear set of ground rules is scary from a server operator
perspective
since some of the responsibility for safety here has to be on the client
knowing
what is and isn't possible to send early.  In large-scale globally
distributed
server environments there may not be a sane way to reliably protect against
replays while providing reasonable performance.  It's the same places where
0RTT anti-replay protections don't work reliably where application layer
anti-replay protections
become fragile and much harder to make work reliably on the server side
than on the client side.

           Erik

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 Mar 2016 10:08:53 +1100

On 24 Mar 2016, at 5:37 AM, Erik Nygren <[email-masked]> wrote:
> over the years to transparently retry POSTs over broken connections by default.)

Very interesting indeed. One can easily imagine an attack; e.g., a captive portal asks for online payment, and gets paid twice.

The advice to use a CSRF token is good, but it's pretty obvious that it's not being followed consistently or well (although maybe it's good enough in the places where it most matters, e.g., online payments). 

Regardless, it seems like we should either change the implementations, or change the spec. 

Cheers,

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Mar 2016 11:04:13 +1100

> over the years to transparently retry POSTs over broken connections by default.)

I'm not sure that this is entirely true, though I might observe that
asking a user:

"Do you want this to work?" Yes / No

In the way that we do is getting pretty close to an automatic retry.
I know that I hit yes often enough in those cases.  I know how to
apportion blame.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Mar 2016 07:48:30 +0100

On Thu, Mar 24, 2016 at 10:08:53AM +1100, Mark Nottingham wrote:
> Very interesting indeed. One can easily imagine an attack; e.g., a captive
> portal asks for online payment, and gets paid twice.
> 
> The advice to use a CSRF token is good, but it's pretty obvious that it's not
> being followed consistently or well (although maybe it's good enough in the
> places where it most matters, e.g., online payments). 
> 

I guess that we should at least document that this behaviour exists and must
be addressed in applications. For many years I've heard users ask if haproxy
could replay failed requests, I respond "no and it must not" then they say
"but my current product XYZ does"... No wonder why a friend received two
similar books the same day (and was charged twice).

It's not very hard to address in applications where this is sensitive (eg:
payment). It requires a transaction ID. It's a good practise against bad
user behaviour as well (eg: people pressing back, then forward and clicking
"Yes" in the warning box). By indicating that this behaviour exists, we can
sensitize developers to this and make them improve their applications.

Note that I'm not trying to defend auto-retry. I'm just saying we know it
exists sometimes.

Willy

---

From: Phil Lello <[email-masked]>
Date: Thu, 24 Mar 2016 11:56:21 +0000

> > Regardless, it seems like we should either change the implementations,
or
> > change the spec.
Theoretically this should be addressed by a nonce in the message flow -
perhaps the http/2 stream should be reusable to provide a built-in for this
scenario? My http/2 experience is still theoretical at present, but I
believe the intent is one stream per request.

Phil