Subject: RE: No validator in 200 response for conditional update

From: Vladimir Lashchev <[email-masked]>
Date: Tue, 6 Nov 2018 14:35:28 -0800 (PST)

Thanks for your replies!

In general, since the spec says that server "MUST NOT" send an ETag in some case so in order to comply with the spec we must implement that logic and it does not matter if it is a corner case or if it may happen only once in our API. In reality it is actually not an edge case and happens quite often (up to single digit % of all requests in high volume concurrent scenarios).

That's why I am trying to understand the reason because that clause calls for a lot of development.

Why "a lot" of development?
Because the clause "... unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent ..." requires us to persistently track previous requests from all clients which is a huge infra requirement for stateless API server farms (which is a typical design for servers with high performance requirements).

Actually, scenario of that 2xx clause (as I read it) is a bit more complicated - it applies in the case when 2 agents (A1 and A2) are talking to the sever and trying to apply semantically same changes or same agent asks for the same change in non-consecutive requests. Here is my understanding:
1) "state change is being requested and the final state is already reflected in the current state" ==> means that requests may not be identical, as long as they ask for the same end-result state.
2) "server MUST NOT send a validator header field in the response unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent " ==> means that:
a) server MUST NOT send ETag to A2 if state change was done by A1, but similar change is requested by A2
b) server MUST NOT send ETag to A1 if state change was done by A1 but there was some other request (even GET without any side-effects) between that original change request and a new, semantically-identical change request.

So expanding on Yishuai's example to include all cases it would be something like:

(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (same change-originating agent, same end-result requested in the immediate prior request)
(step 5) A2 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 6) S --> A1 : 204 No Content (No ETag) (not a change-originating agent, same end-result requested)
(step 7) A1 --> S : GET ... (unconditional, no side-effects)
(step 8) S-> A1 (doesn't matter what is replied here)
(step 9) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 10) S --> A1 : 204 No Content (No ETag) (same change-originating agent, same end-result requested, but not in the immediate prior request)
(step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different end-result requested)

My question is about (step 6) and (step 10) where server MUST NOT send a ETag value back. 
Above is quite complicated logic required without explanation and it is not clear of how it is better/safer than simple and stateless:
(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (any agent, same end-result requested)
 (step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different end-result requested)

Thanks,
Vladimir

-----Original Message-----
From: Julian Reschke <[email-masked]> 
Sent: Monday, October 29, 2018 10:06 AM
To: Yishuai Li <[email-masked]>; Vladimir Lashchev <[email-masked]>
Cc: [email-masked]
Subject: Re: No validator in 200 response for conditional update

On 2018-10-27 08:04, Yishuai Li wrote:
> I am also confused by this clause. Consider the following scenario 
> (for simplicity, the entity-tag is identical to entity, and the all 
> requests have the same target):
> 
> (1) A --> S : PUT "a" (unconditional)
> (2) S --> A : 204 No Content, ETag: "a"
> (3) A --> S : PUT "a", If-Match: "b" (not satisfied)
> (4) S --> A : 204 No Content
> (5) A --> S : PUT "a", If-Match: "b" (not satisfied)
> (6) S --> A : 204 No Content, ETag "a"
> 
> Did I understand the specification correctly that: S MUST NOT send a 
> validator in (4), but MAY do so in (6), because (5) is a duplicate of 
> (3), but (3) is not a duplicate of (1)?
> 
> Thanks,
> Yishuai Li
> ...

Ok, now I understand the question. Ignore my previous reply.

So that's an edge case of an edge case, right?

The spec says:

>    An origin server MUST NOT perform the requested method if a received
>    If-Match condition evaluates to false; instead, the origin server
>    MUST respond with either a) the 412 (Precondition Failed) status code
>    or b) one of the 2xx (Successful) status codes if the origin server
>    has verified that a state change is being requested and the final
>    state is already reflected in the current state of the target
>    resource (i.e., the change requested by the user agent has already
>    succeeded, but the user agent might not be aware of it, perhaps
>    because the prior response was lost or a compatible change was made
>    by some other user agent).  In the latter case, the origin server
>    MUST NOT send a validator header field in the response unless it can
>    verify that the request is a duplicate of an immediately prior change
>    made by the same user agent.

Note the "immediately prior change" - so no (4) did not result in a change, so the rule does not apply and no etag should be sent.

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 7 Nov 2018 14:03:07 +1300

> In general, since the spec says that server "MUST NOT" send an ETag in some case so in order to comply with the spec we must implement that logic and it does not matter if it is a corner case or if…

On a side note;

My understanding of the expected behaviour is that agents doing a lot of
repeats are broken in some way. Probably their author misunderstood that
412 is *not* an error.

A 4xx is just a notice from the server to the agent that the requested
change/fetch was not enacted. Not really errors in the way we tend to
think of errors in software.

So agents receiving back a 412 with ETag matching the one they tried to
set the resource to - should usually treat that as as much a success as
any 2xx they get. It is only when the 412 ETag is different that the
agent needs to figure whether to try again, and HTTP cannot help by
specifying anything about that decision.

> Because the clause "... unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent ..." requires us to persistently track previous requests from…

I fail to see how there is any difference in overhead for a large farm
than for a single server. All that needs to be tracked is what the
current ETag validator is and which agent made it exist. Usually those
would exist anyway in server state as author and version IDs.

In a large farm each stateless server may be tracking a different
version of the object (or not) with its own ETag + agent labels. That is
a given implied by them being stateless. The condition rules from the
request are applied by the individual server against its own object(s)
copy regardless of what other objects might be elsewhere or what might
have happened in old history for the relevant local object(s).

You _can_ do sharing of updates + info between the servers. But that is
up to the implementation and kind of disolves the claim of being stateless.

That said, I don't recall what was behind this RFC wording so am
watching to see what others reply to your specific questions.

HTH
Amos

---

From: Julian Reschke <[email-masked]>
Date: Wed, 7 Nov 2018 08:51:36 +0100

> In general, since the spec says that server "MUST NOT" send an ETag in some case so in order to comply with the spec we must implement that logic and it does not matter if it is a corner case or if…

Are we talking about the same thing?

1) It needs to be a state-modifying request,

2) That comes with a validator,

3) The validator doesn't match, but the current state of the resource does.

So this only happens when multiple clients do concurrent updates of a
resource, and write the same content.

It would be interesting to understand why this happens so frequently -
can you provide more information about the type of the server/service?

> b) server MUST NOT send ETag to A1 if state change was done by A1 but there was some other request (even GET without any side-effects) between that original change request and a new, semantically-i…

In-between GET requests may be covered by "duplicate of an immediately
prior change made by the same user agent" -  but maybe this could be
made more clearer.

> (step 12) S --> A1 : 412 No Content (No ETag) (any agent, different end-result requested)

You always have the choice of not replying with 204 but with 412, in
which case the server logic would be simpler, but clients would need to
do more work.

Which brings me back to my question about what kind of server this is
that shows this type of traffic pattern.

Best regards, Julian

---

From: Yishuai Li <[email-masked]>
Date: Wed, 7 Nov 2018 07:53:45 -0500

As you have said, we are talking about an "edge case of an edge case":
1. If precondition is satisfied, then process; otherwise
2.1. Respond with 412 Precondition Failed, or
2.2. If the proposed change has been reflected, MAY respond with 2xx
(Success), and
2.2.1. If current request is a duplicate of an immediately prior change
made by the same user agent, MAY send etag, otherwise
2.2.2. MUST NOT include etag in the response.

Why do we specify 2.2.1 and 2.2.2? I believe Vladimir has some traffic
pattern that match 2.2, where 2xx Success is more informative than 412
Precondition failed, but determining whether to send an etag or not leads
to extra development effort.

Yishuai

Julian Reschke <[email-masked]> 于2018年11月7日周三 上午2:51写道：

> > In general, since the spec says that server "MUST NOT" send an ETag in
> some case so in order to comply with the spec we must implement that logic
> and it does not matter if it is a corner case or if it may happen only once
> in our API. In reality it is actually not an edge case and happens quite
> often (up to single digit % of all requests in high volume concurrent
> scenarios).
>
> > Because the clause "... unless it can verify that the request is a
> duplicate of an immediately prior change made by the same user agent ..."
> requires us to persistently track previous requests from all clients which
> is a huge infra requirement for stateless API server farms (which is a
> typical design for servers with high performance requirements).
> >
> > Actually, scenario of that 2xx clause (as I read it) is a bit more
> complicated - it applies in the case when 2 agents (A1 and A2) are talking
> to the sever and trying to apply semantically same changes or same agent
> asks for the same change in non-consecutive requests. Here is my
> understanding:
> > 1) "state change is being requested and the final state is already
> reflected in the current state" ==> means that requests may not be
> identical, as long as they ask for the same end-result state.
> > 2) "server MUST NOT send a validator header field in the response unless
> it can verify that the request is a duplicate of an immediately prior
> change made by the same user agent " ==> means that:
> > b) server MUST NOT send ETag to A1 if state change was done by A1 but
> there was some other request (even GET without any side-effects) between
> that original change request and a new, semantically-identical change
> request.
>
> > (step 4) S --> A1 : 204 No Content, ETag: "a" (same change-originating
> agent, same end-result requested in the immediate prior request)
> > (step 10) S --> A1 : 204 No Content (No ETag) (same change-originating
> agent, same end-result requested, but not in the immediate prior request)
> > Above is quite complicated logic required without explanation and it is
> not clear of how it is better/safer than simple and stateless:

---

From: Julian Reschke <[email-masked]>
Date: Wed, 7 Nov 2018 14:01:50 +0100

On 2018-11-07 13:53, Yishuai Li wrote:
> Precondition failed, but determining whether to send an etag or not 
> leads to extra development effort.
> 
> Yishuai
> ...

Well, sending the etag saves work on the client (because it doesn't have 
to refetch the etag if it's interested in it).

On the other hand it's more work on the server. The server implementer 
can choose what's better...

Best regards, Julian

---

From: Yishuai Li <[email-masked]>
Date: Wed, 7 Nov 2018 08:19:28 -0500

I think that sending etag is not a huge work itself, if the server is to
implement etag at all. The difficulty is determining whether to send it or
not, because sending etag in 2.2.2 is violating the specification. Is there
a specific reason for this restriction? If not, shall we remove such
constraint?

Yishuai

---

From: Vladimir Lashchev <[email-masked]>
Date: Tue, 13 Nov 2018 14:39:04 -0800 (PST)

> From: Julian Reschke <[email-masked]> on November 6, 2018 11:52 PM
> and it does not matter if it is a corner case or if it may happen only once in our
> API. In reality it is actually not an edge case and happens quite often (up to
> single digit % of all requests in high volume concurrent scenarios).
> 

Correct.
Duplicated request from other clients for desired end-state that is already achieved.

> It would be interesting to understand why this happens so frequently - can
> you provide more information about the type of the server/service?

It happens in a typical message-based architecture where components exchange messages and some of them are state changes.
All our state change requests carry If-Match for validation purposes.

Why duplicated updates?
Because components are not just ours, some of them are 3rd party, some are behind bad/geo-long networks.
So, messages in order to be guaranteed, can be delivered one or more times.
Network slowness and errors result in connection terminations which lead to retries and retries lead to state change request duplicates.
This happens pretty often (monthly or so).
But during such network congestions it can be pretty high number of message duplicates.

> You always have the choice of not replying with 204 but with 412, in which
> case the server logic would be simpler, but clients would need to do more
> work.

Sending back 200 from the state management endpoint makes sense since desired state is already achieved and it will reduce infra pressure because clients will move forward with such success updates and also less retries/error handling will be happening.

Thanks, Vladimir

---

From: Vladimir Lashchev <[email-masked]>
Date: Tue, 13 Nov 2018 14:52:09 -0800 (PST)

Yishuai summarized the issue correctly.

> From: Yishuai Li <[email-masked]> November 7, 2018 5:19 AM
> I think that sending etag is not a huge work itself, if the server is to implement etag at all. 
> The difficulty is determining whether to send it or not, because sending etag in 2.2.2 is violating the specification. Is there a specific reason for this restriction? If not, shall we remove such constraint?

Exactly.
Sending ETag back all the time is a no-brainer.
There is no additional work here at all since we do send ETag with regular 2xx responses already in state update happy path.
But deciding when to send it and when not to send calls for quite a lot of dev work and previous request tracking infra (ongoing runtime cost).
And we really-really don't want to do this.
But spec says we MUST do this and doesn't explain/hint why.
May be the reason is valid (like security), but we need to know it so we can justify our investment.

> Julian Reschke <mailto:[email-masked]> 于2018年11月7日周三 上午8:02写道：

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 14 Nov 2018 18:35:05 +1300

On 14/11/18 11:21 AM, Vladimir Lashchev wrote:
>> From: Amos Jeffries
>> Sent: Tuesday, November 6, 2018 5:03 PM
>> On a side note;
>>
>> change/fetch was not enacted. Not really errors in the way we tend to think
>> of errors in software.
>>
>> So agents receiving back a 412 with ETag matching the one they tried to set
>> the resource to - should usually treat that as as much a success as any 2xx
>> they get. It is only when the 412 ETag is different that the agent needs to
>> figure whether to try again, and HTTP cannot help by specifying anything
>> about that decision.
> 
> Actually 412 is a proper error as I see it.
> Server denies modification request and indicates that request was not successfully processed (desired resulting state was not achieved).

You are assuming "state" here. Which is wrong and leading you to a wrong
conclusion.

412 states that the _preconditions_ in that individual request were not
obtainable. By itself it does not say anything about any state beyond
the individual request message being responded to.

The ETag validator is the only bit involving any higher-level state and
that has a defined meaning agnostic of the status code.

> I don't see a scenario where client can treat it as a success. It is a failed update in an optimistic concurrency flow.
> 

When the precondition fails because the Etag has already been changed to
a) the desired one, or b) something even newer than the desired one.

The exact edge-edge-case under discussion here is one situation where
the desired state *was* achieved without the preconditions being true.
It just happened to be achieved previously by an earlier change request.

> Anyway, my question is actually about 2xx response and not 412 :-)
> 

And the answer perhaps is "why 2xx when 412 conveys the desired meaning".

>>> Because the clause "... unless it can verify that the request is a duplicate of
>> an immediately prior change made by the same user agent ..." requires us to
>> persistently track previous requests from all clients which is a huge infra
>> requirement for stateless API server farms (which is a typical design for
>> servers with high performance requirements).
>>
>> I fail to see how there is any difference in overhead for a large farm than for
>> a single server. All that needs to be tracked is what the current ETag validator
>> is and which agent made it exist. Usually those would exist anyway in server
>> state as author and version IDs.
> 
> Request #2 can come to any server in the farm and spec calls that all servers should know about previous request from that agent so they can decide whether to send ETag back or not.
> It calls for state sharing among all the servers and thus they become not stateless (which we do not want to have).
> 

That is not my interpretation.

The spec is written in terms of individual request-response
transactions. Not in terms of client-service transactions - which may
span multiple request-response transactions in a stateful way.

>> In a large farm each stateless server may be tracking a different version of
>> the object (or not) with its own ETag + agent labels. That is a given implied by
>> them being stateless. The condition rules from the request are applied by
>> the individual server against its own object(s) copy regardless of what other
>> objects might be elsewhere or what might have happened in old history for
>> the relevant local object(s).
> 
> Spec is defining how a particular server in a farm should behave - it defines API endpoint contract.

Agreed.

> Endpoint is deployed at reverse proxy that is fanning out requests to a farm of identical servers.

"server" does not mean "server state". The underlying HTTP protocol /
contract is stateless. The service layer above usually has state, but
may not.

> We don't have sticky session, so modification request #2 can be processed on a different server from where request #1 was processed.
> Spec defines how client will see the response regardless of how many servers we have, so it should not matter on what worker server it was processed.

Here you assume a relationship between end-servers which is beyond
anything HTTP covers.

The spec / API contract outlines the set of valid responses to any given
request. The meanings for various parts of the message is relative to
the status and the request type.

The spec does *not* dictate statefulness between servers which would
lead to identical behaviour.

Yes there is a design problem inherent in that, but by remaining
strictly stateless HTTP pushes it up to the layer above. There are many
possible solutions to that and IIRC most if not all of them, are
actively used somewhere.

> 
>> You _can_ do sharing of updates + info between the servers. But that is up to
>> the implementation and kind of disolves the claim of being stateless.
> 
> Yes, and we don't want state sharing.
> 

Then you need the client endpoints to be robust against servers being at
different update levels with their internal state.

Such as by using the spec interpretation roughly along the lines I used
earlier to guide design decisions in the client software:

* 5xx are service errors, the client unlikely able do anything to fix
(some exceptions where it can retry).

* 4xx notices that the request did not go exactly as client intended,
with hints to the client about why. Leaving the client to decide if
retry is worth doing based on those hints.

* 2xx everything went exactly as planned. Nothing further for the client
to consider.

If you try to go beyond that to push multiple meaning into the 2xx
status is when you hit problems with whether ETag is possible to send at
all, or what value to send when the server is *more* up to date than the
client.
 Both semantically have the same "I could not do this because I have
object "a"' meaning. The client is the agent which knows best whether
that situation ignorable or needs handling.

AYJ

---

From: Vladimir Lashchev <[email-masked]>
Date: Thu, 29 Nov 2018 15:33:58 -0800 (PST)

> From: Amos Jeffries <[email-masked]>
> Sent: Tuesday, November 13, 2018 9:35 PM
> > Server denies modification request and indicates that request was not
> > successfully processed (desired resulting state was not achieved).
> 
> obtainable. By itself it does not say anything about any state beyond the
> individual request message being responded to.
> The ETag validator is the only bit involving any higher-level state and that has a
> defined meaning agnostic of the status code.

I mean "state" as in the paragraph in question (rfc7232#section-3.1):
"... a) the 412 (Precondition Failed) status code
   or b) one of the 2xx (Successful) status codes if the origin server
   has verified that a state change is being requested and the final
   state is already reflected in the current state of the target
   resource ..."

> a) the desired one, or b) something even newer than the desired one.

Case a) is explicitly covered with 2xx return code in rfc7232#section-3.1 in question
Not sure about case b) because ETag in general cannot be compared with "newer", it is _typically_ a black box like hash or GUID.

> The exact edge-edge-case under discussion here is one situation where the
> desired state *was* achieved without the preconditions being true.

Yes.

To be precise - it applies only if desired state was achieved by immediate previous request from the same exact agent.
It actually doesn't matter if it is edge case or not - server that conforms with RFC MUST implement it (that what RFC says).

> And the answer perhaps is "why 2xx when 412 conveys the desired meaning".

Responding with 2xx provides an optimization by converting some error flows into positive flows.

> > Request #2 can come to any server in the farm and spec calls that all servers
> should know about previous request from that agent so they can decide whether
> to send ETag back or not.
> > It calls for state sharing among all the servers and thus they become not
> stateless (which we do not want to have).
> 
> The spec is written in terms of individual request-response transactions. Not in
> terms of client-service transactions - which may span multiple request-response
> transactions in a stateful way.

Could you please help me understand how this sentence from RFC can be applied to a single request-response:
"unless [server] can verify that the request is a duplicate of an immediately prior change
   made by the same user agent"
To apply such logic IMHO server needs to remember prior changes made by each agent.
And remembering prior changes implies state management on the server farm.

> The spec / API contract outlines the set of valid responses to any given request.
> The meanings for various parts of the message is relative to the status and the
> request type.
> The spec does *not* dictate statefulness between servers which would lead to
> identical behaviour.

Are we still talking about rfc7232#section-3.1?
How the check required by that section can be achieved without statefulness on the server?

> Such as by using the spec interpretation roughly along the lines I used earlier to
> guide design decisions in the client software:
>  ...
> If you try to go beyond that to push multiple meaning into the 2xx status is when
> you hit problems with whether ETag is possible to send at all, or what value to
> send when the server is *more* up to date than the client.
>  Both semantically have the same "I could not do this because I have object "a"'
> meaning. The client is the agent which knows best whether that situation
> ignorable or needs handling.

Going back to the question at hand - what would be your proposal on how to change rfc7232#section-3.1, part with 412 and 2xx + (conditional) ETag responses?

Mine was to remove this sentence:
" In the latter case, the origin server
   MUST NOT send a validator header field in the response unless it can
   verify that the request is a duplicate of an immediately prior change
   made by the same user agent"

VL

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 1 Dec 2018 00:57:19 +1300

>    state is already reflected in the current state of the target

The "state" being mentioned in the spec is the exact octets of the
resource itself. The value / octets which the request is attempting to
change.

The way you have been using "state" was to mean the entire history
information of the resource. Which is very a different type of "state".

The difference between changing a document, and changing the changelog
of a document. The former can be done with some flexibility in the
response payload, the latter cannot.

> And remembering prior changes implies state management on the server farm.

It does require the "immediately prior" state be remembered. Not the
entire history of state, nor state from other servers.

The resource may be sensitive to distributed changes. But that is a
separate issue imposed by the nature of the resource not this RFC. Any
other protocol would face the same issue.

> How the check required by that section can be achieved without statefulness on the server?

The resource gets saved to a file (or equivalent). Changes to it get
labeled with modification time and some label to identify the client
doing the modification. That is all that is needed IMO to meet the spec
requirement.

The server receiving an conditional change request checks it against
those two details - according to the section-3.1 criteria.

Then (always) updates its stored label and mtime values so any followup
request gets based on the new details. Some nil value for the client
label on duplicates so these repeats get detected easily as a mismatch
on the repeats not "immediately prior".

The server can then respond correctly correctly to any clients while it
asynchronously handles any sync with a master copy.

>    made by the same user agent"

Unless someone else can recall why the text is the way it is now. My
preference would also be to go with an Eratta for making that change.

Amos