Subject: Does idempotence include the response codes?

From: cowwoc <[email-masked]>
Date: Tue, 15 Oct 2013 13:43:14 -0400

Hi,

     Can the authors officially weigh in on the meaning of "idempotent" 
with response to response codes? Do idempotent methods have to return 
the same response code in the face of multiple invocations? Or are they 
allowed to return different response codes so long as the state is not 
modified by subsequent requests? See 
http://stackoverflow.com/q/741192/14731 for the discussion that fueled 
this question.

     PS: It would be nice if the spec officially clarified this point as 
I've seen it brought up again and again in different discussions.

Thank you,
Gili

---

From: cowwoc <[email-masked]>
Date: Tue, 15 Oct 2013 13:51:45 -0400

The original question discussed DELETE. Here is a similar question 
I just posted in terms of PUT: http://stackoverflow.com/q/19387764/14731

Gili

On 15/10/2013 1:43 PM, cowwoc wrote:
> the same response code in the face of multiple invocations? Or are 
> they allowed to return different response codes so long as the state 
> is not modified by subsequent requests? See 
>     PS: It would be nice if the spec officially clarified this point 
> as I've seen it brought up again and again in different discussions.
>
> Thank you,
> Gili

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Oct 2013 11:02:47 -0700

On 15 October 2013 10:43, cowwoc <[email-masked]> wrote:
> Do idempotent methods have to return the same response code in the face of
> multiple invocations? Or are they allowed to return different response codes
> so long as the state is not modified by subsequent requests?

I frequently see them returning different answers.  It's pretty common
to see DELETE, 200, DELETE, 404.  But I also see DELETE on a
non-existent resource returning 200.  The only guarantee is that two
attempts produce the same end result in terms of resource state.

I can guess what you might be trying to do, but I don't think that
it's going to be as simple as you might expect.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Oct 2013 20:05:24 +0200

> http://stackoverflow.com/q/741192/14731 for the discussion that fueled

I'll repeat what I said over there:

No, "effect" is not intended to cover the response as well. What you 
propose is that DELETE either always must pass ("200") or fail ("404" or 
"410"), no matter what the state of the server is. This doesn't make any 
sense and furthermore doesn't help the client at all.

> I've seen it brought up again and again in different discussions.

I personally believe it's clear enough, as the other interpretation 
frankly doesn't make any sense.

That being said, what do others think?

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 15 Oct 2013 20:14:36 +0200

Hi Julian,

> That being said, what do others think?

While my understanding has always been that the effect is considered on
the server only, this question made me re-read the definition in p2-4.2.2.

And trying to read it with the other meaning in mind, I find nothing which
tells me I'm wrong :

   Request methods are considered "idempotent" if the intended effect of
   multiple identical requests is the same as for a single request.  Of
   the request methods defined by this specification, the PUT, DELETE,
   and safe request methods are idempotent.

   [the 2 other paragraphs do not prove me wrong either]

For example, people working on a client could easily consider the effect
as "the effect on my client". I guess it would remove that question if we
just add "on the server" after "effect" to remove this possible ambiguity.

Maybe your example about DELETE returning 200 then 404 or 410 would also
be nice to add at the end of the first paragraph quoted above.

Best regards,
Willy

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Oct 2013 11:16:25 -0700

On 15 October 2013 11:05, Julian Reschke <[email-masked]> wrote:
> I personally believe it's clear enough, as the other interpretation frankly
> doesn't make any sense.

That's my take.  DELETE is either a great example to use for
discussing this, or a terrible one, but either way, it's pretty clear
that you can't expect all responses to be the same.

---

From: Gili <[email-masked]>
Date: Tue, 15 Oct 2013 16:17:13 -0400

> be nice to add at the end of the first paragraph quoted above.

Yes. Adding "on the server" and an example would definitely help.

Gili

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 15 Oct 2013 14:24:42 -0700

> Yes. Adding "on the server" and an example would definitely help.

Seems like a reasonable editorial clarification to me.

--
Mark Nottingham   http://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Oct 2013 14:33:24 -0700

> Seems like a reasonable editorial clarification to me.

It's low cost and improves comprehension.  SGTM.

---

From: cowwoc <[email-masked]>
Date: Tue, 15 Oct 2013 18:30:44 -0400

> It's low cost and improves comprehension.  SGTM.

     A lot of my interpretation is coming from reading "RESTful Web 
Services" published in 2007. Unfortunately, I'm seeing more and more 
contradictions between this book and what I'm reading on Stackoverflow. 
Just one example I got today:

http://restfulwebapis.org/RESTful_Web_Services.pdf page 274 says to 
return 301 from operations that rename a resource, but the answer found 
at http://stackoverflow.com/a/19388228/14731 recommends to return 200 
and 301 to subsequent requests to the old URI. To be honest, both 
approach have their downsides and both make sense. The spec doesn't seem 
to point towards one solution or the other which leaves me wondering 
which is the best approach.

     Is there a better/newer book I should read? httpbis contains many 
useful pieces of the puzzle but it's not nearly as "complete" as reading 
a book focused on authoring REST APIs.

Gili

---

From: Philippe Mougin <[email-masked]>
Date: Wed, 16 Oct 2013 00:31:00 +0200

> Seems like a reasonable editorial clarification to me.

It wouldn't work as it would include in its scope methods that actually are not idempotent. This is because "the intended effect" of a request may go far beyond "the server". A method might be idempotent regarding the state of the server but not idempotent regarding other parts of the real world.

BTW the definition of "Safe methods" in the current draft is faulty for a similar reason.

Best,

Philippe Mougin

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 16 Oct 2013 16:34:28 +1300

On 16/10/2013 11:31 a.m., Philippe Mougin wrote:
> It wouldn't work as it would include in its scope methods that actually are not idempotent.

Which ones and how?

>   This is because "the intended effect" of a request may go far beyond "the server".

"the server" is the authoritative copy of the resource. Application 
level choice to use idempotent methods or non-idempotent methods 
determines the systemic effects and behaviour. Not all application 
systems are correct. The ones which break due to this have never been 
correct and are already broken due to their incorrect assumptions about 
HTTP idempotency.

In absence of any evidence to the contrary I also agree with this 
proposed change.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Oct 2013 06:00:17 +0200

> a book focused on authoring REST APIs.

I've read your comment about renaming a company and the risk of reusing
the same name for a new company later. I think you're having your mind
confused because of the difference between "network time" and "human time".

The main idea behind idempotence is to know whether or not a client can
safely resend a request that failed because of network errors. For example,
sending a request over an established connection may result in an error if
the server decided to close at the same time. But the client has no way to
know if the request was considered or not, so it must resend it. If the
previous request was correctly processed, the second one may return an
error, but it's the client's job to handle this. I can give you an example
here with unix commands :

  $ mv a b
  ... phone rings ...
  ... Hmmm where was I. Ah, I was about to rename "a" to "b"
  $ mv a b
  mv: cannot stat `a': No such file or directory
  ... Ah yes, I already did it, let's go on with next operation.

In my opinion it is reasonable to think that during this short retransmit
time, the company name "A" will not be reused to name a new company. And
if it's not true (eg: an application that many people are using in parallel
for a lot of administrative stuff), then it means that the company name is
not a safe key to proceed with operations, so your application needs to use
a permanent ID :

   GET /id-by-name/A

   { "owner" : "smith", "location" : "NY", "id" : "12345" }

   PUT /rename-id/12345

   { "name" : "B" }

But as you can see, the protocol in its current state offers everything
needed to handle this correctly. However it will not make an unsafe
application suddenly safe. The application needs to be safe in its
sequences and use the protocol correctly.

Hoping this helps,
Willy

---

From: cowwoc <[email-masked]>
Date: Wed, 16 Oct 2013 00:37:51 -0400

> sequences and use the protocol correctly.

     Good example. When you talk about "network time" versus "human 
time" are you implying that the server should return 410 GONE and 307 
Temporary Redirect for a short period of time (say 5 minutes) to enable 
clients to retry on delete/rename operations, but after 5 minutes it can 
begin returning 404 at the old address? That seems like a very 
reasonable compromise to me. I just wanted to make sure that it's what 
you meant.

Thanks,
Gili

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Oct 2013 07:13:34 +0200

> reasonable compromise to me. I just wanted to make sure that it's what 

I'm not suggesting anything, it's to the application to decide this based
on what it's trying to achieve. I mean the idempotence is made to cover
the issues that require a retransmit. If your application requires slower
retransmits (eg: you pilot a robot on mars from the earth), then probably
5 minutes will not be enough.

Willy

---

From: cowwoc <[email-masked]>
Date: Wed, 16 Oct 2013 01:53:32 -0400

> 5 minutes will not be enough.

     That's very useful information. All too often I read sections in 
the spec that describe WHAT the implementation should do without an 
explanation of WHY. The why is important because without this context 
the developer can't decide what is reasonable behavior for their 
specific application. In the past I've always erred on the side of 
strictness (retaining idempotent information forever) for fear of doing 
the wrong thing. I don't know how/whether you can address this in the 
spec, but it would be nice if this bit of context was added ("you only 
need to keep idempotence information around long enough for your clients 
retry failed operations" as opposed to covering use-cases like permalinks).

     Anyway, thanks again. I've found this discussion very helpful for 
my own development.

Gili

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Wed, 16 Oct 2013 11:47:28 +0200

> here with unix commands :

I'm not sure I agree with the example, I've seen cases where idempotence
is a much stronger property.

For example, since GET is idempotent, that means a malware checker service
can replay (before or after the browser) get requests to check if the
result is dangerous or not. And it does not affect the result the browser
will receive on its own GET.

If you say mv is an example of idempotence, that means all the broken
websites that transform GETs into implicit POSTs are right, since the
first GET destroys the result the following GET could expect.

IMHO idempotence means a replay of the same command will give the same
result, baring errors, website reorganisations, and data changes (when the
request exposes some dynamic data store)

Regards,

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Wed, 16 Oct 2013 11:52:49 +0200

> request exposes some dynamic data store)

Or to put it more succinctly an idempotent method must not trigger by
itself a state change server-side. Is that clear and short enough for
everyone?

-- 
Nicolas Mailhot

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Oct 2013 12:02:29 +0200

> will receive on its own GET.

GET can retrieve a lot of dynamic information, here it's not about
idempotence since we're suggesting the contents may vary over time. It's
more about cahceability here.

> request exposes some dynamic data store)

But that does not last forever. This is the difference between theory and
reality. When you let your domain expire and it's acquired by a squatter,
your site will return something different whatever you decide. And the GET
you did when you owned the domain will definitely return something different
from what the squatter will return.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Oct 2013 12:03:28 +0200

> itself a state change server-side. Is that clear and short enough for

Yes it can trigger a state change. But it will not trigger another state
change if done *again*.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Oct 2013 12:12:04 +0200

> itself a state change server-side. Is that clear and short enough for

Clear, short, and incorrect.

You are confusing idempotency with safeness.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Oct 2013 12:52:37 +0200

> It's low cost and improves comprehension.  SGTM.

Added "on the server" and mentioned in the already existing example that 
response codes might vary for repeated requests: 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2427>

Best regards, Julian

---

From: cowwoc <[email-masked]>
Date: Wed, 16 Oct 2013 17:38:15 -0400

> BTW the definition of "Safe methods" in the current draft is faulty for a similar reason.
Hi Philippe,

     I don't understand what you mean. Care to elaborate?

Gili

---

From: Mark Baker <[email-masked]>
Date: Thu, 17 Oct 2013 01:20:14 -0400

> Yes. Adding "on the server" and an example would definitely help.

IMO, misunderstandings of this sort will continue so long as people
fail to understand that idempotence and safety are properties of the
request message alone, and are independent of implementation
behaviour. Short of a minor treatise on it - a generalization of
p2-4.2.1 2nd paragraph (re safety) perhaps (which I'm *not* proposing)
- I don't think it will help, but may cause further confusion because
of the mention of the server.

That said, if consensus is for this addition, perhaps it should be "on
the origin server" since, e.g., cache behaviour needn't be idempotent
for idempotent messages, and is rarely safe for safe messages.

Mark.

---

From: Philippe Mougin <[email-masked]>
Date: Thu, 17 Oct 2013 10:24:05 +0200

>> It wouldn't work as it would include in its scope methods that actually are not idempotent.

Methods idempotent with regard to state of the server but not idempotent with regard to other parts of reality.

For example, if I define a new request method, LAUNCH, with the following semantics : "The LAUNCH method requests that a new missile is prepared and launched. The LAUNCH method does not request modifications to the state of the server". 

This method would match the newly proposed definition of idempotent. However, if we consider the intended effect in general, launching multiple missiles has not the same effect as launching one missile. Therefore we do not want this method to be considered idempotent.

Best,

Philippe

PS: I do not encourage people to design methods that launch missiles.

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 18 Oct 2013 00:48:46 +1300

> For example, if I define a new request method, LAUNCH, with the following semantics : "The LAUNCH method requests that a new missile is prepared and launched. The LAUNCH method does not request mod…

... resulting in 1 less missile available in the resource on "the 
server" ... which is a state change for the server.
The assertion "Methods idempotent with regard to state of the server" is 
false for this example.

Even if "the server" is shifted down into the missile itself it has 
undergone a state change from WAITING to LAUNCHED. Possibly with 
implicit LANDED or ABORTED state changes timing down.

> This method would match the newly proposed definition of idempotent. However, if we consider the intended effect in general, launching multiple missiles has not the same effect as launching one mis…

To be idempotent with regard to the server your method URI would have to 
indicate _which_ missile to launch. Then telling the same missile to 
launch multiple times should have no different effect than telling it 
once. That is an idempotent method, both in regards to the server state 
and to reality beyond the server.

> PS: I do not encourage people to design methods that launch missiles.

Too late. You did.

Amos