Subject: Avoiding duplicate entries / Idempotent posting

From: David Powell <[email-masked]>
Date: Tue, 6 Jul 2004 14:01:10 +0100

Hi,

I’ve recently been looking at the Atom API spec, and admittedly haven't spent
much time trawling the list archives yet, so apologies if this has been
discussed:

Does the Atom API provide a way to idempotently POST an entry?  In the case of
some protocol-level errors it is not possible to tell if the entry has been
posted, so the client can't know whether or not to retry the request and risk
posting a duplicate entry. 

Eg:

try {
  POST entry
  if (status==2xx) then OK
  if (status==4xx) then REPORT_ERROR
  if (status==5xx) then RETRY_REQUEST
  ...
} catch IOException {
  // what now? 
}

One way of avoiding this problem (a) is to POST a dummy entry (or a zero-length
POST?) to the PostURI.  This dummy entry would be marked as hidden and wouldn't
show on the feed or the site, but then its EditURI could be used by the client
to PUT the entry to when it is ready to upload it.  The POST would essentially
just be used to reserve a URI, and as PUTing to an EditURI is idempotent, the
client can just repeat the PUT if there is an error.

Another possibility (b) would be to continue to use POST, but to add a
"message-id" field containing a GUID to the entry (in the body or in HTTP?),
and to allow the server to ignore duplicate message-ids.

Either of these techniques could be deployed without affecting existing clients
that do not require idempotent posting.

These techniques would both require extensions to Atom, do people have any other
techniques for avoiding duplicate entries?  I think that it is important to get
this right, especially for Atom clients on mobile devices where connectivity
might be less than ideal.

-- 
Dave

---

From: Joe Gregorio <[email-masked]>
Date: Tue, 6 Jul 2004 09:39:00 -0400

On Tue,  6 Jul 2004 14:01:10 +0100, David Powell <[email-masked]> wrote:
> 
> Hi,
> 
> I've recently been looking at the Atom API spec, and admittedly haven't spent

Only a HTTP response of 201 indicates that an
entry has been created. More verbage should be added
to the spec to make that clearer.

    Thanks,
    -joe

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 6 Jul 2004 07:39:46 -0700

That doesn't guarantee that a post won't be duplicated; for example, if 
the connection drops before the response is received by the client, the 
server will have created the resource, but the client will believe it 
hasn't.

For something more reliable, see:
   http://www.mnot.net/blog/2003/09/13/click_submit_only_once

--
Mark Nottingham     http://www.mnot.net/

---

From: David Powell <[email-masked]>
Date: Tue, 6 Jul 2004 15:57:35 +0100

> to the spec to make that clearer.

Thanks, but perhaps I wasn't clear:

I understand that 201 means that the entry was created, and 4xx/5xx means that
the entry was not created, but I was talking about the case where no status
code reaches the client at all.

The sort of situation where the client has sent the request, but not received a
response, so doesn't know whether the server has failed to process the request,
or processed the request but failed to return the status.

This could be caused by network problems causing a connection to be dropped,
proxies failing, or anything where the client's HTTP library decides to throw
an exception rather than returning the response for whatever reason.

Another example would be if the user was connected via a proxy, and the proxy
returned a 502/503/504 response.  The client wouldn't know whether the POST had
been successfully processed or not.

I have implemented HTTP applications before where these sort of errors have
occurred and it isn't nice to have no deterministic way of knowing what to do
next.

As you might have guessed this request comes from one of those early
interactions with the Internet that has left me scarred – we've all encountered
the following horror story:

  You click on a button on a web page marked "Send" or "Buy" only to spend
  the next 5 minutes staring at a spinning IE logo whilst the realisation
  that you have been jilted sinks in.  Usually these pages have stern red
  text below them that warns you to only click the button once.

Things don't have to be this way.
 
-- 
Dave

---

From: Joe Gregorio <[email-masked]>
Date: Tue, 6 Jul 2004 11:33:26 -0400

> Thanks, but perhaps I wasn't clear:
....
> Things don't have to be this way.

Now I get it, and Mark has pointed out one such solution:

The other possibility is to do a GET on the PostURI
to retrieve a partially filled in atom 'entry' which 
you stuff your content into before submitting
it back to PostURI to create an Entry. A nonce value
in a namespaced element could be added to such
a 'template' to avoid multiple submissions of the same
entry.

The idea of a 'template' entry retrieved from PostURI 
came up previously during a discussion of querying
server capabilities:

http://intertwingly.net/wiki/pie/AtomApiContentNegotiation

   -joe

---

From: David Powell <[email-masked]>
Date: Thu, 8 Jul 2004 23:12:00 +0100

>    -joe

I think that it is important to consider this problem, because as Mark
as shown, this problem is solvable in typical browser-based apps, so I
think that we should ensure that it is also solvable for users of the
Atom API too.  Currently, I don't think it is without some
modifications.

It isn't straightforward to apply Mark's suggestion to Atom directly,
because unlike browser apps, the PostURI is public and there is only
one PostURI. As it is the server's general responsibility to issue
URIs, this would need some extra process that issued unique PostURIs
which adds quite a bit of complexity to what we have at the moment. I
prefer the idea of having a single PostURI for a feed.

Do people think that this issue is worth tackling?

(I'm a bit ignorant of the WG's process - is it bad timing to suggest
new issues at the moment?)

-- 
Dave

---

From: Tim Bray <[email-masked]>
Date: Thu, 08 Jul 2004 15:46:45 -0700

> new issues at the moment?)

Suggesting a new issue is always OK, but to be taken seriously it needs 
to be accompanied by draft language, i.e. to be a Pace. -Tim

---

From: Robert Sayre <[email-masked]>
Date: Thu, 08 Jul 2004 19:22:32 -0400

> Do people think that this issue is worth tackling?

Yes, but perhaps something like PacePutToCreate[1] would do the trick.

Robert Sayre

[1] http://www.intertwingly.net/wiki/pie/PacePutToCreate

---

From: steve jenson <[email-masked]>
Date: Thu, 8 Jul 2004 17:50:36 -0700

> Yes, but perhaps something like PacePutToCreate[1] would do the trick.

If I'm an endpoint that enforces complete control (really, any 
overriding control) over the URI space, should a client be using POST 
instead of PUT? Is it ok for me to _not_ create the Resource with the 
Request-URI asked for if a PUT is sent?

More concretely, if a User PUTs an entry dated 12-25-04 with a 
Request-URI of http://blogger.com/atom/$blogID/2004/11/title.html 
(please note a different month than the entry's date), and I instead 
create it at http://example.com/2004/12/title.html, am I in violation 
of the semantics of PUT?

Also, I understand that a PUT can cause side effects in multiple URIs 
but do I also have to serve this newly created Entry at the 
Request-URI? PUT seems to require that or at least strongly lean in 
that direction.

Thanks,
Steve

---

From: Robert Sayre <[email-masked]>
Date: Thu, 08 Jul 2004 21:24:56 -0400

> create it at http://example.com/2004/12/title.html, am I in violation

Yes, I think so. What you can do is send 301, and let the client decide if
it wants to PUT it there. Not much different from asking for a one-off URI
to POST to, the other suggestion we've had.

> Request-URI? PUT seems to require that or at least strongly lean in

"A single resource MAY be identified by many different URIs. For  example,
an article might have a URI for identifying 'the current version' which is
separate from the URI identifying each particular  version. In this case, a
PUT request on a general URI might result in  several other URIs being
defined by the origin server.

 HTTP/1.1 does not define how a PUT method affects the state of an origin
server."[1]

Although the definition of PUT says the effect on the server state is
undefined, the 201 Response definition[2] states that you have to create the
resource. So it would appear that you have to serve it there, at least for
GET. I'll note that the AtomAPI doesn't currently require the EditURI to be
the same URI created by POSTing to the PostURI.

Robert Sayre

[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6
[2] http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2