Subject: Idempotent partial updates

From: Mike Kelly <[email-masked]>
Date: Mon, 27 Feb 2012 14:18:50 +0000

Hi all,

HTTPbis has amended the semantics of PUT to be unambiguous in its
prevention of partial updates. I cannot understand the rationale
behind this change, and I have a couple of questions for the group:

- Given that 2616 was ambiguous, is it acknowledged that this
represents a breaking change to any existing infrastructure that
relies on PUT requests that are permitted to be partial?

- How is it foreseen that this over-specification of PUT will benefit
the web? i.e. what infrastructure will be able to exist once all PUT
requests are unambiguously non-partial?

- Is it acknowledged that this change will effectively prevent proper
idempotent partial updates on the web?

- In light of the recent proliferation of 'mobile clients' which
operate on relatively slow and unreliable networks where partial
idempotent updates would be ideal; is the loss of idempotent partial
updates considered acceptable?

Cheers,
Mike

---

From: Julian Reschke <[email-masked]>
Date: Mon, 27 Feb 2012 16:15:07 +0100

> HTTPbis has amended the semantics of PUT to be unambiguous in its

"clarified", not "amended". The point being: PUT never allowed partial 
updates. Because people used if for partial updates anyway, we made that 
part clearer.

> relies on PUT requests that are permitted to be partial?

I don't think it ever was ambiguous.

"The PUT method requests that the enclosed entity be stored under the 
supplied Request-URI."

> requests are unambiguously non-partial?

I don't think it's an over-specification. And even if it was, it's not new.

> idempotent partial updates on the web?

No. You can do idempotent partial updates with POST and PATCH (by 
including an If-Match header field).

> updates considered acceptable?

See above: (1) PUT never allowed partial updates, and (2) you don't need 
PUT to make modifications idempotent.

Best regards, Julian

---

From: Mike Kelly <[email-masked]>
Date: Mon, 27 Feb 2012 15:35:52 +0000

> "The PUT method requests that the enclosed entity be stored under the

It was ambiguous enough to lead to real-world implementations which
directly violate it. You've said as much above, and the action to
ammend the semantics

> I don't think it's an over-specification. And even if it was, it's not new.

It is now explicit and non-ambiguous (i.e. prevented). That is new.

This is getting away from the real intention of the question: what are
your thoughts on the benefits of partial prevention vs non-prevention?

> No. You can do idempotent partial updates with POST and PATCH (by including
> an If-Match header field).

I'm aware of this technique. It's neat, but I wouldn't consider it
explicitly idempotent; i.e. it's idempotent nature is not explicit and
is therefore not very visible. This is an issue for intermediary
components, such as HTTP client libraries in mobiles which (if the
request were PUT) could handle the re-issue of failed requests. Afaik,
this infrastructure already exists  right now on the web and works
fine for partial PUT requests.

> See above: (1) PUT never allowed partial updates, and (2) you don't need PUT
> to make modifications idempotent.
>

as above:

(1) Then why were the semantics amended, and why would others have
implemented against the spec in that fashion?

(2) You need PUT to make a request that is visibly idempotent through
the network.

Cheers,
Mike

---

From: Julian Reschke <[email-masked]>
Date: Mon, 27 Feb 2012 16:54:40 +0100

On 2012-02-27 16:35, Mike Kelly wrote:
> ...
> ...

Just because implementations get something wrong doesn't mean the spec 
was ambiguous; witness all the sites that did/do destructive things on 
GET, that do not implement HEAD, etc.

> your thoughts on the benefits of partial prevention vs non-prevention?

You could ask the same question for most methods; PUT means "store" and 
always has. There's no good reason to change that (allowing partial 
updates would be a change).

> is therefore not very visible. This is an issue for intermediary

How is is "not visible"?

> fine for partial PUT requests.

Then I believe the right thing is to request changes to these libraries 
so that they handle PATCH (for instance) better.

> (2) You need PUT to make a request that is visibly idempotent through

Yes, see above.

I think it's pretty clear that I won't convince you (the discussion 
happened before on rest-discuss I believe), and you won't convince me, 
so I'd be interested to hear other people's opinions.

Best regards, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 28 Feb 2012 13:28:12 +1300

I've always had an issue with this concept of "Idempotence".  I really 
wonder whether it actually exists in the real web.

Many servers can and do have side effects for any particular method.

So one cannot claim that any method is in fact Idempotent?  Not in all 
cases anyway.  One could write a script for which POST was idempotent.  
Likewise there are plenty of URIs where GET is not.

I think the concept is not well understood in the web authoring space, 
and no-one really takes much care to make website implementations honour 
the intent of HTTP wrt methods which are supposed to be Idempotent.

My point is, why persist with this language?

Adrien

On 28/02/2012 4:54 a.m., Julian Reschke wrote:
> You could ask the same question for most methods; PUT means "store" 
> and always has. There's no good reason to change that (allowing 
> partial updates would be a change).
>
> Then I believe the right thing is to request changes to these 
> libraries so that they handle PATCH (for instance) better.
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 is released! - http://www.wingate.com/getlatest/

---

From: James M Snell <[email-masked]>
Date: Mon, 27 Feb 2012 16:40:49 -0800

The possible idempotency of a partial update depends entirely on the nature
of the resource being updated, not the method used. At the http level, a
partial update can never be assumed to be idempotent, even if that's how
it's implemented in one specific case. The semantics of put are clear in
that it always must be considered to be idempotent.
On Feb 27, 2012 4:30 PM, "Adrien de Croy" <[email-masked]> wrote:

> I think the concept is not well understood in the web authoring space, and
> no-one really takes much care to make website implementations honour the
> intent of HTTP wrt methods which are supposed to be Idempotent.
>
>> was ambiguous; witness all the sites that did/do destructive things on GET,
>> that do not implement HEAD, etc.
>>
>> always has. There's no good reason to change that (allowing partial updates
>> would be a change).
>>
>> happened before on rest-discuss I believe), and you won't convince me, so
>> I'd be interested to hear other people's opinions.
>>
>> Best regards, Julian
>>
>>
> --
> WinGate 7 is released! - http://www.wingate.com/**getlatest/<http://www.wingate.com/getlatest/>
>
>
>

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 28 Feb 2012 13:54:57 +1300

I wasn't talking so much about partial updates using PUT, but the 
general reliance on the concept of idempotence at all.

Various features in HTTP, such as pipelining, recovering from closes etc 
rely on certain methods being idempotent.  I would propose one can never 
assume any method is idempotent, since that is up to server-side 
implementation.

Whether in practise PUT is usually actually idempotent or not I don't 
have any information.

But my feeling is there is a disconnect between HTTP and those making 
websites in this regard.

On 28/02/2012 1:40 p.m., James M Snell wrote:
>
> The possible idempotency of a partial update depends entirely on the 
> nature of the resource being updated, not the method used. At the http 
> level, a partial update can never be assumed to be idempotent, even if 
> that's how it's implemented in one specific case. The semantics of put 
> are clear in that it always must be considered to be idempotent.
>
> On Feb 27, 2012 4:30 PM, "Adrien de Croy" <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     I've always had an issue with this concept of "Idempotence".  I
>     really wonder whether it actually exists in the real web.
>
>     So one cannot claim that any method is in fact Idempotent?  Not in
>     all cases anyway.  One could write a script for which POST was
>     idempotent.  Likewise there are plenty of URIs where GET is not.
>
>     I think the concept is not well understood in the web authoring
>     space, and no-one really takes much care to make website
>     implementations honour the intent of HTTP wrt methods which are
>     supposed to be Idempotent.
>
>         Just because implementations get something wrong doesn't mean
>         the spec was ambiguous; witness all the sites that did/do
>         destructive things on GET, that do not implement HEAD, etc.
>
>         You could ask the same question for most methods; PUT means
>         "store" and always has. There's no good reason to change that
>         (allowing partial updates would be a change).
>
>         I think it's pretty clear that I won't convince you (the
>         discussion happened before on rest-discuss I believe), and you
>         won't convince me, so I'd be interested to hear other people's
>         opinions.
>
>         Best regards, Julian
>
>
>     --

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 28 Feb 2012 14:18:58 +1300

On 28.02.2012 04:54, Julian Reschke wrote:
> Just because implementations get something wrong doesn't mean the
> spec was ambiguous; witness all the sites that did/do destructive
> things on GET, that do not implement HEAD, etc.
>

This is a side-track argument anyway. To work with partials PUT also 
requires If-Match. Without it PUT becomes non-idempotent and one might 
as well use PATCH.

That side effect is not very visible in the 2616 wording. In the new 
HTTPbis wording there is no choice of getting it wrong, they need to use 
PATCH.

> libraries so that they handle PATCH (for instance) better.

Yes.

> so I'd be interested to hear other people's opinions.

In order to work reliably for partials PUT requires If-Match and 
*additionally* needs some header mechanism to determine the difference 
between full and partial replacement (Range, Content-Type, etc).

In short its *more* complex to use PUT than PATCH.

/2c

AYJ

---

From: Philippe Mougin <[email-masked]>
Date: Tue, 28 Feb 2012 02:36:11 +0100

Le 28 févr. 2012 à 01:28, Adrien de Croy a écrit :

> I've always had an issue with this concept of "Idempotence".  I really wonder whether it actually exists in the real web.
> 

From the latest httpbis draft : "Request methods can also have the property of "idempotence" in that, aside from error or expiration issues, the intended effect of multiple identical requests is the same as for a single request. PUT, DELETE, and all safe request methods are idempotent.  It is important to note that idempotence refers only to changes requested by the client: a server is free to change its state due to multiple requests for the purpose of tracking those requests, versioning of results, etc."

> So one cannot claim that any method is in fact Idempotent?  Not in all cases anyway.  One could write a script for which POST was idempotent.  Likewise there are plenty of URIs where GET is not.
> 
> I think the concept is not well understood in the web authoring space, and no-one really takes much care to make website implementations honour the intent of HTTP wrt methods which are supposed to be Idempotent.
> 

I'd say because it allows us to create distributed systems that benefit from it. Regardless of how well it is understood by some people, I am glad I can benefit from it in the systems I develop.

Philippe Mougin

---

From: Zhong Yu <[email-masked]>
Date: Mon, 27 Feb 2012 19:37:19 -0600

> updates considered acceptable?

How would partial PUT work? Client cuts the whole entity into pieces,
and PUT each one individually? That is not atomic - what happens to a
GET request before the whole thing is finished?

I don't think PATCH addresses OP's concern, that is, reducing
retransmission cost in face of network failure (which should really be
TCP's job).

Zhong

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 28 Feb 2012 02:48:29 +0100

> I would propose one can never assume any method is idempotent, since that is up to server-side implementation.

A number of functions in the HTTP architecture rely on the defined properties of the methods (such as idempotence, safeness).
If the methods no longer have useful properties, we can replace all of them with MAKEITSO or some such.

> Whether in practise PUT is usually actually idempotent or not I don't have any information. 

Plain HTML pages can only trigger GET (links) and POST (forms), so PUT has been a bit of a specialty for the browser web.
Ajax and RESTful Web APIs changed that.  
I seem to remember Amazon AWS relied on range-PUTs heavily for a while (which, I think, is what we are discussing here).
Incidentally, the other REST protocol, CoAP, has them (and needs them for requests with data > desired datagram size), but in a specialized form.

> But my feeling is there is a disconnect between HTTP and those making websites in this regard.

There used to be a strong disconnect a decade ago.
Prefetchers like Google Web Accelerator helped fix that :-)
Security considerations further caused people to keep the difference between methods in mind.
Modern web dev frameworks also helped sites to start conforming.
Nowadays, rookie web developers learn this stuff pretty quickly.

Grüße, Carsten

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 28 Feb 2012 14:56:13 +1300

On 28.02.2012 13:54, Adrien de Croy wrote:
> Various features in HTTP, such as pipelining, recovering from closes
> etc rely on certain methods being idempotent.  I would propose one 
> can
> never assume any method is idempotent, since that is up to 
> server-side
> implementation.

It is required for cacheability assumptions on the response in absence 
of explicit "Cache-Control: public". Since almost no sites explicitly 
send "public" we can't exactly erase that property without causing a 
whole lot of pain to a lot of network administrators.

An HTTP bandwidth increase of between 20% and 50% on certain Tier-1 
network pipes is at stake and not something to play around with.

Intermediary admin already have to violate the specs to a certain 
degree, ignoring no-cache to reduce bandwidth costs wrongly imposed by 
broken server libraries which insist on sending no-cache and no-store on 
static content.

> But my feeling is there is a disconnect between HTTP and those making

IME the disconnect is happening around about the library/framework/SDK 
layer. The documentation for those is spares on HTTP details, and some 
of their defaults are actually causing violations of the HTTP specs. 
This thread on PUT with partials being a case in point. PHP insistence 
on sending prohibitive cache-controls by default has also led to many 
broken web apps. ASP has similar problems. the litany of implementation 
compliance bugs and side effects is _long_.

AYJ

---

From: Martin Thomson <[email-masked]>
Date: Mon, 27 Feb 2012 20:28:25 -0800

I think that the "problem" in this case is one of invention only.  If
you desire the ability to do a partial update of a resource, then you
probably don't have enough resources.  I know that's a gross
generalization, but I haven't seen that comment in the thread.

I agree with Mike that PATCH (or a special POST) aren't visibly
idempotent, which is a crucial characteristic if this is going to
work.

---

From: "Eric J. Bowman" <[email-masked]>
Date: Mon, 27 Feb 2012 22:27:38 -0700

> idempotent, which is a crucial characteristic if this is going to

If 99.9% of partial updates are non-idempotent, wouldn't the need for
idempotent partial update be an edge case, as opposed to crucial?

What we're trying to make visible on the wire is *sender intent* not
idempotency.  The sender doesn't intend to make idempotent vs. non-
idempotent requests.  Idempotency is a property of the request method,
not a sender intent in and of itself.

If the sender intent is partial update, the method is PATCH.  If the
sender intent is full update, the method is PUT -- that intent is what
makes a request idempotent or not.  It wouldn't make sense to determine
idempotency at a glance, but then still need to reverse-engineer sender
intent.

Which is where we'd be if PUT was allowed to sometimes mean partial
update -- we'd need to consider other factors to determine sender
intent, instead of simply looking at the request method.  How would
that be more visible?

If sender intent was based on idempotency, we'd have two methods --
IDEMP and NOTIDEMP -- and wouldn't care about distinguishing full
update from partial update from retrieval.  But, sender intent is about
what action to take, not the properties of that action.  That's what
needs to be visible.

-Eric

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Feb 2012 09:37:45 +0100

> idempotent, which is a crucial characteristic if this is going to

How exactly is it "not visible"?

 From <http://greenbytes.de/tech/webdav/rfc5789.html#rfc.section.2.p.4>:

"A PATCH request can be issued in such a way as to be idempotent, which 
also helps prevent bad outcomes from collisions between two PATCH 
requests on the same resource in a similar time frame. Collisions from 
multiple PATCH requests may be more dangerous than PUT collisions 
because some patch formats need to operate from a known base-point or 
else they will corrupt the resource. Clients using this kind of patch 
application SHOULD use a conditional request such that the request will 
fail if the resource has been updated since the client last accessed the 
resource. For example, the client can use a strong ETag [RFC2616] in an 
If-Match header on the PATCH request."

Best regards, Julian

---

From: Mike Kelly <[email-masked]>
Date: Tue, 28 Feb 2012 10:10:20 +0000

> not a sender intent in and of itself.

No. What we should be trying to make visible "on the wire" are
properties of a request that are useful/valuable for intermediate
processing.

The idempotency of a request is valuable for intermediate processing,
because infrastructure can be developed to re-issue a client request
on network failure.

Having a request guaranteed to be non-partial is not useful or
valuable for intermediate processing, apparently there are no examples
of intermediary mechanisms which leverage this.

Cheers,
Mike

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Feb 2012 11:19:41 +0100

On 2012-02-28 11:10, Mike Kelly wrote:
> On Tue, Feb 28, 2012 at 5:27 AM, Eric J. Bowman<[email-masked]>  wrote:
>> Martin Thomson wrote:
>>>

Just because something isn't useful to intermediaries (with which I 
don't fully agree) doesn't mean it isn't useful to be defined.

Best regards, Julian

---

From: Mike Kelly <[email-masked]>
Date: Tue, 28 Feb 2012 10:23:03 +0000

On Tue, Feb 28, 2012 at 8:37 AM, Julian Reschke <[email-masked]> wrote:
> "A PATCH request can be issued in such a way as to be idempotent, which also
> helps prevent bad outcomes from collisions between two PATCH requests on the
> same resource in a similar time frame. Collisions from multiple PATCH
> requests may be more dangerous than PUT collisions because some patch
> formats need to operate from a known base-point or else they will corrupt
> the resource. Clients using this kind of patch application SHOULD use a
> conditional request such that the request will fail if the resource has been
> updated since the client last accessed the resource. For example, the client
> can use a strong ETag [RFC2616] in an If-Match header on the PATCH request."

The amount of infrastructure that exists right now, or in the near
future, which understands and is built to leverage idempotent PATCH is
tiny in comparison to PUT. Visibility, in practice, depends on the
relative ubiquity of the semantic, and not just on whether a
spec-which-says-so makes it through to RFC.

So, from a purely theoretical stand-point you may be right, but in the
real-world I don't think their visibility is even close to equivalent.

Cheers,
Mike

---

From: Mike Kelly <[email-masked]>
Date: Tue, 28 Feb 2012 10:39:24 +0000

On Tue, Feb 28, 2012 at 10:19 AM, Julian Reschke <[email-masked]> wrote:
> Just because something isn't useful to intermediaries (with which I don't
> fully agree) doesn't mean it isn't useful to be defined.
>

If we're talking about 'visibility on the wire', we should be talking
about facilitating intermediary processing.

I have a couple of further questions:

- why don't you fully agree?
- how is PUT being unambiguously non-partial useful?

Bear in mind applications which interact through HTTP can define
whether or not their use(s) of PUT are non-partial without HTTP being
involved - many already do this.

Cheers,
Mike

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 29 Feb 2012 01:38:26 +1300

> of intermediary mechanisms which leverage this.

What seems not so obvious, is that the PUT request-body is cacheable for 
the request-URI. Middleware can leverage this immediately on a 2xx 
response to start serving it up without wasting further bandwidth 
re-fetching it. People are already hacking away at the code trying to 
get as much as possible cached. Bandwidth reduction is a hot topic in 
the cloud based web app world. You can expect to see implementations 
appearing within the year, now that there is an actual need appearing.

As has been mentioned by others already a lot of HTTP capabilties were 
rare due to HTML limitations affecting the browsers. Custom HTTP agents 
have been using these features invisibly for some time. Browsers XHR 
support and HTML5 features are opening the doors to a great many web 
apps using these features in ways never before seen (outside the dreams 
of the spec writers anyway, maybe not even there).

AYJ

---

From: Mike Kelly <[email-masked]>
Date: Tue, 28 Feb 2012 13:31:19 +0000

On Tue, Feb 28, 2012 at 12:38 PM, Amos Jeffries <[email-masked]> wrote:
> What seems not so obvious, is that the PUT request-body is cacheable for the
> request-URI. Middleware can leverage this immediately on a 2xx response to
> start serving it up without wasting further bandwidth re-fetching it. People
> are already hacking away at the code trying to get as much as possible
> cached.

Do you know of any caching infrastructure that is implemented to this effect?

Does it conflict with the following:

http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-18#section-6.6

"Responses to the PUT method are not cacheable.  If a PUT request
passes through a cache that has one or more stored responses for the
effective request URI, those stored responses will be invalidated"

Cheers,
Mike

---

From: Martin Thomson <[email-masked]>
Date: Tue, 28 Feb 2012 08:30:19 -0800

On 28 February 2012 00:37, Julian Reschke <[email-masked]> wrote:
> "A PATCH request can be issued in such a way as to be idempotent, [...]

"Can be" not "is".  You can issue a non-idempotent PATCH.  Though you
are advised not to, it is possible.  Therefore, an intermediary cannot
assume that the request is safe to retry.

A PUT is by definition idempotent.  A PATCH, like POST, is only
idempotent under (secret, non-obvious) conditions.

--Martin

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Feb 2012 17:49:38 +0100

On 2012-02-28 17:30, Martin Thomson wrote:
> On 28 February 2012 00:37, Julian Reschke<[email-masked]>  wrote:

It can, if it comes with If-Match. Or am I missing something?

> idempotent under (secret, non-obvious) conditions.

That is true, but I'm not sure that is a problem when the conditions are 
well understood and documented.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 28 Feb 2012 18:31:08 +0100

I'm not sure we are communicating.

In the web-as-deployed, partial updates often use the PUT method.
Intermediaries know these are idempotent and can do interesting things to these interactions.

In the web-as-soon-specified, partial updates are now moving to PATCH.
Intermediaries don't know anything and cannot do anything that would rely on idempotence.

Is that a problem?  
A big enough problem that would move us to do something about the point the specs now seem to be converging towards?

I don't know.  I just want to make sure we communicate about this change.

Grüße, Carsten

---

From: Martin Thomson <[email-masked]>
Date: Tue, 28 Feb 2012 10:23:51 -0800

> well understood and documented.

That's the part that needs to be clearer.  Because I'm not certain
that I can look at a given PATCH (or POST) request and make a
determination about idempotence. The problem is that RFC 5789
categorically denies that PATCH is idempotent (then goes on to qualify
as you cited previously.)

Is it always true that a PATCH request with an If-Match header
containing a strong ETag is idempotent?  I think that it is as long as
the contract is observed (i.e. the server pays attention to the
If-Match; the ETag really is strong).

I don't think that any of the other conditional headers can be used to
make a statement of the same certitude.  Of course, there are plenty
of requests that could be non-idempotent, but - if you understand the
context - are actually idempotent.  That's the point of contention: is
it desirable that this property be observable (thanks Carsten)?

I don't see a strong reason to.  (What I think you saw as motivation
to do so, was just an attempt at clarifying a technical point.)  I
tend to think that if this is important, you need to restructure your
resources (my original point).

--Martin

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 28 Feb 2012 11:54:51 -0800

> In the web-as-deployed, partial updates often use the PUT method.

In the Web as defined, standardized, and deployed, those partial
updates using the PUT method are not interoperable with standard
HTTP/1.x servers.  That has been known since the idea was first
brought up and has not changed since then.  We could not make
non-compatible changes to an existing method in 1996, nor 1999,
nor can we do so now in 2012.  There is no such thing as partial
updates using PUT.  At best, there are some standard servers that
actively reject partial updates using PUT in order to prevent such
misbegotten behavior from corrupting their data store.

Any software that uses Content-Range in PUT requests is BROKEN.
Regardless of what we have in the spec, they are still BROKEN.
They simply do not interoperate over HTTP.  They only interoperate
with non-standard servers.

That is why I defined PATCH in 1994.  That is why there is a
proposed standard for PATCH now.  That is why all HTTP servers
that want to support partial updates will either use PATCH or
be forever broken.

Standards are based on what we know will work on the Internet.
We know partial PUT does not work.  Hence, it is not and has never
been part of the HTTP standard.

....Roy

---

From: Mike Kelly <[email-masked]>
Date: Wed, 29 Feb 2012 17:28:49 +0000

On Tue, Feb 28, 2012 at 7:54 PM, Roy T. Fielding <[email-masked]> wrote:
> HTTP/1.x servers.

Please could you be more specific

> That has been known since the idea was first
> nor can we do so now in 2012.

The inference here being there is infrastructure which relies on PUT
requests being non-partial _across the whole of the web_? Do you have
some examples?

> There is no such thing as partial updates using PUT.

Maybe in a theoretical sense, but there are clients and servers
interacting this way on the web right now. Everything seems to be
going OK despite this.

Cheers,
Mike

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 29 Feb 2012 18:46:03 +0100

On Feb 28, 2012, at 20:54, Roy T. Fielding wrote:

> HTTP/1.x servers.  [...]

Thanks, that was part of the input I was looking for.

For the other part of discussion we are having (PATCH vs. idempotence), it would be useful to know what specifically broke when people started using PUT for partial updates.  In particular, it would be good to know if the same breakage occurs with an idempotent version of PATCH.

Grüße, Carsten

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 29 Feb 2012 10:50:23 -0800

> For the other part of discussion we are having (PATCH vs. idempotence), it would be useful to know what specifically broke when people started using PUT for partial updates.  In particular, it woul…

When a server that implemented PUT prior to the introduction of
Content-Range received a partial content body that included such
a range, they would replace the entire resource representation with
the partial body.  That is how PUT was defined to work.  That's how
it was implemented long before ranges existed.  Hence, the introduction
of partial PUT would only be possible with a strong coupling between
client and origin server implementations, which violates the design
of the HTTP.

By design, HTTP method semantics cannot be changed in an incompatible
way without a major version change in the protocol.  Since the WG
refused to bump the protocol version, partial PUT was forbidden.
PATCH was delegated to the WebDAV group to standardize.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Feb 2012 19:54:18 +0100

Yes.

But I *think* there's some terminology confusion here.

When Mike, who brought this up, talks about partial updates with PUT, 
he's referring to a PUT request *without* Content-Range, replacing 
*parts* of the identified resource; and leaving the remainder alone.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 29 Feb 2012 11:00:39 -0800

On Feb 29, 2012, at 10:54 AM, Julian Reschke wrote:

> When Mike, who brought this up, talks about partial updates with PUT, he's referring to a PUT request *without* Content-Range, replacing *parts* of the identified resource; and leaving the remainder alone.

Which is even less interoperable than the Content-Range hack because
now the site cannot even detect it.

As I've told Mike a dozen times already, what he describes is contrary
to the definition of PUT.  It is therefore invalid HTTP.  Whether it is
valid in other protocols is not our problem.

....Roy

---

From: Mike Kelly <[email-masked]>
Date: Wed, 29 Feb 2012 19:09:43 +0000

> valid in other protocols is not our problem.

It can't be unambiguously contrary to 2616, otherwise why would there
have been any need to clarify the semantics via HTTPbis?

Sites can detect whether it is partial vs replace the same way they
distinguish between the intent of a POST requests - it depends on the
resource in question.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Feb 2012 20:37:33 +0100

> have been any need to clarify the semantics via HTTPbis?

I've answered that several times. We clarify things when we see that 
people get them wrong. That doesn't mean we made a normative change, we 
just try to optimize the spec for people who apparently have read 
something into 2616 which isn't there.

> distinguish between the intent of a POST requests - it depends on the

The *site* can't know, unless sender and receiver are closely coupled. 
In which case you can of course do what you want, but don't claim that 
this is HTTP as specified.

Best regards, Julian

---

From: Mike Kelly <[email-masked]>
Date: Wed, 29 Feb 2012 19:59:41 +0000

On Wed, Feb 29, 2012 at 7:37 PM, Julian Reschke <[email-masked]> wrote:
> The *site* can't know, unless sender and receiver are closely coupled. In
> which case you can of course do what you want, but don't claim that this is
> HTTP as specified.
>

Right, it is closely coupled - as is any other interaction driven by
shared understanding between the client and the server. Not everything
needs, or can be, visible on the network and is better off dealt with
via shared understanding between client and server. What is it about
the non-partial'ness of PUT that requires it to be made visible by
HTTP across the web as a whole?

Cheers,
Mike

---

From: Henrik Nordström <[email-masked]>
Date: Wed, 29 Feb 2012 22:46:22 +0100

> HTTP across the web as a whole?

Simply the fact that HTTP/1.x do not allow upgrades of defined methods
in this manner.

If this "partial PUT" is sent to a server that do not understand it
would result in breakage, therefore it can not be accepted as an
extension of the existing PUT method. Instead it needs to use a new
method, making sure that breakage do not occur if the request is
attepmted somewhere where it's not supported.

Regards
Henrik

---

From: Henrik Nordström <[email-masked]>
Date: Wed, 29 Feb 2012 23:18:42 +0100

tis 2012-02-28 klockan 10:23 -0800 skrev Martin Thomson:

> containing a strong ETag is idempotent?

Per the definition of strong etag and If-Match it's safe but not really
idempotent in the view of the client (response differs), but server
state is idempotent. Except that it's not spelled out anywhere that I
know of.

This under the condition that the server do implement If-Match. But
given that it's an MUST condition it's quite reasonable to expect
If-Match to be implemented in the cases of partial updates and similar
if ETag is.

> make a statement of the same certitude.

If-Unmodified-Since can also be considered a strong enough condition
under certain conditions, but it's hard for an intermediary or server to
verify that these conditions are fulfilled as it's dependent on the Date
of the earlier response in relation to it's Last-Modified. But it's
accepted as a safe tradeoff to simply trust that the client only uses
If-Unmodified-Since in situations requiring a strong validator under
such conditions that the carried Last-Modified value is considered
strong.

>  Of course, there are plenty

It's very desirable that it's well known when idempotence can be
unconditionally assumed based on the request alone, as it relates to
when requests can safely be retried at the transport layer in response
to transport failures/events, and also have direct impact on safe use of
transport features such as connection keep-alive and pipelining.

>From this follows that it's important that requests defined as
idempotent is not implemented in an non-idempotent manner as it could
easily result in hard to diagnose timing dependent breakage and
corrupted data.

It also desirable that idempotent methods are used for idempotent
actions, but using requests defined as non-idempotent for idempotent
actions do not cause any breakage, only slight loss of efficiency.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 01 Mar 2012 11:34:56 +1300

On 28/02/2012 2:56 p.m., Amos Jeffries wrote:
>> etc rely on certain methods being idempotent.  I would propose one can
>> never assume any method is idempotent, since that is up to server-side
>> implementation.
>
> broken server libraries which insist on sending no-cache and no-store 
> on static content.

Last time I sampled Cache-control response headers (over couple million 
hits crawling sites), I found a large majority use it to prevent 
caching.  Very few to enable it.  It's a shame.

so moving from a naive HTTP/1.0 style cache to a compliant HTTP/1.1 
style cache actually resulted in a huge reduction in cache utility.  
Without ignoring cache-control directives as you say, it's hard to get 
more than a 10% effective bandwidth benefit from caching, which frankly 
is not worth the pain.

But at least Cache-control as you say is explicit, and doesn't rely on 
any assumptions based purely on the method about whether it is indeed 
safe or not to retry.

So I guess non-compliant websites end up seeing the effect of this when 
clients assuming idempotence of a method cause unwanted side-effects in 
the site, and that's the only real incentive the web developers have to 
fix the non-idempotence.

I wonder whether this could be better explained to web devs.  I can just 
imagine the blank looks "idem-what?".  Or indeed if it's really only a 
problem of my imagining :)

> layer. 

yeah I agree.  Like CMS packages that take the easiest route to dealing 
with potential caching issues instead of using good design.  Even 
revalidation is preferable to regeneration and retransmission.

Regards

Adrien

> The documentation for those is spares on HTTP details, and some of 
> their defaults are actually causing violations of the HTTP specs. This 
> thread on PUT with partials being a case in point. PHP insistence on 
> sending prohibitive cache-controls by default has also led to many 
> broken web apps. ASP has similar problems. the litany of 
> implementation compliance bugs and side effects is _long_.
>
> AYJ
>
>

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 1 Mar 2012 09:56:16 +1100

On 01/03/2012, at 9:34 AM, Adrien de Croy wrote:

> Last time I sampled Cache-control response headers (over couple million hits crawling sites), I found a large majority use it to prevent caching.  Very few to enable it.  It's a shame.
> 
> so moving from a naive HTTP/1.0 style cache to a compliant HTTP/1.1 style cache actually resulted in a huge reduction in cache utility.  Without ignoring cache-control directives as you say, it's hard to get more than a 10% effective bandwidth benefit from caching, which frankly is not worth the pain.

Not sure what you're crawling, but my experience is that effective bandwidth savings is MUCH higher, even on a conservatively configured cache.

And anecdotal evidence suggests it's getting better; see:

http://httparchive.org/trends.php (~45% of responses have caching headers)
http://httparchive.org/interesting.php  (~40% of responses with CC have a max-age > 0)

… and if you restrict to the top 1000 sites, both numbers are higher, about 60% each. Combined with heuristic freshness, what's the problem?

Cheers,

--
Mark Nottingham
http://www.mnot.net/

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 01 Mar 2012 12:00:45 +1300

> Not sure what you're crawling, but my experience is that effective bandwidth savings is MUCH higher, even on a conservatively configured cache.

it was a couple years ago now.

> http://httparchive.org/interesting.php  (~40% of responses with CC have a max-age>  0)

ok, that 40% of 45% = 18% overall.

> … and if you restrict to the top 1000 sites, both numbers are higher, about 60% each. Combined with heuristic freshness, what's the problem?

did you do any tests without heuristic freshness?  That's the area where 
in the past we've had the most problems.

Cheers

Adrien

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 1 Mar 2012 10:02:09 +1100

I'm talking to Steve about getting more fine-grained caching stats in there now…

On 01/03/2012, at 10:00 AM, Adrien de Croy wrote:

> did you do any tests without heuristic freshness?  That's the area where in the past we've had the most problems.
> 
> Cheers
> 
> Adrien
> 
>> 
>> Cheers,
>> 
>> --
>> Mark Nottingham
>> http://www.mnot.net/
>> 
>> 
>> 
>> 
> 
> -- 

--
Mark Nottingham
http://www.mnot.net/

---

From: Henrik Nordström <[email-masked]>
Date: Thu, 01 Mar 2012 00:07:51 +0100

> revalidation is preferable to regeneration and retransmission.

idempotence is not about caching at all.

it's about safely being able to retry the exact same request when
needed, resulting in the expected outcome.

Regards
Henrik

---

From: Mike Kelly <[email-masked]>
Date: Wed, 29 Feb 2012 23:52:14 +0000

> attepmted somewhere where it's not supported.

Upgrading PUT so it is less specific about replacement wouldn't result
in this breakage. Clients don't make requests to servers arbitrarily,
they make them according to whatever application they are fulfilling.
i.e. if an application is operating on the basis that PUT requests to
its resources are replacements, then HTTP relaxing the semantics of
PUT to permit partials would not create breakage.

Cheers,
Mike

---

From: Henrik Nordström <[email-masked]>
Date: Thu, 01 Mar 2012 01:28:29 +0100

> PUT to permit partials would not create breakage.

That's besides the point. To be able to extend PUT like this within
HTTP/1.x it needs to be possible to send the extended request form
arbitrarily without causing breakage. A condition that partial PUT may
only be sent if it's known prior to sending the request that the
receiving server is somehow magically capable of accepting this form of
PUT without causing breakage is not acceptable within HTTP/1.x.

For a change in PUT syntax to be acceptable within HTTP/1.1 you MUST be
able to send such PUT requests to any server without any prior knowledge
of the capabilities of that server and know that the server will process
the request with an acceptable outcome. Storing the partial
representation as the sole representation of the resource is not
regarded as an acceptable outcome so this is not acceptable extension of
PUT within HTTP/1.x.

Regards
Henrik

---

From: Martin Thomson <[email-masked]>
Date: Wed, 29 Feb 2012 16:29:32 -0800

2012/2/29 Mike Kelly <[email-masked]>:
> in this breakage.

PUTv2...which is what PATCH does.

> Clients don't make requests to servers arbitrarily,

That's coupling.  c.f. statements about that being bad.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 29 Feb 2012 16:35:07 -0800

2012/2/29 Henrik Nordström <[email-masked]>:
> actions,

Actually, a server has to be prepared for the consequences if it
treats idempotent requests in a non-idempotent fashion.  Clients (and
intermediaries) should not bear the responsibility for bad code...

> but using requests defined as non-idempotent for idempotent

Depends on the axis you use for measuring efficiency.  I somewhat like
the resiliency afforded by idempotent operations and prefer to use
POST and PATCH in a way that they are effectively idempotent, within
the context of the application.  So retrying doesn't result in waste
of other types (orphaned state, etc).

---

From: Mike Kelly <[email-masked]>
Date: Thu, 1 Mar 2012 01:04:02 +0000

> regarded as an acceptable outcome so this is not acceptable extension of

Ok, so it sounds to me like you are saying that what would happen in
reality is besides the point because there is a rule governing the
design of PUT in HTTP/1.x which must not be broken. Is there any
evidence of that rule producing some benefit to the web?

Cheers,
Mike

---

From: Henrik Nordström <[email-masked]>
Date: Thu, 01 Mar 2012 02:29:51 +0100

> intermediaries) should not bear the responsibility for bad code...

Yes, there is no non- above. What you mention here was covered in
another part.

> of other types (orphaned state, etc).

There is a slight loss of efficiency as intermediaries can not tell
these are idempotent so they can't pipeline them and also need to take
special care when forwarding the request to an existing persistent
server connection as they can not afford running into concurrent server
close condition when forwarding a seemingly non-indempotent request.

Regards
Henrik

---

From: Henrik Nordström <[email-masked]>
Date: Thu, 01 Mar 2012 02:35:41 +0100

> evidence of that rule producing some benefit to the web?

Partial PUT have only been deployed in very isolated use cases and I
have not been in touch of any of them. I am very sure we would have seen
numerous interop issues and breakage if it had been more widely adopted
that PUT can be used in this manner.

To counter your question, is there any evidence that misusing PUT for
partial updates compared to defining a new method for the purpose have
had any positive benefit to the web?

Regards
Henrik