Subject: RE: [Fwd: I-D ACTION:draft-dusseault-http-patch-09.txt]

From: Yaron Goland <[email-masked]>
Date: Wed, 5 Sep 2007 14:01:23 -0700

Don't the default HTTP header semantics already provide the same behavior as Prefer (e.g. pass end-to-end and ignore if not understood)? If so then doesn't Prefer essentially create a new HTTP header system within HTTP? If so wouldn't it be easier to just get rid of the Prefer header and instead use new HTTP headers for values like Content-Returned?

In my original mail I was suggesting the following additional changes to the spec:
1. Put in a note clarifying that 209 is currently only defined for PUT/POST/PATCH.
2. Put in a note clarifying that it is legal (read: possible) to have a situation where after the method (E.g. PUT/POST/PATCH) is executed and before the GET response is generated additional changes might happen.

        Thanks,

                Yaron

> -----Original Message-----
> From: James M Snell [mailto:[email-masked]]
> Sent: Wednesday, August 29, 2007 9:08 PM
> To: Yaron Goland
> Cc: HTTP Working Group
> Subject: Re: [Fwd: I-D ACTION:draft-dusseault-http-patch-09.txt]
>
> Grr... Yaron, I'm sure you're a good person and all but your mail
> clients inability to wrap lines of text sucks. ;-)
>
> In any case.. some comments below (with the quotes from you manually
> wrapped ;-) ..)
>
> Yaron Goland wrote:
> > As part of the Web3S effort I need to define the equivalent of
> "Prefer: content-returned" and 209.
> >
> > My current intention is to introduce a dedicated HTTP request header
> called "Web3SGetResponse"
> > whose semantics are the same as content-returned. The reason why I
> have chosen not to use Prefer
> > is that I am uncomfortable with placing all "safe to ignore" headers
> in HTTP into a single Prefer
> > header. In other words I can't figure out what kind of 'safe to
> ignore' header wouldn't end up in
> > Prefer and putting everything into Prefer just doesn't seem right.
>
> The semantics of Prefer are clear... it's where the client can tell the
> server what behaviors it would like the server to implement without
> placing any requirements on the server.  While it certainly could be
> overloaded to do lots of other things, doing so would definitely be a
> mistake.  That possibility, however, does not lessen the value of the
> Prefer header.
>
> >
> > I choose the name "Web3SGetResponse" to indicate that I'm not trying
> to solve the general problem,
> > I'm just solving a specific problem in the context of Web3S. If a
> more generic solution is agreed upon
> > I would be happy to use it. But my vote in that case would be not to
> use the Prefer header but to
> > introduce a dedicated "please give me a 209" header.
>
> I'm all for solving specific problems with targeted solutions.
> However,
> this one in particular seems to be generic and broadly applicable
> enough
> that a general solution is well waranted.
>
> >
> > 209, on the other hand, makes sense to me. My litmus test for HTTP
> response codes is
> > "Can a client, without domain specific knowledge, do something useful
> with the code?" If the
> > answer is 'no' then re-use an existing code like 200 or 400. If 'yes'
> then adding a new code
> > makes sense. In this case 209 seems to meet the bar so I am just
> using it and referencing>
> > draft-dusseault-http-patch-09 in my spec.
> >
>
> Good to hear.
>
> > In reading sections 3 and 4 it would seem to a naïve reader (e.g. me)
> that
> > Prefer: content-returned and 209 could be used with any method. But I
> could only
> > figure out how to make it make sense for PUT/POST/PATCH. So in my own
> spec, at least
> > for Web3S resources, I have restricted Web3SGetResponse and 209 to
> only be used on
> > PUT/POST/PATCH (although we still use the name UPDATE, but that's a
> whole other discussion).
>
> That's perfectly fine.  Not all status codes make sense for all methods
> in every case, that's true for all of the codes defined in RFC2616 and
> its true for 209.
>
> >
> > In my current spec I threw in some language that basically says that
> a 209 response
> > might include content that goes beyond the changes in the associated
> method. E.g. in
> > theory the following is legal according to my spec:
> > Time 0 - Issue PUT with Web3SGetResponse
> > Time 1 - PUT succeeds
> > Time 2 - Another method that changes the resource's state is
> successfully executed
> > Time 3 - The 209 response body to the PUT received at Time 0 is
> created and returned
> > It isn't clear to me if this behavior is legal according to the PATCH
> spec.
>
> assuming that 0, 1 and 3 all fall within the scope of a single HTTP
> request-response then yes, this should likely be the expected behavior.
>  I wouldn't go so far as to say that it MUST work this way, however.
>
> >
> > Also I put in some text that said that 209s can, in theory, be
> returned
> > even if a client hasn't specified Web3SGetResponse but it isn't
> encouraged.
> > I wasn't sure if this is legal according to the PATCH spec.
>
> Yes, this would be fine.
>
> - James

---

From: Yaron Goland <[email-masked]>
Date: Wed, 5 Sep 2007 14:09:53 -0700

What's the use case for wanting to resolve this race condition? If someone issues a change to a resource and someone else issues a subsequent change to alter the resource's state again then who cares what the interim state was since it is now gone?

In the Web3S case we are explicitly not trying to resolve this race condition. Our interest in 209 is strictly as an optimization. We often find ourselves in a situation where we want to issue a PUT/POST and we need to get the state back. Issuing a PUT/POST followed by a pipelined GET is expensive both because pipelining isn't well supported and because if the PUT/POST failed now the client has to receive and the server has to send a completely useless GET. So having the ability to ask for the GET response in the PUT/POST response is a very useful optimization to us.

        Thanks,

                        Yaron

> From: Henrik Nordstrom [mailto:[email-masked]]
> Sent: Thursday, August 30, 2007 5:01 AM
> To: Yaron Goland
> Cc: James M Snell; HTTP Working Group
> Subject: RE: [Fwd: I-D ACTION:draft-dusseault-http-patch-09.txt]
>
> On ons, 2007-08-29 at 17:34 -0700, Yaron Goland wrote:
>
> a 209 response might include content that goes beyond the changes in
> the associated method. E.g. in theory the following is legal according
> to my spec:
> > Time 0 - Issue PUT with Web3SGetResponse Time 1 - PUT succeeds Time 2
> > - Another method that changes the resource's state is successfully
> > executed Time 3 - The 209 response body to the PUT received at Time 0
> > is created and returned It isn't clear to me if this behavior is
> legal according to the PATCH spec.
> >
> returned even if a client hasn't specified Web3SGetResponse but it
> isn't encouraged. I wasn't sure if this is legal according to the PATCH
> spec.
>
> From what I can tell this family of race conditions is the main
> motivation for the 209 response code.
>
> Consider the following:
>
> Time 0 - Client A issues PUT request
> Time 1 - PUT 'A' succeeds
> Time 3 - Client B issues PUT request
> Time 4 - PUT 'B' succeeds
> Time 5 - The response to the first PUT is generated, reflecting the
> current content which is that of the second PUT, which is not at all
> what the client expects.
>
> Or to make a more interesting example
>
> Time 0 - Client A issues PUT request
> Time 1 - Put succeeds
> Time 3 - Client B issues DELETE request
> Time 4 - DELETE succeeds
> Time 5 - The response to the first PUT is generated, but the resource
> is now gone...
>
> The other approach is using 204/201 with a cache validator. But this do
> not allow the client access to the resulting resource after it's
> request when this race occurs.
>
> Time 0 - Client A issues PATCH request
> Time 1 - PUT A succeeds, etag "1" returned.
> Time 2 - Client B issues PATCH request
> Time 3 - PUT B succeeds, etag "2" returned.
> Time 4 - Client A issues conditional GET using etag "1", which fails
> it's condition as the resource has been changed. It's now impossible
> for Client A to get the results of it's request to verify how the
> modifications was applied.
>
>
>
> So the need for 209 would be to explicitly disallow this family of race
> conditions, enabling the client access to the modified resource just
> after the completetion of it's own request before any other
> modification.
>
> If 209 can not guarantee this then it's no better than using a
> subsequent GET to fetch the modified resource if needed by the client.
>
>
> Regards
> Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 06 Sep 2007 00:45:38 +0200

On ons, 2007-09-05 at 14:09 -0700, Yaron Goland wrote:
> What's the use case for wanting to resolve this race condition? If
> someone issues a change to a resource and someone else issues a
> subsequent change to alter the resource's state again then who cares
> what the interim state was since it is now gone?

From what I understand it's the motivation for the 209 response code in
PATCH, allowing the client to request the patched resource so that it
can verify that the patch was applied like expected.

If the client can not trust that the 209 response to PATCH is infact the
resource after applying the patch then there is not much point in
supporting 209 for PATCH, and it can just as well be specified that the
client should GET the resource if it needs to verify..

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Thu, 06 Sep 2007 09:50:24 +0200

> client should GET the resource if it needs to verify..

Right.

The only reason for 209 IMHO is to avoid the potential race condition. 
If that's not guaranteed, there's no point in having it in the first place.

Best regards, Julian

---

From: Yaron Goland <[email-masked]>
Date: Thu, 6 Sep 2007 14:47:07 -0700

As explained in http://lists.w3.org/Archives/Public/ietf-http-wg/2007JulSep/0398.html we in Web3S land definitely have a need for 209 without the race condition guarantee.

I'm also unclear on a real world scenario where a client would apply the PATCH and then as a matter of necessity need to do an immediate GET that is guaranteed to return exactly the state of the resource as it existed after the PATCH. That sounds more to me like versioning than a normal HTTP method.

So my comments would then be twofold:

#1 - I don't believe that 209 with race condition protection is actually necessary. Although I could easily be convinced that I'm wrong if someone has a killer use case.

#2 - We absolutely need the ability to return a 209 without race condition guarantees. So if 209 is going to be defined with a race condition prevention requirement then we can't use 209 and will need to do something else.

So at this point the question for the group goes something like: We in Web3S land need a 209 equivalent without race condition protection. So should we:

        A) If someone comes up with a killer use case for why 209 needs race condition protection then we can come up with a switch that indicates if the 209 had race condition protection or not?

        B) We should declare the race condition protection use case and the non-race condition protection use case as mutually exclusive and introduce different solutions?

Personally I'm less scared of introducing new HTTP headers than I am of introducing new HTTP response codes (it's a scarcity issue). And we will need to ship something very soon. So my guess is that in the medium term we are going to introduce our own request header and our own response header to be used with 200 to handle our immediate needs. If the PATCH spec ends up introducing a version of 209 that meets our requirements then we'll switch over to that.

        Just my two cents,

                Yaron

> From: Julian Reschke [mailto:[email-masked]]
> Sent: Thursday, September 06, 2007 12:50 AM
> To: Henrik Nordstrom
> Cc: Yaron Goland; James M Snell; HTTP Working Group

---

From: Julian Reschke <[email-masked]>
Date: Fri, 07 Sep 2007 09:04:05 +0200

> In the Web3S case we are explicitly not trying to resolve this race condition. Our interest in 209 is strictly as an optimization. We often find ourselves in a situation where we want to issue a PU…

Introducing a new status code (and new headers to select server 
behavior) just for a performance optimization IMHO isn't a good idea. 
What's clear here is that we have to separate use cases, and 209 as 
proposed can't do both.

With respec to:
	
	"...and because if the PUT/POST failed now the client has to receive 
and the server has to send a completely useless GET."

That can be fixed by using a conditional GET, right? Such as:

PUT /resource HTTP/1.1
If-Match: "123"
...

GET /resource HTTP/1.1
If-None-Match: "123"
...

With the conditional GET, if the PUT failed the client will only get a 
304 Not Modified, so won't have to read the new content (or close the 
connection).

Best regards, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 7 Sep 2007 12:14:51 +0100

Yaron Goland wrote:
> In the Web3S case we are explicitly not trying to resolve this race
> condition. Our interest in 209 is strictly as an optimization. We
> often find ourselves in a situation where we want to issue a
> PUT/POST and we need to get the state back. Issuing a PUT/POST
> followed by a pipelined GET is expensive both because pipelining
> isn't well supported and because if the PUT/POST failed now the
> client has to receive and the server has to send a completely
> useless GET. So having the ability to ask for the GET response in
> the PUT/POST response is a very useful optimization to us.

Chained conditional requests would be a much more useful general
solution to this kind of problem.

Chained conditional requests are where a request will fail if an
earlier request has failed - or other conditions, such as an earlier
request succeeding in a particular way.

To preserve the connectionless nature of HTTP, it would be appropriate
for the references to earlier requests to use some persistent
client-specific id.  Alternatively, it might be acceptable to make
chained conditional requests a feature of persistent connection state.

This would solve a number of issues with pipelining.

Currently, a general purpose client has to go through complicated
heuristic contortions to decide when it is safe to pipeline requests -
and those heuristics are not completely reliable.

If later pipelined requests were sure to abort, with no side effects,
when an earlier request fails, or is more than a certain size, or is
delayed, or whatever (choose your condition), then there would be more
cases when it's safe for a client to send pipelined requests
speculatively, without fore-knowledge of the behaviour of each request.

(Chained conditional requests can be generalised further, to further
improve performance and predictability, if there is interest in the
topic.)

-- Jamie

---

From: Yaron Goland <[email-masked]>
Date: Fri, 7 Sep 2007 10:11:07 -0700

This assumes that the client has an etag for the resource, in many of our use cases they will not. So the if-none-match approach will often fail.

> Sent: Friday, September 07, 2007 12:04 AM
> To: Yaron Goland
> Cc: Henrik Nordstrom; James M Snell; HTTP Working Group
> condition. Our interest in 209 is strictly as an optimization. We often
> find ourselves in a situation where we want to issue a PUT/POST and we
> need to get the state back. Issuing a PUT/POST followed by a pipelined
> GET is expensive both because pipelining isn't well supported and
> because if the PUT/POST failed now the client has to receive and the
> server has to send a completely useless GET. So having the ability to
> ask for the GET response in the PUT/POST response is a very useful
> optimization to us.
>

---

From: James M Snell <[email-masked]>
Date: Fri, 07 Sep 2007 10:22:36 -0700

Yaron Goland wrote:
> [snip]

It's not necessary in every case, which is why the spec does not make it
a MUST.  It should be possible to provide race condition protection
without requiring it.

> #2 - We absolutely need the ability to return a 209 without race condition guarantees. So if 209 is going to be defined with a race condition prevention requirement then we can't use 209 and will n…

Well, I'm wondering if there is some way we can signal in the 209
response whether or not some other change has occurred.  I know warning
codes are not the Cool Thing, but something along those lines would be
helpful to indicate that the resource has been modified before the 209
response was returned.

Do you have a suggestion for what that signal could be? response header?
warning code? something else?

- James

---

From: Yaron Goland <[email-masked]>
Date: Fri, 7 Sep 2007 10:26:27 -0700

Warning codes, in the wild, have, in my experience, failed. Most HTTP clients don't have a way to view them and almost no HTTP client programmers appear to know they exist so I wouldn't want to use a warning code.

My current plan is to just return a header but the semantics of the header are "This is a GET response". What I think you want is a different kind of header that means "there may have been changes to the state of the resource between the time your method was applied and this in-line get response was returned."

BTW, I must admit that I still haven't seen any compelling use case for 209 as it is currently defined (e.g. with race condition protection). Just because something can be done doesn't mean it should be done. Before we put it in a standard we should make sure there is a compelling need for it.

        Just my two cents,

                        Yaron

> Sent: Friday, September 07, 2007 10:23 AM
> To: Yaron Goland
> Cc: Julian Reschke; Henrik Nordstrom; HTTP Working Group
> > #1 - I don't believe that 209 with race condition protection is
> actually necessary. Although I could easily be convinced that I'm wrong
> if someone has a killer use case.
> >
>
> > #2 - We absolutely need the ability to return a 209 without race
> condition guarantees. So if 209 is going to be defined with a race
> condition prevention requirement then we can't use 209 and will need to
> do something else.
> >
>

---

From: James M Snell <[email-masked]>
Date: Fri, 07 Sep 2007 10:34:35 -0700

> BTW, I must admit that I still haven't seen any compelling use case for 209 as it is currently defined (e.g. with race condition protection). Just because something can be done doesn't mean it shou…

Oh, absolutely.  I'm not in any particular rush to standarding 209 or
PATCH.  We need to get some implementation experience with this stuff
before we get that far.

The only reason to define 209 with race condition protection is to make
it possible for a client to verify that their changes were correctly
applied.  If that's not something we need to worry about in general,
then you're right, the race condition protections are unnecessary and
can be dropped.  Dropping it would not stop server implementations that
want to provide such protections from doing so.

So... how about this... we can just say that 209 is the equivalent to a
GET response issued immediately after the successful completion of the
request without saying anything at all about the serialization of
subsequent requests.  Does that work?

- James

---

From: Travis Snoozy <[email-masked]>
Date: Fri, 7 Sep 2007 10:39:35 -0700

On Fri, 7 Sep 2007 12:14:51 +0100, Jamie Lokier <[email-masked]>
wrote:

<snip>
> speculatively, without fore-knowledge of the behaviour of each
<snip>

Section 9.3 outlines when requests become a conditional GET:

The semantics of the GET method change to a "conditional GET" if the
request message includes an If-Modified-Since, If-Unmodified-Since,
If-Match, If-None-Match, or If-Range header field.

As you can see, it's not very extensible. Without a version bump, or
some additional detection mechanism, adding new and more powerful If-*
options doesn't solve anything -- the client still has to be able to
detect whether the server actually supports pipelining, whether it
implements the semantics of the new conditions, and if it supports them
correctly. It's entirely possible for a server with crummy pipelining
to implement new If-* header support either with buginess matching
their pipelining implementation, or (depending on the If-*) correctly
for non-pipelined requests, but without changing the pipelining code to
do the "right thing" in chained scenarios (out-of-order responses,
anyone?).

-- 
Travis

---

From: Yaron Goland <[email-masked]>
Date: Fri, 7 Sep 2007 10:38:39 -0700

That works for me.

________________________________________
From: James M Snell [[email-masked]]
Sent: Friday, September 07, 2007 10:34 AM
To: Yaron Goland
Cc: Julian Reschke; Henrik Nordstrom; HTTP Working Group
Subject: Re: [Fwd: I-D ACTION:draft-dusseault-http-patch-09.txt]

- James

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 7 Sep 2007 19:35:48 +0100

Travis Snoozy wrote:
> options doesn't solve anything

The existing If-* options don't matter to this discussion.  Some
servers ignore them all anyway.

> -- the client still has to be able to
> detect whether the server actually supports pipelining,

Unfortunately there isn't anyway to do this reliably, because of
pipelining bugs in the wild.

> whether it implements the semantics of the new conditions, and if it
> supports them correctly. It's entirely possible for a server with
> crummy pipelining to implement new If-* header support either with
> buginess matching their pipelining implementation, or (depending on
> the If-*) correctly for non-pipelined requests, but without changing
> the pipelining code to do the "right thing" in chained scenarios
> (out-of-order responses, anyone?).

Out of order responses are an improvment, but they are just another
level of hack.  Large reponses still block small responses, sometimes
for a very long time.  When considering out of order responses for
HTTP, maybe alternative multiplex/transport mechanisms like BEEP or
SCTP are worth looking at.

I agree totally with your point, that _merely_ adding new headers
isn't enough to make reliable pipelining of any kind from general
clients to existing servers in the wild.

That's no reason to assume it's impossible to do.  Only that it would
have to be more complicated than simply adding a header, if it's
possible at all.

But first, before thinking about nitty gritty details, there would
need to be a higher level interest in the ideas of chained
conditionals, out of order responses, multiplexing, streaming,
predicated caching and leasing, or whatever people are interested in...

If we have the high level interest, we can look at ways to implement
it them on the wire and try to find something which is backward
compatible, as a real world version even if it has to be ugly, and an
ideal version, perhaps to be deployed in 2022 by phasing out the
hacks.

-- Jamie

---

From: Travis Snoozy <[email-masked]>
Date: Fri, 7 Sep 2007 12:19:09 -0700

On Fri, 7 Sep 2007 19:35:48 +0100, Jamie Lokier <[email-masked]>
wrote:
<snip>
> level of hack.  
<snip>

I prefer to call OOO responses bugs. The implication was not supposed to
be "hey, I chained these, so you can build a dep-tree and process OOO,"
it was supposed to be "Hey, I chained these, and the stupid server went
and processed them in the wrong order, totally ignoring the deps, and
it screwed everything up" -- like servers in the wild -already- do.

> clients to existing servers in the wild.

Indeed --

<snip>

Part of the complete, broader solution would have to involve fixing the
whole RFC so it's not only readable, but actually implementable (it's
neither now). I argue that a certification test suite (at bare minimum)
and a free-to-steal reference implementation are critical as well, to
make any such rewrite successful. There's absolutely no good QA process
for HTTP clients/servers right now, and the "robustness principle" has
given license for all kinds of trash implementations.

> ideal version, perhaps to be deployed in 2022 by phasing out the

Variations and backward-compatible hacks are part of what's turned HTTP
into the crufty monster that it is today. I really don't like the
notion of layering more hacks upon cruft upon incompatibilities upon
heuristics. Yeah, we can make it fit if we bring out a hack saw and
make the round hole square -- but adding more complexity just means
there are more things to misunderstand, mis-implement, and otherwise go
wrong.

-- 
Travis

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 07 Sep 2007 21:44:54 +0200

On fre, 2007-09-07 at 10:11 -0700, Yaron Goland wrote:
> This assumes that the client has an etag for the resource, in many of
> our use cases they will not. So the if-none-match approach will often
> fail.

THe client will have an ETag for the resource IF the server is capable
of providing one, and the client is interested in binding the requests
together.

If the client don't care, then there is no point of 209 either.. (just
have the client do a plain GET after the modification, pretty much the
same thing).

If the server don't care then consider fixing the server. Providing an
ETag for the resource is crucial in order to allow clients to build
meaningful sequences conditional requests.

PUT If-Match, only update if there hasn't been any changes. Returns the
new ETag.

GET If-Match, only get if there hasn't been any changes.

GET If-None-Match, only get if there has been changes.

Without ETag these sequences gets much harder, and ETag is already
there, make use of it.

Don't be scared off by the recent discussions about the meaning of ETag,
those discussions is only related to a couple cornercases where the
specifications is a little vague, and also WebDAV which do not fit into
the HTTP object model very well which makes things quite a bit confusing
when trying to map it to the HTTP object model..

And regarding pipelining, there is certain aspects of pipelining that
one has to accept for the lifetime of HTTP/1.x:

 - Don't pipeline non-idempotent requests unless you know exactly what
you are doing. It's very hard to know how far a pipeline got processed
in case of error.

 - Don't pipeline non-idempotent sequences of requests (i.e. a GET
followed by a DELETE of the same resource). There is no guarantee the
requests will be executed in the exact same order. Reordering of
requests may occur in hops along the HTTP forwarding path, or even in
the requested server if it's a multi-threaded server.

 - Don't pipeline when average response latency is important. A large
response will block the responses to the following requests..

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 07 Sep 2007 22:27:51 +0200

> servers ignore them all anyway.

Not sure how that's relevant here.. Capabilities will never be better
than both endpoints no matter how much the protocol is extended.

> pipelining bugs in the wild.

Yes, just as there will be plenty of bugs in the implementation of 209..

> Out of order responses are an improvment, but they are just another

And a bad one..

> Large reponses still block small responses, sometimes

Yes. SCTP looks quite interesting as a transport for HTTP.

It's also worth noting that when using SCTP the whole concept of
pipelining and persistent connections pretty much goes away for free,
and all you need to continue worring about is non-idempotence and
request reordering..

> clients to existing servers in the wild.

The only reliable method for that is to actively work on getting the
server bugs fixed and broken equipment phased out, or to abandon
pipelining and switch to SCTP or other transports more suited for the
purpose.

There is a number of technical problems with HTTP pipelining, nearly all
from the mixing of transport and protocol where.

> predicated caching and leasing, or whatever people are interested in...

chained conditionals using predicted conditions without using ETag &
If-(None-)Match etc will be very hard to make reliable within HTTP/1.x. 

The existing conditionals work pretty well, but requires responses to
any modifications to reach the client before it can continue the chain..

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 07 Sep 2007 22:36:50 +0200

> there are more things to misunderstand, mis-implement, and otherwise go

Fully agreed.

Trying to specify a protocol while keeping 100% compatibility with any
broken existing implementation just introduces even more breakage.

And no, I do not accept pipelining labeled as unusable only because
there is severs out there doing the wrong thing. With that attitude even
GET is unusable.

But pipelinig is certainly not without problems, some of which
practically needs to have the transport changed before they can be
solved.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 08 Sep 2007 18:59:51 +1200

> This assumes that the client has an etag for the resource, in many of our use cases they will not. So the if-none-match approach will often fail.

I don't know how any sane UA could even begin to contemplate a PATCH 
command if they didn't have an E-Tag for what was being patched.

My own 2c on trying to avoid race conditions, patching et al.

It sounds to me like trying to solve an application problem with the 
protocol. I think the problem if it even deserves to be solved should be 
done in the heuristics of the UA/Server rather than trying to deal with 
it in the protocol where I seriously doubt it can be resolved anyway.

If you have many UAs patching the same entity rapidly (as in some blog 
software which re-writes files), then the concept of getting back the 
patched version you patched is somewhat sketchy.

when you have multiple users trying to submit their copy of what they 
think they have...  Personally I think it's a recipe for disaster, since 
the PATCH command won't work since the entity will have changed before 
the patch gets there, so that patch will fail, and the UA will have to 
cope with that.  In the patch case it's first in wins.  Everyone after 
that has the wrong E-Tag.  "Normal" or "Expected" case however is last 
in wins, i.e if you write a UA and you want your user to "win" the patch 
race, then you submit the whole resource, not a patch for it, because 
that's the only way you can guarantee that the server will receive a 
coherent entity from your command that it can put into its history log 
when the next mod comes in.

If you have Last in wins, then you get a history on the server of all 
mods, and each is coherent in itself.  If you have first in wins, you 
only get the first modification, and no history to roll back if you need 
to resolve conflicts, plus a bunch of UAs that have p*ssed off users, 
since their change couldn't go in, and the UA isn't smart enough to:

a) get a fresh copy
b) try and re-apply the "changes" that the user made to the last copy in 
the context of the new one (flawed concept).
c) try and re-submit it.

even if that would be a "smart" thing to do.  So the user loses their work.

There's a reason why version control software uses last-in-wins 
(sometimes with merging for conflicts), you can roll back to the version 
you want then.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 9 Sep 2007 12:55:44 +0100

Adrien de Croy wrote:
> [...] the PATCH command won't work since the entity will have changed before 
> that has the wrong E-Tag.

It depends on the type of PATCH.  For example, an "append" operation -
or more complicatedly "append this paragraph to my simple blog file".
Both of those can be forms of PATCH which succeed if the original
entity is changed meanwhile.

Perhaps not coincidentally, those forms don't always need an Etag in
the request, either.

> even if that would be a "smart" thing to do.  So the user loses their work.

In my experience, that last-in-wins-with-history results in lost work,
because it succeeds for all users but most of them aren't notified
that their work is dumped _only_ in the history file (assuming you
have one).  They send in their work, get a successful response, assume
they don't need to do anything more, and move on to something else.
They think their work is finished with, but it's effectively lost
because nobody is ensuring it reaches the place that matters.  It's
recoverable, but that is not enough.

> (sometimes with merging for conflicts), you can roll back to the version 

In my experience version control software uses first-in-wins and
everyone else has to resync and fix conflicts locally before they can
submit their changes to a shared server.  CVS and SVN work this way,
for example.  What VC software are you thinking of?

-- Jamie

---

From: Travis Snoozy <[email-masked]>
Date: Sun, 9 Sep 2007 05:37:49 -0700

On Sun, 9 Sep 2007 12:55:44 +0100, Jamie Lokier <[email-masked]>
wrote:

<snip>
<snip>

Preventing unsynchronized checkins is a defining part of source control
systems -- it's one of the primary reasons we have them. Allowing Bob
to check in V3 of a file that fixes bug 42, overtop Alice's V2 checkin
that fixed bug 7 (where they both made their changes starting from V1),
would cause mass confusion, potential loss of work (if the lack of a
merge goes unnoticed) and extra time and energy spent trying to get
everything merged later on (after possibly more changes came in). Bob
must make sure that his changes apply to the repository's HEAD before
he has any business trying to commit those changes, and the system
enforces this.

That said, a "check out, modify, check in" approach may be more
user-friendly than a "sync, merge, check in" approach -- the former
results in access-denied (can't get the lock), so it limits the rate at
which changes can come in, but the latter has the chance to cause
headache-inducing merge conflicts, which even developers hate having to
deal with.

--
Travis

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 10 Sep 2007 00:41:22 +1200

> entity is changed meanwhile.
Why wouldn't you just use a form POST for such an operation?

> recoverable, but that is not enough.
Fair enough.  Just still trying to think of what a poor UA / user would 
do once they've been told their patch won't be accepted because the 
thing they are patching has been changed out from underneath.

> for example.  What VC software are you thinking of?
er.... VSS, with multiple-checkouts enabled

%-P

Adrien

---

From: Julian Reschke <[email-masked]>
Date: Sun, 09 Sep 2007 16:20:15 +0200

> Why wouldn't you just use a form POST for such an operation?

I think that's the wrong question to ask here.... Why would one, if 
PATCH is the more appropriate method for it?

Best regards, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 9 Sep 2007 16:28:16 +0100

> headache-inducing merge conflicts, which even developers hate having to

"check out, modify, check in" is more about locking than
unsynchronised patching.  If PATCH requests were done with locks, we
wouldn't be having this conversation :-)

Both the above methods are more user friendly for some things.

Most groups of software developers use the "sync, merge, check in"
method nowadays, but some use the locking method.  They _used_ to use
the locking method more, but the CVS/SVN manuals have a nice section
which explains how this tends to be a bottleneck more than it helps.

Merge conflicts can be hard, but the locking method can force people
to work outside the version control - by doing an unlocked checkout
(because someone else has the file locked for a while), working, then
waiting until they are able to sync manually, or submitting patches to
the main auther.  Either way, merging conflicts occur.  It's better to
have tools which support that, than doing it manually and prone to
more mistakes.

Analagous things happen at a protocol level with PATCH.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 9 Sep 2007 16:32:14 +0100

> thing they are patching has been changed out from underneath.

Presumably the alternative is that their patch is accepted and
_another_ poor user's patch is "not accepted" by being reverted.

Surely it's better to notify any user immediately that their patch
isn't accepted, than to revert another user's patch silently.  In both
cases, someone's patch isn't accepted, but in the latter case, it's
just dropped silently.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 09 Sep 2007 19:46:02 +0200

> thing they are patching has been changed out from underneath.

Then LOCK, GET, [edit], PATCH, UNLOCK, or depending on the use case

GET, LOCK with If-Match, [edit], PATCH, UNLOCK

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 10 Sep 2007 11:03:59 +1200

right - in other words, the only way to guarantee security on the patch 
is to LOCK, in which case why even bother trying to solve race 
conditions around the PATCH command.  The race condition if anywhere is 
around who gets the LOCK first...:)

---

From: "Mark Baker" <[email-masked]>
Date: Sun, 9 Sep 2007 19:08:53 -0400

Why not PATCH with If-Match?

-- 
Mark Baker.  Ottawa, Ontario, CANADA.         http://www.markbaker.ca
Coactus; Web-inspired integration strategies  http://www.coactus.com

---

From: Travis Snoozy <[email-masked]>
Date: Sun, 9 Sep 2007 16:39:33 -0700

On Mon, 10 Sep 2007 11:03:59 +1200, Adrien de Croy <[email-masked]>
wrote:

> 
> right - in other words, the only way to guarantee security on the
> patch is to LOCK, in which case why even bother trying to solve race 
> conditions around the PATCH command.  The race condition if anywhere
> is around who gets the LOCK first...:)
> 

Careful, though -- you still have to deal with timing locks out, and
dealing with the situation where somebody went LOCK, GET, edit,
timeout (implied UNLOCK), PATCH -- which leads to the same situation of
"Hey, buster -- you lose your work because you didn't get back to us in
a timely manner."

-- 
Travis

---

From: Travis Snoozy <[email-masked]>
Date: Sun, 9 Sep 2007 17:55:12 -0700

> which explains how this tends to be a bottleneck more than it helps.

Which is irrelevant in the general case; it's a heuristic that works in
many cases, but assumes some degree of side-channel communication and
coordination which we cannot assume on the web. This is especially
true if we want to support autonomous or semi-autonomous systems being
able to patch things -- we need to be able to signal intent and
priority for submitting a patch, and a checkout does just that.

> Analagous things happen at a protocol level with PATCH.

Yes, you can always just check out, dump your file, check in. But the
point is that you don't get into the situation where Alice and Bob are
both editing a file simultaneously, and don't find out about this until
checkin. You don't have some mystery "what happens now?" coming when
someone who has checked out a file goes to check in, and clients who
start modifying a checked-out file have advance warning that there
could be merge conflicts. Does that mean clients should jettison merge
support? Not at all. In fact, a client could accept a PATCH back from
the server when acquiring the lock, to update the client view and aid
with merging. But the point is, there needs to be some cross-client
awareness to allow for the coordination that groups like developers
have naturally over the course of working on code with each other.

Checkouts let you make intelligent risk decisions up-front. That is
extremely desirable for conserving bandwidth and branwidth, especially
in cases where there is a lack of communication and coordination
between editors.

-- 
Travis

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 10 Sep 2007 03:36:17 +0200

> around who gets the LOCK first...:)

Why?

For very many applications "fail gracefully on concurrent edit" is the
preferred mode, allowing the application to fall back on merge and
resubmission of the update. On a larger scale this is also less error
prone unless there is lots of interest in authoring the exact same
resource.. There is generally less loss in productivity from the
overhead of having to deal with a merge when a concurrent edit should
happen than having to fight with stale locks which is far more common.

And even with locks the application should be capable of handle merges
in case the lock has timed out or been overridden, at least if the lock
is held over any longer period of time (minutes or more..) which is
generally the case when there is humans involved..

Sequence "fail gracefully on concurrent edit" is simply

GET

[edit]

PATCH with If-Match.

[if error, perform a merge of the changes]

The LOCK sequence is only needed if the application wants to try
blocking others from even trying to author the same resource at the same
time.

Regards
Henrik