Subject: Etag-on-write, draft -04

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Wed, 29 Nov 2006 16:25:21 -0800

Julien, Lisa, and I have been having a discussion about this  
offline, which should be recorded here.  Here is a glommed-together  
version of the discussion so far.

	-wsv

On Nov 28, 2006, at 4:09 PM, Lisa Dusseault wrote:

> Wilfredo, have you thought about the backwards-compatibility issue  
> with introducing a new header in order to ensure the same behavior  
> that's common now and that existing clients depend on?  I think that  
> issue might hit your client the worst, as well as impacting Xythos  
> WFC, but I don't know exactly how the Mac client works.
>
> I mentioned this on the HTTP list Sept 6 and 12, and Jamie Lokier  
> wrote a far better explanation of the tradeoffs on Sept 12,  
> explaining how the tradeoffs depend on how deployed clients behave  
> as well as deployed servers.   Roy chimed in suggesting a solution  
> that manages the case Jamie outlined where we try to be compatible  
> with deployed clients.  Jamie replied (Sept 14) with a slight  
> variation on that solution that shows how easy it would be for  
> servers to implement.  I believe Roy's solution to be the best I've  
> seen though it requires a little elaboration about what "can use in  
> in future conditional requests" means.
>
> http://lists.w3.org/Archives/Public/ietf-http-wg/2006JulSep/

On Nov 28, 2006, at 4:27 PM, Julian Reschke wrote:

> The Xythos client always assumes that content isn't rewritten. And  
> if no ETag is returned, it uses the Last-Modified date as cache key.  
> So it's already broken with respect to servers that have to rewrite.
>
> Roy never made a proposal, and didn't answer when asked for  
> clarification/confirmation.
>
> Assuming he meant that the server is supposed to make sure that  
> strong etags work as (IMHO broken) clients expect: that wouldn't  
> cover the case where the server *has* to rewrite content, such as  
> when the backing store doesn't provide the capability of storing  
> binary content (think XML or ICS). I think we all agree that in  
> cases like these we still want the ability to do an PUT/If-Match  
> request, and that's what the extension header in my draft provides  
> (the client gets the etag upon PUT but with a warning attached, and  
> then it's the client who can decide whether it wants to refetch the  
> content or not).

On Nov 28, 2006, at 5:10 PM, Lisa Dusseault wrote:

> Yes, we've been through this before -- a server that has to rewrite  
> content, so it issues a "not-valid-etag" in the ETag header in the  
> response to the PUT request, then issues a *second* ETag for the  
> rewritten content (this could go in a separate new header for  
> optimization with future clients) would work perfectly with Xythos  
> WFC.
>
> In other words, even for a client that assumes that servers do not  
> rewrite, like Xythos WFC, there's an easily-implementable way for  
> servers to that do rewrite to have backwards compatibility.
>
> [Re: Roy's proposal]
>
> Well, Jamie elaborated on what I describe as Roy's proposal.  I may  
> have been reading more into Roy's email than you did, but apparently  
> Jamie understood it the same way I did.
>
> But your proposal only works with upgraded clients.  Roy's proposal,  
> or Jamie's or my understanding of it, works with existing deployed  
> clients as well because it mimics the behavior of a change-by-a- 
> client even though it's a change-by-a-server-after-PUT.

On Nov 29, 2006, at 1:10 AM, Julian Reschke wrote:

> I'm not sure I follow.
>
> So are you proposing to return a dummy etag (that will always fail  
> If-Match tests) upon PUT, in order to fix clients that make an  
> assumption not licensed by RFC2616? It seems to me that it would be  
> easier to fix the client (which is in active development), instead  
> of having to change potentially many servers.
>
> [Re: easily-implementable wayto have backwards compatibility]
>
> Well, if you really think this is a good idea, please go ahead and  
> write down a specific proposal, like I did :-).
>
> [Re: Roy/Jamie proposal]
>
> 1) I think we do agree that there is an interoperability problem  
> that needs to be solved. There are clients out there not doing the  
> right thing, because the server behaves in a way they don't expect.
>
> 2) As far as I can tell, the behavior of these servers is legal  
> according to RFC2616. This is explained in <http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-04.html#rfc.section.1.3 
> > -- in the last months I haven't seen not a single comment stating  
> that this analysis is incorrect. If you think it's incorrect, please  
> be more specific.
>
> 3) We can't change deployed code, and we can't make incompatible  
> changes to RFC2616.
>
> Thus, I think the best thing we can do is to clarify the situation,  
> and add code that make things easier for future clients. And that's  
> exactly what my document does.

On Nov 29, 2006, at 10:58 AM, Wilfredo Sánchez Vega wrote:

>  [Re: original question]
>
>   When that behavior is known the be broken, I guess I don't really  
> sympathize with the clients in question.  A DAV server is not  
> required to store content octet-by-octet.  Nothing says that they  
> should, and nothing the server does says that it did.
>
>   By "your client", I assume that you mean WebDAV FS.  (We have  
> others.)  WebDAV FS does in fact depend on octet-by-octet storage,  
> as far as I can tell (I don't work on it, per se).  But WebDAV FS  
> expect WebDAV to behave like a file system.  A WebDAV server that  
> does not behave like a file system will not work with WebDAV FS.  I  
> do not take this to mean that all servers should behave like a file  
> system, but rather that they should if that's how they want to be  
> used.
>
>   A server that does content modification on PUT will make WebDAV FS  
> unhappy no matter what the headers say.  So don't use WebDAV FS as a  
> client on those servers.  It's not appropriate.  But right now,  
> WebDAV FS has no way to distinguish between a server that does  
> content modification and one that does not.  This header allows a  
> client to do so, and you can then expect that WebDAV FS should, if  
> the authors care to, use that information to do something more  
> useful when it detects content modification.  Perhaps it can do a  
> reload.  Perhaps it can tell the user that it's using a server that  
> isn't supported.  I don't know, but I don't really care that much,  
> because the servers that behave like a file system will continue to  
> do so and will continue to work just fine.
>
>   Let's not assume that those are the only kinds of servers that we  
> need in the world, and accept that not all DAV clients are  
> appropriate for all DAV servers.  It's already the case today, and I  
> don't see this changing, or even as a problem.
>
>   CalDAV clients aren't going to find vanilla DAV servers all that  
> exciting.  And I certainly don't expect that the iCal server I'm  
> writing needs to support authoring via WebDAV FS in a calendar  
> collection.  It's a ridiculous use case.
>
>   ETag behavior on WebDAV FS is already broken in the most common  
> case, by the way.  I frequently have a document open on a DAV  
> volume, and a second after I save it, my editor thinks the document  
> changed (because Apache changed the ETag that it never should have  
> sent on PUT in the first place).  Thus, the most common case (Apache  
> server) is already broken today, partly for reasons (the weak- 
> >strong change) that are not addressed in this draft at all, because  
> they are unrelated, and arguably an Apache implementation problem.
>
>   My point being that as a server implementor, I have some clients  
> doing one thing, and other clients doing another.  Changing behavior  
> annoys some and make others happy.  Meh.  Give me a standard so that  
> at least new clients can behave themselves sanely.

On Nov 29, 2006, at 1:38 PM, Lisa Dusseault wrote:

>  [Re: no sympathy for broken clients]
>
> That's arguably true (OTOH arguably an unstated assumption that was  
> reasonable for those clients to make), but maybe we can do something  
> sensible with those clients anyway rather than argue what was meant  
> by an old spec and assumed by its authors and readers.
>
>
>>  A server that does content modification on PUT will make WebDAV FS  
>> unhappy no matter what the headers say.
>
> I'm just not understanding everything, putting this statement  
> together with the rest of your email.  In this statement, you imply  
> that even if that server treats content modification on or after a  
> PUT exactly the same as it would a content modification by a client  
> after the PUT -- by issuing one ETag in the response to the PUT, and  
> another ETag immediately afterward.
>
> Then later on in your email you explain how Apache issuing first a  
> weak ETag, then a strong one, makes WebDAV FS think that the  
> document has changed.  This sounds perfect for what has been  
> suggested -- that if a content-modifying server issued a strong ETag  
> and then another strong ETag, WebDAV FS would think the document has  
> been changed, which it has.
>
> I don't understand why there's a problem with this approach.  It  
> works well with deployed clients.  It's perfectly compatible with  
> complementary approaches that also advertise that the server does  
> content modification and that modification has been done in this case.

On Nov 29, 2006, at 1:58 PM, Julian Reschke wrote:

>> That's arguably true (OTOH arguably an unstated assumption that was  
>> reasonable for those clients to make),
>
> For the record: I do not think it was a reasonable assumption.
>
>> but maybe we can do something sensible with those clients anyway  
>> rather than argue what was meant by an old spec and assumed by its  
>> authors and readers.
>
> Hm, so you *do* want to make an incompatible change to RFC2616?
>
>>>   A server that does content modification on PUT will make WebDAV  
>>> FS unhappy no matter what the headers say.
>>
>> I'm just not understanding everything, putting this statement  
>> together with the rest of your email.  In this statement, you imply  
>> that even if that server treats content modification on or after a  
>> PUT exactly the same as it would a content modification by a client  
>> after the PUT -- by issuing one ETag in the response to the PUT,  
>> and another ETag immediately afterward.
>
> ...I think there's something missing in this sentence. If it does  
> this that, what then?
>
>> Then later on in your email you explain how Apache issuing first a  
>> weak ETag, then a strong one, makes WebDAV FS think that the  
>> document has changed.  This sounds perfect for what has been  
>> suggested -- that if a content-modifying server issued a strong  
>> ETag and then another strong ETag, WebDAV FS would think the  
>> document has been changed, which it has.
>
> But that behaviour breaks all clients that indeed would work well  
> with "harmless" content rewriting, such as when ICS entities are  
> normalized. Those clients could go on editing the resource, because  
> the transformation by the server obeys the rule:
>
> 	t(e2(e1(c))) = t(e2(t(e1(c))))
>
> (<http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-04.html#rfc.section.1.2.p.11 
> >)
>
>> I don't understand why there's a problem with this approach.  It  
>> works well with deployed clients.  It's perfectly compatible with  
>> complementary approaches that also advertise that the server does  
>> content modification and that modification has been done in this  
>> case.
>
> It breaks existing clients that do not care about servers rewriting  
> the content the same way every time the resource is PUT. This is  
> *only* a problem when the client tries to do a GET/range request. I  
> think neither Xythos WFC and Apples' WebDAV client fall into that  
> category, though.

On Nov 29, 2006, at 3:13 PM, Wilfredo Sánchez Vega wrote:

>> That's arguably true (OTOH arguably an unstated assumption that was  
>> reasonable for those clients to make),
>
>   I will strongly disagree that this was a reasonable assumption to  
> make.  It is well known that a DAV server is allowed to rewrite  
> content, and nothing in any spec that I'm aware of says otherwise.
>
>>>   A server that does content modification on PUT will make WebDAV  
>>> FS unhappy no matter what the headers say.
>>
>> I'm just not understanding everything, putting this statement  
>> together with the rest of your email.  In this statement, you imply  
>> that even if that server treats content modification on or after a  
>> PUT exactly the same as it would a content modification by a client  
>> after the PUT -- by issuing one ETag in the response to the PUT,  
>> and another ETag immediately afterward.
>
>   What is the point of issuing an ETag on PUT that is not useful for  
> any request that follows?  It's completely misleading because it  
> implies that the client has something that should be cached, when it  
> fact it already has a stale copy.  That's bogus.
>
>> Then later on in your email you explain how Apache issuing first a  
>> weak ETag, then a strong one, makes WebDAV FS think that the  
>> document has changed.  This sounds perfect for what has been  
>> suggested -- that if a content-modifying server issued a strong  
>> ETag and then another strong ETag, WebDAV FS would think the  
>> document has been changed, which it has.
>
>   WebDAV FS isn't the only DAV client in the world, and it's not  
> even a very interesting one.  The correct thing for a server to do  
> with today's WebDAV FS is not return an ETag on PUT, and let it use  
> the last-modified header instead.
>
>   This has the pleasant effect of not breaking clients that don't  
> make the same bad assumptions that WebDAV FS is making.
>
>> I don't understand why there's a problem with this approach.  It  
>> works well with deployed clients.  It's perfectly compatible with  
>> complementary approaches that also advertise that the server does  
>> content modification and that modification has been done in this  
>> case.
>
>   My point with the Apache weak->strong problem is that no, the  
> status quo does not work well today with the most popular web server  
> in use today and with what is probably one of the most widely used  
> DAV authoring clients in use today, and we should stop pretending  
> that it does.

On Nov 29, 2006, at 3:45 PM, Lisa Dusseault wrote:

>>   What is the point of issuing an ETag on PUT that is not useful  
>> for any request that follows?  It's completely misleading because  
>> it implies that the client has something that should be cached,  
>> when it fact it already has a stale copy.  That's bogus.
>
> So what would be the client behavior if the PUT response contained  
> no ETag in the ETag header (and no last-modified header)?  Then the  
> client could not assume that the content had not been modified,  
> which would be correct behavior.  Later if the client does a HEAD,  
> PROPFIND or other, the server could provide the post-modification  
> ETag.
>
> This could be less bogus than the proposal for a "never-valid" ETag  
> that's not useful for any request that follows.
>
> What I'd like to come up with is proposals that allow a new client  
> to work well with existing servers, and a new server to work well  
> with existing clients.  Sometimes there are cheap things you can do  
> to make those cases work even if there have been bad assumptions in  
> the past.
>
> Right now we have three interesting things to flag (no change,  
> harmless change, big change) and our existing tools only allow us to  
> flag two of those things.  The major question is whether "harmless  
> modifications" should be flagged in such a way so that:
> 	1. Deployed clients can't distinguish a harmless modification from  
> the case where no modification occurred; only upgraded clients can  
> tell the difference by using a new header
> 	or 2.  Deployed clients can't distinguish between a harmless  
> modification and a major modification done perhaps by another  
> client; only upgraded clients can tell the difference by using a new  
> header
>
> I believe the proposal in the draft describes case 1, whereas case 2  
> would be safer.
>
> In any case, major server modifications (non-harmless) should never  
> treated in such a way that they appear as if no change occurred to  
> deployed-base clients.  Do we generally agree on that?

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Wed, 29 Nov 2006 16:51:32 -0800

> to make those cases work even if there have been bad assumptions in  

   Sure.  I don't think your solutions are cheap long-term, is all.   
They are convenient for today's clients, but I'd rather do the right  
thing for future clients.  There aren't that many clients today, and I  
don't think Julian's spec makes life practically hard for existing  
clients.

   In the case where a server does not modify content, things continue  
to work as "well" as they work today.  In the case where the server  
does modify content, existing clients are already broken.  I don't  
believe that today's clients really care, because they aren't designed  
to work with such servers in any case.

   I therefore don't think that catering to today's broken clients  
should prevent us from pursuing the best option for tomorrow's  
(hopefully) less-broken clients.

> flag two of those things.

   False.  Existing tools don't allow us to flag *any* of these things  
in the reply to a PUT request.  Some clients are assuming "no change"  
without any real justification, other than the fact that most servers  
out there seem to not change the data.  The only way to really know  
this today is to follow a PUT with a GET, in which case, you arguably  
can identify all three cases.

>   The major question is whether "harmless modifications" should be  
> flagged in such a way so that:

   No, because one of the existing client scenarios is the file system  
clients, and those clients want harmless modifications to be refreshed  
as much as they want major ones.  When I open a document on my Mac, I  
expect to see the document as it exists on the file server.  If the  
file server made "harmless" changes, I expect them to be there.

   For those clients, all servers that modify content appear broken.   
The only way for those clients to reliably work with such a server is  
not to trust the ETag on PUT at all.  If they don't do that today, the  
workarounds you are suggesting don't reliably fix them.

   In any case, I am unaware of any cases where someone mounts a  
content-modifying DAV service as a file system and expects it to  
work.  So the use case you want us to design around seems theoretical.

	-wsv

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 30 Nov 2006 03:35:14 +0100

> > they are unrelated, and arguably an Apache implementation problem.

Can you please elaborate on what the actual problem with the Apache
approach is? To me it smells like this is actually a slightly dumb
client not understanding ETag semantics, and nothing wrong at the
server.

I haven't looked in detail at what Apache does, but I assume the ETag as
such is still the same, just that it's now a strong ETag. If so then
what's the problem? It's still the same ETag indicating that the content
has not changed in semantics, but octet equality is not guaranteed
(well, it pobably is in the Apache case, but can not be counted in by
specs). The temporary weak ETag in this context has the potential of
being a much stronger validator than Last-Modified ever can be.

The only thing I see troublesome with this approach is the silly fact
that the use of the weak etag is not allowed as If-Match condition on
subsequent PUT when saving the next change to the same document. This
forces the client to first do a HEAD to get the strong ETag, make it's
own compare to ensure the content is still semantically equivalent and
then PUT using the strong ETag.. or alternatively ignore this silly
restriction and use the weak ETag in PUT If-Match assuming the server
will swallow it even if tecnically not allowed..

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 30 Nov 2006 14:50:29 -0800

>> if no ETag is returned, it uses the Last-Modified date as cache  
>> key. So it's already broken with respect to servers that have to  
>> rewrite.
>>

I don't like repeating myself over and over again just because you
want to add an unnecessary feature to HTTP.

My opinion hasn't changed -- the extension is completely unnecessary,
RFC 2616 already defines what sending an etag on PUT means,
and every single relevant case can be handled by a simple clarification.

My proposal was:

     require that ETag in a response to PUT means that the client
     can use that entity tag in future conditional requests (that
     includes IMS, If-Match, and Range-based conditional requests).

There is absolutely no reason for the server to send an ETag on a PUT
response if the server does not intend to support that functionality.
I am quite sure that none of the server vendors will care about the
addition of that clarification provided that the requirement is
properly stated as an interface constraint and not an implementation
constraint.

OTOH, the mechanism for how the server "stores" the representation,  
whether
or not it is stored byte-for-byte, and how a server might otherwise  
manage
to accomplish the feat of handling a conditional request without  
preserving
byte-level equality is none of our business and should never appear  
in an
IETF specification regarding HTTP.

My proposal does solve every single case that has been described so far,
including repeatable server-modifications such as normalization, and
works with existing clients.

In regard to Apache mod_dav's use of weak entity tags, that is a  
separate
issue and can be fixed without changing HTTP at all.  It is merely an
implementation quirk having to do with the way mod_dav reuses the
file space as a storage mechanism (and thus a separate handler for GET).
I can very easily turn it off, as can anyone with access to the config
files.  The right solution, though, is to use a property-based back-end
and store the MD5 on write, which would allow the weak designation to be
removed and actually solve the real problem of clients wanting a strong
etag returned from PUT.  Patches are welcome.

....Roy