Subject: Re: Etag-on-write, draft -04

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Thu, 30 Nov 2006 20:03:03 -0800

Well, two things.

   One, the client doesn't really know that the weak ETag is going to  
become a strong ETag later, unless it makes assumptions about Apache's  
behavior.

   Two, I don't think you can assume that "X" and W/"X" are at all  
related to each other.  In Apache's case, there is no such guarantee,  
because the same weak ETag may be issues to different versions of the  
document (if it is edited more than once in the same 1-second window),  
the data you got with an ETag of W/"X" may be completely different  
than the data that later exists with an ETag of "X", or even the same  
W/"X" you got.

   Apache's behavior works more-or-less for caching proxies, but it's  
not really great for authoring clients.  The weak->strong promotion  
itself isn't broken per se, but the same data having a changed ETag is  
a bit of a bother.  That different data exist with the same (weak,  
brief) ETag seems like a violation.

	-wsv

On Nov 29, 2006, at 6:35 PM, Henrik Nordstrom wrote:

> ons 2006-11-29 klockan 16:25 -0800 skrev Wilfredo Sánchez Vega:
>
>>>   ETag behavior on WebDAV FS is already broken in the most common
>>> case, by the way.  I frequently have a document open on a DAV
>>> volume, and a second after I save it, my editor thinks the document
>>> changed (because Apache changed the ETag that it never should have
>>> sent on PUT in the first place).  Thus, the most common case (Apache
>>> server) is already broken today, partly for reasons (the weak-
>>>> strong change) that are not addressed in this draft at all, because
>>> they are unrelated, and arguably an Apache implementation problem.
>
> Can you please elaborate on what the actual problem with the Apache
> approach is? To me it smells like this is actually a slightly dumb
> client not understanding ETag semantics, and nothing wrong at the
> server.
>
> I haven't looked in detail at what Apache does, but I assume the  
> ETag as
> such is still the same, just that it's now a strong ETag. If so then
> what's the problem? It's still the same ETag indicating that the  
> content
> has not changed in semantics, but octet equality is not guaranteed
> (well, it pobably is in the Apache case, but can not be counted in by
> specs). The temporary weak ETag in this context has the potential of
> being a much stronger validator than Last-Modified ever can be.
>
> The only thing I see troublesome with this approach is the silly fact
> that the use of the weak etag is not allowed as If-Match condition on
> subsequent PUT when saving the next change to the same document. This
> forces the client to first do a HEAD to get the strong ETag, make it's
> own compare to ensure the content is still semantically equivalent and
> then PUT using the strong ETag.. or alternatively ignore this silly
> restriction and use the weak ETag in PUT If-Match assuming the server
> will swallow it even if tecnically not allowed..
>
> Regards
> Henrik

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Thu, 30 Nov 2006 20:12:10 -0800

IIRC, the weak ETag generation is in the core code, not mod_dav, so  
it's also there in the case where edits happen outside of DAV  
(possible outside of httpd altogether).

   I agree that the MD5 solution is a good one; it's what the Twisted  
server now does, and that works swell.  But, as would be the case for  
Apache, it works for DAV, but not the generic server, since it doesn't  
have a property store.

   In any case, I agree that this is an implementation, not protocol,  
issue.

	-wsv

On Nov 30, 2006, at 2:50 PM, Roy T. Fielding wrote:

> In regard to Apache mod_dav's use of weak entity tags, that is a  
> separate
> issue and can be fixed without changing HTTP at all.  It is merely an
> implementation quirk having to do with the way mod_dav reuses the
> file space as a storage mechanism (and thus a separate handler for  
> GET).
> I can very easily turn it off, as can anyone with access to the config
> files.  The right solution, though, is to use a property-based back- 
> end
> and store the MD5 on write, which would allow the weak designation  
> to be
> removed and actually solve the real problem of clients wanting a  
> strong
> etag returned from PUT.  Patches are welcome.

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Thu, 30 Nov 2006 21:41:35 -0800

Roy-

> My opinion hasn't changed -- the extension is completely unnecessary,
> RFC 2616 already defines what sending an etag on PUT means,
> and every single relevant case can be handled by a simple  
> clarification.
>
> My proposal was:
>
>     require that ETag in a response to PUT means that the client
>     can use that entity tag in future conditional requests (that
>     includes IMS, If-Match, and Range-based conditional requests).
>
> There is absolutely no reason for the server to send an ETag on a PUT
> response if the server does not intend to support that functionality.
> I am quite sure that none of the server vendors will care about the
> addition of that clarification provided that the requirement is
> properly stated as an interface constraint and not an implementation
> constraint.

   I think Julian's text implies the same thing, but I agree that  
phrasing it terms of "future conditional requests" is probably more  
relevant that phrasing it in terms of "same as a subsequent HEAD  
request".

   Note also that this draft addresses more than PUT.  Since it is  
possible that other methods may alter the ETag, it also clarifies  
that.  I also think the t(e2(e1(c))) = t(e2(t(e1(c)))) is very useful  
for client authors.

> OTOH, the mechanism for how the server "stores" the representation,  
> whether
> or not it is stored byte-for-byte, and how a server might otherwise  
> manage
> to accomplish the feat of handling a conditional request without  
> preserving
> byte-level equality is none of our business and should never appear  
> in an
> IETF specification regarding HTTP.

   I think that in terms of what ETag is meant for, you are totally  
right.

   I also think that there are clients in the world that would like to  
know that if they did a an immediate subsequent GET of the same  
variant, that the data would look different.  Several client authors  
have expressed interest in this information, which is what the Entity- 
Transform is meant to address.

   Example A.1 describes a Subversion server.  Let's say that the  
client isn't the Subversion client, but a text editor, and we have  
auto-versioning enabled on the server.

   My editor saves a document with "$Id$" tags in it.  The server says  
OK and gives me an ETag.

   You are arguing, I think, that my editor has no reason to care that  
the $Id$ tag portion was modified, since subsequent edits will result  
in the same entity on the server, regardless of whether it refreshes  
the modified $Id$ tag in the document, so a clarified ETag should be  
sufficient.

   This is probably true for a lot of editing situations.  In the case  
of normalization of, say, iCalendar data, I think you are right.

   However, in the Subversion example, I think the edit does care,  
because the point of the $Id$ tag is that the user can see what  
version of the document they are looking at when they view the file.   
In this case, knowing that the data was modified will let the editor  
know that a subsequent GET request would be a good idea.  A Subversion  
client wouldn't have to do this, in that it can probably assume that  
it knows what the server changed, and make the same edit itself.  But  
a generic text editor would not.

   I do agree that many client authors are all bent out of shape about  
the data changing when they needn't be, and it's probably best if they  
don't refresh the data unnecessarily, but I do think there are valid  
use cases for having that information.

> My proposal does solve every single case that has been described so  
> far,
> including repeatable server-modifications such as normalization, and
> works with existing clients.

   Is the above use case covered?  I'm not seeing how.

   The ETag this has obviously caused a lot of heartburn.  I'd be  
interested to know whether you think the rest of the draft is good,  
absent the new header.  It's clear that some clarification is needed,  
and I think this is a good start.  No?

	-wsv

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 01 Dec 2006 09:53:59 +0100

tor 2006-11-30 klockan 20:03 -0800 skrev Wilfredo Sánchez Vega:

> related to each other.

HTTP specs do... see RFC2616 13.3.3, The weak comparison function.

> In Apache's case, there is no such guarantee,  

Then Apache is somewhat broken.

Two identical weak etags SHOULD only be used if the objects is
semantically equivalent, but not necessarily octet equivalent. Two
different edits can not be considered semantically equivalent in this
context.  RFC 2616 13.3.4.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 01 Dec 2006 10:14:28 +0100

> HTTP specs do... see RFC2616 13.3.3, The weak comparison function.

You are right. However, the fact that this discussion has been going on 
for almost one year now, and nobody mentioned that before, may indicate 
that RFC2616bis potentially could make that clearer.

> Then Apache is somewhat broken.

Yes, at least when the resource can be authored.

> ...

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 1 Dec 2006 22:00:56 -0800

On Nov 30, 2006, at 9:41 PM, Wilfredo Sánchez Vega wrote:
>   I also think that there are clients in the world that would like  
> to know that if they did a an immediate subsequent GET of the same  
> variant, that the data would look different.  Several client  
> authors have expressed interest in this information, which is what  
> the Entity-Transform is meant to address.

Which, again, is why it is completely unnecessary.  If the entity being
edited had an etag, and the result from performing a PUT did not include
an etag (or included only a weak etag), then the client can conclude  
that
the data would look different.  Whether or not it does happen to be
different when a subsequent GET is made is a distributed systems problem
that won't disappear, regardless of Entity-Transform.  The client should
be assuming a transform has occurred if it does not receive a strong  
etag,
since it has no alternative available to it other than to perform a
subsequent GET.  The Entity-Transform provides no useful information
and may, in fact, be completely wrong if there is an intermediary in
the middle doing transformations of its own -- the spec already
anticipated that for etag, but not for future extension headers.

>   My editor saves a document with "$Id$" tags in it.  The server  
> says OK and gives me an ETag.
>
>   You are arguing, I think, that my editor has no reason to care  
> that the $Id$ tag portion was modified, since subsequent edits will  
> result in the same entity on the server, regardless of whether it  
> refreshes the modified $Id$ tag in the document, so a clarified  
> ETag should be sufficient.

Or simply not returning the etag at all, which tells the client it needs
to obtain the new representation that contains the current Id info.

>   This is probably true for a lot of editing situations.  In the  
> case of normalization of, say, iCalendar data, I think you are right.
>
> version of the document they are looking at when they view the  
> file.  In this case, knowing that the data was modified will let  
> the editor know that a subsequent GET request would be a good  
> idea.  A Subversion client wouldn't have to do this, in that it can  
> probably assume that it knows what the server changed, and make the  
> same edit itself.  But a generic text editor would not.

Actually, the Subversion client wouldn't know the contents anyway,
since only the server knows the variable replacement contents even
for something as simple as ID.  That is why etag should not be sent
in that case, and the result works with any client. Even if the
Subversion client thought it knew what the transformation would be,
and what the Id value should be, it still wouldn't know if any other
transformations might have occurred and thus would have to do a GET
(or receive some other form of notification that would tell it how to
transform its current representation to the current form).  A far more
useful extension would be to define a new 2xx response to PUT that
includes both the new metadata and a patch that would allow the
client to implement a matching transformation on its own copy.

OTOH, it would also be more sensible for the Subversion client to strip
the contents within $Id$ itself, before the PUT, and thus both client
and server would have the same content and a strong etag could be
returned.  In other words, the client uses its knowledge of the resource
definition to remove the unnecessary bits at the application layer
above the HTTP processing, thus making the HTTP interaction more
efficient.

I have no idea what the current Subversion client does in that  
situation,
so this is all hypothetical.

>   I do agree that many client authors are all bent out of shape  
> about the data changing when they needn't be, and it's probably  
> best if they don't refresh the data unnecessarily, but I do think  
> there are valid use cases for having that information.
>
>> My proposal does solve every single case that has been described  
>> so far,

Yes.  A strong ETag in a PUT result means that the entity has not been
transformed in an octet-significant way.  No strong ETag in a PUT result
means that it may have been transformed and thus a refresh is necessary
to obtain both the current form and the current ETag.

> absent the new header.  It's clear that some clarification is  
> needed, and I think this is a good start.  No?

2616 already says it, just not in a way that is readable by anyone
(including the original authors).  The draft wastes a great deal of
text justifying various ways that people have misread 2616 and then
proposes an extension header that doesn't solve any real problem
and adds the very significant problem that no current implementation
is prepared to deal with a new response header that isn't metadata.
I'd rather just fix the problematic text with a clarification.
It explains the intended function of etag without really changing
anything in the protocol -- it just documents an undocumented  
assumption.

The single sentence I supplied is sufficient to solve the problem
of clients not knowing whether or not they need to do a GET after
a PUT, and answering it solves the question of interoperability.
If we want to go further and actually grapple with the feature that
clients would really like to have -- namely, the content of the
transformed bits without an extra round trip -- then half-measures
like Entity-Transform are not good enough.  That feature requires the
transformation to be sent in the response, which would mean a new
response status code and associated semantics.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 02 Dec 2006 12:34:16 +0100

Hi,

given the fact that more than a few persons were confused about the weak 
matching function, I'd propose to add an example here, to appear below 
the definitions in 13.3.3:

     The example below shows the results for a set of entity tag pairs,
     and both the weak and strong comparison function results:

          +--------+--------+-------------------+-----------------+
          | ETag 1 | ETag 2 | Strong Comparison | Weak Comparison |
          +--------+--------+-------------------+-----------------+
          | W/"1"  | W/"1"  | no match          | match           |
          |        |        |                   |                 |
          | W/"1"  | W/"2"  | no match          | no match        |
          |        |        |                   |                 |
          | W/"1"  | "1"    | no match          | match           |
          |        |        |                   |                 |
          | "1"    | "1"    | match             | match           |
          +--------+--------+-------------------+-----------------+

Best regards, Julian

Julian Reschke schrieb:
>>> In Apache's case, there is no such guarantee,  because the same weak 
>>> ETag may be issues to different versions of the  document (if it is 
>>> edited more than once in the same 1-second window),  the data you got 
>>> with an ETag of W/"X" may be completely different  than the data that 
>>> later exists with an ETag of "X", or even the same  W/"X" you got.
>>

---

From: Julian Reschke <[email-masked]>
Date: Sat, 02 Dec 2006 22:00:36 +0100

Hi,

before I get to Roy's mail, I'd like to point out how we got here.

During autumn 2005, a requirement to return strong etags made it into 
RFC2518bis. After some discussion over in the WebDAV WG we realized that 
RFC2616 isn't very clear about what an ETag in a PUT response means.

With that question I went to *this* mailing list; see thread starting at 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2005OctDec/0013.html>. 
After some discussion I went away with the impression that there was 
consensus that the entity tag is for the entity that was stored, not for 
what you sent. And of course everybody seems to agree that a server may 
rewrite content sent with PUT, such as for keyword substitution, object 
re-serialization (think XML or ICS), charset translation, whatever.

The IESG decided that this issue should be resolved, but in a separate 
spec, because it affects many other specs, most of which not based on 
WebDAV.

In the meantime, the first draft that was supposed to do this has long 
expired (<http://tools.ietf.org/html/draft-whitehead-http-etag-00>). 
Also, the IESG has approved (for the standards track!!!!) one 
specification requiring the server to return a strong ETag, although 
content always will be rewritten (XCAP), and another one, requiring 
*not* to return a strong ETag in that case (CALDAV). That essentially 
means that a single HTTP resource never can be both XCAP and CALDAV 
compliant. This may not be a problem in practice, but the thought that 
different Standards Track specifications profile RFC2616 in completely 
opposite directions makes me shudder.

So when I started work on my draft, I did that based on the assumption 
that there *is* consensus on what RFC2616 says, and attempted both a 
proposed clarification, and an extension helping in some of the use 
cases. The key part here is 
<http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-04.html#rfc.section.1.3, 
which tries to explain how we came to the conclusion about the 
aforementioned ETag behavior, and 
<http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-04.html#rfc.section.3>, 
proposing clarifications for RFC2616.

Roy seems to argue from the point of view what RFC2616 *should* be 
saying. That's valid as well, but I really don't see how that helps 
unless we come to a consensus that making an actual change to the spec 
is OK with respect to servers that may break because of that. If we do 
come to that consensus, I'll be happy to either let the draft expire, or 
  to rewrite it based on the new information.

OK, now for my comments...

Roy T. Fielding schrieb:
>> have expressed interest in this information, which is what the 
>> Entity-Transform is meant to address.
> 
> an etag (or included only a weak etag), then the client can conclude that

...I'd say *may* look different. Many servers never return an ETag upon PUT.

> be assuming a transform has occurred if it does not receive a strong etag,

First of all, this argument seems to be based on the assumption that 
content-rewriting servers do not return ETags, which we know is not 
true, and is even required in XCAP. Are these servers non-compliant to 
RFC2616? I don't think so.

Furthermore, I don't get the part about the intermediary. Even if the 
server didn't rewrite the content, and returned a strong ETag, an 
intermediary may have rewritten the content, right?

> to obtain the new representation that contains the current Id info.

But returning the ETag is very useful, and completely harmless unless 
the client tries a byte range request. The final status of the result 
will be the same, no matter whether the client does a 
PUT/GET(refresh)/PUT or a PUT/PUT request, so it's pointless to refresh 
the local copy unless somebody is interested in the newly substituted 
keywords.

> in that case, and the result works with any client. Even if the

Not returning an ETag surely will always work, because the source of 
confusion is eliminated. But instead, you now have to resolve the lost 
update problem, for which the ETag previously would have been the solution.

> above the HTTP processing, thus making the HTTP interaction more

Well, all of this isn't necessary if the server does return the strong 
ETag, and the client just keeps authoring that resource.

> I have no idea what the current Subversion client does in that situation,
>>> My proposal does solve every single case that has been described so far,

I don't think this is was RFC2616 says, and what (some) servers do 
today. And even if we could simply ignore that, this approach would make 
authoring of resources on content-rewriting servers hard, because you 
wouldn't be able to use ETags anymore (that is, you would always have to 
refresh the local copy, although it may be completely unneeded).

> text justifying various ways that people have misread 2616 and then

OK, it would be really helpful if you could point out what actually was 
misread.

> is prepared to deal with a new response header that isn't metadata.

Many protocols have defined new response headers that aren't metadata, 
and I'm not sure what problem that causes in practice. This being said, 
the value of Entity-Transform could actually be seen as metadata, and 
also uses a format which makes it impossible that stale copies of the 
header cause harm in the client.

> anything in the protocol -- it just documents an undocumented assumption.
> 

As far as I can tell, this has the price of making it incompatible with 
RFC2616, and also defies efficient ETag-based authoring on servers that 
do rewrite the content.

> response status code and associated semantics.

Which would be interesting, in particular in combination with PATCH.

Going back to a previous mail:

>> On Nov 28, 2006, at 4:27 PM, Julian Reschke wrote:
>>
>>> The Xythos client always assumes that content isn't rewritten. And if no ETag is returned, it uses the Last-Modified date as cache key. So it's already broken with respect to servers that have to rewrite.
>>>
>>> Roy never made a proposal, and didn't answer when asked for clarification/confirmation.
> 
> I don't like repeating myself over and over again just because you
> want to add an unnecessary feature to HTTP.

Roy, nobody asked you to *repeat* yourself. I was asking because I 
wasn't sure I understood what you said, and it turns out I in fact did not.

> and every single relevant case can be handled by a simple clarification.
> 
> My proposal was:
> 

And the ambiguity here was that it's not clear what "can use" actually 
means.

In my server I can return a strong ETag although the content was 
rewritten, and that ETag can be used both for PUT/If-Match (PUT will 
succeed), GET/If-None-Match and in GET/If-Match+Range (server will 
return the full content because it knows the ETag was sent back upon PUT).

> response if the server does not intend to support that functionality.

So if it does support that functionality, can it return the strong ETag 
even though content was rewritten?

> OTOH, the mechanism for how the server "stores" the representation, whether
> or not it is stored byte-for-byte, and how a server might otherwise manage
> to accomplish the feat of handling a conditional request without preserving
> byte-level equality is none of our business and should never appear in an

Yes. Did anybody suggest that?

> My proposal does solve every single case that has been described so far,

At this point (sorry!) I am still not sure what your precise proposal 
is. Does it require the ETag to be usable as expected, or does it 
require not to return an ETag when the server did rewrite the contents? 
This mail and the mail above seem to be contradictory with respect to this.

> In regard to Apache mod_dav's use of weak entity tags, that is a separate
> file space as a storage mechanism (and thus a separate handler for GET).
> files.  The right solution, though, is to use a property-based back-end
> and store the MD5 on write, which would allow the weak designation to be
> removed and actually solve the real problem of clients wanting a strong

Last time I checked, the ETag handling was completely independent of 
Apache/moddav.

Best regards, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 03 Dec 2006 00:32:19 +0100

> ...I'd say *may* look different. Many servers never return an ETag upon PUT.

I'd say that no matter if an strong or weak etag is returned the client
can assume that the result is based on what it sent in it's PUT request
for as long as the etags compare true, and that there under those
conditions usually is no need to refetch the object to continue
authoring. But returning a weak etag does signal that quite likely
something will differ on a GET for the object.

Also, since the weak etag can not be used in future conditional PUT the
agent better fetch a new copy if a weak etag was returned in response to
PUT if actual details of the content is important. Making the weak etag
mainly suitable for detecting if there has been important edits
modifying the actual content between the PUT and subsequent GET/HEAD.

> RFC2616? I don't think so.

They are compliant in my view. And absolutely certainly so if returning
a weak etag.

> intermediary may have rewritten the content, right?

Right. Content-modifying intermediaries is on their own. HTTP specs is
not too detailed on what those may or may not do but anyone who have
grasped ETag would consider it a functional requirement that content
rewriting intermediaries also remap ETags suitably or at least degrade
them to weak. If not serious confusion may arise.

> the local copy unless somebody is interested in the newly substituted 

True, and in this scenario weak ETags maps extremely well. But I would
not argue that it's sane to return a strong ETag when doing keyword
substitution on the PUT entity.

> update problem, for which the ETag previously would have been the solution.

Yes, and a weak etag is sufficient for solving the lost update problem.
If it wasn't for the stupid fact that a weak ETag can not be used in a
PUT/PUT If-Match sequence if following the spec.

> ETag, and the client just keeps authoring that resource.

Well.. lets return to the Subversion use case again.

The keyword substitution may or may not be important depending on how
the data continues to be used on the client. If we only consider
authoring then it's not very important as the substitution will be
redone again on the next PUT so a PUT/PUT sequence should work out fine
(unless subversion substitutes $Log$ like CVS does...). But it's not the
case for any other use of the content where the substituted keywords may
actually be important (embedded in version identifiers etc).

> refresh the local copy, although it may be completely unneeded).

I don't think RFC2616 says neither yes or no here. Actually I don't
think automatic rewriting of the posted entity was considered at all in
the specification of PUT other than "we don't specify how any of that is
done".

The language used in 9.6 does not leave much room for automatic entity
modifications as part of the PUT processing.

> header cause harm in the client.

Same here. To me this new header allows for reasonable signaling of
server-side PUT entity processing where RFC2616 hasn't been very clear.

But I am also of the view that just making a better definition of weak
etags would go pretty much as far by defining that a strong etag is
octet equal to the PUT entity, while a weak etag may differ in
non-important aspects as is always the case on weak etags.

So I can only second Roy here.

> RFC2616, and also defies efficient ETag-based authoring on servers that 

I don't agree.

Servers which do rewrite the content is free to exploit weak etags in
their PUT responses, signaling that "yes, it means the same but may
differ..", and is frankly about the only way I can make content
rewriting servers fit in the 2616 definition of PUT where no provision
for content rewriting is provided..

And I can't see how using an ETag returned from PUT in subsequent
conditional requests would be incompatible. The returned ETag is after
all supposed to represent the entity created from the PUT request.

> The Xythos client always assumes that content isn't rewritten. And if
> no ETag is returned, it uses the Last-Modified date as cache key. So
> it's already broken with respect to servers that have to rewrite.

And noone would argue that using Last-Modified as a guarantee that the
content has not been rewritten is a good thing to do. Last-Modified
recently is clearly defined as a weak validator which does not guarantee
octet equality.

But it would be unwise for a content rewriting server to preserve
Last-Modified if supplied by the client. A Last-Modified some time ago
may be considered a strong validator almost like a strong ETag.

> And the ambiguity here was that it's not clear what "can use" actually 

strong validator by definition means octet equality. I guess the
question is "of what"?

> return the full content because it knows the ETag was sent back upon PUT).

I don't read 2616 in this way, but I understand that others do.

The PUT request entity is defined as the entity to be stored under the
URL. No mention of any possible rewriting of the content here. But it's
intentionally excluded how the server stores the entity.

A strong ETag is defined to guarantee octet equality, plus imho
reasonable equality in entity headers. Note that the ETag refers to the
entity, not the entity-body.

> even though content was rewritten?

In my view of 2616 it should not, but it's not strongly spelled out so I
may be wrong.

But with the crippled definitions of weak etags little other choice is
available.. Your proposed draft is a way forward which allows for use
cases to continue use If-Match based on an ETag from a rewritten PUT in
a well defined manner.

Perhaps an alternative would be to more clearly define the purpose of
weak etags, and also extend their scope to allow them to be used in most
operations where octet equality is not really a strict requirement (i.e.
most operations except for If-Range).  To make this fit nicely in HTTP
may require a split in three categories of validators:

 * very weak. Give no strong guarantees at all. In this category one
finds "Last-Modified: justnow" (based on the Date of the same response,
not some clock). But this may be seen as a weak validator if one wants
to or if it by other means outside of HTTP is known for certain that two
edits within the same second will not occur (may even be upgraded to a
strong validator in such case).

 * weak. Semantic equivalence. In this category one finds weak etags or
Last-Modified being on the correct side of time but not equal to what
was matched against.

 * strong. Full equivalence on octet level and in case of ETag semantic
equivalence of important entity headers as well. Entity headers is part
of an entity and should not be ignored, but octet equivalence is not
defined or realistic. In this category one finds strong etags and
last-modified some time ago.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 3 Dec 2006 00:25:19 +0000

> the modified $Id$ tag in the document, so a clarified ETag should be  

I don't think that's what Roy is arguing.

I think what Roy is saying is, if the client receives a strong Etag on
PUT, the client can use it in subsequent conditional and byte-range
requests to confirm that the representation held by the client is
still current, byte for byte.  (It has to be byte for byte otherwise
range requests don't work).  The representation held by the client is
of course what it sent in PUT, because it can't be anything else - it
doesn't have anything else.

That is quite the opposite of what you say he argues, which is "my
editor has no reason to care" that the document was modified by the
server.  On the contrary, Roy's statement on Etag grants the client
the ability to check whether the document was modified.

> a generic text editor would not.

If the editor needs to show the user what would be received by a
subsequent GET, it should do this:

    1. PUT /file
       => Receive Etag in response from PUT

    2. GET /file
       If-None-Match: <etag received in step 1>

The server modifies the file after it's PUT - it replaces $Id$ with
something else.

So the GET will retrieve the new contents - exactly what you asked for.

If the server does not transform the file at all, step 2 will simply
return a "Not Modified" response.

Also exactly what you asked for.

> use cases for having that information.

And Roy's statement on the meaning Etag gives you that information.

>   Is the above use case covered?  I'm not seeing how.

Yes, as I described :)

> and I think this is a good start.  No?

I am surprised at the debate, because it seems quite obvious to me
what a strong Etag is "supposed" to mean as a concept.

That is: if you have received a strong Etag from the server, it means
you may use it for conditional and byte-range GET requests in future,
to confirm the client's stored representation is byte for byte
accurate.

That's what it means for everything else.  I don't see why people want
to give it a different meaning when it's in the response from a PUT.

(And in case anyone's confused, I think that means I agree with Roy).

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Sun, 03 Dec 2006 14:33:04 +0100

> something will differ on a GET for the object.

Yes.

> modifying the actual content between the PUT and subsequent GET/HEAD.

Yes. So that solves the lost update problem (the refetch is safe not to 
lose somebody else's update), but it still requires the client to refetch.

> They are compliant in my view. And absolutely certainly so if returning

Yes.

> them to weak. If not serious confusion may arise.

+1. Contrary what some say, this topic is already complicated enough. 
Let's get an understanding of how authoring works in the absence of 
these kinds of intermediaries first.

> substitution on the PUT entity.

The approach using the weak ETag has the problem requiring an additional 
GET.

In general, with APP, CALDAV and XCAP we have several protocols where it 
will be the *default* case that content rewriting happens. That's 
because they will usually store content not as supplied by the client, 
but as the serialization of an internal representation. XCAP will run 
everything through an XML parser, APP many resources, and store the 
result as some kind of serialization of their object model (which may be 
the XML Infoset or something else). CALDAV servers in general will use 
ICS parsers.

I think in the end we should be able to come up with a solution where 
these resources on these servers can be written to with no need of a 
refetch. Optimally, that solution is compliant to RFC2616 (I would even 
say it's required to be so, so that's why it's essential to clarify what 
RFC2616 really is saying).

I'd also like to mention that in many cases, an authoring session looks like

1) Client application starts, gets the content (GET)
2) User does edits, saves (PUT)
3) User does more edits, saves (PUT)
...
n) User is done, final PUT

Ir would be good if at least a refetch GET could be avoided for every 
but the last PUT.

> PUT/PUT If-Match sequence if following the spec.

So, can we resolve all of this by relaxing that requirement?

> actually be important (embedded in version identifiers etc).

That would indicate there's a real use case for have more information, 
either by returning diff information as proposed by Roy, or by adding 
more detailed information to Entity-Transform.

> modifications as part of the PUT processing.

"HTTP/1.1 does not define how a PUT method affects the state of an 
origin server."

> So I can only second Roy here.

I do agree that upgrading weak ETags so they become usable for authoring 
would be a potential solution to that problem. We would still need to 
clarify RFC2616 with respect to returning ETags on PUT at all, and also 
with requirements on strong ETags. And when making that change, we need 
to take into account that we may break existing servers.

> all supposed to represent the entity created from the PUT request.

OK, let me rephrase this: RFC2616 allows servers to rewrite content on 
PUT, and it also allows to return a strong ETag. Servers do that today. 
Now I hear that this may be a bad idea, and not what the authors of 
RFC2616 intended, but it I don't see how the actual text of RFC2616 can 
be understood differently.

So if this is really a bug in RFC2616, it would a good thing to add that 
to the errata list ASAP, and to tell the RFC-Editor not to publish XCAP, 
because it violates that requirement.

> strong validator by definition means octet equality. I guess the

Yes.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 03 Dec 2006 14:41:29 +0100

> (And in case anyone's confused, I think that means I agree with Roy).

Jamie,

the issue is that RFC2616 doesn't clearly define what an ETag means for 
PUT resulting in a status 200.

For 201 Created it *does* say 
(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.10.2.2>):

"A 201 response MAY contain an ETag response header field indicating the 
current value of the entity tag for the requested variant just created, 
see Section 14.19."

IMHO the only way to read this is that you're getting an ETag for what 
the server stored, not what you sent.

So, again, this seems to turn into a discussion about what RFC2616 
*should have been saying*, not what it actually says today.

Now if there's broad agreement that RFC2616 is broken with respect to 
this, let's fix it, potentially breaking some existing servers. But 
let's be clear about that really is a change.

Best regards, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 03 Dec 2006 15:44:50 +0100

> lose somebody else's update), but it still requires the client to refetch.

Or at least a HEAD roundtrip to verify the ETag, assuming the weak etag
it has been upgraded to a strong etag with the same value signaling that
the entity has been finalized. Or the specs changed to allow the weak
etag to be used as guard in future authoring.

Related note: If the server can not issue a strong ETag for the entity
even on a GET/HEAD then the lost update problem can not be fully solved
without changing the specs to allow for weak etags in PUT If-Match.

> The approach using the weak ETag has the problem requiring an additional 

Yes, and only because the specs is afraid of weak etags of any other
condition which may be uncertain.

> RFC2616 really is saying).

My view is that the best solution is to broaden the allowed use of weak
etags.

Many of these servers can not guarantee a octet equality on the objects
anyway. A slight change in software may change of the objects is
rendered back in the transmitted format, requiring all objects to have
new ETags assigned to them. CalDAV is one such example where the store
may be using a completely different format than seen on the wire.

> Ir would be good if at least a refetch GET could be avoided for every 

Which allowing weak etags in PUT If-Match would do just fine.

> So, can we resolve all of this by relaxing that requirement?

I think so.

A PUT returning a weak etag is sufficient for binding the PUT entity
together with the server entity when octet equality is not guaranteed.

The server then upgrading this to a strong ETag (with the same value) on
the provided resource allows for full utilization of the ETag on GET
request including If-Range.

And it's quite obvious that the returned weak etag does not guarantee
octet equality, so clients needing the exact representation know they
need to GET the finalized entity. And this works out even if the server
is still returning weak etags in the GET.

> more detailed information to Entity-Transform.

Neither works out very well I am afraid. The diff approach is the more
interesting I think, but this has to be left to the application. It's
not realistic for HTTP to specify a diff format which works for all
applications.

> "HTTP/1.1 does not define how a PUT method affects the state of an 

Yes, but what state?

You also have

 "The PUT method requests that the enclosed entity be stored under the
supplied Request-URI."

and

 "Unless otherwise specified for a particular entity-header, the
entity-headers in the PUT request SHOULD be applied to the resource
created or modified by the PUT."

when combined kind of rules out rewriting as in this set you have things
like Content-Length and Content-MD5.

> to take into account that we may break existing servers.

The specs does allow (and even clearly recommends in 201) ETags to be
returned, in all positive responses to PUT.

The weak area as I see it is specifying how servers not providing octet
equality should act to not confuse clients regarding the state of the
entity. It's quite obvious from this discussion that the RFC can be
understood in more than one way even after careful reading.

Regards
Henrik

---

From: "Larry Masinter" <[email-masked]>
Date: Sun, 3 Dec 2006 07:07:35 -0800

> let's be clear about that really is a change.

Well, you might find yourself going back to the early discussions
about etag and PUT (from 1996) to decide whether this 'really is a change'
rather than an editorial issue or clarification.

http://lists.w3.org/Archives/Public/ietf-http-wg-old/1996MayAug/0298.html
> While this isn't much of an issue today, it may become one.

Personally (having been in many of those early discussions) I'd
say this was a clarification rather than a 'change'.

Larry

---

From: "Justin Erenkrantz" <[email-masked]>
Date: Mon, 4 Dec 2006 10:48:06 +0100

> so this is all hypothetical.

FWIW, Subversion does the keyword translation locally on the client
side (in libsvn_wc).  In other words, it follows the second
definition.  This is why if you do a HTTP GET, you don't see the
keywords expanded.  Only the Subversion client has enough knowledge to
do the translations - the client must untranslate the keywords before
sending it up to the server.

HTH.  -- justin

---

From: "Justin Erenkrantz" <[email-masked]>
Date: Mon, 4 Dec 2006 10:57:40 +0100

> Last time I checked, the ETag handling was completely independent of

No - ETag is handled by mod_dav through its getetag hook.  The catch
here is that the simplistic mod_dav_fs doesn't use that and defers to
the core handlers - which uses inodes and last-modified resources -
which can produce weak etags when the last-modified date is the
current second.  (There is a #ifdef that'll make mod_dav_fs not defer
to the core though.)

FWIW, Subversion's mod_dav_svn uses its own ETag implementation -
based on rev # and resource; and will never produce weak etags -
except for collections - which always produce weak ETags (as
Subversion includes header and footer sugar that can change over
time).

HTH. -- justin

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 8 Dec 2006 18:06:17 -0800

Just one question from all this conversation I just caught up on  
(didn't realize Wilfredo had posted to the main list).

If a server doesn't return an ETag at all in response to PUT , isn't  
the client likely to assume that the server wants it to use the Last- 
Modified value instead?  In which case today's clients will all  
assume they already have the content (what they PUT)  and never  
download what the server actually stored.

Lisa

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 8 Dec 2006 18:47:38 -0800

On Dec 8, 2006, at 6:06 PM, Lisa Dusseault wrote:

> If a server doesn't return an ETag at all in response to PUT ,  
> isn't the client likely to assume that the server wants it to use  
> the Last-Modified value instead?  In which case today's clients  
> will all assume they already have the content (what they PUT)  and  
> never download what the server actually stored.

I hope not, since the Last-Modified is an entity-header and refers
only to the content of the response message.  ETag is a response
header because it is resource metadata, not message metadata.

....Roy

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 10 Dec 2006 17:19:27 +0100

fre 2006-12-08 klockan 18:06 -0800 skrev Lisa Dusseault:

> the client likely to assume that the server wants it to use the
> Last-Modified value instead?  In which case today's clients will all

Well.. the Last-Modified in that scenario will by definition be weak
unless it was provided by the client and preserved by the server.

Specs is quite clear on the fact that clients can not safely assume
octet equality with a weak condition.

Regards
Henrik