Subject: Re: Etag-on-write, 4rd attempt (== IETF draft 03), was: I-D ACTION:draft-reschke-http-etag-on-write-03.txt

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Sat, 4 Nov 2006 13:33:53 -0800

Julian-

   I apologize for the absurdly slow response, but here goes.  Overall  
feedback: I think this is excellent.

Section 1.2:

   I'm not sure the "Furthermore" note is interesting.  Of course it  
is possible that one a resource is put onto a server that it's body  
may change again later.  Whether that happens via a PUT request, or  
via PROPPATCH, or even outside of HTTP altogether isn't really all  
that relevant.  What matters is that changed content requires a  
changed ETag, and that's already quite clear in HTTP.

   OK, this becomes clearer when we get to section 5...

Section 1.2:

   The t(e2(e1(c))) = t(e2(t(e1(c)))) explanation is very useful.

   As a general note, I still don't quite understand the widespread  
aversion to weak ETags, and I think that the this editing property  
show why weak ETags *do* work in an authoring environment.  That the  
server might give you a different-but-equivalent representation isn't  
a problem if any "equivalent" version of the document can be edited by  
a client and sent back to the server and produce an "equivalent"  
result in the end.

   I'm rather annoyed that CalDAV was short-sighted here and insisted  
that all calendar resources must have strong ETags when a weak ETag  
could be entirely appropriate, but then I think the entire section  
5.3.4 in CalDAV was hasty and poorly thought out.

Section 4:

   I think it's useful here to say something about weak ETags.  In  
particular, I would expect an entity-transform-keyword of "identity"  
followed by a weak entity-tag to indicate that a semantically  
equivalent entity was stored, not an octet-by-octet copy.  Thus, a  
user agent MUST NOT assume an octet-by-octet entity was stored if a  
weak ETag was assigned.

   I suppose another entity-transform-keyword could indicate semantic  
equivalence, but that's already covered by the nature of the ETag.

Section 4.1:

   What's an example of a situation in which a server would send you a  
stale Entity-Tag header?

Section A.1:

   This example works well to illustrate what you are trying to  
explain, but, for what it's worth, doesn't work well for a Subversion  
client per se.  The reason being that the whole point of the $Revision 
$ keyword is that it shows you want revision of a document you have  
checked out in your working copy.  That is, which is doesn't matter in  
terms of the integrity of the authoring process (what you are  
explaining), it does matter for it's intended usage, which is as an  
informational bit of text maintained by the subversion software in  
your document.

   That said, in this particular case, the client should understand  
this aspect of how the protocol works, and it should be able to modify  
the $Revision$ text in the document itself without having to follow an  
edit with a GET request, assuming that it knows that the ETag == the  
revision number.  That may not work for other keywords, though.

Section A.2:

   This is a particularly notable example if the client did a LOCK  
operation, followed by the PUT and PROPPATCH in the example, in which  
case a client might have more reason to assume that the content hasn't  
been modified since the PUT operation, since no other client could  
have made an edit in the mean time.

Section A.2:

   Note that in Request (3), the title property sent contains spaces  
and newlines which were removed in the new entity body shown.  So even  
in the PROPPATCH example, there is some server-side transformation of  
sent data going on.  Obviously the Entity-Transform header can't help  
us here due to the Multi-Status response, but one might imagine an  
equivalent sort of thing in a new child of the the propstat element.

More on Entity-Transform:

   An Entity-Transform example with a transformed entity might be  
useful.  (I like lots of examples.)  Perhaps an example of PUTing JPEG  
image data where the server stores it as PNG?  It's easy to turn this  
into a rat-hole, I suppose; what if the transform is lossy (PNG- 
 >JPEG), etc.?  Is that worth exploring here, or too zany?

   Perhaps limiting the example to something that's easily  
transformable both ways is best.  iCalendar <-> hCalendar, for  
example.  Of interest here is the change of Content-Type.

   I reckon that you may have lets this out intentionally, to be  
addressed later.  I'm just trying to think of cases where this draft  
might make that sort of operation harder than necessary... not  
thinking of any as yet.

More on weak ETags:

   I think there is room for a weak ETag example here, new style.   
I'll assume for now that my comment about the "identity" entity- 
transform-keyword makes sense:

 >> Request (1)

   PUT /cal.ics HTTP/1.1
   Host: example.com
   Content-type: text/icalendar

   BEGIN:VCALENDAR
   CALSCALE:GREGORIAN
   VERSION:2.0
   BEGIN:VEVENT
   UID:00B33BAF-30A0-4C2A-A93A-8236318F8002
   DTSTART;VALUE=DATE:20020212
   DTEND;VALUE=DATE:20020213
   DTSTAMP:20041015T171054Z
   RRULE:FREQ=YEARLY;INTERVAL=1;BYMONTH=2
   SUMMARY:Lincoln's Birthday
   END:VEVENT
   END:VCALENDAR

 >> Response (1)

   HTTP/1.1 204 NO_CONTENT
   Content-Type: text/plain
   Entity-Transform: "identity"
   ETag: "W/1"

Because the server stores the iCalendar data in an intermediate  
format, it is possible that the entity changes in a semantically  
equivalent manner between requests, for example:

 >> Request (2)

   GET /test.html HTTP/1.1
   Host: example.com

 >> Response(2)

   HTTP/1.1 200 OK
   Content-Type: text/icalendar
   ETag: "W/1"

   BEGIN:VCALENDAR
   CALSCALE:GREGORIAN
   VERSION:2.0
   BEGIN:VEVENT
   DTSTAMP:20041015T171054Z
   RRULE:FREQ=YEARLY;INTERVAL=1;BYMONTH=2
   SUMMARY:Lincoln's Birthday
   DTSTART;VALUE=DATE:20020212
   DTEND;VALUE=DATE:20020213
   UID:00B33BAF-30A0-4C2A-A93A-8236318F8002
   END:VEVENT
   END:VCALENDAR

In this case, the iCalendar properties in the event are in a different  
order.  Perhaps the server retrieves these values from separate  
sources and renders them as they arrive; the timing may be different  
in future requests, but the data is semantically equivalent in all  
cases.

Regardless, the result is similar to the example in appendix A.2; the  
client can continue editing the cached local copy despite the  
differences that exist in the stored entity on the server.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 05 Nov 2006 19:41:54 +0100

> feedback: I think this is excellent.

Good to hear.

> Section 1.2:
> 
>   I'm not sure the "Furthermore" note is interesting.  Of course it is 
> possible that one a resource is put onto a server that it's body may 
> change again later.  Whether that happens via a PUT request, or via 
> PROPPATCH, or even outside of HTTP altogether isn't really all that 
> relevant.  What matters is that changed content requires a changed ETag, 
> and that's already quite clear in HTTP.

I thought it was worthwhile mentioning this because I've heard from 
clients struggling with this problem (at interop events).

>   The t(e2(e1(c))) = t(e2(t(e1(c)))) explanation is very useful.

Thanks.

> aversion to weak ETags, and I think that the this editing property show 
> why weak ETags *do* work in an authoring environment.  That the server 
> might give you a different-but-equivalent representation isn't a problem 
> if any "equivalent" version of the document can be edited by a client 
> and sent back to the server and produce an "equivalent" result in the end.

I do agree in general. The problem is that to include weak Etag 
discussion into this draft would mean to make it more complicated, and 
there are so many people not getting weak etags that it would likely 
complicate and delay this specific issue.

> could be entirely appropriate, but then I think the entire section 5.3.4 
> in CalDAV was hasty and poorly thought out.

Right. In particular it defeats etag support in editing calendar entries 
when the server has rewrite the contents (for instance because the back 
end is a calendaring system, not a "generic" store).

So -- out of curiosity -- what is Apple's server doing with respect to 
this requirement?

> followed by a weak entity-tag to indicate that a semantically equivalent 
> entity was stored, not an octet-by-octet copy.  Thus, a user agent MUST 
> NOT assume an octet-by-octet entity was stored if a weak ETag was assigned.

Well, the way it's written a weak ETag can never appear with "identity". 
I just added an attempt to clarify that, see 
<http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-latest.html#rfc.section.4>).

Note that the extensibility rules would allow future definitions, such 
as "semantic-identity" to be used with weak ETags.

> equivalence, but that's already covered by the nature of the ETag.

I think strong vs weak is somehow orthogonal to rewrite vs no-rewrite, 
so the Entity-Transform header may be useful for weak ETags as well.

>   What's an example of a situation in which a server would send you a 

A server wouldn't, but a client might get one due to a proxy 
understanding the header as entity header.

> Section A.1:
> 
>   This example works well to illustrate what you are trying to explain, 
> but, for what it's worth, doesn't work well for a Subversion client per 
> se.  The reason being that the whole point of the $Revision$ keyword is 
> that it shows you want revision of a document you have checked out in 
> your working copy.  That is, which is doesn't matter in terms of the 
> integrity of the authoring process (what you are explaining), it does 
> matter for it's intended usage, which is as an informational bit of text 
> maintained by the subversion software in your document.
> 
>   That said, in this particular case, the client should understand this 
> aspect of how the protocol works, and it should be able to modify the 
> $Revision$ text in the document itself without having to follow an edit 
> with a GET request, assuming that it knows that the ETag == the revision 
> number.  That may not work for other keywords, though.

Hm. So do you propose to use a different keyword, which the client would 
be able to update itself?

> been modified since the PUT operation, since no other client could have 
> made an edit in the mean time.

OK, so would it make sense to add a note stating that in a case like 
this, even the presence of a WebDAV lock wouldn't have helped the client?

> Section A.2:
> 
>   Note that in Request (3), the title property sent contains spaces and 
> newlines which were removed in the new entity body shown.  So even in 
> the PROPPATCH example, there is some server-side transformation of sent 
> data going on.  Obviously the Entity-Transform header can't help us here 
> due to the Multi-Status response, but one might imagine an equivalent 
> sort of thing in a new child of the the propstat element.

Good catch - I just fixed the example so that the whitespace indeed 
matches 
(<http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-latest.html#rfc.section.A.2>).

> into a rat-hole, I suppose; what if the transform is lossy (PNG->JPEG), 
> etc.?  Is that worth exploring here, or too zany?

I guess I should add the Entity-Transform header to a variant of the 
example in Appendix A.1...

>   Perhaps limiting the example to something that's easily transformable 
> both ways is best.  iCalendar <-> hCalendar, for example.  Of interest 
> here is the change of Content-Type.
> 
> might make that sort of operation harder than necessary... not thinking 
> of any as yet.

A server always has the freedom not to send the header, so I don't think 
it ever gets harder...

> More on weak ETags:
> 
>   I think there is room for a weak ETag example here, new style.  I'll 
> assume for now that my comment about the "identity" 
> entity-transform-keyword makes sense:
> ...

Understood, but I think starting to talk about weak etags in this 
context will make it much harder to get this done. When I wrote this, I 
was hoping to prevent other authors to profile HTTP in incompatible 
ways; exactly what we now have with XCAP and CALDAV. So timing is really 
important here; I don't want it to take years :-) .

Best regards, Julian

---

From: Lisa Dusseault <[email-masked]>
Date: Sun, 5 Nov 2006 13:29:43 -0800

Speaking for myself, my aversion is to ambiguously-defined weak  
ETags.  You could probably define something called "Weak ETags" and  
say more about how they work than RFC2616 does, and create something  
useful.  I'm sure your idea of what weak ETags do is a sane one and  
if everybody agreed we'd have interoperability.  But we've had  
interoperability problems around weak ETags and that's the root cause  
of my aversion.

Lisa

On Nov 4, 2006, at 1:33 PM, Wilfredo Sánchez Vega wrote:

> show why weak ETags *do* work in an authoring environment.

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 05 Nov 2006 23:32:06 +0100

sön 2006-11-05 klockan 13:29 -0800 skrev Lisa Dusseault:
> Speaking for myself, my aversion is to ambiguously-defined weak ETags.

In what sense is the current definition of weak ETags ambiguous?

Sure, it's not always obvious when it's reasonable to use them, but the
definition and protocol implications is quite clear to me..

Also some of the RFC2616 restrictions on weak validators can be
questioned. I.e. whats actually wrong with using a weak If-Match in
DELETE/PUT?

Regards
Henrik

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Sun, 5 Nov 2006 17:54:38 -0800

Well, that's going to be dependent on the delivered format.  Many  
formats have multiple representations which are considered  
equivalent.  This includes iCalendar, which is why CalDAV's shoo-shoo- 
go-away stance on weak ETags is, well, weak.

   The example I gave, where the order of iCalendar properties changes  
within an iCalendar component, is what I have in mind in that context.

	-wsv

On Nov 5, 2006, at 1:29 PM, Lisa Dusseault wrote:

> interoperability problems around weak ETags and that's the root  
> cause of my aversion.
>
> Lisa
>

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Mon, 6 Nov 2006 11:18:58 -0800

And to clarify, so long as Julian's formula holds true:

	t(e2(e1(c))) = t(e2(t(e1(c))))

   A client doesn't necessarily need to care whether an ETag is strong  
or weak.

   CalDAV is a prime example of where weak ETags are appropriate.  If  
the underlying data store isn't iCalendar, and the server needs to  
render iCalendar data every time a resource is requested, it's  
entirely possible that the iCalendar data is semantically but not  
binary equal every time.  Yes, you can make sure your iCalendar  
generation is deterministic, but if you change the software, you need  
to change the strong ETags for every resource on your server along  
with it.  That's inane.  CalDAV dropped the ball here.

	-wsv

On Nov 5, 2006, at 5:54 PM, Wilfredo Sánchez Vega wrote:

> equivalent.  This includes iCalendar, which is why CalDAV's shoo- 
> shoo-go-away stance on weak ETags is, well, weak.
>
>   The example I gave, where the order of iCalendar properties  
> changes within an iCalendar component, is what I have in mind in  
> that context.
>
> 	-wsv
>
>
>> say more about how they work than RFC2616 does, and create  
>> something useful.  I'm sure your idea of what weak ETags do is a  
>> sane one and if everybody agreed we'd have interoperability.  But  
>> we've had interoperability problems around weak ETags and that's  
>> the root cause of my aversion.
>>
>> Lisa
>>

---

From: Helge Hess <[email-masked]>
Date: Mon, 6 Nov 2006 20:35:06 +0100

On Nov 6, 2006, at 20:18, Wilfredo Sánchez Vega wrote:
>   CalDAV is a prime example of where weak ETags are appropriate.   
> If the underlying data store isn't iCalendar, and the server needs  
> to render iCalendar data every time a resource is requested, it's  
> generation is deterministic, but if you change the software, you  
> need to change the strong ETags for every resource on your server  
> along with it.  That's inane.

Agreed.

However, if this is insane, don't we need to extend if-match to allow  
for weak etags? That is, remove this sentence in rfc2616 14.24:
   A server MUST use the strong comparison function (see section 13.3.3)
   to compare the entity tags in If-Match.

Greets,
   Helge
-- 
Helge Hess
http://docs.opengroupware.org/Members/helge/

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Mon, 6 Nov 2006 13:52:30 -0800

It does seem that extending If-Match would be useful.

	-wsv

On Nov 6, 2006, at 11:35 AM, Helge Hess wrote:

> However, if this is insane, don't we need to extend if-match to  
> allow for weak etags? That is, remove this sentence in rfc2616 14.24:

---

From: Helge Hess <[email-masked]>
Date: Tue, 7 Nov 2006 00:08:32 +0100

On Nov 6, 2006, at 20:18, Wilfredo Sánchez Vega wrote:
> Yes, you can make sure your iCalendar generation is deterministic,  
> but if you change the software, you need to change the strong ETags  
> for every resource on your server along with it.  That's inane.

Thinking about it, I suppose it isn't that insane or impractical.

If the binary representation changes if the software changes, you  
"just" need to embed a software version identifier in the etag? (eg  
the iCal generation framework version [A/B/C...]) That doesn't sound  
too hard / inappropriate.

Greets,
   Helge

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 07 Nov 2006 14:33:53 +0100

>    to compare the entity tags in If-Match.

Which in itself is a rather stupid restriction in RFC2616, especially if
considering that "If-Match: *" and "If-Unmodified-Since" is allowed
which both are very weak match conditions.

The fact that weak etags is forbidden in If-Range is quite natural as
range merging is very dependent on octal equivalence, but the reasoning
of the other specified limitations of weak etags is not so easy to
understand and seriously limits the usability of weak etags which
despite their weak property is still stronger than Last-Modified (weak
etags guarantee semantic equivalence, which Last-Modified doesn't).

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 07 Nov 2006 17:01:57 +0100

> etags guarantee semantic equivalence, which Last-Modified doesn't).

Actually, a workaround is to use the WebDAV "If" header instead; see 
<http://greenbytes.de/tech/webdav/draft-ietf-webdav-rfc2518bis-15.html#rfc.section.10.4.4>.

Best regards, Julian

---

From: Wilfredo Sánchez Vega <[email-masked]>
Date: Tue, 7 Nov 2006 14:10:01 -0800

It's not hard for a server to implement, but it's rather silly to  
invalidate caches for all resources when it shouldn't be necessary.

	-wsv

---

From: Helge Hess <[email-masked]>
Date: Tue, 7 Nov 2006 23:29:17 +0100

> invalidate caches for all resources when it shouldn't be necessary.

I have no specific preference on this issue, but supporting two kinds  
of etags doesn't exactly reduce complexity in the HTTP stack and it  
might be worth dropping one of them (that is, only support strong  
etags).

Invalidated caches after software upgrades most likely won't be a  
major real world issue (how often do you actually change the  
generator). In fact I would tend to claim that implementing weak  
etags properly is very likely error prone (hard work to make the  
generator developer properly takes care of ensuring "ABI  
compatibility", much simpler/reliable to bump the version and drop  
the cache).

Greets,
   Helge

>> "just" need to embed a software version identifier in the etag?  
>> (eg the iCal generation framework version [A/B/C...]) That doesn't  
>> sound too hard / inappropriate.