Subject: RE: ETags and concurrency control

From: "Brian Smith" <[email-masked]>
Date: Sun, 27 Apr 2008 22:01:15 -0700

Pablo Castro wrote:
> Weak ETags: the HTTP spec is pretty explicit that 
> ETag/if-match headers require bit-by-bit equality of the 
> resource, and if the resource changes a single bit then the 
> ETag should be different. There is a concept of "weak ETags" 
> that can be equal if the resource is "semantically 
> equivalent", but the spec indicates that is only to be used 
> in GETs. So for the cases where a developer uses something 
> that's not a reliable timestamp as an ETag we end up 
> deviating somewhat from the spec. I'm sure that other folks 
> have ran into this before.

(from the blog post):
> You can also used a relaxed form of ETags where the entity
> might change but the ETag stay the same. It's not
> completely HTTP compliant and may confuse intermediate
> systems, but it may be your only option in some scenarios.

Nobody should implement optimistic concurrency in SQL without using a
version field (e.g. timestamp) or checking every single value in the row (or
at least the ones being updated). Otherwise, you don't have optimistic
concurrency; you have a giant mess. The same applies to ETags; if the server
doesn't vary the ETag when the resource (representation) changes, then you
will end up with a huge mess. 

You can use MD5, SHA-1, or SHA-2 hashing to keep the ETag at a reasonable
size. Many backup systems and distributed filesystems are using SHA-* hashes
for duplicate block detection. You should be able to find quite a few
whitepapers and research papers that detail the (minute) risks associated
with using these hashes for matching.

I can't think of a reason that weak ETag comparison should be forbidden for
non-safe HTTP requests (PUT or DELETE), as long as people are using matching
weak ETags only for "semantically equivalent" representations. Strong ETag
comparison should only be needed for range requests. Try bringing it up on
the HTTP Working Group mailing list ([email-masked]) and see what they
have to say.

Regards,
Brian

---

From: Robert Siemer <[email-masked]>
Date: Mon, 28 Apr 2008 13:44:47 +0200

> the HTTP Working Group mailing list ([email-masked]) and see what they

That raises three issues:

1) It's not in RFC2616 (weak comparison for non-GET) and so it's not on 
   RFC2616bis charta.

2) Weak ETags don't mean "semantically equivalent" anymore. They mean 
   nothing now (see i101). As of today there is no replacement text 
   proposed for i101, but weak ETags could get degraded to something way 
   weaker than Last-Modified.

   (mail archive and
   http://www3.tools.ietf.org/wg/httpbis/trac/ticket/101
   )

3) the upgrade/downgrade questions will come up again:
   match_weak("xyz", W/"xyz")

   is 'true' or 'false'?

And 2) makes even future considerations for extending the weak etag 
mechanism much harder.

Regards,
Robert

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 28 Apr 2008 21:27:51 +0200

> 1) It's not in RFC2616 (weak comparison for non-GET) and so it's not on 

Not convinced. The current limitations on weak etags is just silly with
the exception of If-Range..

In my view it's a specification error that validators based on
Last-Modified is allowed in more places than weak etag based ones.

>    weaker than Last-Modified.

As specifiedin RFC2616 weak ETags is already less useful as cache
validator than Last-Modified, even if even the most trivial
implementations do generate a much stronger weak ETag cache validator
than Last-Modified.

>    match_weak("xyz", W/"xyz")

True. We have already beaten that discussion to death I think.
Specifications is very clear on this.

The error many seems to get trapped in is thinking that because of this
clients may upgrade the received weak etag as a strong one in order to
be able to acutally use the etag for anything useful, but this MUST NOT
be done and breaks the whole scheme in obvious ways.

Regards
Henrik

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 30 Apr 2008 21:46:04 -0700

Henrik Nordstrom wrote:
> > 1) It's not in RFC2616 (weak comparison for non-GET) and so 
> >    it's not on RFC2616bis charta.
> 
> Not convinced. The current limitations on weak etags is just 
> silly with the exception of If-Range..
>

I agree with Henrik. The weak validator stuff in RFC 2616 doesn't make any sense. The point of the working group is to fix things so that the specification makes sense.

> 
> > 2) Weak ETags don't mean "semantically equivalent" anymore. 
> >    They mean nothing now (see i101). As of today there is no
> >    replacement text proposed for i101, but weak ETags could
> >    get degraded to something way weaker than Last-Modified.
> 

The proposed resolution for i101 just changes one kind of hand waving ("semantic equivalent") for another ("good enough, from the server's point of view"). To me, the two phrases mean exactly the same thing.

AFAICT, The issue with i101 is that some servers do not general *useful* weak ETags; at least as Apache seems to just generate weak ETags that it will never match (which is inefficient but not totally broken). I'm not sure that their behavior is a reason to change the spec.--no harm, no foul. Besides, the existence of noncompliant implementations should not force the definition of compliance to change when that definition has useful properties.

To me, "semantically equivalent" is something that is definitely vague. However, the use of strong ETags for range requests makes things clearer: If you can guarantee that your server will always generate a byte-for-byte identical representation (usable for range requests) for a given ETag, use a strong one; otherwise, if you think that generating an ETag makes sense at all, use a weak one. For example, mod_deflate should never return a strong ETag because the entity it generates is dependent on the system configuration. With a strong ETag, mod_deflate needs to ensure that the ETag changes whenever the mod_deflate configuration changes and whenever the system's zlib changes; with a weak ETag, it could continue to ignore these little details (like it does now when generating strong ETags.)

In fact, I would say that weak ETags should be the default choice, and strong ETags should only be used when the application has specifically ensured that there is a one-to-one correspondence between the ETag and the byte stream that comprises the entity--in other words, only use strong ETags when you could support range requests (whether you support range requests or not). The restriction against using weak ETags in PUT and DELETE requests forces applications to use strong ETags in situations where they are not guaranteeing this one-to-one correspondence.

The alternative would be to deprecate weak ETags and discourage their use, weaken the definition of strong ETag to match what weak ETags were original for, and then say that strong ETags are only guaranteed to have a one-to-one correspondence with entities if the server supports range requests for that resource. That seems to be the effective result of the proposed i101 resolution.

> The error many seems to get trapped in is thinking that 
> because of this clients may upgrade the received weak ETag
> as a strong one in order to be able to acutally use the etag
> for anything useful, but this MUST NOT be done and breaks
> the whole scheme in obvious ways.

Agreed 100%.

Regards,
Brian

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 01 May 2008 17:48:15 +1200

I have to admit to always having a philosophical problem with the entire 
concept of weak etags.

semantic equivalence of say HTML content can surely only be reliably 
determined by a human?

Which means that to save client caches, a human has to decide whether a 
change to an entity is semantically significant or not when the change 
is made, and there needs to be a system to propagate this information 
via the server to the client - all to save some poor client who honours 
weak tags from the task of getting the up-to-date version if it didn't 
really change all that much (who decides?).

Doesn't this basically just make weak etags administratively burdensome 
and basically pointless, whose only possible affect is to prevent people 
getting fresh copies of data?  Why change something if it isn't 
significant?  Who really does this, and who really does use weak etags?

Do browsers warn when they choose not to get an up-to-date copy of 
something because of a weak etag?  Does the user ever find out that they 
didn't get the latest copy because someone decided that even though it 
changed, it wasn't significant?  Significance is a subjective concept, 
so it's impossible to reliably unilaterally decide whether a change 
truly is or is not significant. The only potential benefit is to the 
server operator in potentially reducing load, trading off against the 
information they provide being fresh or not.  I think most businesses 
would rather have their clients have the information they wish to 
provide them.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 30 Apr 2008 23:46:33 -0700

Pablo Castro wrote:
> 
> Putting the caching issue aside for a second

Once you add the caching issue back in, the use of weak ETag equivalence for the types of things you describe below doesn’t make sense. When multiple representations share a common weak ETag, that means there is no useful difference between them, so it doesn't matter which representation the client has, and it doesn't matter which representation is returned to the client. In the situation you describe below, the client is likely to be very interested in those secondary bits of information, even if it doesn't intend to (or can't) update them.

> think about the 
> case of side-effecting operations. You have some version of a 
> resource you got from a previous GET. Now you want to update 
> it only if the version on the server is consistent with the 
> version you originally got (arguably you made your 
> modification decisions based on that state). At that point 
> you'd issue a PUT or DELETE with an if-match header 
> containing the value you got in the ETag. If an aspect of the 
> resource that does not affect its semantics (as defined by 
> the author of the service) has changed, a weak ETag would 
> allow the PUT to go through successfully even if the resource 
> wasn't bit-by-bit identical.

Agreed.

> On the Astoria side of things we ran into this often. When 
> people expose resources made of data coming from databases, 
> not all the columns in a record matter (some are redundant, 
> some are de-normalized data just carried around, some are 
> non-comparable such as blobs or xml cells). You still want 
> concurrency checks on the rest in those cases.

The data is that is redundant, denormalized, or difficult to compare is still semantically significant; otherwise, why would the service return it in the first place? All of that data needs to be taken into consideration when generating the ETag, whether it is weak or strong. 

It sounds like you are trying to solve the "partial update" problem that is often discussed on rest-discuss and elsewhere. IMO, the best solution to that for the situation you describe is to give the updatable part of the resource its own URI with its own separate ETag(s) so you can PUT and DELETE at that URI using normal HTTP semantics. After all, the client and the server already need to have a shared understanding of what is updatable and what isn't. That would make your use of PUT and DELETE fit in very well with commonly accepted practice. Doing otherwise seems likely to result in Astoria making it way too easy to create services that simply don't work (especially in conjunction with caching proxies).

Cheers,
Brian