Subject: Our old friends, weak ETags

From: Mark Nottingham <[email-masked]>
Date: Fri, 20 Jul 2012 13:25:53 +1000

I just received an interesting bug report on REDbot; <https://github.com/mnot/redbot/issues/109>

"""
When an ETag is marked weak with "W/", it need not change across different content encodings. "Weak" means the entity is semantically equivalent but not bit-equivalent. But Redbot complains that it doesn't change with different content encodings.
"""

Looking into this, a couple of things pop up:

1) The obvious question, whether two different *negotiated* representations of the same resource can (or should) have the same weak ETag. Our current definition is this:

"""
In contrast, a "weak validator" is a representation metadata value that might not be changed for every change to the representation data. This weakness might be due to limitations in how the value is calculated, such as clock resolution or an inability to ensure uniqueness for all possible representations of the resource, or due to a desire by the resource owner to group representations by some self-determined set of equivalency rather than unique sequences of data. An origin server should change a weak entity-tag whenever it considers prior representations to be unacceptable as a substitute for the current representation. In other words, a weak entity-tag ought to change whenever the origin server wants caches to invalidate old responses.
""" <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p4-conditional.html#weak.and.strong.validators>

Strictly speaking, I don't think this is a problem for caches; following the rules for reusing a stored response <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p6-cache.html#constructing.responses.from.caches>, a cache doesn't use the ETag to select a representation (what we used to call a variant) from a pool of many. 

Having said that, I'm still a bit uneasy. An ETag is supposed to be scoped to an entire resource, not just the selected representation. Because the response is negotiated, I'm tempted to argue that a compressed response is NOT semantically equivalent to an uncompressed one, just as a French response isn't equivalent to an English one, because the client has stated they don't understand English.

Thoughts? Is it worth clarifying this, or is it acceptable to have two different negotiated representations of the same resource share a weak ETag?

2) In the definitions of If-Match and If-None-Match, we don't specify whether the weak or strong comparison function is to be used when these validations actually occur, although we spend a lot of text on when to use weak vs. strong ETags themselves. 

Now, you might say that an origin server can decide whether to use the weak or strong function, but an intermediary or client cache doesn't have license to do weak comparison, and could cause a lot of trouble if it did. AFAICT we don't specify this, but I think we should.

Again, thoughts?

Thanks,

--
Mark Nottingham   http://www.mnot.net/

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 20 Jul 2012 04:30:46 +0000

for us (shared cache), weak ETags are nothing more than a PITA.

But we don't index on them (not alone), and so I think it would be no 
problem for us if they were used amongst different representations, as 
long as the server had properly previously set vary.

But I'd be interested to know

a) does anyone actually use them?
b) do they work?
c) do we still need them?

Regards

Adrien

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 20 Jul 2012 07:48:41 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

>Strictly speaking, I don't think this is a problem for caches; following 
>the rules for reusing a stored response 

I can confirm:  We have had a heck of a time to find out how/what to
do about eTags when Varnish gzip's objects to save bandwidth.

I think the poster-boy example we came up with was this:

Request:
	(routed directly to server)
	GET /bla, range=0-10kbyte, A-E=gzip
Response:
	10 kbyte of uncompressed bla, E-tag=foo

Request:
	(routed to cache, which holds gzip'ed copy of object)
	GET /bla, range=10-20kbyte, A-E=gzip, If-Match=foo
Response:
	(fetches object from server (E-tag=foo))
	(gzips to save bandwidth)
	10 kbyte of gzip'ed bla, [...]

It is our interpretation that current semantics demand that the
cache modify the ET when it does the gzip'ing of the object.

But it is not at all obvious how it can safely do so, since that
ET can end up back on the origin server if the third request
gets routed directly to the server.

Absent knowledge about how the server produces ETs, the
cache as no way of knowing how to produce a non-colliding one.

My conclusion was that ETs were specified wrong with respect to
range and gzip:  ETs should always refer to the complete underlying
objects bits in raw form, and not be affected by range or compression.

If that was the case, the client would not send A-E=gzip on
the second request, having gotten uncompressed bits back on
the first and there would be no issue.

I'd love to be told there are a simpler solution...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 24 Jul 2012 00:39:40 -0600

> a) does anyone actually use them?

Yes.

> b) do they work?

I've never heard that they don't, nor run into any problems using them.

> c) do we still need them?

Yes, otherwise I have to cache-expire for all changes to resource state,
without the option to declare certain changes insignificant.  Take a
"weblog entry" resource which includes a # of comments counter, with a
link to a comment thread.  XHR is used to synch the count, eventually.

I only change the Etag if the entry itself is edited.  Since metadata
about the entry may change in the meantime, which isn't significant
enough to warrant a new Etag in my caching scheme, I preface with W/ to
indicate variance from the origin server not requiring cache expiration,
i.e. semantic equivalence.

-Eric

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 24 Jul 2012 19:11:10 +1200

> link to a comment thread.  XHR is used to synch the count, eventually.

What type of server do you have which does not send strong ETag on 
binary-level changes anyway?

If a cache called in asking for updates with one of your weak-ETag would 
your server send a new copy or a 304 even if some changes had taken place?
    If its going to emit a new copy why bother sending W/ in the first 
place? a new Etag can goes out and the cache is refreshed with that 
immediately.
    If its not going to emit a new copy on request, why not just send 
the old strong ETag? are you emitting 304 with a different weak-ETag?

  ... Range requests that make it to the origin make things weird, but 
they can be special-cased to either change the ETag to something more 
current before handling, or to use an old copy from which the ETag was 
generated.

> i.e. semantic equivalence.

I assume you mean meta data inside the object itself (that counter, 
keywords list etc) ?

AYJ

---

From: Julian Reschke <[email-masked]>
Date: Tue, 24 Jul 2012 09:27:11 +0200

On 2012-07-24 09:11, Amos Jeffries wrote:
> ...

As far as I recall, Apache httpd/moddav sends a (timestamp-based) weak 
etag in PUT responses if it doesn't have sufficient time resolution. 
Once the change time for the underlying object is in the past, the ETag 
changes to a strong one.

Best regards, Julian

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 24 Jul 2012 11:05:25 -0600

Amos Jeffries wrote:
> Eric J. Bowman wrote:
> > Yes, otherwise I have to cache-expire for all changes to resource
> > state, without the option to declare certain changes
> > insignificant.  Take a "weblog entry" resource which includes a #
> > of comments counter, with a link to a comment thread.  XHR is used
> > to synch the count, eventually.
> 

My own.

>
> If a cache called in asking for updates with one of your weak-ETag
> would your server send a new copy or a 304 even if some changes had
> taken place?
>

If the changes were insignificant, then 304, because the origin server
is sending fresh content with the same W/Etag being used to validate.
The Etag is only changed for significant updates.

>
> If its going to emit a new copy why bother sending W/ in
> the first place? a new Etag can goes out and the cache is refreshed
> with that immediately.
>

Because I don't want to refresh the cache.

> the old strong ETag? are you emitting 304 with a different weak-ETag?

The Etag stays the same, which is why it has to be weak; otherwise I'd
be sending byte-for-byte-different representations with the same strong
Etag, which would be incorrect.

>
>   ... Range requests that make it to the origin make things weird,
> but they can be special-cased to either change the ETag to something
> more current before handling, or to use an old copy from which the
> ETag was generated.
> 

Or, the origin can be special-cased to ignore range requests with W/.
;-)

>
> > I only change the Etag if the entry itself is edited.  Since
> > metadata about the entry may change in the meantime, which isn't
> > significant enough to warrant a new Etag in my caching scheme, I
> > preface with W/ to indicate variance from the origin server not
> > requiring cache expiration, i.e. semantic equivalence.
> 

Yes; without W/ I'd have to break those out as their own resources,
adding more round trips.

-Eric