Subject: Re: ETags and concurrency control

From: Werner Baumann <[email-masked]>
Date: Fri, 02 May 2008 19:37:47 +0200

I want to remind of how Apache and IIS use weak etags:

They create weak etags when the time of the request is within the same 
second as the last modified date of the resource. This weak etags simple 
mean:
- if the etag matches, the entity is most probably unchanged, but there 
is a small chance, that it changed.
- if the entity changed, it may have changed into anything; there is 
absolutely no check for anything like semantic equivalence.

 From this common use of weak etags, it makes sense to not allow weak 
etags in PUT (lost update problem), but to allow it in full body GET (in 
case the entity changed, the damage is usually small).

The spec is inconsistent in mixing up this kind of "unreliable" etag 
with semantic equivalence. The issue may be resolved to either side. But 
of course, associating "semantic equivalence" with weak etags will 
render most servers none compliant.

Werner

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 02 May 2008 20:40:18 +0200

> case the entity changed, the damage is usually small).

Wait a minute here. The "weak" ETags generated by Apache isn't that
weak. For Apache in default configuration to generate the same weak ETag
for two different versions the following need to all be true

- The update was done in-place by overwriting parts of the file,
preserving the same inode number.
- The update MUST be within the same sub-second as the previous update
- The update MUST NOT change the file size.
- The inode change timestamp must also not change by the update.

This can practically only happen if there is other processes updating
the file content directly outside Apache.

The reason why Apache sends weak ETag on content modified in the last
second is because the default configuration assumes there will be other
processes running on the server "randomly" overwrite parts of published
files many times within the same second.

For normal HTTP use where updates is done using PUT, or nearly all
normal edits the above isn't true and Apache may just as well send a
strong ETag without any loss of guarantee.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 May 2008 20:52:26 +0200

Werner Baumann wrote:
> ...

I'd say that it's the server's choice to generate these kinds of ETags. 
Nut if it does, and they aren't reliable enough for a write operation, 
it could still reject the request... My understanding was that we want 
*allow* servers to support weak etags on write operations, not require.

> ...

BR, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 2 May 2008 19:55:40 +0100

> preserving the same inode number.

Or the inode is recycled when the old file is deleted and a new one
created.

> - The update MUST be within the same sub-second as the previous update

Or the clock moves backwards due to a correction, or someone writes a
similar file using a timestamp-preserving copy (like cp -p, rsync -t).

NB: Both of these break strong Etags for updates _not_ in the last
second too - Apache's algorithm is not watertight.

> - The update MUST NOT change the file size.

Quite common, when editing a file of the same name.

> - The inode change timestamp must also not change by the update.

If the modification time is in the same second, you can be quite
confident the change timestamp will be in the same second too.

> the file content directly outside Apache.

That's probably the most common way files in Apache are updated.

> files many times within the same second.

Or non-randomly.  Every way I've seen files updated and published
through Apache other than WebDAV (FTP, rsync, scp) can trigger these
problems occasionally.

Clearly, it does screw up range requests.

But also: after doing an update, then you run a client to GET the
file, perhaps to verify it's serving the right content, you expect to
get the file you have just updated.  If weak Etags are used in
caching, this is not guaranteed any more.

> strong ETag without any loss of guarantee.

That's right.  I believe it can be configured to do so, if you can
confirm all these guarantees.

Dynamically generated content from databases, blogs, wikis etc. can
also use strong Etags in the same way for precise cache validation, if
you can confirm the tags precisely.  So can backends serving ordinary
files modified by other processes, if you can use something like
Linux's F_LEASE or inotify to be informed of file changes
synchronously.  But these aren't the simplest of configurations.

-- Jamie

---

From: Werner Baumann <[email-masked]>
Date: Fri, 02 May 2008 21:31:40 +0200

> *allow* servers to support weak etags on write operations, not require.
Take the case of Apache-style weak etags (based on insufficient time 
resolution).
If the server cannot know whether the the entity changed since the etag 
was generated, it will always have to reject the request. Allowing weak 
etags in these requests will only result in an additional round trip.
If the server can know whether the entity changed since the etag was 
generated, there is no need for the etag to be weak, it *is* strong.

Things are quite different, when you think of weak etags meaning not 
byte-by-byte, but semantically equal. But there is the whole mess. These 
are two completely different and unrelated concepts.
- insecure (or unreliable): there is a small chance, that the entity 
changed without changing the etag, and the change may be completely 
arbitrary.
- semantically equivalent, though not byte-by-byte equal.

As long as you try to get this two concepts into one definition of weak 
etag, it will stay a confusing mess.

Werner

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 2 May 2008 20:37:55 +0100

> etag, it will stay a confusing mess.

Plus, when you serve a "semantically equivalent" weak ETag, you can't
serve a strong ETag at the same time.

This is a silly limitation: supporting weak comparison (whatever it
means) shouldn't prevent strong caching from being possible!

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 May 2008 17:09:41 +1000

Yes. I tend to think that weakness doesn't help here; there's no  
guarantee about the nature of the change, and if the server is that  
unsure about the actual content, it shouldn't be sending an ETag at  
all (or, more likely, allow administrators to configure whether or not  
ETags are sent, with a sensible default, depending on how they use the  
FS).

Adding weakness to the mix doesn't provide any benefit (as many have  
pointed out), and just muddies the concept.

On 03/05/2008, at 5:31 AM, Werner Baumann wrote:
>
> Julian Reschke wrote:
>> I'd say that it's the server's choice to generate these kinds of  
>> ETags. Nut if it does, and they aren't reliable enough for a write  
>> operation, it could still reject the request... My understanding  
>> was that we want *allow* servers to support weak etags on write  
>> operations, not require.
> If the server cannot know whether the the entity changed since the  
> etag was generated, it will always have to reject the request.  
> Allowing weak etags in these requests will only result in an  
> additional round trip.
> byte-by-byte, but semantically equal. But there is the whole mess.  
> These are two completely different and unrelated concepts.
> As long as you try to get this two concepts into one definition of  
> weak etag, it will stay a confusing mess.
>
> Werner
>

--
Mark Nottingham     http://www.mnot.net/