Subject: PROPOSAL: Weak Validator definition [i101]

From: Mark Nottingham <[email-masked]>
Date: Tue, 11 Mar 2008 16:03:15 +1100

The only proposal I've seen for this is to remove the notion of  
"semantic equivalence" and replace it with (roughly) "good enough,  
from the server's point of view".

--
Mark Nottingham     http://www.mnot.net/

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 13 Mar 2008 23:12:13 +0100

> from the server's point of view".

Isn't that what the specs already says?

The spec does not say "semantic equivalence". It says 'could be
substituted for each other with no significant change in semantics'. and
'A validator that does not always change when the resource changes is a
"weak validator."'.

Note: The full discussion regarding weak/strong is found in the "Weak
and Strong Validators" section, not the definition of ETag.

Regards
Henrik

---

From: Werner Baumann <[email-masked]>
Date: Fri, 14 Mar 2008 21:35:31 +0100

> and Strong Validators" section, not the definition of ETag.
The spec says in 13.3.3 Weak and Strong Validators:
"One can think of a strong validator as one that changes whenever the
  bits of an entity changes, while a weak value changes whenever the
  meaning of an entity changes. Alternatively, one can think of a strong
  validator as part of an identifier for a specific entity, while a weak
  validator is part of an identifier for a set of semantically
  equivalent entities."

Why do you think, it does not say "semantically equivalent entities"?

The spec is self-contradictory and the parts that state "semantic 
equivalence" should be removed.

Werner

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 14 Mar 2008 22:11:24 +0100

> equivalence" should be removed.

Agreed. All definitons of weak should be based on "SHOULD change when
the entity changes in a semantically significant way".

Regards
Henrik

---

From: Werner Baumann <[email-masked]>
Date: Sat, 15 Mar 2008 09:55:30 +0100

> the entity changes in a semantically significant way".

Using SHOULD only would turn a strong self-contradiction into a weak 
self-contradiction.
I still suggest to remove "semantic" from all validator related stuff 
and instead use wording as proposed by Mark Nottingham: "good enough,
from the server's point of view".

I originally raised this issue, because I believed in weak etags 
guaranteeing something like "semantic equivalence" and I was 
disappointed when it turned out to be an illusion. My experience is 
restricted to Apache and IIS. Weak etags created by this two servers are 
not related to semantics in any way. They are weak only because of the 
limited resolution of Last-Modified-Date.

Are there any implementations of weak validators, that refer to semantic 
equivalence?

Werner

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 15 Mar 2008 18:36:01 +0100

> Are there any implementations of weak validators, that refer to semantic 

Quite likely there is many examples out there in the wild along the
lines of the pageviews counter example found in the RFC, but semantic
equivalence fits within "good enough from the servers point of view".
But probably not as many as there should be as most people don't
understand the cache model of HTTP until it's too late to make their web
application fit..

Regards
Henrik

---

From: Werner Baumann <[email-masked]>
Date: Sat, 15 Mar 2008 22:40:55 +0100

> equivalence fits within "good enough from the servers point of view".

I am not sure whether this is meant by you the way I understand it: Do 
you mean "good enough from the servers point of view" is o.k., because 
it includes your understanding of semantic equivalence? Or is it meant 
the other way round?
To be pedantic about pageview counters (what I ain't): a change in the 
pageview count is a semantic change. But I agree, it would be a good 
example for weak etags, because it is not important. I, personally, 
would accept strong etags in this case (although this violates the spec).

> understand the cache model of HTTP until it's too late to make their web

To be honest: I would not know, how to get this working with Apache. 
Apache's default etag-handling can not deal with this kind of weak 
etags. You would have to override apache's etag-handling and do it all 
by your own. Quite a lot of effort for a toy like pageview counters.

Werner

---

From: Robert Siemer <[email-masked]>
Date: Sun, 16 Mar 2008 16:23:35 +0100

> Are there any implementations of weak validators, that refer to semantic 

There are. At least some of my CGI scripts use them. - I would not 
discard that many other CGIs do the same.

To see no useful weak etag implementations within the static file 
serving code among common servers does not surprise me at all. - How 
should they know about semantic equivalence?

I still don't know why this mecanism has to be an illusion. 

Robert

---

From: Werner Baumann <[email-masked]>
Date: Sun, 16 Mar 2008 22:35:33 +0100

> I still don't know why this mecanism has to be an illusion. 
I don't say, it *has to be* an illusion. I say it *is* an illusion, when 
confronted with current practice. And the spec is self-contradictory, 
because it contains two mutual exclusive definitions of weak etags.

You can resolve this to either side. But the only realistic way seems to 
be to adapt the spec to current practice.

Werner

---

From: Robert Siemer <[email-masked]>
Date: Mon, 17 Mar 2008 03:44:29 +0100

> be to adapt the spec to current practice.

Current practice is to deliver weak etags that never match later on. 
These are based on "weak last-modified" dates. I hope that this 
useless practice ("we always generate ETags") never makes it into the 
spec!

As clients will do the same independent on which side we pull, I don't 
see "semantic equivalence" already lost.

Robert

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 17 Mar 2008 17:37:20 +1100

Hmm. Doesn't resolving it on the side of keeping semantic equivalence  
beg the question of what semantic equivalence is -- and is there any  
way to define it except in a server-specific fashion?

---

From: Lisa Dusseault <[email-masked]>
Date: Mon, 17 Mar 2008 14:57:07 -0700

Mark said he'd only seen one proposal on this issue.  There are so  
many ways this could be resolved, I'm following the advice of my  
Systems Design professor who said "You have no idea if you have a  
good solution, if you have only thought of one solution."  How about  
these:

Strawman proposal "Document Apache": Weak ETags could be documented  
the way they are used today in Apache.  I believe this effectively  
delays a second PUT until a second has passed since the PUT that  
cause the weak ETag to be created; then the weak ETag is converted to  
a strong one that the client can consider as equivalent.  Be more  
clear about how "W/[tag]" ~= "[tag]" for any value of [tag] (that  
doesn't begin with W/).   The value of weak ETag therefore becomes  
one of conserving the server's resources by using a cheap way to  
calculate ETags, that happens to also push clients to delay changes  
that contain If-Match conditionals.

Strawman proposal "Read-Only": Let the server decide when to use weak  
ETags, but only on read-only resources (writable resources must offer  
strong ETags or none.)  Document the server's goal here is to allow  
caches to serve old resources if they are similar enough to the most- 
recent version.  Thus, the server's test here is not "Are the  
versions semantically equivalent", but "Should caches refetch."

Strawman proposal "Die die die": get rid of weak Etags.  Do this by  
making the W/ prefix simply part of the ETag.  Alternatively, do this  
deprecating: recommend clients to ask again or not use etags that  
begin with W/.

In any case, I find that the paragraph on modification time is  
inconsistent with any language about semantic equivalence; there's no  
relationship between whether a resource was modified in the last  
second and whether it's semantically equivalent.

Lisa

On Mar 16, 2008, at 11:37 PM, Mark Nottingham wrote:

>
> Hmm. Doesn't resolving it on the side of keeping semantic  
> equivalence beg the question of what semantic equivalence is -- and  
> is there any way to define it except in a server-specific fashion?
>
>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2008 09:44:07 +1100

This crossed my mind as well... Weak ETags are in use today, but can  
we find a situation where they're actually improving things, and  
getting interoperability?

On 18/03/2008, at 8:57 AM, Lisa Dusseault wrote:

> making the W/ prefix simply part of the ETag.  Alternatively, do  
> this deprecating: recommend clients to ask again or not use etags  
> that begin with W/.

---

From: Robert Siemer <[email-masked]>
Date: Tue, 18 Mar 2008 00:47:40 +0100

> getting interoperability?

What is so bad with my CGI example? If I change something in the script 
that will make it's output different but has no semantic importance, I 
use weak etags. - You could argument that it is not worth caching at 
all, but please, get rid of "same second as now"-last-modified headers 
first. And that way the whole weak validator story can go.

(I don't see any interop issues with browsers on weak etag resources.)

Robert

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2008 11:23:22 +1100

Are you seeing any clients who are using the weak etags in a way  
that's different from how they'd behave if you sent a strong etag?

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 17 Mar 2008 17:50:27 -0700

On Mar 17, 2008, at 2:57 PM, Lisa Dusseault wrote:
> cause the weak ETag to be created; then the weak ETag is converted  
> to a strong one that the client can consider as equivalent.  Be  
> more clear about how "W/[tag]" ~= "[tag]" for any value of [tag]  
> (that doesn't begin with W/).   The value of weak ETag therefore  
> becomes one of conserving the server's resources by using a cheap  
> way to calculate ETags, that happens to also push clients to delay  
> changes that contain If-Match conditionals.

Just to be clear, this is only true for Apache when the core file
handler is being used to process responses and the default ETag
generation is configured to use second-resolution last-modified
dates.  For example, Apache with Subversion on the back-end has
very different ETags.  The reason for making the second-resolution
entity-tag weak in this case is the possibility that two updates
made within the same second have the potential to confuse a later
Range request based on the first update.

Apache is supposed to be able to return 304 to a GET based on
weak comparison of entity-tags.  There is a bug report that says
we don't do that at the moment, but I haven't had time to figure
out if (or why) that is the case.

> Strawman proposal "Read-Only": Let the server decide when to use  
> weak ETags, but only on read-only resources (writable resources  
> must offer strong ETags or none.)  Document the server's goal here  
> is to allow caches to serve old resources if they are similar  
> enough to the most-recent version.  Thus, the server's test here is  
> not "Are the versions semantically equivalent", but "Should caches  
> refetch."

I don't believe that the server is capable of knowing such a thing.
The vast majority of resources that are "read-only" are, in fact,
writable behind the interface, and it is primarily those writes
that are at risk of the twice-in-one-second updates.  In any case,
it is an implementation detail -- HTTP has no business limiting the
implementation behind the interface.

> this deprecating: recommend clients to ask again or not use etags  

Old servers will exist for a long time.

> inconsistent with any language about semantic equivalence; there's  
> no relationship between whether a resource was modified in the last  

Agreed, I see no reason to say anything about semantics here.
It should just be defined in terms of what the server is saying
and let the client adjust its behavior as it deems appropriate.

....Roy

---

From: Robert Siemer <[email-masked]>
Date: Tue, 18 Mar 2008 12:22:42 +0100

> that's different from how they'd behave if you sent a strong etag?

Only for full-body conditional GETs the answer is "no".

I think the big questions is: Do we want to cache documents that just 
changed when we asked for them AND could have changed once again in that 
second?

Apaches weak etags would provoke that, which for me makes no sense. - 
The server obviously does not care if some client stays with old and 
possibly completely "wrong" data. This is what we have now and we could 
judge:

a) that's fine, the server sends weak etags whenever it likes
b) that's not fine, weak etags stand for "semantic equivalence"
   -it is still the servers task to rule something equivalent
   -but "semantic equivalence" is definitively content based, not 
    modification time based

I personally don't like careless server behaviour like a); a 
document that changes "so often" is in my view probably out of date. 
Servers should not generate an ETag in this situation if they have no 
better means.

Robert

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 14:27:38 +0100

> it includes your understanding of semantic equivalence? Or is it meant 

Basically that I am ok with the change.

> would accept strong etags in this case (although this violates the spec).

strong ETags MUST NOT be used in that case as the object isn't binary
equivalent. A property of strong ETags is that it's possible to merge
ranges of the object received in different responses, and this would
result in garbage if the same ETag is shared among two slightly
different objects (except in very rare cases).

> etags.

Apaches default etag handling can note deal with weak etags. But any
CGI/PHP script or similar you use for generating content can do so just
fine.

> You would have to override apache's etag-handling and do it all 

Change the example to "visitor counter" and it may make more sense.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 15:12:25 +0100

> I still don't know why this mecanism has to be an illusion. 

It's an illusion only because the meaning of rough semantic equivalence
(or no significant change in semantics) isn't defined in technical terms
and means different things to different people, and can not be enforced
by the protocol. 

But this does not make weak validators a useless feature by any means.
It's a very interesting aspect of HTTP. Just means we need to get the
language cleaned up so people do not get so confused on what the specs
really means.

In the spec conditions using weak etags is placed pretty much equal to
conditions based on Last-Modified which has no semantic guarantee at
all, even if one MAY deduce some strength from modification time when
some time has passed.

Weak validators (and weak etags in specific) is a fuzzy feature of the
protocol which can be used for many interesting applications. But the
only "semantic equivalence" that can really be counted on is that a
strong ETag guarantees equality down to the octet, and that a weak etag
(or a condition based on only Last-Modified) signals that the object
most likely have not changed meaning in any significant manner.

The problem with "semantic equivalence" is a language thing, defining
"semantic equivalence". What the spec means to say is basically this
(from "Entity tags")

   A "weak entity tag," indicated by the "W/" prefix, MAY be shared by
   two entities of a resource only if the entities are equivalent and
   could be substituted for each other with no significant change in
   semantics. A weak entity tag can only be used for weak comparison.

or this from "Weak and strong validators"

   However, there might be cases when a server prefers to change the
   validator only on semantically significant changes, and not when
   insignificant aspects of the entity change. A validator that does not
   always change when the resource changes is a "weak validator."

not strict "semantic equivalence" as it mistakenly says in other
sections or paragraps.

What has got people upset about weak etags is the (in my opinion) valid
assumption taken by Apache and a few other implementations that if an
object changes twice in the same second it's most likely not a
significant change in semantics. Which isn't a bad assumption, but also
one which can not be guaranteed to 100%, and because it can not be
guaranteed it must not be used, right? (sure... never heard of a weak
condition which can not be guaranteed have you..)

The world is full of such things which fall into the "most likely"
category, and if we did not make use of such assumptions when it makes
sense then most things would take a grinding halt, not only HTTP..

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 15:50:41 +0100

> useless practice ("we always generate ETags") never makes it into the 

It's imho not a bad practice. Returning a reasonable weak ETag is better
than just Last-Modified. In Apache even the weak ETag is very likely to
change on subsecond modification (change in size or inode). The weak
ETag stays unchanged on modification only if the object is overwritten
in-place with no change in size. (assuming the default configuration for
ETag generation, administrators may relax this to just "no change in
modification time or size")

You should not read much more into a weak etag than that it's a weak
validator, in which category "modification time" is included. Both means
'is likely "good enough" to be equivalent'. Not 'is "good enough" to be
equivalent'. A weak validator signals "likely to be considered equal by
the recipient". A weak etag probably (but not guaranteed to) signals
this stronger than just last-modified, but still is just "likely to be
considered equal".

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 19:51:04 +0100

> getting interoperability?

There is no real interoperability problem with weak etags today, other
than some servers not implementing weak matches in their default
filesystem based conditional processing.

Clients behave well, and servers using (not only sending) weak etags can
make good use of them.

The only real problem is this discussion, of which maybe 80% is fud from
not understanding how servers use weak etags today and people drawing
far going conclusions from just a small part of the picture, 10%
misunderstandings of what weak etags may be used for, 5% total confusion
and 5% actual discussion moving forward.

Personally I wish we could extend weak conditionals to be allowed in
If-Match processing for most methods which would make the protocol more
complete for the situations where one need (or wants) to use weak etags,
but thats outside our scope.

For whats within our scope the existing definition of weak etags and
weak conditionals is very sane, if one ignores the confusion about what
is actually meant by "semantic equivalent" which is a product of the
language used in the descriptive text outside of the actual definitions
combined with those 10% who wants to use etags for more than they are
(part of that also visible in the meaning of ETag in response to PUT
discussion).

Getting rid of the "semantic equivalence" mistakes, sticking to "no
significant semantic difference in point of view of the server" whould
help a lot in keeping people on track on what a weak etag is and meant
to be used for. Together with a reminder that the only 2616 operation
allowing a weak condition is GET If-None-Match, which for most uses of
weak etag is quite sufficient, but falls short for extensions like
WebDAV or CalDAV who have additional needs (CalDAV in particular as it's
application scope makes strong ETags quite unsuited, with a lot of not
fully deterministic server-side processing going on affecting the octet
equivalence)

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 20:06:49 +0100

> a) that's fine, the server sends weak etags whenever it likes

In the Apache case I say it's fine. The weal ETag returned by Apache is
built from a number of sources in fact making it a quite strong etag.
The only reason why it's weak is because it is possible there is a local
process outside of Apache which modifies the file in such manner that
stat() of the file would not notice any difference (no change in size,
modification time or inode number).

The likelyhood that you actaully see two semantically significant
objects with the same weak ETag from Apache is very low in nearly all
applications, even if you have a lot of sub-second updates of the
object. And if that becomes a problem for you then a small change in
your server-side application to write to new files and rename instead of
overwriting parts of existing files would make the server behave well.

> Servers should not generate an ETag in this situation if they have no 

That weak ETag is a lot stronger conditional than Last-Modified on such
objects. Last-Modified would not notice nearly any of the sub-second
updates, while the weak ETag used by Apache detects almost all except
for a very specific case.

Worth noting in this discussion is that Apache has a similar cornercase
where it's strong ETags malfunctions, and thats if you run a backend
application which modifies parts of files without touching the last
modification date. Thats somewhat less likely than an application
rewriting parts of a file multiple time in a sub-second but both is
quite unlikely in nearly all real life uses.

The only problem I have with how Apache uses weak etags is that it can
not match then in a subsequent If-None-Match, not even within the same
second. But that's a minor implementation detail and not a protocol
violation (only sub-optimal behavior).

Regards
Henrik

---

From: Robert Siemer <[email-masked]>
Date: Tue, 18 Mar 2008 22:39:06 +0100

Nothing comes to a "grinding halt" if we don't cache these responses. 
You say the likeliness is high that a same second change is small, I say 
the likeliness is high that the change is significant.

If weak etags mean "it was the only feasible way to generate one" then I 
would advocate to stop using them client side. They just degrade cache 
transparency with no real benefit.

If I hit five times the "reload" button of my browser, what should 
happen? Should there be space for the possibility to see the wrong page 
because the browser used a weak etag in a conditional request?

Robert

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 18 Mar 2008 23:00:24 +0100

> Nothing comes to a "grinding halt" if we don't cache these responses. 

I didn't say that. I said that the wold will come to a grinding halt if
everything taking place needs to be 100% guaranteed. For most things
it's good enough if it's within reasonable doubt.

> the likeliness is high that the change is significant.

I say that the assumption that it's very unlikely that there is a
significant in-place edit (overwrite, not replace) change within the
same second and which do not change the filesize is for most use cases a
perfectly valid assumtion.

I do not say that this is true in every possible imagineable situation,
but there is a lot of things that is not true in every situation. And
for that reason there is knobs to tune these details in the web server.

> transparency with no real benefit.

I don't agree at all with this.

> because the browser used a weak etag in a conditional request?

Sure, just as will happen if just Last-Modified was sent. But with the
weak etag this corner case is a lot smaller. See below.

In this specific case the weak etag do improve things a lot compared to
just Last-Modified as the weak etag is built from more sources than just
last modification time. Based on the Apache algorithm (and assuming
Apache gets fixed to process the conditional with a positive outcome)
the weak etag in If-None-Match detects changes in the following
conditions where If-Modified-Since does not:

- Some text was added or removed in the same second. (different file
size)

- File was completely replaced by a different file (different inode from
write -> rename).

- An older backup of the file was restored, including the older
modification time (different modification time in the past).

And in addition to this the weak etags is very useful in cases where the
server actually knows the semantics.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Mar 2008 22:22:35 +1100

I haven't seen any support for Lisa's straw-man proposals, and  
discussion seems to confirm the below as the direction we want to take  
this. Closing this one (although there may be other issues nearby...).