Subject: Re: NEW ISSUE: example for matching functions, was: Weak and strong ETags

From: Julian Reschke <[email-masked]>
Date: Mon, 28 May 2007 13:36:40 +0200

Hi,

I'm currently going through 
<http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/draft-lafon-rfc2616bis-latest.html>, 
checking for changes that currenly do not appear in the issues list at 
<http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/>.

This is one of these... (in context: 
<http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/draft-lafon-rfc2616bis-latest.html#rfc.section.13.3.3>):

Julian Reschke wrote:
> 
> Hi,
> 
> given the fact that more than a few persons were confused about the weak 
> matching function, I'd propose to add an example here, to appear below 
> the definitions in 13.3.3:
> 
>     The example below shows the results for a set of entity tag pairs,
>     and both the weak and strong comparison function results:
> 
>          +--------+--------+-------------------+-----------------+
>          | ETag 1 | ETag 2 | Strong Comparison | Weak Comparison |
>          +--------+--------+-------------------+-----------------+
>          | W/"1"  | W/"1"  | no match          | match           |
>          |        |        |                   |                 |
>          | W/"1"  | W/"2"  | no match          | no match        |
>          |        |        |                   |                 |
>          | W/"1"  | "1"    | no match          | match           |
>          |        |        |                   |                 |
>          | "1"    | "1"    | match             | match           |
>          +--------+--------+-------------------+-----------------+
> 
> Best regards, Julian

(a) Do we have agreement that this example is correct?

(b) Is there consensus to have it included?

Best regards, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 28 May 2007 19:36:17 +0100

> (a) Do we have agreement that this example is correct?

I'm surprised, as I'd expected that particular combination to be a
non-match for the weak comparison.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Mon, 28 May 2007 21:16:40 +0200

> non-match for the weak comparison.

So was I some time ago, and that's why I think we need an example.

<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.13.3.3> says:

"The only function that the HTTP/1.1 protocol defines on validators is 
comparison. There are two validator comparison functions, depending on 
whether the comparison context allows the use of weak validators or not:

     * The strong comparison function: in order to be considered equal, 
both validators MUST be identical in every way, and both MUST NOT be weak.
     * The weak comparison function: in order to be considered equal, 
both validators MUST be identical in every way, but either or both of 
them MAY be tagged as "weak" without affecting the result.

An entity tag is strong unless it is explicitly tagged as weak. Section 
3.11 gives the syntax for entity tags."

Earlier it says:

"Entity tags are normally "strong validators," but the protocol provides 
a mechanism to tag an entity tag as "weak." One can think of a strong 
validator as one that changes whenever the bits of an entity changes, 
while a weak value changes whenever the meaning of an entity changes. 
Alternatively, one can think of a strong validator as part of an 
identifier for a specific entity, while a weak validator is part of an 
identifier for a set of semantically equivalent entities."

So it seems that the relation between entity tags and validators could 
be clearer...

Best regards, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 28 May 2007 21:17:50 +0200

> (a) Do we have agreement that this example is correct?

Yes, imho it is what the RFC says. But see below.

> (b) Is there consensus to have it included?

Not sure, might loose the context somewhat as it's not only about ETag
but also Last-Modified which also has strong/weak properties.

The language wrt the weak compare function isn't really very complex.

      - The weak comparison function: in order to be considered equal,
        both validators MUST be identical in every way, but either or
        both of them MAY be tagged as "weak" without affecting the
        result.

But there is the small questionmark on if this is what was intended for
ETag, or if the optional weakness in the context was only intended for
weak Last-Modified values.. (less than 1 minute before Date).

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Mon, 28 May 2007 21:40:04 +0200

> but also Last-Modified which also has strong/weak properties.

Another example?

> weak Last-Modified values.. (less than 1 minute before Date).

Ah, and that's probably why people are surprised.

I guess we should identify this as a separate issue.

Best regards, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 28 May 2007 22:45:14 +0200

> > but also Last-Modified which also has strong/weak properties.

A bit hard to make an intelligible weak/strong compare example for
Last-Modified in such table. you'll need full Last-Modified + Date
headers, and a bit of explanation to make people actually see whats
imporant about the difference between them..

> Ah, and that's probably why people are surprised.

Could be, but I have my doubts.. More likely they assume what the weak
comparison function means without actually reading 13.3.3 at all.
	
Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 29 May 2007 14:51:22 +0200

> comparison function means without actually reading 13.3.3 at all.

OK, I re-read section 13.3, and I still think (agreeing with you) that 
the weak comparison of "1" and W/"1" yields true. At least this seems to 
be the only interpretation of 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.13.3.3.p.7> 
that makes sense:

"...The weak comparison function: in order to be considered equal, both 
validators MUST be identical in every way, but either or both of them 
MAY be tagged as "weak" without affecting the result...."

...as "tagging as weak" is only defined for ETags. It seems to me that 
what we need to clarify is how to extract the validator from a given 
etag. Looking at 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.11>:

    entity-tag = [ weak ] opaque-tag
    weak       = "W/"
    opaque-tag = quoted-string

...the validator would be the "opaque-tag" component.

Best regards, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 29 May 2007 15:22:56 +0100

> comparison function means without actually reading 13.3.3 at all.

:-)

On this topic, if there are some clients (or proxy caches) who require
byte-exact revalidations - i.e. strong comparison - and some clients
who are fine with semantically equivalent revalidations - i.e. weak
comparison - is that currently possible?

Is that done by the server sending a weak Etag in a response, and
giving clients the option to send a weak or strong version of that
Etag in If-None-Match later?

If so, do deployed clients actually do that?  I.e. those requiring a
strong comparison, do they strip away the W/ if it was present in an
earlier response's Etag?

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 29 May 2007 19:22:53 +0200

> <http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.11>:

The language "tagging as weak" is only defined for ETag, but it's for
the compare function equally valid for Last-Modified. But on the other
hand it's mostly the clients responsibility to judge if a Last-Modified
is weak or not.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 29 May 2007 19:50:53 +0200

> comparison - is that currently possible?

I expect most to not consider W/"1" and "1" as equal. Also consider it a
corner case which generally do not make any difference for any practical
operation of the protocol. The weak comparison function is only allowed
in simple cache validations.

I also expect many servers and proxies to not care very much about
weak/strong, just following the instructions by the client.

The most common example of where the same ETag exists in both strong and
weak is Apache, which uses weak ETags for objects modified very recently
(Last-Modified == Date), and upgrades the ETag to a strong one with the
same value when a bit of time has passed.

> Etag in If-None-Match later?

Clients can not upgrade an weak ETag to a strong. Only the server can do
that. Doesn't make sense for a client to do this.

> strong comparison, do they strip away the W/ if it was present in an

Any one requiring strong compare must consider any comparison where
either side is weak as false or invalid depending on the context (i.e.
servers seeing If-Range using and weak etag should consider the request
invalid).

Clients constructing conditionals which need to be strong MUST only do
so using strong validators. If the client do not have a strong validator
and require one then it MUST NOT create the conditional.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 29 May 2007 21:01:59 +0100

> that. Doesn't make sense for a client to do this.

Well, clients can compare a weak Etag with a strong one and, according
to this thread, that sometimes returns true.  That means there's a
relationship between weak and strong Etags with the same value except
the W/.

So why not permit clients to convert a weak one to a strong one in
If-None-Match?  (For example, clients doing byte-range requests need a
strong validator every time.  It would be useful for that.)

> and require one then it MUST NOT create the conditional.

But what about the other way around?  Clients which require only weak
comparison, but the server must send a strong Etag so that _other_
clients, which require a strong comparison for the same resource, can
use conditionals?

It seems that it would be useful if a server could specify two Etag
for the same resource - a weak one, and a strong one.

That way, clients needing only semantic equivalence could use the weak
one, and clients needing byte equivalence could use the strong one.

Currently, a server can support conditionals for one type of client
but not the other.  That seems to be an unnecessary limitation.

Either permitted clients to convert a weak Etag to a strong one in
If-None-Match, or permitting multiple Etags in a response, would allow
conditionals by both types of clients.

(That would also improve cache hits with Apache's method - which uses
a weak Etag for resources with a recent time, then converts to a
strong Etag with the same value after the time has passed).

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 29 May 2007 22:27:09 +0200

> strong validator every time.  It would be useful for that.)

Because the validator isn't strong and as such do not guarantee the
entity will be binary equivalent, only semantically equivalent.

Range merging is only safe if there is a strong indication from the
server that the parts come from the same entity. For this reason
If-Range only supports strong ETag:s as it's only when the server has
given the response a strong ETag one can assume binary equivalence.

If-None-Match is the only conditional allowing weak ETag. This is used
for cache validations and Vary probing.

> for the same resource - a weak one, and a strong one.

Why? A strong one can always be used.

The only reasons to specify a weak ETag is if you can't or don't want to
fulfill the uniqueness requirements of a strong one.

can't: When binary equivalence can not be guaranteed between two
requests. For example dynamic compression with a embedded timestamp, or
when there is other "random" elements in each response.

don't want to: When each response is unique, but it's at the same time
not very important that the user always gets the most current. For
example a hit counter.

> but not the other.  That seems to be an unnecessary limitation.

???

Can you give some examples please. I just don't see what you are talking
about here..

> strong Etag with the same value after the time has passed).

Already the case, if Apache follows the RFC.. The initial response will
have a weak ETag, which is then used in If-None-Match and should compare
true even after it has been upgraded to a strong one.

But please note that the Apache use of weak etags here is a bit flawed.
It's not really a good application of weak etags. But it's better than
not sending an ETag at all..

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 29 May 2007 22:36:25 +0200

Henrik Nordstrom wrote:
> ...
> ...

Speaking of which, does Apache httpd indeed implement the weak 
comparison function with respect to this?

Best regards, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 29 May 2007 23:27:50 +0100

> true even after it has been upgraded to a strong one.

You're right.

(But beware: If the client sends the weak ETag in If-None-Match, it
can get the _wrong_ response if the resource has changed during that
small time window (i.e. in the same second)).

Clients which require an accurate response after the request is sent
must send a strong Etag, or none at all.

In this case, converting the weak one to a strong one in the client
doesn't help, as you point out.

> Why? A strong one can always be used.

Because for some clients, a strong one is excessive (weak is good
enough), while for other clients accessing the same resource, a strong
one is required.

> Can you give some examples please. I just don't see what you are talking

For example: if there's a client like, oh, a calender application that
may not depend on the exact form of XML.  A weak Etag is fine for that.

But if you want to fetch the response for some other application like
displaying the XML, and have a proxy cache which converts
unconditional requests to conditional ones, then that must use a
strong Etag, or none at all.

Also, if you have an application that fetches the XML in pieces using
byte-range conditional requests - or a proxy which does that - it must
use a strong Etag.

It's reasonable to want a server to support all those clients with
efficient caching.

If it serves a strong Etag, the calendar client (for which weak
comparison would be fine) will result in unnecessary network traffic
sometimes - when a weak match would be fine, but a strong match fails.

If it serves a weak Etag, then the other client applications which
require a strong comparison can't use it and will result in
unnecessary network traffic.

Either way, the limitation means redundant network traffic that can be
avoided if either (a) the server can send two Etags, or (b) the client
can convert a weak Etag to a strong one, when it requires a strong one
or none at all.

Note that allowing (b) would require the table which started this
thread to be changed: request's strong Etag + resource's weak Etag
would return false, while requests's weak Etag + resource's strong
Etag would return true.

> fulfill the uniqueness requirements of a strong one.

My point is that "if you want to fulfull the uniqueness requirements"
is a property of the client application (and also how the data will be
used), not just the server.

A server can legitimately have reason to support clients which require
strong validation, as well as clients for which weak validation is
good enough, for the same resource.

> not very important that the user always gets the most current. For

Even with a hit counter, some clients may benefit from a byte-range
conditional request on it (e.g. if it's a large image file, and the
client checks the first few x00 bytes before deciding whether to get
the rest).  The weak Etag isn't enough for them.  For other clients,
it will be enough.

See other examples above.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 30 May 2007 02:12:10 +0200

> sometimes - when a weak match would be fine, but a strong match fails.

Point taken.

But I suspect that in many of these cases Last-Modified is a sufficient
validator.

Extending HTTP to support more than one ETag is not possible. It's
pretty hardcoded at a single ETag per entity. Also this would make
handling of Vary even more complex than it is today..

> can convert a weak Etag to a strong one, when it requires a strong one

I still don't see how converting the ETag to a strong one would work in
the context, no matter what the compare table looks like. The weak etag
simply do not carry the information you need as it's possibly shared by
multiple entities and if you just convert it to a strong ETag by
removing the weakness flag then you still don't know what it being
compared..

Related to this I still maintain that the restrictions on weak etags is
too strict, and should be allowed in If-Match for most operations if not
all. But that's a separate issue from the comparison function.

Regards
Henrik