Subject: Re: #306: does etag value really use quoted-string

From: Julian Reschke <[email-masked]>
Date: Thu, 01 Sep 2011 19:44:03 +0200

On 2011-08-30 13:35, Mark Nottingham wrote:
> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/306>
>
> IIRC we checked various implementations and found that many don't handle ETags as a quoted-string; therefore, the safe/sensible thing to do is to re-define ETag as a something else; i.e., something with the same syntax, but without any special semantic for backslashes.
>
> We might also caution against including backslashes in them in prose, since it isn't interoperable.
>
> Thoughts / objections?

Sounds right to me.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 01 Sep 2011 19:56:15 +0200

On 2011-08-30 13:35, Mark Nottingham wrote:
> ...
> ...

Right now, the ABNF also allows

- OWS (optional whitespace)
- obs-text (octets >= 128)

Do we need to keep those in the grammar?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 01 Sep 2011 20:59:48 +0200

On 2011-08-30 13:35, Mark Nottingham wrote:
> --
> Mark Nottingham   http://www.mnot.net/

Proposed patch: 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/306/306.diff>

The definition would then read:

2.3.  ETag

    The ETag header field provides the current entity-tag for the
    selected representation.  An entity-tag is an opaque validator for
    differentiating between multiple representations of the same
    resource, regardless of whether those multiple representations are
    due to resource state changes over time, content negotiation
    resulting in multiple representations being valid at the same time,
    or both.  An entity-tag consists of an opaque quoted string, possibly
    prefixed by a weakness indicator.

      ETag       = entity-tag

      entity-tag = [ weak ] opaque-tag
      weak       = %x57.2F ; "W/", case-sensitive
      opaque-tag = DQUOTE *etagc DQUOTE
      etagc      = %x21 / %x23-7E / obs-text
                 ; VCHAR except double quotes, plus obs-text

       Note: Previously, opaque-tag was defined to be a quoted-string
       ([RFC2616], Section 3.11), thus some recipients might perform
       backslash unescaping.  Servers therefore ought to avoid backslash
       characters in entity tags.

       ...

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 2 Sep 2011 10:30:26 +1000

Looks right to me.

I'm a *little* concerned about how productions like this will be handled:

If-Match: "abc\", "def"

but that's a corner case...

On 02/09/2011, at 4:59 AM, Julian Reschke wrote:

> Proposed patch: <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/306/306.diff>
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Sep 2011 10:23:20 +0200

> but that's a corner case...

Maybe we need to do this differently. Leave the syntax definition alone, but

- explain that senders should not use escapes, and that they might be 
screwed if they need a DQUOTE in an etag (also to avoid OWS)

- explain that recipients should really really treat etags as opaque, 
and just keep and return what they got from senders

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 3 Sep 2011 10:36:09 +1000

On 02/09/2011, at 6:23 PM, Julian Reschke wrote:

> - explain that senders should not use escapes, and that they might be screwed if they need a DQUOTE in an etag (also to avoid OWS)
> 
> - explain that recipients should really really treat etags as opaque, and just keep and return what they got from senders

I'd prefer to have the BNF be honest, but maybe that's not possible. Give it a try?

Cheers,

---

From: Yves Lafon <[email-masked]>
Date: Mon, 5 Sep 2011 09:17:36 -0400 (EDT)

>                ; VCHAR except double quotes, plus obs-text

How about removing %x5C as well '\' and note that implementations were not 
handling \ as a quoted-string as it was defined before, leading to 
non-interoperable parsing of such ETags.

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Julian Reschke <[email-masked]>
Date: Mon, 05 Sep 2011 15:27:02 +0200

On 2011-09-05 15:17, Yves Lafon wrote:
> How about removing %x5C as well '\' and note that implementations were
> not handling \ as a quoted-string as it was defined before, leading to

The reason I didn't do it is that I really don't believe \ is 
non-interoperable in practice, as the clients really do not treat the 
(most?) etags as quoted strings but as opaque values; in which case 
there is no problem in practice...

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 06 Sep 2011 17:33:59 +1200

> there is no problem in practice...

I am also struggling to understand what the interoperability problem 
here is that can't be fixed with regular software compliance updates.

The broken pieces are already non-compliant with RFC 2616 quoted-string 
definitions. So no problem making anything existing suddenly 
non-compliant by keeping them. Quite the opposite, removing 
quoted-string makes all the existing un-escaping software non-compliant.
  I'm thinking Squid here where we have rolled out quoted-string escape 
handling into the recent stable releases of all popular distros to meet 
the RFC 2616 compliance checklist. This unescaping will be hitting the 
Enterprise sector in the next upgrade cycle.

There was an analysis posted of UA agents behaviour earlier, which is 
useful info but seems irrelevant to any visible problem. UA is handling 
only its own cache. Middleware which is opaque will be less efficient, 
but work. It is the servers alone which will have to handle a mix of 
correctly unescaped and opaque traffic inbound. Especially if escaping 
was originally done wrong by the itself or its mirrors (\Z \G etc).
  Worst case there seems to be inefficiency and excess server load on 
the broken machinery itself.

If I have missed something, perhapse one of you pushing for removal of 
quoted-string can explain for me?

AYJ

---

From: Julian Reschke <[email-masked]>
Date: Tue, 06 Sep 2011 09:36:06 +0200

> here is that can't be fixed with regular software compliance updates.

The problem is that regular software compliance updates do not happen 
everywhere.

> quoted-string makes all the existing un-escaping software non-compliant.

It would, but for an edge case we thought it's used.

Do you have different data? Is anybody *producing* Etags with backslashes?

> Enterprise sector in the next upgrade cycle.

That's interesting.

> but work. It is the servers alone which will have to handle a mix of

I think it'll break for If-Match -- clients might not be able to PUT to 
resource when guarding against lost updates.

> Worst case there seems to be inefficiency and excess server load on the
> broken machinery itself.
>

I'm not pushing for it, but there's clearly a problem right now in that 
what the spec says isn't implemented by most servers.

One way to address this would be to leave the definition alone, but to 
warn producers that if they use escapes they may be in trouble. (Another 
thing we need to consider is whitespace handling in quoted strings...)

Best regards, Julian

---

From: Karl Dubost <[email-masked]>
Date: Thu, 8 Sep 2011 15:40:59 +0200

> Do you have different data? Is anybody *producing* Etags with backslashes?

I made a quick survey on Alexa Top 500

39 sites have an "etag"
   36 ETag
    3 Etag

On these 39 sites,
    5 using ETag: W/"chars" 
    3 NOT using double quotes at all
    1 being ETag: GgZz"ba1e66504638a1a318ce1edb1c843c8a"

none using the char backslash "\"  

ps: It is a very quick dirty survey, I could do on a  lot more Web sites (around 1million sites) if interested, and with a proper way of doing it. (following redirects for example, using a GET instead of a HEAD, etc.)

-- 
Karl Dubost - http://dev.opera.com/
Developer Relations & Tools, Opera Software