Subject: #307 (untangle Cache-Control ABNF)

From: Julian Reschke <[email-masked]>
Date: Sun, 10 Jun 2012 21:52:16 +0200

This is a left-over TODO from previous ABNF discussions -- Cache-Control 
currently defines directives as part of it's value ABNF; which makes it 
look as if generic parsers need to special-case the predefined 
directives. Hopefully we all agree that parsers should be generic (as we 
suggest for new header fields in P2).

Thus I'd like to simplify the Cache-Control ABNF to the bare minimum, 
and then define the individual directives in separate subsections. 
Here's how this might look:

3.2.  Cache-Control

    The "Cache-Control" header field is used to specify directives for
    caches along the request/response chain.  Such cache directives are
    unidirectional in that the presence of a directive in a request does
    not imply that the same directive is to be given in the response.

    A cache MUST obey the requirements of the Cache-Control directives
    defined in this section.  See Section 3.2.3 for information about how
    Cache-Control directives defined elsewhere are handled.

       Note: HTTP/1.0 caches might not implement Cache-Control and might
       only implement Pragma: no-cache (see Section 3.4).

    A proxy, whether or not it implements a cache, MUST pass cache
    directives through in forwarded messages, regardless of their
    significance to that application, since the directives might be
    applicable to all recipients along the request/response chain.  It is
    not possible to target a directive to a specific cache.

    Cache directives are identified by a token, to be compared case-
    insensitively, and have an optional argument, that can use both token
    and quoted-string syntax.  Recipients MUST accept both forms.

      Cache-Control   = 1#cache-directive

      cache-directive = token [ "=" ( token / quoted-string ) ]

    For the cache directives defined below, no argument is defined (nor
    allowed) otherwise stated otherwise.

3.2.1.  Request Cache-Control Directives

3.2.1.1.  no-cache

    The no-cache request directive indicates that a cache MUST NOT use a
    stored response to satisfy the request without successful validation
    on the origin server.

3.2.1.2.  no-store

    The no-store request directive indicates that a cache MUST NOT store
    any part of either this request or any response to it.  This
    directive applies to both private and shared caches.  "MUST NOT
    store" in this context means that the cache MUST NOT intentionally
    store the information in non-volatile storage, and MUST make a best-
    effort attempt to remove the information from volatile storage as
    promptly as possible after forwarding it.

    This directive is NOT a reliable or sufficient mechanism for ensuring
    privacy.  In particular, malicious or compromised caches might not
    recognize or obey this directive, and communications networks might
    be vulnerable to eavesdropping.

    Note that if a request containing this directive is satisfied from a
    cache, the no-store request directive does not apply to the already
    stored response.

3.2.1.3.  max-age

    Argument syntax:

       delta-seconds (see Section 1.5)

    The max-age request directive indicates that the client is unwilling
    to accept a response whose age is greater than the specified number
    of seconds.  Unless the max-stale request directive is also present,
    the client is not willing to accept a stale response.

3.2.1.4.  max-stale

    Argument syntax:

       delta-seconds (see Section 1.5)

    The max-stale request directive indicates that the client is willing
    to accept a response that has exceeded its expiration time.  If max-
    stale is assigned a value, then the client is willing to accept a
    response that has exceeded its expiration time by no more than the
    specified number of seconds.  If no value is assigned to max-stale,
    then the client is willing to accept a stale response of any age.

3.2.1.5.  min-fresh

    Argument syntax:

       delta-seconds (see Section 1.5)

    The min-fresh request directive indicates that the client is willing
    to accept a response whose freshness lifetime is no less than its
    current age plus the specified time in seconds.  That is, the client
    wants a response that will still be fresh for at least the specified
    number of seconds.

3.2.1.6.  no-transform

    The no-transform request directive indicates that an intermediary
    (whether or not it implements a cache) MUST NOT change the Content-
    Encoding, Content-Range or Content-Type request header fields, nor
    the request representation.

3.2.1.7.  only-if-cached

    The only-if-cached request directive indicates that the client only
    wishes to obtain a stored response.  If it receives this directive, a
    cache SHOULD either respond using a stored response that is
    consistent with the other constraints of the request, or respond with
    a 504 (Gateway Timeout) status code.  If a group of caches is being
    operated as a unified system with good internal connectivity, a
    member cache MAY forward such a request within that group of caches.

3.2.2.  Response Cache-Control Directives

3.2.2.1.  public

    The public response directive indicates that a response whose
    associated request contains an 'Authentication' header MAY be stored
    (see Section 2.7).

3.2.2.2.  private

    Argument syntax:

       #field-name

    The private response directive indicates that the response message is
    intended for a single user and MUST NOT be stored by a shared cache.
    A private cache MAY store the response.

    If the private response directive specifies one or more field-names,
    this requirement is limited to the field-values associated with the
    listed response header fields.  That is, a shared cache MUST NOT
    store the specified field-names(s), whereas it MAY store the
    remainder of the response message.

    The field-names given are not limited to the set of standard header
    fields defined by this specification.  Field names are case-
    insensitive.

    Note: This usage of the word "private" only controls where the
    response can be stored; it cannot ensure the privacy of the message
    content.  Also, private response directives with field-names are
    often handled by implementations as if an unqualified private
    directive was received; i.e., the special handling for the qualified
    form is not widely implemented.

    Note: For compatibility with non-robust recipients, even a single-
    entry list of field names SHOULD be sent using the quoted-string
    syntax.

3.2.2.3.  no-cache

    Argument syntax:

       #field-name

    The no-cache response directive indicates that the response MUST NOT
    be used to satisfy a subsequent request without successful validation
    on the origin server.  This allows an origin server to prevent a
    cache from using it to satisfy a request without contacting it, even
    by caches that have been configured to return stale responses.

    If the no-cache response directive specifies one or more field-names,
    then a cache MAY use the response to satisfy a subsequent request,
    subject to any other restrictions on caching.  However, any header
    fields in the response that have the field-name(s) listed MUST NOT be
    sent in the response to a subsequent request without successful
    revalidation with the origin server.  This allows an origin server to
    prevent the re-use of certain header fields in a response, while
    still allowing caching of the rest of the response.

    The field-names given are not limited to the set of standard header
    fields defined by this specification.  Field names are case-
    insensitive.

    Note: Most HTTP/1.0 caches will not recognize or obey this directive.
    Also, no-cache response directives with field-names are often handled
    by implementations as if an unqualified no-cache directive was
    received; i.e., the special handling for the qualified form is not
    widely implemented.

    Note: For compatibility with non-robust recipients, even a single-
    entry list of field names SHOULD be sent using the quoted-string
    syntax.

3.2.2.4.  no-store

    The no-store response directive indicates that a cache MUST NOT store
    any part of either the immediate request or response.  This directive
    applies to both private and shared caches.  "MUST NOT store" in this
    context means that the cache MUST NOT intentionally store the
    information in non-volatile storage, and MUST make a best-effort
    attempt to remove the information from volatile storage as promptly
    as possible after forwarding it.

    This directive is NOT a reliable or sufficient mechanism for ensuring
    privacy.  In particular, malicious or compromised caches might not
    recognize or obey this directive, and communications networks might
    be vulnerable to eavesdropping.

3.2.2.5.  must-revalidate

    The must-revalidate response directive indicates that once it has
    become stale, a cache MUST NOT use the response to satisfy subsequent
    requests without successful validation on the origin server.

    The must-revalidate directive is necessary to support reliable
    operation for certain protocol features.  In all circumstances a
    cache MUST obey the must-revalidate directive; in particular, if a
    cache cannot reach the origin server for any reason, it MUST generate
    a 504 (Gateway Timeout) response.

    The must-revalidate directive ought to be used by servers if and only
    if failure to validate a request on the representation could result
    in incorrect operation, such as a silently unexecuted financial
    transaction.

3.2.2.6.  proxy-revalidate

    The proxy-revalidate response directive has the same meaning as the
    must-revalidate response directive, except that it does not apply to
    private caches.

3.2.2.7.  max-age

    Argument syntax:

       delta-seconds (see Section 1.5)

    The max-age response directive indicates that the response is to be
    considered stale after its age is greater than the specified number
    of seconds.

3.2.2.8.  s-maxage

    Argument syntax:

       delta-seconds (see Section 1.5)

    The s-maxage response directive indicates that, in shared caches, the
    maximum age specified by this directive overrides the maximum age
    specified by either the max-age directive or the Expires header
    field.  The s-maxage directive also implies the semantics of the
    proxy-revalidate response directive.

3.2.2.9.  no-transform

    The no-transform response directive indicates that an intermediary
    (regardless of whether it implements a cache) MUST NOT change the
    Content-Encoding, Content-Range or Content-Type response header
    fields, nor the response representation.

3.3 ...

Feedback appreciated, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 11 Jun 2012 22:36:22 +1000

+1; do you have a diff?

On 11/06/2012, at 5:52 AM, Julian Reschke wrote:

> This is a left-over TODO from previous ABNF discussions -- Cache-Control currently defines directives as part of it's value ABNF; which makes it look as if generic parsers need to special-case the predefined directives. Hopefully we all agree that parsers should be generic (as we suggest for new header fields in P2).
> 
> Thus I'd like to simplify the Cache-Control ABNF to the bare minimum, and then define the individual directives in separate subsections. Here's how this might look:
> 
> 3.2.  Cache-Control
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jun 2012 14:42:08 +0200

> +1; do you have a diff?

Always :-) -> 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/307/307.diff>

That being said, that's for the XML source only.

If this is going into the right direction I'll do some more tuning (the 
IANA template generation needs to be adjusted) and post a new diff later 
today.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 11 Jun 2012 22:44:22 +1000

Assuming that the text isn't changing in substance, yes.

I think this is verging on editorial; if anyone has concerns about this direction, please speak up; otherwise, Julian go ahead.

Cheers,

On 11/06/2012, at 10:42 PM, Julian Reschke wrote:

> Always :-) -> <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/307/307.diff>
> 
> If this is going into the right direction I'll do some more tuning (the IANA template generation needs to be adjusted) and post a new diff later today.
> 
> Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jun 2012 14:51:38 +0200

On 2012-06-11 14:44, Mark Nottingham wrote:
> ...

It's a bit more than editorial, because it tries to address the mismatch 
between the ABNFs for predefined directives (special-cased), and 
extension directives (token *and* quoted-string).

Highlighting these changes:

   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

3.2.2.2.  private

   Argument syntax:

      #field-name

   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   ....

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 11 Jun 2012 23:02:15 +1000

On 11/06/2012, at 10:51 PM, Julian Reschke wrote:

> It's a bit more than editorial, because it tries to address the mismatch between the ABNFs for predefined directives (special-cased), and extension directives (token *and* quoted-string).

Ah.

>  and quoted-string syntax.  Recipients MUST accept both forms.

The MUST is going too far here. I think that individual parameters can (and do) constrain the syntax.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jun 2012 15:16:24 +0200

> The MUST is going too far here. I think that individual parameters can (and do) constrain the syntax.

Well, they should not. It's pointless. Do you expect a parser for 
cache-control to special-case every constraint it knows about? What 
happens with an extension header that has constrained syntax? Some 
parsers will apply the constraint, some won't.

It's the same discussion we've had for other fields (say 
WWW-Authenticate, Expect, Prefer, STS, ...); and the answer should be 
the same.

We can't change what's out there, but we need to provide guidance how to 
fix these parsers to make them reliable.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 12 Jun 2012 06:58:46 +1000

On 11/06/2012, at 11:16 PM, Julian Reschke wrote:

> Well, they should not. It's pointless. Do you expect a parser for cache-control to special-case every constraint it knows about? What happens with an extension header that has constrained syntax? Some parsers will apply the constraint, some won't.
> 
> It's the same discussion we've had for other fields (say WWW-Authenticate, Expect, Prefer, STS, ...); and the answer should be the same.
> 
> We can't change what's out there, but we need to provide guidance how to fix these parsers to make them reliable.

If we take this path, we'll be making any cache that doesn't correctly parse

  Cache-Control: max-age="5"

non-conformant. I don't think that's a reasonable thing to do; it's not improving security or interoperability (our two "major" criteria), only library reusability.

I *thought* we'd already discussed this; once I find some coffee, perhaps I'll dig in the archives.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jun 2012 23:32:59 +0200

> I *thought* we'd already discussed this; once I find some coffee, perhaps I'll dig in the archives.

You are right; this is a case where we should just advise ("ought to"), 
not yell.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Jun 2012 22:57:24 +0200

On 2012-06-11 23:32, Julian Reschke wrote:
>> non-conformant. I don't think that's a reasonable thing to do; it's
>> not improving security or interoperability (our two "major" criteria),
>> only library reusability.
>>

OK, revised patch at 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/307/307.2.diff>.

Text for that section with important parts highlighted...:

3.2.  Cache-Control

--- NEW ---
    Cache directives are identified by a token, to be compared case-
    insensitively, and have an optional argument, that can use both token
    and quoted-string syntax.  For the directives defined below,
    recipients ought to accept both forms, even if one is documented to
    be preferred.  For any other directive, recipients MUST accept both
    forms.

--- NEW ---

3.2.1.1.  no-cache

3.2.1.2.  no-store

3.2.1.3.  max-age

    Argument syntax:

--- NEW ---
    Note: the preferred syntax for this argument is "token".  For
    compatibility with non-robust recipients, arguments SHOULD be sent
    using the token syntax.
--- NEW ---

3.2.1.4.  max-stale

    Argument syntax:

--- NEW ---
    Note: the preferred syntax for this argument is "token".  For
    compatibility with non-robust recipients, arguments SHOULD be sent
    using the token syntax.
--- NEW ---

3.2.1.5.  min-fresh

    Argument syntax:

3.2.1.6.  no-transform

3.2.1.7.  only-if-cached

3.2.2.1.  public

3.2.2.2.  private

    Argument syntax:

       #field-name

--- NEW ---
    Note: the preferred syntax for this argument is "quoted-string".  For
    compatibility with non-robust recipients, even a single-entry list of
    field names SHOULD be sent using the quoted-string syntax.
--- NEW ---

3.2.2.3.  no-cache

    Argument syntax:

       #field-name

--- NEW ---
    Note: the preferred syntax for this argument is "quoted-string".  For
    compatibility with non-robust recipients, even a single-entry list of
    field names SHOULD be sent using the quoted-string syntax.
--- NEW ---

3.2.2.4.  no-store

3.2.2.5.  must-revalidate

3.2.2.7.  max-age

    Argument syntax:

--- NEW ---
    Note: the preferred syntax for this argument is "token".  For
    compatibility with non-robust recipients, arguments SHOULD be sent
    using the token syntax.
--- NEW ---

3.2.2.8.  s-maxage

    Argument syntax:

--- NEW ---
    Note: the preferred syntax for this argument is "token".  For
    compatibility with non-robust recipients, arguments SHOULD be sent
    using the token syntax.
--- NEW ---

3.2.2.9.  no-transform

...yes, it's slightly more verbose, but I think it's worth the 
additional lines because it makes clear where a specific syntax is 
needed for historic reasons (only).

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jun 2012 13:17:19 +1000

>   allowed) otherwise stated otherwise.

A few things:

* You need to qualify the requirement so that it applies only to directives that take an argument

* The MUST still seems fairly strong; I'd be more comfortable with a SHOULD

* Isn't the requirement more appropriate in 3.2.3 Cache Control Extensions?

>   compatibility with non-robust recipients, arguments SHOULD be sent

Here and elsewhere, the quotes around the token are confusing, and the language is slightly revisionist (max-age, for example, has never been defined to use quotes).

Suggest replacing with:

Note: This cache-control directive uses the token form of the argument; e.g., 'max-age=5', not 'max-age="5". Senders SHOULD NOT use the quoted-string form.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2012 09:18:05 +0200

> * You need to qualify the requirement so that it applies only to directives that take an argument

Yes:

    Cache directives are identified by a token, to be compared case-
    insensitively, and have an optional argument, that can use both token
    and quoted-string syntax.  For the directives defined below that
    define arguments, recipients ought to accept both forms, even if one
    is documented to be preferred.  For any other directive, recipients
    MUST accept both forms.

> * The MUST still seems fairly strong; I'd be more comfortable with a SHOULD

The MUST is essential otherwise you can't deploy extension directives in 
a reliable way. What would be a reason not to do this? (recall the ABNF 
requires it, and always has)

> * Isn't the requirement more appropriate in 3.2.3 Cache Control Extensions?

The requirement is about how to parse the header field; it's not 
specific to extensions. Even if you do not understand a single extension 
(and do not plan to), you still have to skip over them properly while 
parsing.

> Note: This cache-control directive uses the token form of the argument; e.g., 'max-age=5', not 'max-age="5". Senders SHOULD NOT use the quoted-string form.

I made that:

    Note: This directive uses the token form of the argument syntax;
    e.g., 'max-age=5', not 'max-age="5"'.  Senders SHOULD NOT use the
    quoted-string form.

And for the other (#field-name) case:

    Note: This directive uses the quoted-string form of the argument
    syntax.  Senders SHOULD NOT use the token form (even if quoting
    appears not to be needed for single-entry lists).

Complete patch: 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/307/307.3.diff>

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jun 2012 20:32:06 +1000

>   is documented to be preferred.  For any other directive, recipients

You've managed to qualify it for those defined below, but not for "any other directive."

> The MUST is essential otherwise you can't deploy extension directives in a reliable way. What would be a reason not to do this? (recall the ABNF requires it, and always has)

How so? If I define foo to have an unquoted numeric argument, what breaks? A generic parser can parse quoted and token, but that doesn't mean that all new directives need to allow both forms.

> The requirement is about how to parse the header field; it's not specific to extensions. Even if you do not understand a single extension (and do not plan to), you still have to skip over them properly while parsing.

I see. I understand what you're trying to do, but I think it's too easy to misread it as "all new directives must be defined to allow both forms", which is both unnecessary and not realistic. Some rewording might help, but I question whether it's really necessary to put requirements around this; as you say, reading the BNF should make it clear what a generic parser needs to do.

>   appears not to be needed for single-entry lists).

Looks good, thanks.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2012 12:55:38 +0200

> You've managed to qualify it for those defined below, but not for "any other directive."

Indeed... How about:

    Cache directives are identified by a token, to be compared case-
    insensitively, and have an optional argument, that can use both token
    and quoted-string syntax.  For the directives defined below that
    define arguments, recipients ought to accept both forms, even if one
    is documented to be preferred.  For any directive not defined by this
    specification, recipients MUST accept both forms.

? 
(<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/307/307.4.diff>)

> How so? If I define foo to have an unquoted numeric argument, what breaks? A generic parser can parse quoted and token, but that doesn't mean that all new directives need to allow both forms.

It means that you add new special cases to something that is already 
unnecessarily complex. Why would you want to do that?

> I see. I understand what you're trying to do, but I think it's too easy to misread it as "all new directives must be defined to allow both forms", which is both unnecessary and not realistic. Some…

No, that's actually the *intent*. I believe it's both necessary *and* 
realistic.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 14 Jun 2012 13:23:14 +1000

On 13/06/2012, at 8:55 PM, Julian Reschke wrote:
> ? (<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/307/307.4.diff>)

Still managing to avoid it. The last sentence needs to say something like:

"""
For any other directive that accepts an argument, recipients MUST accept both forms.
"""

> It means that you add new special cases to something that is already unnecessarily complex. Why would you want to do that?

What special cases? 

For example, if I define a new directive that takes an URL as its argument, I might decide to say that it should ONLY use the quoted form, because I know that people will mess it up if they use token. How does that make things more complex? Your generic parser works the same.

> No, that's actually the *intent*. I believe it's both necessary *and* realistic.

So, I didn't see what you're trying to do. However, above you say it's not specific to extensions, but as you say later, that's the effect (and intent).

I think it's going too far; in similar situations we haven't laid down such draconian rules.

E.g., we don't place ANY conformance requirements on new headers; it's all advice: <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#considerations.for.creating.header.fields>

Likewise for auth schemes: <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p7-auth.html#considerations.for.new.authentication.schemes>

We don't place any conformance requirements on media type parameters either: <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#media.types>; we only note that they can be transmitted in either form. Why not use similar language here?

What I'd really like, though, is to hear what someone else things, so this isn't just a back-and-forth between Julian and I. Anyone?

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 14 Jun 2012 14:29:34 +1000

>>> I see. I understand what you're trying to do, but I think it's too
>>> easy to misread it as "all new directives must be defined to allow both
>>> forms", which is both unnecessary and not realistic.

> So, I didn't see what you're trying to do. However, above you say it's
> not specific to extensions, but as you say later, that's the effect
> (and intent).
>

My interpretation of Julian's crusade, using max-age as an example:

1. The syntax for a generic parameter is: token ["=" (token / quoted-string)].

2. A generic parser will accept max-age=5 and max-age="5" and max-age="\5", and simply make a one-character string available to the next stage (eg make it available in a dictionary).

3. handleMaxAge() code will use the 1-char string, checking it matches delta-seconds. handleMaxAge() doesn't know, nor care, which of the 3 forms in step 2 was received.

4. A developer wants to set max-age. He learns it is a decimal number in seconds. He doesn't closely read the spec. He sees some other xxx="yyy" parameters. He tries max-age="5". It works. He deploys his code.

5. Months later, after the developer's code has been working fine in systems X & Y, it fails with system Z. Z wasn't using a generic parser. Z enforced the max-age=token restriction.
=> Interop failure

Syntactically there is no need to allow max-age=quoted-string, but if some systems except that then other systems will start relying on it, then interop will fail with systems that don't allow it.

If, like system Z, you don't use a generic parser then Julian's rule is annoying. It means you have to support quoted-string for a field that shouldn't need it just in case a developer is sloppy. It doesn't seem fair. You have to do extra work up-front (supporting quoted-string, probably by switching to a generic parser) to save possible interop pain later.

Julian's rule encourages use of generic parsers (a good thing); and ensuring all developers diligently read our specs is not realistic. So (grudgingly) it is probably the best approach (or at least it would be if it supported any Unicode string value).

--
James Manger

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Jun 2012 08:49:55 +0200

> Julian's rule encourages use of generic parsers (a good thing); and ensuring all developers diligently read our specs is not realistic. So (grudgingly) it is probably the best approach (or at least…

Thanks James. Worth bookmarking :-)

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Jun 2012 09:29:47 +0200

> For any other directive that accepts an argument, recipients MUST accept both forms.

I believe that all new directives should not only be parseable with a 
generic parser, but also that senders can *rely* on them being parsed 
that way; see James' explanation.

> For example, if I define a new directive that takes an URL as its argument, I might decide to say that it should ONLY use the quoted form, because I know that people will mess it up if they use tok…

It's a special case in libraries that *construct* the header field. To 
conform to the SHOULD requirement, they'd have to understand the new 
directive.

> So, I didn't see what you're trying to do. However, above you say it's not specific to extensions, but as you say later, that's the effect (and intent).

Optimally, it would apply to all directives, but we can't do that 
without making existing code non-compliant (although I'm pretty sure 
that most existing code is non-compliant anyway).

> I think it's going too far; in similar situations we haven't laid down such draconian rules.

I would call that the opposite of draconian; I want to make *more* 
things conforming on the recipient side.

> E.g., we don't place ANY conformance requirements on new headers; it's all advice: <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#considerations.for.creating…

Optimally we would, but we don't have a mandate for that.

> Likewise for auth schemes: <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p7-auth.html#considerations.for.new.authentication.schemes>

That is incorrect. We've had these requirements all the time based on 
the ABNF.

Looking at RFC 2616:

Cache-Control   = "Cache-Control" ":" 1#cache-directive

    cache-directive = cache-request-directive
         | cache-response-directive

    cache-request-directive =
           "no-cache"                          ; Section 14.9.1
         | "no-store"                          ; Section 14.9.2
         | "max-age" "=" delta-seconds         ; Section 14.9.3, 14.9.4
         | "max-stale" [ "=" delta-seconds ]   ; Section 14.9.3
         | "min-fresh" "=" delta-seconds       ; Section 14.9.3
         | "no-transform"                      ; Section 14.9.5
         | "only-if-cached"                    ; Section 14.9.4
         | cache-extension                     ; Section 14.9.6

     cache-response-directive =
           "public"                               ; Section 14.9.1
         | "private" [ "=" <"> 1#field-name <"> ] ; Section 14.9.1
         | "no-cache" [ "=" <"> 1#field-name <"> ]; Section 14.9.1
         | "no-store"                             ; Section 14.9.2
         | "no-transform"                         ; Section 14.9.5
         | "must-revalidate"                      ; Section 14.9.4
         | "proxy-revalidate"                     ; Section 14.9.4
         | "max-age" "=" delta-seconds            ; Section 14.9.3
         | "s-maxage" "=" delta-seconds           ; Section 14.9.3
         | cache-extension                        ; Section 14.9.6

    cache-extension = token [ "=" ( token | quoted-string ) ]

So for any field not defined by RFC 2616, recipients always had to 
accept none/token/quoted-string. This is not new.

It says:

> The type/subtype MAY be followed by parameters in the form of attribute/value pairs.
>
>   parameter      = attribute "=" value
>   attribute      = token
>   value          = word
>
> The type, subtype, and parameter attribute names are case-insensitive. Parameter values might or might not be case-sensitive, depending on the semantics of the parameter name. The presence or absence of a parameter might be significant to the processing of a media-type, depending on its definition within the media type registry.
>
> A parameter value that matches the token production can be transmitted as either a token or within a quoted-string. The quoted and unquoted values are equivalent.

So I don't see any language here that would allow a parameter to 
restrict the serialization within the header field.

> What I'd really like, though, is to hear what someone else things, so this isn't just a back-and-forth between Julian and I. Anyone?

Yes, more feedback appreciated. I think it's important to be as 
consistent as possible (without making existing code non-compliant), 
otherwise it's hard to see how we can argue for consistency and 
simplicity in new header fields (STS comes to mind).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Jun 2012 12:51:08 +0200

> E.g., we don't place ANY conformance requirements on new headers; it's all advice: <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#considerations.for.creating…

And indeed we don't have this kind of advice for Cache-Control 
directives, but I think we should.

I just opened <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/360>:

> For new cache control directives, we should recommend to document
>
>     what it means for a directive to be specified multiple times (invalid?)
>
>     what it means for an argument-less directive to have an argument
>
>     what it means for a argument-required directive not to have an argument

I believe we should improve the spec here no matter what we do about 
issue #307.

My 2 cents:

> For new cache control directives, we should recommend to document
>
>     what it means for a directive to be specified multiple times (invalid?)

I think the least surprising approach is to recommend to make this 
invalid. Mark noted that there may be cases out there where it's both 
valid and useful; feedback appreciated...

>     what it means for an argument-less directive to have an argument
>
>     what it means for a argument-required directive not to have an argument

A simple approach would be to recommend to make a missing argument to be 
equivalent to a zero-length string argument.

Re #307: optimally, these rules would be the same for all cache 
directive, or for at least all new directives. But if we can't do that, 
we should at least recommend a common approach.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 14 Jun 2012 20:54:28 +1000

On 14/06/2012, at 5:29 PM, Julian Reschke wrote:

> I believe that all new directives should not only be parseable with a generic parser, but also that senders can *rely* on them being parsed that way; see James' explanation.

And that's fine; I'm just wondering why we're encouraging this with a MUST, where we haven't done so anywhere else, AFAICT.

> It's a special case in libraries that *construct* the header field. To conform to the SHOULD requirement, they'd have to understand the new directive.

OK.

> That is incorrect. We've had these requirements all the time based on the ABNF.

Yes, but we haven't surfaced them as conformance requirements for auth schemes like we are here. Why is this different?

> So I don't see any language here that would allow a parameter to restrict the serialization within the header field.

I think what I'm most concerned about here is that (in all of these cases, not just CC directives), we're requiring people to read between the lines. If we want to assure interoperability, we need to spell it out, e.g.,:

"""
Receivers of Cache-Control directives ought to be able to parse all three forms of directive (without a value, with a token value, and with a quoted-string value) for all directives. 
"""

And, separately in the cache control extensions section:

"""
New cache control extension directives are encouraged to:
   - Define whether or not they allow or require a parameter
   - If they do accept a parameter, allow it to occur in either token or quoted-string form (restricting to one form requires sending libraries to special-case the directive)
   - If they do accept a parameter, define how a directive without one should be interpreted
   - If they do not accept a parameter, define how a parameter with one should be interpreted
   - If they do accept a parameter, define how multiple occurrences should be handled
"""

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Jun 2012 13:55:31 +0200

> And that's fine; I'm just wondering why we're encouraging this with a MUST, where we haven't done so anywhere else, AFAICT.

I agree that we should be consistent about this in the spec 
independently of the header field. *Are* we currently inconistent?

> Yes, but we haven't surfaced them as conformance requirements for auth schemes like we are here. Why is this different?

I don't think it's different.

The header field definition needs to define how to parse the header 
field, including future extensions. Extensions can constrain 
parameter/argument values *after* parsing (like: *DIGIT), but shouldn't 
constrain the base syntax (for the reasons we have discussed many many 
times).

The reason why we're discussing this here is that we need to 
special-case the existing cache control directives for now, thus we also 
have text that tries to clarify that the situation is different for any 
other directive. Maybe that's where the disconnect comes from?

> I think what I'm most concerned about here is that (in all of these cases, not just CC directives), we're requiring people to read between the lines. If we want to assure interoperability, we need…

+1

> Receivers of Cache-Control directives ought to be able to parse all three forms of directive (without a value, with a token value, and with a quoted-string value) for all directives.

I'd prefer this to be a MUST for any directive not defined in HTTPbis.

> And, separately in the cache control extensions section:

A few minutes earlier, I opened the new ticket #360 for this...

>     - If they do accept a parameter, define how multiple occurrences should be handled

Indeed. Or alternatively, we could add additional constraints, see 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2012AprJun/0668.html>.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 15 Jun 2012 09:40:54 +1000

On 14/06/2012, at 9:55 PM, Julian Reschke wrote:

> I agree that we should be consistent about this in the spec independently of the header field. *Are* we currently inconistent?
> 
>> ...
> The header field definition needs to define how to parse the header field, including future extensions. Extensions can constrain parameter/argument values *after* parsing (like: *DIGIT), but shouldn't constrain the base syntax (for the reasons we have discussed many many times).
> 
> The reason why we're discussing this here is that we need to special-case the existing cache control directives for now, thus we also have text that tries to clarify that the situation is different for any other directive. Maybe that's where the disconnect comes from?

I *think* the only point of disagreement here is whether this (i.e., how to parse "non-core" cc directives) is advisory ("ought to...") or a conformance requirement ("MUST...").

I can't find anywhere else in our specs where we place conformance requirements on being able to parse multiple paths in the ABNF. Can you?

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Sun, 17 Jun 2012 18:20:30 +0200

> I can't find anywhere else in our specs where we place conformance requirements on being able to parse multiple paths in the ABNF. Can you?

(I'm not totally sure what you mean by "multiple paths" here, but I'll 
assume we're still talking about the quoted-string/token choice).

I would argue it's implicit in how the ABNF defined.

When the ABNF indicates that an extension parameter or directive uses

   quoted-string / token

then, yes, this means recipients MUST support both.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Jun 2012 11:41:54 +1000

On 18/06/2012, at 2:20 AM, Julian Reschke wrote:

> (I'm not totally sure what you mean by "multiple paths" here, but I'll assume we're still talking about the quoted-string/token choice).
> 

Julian -

I'm not disputing that. What I've been trying to ask, in so many ways, is why THIS case differs from all of the other cases. 

I.e., AFAICT, we rely on the ABNF in every similar case, without making an explicit requirement with RFC2119 language. Why is it necessary to have the ABNF *and* an explicit requirement here?

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Jun 2012 04:06:51 +0200

* Mark Nottingham wrote:
>What I've been trying to ask, in so many ways, is why THIS case differs
>from all of the other cases. 

If you name a couple, I assume Julian will happily explain where the
difference is, if there is any; he in fact asked for evidence that the
drafts are currently inconsistent in this regard, as I understood it,
anyway. (I'd offer some if I understood your concern well enough...)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Jun 2012 12:12:57 +1000

> anyway. (I'd offer some if I understood your concern well enough...)

I did, way back when...

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Jun 2012 05:45:02 +0200

>> anyway. (I'd offer some if I understood your concern well enough...)

Okay; Cache-Control is not different from media type parameters here,
but media type parameters already have similar language, "The quoted
and unquoted values are equivalent." Cache-Control could use the same
language if we did not mind that implementations that have trouble 
with `max-age="5"` would become (more clearly) non-conforming.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Jun 2012 08:11:32 +0200

> I.e., AFAICT, we rely on the ABNF in every similar case, without making an explicit requirement with RFC2119 language. Why is it necessary to have the ABNF *and* an explicit requirement here?

The difference is that Cache-Control *does* have special cases for 
historical reasons, whereas I think that is not the case for other headers.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Jun 2012 17:40:42 +1000

I think the underlying issue here is that we don't explicitly require recipients to be able to parse ABNF-conformant messages, even though we require senders to generate them.

I've created <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/361> to track. One we figure that out, these details of #307 should become clear.

Cheers,

On 19/06/2012, at 4:11 PM, Julian Reschke wrote:

> The difference is that Cache-Control *does* have special cases for historical reasons, whereas I think that is not the case for other headers.
> 
> Best regards, Julian
>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Jun 2012 09:57:13 +0200

> I've created <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/361> to track. One we figure that out, these details of #307 should become clear.

Thanks for that.

I believe that's a case where we started with something consistent 
(2616), and added a normative requirement (for the sending side) to 
clarify things, just to make the other case (recipients) less clear.

We currently have in 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-19.html#rfc.section.2.5>:

"Senders MUST NOT generate protocol elements that do not match the 
grammar defined by the ABNF rules for those protocol elements.

Unless otherwise noted, recipients MAY attempt to recover a usable 
protocol element from an invalid construct. HTTP does not define 
specific error handling mechanisms except when they have a direct impact 
on security, since different applications of the protocol require 
different error handling strategies. For example, a Web browser might 
wish to transparently recover from a response where the Location header 
field doesn't parse according to the ABNF, whereas a systems control 
client might consider any form of error recovery to be dangerous."

So, for consistency, we should insert:

"Recipients MUST accept all protocol elements matching the ABNF rules 
defined for them."

Maybe we need "Unless otherwise noted" because of some special cases for 
"obs-*".

We should also make the placement of this section consistent; in the 
other sections we have the equivalent text in Section 1.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Jun 2012 18:10:45 +1000

On 19/06/2012, at 5:57 PM, Julian Reschke wrote:

> "Recipients MUST accept all protocol elements matching the ABNF rules defined for them."

"accept" is ambiguous here. "Be able to parse" is more specific, I think.

> Maybe we need "Unless otherwise noted" because of some special cases for "obs-*".

yes.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 20 Jun 2012 19:23:33 +0200

>> Maybe we need "Unless otherwise noted" because of some special cases for "obs-*".

Proposed change:

339,347c339,349
<  Unless noted otherwise, Recipients MAY take steps to recover a usable
<  protocol element from an invalid construct.  However, HTTP does not
<  define specific error handling mechanisms, except in cases where it
<  has direct impact on security.  This is because different uses of the
<  protocol require different error handling strategies; for example, a
<  Web browser may wish to transparently recover from a response where
<  the Location header field doesn't parse according to the ABNF,
<  whereby in a systems control protocol using HTTP, this type of error
<  recovery could lead to dangerous consequences.
---
 >  Unless noted otherwise, Recipients MUST be able to parse all protocol
 >  elements matching the ABNF rules defined for them and MAY take steps
 >  to recover a usable protocol element from an invalid construct.
 >  However, HTTP does not define specific error handling mechanisms,
 >  except in cases where it has direct impact on security.  This is
 >  because different uses of the protocol require different error
 >  handling strategies; for example, a Web browser may wish to
 >  transparently recover from a response where the Location header field
 >  doesn't parse according to the ABNF, whereby in a systems control
 >  protocol using HTTP, this type of error recovery could lead to
 >  dangerous consequences.

(Change is just in the first sentence)

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 21 Jun 2012 09:55:24 +1000

> (Change is just in the first sentence)

Looks good to me.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 21 Jun 2012 11:03:28 +0200

>> (Change is just in the first sentence)

Applied with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1682>.

(I have opened a separate ticket as a reminder to re-align P1 with the 
other parts).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 24 Jun 2012 20:30:59 +0200

In the meantime, we have opened and resolved issue #361, by adding text 
that requires recipients to process all syntax variations allowed by the 
ABNF (see <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1682>).

With that, I'd like to go back to this issue; here's the text I proposed 
before we discussed #361, with a few inline comments prefixed "JR:".

3.2.  Cache-Control

JR: text below is new:

JR: I believe Mark wasn't happy with: "For any directive not defined by 
this specification, recipients MUST accept both forms."

Given what we did for issue #361, this would now repeat what we said 
about recipient requirements based on the ABNF. I still think it's good 
to state, because previously the ABNF was ambigouos; for instance,

   max-age=5

would have two ABNF paths match (as both "max-age" and 
"cache-extension"), while

   max-age="5"

would match just "cache-extension", and the spec didn't really say what 
that means.

We need to address this problem for the existing directives, and SHOULD 
avoid it for new directives; this proposal is my attempt to do so).

JR: remaining text is almost the same; except for the introduction of 
subsections, ABNF for parameter values, and notes about the "preferred" 
notation.

3.2.1.1.  no-cache

3.2.1.2.  no-store

3.2.1.3.  max-age

    Argument syntax:

3.2.1.4.  max-stale

    Argument syntax:

    Note: This directive uses the token form of the argument syntax;
    e.g., 'max-stale=10', not 'max-stale="10"'.  Senders SHOULD NOT use
    the quoted-string form.

3.2.1.5.  min-fresh

    Argument syntax:

    Note: This directive uses the token form of the argument syntax;
    e.g., 'min-fresh=20', not 'min-fresh="20"'.  Senders SHOULD NOT use
    the quoted-string form.

3.2.1.6.  no-transform

3.2.1.7.  only-if-cached

    The only-if-cached request directive indicates that the client only
    wishes to obtain a stored response.  If it receives 3ive, a cache
    SHOULD either respond using a stored response that is consistent with
    the other constraints of the request, or respond with a 504 (Gateway
    Timeout) status code.  If a group of caches is being operated as a
    unified system with good internal connectivity, a member cache MAY
    forward such a request within that group of caches.

3.2.2.1.  public

3.2.2.2.  private

    Argument syntax:

       #field-name

3.2.2.3.  no-cache

    Argument syntax:

       #field-name

3.2.2.4.  no-store

3.2.2.5.  must-revalidate

3.2.2.7.  max-age

    Argument syntax:

3.2.2.8.  s-maxage

    Argument syntax:

    Note: This directive uses the token form of the argument syntax;
    e.g., 's-maxage=10', not 's-maxage="10"'.  Senders SHOULD NOT use the
    quoted-string form.

3.2.2.9.  no-transform

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 25 Jun 2012 09:29:47 +1000

Works for me.

On 25/06/2012, at 4:30 AM, Julian Reschke wrote:

> In the meantime, we have opened and resolved issue #361, by adding text that requires recipients to process all syntax variations allowed by the ABNF (see <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1682>).
> 
> With that, I'd like to go back to this issue; here's the text I proposed before we discussed #361, with a few inline comments prefixed "JR:".
> 
> 3.2.  Cache-Control
> 
> JR: I believe Mark wasn't happy with: "For any directive not defined by this specification, recipients MUST accept both forms."
> 
> Given what we did for issue #361, this would now repeat what we said about recipient requirements based on the ABNF. I still think it's good to state, because previously the ABNF was ambigouos; for instance,
> 
>  max-age=5
> 
> would have two ABNF paths match (as both "max-age" and "cache-extension"), while
> 
>  max-age="5"
> 
> would match just "cache-extension", and the spec didn't really say what that means.
> 
> We need to address this problem for the existing directives, and SHOULD avoid it for new directives; this proposal is my attempt to do so).
> 
> JR: remaining text is almost the same; except for the introduction of subsections, ABNF for parameter values, and notes about the "preferred" notation.
> 
>

---

From: Julian Reschke <[email-masked]>
Date: Mon, 25 Jun 2012 09:19:59 +0200

> Works for me.

-> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1695>

Best regards, Julian