Subject: Re: #231: Considerations for new headers

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Oct 2011 10:26:31 +0200

On 2011-08-28 17:08, Julian Reschke wrote:
> On 2011-08-26 17:57, Julian Reschke wrote:
>> On 2011-07-01 07:05, Mark Nottingham wrote:
>>> Revised proposal:
>>> ...
>>
>> +1 - we may want to add more things or tune, but we should get this in.
>>
>> Proposed patch:
>> <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/231/231.diff>
>>
>> ...
>
> ...applied (minus some typos) as
> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1417>.
> ...

There recently were some interesting discussion in other Working Groups, 
namely:

websec - implications of not using the list syntax when the header field 
*does* occur multiple times (Origin, Strict-Transport-Security)

hybi - when using params, whether it makes sense to disallow one of the 
two notations on the wire (token vs quoted-string)

oauth - how to I18N parameters (there's a proposal to use JSON escapes 
inside quoted-string, which IMHO conflicts with the definition in HTTPbis).

So it's good that we have started work on this (header field syntax 
recommendations), but we probably need more:

>    o  Whether the field is a single value, or whether it can be a list
>       (delimited by commas; see Section 3.2 of [Part1]).

add...: "If it does not use the list syntax, how to treat messages where 
the header field occurs multiple times (a sensible default would be to 
ignore the header field, but this may not always be the right choice)."

Also, we probably should have a single place that defines the "param" 
ABNF construct, like

   param = token OWS "=" OWS ( token / quoted-string )

or maybe

   param = token OWS [ "=" OWS ( token / quoted-string ) ]

and then clarify that predefined params never ever should parse 
different from extension parameters; for instance, they should always 
allow both notations.

See also <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/307> and 
<http://trac.tools.ietf.org/wg/httpbis/trac/wiki/HeaderFieldTypes>.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 11 Oct 2011 09:56:04 +1100

+1

On 10/10/2011, at 7:26 PM, Julian Reschke wrote:

> There recently were some interesting discussion in other Working Groups, namely:
> 
> websec - implications of not using the list syntax when the header field *does* occur multiple times (Origin, Strict-Transport-Security)
> 
> hybi - when using params, whether it makes sense to disallow one of the two notations on the wire (token vs quoted-string)
> 
> oauth - how to I18N parameters (there's a proposal to use JSON escapes inside quoted-string, which IMHO conflicts with the definition in HTTPbis).
> 
> So it's good that we have started work on this (header field syntax recommendations), but we probably need more:
> 
> add...: "If it does not use the list syntax, how to treat messages where the header field occurs multiple times (a sensible default would be to ignore the header field, but this may not always be the right choice)."
> 
> Also, we probably should have a single place that defines the "param" ABNF construct, like
> 
> and then clarify that predefined params never ever should parse different from extension parameters; for instance, they should always allow both notations.
> 
> See also <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/307> and <http://trac.tools.ietf.org/wg/httpbis/trac/wiki/HeaderFieldTypes>.
> 
> Best regards, Julian

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 11 Oct 2011 09:25:25 +0200

>> add...: "If it does not use the list syntax, how to treat messages where the header field occurs multiple times (a sensible default would be to ignore the header field, but this may not always be…

-> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1446>

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 13 Oct 2011 19:52:55 +0200

> <http://trac.tools.ietf.org/wg/httpbis/trac/wiki/HeaderFieldTypes>.

Proposal:

"Many header fields use a format including named parameters (for 
instance, Content-Type). Parameter values should always allow both 
unquoted (token) and quoted (quoted-string) values:"

If we later get around to defining concrete ABNF building blocks, we can 
still rephrase this...

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 14 Oct 2011 08:35:07 +1100

WFM

On 14/10/2011, at 4:52 AM, Julian Reschke wrote:

> "Many header fields use a format including named parameters (for instance, Content-Type). Parameter values should always allow both unquoted (token) and quoted (quoted-string) values:"
> 
> If we later get around to defining concrete ABNF building blocks, we can still rephrase this...
> 
> Best regards, Julian

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 14 Oct 2011 12:12:06 +1100

> unquoted (token) and quoted (quoted-string) values:"

What is the benefit of requiring that both syntaxes are supported?

I cannot see how a sender needs the flexibility to send a given parameter in either form. Do apps often pass string values to a generic library or framework that then decides between the token or quoted-string syntax? That seems quite unlikely.

A receiver might prefer to use a generic parser (that accepts token or quoted-string for any parameter value) instead of one that knows the extra restrictions on particular parameters. That parser will still work: it will accept all valid values. The only issue is that the parser can accept invalid values as well (eg accept a token value when a spec says a quoted-string must be sent). That is, the only issue is that the parser might be too lenient. The receiving app could enforce the extra parameter-specific restrictions after the generic parser has passed up the result, though many receivers might not bother (or the generic parser might have discarded the info).

Is the motivation for this proposal that:
* Some receivers will accept a value as either a token or quoted-string regardless of the parameter's definition.
* Being lenient (despite the IETF mantra) can be a security risk.
* By allowing token and quote-string in a parameter's definition, such receivers are not being lenient so there is less security risk.

Or is the motivation that:
* Some receivers will accept a value as either a token or quoted-string regardless of the parameter's definition.
* Some senders will send a value as a token even if the definition says quoted-string (and vice versa) because it works with some lenient receivers.
* Those senders will not interoperate with other (non-lenient) receivers.

Perhaps a better rule would be that a receiver of named parameters in a header field should be allowed to accept token or quoted-string (or RFC5987) values. Hence, specs should not assign different semantics to these different forms even if the spec requires senders to use one particular syntax.

--
James Manger

---

From: Julian Reschke <[email-masked]>
Date: Fri, 14 Oct 2011 08:56:59 +0200

> A receiver might prefer to use a generic parser (that accepts token or quoted-string for any parameter value) instead of one that knows the extra restrictions on particular parameters. That parser…

A receiver using a lenient parser will accept all values, and this 
creates an interop issue. If the majority of recipients accepts both 
syntaxes, then, de facto it's required to do so.

It's very unlikely that recipients using a generic parser will reject a 
certain syntax *after* parsing. The simple reason for that is that 
generic parsers usually do not retain the information whether the value 
was quoted or not.

> * Those senders will not interoperate with other (non-lenient) receivers.

Mainly the second, informed by the problems we have seen with existing 
parameters (think Content-Type/charset or WWW-Authenticate/Realm).

> Perhaps a better rule would be that a receiver of named parameters in a header field should be allowed to accept token or quoted-string (or RFC5987) values. Hence, specs should not assign different…

I don't see the advantage. Special rules per header field or even per 
parameter make it extremely unlikely that recipients will behave 
differently. If they try (by writing custom parsers), the risk of 
getting things wrong actually will be much higher.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 16 Oct 2011 14:33:21 +0200

Ok, for now I have added:

    Many header fields use a format including named parameters (for
    instance, Content-Type, defined in Section 6.8 of [Part3]).  Allowing
    both unquoted (token) and quoted (quoted-string) syntax for the
    parameter value enables recipients to use existing parser components.
    Also, the meaning of a value ought to be independent of the syntax
    variant used for it (for an example, see the notes on parameter
    handling for media types in Section 2.3 of [Part3]).

<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1448>

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 27 Oct 2011 21:57:05 +0200

On 2011-10-11 09:25, Julian Reschke wrote:
>>> add...: "If it does not use the list syntax, how to treat messages
>>> where the header field occurs multiple times (a sensible default
>>> would be to ignore the header field, but this may not always be the
>>> right choice)."
>> ...
>

I added more, triggered by a recent discussion over in the websec WG 
about "Origin:":

        Furthermore, intermediaries and software libraries might combine
        multiple header field instances into a single one, despite the
        header field not allowing this.  A robust format enables
        recipients to discover these situations (good example: "Content-
        Type", as the comma can only appear inside quoted strings; bad
        example: "Location", as a comma can occur inside a URI).

-> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1458>

Best regards, Julian