Subject: allowable characters in token as used in parameter ABNF

From: Julian Reschke <[email-masked]>
Date: Fri, 05 Feb 2010 16:40:12 +0100

Hi,

this came up while discussing Mark's Link Header draft.

For link extensions, it currently uses

   link-extension    = token [ "=" ( token | quoted-string ) ]

(<http://greenbytes.de/tech/webdav/draft-nottingham-http-link-header-07.html#rfc.section.5>)

This is consistent with HTTP/1.1's use of parameters:

   parameter               = attribute "=" value
   attribute               = token
   value                   = token | quoted-string

(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.6>)

(Reminder: this is the old RFC2616 ABNF style, allowing LWS between 
tokens and words etc)

However, for the (non-extended) parameter "type", the draft says:

   "type" "=" type-name "/" subtype-name

...so it uses a syntax that wouldn't be allowed for link-extensions ("/" 
is not allowed in tokens.

Testing with the two UAs that do something with the Link header (Opera 
and FF) show that they happily accept both quoted and unquoted type 
parameters.

So minimally,

   "type" "=" DQUOT type-name "/" subtype-name DQUOT

should be allowed as well.

However, special-casing the syntax for predefined parameters feels lame 
-- it would be good if you could use a generic parser to get all components.

So one way out of this would be to require that *all* parameters follow 
the pattern

   parameter               = attribute "=" value
   attribute               = token
   value                   = token | quoted-string

 From a purity point of view, that probably would be best. However, we 
have evidence (see above) that some implementations do not require 
quoted-string for certain characters that *are* forbidden in token.

Of course this could be addressed by saying "be lenient in what you 
accept", and be done with it.

Another approach that we'd like to discuss is to widen the set of 
characters that are allowed in unquoted parameter values, such as

   parameter               = attribute "=" value
   attribute               = token
   value                   = xtoken | quoted-string

and make xtoken extend token, also allowing certain harmless characters, 
such as "/".

If we do this, we'd really like to do this consistently in both 
draft-nottingham-http-link-header and draft-reschke-rfc2231-in-http, and 
potentially even define it in HTTPbis (maybe just as a recommended 
syntax component for new header fields).

Feedback appreciated,

Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Fri, 05 Feb 2010 16:46:25 +0100

> syntax component for new header fields).

Don't many headers accept more bytes there? E.g. cookie related headers.  
Making HTTP more flexible makes sense to me. Or maybe study some existing  
implementations first to see how this is commonly implemented. Do many use  
a generic parser?

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Julian Reschke <[email-masked]>
Date: Fri, 05 Feb 2010 16:59:43 +0100

Anne van Kesteren wrote:
> On Fri, 05 Feb 2010 16:40:12 +0100, Julian Reschke 
> <[email-masked]> wrote:
>> draft-nottingham-http-link-header and draft-reschke-rfc2231-in-http, 
>> and potentially even define it in HTTPbis (maybe just as a recommended 

Indeed, Cookies (as specced in RFC 2109) use that pattern as well.

> Making HTTP more flexible makes sense to me. Or maybe study some 
> existing implementations first to see how this is commonly implemented. 

Well, so far we're not discussing any normative change to HTTP/1.1 
itself; but for two new IDs draft use this pattern.

But we may want to add this as advice for new header definitions.

> Do many use a generic parser?

That's a good question.

Even if it's not the case today it would be cool if it could be done at 
least for new stuff.

BR, Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Fri, 05 Feb 2010 17:12:00 +0100

> Even if it's not the case today it would be cool if it could be done at  

Why exactly?

---

From: Julian Reschke <[email-masked]>
Date: Fri, 05 Feb 2010 17:17:03 +0100

Anne van Kesteren wrote:
> On Fri, 05 Feb 2010 16:59:43 +0100, Julian Reschke 
>> Even if it's not the case today it would be cool if it could be done 
>> at least for new stuff.
> 
> Why exactly?

So that you don't need to come up with a new parser for each of them?

(Am I missing something here?)

BR, Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Fri, 05 Feb 2010 17:22:12 +0100

> (Am I missing something here?)

Well, I don't really see the drawback in allowing more bytes by default.  
It seems that you always need a specific parser at some point except for  
headers that take fixed token values, but for those being more lenient is  
not an issue. Therefore I was wondering whether a concept of generic  
parser is even used/needed in implementations today. Or maybe they have  
such a concept, but it already is far more lenient so it can also cope  
with e.g. Link and Cookie-related headers. And maybe Authorization? And  
custom set headers through setRequestHeader() per chance? Should  
setRequestHeader() impose less strict requirements than it does now?

---

From: Julian Reschke <[email-masked]>
Date: Fri, 05 Feb 2010 17:25:46 +0100

Anne van Kesteren wrote:
> ...

...by default in what?

> headers that take fixed token values, but for those being more lenient 

so let's s/parser/parser component/

> is not an issue. Therefore I was wondering whether a concept of generic 

You tell me. What does Opera do here?

> ...

Sounds a bit as if you're now referring to the non-ASCII issue. That's 
orthogonal, I think.

BR, Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Fri, 05 Feb 2010 17:34:11 +0100

>> Well, I don't really see the drawback in allowing more bytes by default.

The generic value production.

>> It seems that you always need a specific parser at some point except  
>> for headers that take fixed token values, but for those being more  
>> lenient
>

Sure.

> You tell me. What does Opera do here?

Yngve is in a better position to answer that I think.

---

From: Dan Winship <[email-masked]>
Date: Fri, 05 Feb 2010 13:04:07 -0500

> Indeed, Cookies (as specced in RFC 2109) use that pattern as well.

RFC 2109 isn't used though. Set-Cookie and Cookie are complete
disasters, syntax-wise, and are almost certainly treated as
special-cases even by clients that otherwise use a generic parser.

>> Do many use a generic parser?

Evidence from Digest auth interoperability bugs is that some (probably
most) do, but some don't. Lots of people generate
WWW-Authenticate/Authorization headers under the assumption that the
receiving implementation will just parse it as "token 1#parameter". But
other people have written parsers that require that the parameters are
quoted if and only if they are quoted in RFC 2617 sections 3.2.1 and
3.2.2. (So eg, the "response" parameter MUST be quoted, but the "nc"
parameter MUST NOT be, even though they are both syntactically tokens.)
And then those people file bugs (and write Internet Drafts:
http://tools.ietf.org/html/draft-smith-sipping-auth-examples-01#section-2.1)
yelling at the people who assumed the grammar was generic.

-- Dan

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Feb 2010 13:53:20 -0800

> special-cases even by clients that otherwise use a generic parser.

Yeah, I wouldn't use the Set-Cookie header as a syntactic model for
anything.  :)

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 9 Feb 2010 23:36:35 +1100

Speaking personally --

I'm torn here. On one hand, I'd very much like to see best practice promoted here, because the wild-west situation of HTTP header parsing is one of the things I really dislike, and suspect causes a lot of problems.

OTOH, we don't have any implementers stepping up and saying that they're eager, and in this situation it may be too easy to specify the wrong thing.

AIUI the most liberal form of xtoken would be 1*VCHAR without DQUOTE, "," or ";". Correct?

If we can get agreement to that, I think we could document that as the construct for link-extension in the link draft, and perhaps recommend it in section 4 of Julian's draft when talking about how to accommodate extensibility. 

Beyond that, we have a bit more time to figure out if it's useful in httpbis as well.

Thoughts?

On 06/02/2010, at 2:40 AM, Julian Reschke wrote:

> (Reminder: this is the old RFC2616 ABNF style, allowing LWS between tokens and words etc)
> 
> ...so it uses a syntax that wouldn't be allowed for link-extensions ("/" is not allowed in tokens.
> 
> Testing with the two UAs that do something with the Link header (Opera and FF) show that they happily accept both quoted and unquoted type parameters.
> 
> So minimally,
> 
> However, special-casing the syntax for predefined parameters feels lame -- it would be good if you could use a generic parser to get all components.
> 
> So one way out of this would be to require that *all* parameters follow the pattern
> 
> From a purity point of view, that probably would be best. However, we have evidence (see above) that some implementations do not require quoted-string for certain characters that *are* forbidden in token.
> 
> Of course this could be addressed by saying "be lenient in what you accept", and be done with it.
> 
> Another approach that we'd like to discuss is to widen the set of characters that are allowed in unquoted parameter values, such as
> 
> and make xtoken extend token, also allowing certain harmless characters, such as "/".
> 
> If we do this, we'd really like to do this consistently in both draft-nottingham-http-link-header and draft-reschke-rfc2231-in-http, and potentially even define it in HTTPbis (maybe just as a recommended syntax component for new header fields).
> 
> Feedback appreciated,
> 
> Julian
> 
> 

--
Mark Nottingham     http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Feb 2010 14:34:18 +0100

> AIUI the most liberal form of xtoken would be 1*VCHAR without DQUOTE, "," or ";". Correct?

That will make it:

"!" / "#" / "$" / "%" / "&" / "'" / "(" / ")" / "*" / "+" / "-" / "." / 
"/" / DIGIT / ":" / "<" / "=" / ">" / "?" / "@" / ALPHA / "[" / "\" / 
"]" / "^" / "_" / "`" / "{" / "|" / "}" / "~"

which includes the following non-token characters:

"(" / ")" / "/" / ":" / "<" / "=" / ">" / "?" / "@" / "[" / "\" / "]" / 
"{" / "}"

"(" and ")" *might* become a problem in headers that allow comments.

"\" might be confused with the escape character in quoted strings and 
comments.

Other than that, I don't see a problem.

> If we can get agreement to that, I think we could document that as the construct for link-extension in the link draft, and perhaps recommend it in section 4 of Julian's draft when talking about how…

Sounds good.

> Beyond that, we have a bit more time to figure out if it's useful in httpbis as well.

I think for httpbis it will be valuable if we add a section with 
guidance on defining new header fields (which we should). Not sure 
whether we'd want to change any of the existing ABNF...

Best regards, Julian