Subject: Re: suggestions for examples and explication wrt ABNF and header fields in draft-ietf-httpbis-p1

From: =JeffH <[email-masked]>
Date: Sat, 09 Jan 2010 14:40:09 -0800

Thanks for addressing this stuff Julian.

 >> In Section 1.2.2 Basic Rules...
 >>
 >>  >    Many HTTP/1.1 header field values consist of words separated by
 >>                                                    ^^^^^
 >>                                                    tokens
 >
 > I don't think "token" would be correct here.

Here's my rationale: "word" is used only twice in 
draft-ietf-httpbis-p1-messaging with respect to composition and parsing of 
(header field) string values (both occurrences being in "1.2.2.  Basic Rules"), 
yet what constitutes a "word" in this context is not defined anywhere in the 
spec, and seems superfluous. Tokens, however, are clearly defined in this 
context, and the word "token" is a term of art in the text parsing discipline 
(at least it was when I took my compilers class :)

 >>  >    whitespace or special characters.  These special characters MUST be
 >>  >    in a quoted string to be used within a parameter value (as defined in
 >>  >    Section 6.2).
 >>
 >> The "special characters" aren't mentioned anywhere in
 >> draft-ietf-httpbis-p* spec set other than the above, nor are they
 >> defined in ABNF. Inspection of draft-ietf-httpbis-p1 and RFC2616 reveals
 >
 > Yes. This happened when we rewrote the ABNF not to use "prose" rules
 > anymore.

as I suspected.

 > Unfortunately ABNF doesn't have syntax for something like "this
 > set of characters, except for that other set".

yep.

 > I have looked at this and decided to leave the prose alone for now. I
 > changed the ABNF to:
 >
 >      token          = 1*tchar
 >
 >      tchar          = "!" / "#" / "$" / "%" / "&" / "'" / "*"
 >                     / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
 >                     / DIGIT / ALPHA
 >                     ; any VCHAR, except special
 >
 >      special        = "(" / ")" / "<" / ">" / "@" / ","
 >                     / ";" / ":" / "\" / DQUOTE / "/" / "["
 >                     / "]" / "?" / "=" / "{" / "}"
 >
 >
 > (see <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/744>).
 >
 > This introduces an unused ABNF production, but I think that's ok for
 > clarity.

oh, ok, cool, that works for me.

thanks,

=JeffH

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jan 2010 17:23:31 +0100

=JeffH wrote:
> draft-ietf-httpbis-p1-messaging with respect to composition and parsing 
> of (header field) string values (both occurrences being in "1.2.2.  
> Basic Rules"), yet what constitutes a "word" in this context is not 
> defined anywhere in the spec, and seems superfluous. Tokens, however, 
> are clearly defined in this context, and the word "token" is a term of 
> art in the text parsing discipline (at least it was when I took my 
> compilers class :)

Yes. I don't claim that word == token, otherwise I would already have 
made that change.

RFC2616 said:

"The grammar described by this specification is word-based. Except where 
noted otherwise, linear white space (LWS) can be included between any 
two adjacent words (token or quoted-string), and between adjacent words 
and separators, without changing the interpretation of a field." -- 
<http://greenbytes.de/tech/webdav/rfc2616.html#basic.rules>

So what got lost is the explanation word = token / quoted-string. 
Apparently we need to resurrect that part.

> ...

BR, Julian