Subject: Implied LWS questions

From: Julian Reschke <[email-masked]>
Date: Fri, 06 Jun 2008 16:18:05 +0200

<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#implied.LWS>:

   "The grammar described by this specification is word-based. Except 
where noted otherwise, linear white space (LWS) can be included between 
any two adjacent words (token or quoted-string), and between adjacent 
words and separators, without changing the interpretation of a field. At 
least one delimiter (LWS and/or separators) MUST exist between any two 
tokens (for the definition of "token" below), since they would otherwise 
be interpreted as a single token."

<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#rfc.section.3.1>

"The version of an HTTP message is indicated by an HTTP-Version field in 
the first line of the message. HTTP-Version is case-sensitive.

   HTTP-Version   = "HTTP" "/" 1*DIGIT "." 1*DIGIT"

So, do "HTTP" and "/" qualify as instances of quoted-string?

What about 1*DIGIT? That's definitively not a quoted string, but it 
could be parsed as token.

So, after applying the implied LWS rule, what is the ABNF for HTTP-Version?

1) HTTP-Version = "HTTP" *LWS "/" *LWS 1*DIGIT *LWS "." *LWS 1*DIGIT

or

2) HTTP-Version = "HTTP" *LWS "/" 1*DIGIT "." 1*DIGIT

Looking futher down... In 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#rfc.section.3.2>, 
the specification imports BNF rules from RFC2396:

   absoluteURI   = <absoluteURI, defined in [RFC2396], Section 3>
   authority     = <authority, defined in [RFC2396], Section 3.2>
   fragment      = <fragment, defined in [RFC2396], Section 4.1>
   path-absolute = <abs_path, defined in [RFC2396], Section 3>
   port          = <port, defined in [RFC2396], Section 3.2.2>
   query         = <query, defined in [RFC2396], Section 3.4>
   relativeURI   = <relativeURI, defined in [RFC2396], Section 5>
   uri-host      = <host, defined in [RFC2396], Section 3.2.2>

...

   http-URL = "http:" "//" uri-host [ ":" port ]
              [ path-absolute [ "?" query ]]

So, does http-URL allow *LWS anywhere? It's certainly not supposed to, 
but I think the rules allow is between "http:" and "//".

I'd really like to make progress on the ABNF thingy, and thus get rid of 
the implied LWS rule, but I'm not totally sure how to proceed? (I came 
here because I attempted mechanical conversion, which led to these 
questions).

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 6 Jun 2008 17:32:03 +0200

> What about 1*DIGIT? That's definitively not a quoted string,
> but it could be parsed as token.

An 822-parser would "see" any <specials> including ".", but 
not "/".  Based on that the old syntax could be misleading,
how about this:

| HTTP-Version   = ( "HTTP/" 1*DIGIT "." 1*DIGIT" ) / obs-version
|
| obs-version    = "HTTP/" 1*DIGIT *WSP "." *WSP 1*DIGIT

No *WSP before or after the slash.  IIRC we already agreed
that there can't be any folding, otherwise LWS would result
in [FWS] instead of *WSP.

Please don't say *LWS, more than one adjacent LWS makes no
sense.  A single LWS already allows multiple line foldings,
a *LWS buys you nothing apart from confusing readers... :-)

> the specification imports BNF rules from RFC2396:

RFC 3986 has an appendix with translations of old constructs. 
 
> does http-URL allow *LWS anywhere?

No, RFC 2396 has no #-LWS horrors, and STD 66 is anyway clean.
RFC 2616 didn't introduce LWS in RFC 2396 http URIs <shudder />

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Fri, 06 Jun 2008 17:46:28 +0200

> | obs-version    = "HTTP/" 1*DIGIT *WSP "." *WSP 1*DIGIT

Well, for now I'm not trying to change anything, just to come to a 
common understanding about what RFC2616 *really* says.

> in [FWS] instead of *WSP.

Again, see above.

> a *LWS buys you nothing apart from confusing readers... :-)

The rule is called "implied *LWS", but then it doesn't state whether 
it's really "*LWS" or "[ LWS ]".

And yes, *LWS and [ LWS ] are different; the latter only allows one CRLF.

> RFC 2616 didn't introduce LWS in RFC 2396 http URIs <shudder />

I do agree that it would be non-sense to introduce it; right now I'm 
just trying to understand whether there *is* a mechanical way to 
transform the RFC2616 ABNF so that the implied LWS rule can go.

BR, Julian

---

From: Dan Winship <[email-masked]>
Date: Fri, 06 Jun 2008 12:17:18 -0400

> So, do "HTTP" and "/" qualify as instances of quoted-string?

I think the grammar is eating your brain :-). "quoted-string" means that
the *result* is quoted, not the *rule*. The strings generated by those
rules don't have quotes around them, so no, they're not quoted strings.
But "HTTP" is a token, and "/" is a separator, so yes, *LWS is allowed
between them.

> 2) HTTP-Version = "HTTP" *LWS "/" 1*DIGIT "." 1*DIGIT

I would say

  3) HTTP-Version = "HTTP" *LWS "/" *LWS 1*DIGIT "." 1*DIGIT

Because "/" is a separator and "HTTP" and 1*DIGIT are tokens, but "." is
neither a separator nor a token.

> but I think the rules allow it between "http:" and "//".

As I read it, you can't put *LWS between "http:" and "//", because
neither of those strings is a token, a quoted string, or a separator.
(I'm assuming you're not allowed to treat the string "http:" as though
it was the same as "http" ":".) However, you could have *LWS around the
":" separating host and port, or the "?" separating path and query.

But I'd totally believe you if you claimed that 2616 never meant for
implied *LWS to apply inside rules imported from other specs.

Request-Line has the same problem as http-URL with *LWS around "?".
Given that there are already rules in the running text stating that you
can't have CR or LF in the Request-Line or Status-Line (meaning that any
implied *LWS appearing there is not really LWS anyway...), we should
probably just say that the implied *LWS rule was also not meant to apply
to the Request-Line and Status-Line. (This is already implied by the
"Tolerant Applications" appendix; there would be no need to say that
apps "SHOULD accept any amount of SP or HTAB characters between fields"
if implied *LWS was in effect there.)

Another major oddity of implied *LWS is this:

    chunk          = chunk-size [ chunk-extension ] CRLF
                     chunk-data CRLF
    chunk-extension= *( ";" chunk-ext-name [ "=" chunk-ext-val ] )

Since chunk-size can be parsed as a token, and chunk-extension starts
with a separator, you can in theory put *LWS between them. Which makes
the parsing ambiguous because there's no way to distinguish a
chunk-extension preceded by CR LF SP from a chunk-data that just happens
to look like a chunk-extension.

-- Dan

---

From: Geoffrey Sneddon <[email-masked]>
Date: Fri, 6 Jun 2008 18:08:12 +0100

On 6 Jun 2008, at 15:18, Julian Reschke wrote:

> "The version of an HTTP message is indicated by an HTTP-Version  
> field in the first line of the message. HTTP-Version is case- 
> sensitive.
>

Exactly per RFC2616, I'd say Dan's 3).

However, in the real world everything is rather crazy (this is testing  
by the comparatively sane 2) with a single space character). Saf/Mac  
treats it as a HTTP/0.9 response (i.e., all the data sent by the  
server is the response); Firefox goes crazy (the status code is given  
as 200, and the status text is "200 OK"), but the rest is as would be  
the case without any *LWS (it seems to just split on the first two SP  
characters); Opera copes with a single SP there per 1) or 2); IE is  
even crazier, with a status code of -9 and the status text of "200  
OK", though everything after the response line is fine; Apache seems  
to cope fine; HTTP.sys (in IIS/7.0) returns 400 Bad Request. I expect  
stuff gets even madder once you use CRLF SP as that *LWS (like stuff  
being treated as headers and the like), or even just multiple spaces.

I think it's safe to say that no specific behaviour is needed in the  
real world, which is why I'd lean towards following Saf/Mac and  
HTTP.sys: either outright rejecting it as invalid, or falling back to  
treating it as HTTP/0.9 (this is, FWIW, I do in my ultra-early-draft  
HTTP parsing spec).

As for http-url, I'd say it doesn't allow it anywhere.

--
Geoffrey Sneddon
<http://gsnedders.com/>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 06 Jun 2008 19:47:02 +0200

> I think the grammar is eating your brain :-). "quoted-string" means that

It sure is.

Will get back to it next week :-)

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 6 Jun 2008 20:31:16 +0200

Julian Reschke wrote:

> The rule is called "implied *LWS", but then it doesn't 
> state whether it's really "*LWS" or "[ LWS ]".

> And yes, *LWS and [ LWS ] are different; the latter

You are right, *LWS is the same as [LWSP] or [FWS], it
is not [LWS].  And it's certainly eating *my* brain. :-(

> right now I'm just trying to understand whether there
> *is* a mechanical way to transform the RFC2616 ABNF so
> that the implied LWS rule can go.

There is a way to translate # constructs, outside of #
I'm not sure.  I guess they didn't replace 822 by 2234
and 2822, because everybody understood 822... <gd&r>   

 Frank

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 08 Jun 2008 11:59:59 +0200

> But "HTTP" is a token, and "/" is a separator, so yes, *LWS is allowed

But it is also a typical case where LWS SHOULD NOT be allowed, as many
implementations will fail badly if there is LWS here.

> ":" separating host and port, or the "?" separating path and query.

No. Space is explicitly disallowed in HTTP URLs, and LWS is parsed as a
single space, not "nothing".

There is special rules for URLs in textual context such as email body,
where LWS (including folding) is ignored. But this does not apply to
HTTP.

> to look like a chunk-extension.

Ouch..

This is a typical case where we need to explicitly forbid implied LWS,
and at most allow *(SP|HT) around the ; and chunk-size.

    chunk          = *(SP|HT) chunk-size *(SP|HT) [ chunk-extension ] CRLF
                     chunk-data CRLF
    chunk-extension= *( ";" *(SP|HT) chunk-ext-name [ "=" chunk-ext-val ] *(SP|HT))

I am sure there is better productions for this...

Regards
Henrik