Subject: progress on BNF conversion

From: Julian Reschke <[email-masked]>
Date: Tue, 13 Nov 2007 16:34:47 +0100

Hi,

in the spirit of making small incremental steps, I'd like to propose 
that we first fix the BNF so that it actually becomes parseable with a 
parser written for the 822/2616 BNF format.

The current issues I'm aware of are:

1) missing whitespace, such as in

    Accept-Charset = "Accept-Charset" ":"
            1#( ( charset | "*" )[ ";" "q" "=" qvalue ] )

which should be

    Accept-Charset = "Accept-Charset" ":"
            1#( ( charset | "*" ) [ ";" "q" "=" qvalue ] )

2) multi-line prose values, such as

     field-content  = <the OCTETs making up the field-value
                      and consisting of either *TEXT or combinations
                      of token, separators, and quoted-string>

3) prose values containing prose delimiters, such as

     qdtext         = <any TEXT except <">>

4) illegal characters in rule names, such as in

     http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]]

5) duplicate rule names (which are case-insensitive), such as with

    trailer        = *(entity-header CRLF)
    Trailer  = "Trailer" ":" 1#field-name

6) attempts to do something in BNF that just does not work :-):

     chunk          = chunk-size [ chunk-extension ] CRLF
                      chunk-data CRLF
     chunk-size     = 1*HEX
     last-chunk     = 1*("0") [ chunk-extension ] CRLF

     chunk-extension= *( ";" chunk-ext-name [ "=" chunk-ext-val ] )
     chunk-ext-name = token
     chunk-ext-val  = token | quoted-string
     chunk-data     = chunk-size(OCTET)

(note the chunk-date rule in the last line).

7) editorial nit: Bill's ABNF parser (BAP) prefers to see all rule names 
to be indented by the same amount; I think this is just a matter of 
editorial quality and we simply should make the indentations consistent.

Here are my proposals to fix the individual issues:

1) just fix it.

2) try to get rid of prose value; when not possible, replace with a 
shorter one and add the remaining text as BNF comment.

3) Use DQUOTE instead of <">

4) Replace "_" with "-". In some cases we currently import rules from 
other older specs in which case we can write this as:

    abs-path = <abs_path defined in ...>

5) Keep the canonical rule names for the header productions, replace the 
other ones.

6) just fix it.

7) fix the indentation.

(Would it make sense to open a separate issue for this collection of 
problems?)

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 13 Nov 2007 09:56:42 -0800

On Nov 13, 2007, at 7:34 AM, Julian Reschke wrote:
> in the spirit of making small incremental steps, I'd like to  
> propose that we first fix the BNF so that it actually becomes  
> parseable with a parser written for the 822/2616 BNF format.

I don't understand.  That is a lot harder than just replacing with
the standard ABNF.  Note that HTTP is the basis for a hundred or so
other IETF standards track specifications, so the pressure to update
the ABNF is greater than the sum of all other 2616 issues.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 13 Nov 2007 19:29:25 +0100

> the ABNF is greater than the sum of all other 2616 issues.

Seems there's a misunderstanding here.

All the issues I listed need to be resolved if we move to the RFC4234 
format anyway.

What would be left then would be:

- replacing "|" by "/"

- deciding what to do with #element and 1#element (keep this ABNF 
extension or expand the rules (*))

- deciding about implicit LWS

So I was just trying to break this down into smaller chunks...

Best regards, Julian

(*) I know have got a hack for BillF's ABNF parser automating that in 
case we want to do that.

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 14 Nov 2007 11:31:39 +0100

Julian Reschke wrote:

> - deciding what to do with #element and 1#element (keep this
> ABNF extension or expand the rules (*))

Certainly "expand", it's the worst part of the old syntax, it
permits ASCII art consisting of commata and linear white space
over multiple lines.

> So I was just trying to break this down into smaller chunks.

Isn't it simpler to track the changes when fixing the ABNF is
one of the _last_ steps ?  It's also not really difficult, on
the SASL list it was tested with a 2831bis draft (RFC 2831 as
a variant of 2617 inherited the "old" syntax with the #-rule).

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Wed, 14 Nov 2007 13:23:33 +0100

> permits ASCII art consisting of commata and linear white space

Sure. But can we get rid of that? At least not as a requirement for the 
recipient to accept.

> a variant of 2617 inherited the "old" syntax with the #-rule).

And kept it. Interesting data point.

Looking at a version of the HTTP grammar which expands the # rule, I 
must say it's not pretty :-).

Best regards, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 14 Nov 2007 13:39:28 +0100

Julian Reschke wrote:
 
> Sure. But can we get rid of that?

Dunno, I'd certainly love to get rid of it.  That's a question
for the "implementation and interoperability report".  If it
does not really work in practice you could be free to drop it.

>> on the SASL list it was tested with a 2831bis draft (RFC 2831
>> as a variant of 2617 inherited the "old" syntax with the #-rule)
 
> And kept it. Interesting data point.

Actually the SASL folks decided to move RFC 2831 to historic 
soon after they saw the expanded ABNF for 2831bis, but there
were other reasons for this decision.  And at that point in
time I didn't know that 2831 md5-sess and 2617 md5-sess might
be incompatible (the fact is hidden in an 2617 erratum).

> Looking at a version of the HTTP grammar which expands the
> # rule, I must say it's not pretty :-).

Truth in advertising.  Compare
<http://article.gmane.org/gmane.ietf.sasl/2758>
<http://article.gmane.org/gmane.ietf.sasl/2759>
It's not _too_ ugly if you're forced to keep it.

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Fri, 23 Nov 2007 15:55:09 +0100

Hi,

in the meantime I have made some progress changing the BNF productions 
so that they actually become parseable; see 
<http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/draft-lafon-rfc2616bis-latest.html> 
and 
<http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/draft-lafon-rfc2616bis-latest-from-previous.abnf.diff.html>.

One thing I'm currently looking at is to remove rules defined in prose 
when possible. For instance:

      token          = 1*<any CHAR except CTLs or separators>
      separators     = "(" | ")" | "<" | ">" | "@"
                     | "," | ";" | ":" | "\" | <">
                     | "/" | "[" | "]" | "?" | "="
                     | "{" | "}" | SP | HT

could be rewritten as

     separators     = "(" | ")" | "<" | ">" | "@"
                    | "," | ";" | ":" | "\" | DQUOTE
                    | "/" | "[" | "]" | "?" | "="
                    | "{" | "}" | SP | HT

     tchar          = "!" | "#" | "$" | "%" | "&" | "'" | "*" | "+" | "-"
                    | "." | "^" | "_" | "`" | "|" | "~" | DIGIT | ALPHA
                      ; any CHAR except CTLs or separators
     token          = 1*tchar

(unless I broke it, of course :-).

Which leads me to the obvious question: was it really intended that 
things like method names and header names can contain all these characters?

BR, Julian

(planning to submit a draft defining the "!" method).

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 26 Nov 2007 09:40:47 +0100

>                    | "," | ";" | ":" | "\" | DQUOTE

18 + HT.
 
>                    | "." | "^" | "_" | "`" | "|" | "~" | DIGIT | ALPHA

15 + DIGIT + ALPHA.  15 + 10 + 52 + 18 = 95, 95 + CTL = 128,
yes, that looks good... :-)

> Which leads me to the obvious question: was it really intended
> that things like method names and header names can contain all
> these characters?

It mirrors MIME (RFC 2045), maybe it makes more sense to mirror
<attribute-char> in RFC 2231:

| attribute-char := <any (US-ASCII) CHAR except SPACE, CTLs,
|                    "*", "'", "%", or tspecials>

Please note that the MIME <tspecials> don't include "{" and "}",
it's not exactly the same set as the HTTP <separators>.  Maybe
the draft should mention this difference from MIME <tspecials> ?

Or mention the difference from RFC 2822 (= 2822upd) <specials>,
that would be WSP (SP + HT), "/", "?", "=", "{", and "}" (+7),
excluding "." (-1), 13 +7 -1 = 18 + HT as above.

> (planning to submit a draft defining the "!" method).

LOL.

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Fri, 30 Nov 2007 14:26:11 +0100

Hi,

I've created a few slides summarizing the current state of things WRT to 
the conversion to the RFC4234bis ABNF syntax -- see 
<http://greenbytes.de/tech/webdav/httpbis-bnf.pdf>.

BR, Julian