Subject: Collected ABNF for HTTPbis

From: Julian Reschke <[email-masked]>
Date: Sun, 17 May 2009 17:33:49 +0200

Hi,

I discussed generating a collected ABNF for Parts 1 through 7 with Björn 
Höhrmann (see <http://jabber.ietf.org/logs/httpbis/2009-05-16.txt> and 
<http://jabber.ietf.org/logs/httpbis/2009-05-17.txt>).

Generating it turned out to be a *bit* tricky:

- we had an unintended name collision between "parameter" in Part 1 and 
3 (fixed with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/583>)

- there are a few duplicated rules (harmless)

- some rules have a generic version in Part 1 (for instance, "Method"), 
and a more specific in Part 3; this is a known TODO

Working around the two latter issues, we can generate 
<http://greenbytes.de/tech/webdav/httpbis.abnf> which I'll keep 
up-to-date from now, for those who want to play around with it (feedback 
   appreciated).

Best regards, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 17 May 2009 18:40:28 -0500

Julian Reschke wrote:
> up-to-date from now, for those who want to play around with it
> (feedback appreciated).

The collected ABNF is something that is best published separately, probably
by somebody other than the IETF. Generating the collected ABNF is useful for
its diagnostic checks, as you noted. But, it shouldn't be published as part
of the specification.

Many other IETF specifications do not publish a collected ABNF. In fact,
very few seem to do so. SMTP and IMAP are two examples of specifications
analogous to HTTP that don't provide the collected ABNF.

As Julian said before, the collected ABNF exists in order to desugar the
"#X" ABNF extension into true ABNF syntax. AFAICT, that isn't necessary
since ABNF processing tools seem to handle the extension anyway. But, also,
the "#X" extension is really a huge readability advantage. Every usage of
"#X" is easily replaced with X (COMMA-ows X)* where COMMA-ows is just "OWS
COMMA OWS". That doesn't justify the extra complication. Just desugar all
the inline productions, remove the #X extension, remove the collected ABNF.

Obviously, the collected ABNF is redundant with the rest of the
specification. When there is such redundancy, it is important to make it
clear how to resolve differences that arise from the redundancy. My recent
suggestions for cutting text out of the spec. have all been aimed at solving
this problem by removing redundancy. If the inline grammar is normative and
the collected ABNF is normative, then readers are forced to either trust
that they are in sync and/or double-check that they are in sync. If for some
reason the collected ABNF doesn't match the inline grammar then there will
be much confusion about which one to use. It is much easier on the reader to
explicitly say the collected ABNF is non-normative to avoid the issue. The
issue can be avoided altogether by removing the collected ABNF from the
specification.

It doesn't make sense to create a collected ABNF and then split it into
multiple parts like it is in the current drafts. Yet, collecting it in that
goes against the reasons for splitting the specification into multiple parts
in the first place. Not only that, but it would be a lot more useful to
implementers if it included the grammar productions from other
specifications that are currently included by reference (from RFC3986,
RFC1864, RFC4647, RFC5322). At the same time, the HTTP specification should
not duplicate those other specifications' extensive grammar productions. 

The final issue is the one I mentioned before: The collected ABNF would be
much easier to use directly if it was published without the IETF
boilerplate. I checked into this and it seems like there's no accepted
process for including such plain-text attachments in the official version of
IETF specifications.

Anybody can already use the same tool the editors are using to generate the
collected ABNF from the inline grammar. AFAICT, anybody can then publish
this "true" collected ABNF as a plain-text file. And, they can publish the
grammar in whatever format is most usable for then: ABNF syntax, Flex/Bison
syntax, etc.

Regards,
Brian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 18 May 2009 02:21:42 +0200

>the inline productions, remove the #X extension, remove the collected ABNF.

If you would like to suggest changes to the drafts, the best way to do
that would be to start a new thread. As for the suggestions above, the
expansion you give is incorrect as it fails to account for empty items
in lists, and there is little evidence of widespread support for the
list syntax, for example, Chris Newman's abnf.c, bap, abnfgen, and my
own Parse::ABNF Perl module do not support it out of the box.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 17 May 2009 21:20:27 -0500

Bjoern Hoehrmann wrote:
> Brian Smith wrote:
> >As Julian said before, the collected ABNF exists in order to desugar
> > the "#X" ABNF extension into true ABNF syntax. AFAICT, that isn't
> > necessary since ABNF processing tools seem to handle the extension
> > anyway. But, also, the "#X" extension is not really a huge
> > readability advantage. Every usage of "#X" is easily replaced with
> > X (COMMA-ows X)* where COMMA-ows is just "OWS COMMA OWS". That
> > doesn't justify the extra complication. Just desugar all
> As for the suggestions above, the expansion you give is incorrect as
> it fails to account for empty items in lists

There are only four places that allow empty lists, and they are all of the
same form as "TE":

     TE        = "TE" ":" OWS TE-v
     TE-v      = #t-codings

They all can be rewritten using that expansion:
     
     TE        = "TE" ":" OWS [TE-v]
     TE-v      = t-codings (COMMA-ows t-codings)*

Every other use is of the form 1#rule.

> and there is little evidence of widespread support for the

All the more reason to get rid of it.

Regards,
Brian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 18 May 2009 05:16:17 +0200

>Every other use is of the form 1#rule.

I am not sure what you are trying to say here. This is a matter of empty
list items, not empty lists and the difference between 0#rule and 1#rule
is the difference between [ 1#rule ] and 1#rule. Your rewrite disallows
constructs that are valid under RFC 2616 and the current draft for which
support is required and recommended respectively.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Brian Smith" <[email-masked]>
Date: Sun, 17 May 2009 23:06:38 -0500

Bjoern Hoehrmann wrote:
> I am not sure what you are trying to say here. This is a matter of
> empty list items, not empty lists and the difference between 0#rule and
> 1#rule is the difference between [ 1#rule ] and 1#rule. Your rewrite
> disallows constructs that are valid under RFC 2616 and the current draft
> for which support is required and recommended respectively.

I understand what you mean now. I grabbed the wrong expansion from the spec;
I should have started with the more complicated one:

   #element  => [ ( "," / element ) *( OWS "," [ OWS element ] ) ]
   1#element => *( "," OWS ) element *( OWS "," [ OWS element ] )

which is effectively (considering the contexts in which those rules are
used) just:

   #element  => [element] *(list-sep [element])
   1#element => *LIST-SEP element *(LIST-SEP [element])
   LIST-SEP  => OWS "," OWS

Which, desugared, results in rules such as:

   TE        => "TE" ":" OWS TE-v
   TE-v      => [t-codings] *(LIST-SEP [t-codings])

I do admit that isn't super readable. But, the naming "-v" convention
(probably better "-list") and the avoidance of an ABNF extension make up for
it. If getting rid of the #rule extension avoids having to deal with the
collected ABNF, then this is a reasonable tradeoff. 

- Brian