Subject: [abnf-discuss] ABNF colloquialism for end-of-line

From: Sean Leonard <[email-masked]>
Date: Wed, 15 Nov 2017 19:54:36 +0800

Hello ABNF “Doctors”:

On a recent thread on cbor@ <https://mailarchive.ietf.org/arch/msg/cbor/ZL2NvalH6jmSVqfvkBH4af3BHsE>, the following question arose:
What is the best ABNF colloquialism for end-of-line?

CR LF is the “Internet standard newline”, with rule name <CRLF> [RFC5234]. However, there is a desire amongst some specification writers to admit Unix LF as well. I have not heard of people clamoring for bare CR; however if bare LF is on the table then bare CR might be on the table too, for completeness. A quite small minority of published RFC specifications define an end-of-line production as CRLF / LF, and an even smaller minority of published RFC specifications allow for bare CR as well.

So I just wanted to see what ABNF people’s opinions are on this one.

Thanks,

Sean

~~~~~~~
2017-11-15 Facts about CRLF and new lines in ABNF in RFC Series
Sean Leonard

Facts about CRLF and new lines:

Surveyed ABNF of all RFCs between RFC 2000 to RFC 7999.

“What is the end-of-line marker called, other than CRLF?” (by search of CRLF)

Answer: Most call it CRLF. Amongst the minority, EOL is the most common, followed by line-break or lineBreak. Most define the end-of-line marker as CR LF; in the minority, most in the minority use CRLF / LF, and a smaller subset uses CR / LF (aka any of bare CR, bare LF, or CRLF).

RFC	Hint	Name	=	Definition
2705	Megaco	EOL	=	CRLF / LF
2967*	TISDAG	nl	=	%d13 %d10
2967*	TISDAG	SEP	=	(CR LF) | LF   <-- NOT actually ABNF 
3108	SDP	EOL	=	(CR / LF / CRLF)
3435	Megaco	EOL	=	CRLF / LF
3780	SMIng	lineBreak =	CRLF / LF
4997	ROHC-FN	CRLF	=	%x0A / %x0D.0A
4997	ROHC-FN	NL	=	COMMENT / CRLF
4997	ROHC-FN	COMMENT	=	"//" *(SP / HTAB / VCHAR) CRLF
5228*	Sieve	(Sieve explicitly limits to CRLF-only in text)
5234*	ABNF	c-nl	=	comment / CRLF
5234*	ABNF	comment =	";" *(WSP / VCHAR) CRLF
6020	YANG	line-break =	CRLF / LF
7208*	SPF	CRLF	=	<standard end-of-line token as per [RFC5322]>
7468	PKIXt	eol	=	CRLF / CR / LF
7468	PKIXt	eolWSP	=	WSP / CR / LF
7468	PKIXt	W	=	WSP / CR / LF / %x0B / %x0C

“How is ‘nl’ defined?” (by search of ‘nl’)

Answer: Always CRLF.

RFC	Hint	Name	=	Definition
2957	whoispp	nl	=	%d13 %d10
2958	whoispp	nl	=	%d13 %d10
2967*	TISDAG	nl	=	%d13 %d10
4997	ROHC-FN	NL	=	COMMENT / CRLF
5234*	ABNF	c-nl	=	comment / CRLF
5234*	ABNF	comment =	";" *(WSP / VCHAR) CRLF
6787*	MRCPv2	phrase-nl =	"Phrase-NL" ":" 1*UTFCHAR CRLF

“How is ’newline’ used?” (by search of ‘newline’) 

Answer: ‘newline’ is never defined as a rule name. When it appears in comments, it is *always* associated with CR LF only. (Same as ‘nl’.)

“How is ‘0A’ used?” (by search of 0A)

Answer: ‘0A’ is almost always defined as LF or used as LF; no additional interesting variations were uncovered. The only interesting variation is RFC 4464, which has %x0a / %x0d / (others) as whitespace generically, but that protocol is not line-oriented.

* items do not address the premise of the inquiry, but is included anyway for completeness.
~~~~~~~

---

From: Dave Crocker <[email-masked]>
Date: Wed, 15 Nov 2017 07:37:06 -0800

> So I just wanted to see what ABNF people’s opinions are on this one.

Let's start from first principles:

      Interoperability is improved by simplicity. Simplicity is improved 
by having a single, canonical form for data; participants that do not 
use that form as their 'native' form convert to the canonical form. Core 
ABNF rules are designed to aid simplicity.

      When designing abnf rulesets decide whether the goal is 
interoperability -- that is, common representation for participants in a 
standardized protocol -- or is to describe raw, non-canonical data; that 
is, the goal is describing a range of native data.  Clarity about this 
distinction aids in making design choices for rules.

Given that the thread in CBOR says 'matching rules', I'm guessing that 
the goal here is to describe freeform data coming from the net.  Hence, 
requiring a simple, canonicalized data form is not appropriate.  (This 
is an essential point; if it's not correct, then what follows won't be 
either.)

Note that the term "Internet standard newline" in RFC5234 is merely a 
comment to the ABNF Core Rule definition of CRLF.

What you most definitely do /not/ want is to redefine crlf to cover both 
crlf and lf.

But I gather you /do/ want a rule that covers both CRLF and LF.

So define one.

For example:

    cbor-nl:  crlf / lf

Is there some problem with using this?

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Sean Leonard <[email-masked]>
Date: Thu, 16 Nov 2017 07:35:52 +0800

Hello Dave,

> On Nov 15, 2017, at 11:37 PM, Dave Crocker <[email-masked]> wrote:
> 
>     Interoperability is improved by simplicity. Simplicity is improved by having a single, canonical form for data; participants that do not use that form as their 'native' form convert to the canonical form. Core ABNF rules are designed to aid simplicity.
> 
>     When designing abnf rulesets decide whether the goal is interoperability -- that is, common representation for participants in a standardized protocol -- or is to describe raw, non-canonical data; that is, the goal is describing a range of native data.  Clarity about this distinction aids in making design choices for rules.
> 
> 
> Given that the thread in CBOR says 'matching rules', I'm guessing that the goal here is to describe freeform data coming from the net.  Hence, requiring a simple, canonicalized data form is not appropriate.  (This is an essential point; if it's not correct, then what follows won't be either.)

The original thread was about the Matching Rules appendix, which was based on the adjacent ABNF grammar for CDDL (CBOR Data Definition Language, or whatever the acronym is these days). It’s really to describe the grammar that specification writers are to write, akin to ABNF itself, ASN.1, programming language code, etc.

> 
> Note that the term "Internet standard newline" in RFC5234 is merely a comment to the ABNF Core Rule definition of CRLF.

Yes. Oh, I forgot the reference that standardizes CRLF as the actual Internet standard; I guess the latest is RFC 5198?

> 
> What you most definitely do /not/ want is to redefine crlf to cover both crlf and lf.

Yeah, I think that is dumb, but (as I documented) a handful of RFCs do just that.

> Is there some problem with using this?

My opinion is that newline === CRLF in IETF Internet standards, and since ABNF is an Internet standard, the concept of newline === CRLF. In a local operating environment, one can convert CRLF to whatever else. For example, the C standard library treats \n (aka LF) as the end-of-line marker, and converts appropriately on ingress/egress. Ditto for HTML/XML. (This is congruent with your first paragraph on simplicity.)

I understand why people want to define EOL = CRLF / LF; however, ABNF producers such as abnfgen will (probably) randomly output sample data with mixed line endings. The author’s intent of EOL = CRLF / LF is to say that a format (rather than a wire protocol) ought to accept either CRLF or LF as line endings in a file, but it has the effect of saying that CRLF/LF can be randomly interspersed in the same PDU. (And, in so doing, it privileges LF but denies CR, or NEL, or LS / PS, which don’t have anything better or worse for them going when it comes to line endings.)

In my own RFC 7468, I said:
 eol        = CRLF / CR / LF
 eolWSP     = WSP / CR / LF
 W          = WSP / CR / LF / %x0B / %x0C ; whitespace

and while I don’t view that as a mistake necessarily, I’m just pointing out that probably a better way to handle the issue would have been to say that CRLF *is* the end-of-line marker, and that when ingressing/egressing data, to treat whatever the local convention is, as if it’s CRLF in the ABNF.

One reason to have a uniform ABNF for line endings (i.e., CRLF) is inter-specification reuse. For example, one protocol such as URI [RFC3986] has ABNF productions that are reused throughout many other specifications, including specs that are CRLF oriented (mail...) and specs that are just LF oriented (HTML). URIs do not include line breaks, but if you want to reuse a production that includes line breaks in one RFC, but the line ending specs don’t match, then it will be a problem.

Sean

> _______________________________________________
> abnf-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/abnf-discuss

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 16 Nov 2017 12:18:04 +1000

Hi Sean,

[trimming a bunch]

> I understand why people want to define EOL = CRLF / LF; however, ABNF producers such as abnfgen will (probably) randomly output sample data with mixed line endings. The author’s intent of EOL = CRL…

[aside]
... except that they don't have 7-bit ASCII codes (NEL collides with U+2026).
[/aside]

> and while I don’t view that as a mistake necessarily, I’m just pointing out that probably a better way to handle the issue would have been to say that CRLF *is* the end-of-line marker, and that whe…

An "end-of-line marker" is different from a "newline character," so if
you want a 'best' colloquialism I'd say some variation of "newline" is
the way to go.  But there's no standard definition of "newline" in any
core ABNF specs, because there's no standard definition outside of
ABNF.

If authors want to write rules for parsing a byte sequence into
tokens, defined in ABNF, and they want to allow Windows or UNIX or
Apple][ newlines, I'm completely with your and Dave's `CRLF / LF /
CR`-type of definitions.  If they want to use reuse the same ABNF for
*generating* byte sequences, and want to mandate/suggest/implore that
implementers use one or other as the canonical form, can they not do
that by talking to the human reader?

~~~
   foobar  = freb *( blat ) foobar-newline
   foobar-newline = ( CRLF / LF / CR )  # when parsing liberally
                 / CRLF    # when generating conservatively
~~~

I think that's the point you're making... but I don't quite get why
there's any source for confusion.  As a wild stab, is it because of an
over-reliance on tools to turn ABNF into running code?

> One reason to have a uniform ABNF for line endings (i.e., CRLF) is inter-specification reuse. For example, one protocol such as URI [RFC3986] has ABNF productions that are reused throughout many ot…

[aside]
HTML isn't particularly LF-oriented, unless you consider &#10; vs
&#13; a significant skew.

<https://www.w3.org/TR/html5/syntax.html#newlines>:
~~~
Newlines in HTML may be represented either as "CR" (U+000D)
characters, "LF" (U+000A) characters, or pairs of "CR" (U+000D), "LF"
(U+000A) characters in that order.
Where character references are allowed, a character reference of a
"LF" (U+000A) character (but not a "CR" (U+000D) character) also
represents a newline.
~~~
[/aside]

But yeah, I think there's a good reason CRLF is called "CRLF".
There's nothing stopping someone from minting a compelling and useful
"NEWLINE" ABNF rule that ends up being widely adopted, aside from
natural forces.

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Nov 2017 10:35:48 +0800

Hi Dave,

> Given that the thread in CBOR says 'matching rules', I'm guessing that the goal here is to describe freeform data coming from the net.  Hence, requiring a simple, canonicalized data form is not app…

The thread title unfortunately is misleading.

The ABNF is not for on-the wire packets, but for defining the syntax of the CDDL language (which then defines the syntax of the on-the-wire data items).

So this ABNF is about files on computers, which probably run a form of Linux/Unix or Windows (and very likely not pre-2001 classic MacOS).  So

   EOL = [CR] LF

is probably the right way to describe line ends for these files.

The context for this (with one exception) is

   NL = COMMENT / EOL
   COMMENT = ";" *(SP / VCHAR) EOL
   VCHAR = %x21-7E

Maybe NL is not the best name for the encompassing rule, but it is used only in one additional rule:

   WS = SP / NL

Grüße, Carsten

PS.: If ABNF had cuts, I’d write

   EOL = [CR ^] LF

to get better errors on stray CRs.  Maybe not so important these days.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 16 Nov 2017 07:31:19 -0800

Carsten,

> So this ABNF is about files on computers, which probably run a form of Linux/Unix or Windows (and very likely not pre-2001 classic MacOS).  So

I take your point, but suspect there is still an issue.  At the least, 
being clear /and explicit/ about this in the specification document(s) 
will be helpful.

The issue I suspect is the intended portability of the file.  If the 
file is intended to be portable, then it, too, needs to be in a 
canonical form.  It's a type of 'over the wire' even though it isn't 
part of a wire protocol.

This, then, would require separate translation from native, local 
representation to the canonical form.  But that's a pretty simple 
definition effort.

> is probably the right way to describe line ends for these files.

Possibly, unless folk really want

    EOL = *CR LF

> Maybe NL is not the best name for the encompassing rule, but it is used only in one additional rule:

Yeah, maybe not the best.  If ruleset is not really treating it 
semantically as end of line.

So while there is an historical basis for saying EOL, I'd think that in 
this context, it would sufficient and simpler just to have:

   WS = SP / CR / LF

(why not also include TAB?)

> to get better errors on stray CRs.  Maybe not so important these days.

Sorry.  Not sure what you mean.  "cuts"? "^"?

d/

---

From: Sean Leonard <[email-masked]>
Date: Fri, 17 Nov 2017 05:26:34 +0800

> On Nov 16, 2017, at 11:31 PM, Dave Crocker <[email-masked]> wrote:
> 
> Carsten,
> 
>>> [...]
> Sorry.  Not sure what you mean.  "cuts"? "^”?

To add some color to this point, “cuts” was discussed in the CBOR WG in the context of CDDL. It is a technique from Parsing Expression Grammars. Here is overview:

http://www.romanredz.se/papers/CSP2014.pdf

https://en.wikipedia.org/wiki/Parsing_expression_grammar
(see soft cut)

Basically it commits the parser at a particular point, so that it does not backtrack.

However, PEGs are ordered; ABNF is unordered. With ABNF (as presently constituted), all alternatives in a choice are considered simultaneously (order is not relevant). Even if you match one alternative, you’re supposed to try all other alternatives.

EOL = [CR ~CUT~] LF
EOL = CR ~CUT~ LF / LF

are equivalent: the point being that once a CR is encountered, LF is supposed to follow...and if LF does not follow, then the parser fails but throws an error that EOL did not match because there was no LF after the CR (because it’s more specific). However, cuts also takes the initial production CR out of the equation, so if you have:

EOL = CR ~CUT~ LF / LF / CR "foo"

the last alternative will never be reached, because it’s order-sensitive: the ~CUT~ will consume the first CR, so CR "foo" can never be matched. The equivalent technology in regular expressions is “atomic grouping”.

With a context-free grammar, it is better to think about all alternatives simultaneously.

Sean

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 18 Nov 2017 10:57:27 +0100

On Nov 16, 2017, at 23:31, Dave Crocker <[email-masked]> wrote:
> I take your point, but suspect there is still an issue.  At the least, being clear /and explicit/ about this in the specification document(s) will be helpful.

Right.

> The issue I suspect is the intended portability of the file.  If the file is intended to be portable, then it, too, needs to be in a canonical form.  It's a type of 'over the wire' even though it isn't part of a wire protocol.

Well, software development has focused on Unix line ends, tolerating DOS line ends in some spaces, for a while.
That seems to work for so many languages, we can just emulate that.

Let’s take a page from RFC 7950 (YANG 1.1):

   line-break          = CRLF / LF

That is essentially the same I was proposing, but the explicit name “line-break” is probably better than NL by some.

> This, then, would require separate translation from native, local representation to the canonical form.  But that's a pretty simple definition effort.

Again, I think that most source control systems and programmers’ editors know how to do that.

> Possibly, unless folk really want

We don’t want to tolerate more than one CR here; these would be isolated CRs in todays line end worlds.

> So while there is an historical basis for saying EOL, I'd think that in this context, it would sufficient and simpler just to have:
> 
>  WS = SP / CR / LF

That generally works, except in certain strings, where it is good to identify actual line ends.

> (why not also include TAB?)

I can’t find the string TAB in RFC 20; I think you probably mean HT.
HT is evil(**) and, approximately since the time 300 bit/s LA36 terminals(***) went out of use, should never be used(*).
Easy fix as applied here: Simply don’t allow HT in specification source files.

Grüße, Carsten

(*) Outside certain very sheltered environments such as Linux Kernel development.
(**) RFC 7386/7396 should be proof enough here.
(***) https://archive.org/details/handbookofintera00duan p. 82; additionally insert fond memories here…

---

From: Paul Kyzivat <[email-masked]>
Date: Sat, 18 Nov 2017 12:54:30 -0500

From a practical perspective, when parsing things that need EOL marks, 
ISTM that you can generally get good results by treating LF as EOL, and 
CR as WS. It works for CRLF, LFCR and naked LF. (I don't recall anything 
in the last few decades that treats a naked CR as EOL.)

This of course isn't enough to know what forms to produce when 
generating text.

	Thanks,
	Paul

---

From: Dave Crocker <[email-masked]>
Date: Sun, 19 Nov 2017 10:56:10 -0800

> To add some color to this point, “cuts” was discussed in the CBOR WG in the context of CDDL. It is a technique from Parsing Expression Grammars. Here is overview:
...
> However, PEGs are ordered; ABNF is unordered. With ABNF (as presently constituted), all alternatives in a choice are considered simultaneously (order is not relevant). Even if you match one alterna…

(Disclaimer:  What follows is pure personal opinion.)

I'll offer this for consideration, without also offering any specific 
action...

Why did ABNF become popular?

(For a time, RFC 733 and then RFC 822 were the most-cited RFCs.  It 
turned out this was not due to the email portion but folk were re-using 
the ABNF meta-specification, which is why the later, revision effort to 
RFC 822 split the ABNF text out into its own RFC.)

At the time ABNF was defined in the latter 1970s, most specs provided 
their own variation of BNF.  Everyone wanted tailoring to the basic 
tool.  But while folk have often wanted to enhance ABNF, over the years, 
the 'let's define a new variant' tendency mostly died out -- ignoring 
the much more recent move towards JSON...  Why did this popularity happen?

Languages need to balance expressive power against human usage 
complexity.  Enough but not too much, of each.  ABNF seemed to strike a 
good balance.  (I like to think the documentation clarity in RFC 733 
also helped, but then I'm quite biased about this, given how much effort 
I put into that aspect of the work...)

I think the biggest danger in creating a meta-language is specification 
obscurity.  The tendency to want to add features can too-easily create 
too much complexity for easy human comprehension.  The result is that 
seemingly-simple specifications can too-easily have implications that 
are not understood by most readers.

Computers are not the target audience for computer languages.  Human 
readers are.  Subtle effects (nevermind side-effects) are very easily 
missed by human readers.

d/

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 19 Nov 2017 22:08:23 +0100

> On Nov 19, 2017, at 19:56, Dave Crocker <[email-masked]> wrote:
> 
> Computers are not the target audience for computer languages.  Human readers are.  

+10k.

This is so obvious once one starts to think about why we need these languages in the first place.
Unfortunately, considerations about implementation dominate the literature about formal languages.

(Computer history aficionados will remember Algol68, a ground-breaking, highly innovative, but also very practical computer language.  Unfortunately, the theoretically brilliant, but an unusable for humans, two-level Van Wijngaarden grammar was used to specify it.  This completely destroyed adoption of Algol 68.)

> Subtle effects (nevermind side-effects) are very easily missed by human readers.

This is one of the reasons why context-free grammars are so practical.
(In reality they are usually a bit of whitewashing because symbol tables are invariably used to add some context-sensitivity, but that can be considered to be on the semantic level, even if it sometimes isn’t.)

Cuts were initially invented to limit the decision space of Prolog programs to fit the memory available.
When Parse Expression Grammars became implementable, the same approach was useful to limit their memory requirements as well.
Two implementation concerns.

Cuts also have been used to improve error messages from Parse Expression Grammars.  
Now this is about usability of a tool (such as a compiler or validator) for humans and thus highly relevant.

In CDDL, we have discussed cuts also for improving error messages, which maybe is something that can be left for a CDDL 2.0.  However, it recently became clear that CDDL users actually do want some context-sensitive properties for the grammar — being context-free is counter-intuitive in certain limited situations (the “map” container CBOR inherited from JSON also is inherently context sensitive).  Hence our renewed interest in cuts here, for actually obtaining those limited forms of context sensitivity.  (We are still discussing how to do this without turning CDDL into a monster; certainly we don’t want another van Wijngaarden effect.)

Grüße, Carsten

---

From: Paul Kyzivat <[email-masked]>
Date: Sun, 19 Nov 2017 19:19:45 -0500

>> Computers are not the target audience for computer languages.  Human readers are.

I agree that the human audience is of major importance.
But mechanical verifiability of ABNF is also important.

The existing ABNF definition fits that need if you don't do something 
when using it to break that verifiability. Unfortunately, the ABNF as 
used in the IETF is rarely completely verifiable without a bunch of 
manual tweaking.

The end result of this is that often the ABNF in drafts either isn't 
formally verified (relying on eyeballing by reviewers), or is 
incompletely verified. (Superficially verifying that the ANBF text in a 
draft is well formed with the exception of undefined references that, 
based on informal notation in the text, are defined elsewhere.)

Going beyond that requires a bunch of manual work that is error prone.

Ideally IdNits would verify ABNF in drafts as a matter of course. IIUC 
doing so has been included as a requirement for the new tooling. Given 
the current situation what IdNits can potentially do is very limited.

	Thanks,
	Paul

---

From: Dave Crocker <[email-masked]>
Date: Sun, 19 Nov 2017 16:54:08 -0800

On 11/19/2017 4:19 PM, Paul Kyzivat wrote:
>>> readers are.

I of course do not mean that computers are irrelevant.  However 
'verifiability' invokes formal methods and those typically aren't a 
concern for typical specification efforts.  Machine processing, yes. 
Mathematical proofs, no.

I gather you are claiming otherwise.  I haven't noticed any history of 
formal verification for IETF-related specifications, so perhaps you have 
some pointers?

> the current situation what IdNits can potentially do is very limited.

IdNits hardly seems to qualify as 'verification'.  So I'm probably not 
understanding exactly what problem you are concerned about.  Details?

d/

---

From: Paul Kyzivat <[email-masked]>
Date: Sun, 19 Nov 2017 20:13:42 -0500

Dave,

> understanding exactly what problem you are concerned about.  Details?

I'm not talking about verifying/proving that the ABNF does what was 
"intended". I'm willing to grant (for purposes of IETF documents) that 
verifying the ABNF reflects the *intent* is done by the human review 
process.

I'm simply talking about verifying that the ABNF is entirely well 
formed, that all rulenames have definitions, and that there are no 
multiple definitions. (Multiple instances of the same rulename on the 
left of  an "=".)

	Thanks,
	Paul

---

From: Dave Crocker <[email-masked]>
Date: Sun, 19 Nov 2017 17:20:07 -0800

> multiple definitions. (Multiple instances of the same rulename on the 

ahh. syntactic validation.  Hmmm... I'd had the impression that that was 
pretty common these days.

d/

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 20 Nov 2017 07:50:02 +0100

> On Nov 20, 2017, at 02:13, Paul Kyzivat <[email-masked]> wrote:
> 
> Dave,
> 
>> I of course do not mean that computers are irrelevant.  However 'verifiability' invokes formal methods and those typically aren't a concern for typical specification efforts.  Machine processing, yes. Mathematical proofs, no.
>> I gather you are claiming otherwise.  I haven't noticed any history of formal verification for IETF-related specifications, so perhaps you have some pointers?
>>> Ideally IdNits would verify ABNF in drafts as a matter of course. IIUC doing so has been included as a requirement for the new tooling. Given the current situation what IdNits can potentially do is very limited.
>> IdNits hardly seems to qualify as 'verification'.  So I'm probably not understanding exactly what problem you are concerned about.  Details?
> 
> I'm not talking about verifying/proving that the ABNF does what was "intended". I'm willing to grant (for purposes of IETF documents) that verifying the ABNF reflects the *intent* is done by the human review process.

Right, but tools like abnfgen can help enormously in this task — the reviewer generates a few matching instances and eyeballs whether they look like what was intended.  Validating a few manually generated instances against the ABNF is the other computer-assisted process in validation.

> I'm simply talking about verifying that the ABNF is entirely well formed, that all rulenames have definitions, and that there are no multiple definitions. (Multiple instances of the same rulename on the left of  an "=“.)

If the YANG people can do validation automatically upon submission, why can’t we?

(That would probably need the XML, with properly marked up [type=“abnf”] artwork elements.  But we could encourage submitting the XML for drafts that cite RFC 5234 normatively.)

Grüße, Carsten

---

From: Sean Leonard <[email-masked]>
Date: Mon, 20 Nov 2017 00:35:52 -0800

> (That would probably need the XML, with properly marked up [type=“abnf”] artwork elements.  But we could encourage submitting the XML for drafts that cite RFC 5234 normatively.)

Note: reaex (SeanTek's Regular Expression-Based ABNF Extractor) 
<https://tools.ietf.org/tools/table> properly and accurately grabs ABNF 
out of plain text RFCs and Internet-Drafts each and every time. Includes 
automatic comment generation to annotate where in the RFC the ABNF came 
from. Tested on all of the RFCs in the RFC series.*

Sean

*Succeeds when it's actually ABNF. Skips over stuff that is not actually 
ABNF (including ABNF typos). I think a few superseded RFCs had typos and 
they don't count.

---

From: Sean Leonard <[email-masked]>
Date: Mon, 20 Nov 2017 01:17:18 -0800

With respect to the original topic about end-of-line, I think the 
solution has only 10% to do with ABNF per-se, and 90% to do with the 
surrounding computing environment. Therefore, I have come up with the 
following conventions based on inspection of a large quantity of RFCs:

Some protocols are binary-oriented (unit is based on octet), others are 
text-oriented (unit is based on code point, e.g., ASCII, or Unicode/UTF-8).

Within the text-oriented protocols, some are line-oriented and some are 
whitespace-oriented. (Some are oriented towards both.)

CRLF is the Internet standard newline. (RFC 5198) LF is the XML standard 
newline. (https://www.w3.org/TR/xml/ Section 2.11) CRLF is the Windows 
standard newline. LF is the Unix standard newline. (No citations 
necessary.) And etc.

A format is a "protocol" for our purposes if the format is sent over the 
Internet. Specifically, RFC 2046 says:

    The canonical form of any MIME "text" subtype MUST always represent a
    line break as a CRLF sequence.  Similarly, any occurrence of CRLF in
    MIME "text" MUST represent a line break.  Use of CR and LF outside of
    line break sequences is also forbidden.

Therefore:
When ABNF is used to describe a line-oriented protocol that has to do 
with the Internet, it ought to use <CRLF>.

When ABNF is used to describe a line-oriented protocol that does not 
have to do with the Internet, it can define line markers "to taste"; a 
common convention is <EOL>. But don't define EOL = CRLF because there is 
no point. EOL is an indication to the reader that something is going on 
with end-of-line markers, but there needs to be a good reason. Don't 
redefine CRLF (or LF for that matter) either, because that confuses readers.

When ABNF is used to describe a whitespace-oriented (but not 
line-oriented) protocol, it is acceptable to define whitespace as a glob 
of any of SP, HTAB, CR, and LF. A key example is JSON RFC 7159, which is 
whitespace-oriented, but does not care about lines. It is media-typed as 
application/json, not text/json, for that and related reasons.

When ABNF is used to describe a binary protocol, do whatever, but don't 
use <CRLF> or <EOL> rule names in a binary protocol definition since 
those conventions imply a text-oriented protocol.

When in doubt, ask: "If this format were to be dumped into a MIME part, 
would this format foreseeably be transmitted as text/* such as 
text/plain or message/* such as message/http? Or is it going to have to 
be transmitted as application/* such as application/xml?" And there's a 
reasonable guide.

If there is disagreement about line endings and it matters to the 
protocol, say something about that during "preprocessing", rather than 
in the (A)BNF definition. Example is CSS3: Section 3.3 of 
<https://www.w3.org/TR/css-syntax-3/>.

Regards,

Sean

On 11/19/2017 10:56 AM, Dave Crocker wrote:
>> To add some color to this point, “cuts” was discussed in the CBOR WG 
>> in the context of CDDL. It is a technique from Parsing Expression 
>> Grammars. Here is overview:
> ...
>> However, PEGs are ordered; ABNF is unordered. With ABNF (as presently 
>> constituted), all alternatives in a choice are considered 
>> simultaneously (order is not relevant). Even if you match one 
>> alternative, you’re supposed to try all other alternatives.
>
>
> turned out this was not due to the email portion but folk were 
> re-using the ABNF meta-specification, which is why the later, revision 
> effort to RFC 822 split the ABNF text out into its own RFC.)
>
> tool.  But while folk have often wanted to enhance ABNF, over the 
> years, the 'let's define a new variant' tendency mostly died out -- 
> ignoring the much more recent move towards JSON... Why did this 
> popularity happen?
>
> complexity.  Enough but not too much, of each.  ABNF seemed to strike 
> a good balance.  (I like to think the documentation clarity in RFC 733 
> also helped, but then I'm quite biased about this, given how much 
> effort I put into that aspect of the work...)
>
> I think the biggest danger in creating a meta-language is 
> specification obscurity.  The tendency to want to add features can 
> too-easily create too much complexity for easy human comprehension.  
> The result is that seemingly-simple specifications can too-easily have 
> implications that are not understood by most readers.
>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 20 Nov 2017 12:35:30 -0500

> (That would probably need the XML, with properly marked up [type=“abnf”] artwork elements.  But we could encourage submitting the XML for drafts that cite RFC 5234 normatively.)

Yes, and I gather that is what is being planned for the future. This 
will work ok if the ABNF is complete and self contained. But there are 
large numbers of important RFCs where it isn't, and in a practical sense 
can't be. For instance RFC6665 is an extension to RFC3261. It includes 
the following as a prelude to the ABNF it includes:

    The Augmented BNF [RFC5234] definitions for the various new and
    modified syntax elements follows.  The notation is as used in
    [RFC3261], and any elements not defined in this section are as
    defined in SIP and the documents to which it refers.

The end result of this is that if you attempt to verify the syntax in a 
standalone way all of the referenced rules are undefined.

(This RFC has the additional issue of using '=' to redefine a rule 
defined in RFC3261. This is historic - it is a revision of RFC3265 that 
was defined prior to the addition of '=/' to ABNF.

There are large number of RFCs that use techniques something like this. 
More modern ones use '=/' and sometimes do something like:

	foo = <defined in RFCxxxx>

That construct eliminates the undefined rules, but the verification then 
fails to check lots of other things.

Another issue is that there is ambiguity about the status of the Core 
Rules defined in RFC5234. Can anyone writing an ABNF grammar assume that 
the core rules are predefined for their use? Or must the grammar 
indicate specifically (how?) that it is implicitly including their 
definitions? If they are automatically present, then they are reserved 
rule names, and cannot be defined differently in any ABNF grammar.

These are impediments to thorough verification of ABNF syntax.

	Thanks,
	Paul

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 20 Nov 2017 18:45:18 +0100

Hi Paul,

I think these are good observations for a future FDT (ABNFplus :-) or CDDL 2.0).

> Yes, and I gather that is what is being planned for the future. This will work ok if the ABNF is complete and self contained. But there are large numbers of important RFCs where it isn't, and in a practical sense can't be. For instance RFC6665 is an extension to RFC3261. It includes the following as a prelude to the ABNF it includes:
> 

Of course, YANG solves this by defining an import/export interface.
ABNF could grow a convention that makes something like

> 	foo = <defined in RFCxxxx>

actually well-defined.
(We have discussed this for CDDL and didn’t want to include this in this round.)

> Another issue is that there is ambiguity about the status of the Core Rules defined in RFC5234. Can anyone writing an ABNF grammar assume that the core rules are predefined for their use? Or must the grammar indicate specifically (how?) that it is implicitly including their definitions? If they are automatically present, then they are reserved rule names, and cannot be defined differently in any ABNF grammar.

Right.  In CDDL, we have a defined “prelude” that is added to all specifications, effectively becoming a set of reserved names.  ABNF could use the above convention with RFC 5234 as the source, or grow a wholesale “import RFC5234” style statement.

Grüße, Carsten

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 20 Nov 2017 13:04:37 -0500

> Right.  In CDDL, we have a defined “prelude” that is added to all specifications, effectively becoming a set of reserved names.  ABNF could use the above convention with RFC 5234 as the source, or…

Indeed it is possible to address these issues with some extensions to 
ABNF. But there is resistance to making extensions to ABNF for fear of 
making it too complex or "formal" for users.

What I'm trying to do here is make the case that there are some issues 
that need solutions. If we can get agreement on those then we can 
discuss how to do so without confusing users.

	Thanks,
	Paul