Subject: Re: [Cbor] [Technical Errata Reported] RFC8610 (6527)

From: Carsten Bormann <[email-masked]>
Date: Sun, 11 Apr 2021 18:51:16 +0200

Hi Sean,

thank you for submitting this errata report.

Without going into all the details here:

CDDL imports JSON syntax (Section 7 of RFC 8259) for text strings, and that has been somewhat simplified in the ABNF (which allows more than the RFC 8259 rules then actually allow).
However, there is one omission in the ABNF: the \u syntax, which is somewhat complicated in JSON because it is followed either by 4 hex digits that are not in the range d800 to dfff or by 4 hex digits in the range d800 to dbff, another \u, and four more hex digits dc00 to dfff.
Someone needs to sit down and write up the ABNF for that (or find some ABNF that already has done the work).

I’m not sure this update should all be put into an errata item; maybe we should pursue writing this up in a document that updates RFC 8610 that could then also add a less unwieldy syntax for non-BMP code points such as the ubiquitous \u{…}.

It seems people haven’t used a lot of \u notation with CDDL before: except for the C0 control characters not covered by JSON escapes, the literal character usually works well enough.

Grüße, Carsten

> On 2021-04-11, at 18:10, RFC Errata System <[email-masked]> wrote:
> 
> The following errata report has been submitted for RFC8610,
> "Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures".
> 
> --------------------------------------
> You may review the report below and at:
> https://www.rfc-editor.org/errata/eid6527
> 
> --------------------------------------
> Type: Technical
> Reported by: Sean Bartell <[email-masked]>
> 
> Section: B
> 
> Original Text
> -------------
>     SESC = "\" (%x20-7E / %x80-10FFFD)
> 
> Corrected Text
> --------------
>     SESC = "\" (%x22 / %x27 / %x2F / %x5C / %x62 / %x66 / %x6E / %x72 / %x74 / %x75 4HEXDIG )
> 
> Notes
> -----
> The ABNF rule for string escape codes should specify which escape codes are allowed, and also specify that "\u" must be followed by four hex digits. My suggested correction is based on JSON, but with the addition of "\'" (single quote) for use in text-based byte strings (section G.2). (Presumably, characters outside the BMP must be escaped using a surrogate pair, as in JSON.)
> 
> Instructions:
> -------------
> This erratum is currently posted as "Reported". If necessary, please
> use "Reply All" to discuss whether it should be verified or
> rejected. When a decision is reached, the verifying party  
> can log in to change the status and edit the report, if necessary. 
> 
> --------------------------------------
> RFC8610 (draft-ietf-cbor-cddl-08)
> --------------------------------------
> Title               : Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures
> Publication Date    : June 2019
> Author(s)           : H. Birkholz, C. Vigano, C. Bormann
> Category            : PROPOSED STANDARD
> Source              : Concise Binary Object Representation Maintenance and Extensions
> Area                : Applications and Real-Time
> Stream              : IETF
> Verifying Party     : IESG

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 13 Apr 2021 14:13:22 +0200

> Someone needs to sit down and write up the ABNF for that (or find some ABNF that already has done the work).

I was too lazy to find some ABNF so I wrote my own.

76c76,84
< SESC = "\" (%x20-7E / %x80-10FFFD)
---
> 
> SESC = "\" ( %x22 / %x2F / %x5C / %x62 / %x66 / %x6E / %x72 / %x74 /
>              (%x75 hexchar) )
> 
> hexchar = ((DIGIT / "A"/"B"/"C" / "E"/"F") 3HEXDIG) /
>            ("D"
>             (( %x30-37 2HEXDIG ) /
>              (("8"/"9"/"A"/"B") 2HEXDIG "\" %x75 "D"
>               ("C"/"D"/"E"/"F") 2HEXDIG )))
79c87
< BCHAR = %x20-26 / %x28-5B / %x5D-10FFFD / SESC / CRLF
---
> BCHAR = %x20-26 / %x28-5B / %x5D-10FFFD / SESC / "\'" / CRLF

The second change is necessary as SESC is now narrowly restricted to the escape combinations that JSON allows in text strings, and \' is not among those.

You can play with this changed grammar in cddlc version 0.0.3 (`gem update` if needed).

> I’m not sure this update should all be put into an errata item; maybe we should pursue writing this up in a document that updates RFC 8610 that could then also add a less unwieldy syntax for non-BM…

I’m still not entirely sure we want to handle this as a bog-standard errata item.
Opinions welcome.

Grüße, Carsten

---

From: Barry Leiba <[email-masked]>
Date: Tue, 13 Apr 2021 09:07:18 -0400

Indeed, this one screams "Hold for Document Update" to me, as it's
something that rather more complex.  It's clear that this isn't just
an "oops" when we published, but, rather a (valid) rethinking that
needs broader review and consensus on the solution.

Barry

---

From: Francesca Palombini <[email-masked]>
Date: Tue, 13 Apr 2021 13:29:49 +0000

I agree with Barry in that this seems to require more thinking and broader review and consensus. Carsten, you seem to imply that this should instead be rejected, with the motivation that it would need a new document (updating 8610), is that right ?

(For everybody not familiar with the errata status definition: https://www.rfc-editor.org/errata-definitions/
 Hold for document update: “The erratum is not a necessary update to the RFC. However, any future update of the document might consider this erratum, and determine whether it is correct and merits including in the update.”
Rejected: “The erratum [...] proposes a change to the RFC that should be done by publishing a new RFC that replaces the current RFC. In the latter case, if the change is to be considered for future updates of the document, it should be proposed using channels other than the errata process, such as a WG mailing list.”)

Francesca

On 13/04/2021, 15:07, "Barry Leiba" <[email-masked]> wrote:

    Barry

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 13 Apr 2021 16:03:31 +0200

Hi Francesca,

I must admit that I’m not really comfortable with the errata process.

This errata report points to a problem with the ABNF definition of CDDL that keeps it from upholding the promise in Section 3.1 (top of page 16).
(Note that the note in the same paragraph is also about ABNF, but about how CDDL is different from ABNF, not about details of using the JSON string form and describing that in ABNF.  This stuff is all so meta...)

So is an update necessary?  The problem does not break existing specifications, as far as I know.  I would expect that implementers of CDDL will want to do something similar to what I did for the cddlc tool; the conflict between ABNF spec and text needs to be resolved in some way and a search for a minimal way to do so will come up with a solution similar to mine.
It is less clear that this minimal way is what we actually want to limit ourselves to.
But we *could* apply this hot fix and think about better ways to handle literals later, in CDDL 2.0 — draft-bormann-cbor-cddl-freezer has a Section 3 about literal syntax.

On the other hand, a workaround for this gap will be available with .cat, which I think we will be moving forward even faster than the next CDDL version.  Having an errata report that is not just rejected but acknowledges the problem and possibly points to this workaround for now would be helpful to users.

Grüße, Carsten

> On 2021-04-13, at 15:29, Francesca Palombini <francesca.palombini=[email-masked]> wrote:
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: "Sean Bartell" <[email-masked]>
Date: Tue, 13 Apr 2021 18:54:15 -0500

> >> The second change is necessary as SESC is now narrowly restricted to the escape combinations that JSON allows in text strings, and \' is not among those.

This ABNF looks correct to me, but it obfuscates more than it clarifies. I would suggest following the example of the JSON RFC (https://www.rfc-editor.org/rfc/rfc8259.html#section-7). The ABNF there allows arbitrary hex digits (%x75 4HEXDIG), but the text clarifies how non-BMP characters are handled. A similar situation already exists for base64 bytestrings, for which CDDL's ABNF allows invalid base64.

It's also not clear to me whether \' should be allowed in text strings, or whether \" should be allowed in byte strings. Allowing them would make both types of strings consistent with each other, but prohibiting them would let text strings use the exact same syntax as JSON. And one more issue: the ABNF currently allows CRLF in unprefixed byte strings, and I'm not sure whether that's intentional.

> > (For everybody not familiar with the errata status definition: https://urldefense.com/v3/__https://www.rfc-editor.org/errata-definitions/__;!!DZ3fjg!soSEF4XBhOUPRdN5wSzvAeVKOyu_4HkZe_d06IEV4huYgnyqgimjYJH9OeHL0LITxA8$ 

More guidance on the errata process (https://www.ietf.org/about/groups/iesg/statements/processing-rfc-errata/): "Changes that modify the working of a protocol to something that might be different from the intended consensus when the document was approved should be either Hold for Document Update or Rejected. Deciding between these two depends on judgment. Changes that are clearly modifications to the intended consensus, or involve large textual changes, should be Rejected. In unclear situations, small changes can be Hold for Document Update."

As I interpret it, if this is a "small change", and there was no clear "intended consensus" about which escape codes should be allowed, the erratum can be marked as "Hold for Document Update".

FWIW, I'm not actually using CDDL at the moment. I'm using the extended diagnostic notation, and I was using the CDDL ABNF to narrow down my choice of syntax.

Sean

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 14 Apr 2021 07:53:50 +0200

> This ABNF looks correct to me, but it obfuscates more than it clarifies.

I don’t agree — it is a very clear statement of what is allowed.

> I would suggest following the example of the JSON RFC (https://www.rfc-editor.org/rfc/rfc8259.html#section-7). The ABNF there allows arbitrary hex digits (%x75 4HEXDIG), but the text clarifies how non-BMP characters are handled.

RFC 8259 is a product of a process that was more political than technical.  There was a lot of discussion of unpaired surrogates etc., spilling over from UTF-16 land.
There was always a danger of ECMA bursting out of the fragile agreement to align ECMA 404 and RFC 8259, and that precarious situation limited the amount of fixing that could be done between 7159 and 8259.
While RFC 8259 makes it clear that the JSON text itself needs to be UTF-8, it wasn't really possible to summon the courage to write text that explicitly disallows constructing invalid text from escapes in text string data items.

To summarize, RFC 8259 is in many ways not a good model for how to specify things.

As my ABNF proposal shows, it is easy to specify how surrogates can be used in the ABNF, and doing that saves a validation process step in the implementation.

> A similar situation already exists for base64 bytestrings, for which CDDL's ABNF allows invalid base64.

Yes, but it is much harder to do an ABNF that combines all the syntax for whitespace, comments, *and* that for base64.

> It's also not clear to me whether \' should be allowed in text strings,

The text in section 3.1 doesn’t support allowing that:

   o  Text strings are enclosed by double quotation '"' characters.
      They follow the conventions for strings as defined in Section 7 of
      [RFC8259].  […]

> or whether \" should be allowed in byte strings.

It is:

      If unprefixed, the string is
      interpreted as with a text string, except that single quotes must
      be escaped and that the resulting UTF-8 bytes are marked as a byte
      string (major type 2).

Note the absence of a statement like “double quotes must not be escaped”.

> Allowing them would make both types of strings consistent with each other, but prohibiting them would let text strings use the exact same syntax as JSON.

JSON doesn’t have byte strings, but for text strings we indeed should have full alignment.

> And one more issue: the ABNF currently allows CRLF in unprefixed byte strings, and I'm not sure whether that's intentional.

It is.
(And that is important for draft-ietf-cbor-cddl-control.)

> FWIW, I'm not actually using CDDL at the moment. I'm using the extended diagnostic notation, and I was using the CDDL ABNF to narrow down my choice of syntax.

Maybe providing ABNF for EDN would be a good idea, after all.
It *is* subtly different in a number of details from CDDL syntax.
One formal interpretation of EDN syntax (albeit not in ABNF form, as it predates the abnftt tool) can be found in https://github.com/cabo/cbor-diag/blob/master/lib/cbor-diag-parser.treetop (and, yes, this contains the equivalent of the surrogate pair handling that is in the ABNF I proposed).

Grüße, Carsten

---

From: RFC Errata System <[email-masked]>
Date: Sun, 11 Apr 2021 09:10:45 -0700 (PDT)

Section: B

Original Text
-------------

Corrected Text
--------------

Notes
-----

Instructions:
-------------

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 14 Apr 2021 08:25:38 -0600

> I don’t agree — it is a very clear statement of what is allowed.

I'd like to see separate ABNF definitions for UNICODE_SCALAR_VALUE, HIGH_SURROGATE, and LOW_SURROGATE, and then see 'hexchar' defined in terms of those. I think that would make it clearer what is going on.

--
Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 14 Apr 2021 08:29:27 -0600

I just wrote:

> I'd like to see separate ABNF definitions for UNICODE_SCALAR_VALUE,
> HIGH_SURROGATE, and LOW_SURROGATE, and then see 'hexchar' defined in
> terms of those. I think that would make it clearer what is going on.

Of course it can't be UNICODE_SCALAR_VALUE, but something like NONSURROGATE that conveys the meaning "any 4-digit hex value excluding surrogates."

---

From: "Sean Bartell" <[email-masked]>
Date: Wed, 14 Apr 2021 09:53:02 -0500

On Wed, Apr 14, 2021, at 09:29, Doug Ewell wrote:
> Of course it can't be UNICODE_SCALAR_VALUE, but something like 
> NONSURROGATE that conveys the meaning "any 4-digit hex value excluding 
> surrogates."

I'd be happy with handling surrogate pairs this way; it's much clearer than putting everything in one rule. Other than that, I don't have any particular opinion about Carsten's proposals.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 14 Apr 2021 18:18:19 +0200

Nice!

hexchar = non-surrogate / (high-surrogate "\" %x75 low-surrogate)
non-surrogate = ((DIGIT / "A"/"B"/"C" / "E"/"F") 3HEXDIG) /
                ("D" %x30-37 2HEXDIG )
high-surrogate = "D" ("8"/"9"/"A"/"B") 2HEXDIG
low-surrogate = "D" ("C"/"D"/"E"/"F") 2HEXDIG

Now in use in
https://github.com/cabo/cddlc/blob/master/lib/parser/cddl.abnf.orig

Grüße, Carsten

> On 2021-04-14, at 16:29, Doug Ewell <[email-masked]> wrote:
> 
> I just wrote:
>