Subject: [Cbor] [Technical Errata Reported] RFC8610 (6278)

From: RFC Errata System <[email-masked]>
Date: Thu, 3 Sep 2020 17:17:02 -0700 (PDT)

The following errata report has been submitted for RFC8610,
"Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures".

--------------------------------------
You may review the report below and at:
https://www.rfc-editor.org/errata/eid6278

--------------------------------------
Type: Technical
Reported by: Eric Seppanen <[email-masked]>

Section: Appendix B

Original Text
-------------
BCHAR = %x20-26 / %x28-5B / %x5D-10FFFD / SESC / CRLF

Corrected Text
--------------
BCHAR = %x20-26 / %x28-5B / %x5D-7E / %x80-10FFFD / SESC / CRLF

Notes
-----
Between draft 6 and 7 of the RFC, the ASCII DELETE character 0x7F was excluded from the character set allowed in text literals.  I believe it should also be excluded from the character set of bytestring literals.

A 0x7F byte could not be decoded if included in a hex or base64 bytestring, leaving only UTF-8 bytestrings.  Since there is no discussion in the text of any reason to allow this character in a UTF-8 bytestring when it was not allowed in a UTF-8 text string, I suspect this was an oversight.

Instructions:
-------------
This erratum is currently posted as "Reported". If necessary, please
use "Reply All" to discuss whether it should be verified or
rejected. When a decision is reached, the verifying party  
can log in to change the status and edit the report, if necessary. 

--------------------------------------
RFC8610 (draft-ietf-cbor-cddl-08)
--------------------------------------
Title               : Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures
Publication Date    : June 2019
Author(s)           : H. Birkholz, C. Vigano, C. Bormann
Category            : PROPOSED STANDARD
Source              : Concise Binary Object Representation Maintenance and Extensions
Area                : Applications and Real-Time
Stream              : IETF
Verifying Party     : IESG

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 4 Sep 2020 02:42:07 +0200

> Between draft 6 and 7 of the RFC, the ASCII DELETE character 0x7F was excluded from the character set allowed in text literals.  I believe it should also be excluded from the character set of bytes…

I agree.  DEL (0x7F, also known as rubout) was removed based on a comment from Adam Roach:

https://mailarchive.ietf.org/arch/msg/cbor/jzJ297jNm5EoJkJj-n8R43QSOP4/
(Search for “7F”)

Which we implemented in:

https://github.com/cbor-wg/cddl/commit/bf3946

… apparently missing out on BCHAR, as Adam also happened to do.
(Note that BCHAR had gained 0x7F with
https://github.com/cbor-wg/cddl/commit/33b98a
where we added beyond-ASCII characters; this also gave it to SCHAR as well as SESC.)

Note that we never worried to exclude C1 control characters or other characters that might look weird in a specification (trying to do that exhaustively is just a bottomless pit).  So I believe the practical effect of missing out excluding 0x7F from BCHAR is near nil (except maybe in an obfuscated CDDL contest).
But it sure would be nice to be consistent (even if it is not *required* to be consistent here).

My verdict would be: Hold for document update.

Grüße, Carsten

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 3 Sep 2020 20:06:42 -0600

> Corrected Text

Actually, if the range is meant to end at U+10FFFD (thus excluding the noncharacters U+10FFFE and U+10FFFF), then it should also exclude the following:

1. the U+xxFFFE and U+xxFFFF noncharacters in every other plane
2. noncharacters U+FDD0 through U+FDEF
3. surrogate code points 0xD800 through 0xDFFF, which aren't even Unicode scalar values (hence not marked with "U+" notation)

Of course this goes for the definitions of PCHAR, SCHAR, and SESC as well.

A new symbol, NONASCII, could resolve all this:

NONASCII = %x80-D7FF / %xE000-FDCF / %xFDF0-FFFD
         / %x10000-1FFFD
         / %x20000-2FFFD
         / %x30000-3FFFD
         / %x40000-4FFFD
         / %x50000-5FFFD
         / %x60000-6FFFD
         / %x70000-7FFFD
         / %x80000-8FFFD
         / %x90000-9FFFD
         / %xA0000-AFFFD
         / %xB0000-BFFFD
         / %xC0000-CFFFD
         / %xD0000-DFFFD
         / %xE0000-EFFFD
         / %xF0000-FFFFD
         / %x100000-10FFFD
[...]
SCHAR = %x20-21 / %x23-5B / %x5D-7E / NONASCII / SESC
SESC = "\" (%x20-7E / NONASCII)
[...]
BCHAR = %x20-26 / %x28-5B / %x5D-7E / NONASCII / SESC / CRLF
[...]
PCHAR = %x20-7E / NONASCII

--
Doug Ewell | Thornton, CO, US | ewellic.org

---

From: Barry Leiba <[email-masked]>
Date: Fri, 4 Sep 2020 15:22:27 -0400

Still more reason why it's HFDU: the fix for this isn't
straightforward, and should be handled as part of a revision.

Barry

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 4 Sep 2020 21:42:06 +0200

The below is exactly what should never happen: having to encode detailed (and somewhat arcane) knowledge about a standard in the bowels of another, mostly unrelated standard. I think we hit the right balance  with VCHAR and SCHAR, we just forget to fix BCHAR.  But we can discuss that when we do the bis...

Sent from mobile, sorry for terse

> On 4. Sep 2020, at 05:03, Doug Ewell <[email-masked]> wrote:
> 
> 
>> 
>> Corrected Text
>> --------------

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 04 Sep 2020 21:28:28 -0400

Carsten Bormann <[email-masked]> writes:
> The below is exactly what should never happen: having to encode
> detailed (and somewhat arcane) knowledge about a standard in the
> bowels of another, mostly unrelated standard.

Idealistically, that suggests that the one standard (Unicode) should
define an "exported concept" that is what the other standard (CBOR)
needs to refer to without further complication.  The natural consequent
of that is to ask whether Unicode might already define a term which is
what CBOR needs, but given the people who have read this message thread,
if it existed, it would already have been mentioned.

Dale

---

From: "Doug Ewell" <[email-masked]>
Date: Sat, 5 Sep 2020 12:51:05 -0600

Carsten Bormann wrote:

> bowels of another, mostly unrelated standard. I think we hit the right
> balance  with VCHAR and SCHAR, we just forget to fix BCHAR.  But we
> can discuss that when we do the bis...

To be sure, it should not be necessary to specify this stuff in CBOR. This and other Unicode-specific rules should be part of the ABNF spec (Section B.1, "Core Rules") to go with all the ASCII-specific rules already there.

Ending the range of allowable characters at 10FFFD solves only a tiny part of the "exclude noncharacters" problem. If we don't want to solve the whole problem, then don't solve any of it: just end the range at 10FFFF and pretend we don't know about noncharacters.

For that matter, even the value 10FFFD (or 10FFFF) can be considered detailed (and somewhat arcane) knowledge about Unicode. Removing that is left as an exercise.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 6 Sep 2020 07:22:45 +0200

We didn’t set out to mirror all the exclusion of Unicode in the ABNF. 
But the range has to end somewhere, and we happen to know the largest number is 10FFFD, so we took that.  Again, I think we struck the right balance. 

> On 5. Sep 2020, at 21:20, Doug Ewell <[email-masked]> wrote:
>