Subject: [Cbor] Feature requests for CBOR and CDDL

From: Jeffrey Yasskin <[email-masked]>
Date: Tue, 28 Mar 2017 09:44:47 -0700

In the course of writing https://github.com/dimich-g/webpackage/pull/36, I
ran into the following issues (in addition to the ones Carsten forwarded
yesterday). Please only give them the weight of personal thoughts, since
the PR hasn't been accepted by the other folks working on the web-package
format yet.

   1. Re
   https://tools.ietf.org/html/draft-bormann-cbor-tags-oid-06#section-13,
   https://github.com/dimich-g/webpackage/pull/36 wants to embed
   certificates into CBOR data. At the moment, I'm just saying they're X.509
   binary strings, but if there were a standard way to embed them, I'd use
   that.
   2. The web packaging spec wants to forbid indefinite-length encodings in
   several places, for example to facilitate using a section of mmapped CBOR
   data as another file type. It'd be nice to have a way to say that in CDDL.
   3. The web packaging spec wants to prefix several things with their byte
   lengths in order to make it easier to skip parsing uninteresting
   structures. This is doable by specifying things as "bstr .cbor real-type"
   or "#6.24(bstr) .cbor real-type", but those expressions don't really
   capture the intent.
   4. I've been using ".within" annotations to assert that an extensible
   structure has a particular generic expression, but they turn out to be
   inconvenient to use as expressions.
   https://w3c.github.io/webauthn/#generating-an-attestation-object's
   misuses .within as a statement (and also gets it backwards, IIUC), and this
   would be more convenient for my uses too.

Jeffrey

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 28 Mar 2017 12:17:36 -0500

Hi Jeffrey,

> On 28 Mar 2017, at 11:44, Jeffrey Yasskin <[email-masked]> wrote:
> 
> In the course of writing https://github.com/dimich-g/webpackage/pull/36, I ran into the following issues (in addition to the ones Carsten forwarded yesterday). Please only give them the weight of personal thoughts, since the PR hasn't been accepted by the other folks working on the web-package format yet.

Many thanks for making these needs known.

Let me try to provide my view, or possibly restate the objective in a way that we can immediately acted on.

> 	• Re https://tools.ietf.org/html/draft-bormann-cbor-tags-oid-06#section-13, https://github.com/dimich-g/webpackage/pull/36 wants to embed certificates into CBOR data. At the moment, I'm just saying they're X.509 binary strings, but if there were a standard way to embed them, I'd use that.

There are two ways to do this in CBOR:

1) define a tag.  This is, e.g. proposed for ASN.1 object IDs, and of course could be done for other common data structures.

2) use the application layer structure you are building to indicate how to interpret the byte string (e.g., by putting it into a map with a certain field name).  This saves expending a couple of bytes for the tag (pretty much irrelevant for an X.509 cert), and may be sufficient if you are not expecting special handling from the CBOR decoder for X.509 certs.

… on to CDDL:

> 	• The web packaging spec wants to forbid indefinite-length encodings in several places, for example to facilitate using a section of mmapped CBOR data as another file type. It'd be nice to have a way to say that in CDDL.

This class of “encoding hints" is hinted at in the CDDL spec, but CDDL does not do much there.
We do have float vs. float16 vs. float32 vs. float64, but the current tool does not actually act on this (encoding is transparent for the CDDL tool).

Syntactically, definite/indefinite would need to label strings, arrays and maps as to whether indefinite encoding is fine.  This, of course, should be done in a non-intrusive way.

So far, most of the constraints of this type have been global, i.e. an additional sentence in the spec says “definite only” or “float16 or float32 only", and there hasn’t been a perceived need to actually identify this in individual elements of the grammar defined by CDDL.  If that starts to be necessary, CDDL indeed would need to grow a feature.

> 	• The web packaging spec wants to prefix several things with their byte lengths in order to make it easier to skip parsing uninteresting structures. This is doable by specifying things as "bstr .cbor real-type" or "#6.24(bstr) .cbor real-type", but those expressions don't really capture the intent.

bstr .cbor embedded-type
is indeed the way to do this.
In which way does this not capture the intent?

(Maybe we should define a standard generic of the form
embedded<T> = bstr .cbor T
in the prelude, so there is a more common, more declarative way to do this.)

> 	• I've been using ".within" annotations to assert that an extensible structure has a particular generic expression, but they turn out to be inconvenient to use as expressions. https://w3c.github.io/webauthn/#generating-an-attestation-object's misuses .within as a statement (and also gets it backwards, IIUC), and this would be more convenient for my uses too.

Indeed, the reason that these operators are called annotations is that they often*) add information to what would be a self-sufficient grammar without them, and it would be convenient to turn them into a separate construct (“statement”).  This has to be balanced against complicating the overall structure of CDDL by providing this additional kind of statement.  I don’t have an opinion what should be done here; maybe developing a proposal for statement-type annotations would be the next step to better be able to gauge the impact.

Grüße, Carsten

*) Annotations also are a genuine extension point of CDDL, which can be used for any two-argument operator.

---

From: Sean Leonard <[email-masked]>
Date: Tue, 28 Mar 2017 18:35:44 -0500

> On Mar 28, 2017, at 12:17 PM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Jeffrey,
> 

Yes, that is the essence of the issue that Section 13 discusses, and I think that is a function of what the sender is trying to describe, and what foreseeable receivers are expected to do or understand with less than specific programming.

All tags and other metadata can be omitted in CBOR. The same can be said for MIME types: all blobs can be transmitted as application/octet-stream. But, you get less functionality on the receiving end unless the receiver is specifically programmed to do a specific thing in a protocol flow without that metadata. In the case of embedding certificates into CBOR data, what would you like or want a receiver to do that is not fully aware of the web-package schema, but is aware of the global meaning of CBOR tags? A debugger can cue off the CBOR tag (or the mechanism that Section 13 proposes, which is fancy metadata) to show something useful beyond just a blob of bytes.

Regards,

Sean

PS Object Identifiers are not specific to ASN.1.

---

From: Jeffrey Yasskin <[email-masked]>
Date: Wed, 29 Mar 2017 16:04:31 -0700

On Tue, Mar 28, 2017 at 10:17 AM, Carsten Bormann <[email-masked]> wrote:

> > In the course of writing https://github.com/dimich-g/webpackage/pull/36,
> I ran into the following issues (in addition to the ones Carsten forwarded
> >       • Re https://tools.ietf.org/html/draft-bormann-cbor-tags-oid-
> 06#section-13, https://github.com/dimich-g/webpackage/pull/36 wants to
> embed certificates into CBOR data. At the moment, I'm just saying they're
> X.509 binary strings, but if there were a standard way to embed them, I'd
> use that.
>
> 1) define a tag.  This is, e.g. proposed for ASN.1 object IDs, and of
> course could be done for other common data structures.
>
> 2) use the application layer structure you are building to indicate how to
> interpret the byte string (e.g., by putting it into a map with a certain
> field name).  This saves expending a couple of bytes for the tag (pretty
> much irrelevant for an X.509 cert), and may be sufficient if you are not
> expecting special handling from the CBOR decoder for X.509 certs.
>

Yep, (2) is basically what I'm doing now, and (1) is the fix I imagine. I
only imagine (1) helping with automated debugging interfaces -- the
application itself knows what to expect. It's totally possible that I've
been too eager to use tags, and that I should mostly avoid them when
designing formats on top of CBOR.

… on to CDDL:
>
> >       • The web packaging spec wants to forbid indefinite-length
> encodings in several places, for example to facilitate using a section of
> mmapped CBOR data as another file type. It'd be nice to have a way to say
> that in CDDL.
>
> We do have float vs. float16 vs. float32 vs. float64, but the current tool
> does not actually act on this (encoding is transparent for the CDDL tool).
>
> Syntactically, definite/indefinite would need to label strings, arrays and
> maps as to whether indefinite encoding is fine.  This, of course, should be
> done in a non-intrusive way.
>
> So far, most of the constraints of this type have been global, i.e. an
> additional sentence in the spec says “definite only” or “float16 or float32
> only", and there hasn’t been a perceived need to actually identify this in
> individual elements of the grammar defined by CDDL.  If that starts to be
> necessary, CDDL indeed would need to grow a feature.
>

Yep, a global constraint is enough for my spec, possibly with local
exceptions in some cases. We'll need a custom validity checker anyway
because of some other constraints, but it'd be nice to take advantage of
the standard one as much as possible.

>       • The web packaging spec wants to prefix several things with their
> byte lengths in order to make it easier to skip parsing uninteresting

I'm using bstr encoding to provide an explicit byte-length. COSE uses bstr
encoding in payloads to "guarantee" that a particular byte sequence isn't
changed by intermediate processing. There might be other purposes for
wrapping something into a bstr, and .cbor doesn't distinguish them.

It's not a terrible problem, and I'm not going to be upset if y'all decide
not to change anything here, or just to provide a generic.

> >       • I've been using ".within" annotations to assert that an
> extensible structure has a particular generic expression, but they turn out
> to be inconvenient to use as expressions. https://w3c.github.io/
> webauthn/#generating-an-attestation-object's misuses .within as a
> statement (and also gets it backwards, IIUC), and this would be more
> convenient for my uses too.
>
> Indeed, the reason that these operators are called annotations is that
> they often*) add information to what would be a self-sufficient grammar
> without them, and it would be convenient to turn them into a separate
> construct (“statement”).  This has to be balanced against complicating the
> overall structure of CDDL by providing this additional kind of statement.
> I don’t have an opinion what should be done here; maybe developing a
> proposal for statement-type annotations would be the next step to better be
> able to gauge the impact.
>

I agree with the tradeoff you describe. I don't expect to write that
proposal myself.

Thanks,
Jeffrey

Grüße, Carsten