Subject: Re: [Cbor] Record proposal

From: Emile Cormier <[email-masked]>
Date: Wed, 1 Dec 2021 19:57:23 -0400

I'd like to point out that this proposal as it's currently written will
result in loss of information with CBOR decoders that don't understand the
tags, or when the data elements are transcoded to another protocol such as
JSON. This is due to the tags carrying information (record IDs) that the
application needs in order to make sense of the received payload. The
proposed tags do more than provide "semantic meaning"; they also carry
information.

This practice of bundling numeric information into the tags is making it so
that these CBOR items are not interoperable when transcoded to JSON (think
web browser), or when a CBOR decoder doesn't understand the tags and
doesn't propagate them up to the application.

On Mon, Nov 29, 2021 at 9:35 AM Kris Zyp <[email-masked]> wrote:

> To follow up, I did update the proposal/spec to use the tag id range
> of 57342-57599. Let me know if you would prefer different ids.
> Thanks,
> Kris
>
> On Wed, Nov 17, 2021 at 10:52 PM Kris Zyp <[email-masked]> wrote:
> >
> > That sounds reasonable to me.
> >
> > To clarify a little bit of the rationale for this:
> > The purpose of a global registry, as I understand it, is to define the
> > global tag ids that need to be known by independent encoders and
> > decoders prior to their encoding and decoding interaction. However,
> > ids that can be communicated within a message for the scope of that
> > message, don't require global allocation, if they are only used within
> > a specified scope. This is just like in a programming language: a
> > variable from an inner block may shadow another variable from an outer
> > block or a global scope temporarily within its block without affecting
> > the outer or global scope (this doesn't require any coordination with
> > a registry of globals). And a decoder that really implements this
> > proposal would, by definition, be implementing this behavior of
> > temporary tag reassignment/shadowing, if it really is a true record
> > implementation (i.e. a real record implementation wouldn't see
> > gobbledygook if it is conformant).
> >
> > Also allowing dynamic/temporary allocation permits potential use of
> > shorter/more efficient tag ids based on known use of other tags, and
> > in the case of record references, these may occur much more frequently
> > in a data item/document compared record definitions, and be much more
> > sensitive to size differences. Furthermore, this also sidesteps issues
> > with having to figure out the appropriate size of a range of ids to
> > globally allocate. How many ids should be allocated? In terms of
> > percentage of available space, registering/using 256 ids from the tag
> > 1+2 range seems like it is effectively equivalent as registering 1 id
> > from the tag 1+1 range (and tag 1+4 would definitely be undesirable
> > due to the high frequency and sensitivity to size). And I have had
> > users of my implementations say they need a few hundred structures to
> > be defined and referenceable, so even 256 may be somewhat limiting to
> > some users.
> >
> > But that being said, if you are thinking that it may be too onerous
> > for (record) implementators to support this type of potentially
> > dynamic/temporary tag reassignment with possibly conflicting ids (as
> > opposed to using a more stable table of tags or tag ranges), I can
> > understand that concern. What size of chunk of tag ids (above 32768)
> > do you think would be appropriate to allocate? Would a chunk of 256 be
> > reasonable?
> >
> > Anyway, thank you for your help and feedback!
> > Thanks,
> > Kris
> >
> >
> >
> > On Wed, Nov 17, 2021 at 1:52 PM Carsten Bormann <[email-masked]> wrote:
> > >
> > > Hi Kris,
> > >
> > > thank you for the update.
> > > I still need to take a closer look, but I have one immediate reaction:
> > > You probably shouldn’t induce implementations to “hijack” a tag number.
> > > The handling of a tag number that has actually been allocated for
> something else may be deeply embedded into the CBOR decoder; a record
> implementation then would only see gobbledygook.
> > > So I think we should allocate some space for tag-allocated tags — I
> don’t see a big problem with allocating a chunk above 32768 just for the
> record proposal.
> > >
> > > Grüße, Carsten
> > >
> > >
> > > > On 2021-11-17, at 16:16, Kris Zyp <[email-masked]> wrote:
> > > >
> > > > I have updated my tag registration proposal/submission at
> > > > https://github.com/kriszyp/cbor-records to use 1+2 tag entries,
> which
> > > > hopefully makes this a much less intrusive registry entry. I have
> also
> > > > updated the proposed tag definitions to also support up-front
> > > > declaration of a set of record structure definitions for a data item
> > > > ("around" that data item, like the packed approach, as you
> suggested),
> > > > in addition to the current inline record definitions (which can be
> > > > scoped with record definitions and should follow a well-specified
> > > > order for when they can be referenced by subsequent references). I
> > > > hope this offers flexibility for encoders that have all structures
> > > > known a-priori and streaming encoders (that do not), while still
> > > > maintaining nearly the same mechanics for decoders. Let me know if
> you
> > > > think this looks reasonable.
> > > > Thank you!
> > > > Kris
> > > >
> > > > On Thu, Nov 11, 2021 at 9:15 PM Kris Zyp <[email-masked]> wrote:
> > > >>
> > > >>> Actually, stats would be very interesting.
> > > >>> I was assuming that the 1+1 setup comes with a number of 1+2
> referencing records, the hit from going to 1+2 there as well would be
> relatively insignificant.
> > > >>> Number are better than assumptions!
> > > >>
> > > >> You are definitely right, using 1+2 tag for defining records is
> pretty
> > > >> insignificant (around a quarter of percent in my tests). Anyway, I
> put
> > > >> together some tests comparing CBOR packed, record structures, and
> > > >> combinations with a couple test data structures, with my library,
> for
> > > >> the sake of further comparisons:
> > > >> https://gist.github.com/kriszyp/b623b85d2dc25ac9e3b07d8f39df9307
> > > >>
> > > >> Anyway, seeing this, I am happy to go ahead and update my proposal
> to
> > > >> use a 1+2 tag for defining records. And thinking about this, I don't
> > > >> think my proposal necessarily even needs to mandate the tag ids used
> > > >> for referencing records since those are dynamically assigned and
> > > >> explicitly specified by the encoder itself (encoder obviously must
> not
> > > >> conflict and use tag ids that will be used for other purposes),
> albeit
> > > >> can encourage a certain range (presumably from the first-come
> > > >> first-serve range).
> > > >>
> > > >> Do you have any preference for a tag id to use? It looks like 279 is
> > > >> the next in the contiguous block, but sounds like choosing aesthetic
> > > >> characters is the new preference (29299/"rs" perhaps).
> > > >>
> > > >> Anyway, thanks again for the helpful feedback, really appreciate it!
> > > >>
> > > >> Thanks,
> > > >> Kris
> > > >
> > > > _______________________________________________
> > > > CBOR mailing list
> > > > [email-masked]
> > > > https://www.ietf.org/mailman/listinfo/cbor
> > >
>
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor
>

---

From: Kris Zyp <[email-masked]>
Date: Wed, 1 Dec 2021 22:34:37 -0700

I certainly empathize with the idea of keeping tags as conceptually
distinct in terms of just adding some extra non-transformational semantic
description to the underlying CBOR data structure. However, there is a wide
range of "meanings" in tags, and realistically, with many, or even most
tags, you can't really "make sense" of a payload without the "semantic
meaning"; the meaning is what provides the direction for how to make sense
of the data. For many tags, something needs to understand the data beyond
just a raw CBOR structure.

Without the semantic meaning, there isn't data loss; raw CBOR structures
could always be transcoded to JSON or anything else (with conventions for
tags and such), but those basic data structures could still be (and needs
to be) interpreted by a decoder/app that understands how the tags encode
things like the distinction between a record and map to properly translate
to various language constructs (failure to distinguish these and conflating
them all into a single JSON object or map would be the real data loss). And
likewise there are many other tags that also have significant meaning for
interpretation, such as the packed-CBOR tags, tags 1-5, 28/29, 100, 256/25,
and I'm sure many more, that do not make useful sense without understanding
the tags. Of course a decoder can always simply defer interpretation of
tags and decode CBOR to a set of tags, array, maps, etc., and leave the
interpretation to a higher level component.

Thanks,

Kris

*From: *Emile Cormier <[email-masked]>
*Sent: *December 1, 2021 4:57 PM
*To: *Kris Zyp <[email-masked]>
*Cc: *Carsten Bormann <[email-masked]>; Christian Amsüss
<[email-masked]>; [email-masked]
*Subject: *Re: [Cbor] Record proposal

> > > https://github.com/kriszyp/cbor-records to use 1+2 tag entries, which
> > > hopefully makes this a much less intrusive registry entry. I have also
> > > ("around" that data item, like the packed approach, as you suggested),
> > > maintaining nearly the same mechanics for decoders. Let me know if you
pretty
put
> > >> combinations with a couple test data structures, with my library, for
> > >> Anyway, seeing this, I am happy to go ahead and update my proposal to
not
albeit

---

From: Emile Cormier <[email-masked]>
Date: Thu, 2 Dec 2021 02:18:29 -0400

> just a raw CBOR structure.

But it should be possible for applications to agree in advance on the
meaning of the data and still be able to interpret it in the absence of
tags. This practice of embedding an extra number via a tag range is ruining
this ability for applications to agree in advance on the meaning of the
data in the absence of tags.

Your proposed tag range of 57344 - 57599 could simply be replaced by a
single tag followed by an array of size 2. The first element of that array
would be the record ID that would otherwise be encoded via the tag value.
In this way, an application expecting a record set would still be able to
interpret the data in the absence of tags, because the record IDs appear
explicitly in the data.

In section 3.4 of the spec, it says: "If a tag requires further structure
to its content, this structure is *provided by the enclosed data item*."
(emphasis mine). This clause is being violated by tags that rely on tag
range to convey additional data.

> tags and such)

Well, that's the problem: there is no convention for tags in JSON. A
CBOR-to-JSON converter now needs to understand your record set tags, and
generate a modified structure for JSON that contains the record IDs that
would otherwise be lost.

This practice of using a tag range to avoid the few measly bytes of a
two-element tuple is making it so that once an application adopts these
tags, it can never switch to another encoding other than CBOR.

I'm afraid CBOR can no longer be considered as the binary equivalent of
JSON plus byte arrays. Instead of tags enhancing data with semantic meaning
hints, it's effectively extending the number of data types. This is what
I've realized with signed CBOR bignums which also rely on tag ranges (a
range of two): they are now a new data type that my decoder needs to
interpret if I don't want the application layer to be concerned with tags.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 2 Dec 2021 08:35:22 +0100

On 2. Dec 2021, at 07:18, Emile Cormier <[email-masked]> wrote:
> I certainly empathize with the idea of keeping tags as conceptually distinct in terms of just adding some extra non-transformational semantic description to the underlying CBOR data structure. However, there is a wide range of "meanings" in tags, and realistically, with many, or even most tags, you can't really "make sense" of a payload without the "semantic meaning"; the meaning is what provides the direction for how to make sense of the data. For many tags, something needs to understand the data beyond just a raw CBOR structure.
> 
> 
> But it should be possible for applications to agree in advance on the meaning of the data and still be able to interpret it in the absence of tags.

I do not agree that this is a worthwhile goal.
Ensuring this causes gymnastics that simply does not provide that much benefit.

> This practice of embedding an extra number via a tag range is ruining this ability for applications to agree in advance on the meaning of the data in the absence of tags.

There is no rule that CBOR data need to make sense after removal of tags.

For instance, in YANG-CBOR [1], tag 47 means that a given map key is an absolute SID, not a relative SID delta.
An implementation that discards these tags is not going to work very well.

[1]: https://www.ietf.org/archive/id/draft-ietf-core-yang-cbor-17.html
[2]: https://www.ietf.org/archive/id/draft-ietf-core-yang-cbor-17.html#sid

> Your proposed tag range of 57344 - 57599 could simply be replaced by a single tag followed by an array of size 2.

Yes.  Encoding this needs some additional bytes.

> In section 3.4 of the spec, it says: "If a tag requires further structure to its content, this structure is provided by the enclosed data item." (emphasis mine). This clause is being violated by tags that rely on tag range to convey additional data.

No, a tag that expresses information in its tag number simply may not "require further structure to its content”, so the cited statement of fact simply doesn’t apply.

> Without the semantic meaning, there isn't data loss; raw CBOR structures could always be transcoded to JSON or anything else (with conventions for tags and such)

That doesn’t work in general: An array with two elements could be a valid data item as well, so you get ambiguity.
Tags provide with a clean way to extend the data model.

I think you are better off accepting that the CBOR generic data model is a superset of that of JSON.

> Well, that's the problem: there is no convention for tags in JSON. A CBOR-to-JSON converter now needs to understand your record set tags, and generate a modified structure for JSON that contains the record IDs that would otherwise be lost.

Generic CBOR-to-JSON converters very well can be designed so they are round-trippable without information loss (*).
It’s just a bit ugly.

> This practice of using a tag range to avoid the few measly bytes of a two-element tuple is making it so that once an application adopts these tags, it can never switch to another encoding other than CBOR.

I don’t know why it would want to, but it can, see above.

> I'm afraid CBOR can no longer be considered as the binary equivalent of JSON plus byte arrays.

It never was!
A simplistic view of its generic data model is JSON plus bytes plus extensibility via tags.

> Instead of tags enhancing data with semantic meaning hints, it's effectively extending the number of data types.

Yes!

> This is what I've realized with signed CBOR bignums which also rely on tag ranges (a range of two): they are now a new data type that my decoder needs to interpret if I don't want the application layer to be concerned with tags.

Handling tags inside generic codecs is definitely one implementation strategy; it requires the tagged information to be representable by platform types (which often is not a problem for bignums).  I’m not sure how a platform type for Kris’ records would look like; I’m sure one can be defined in many platforms.

Grüße, Carsten

(*): E.g., see https://github.com/cabo/cbor-diag/blob/master/lib/cbor-transform-j.rb
This isn’t perfect (it would also need code to handle certain strings to obtain full data transparency), but it was good enough for one application.

---

From: Emile Cormier <[email-masked]>
Date: Thu, 2 Dec 2021 05:48:54 -0400

> Tags provide with a clean way to extend the data model.

For the record, the "Without the semantic meaning, there isn't data
loss..." remark was by Kris, not me.

I think it would have been cleaner for extensions to the data model to be
done purely in terms of the core data model. All tags should do is
*identify* the new data type and not provide a few extra bytes of storage
to that new data type.

Well it seems the genie is out of the bottle in terms of tag ranges being
used to "cheaply" store a few bytes of extra information and I'm not going
to convince you of putting the genie back in. :-)