Subject: Re: [Cbor] Record proposal

From: Kris Zyp <[email-masked]>
Date: Wed, 10 Nov 2021 22:31:55 -0700

Just checking on the status of this tag registration, any update on its status?
Thanks,
Kris

On Sat, Oct 23, 2021 at 1:49 PM Kris Zyp <[email-masked]> wrote:
>
> Just checking to see if you had a chance to discuss this further?
> Thanks,
> Kris
>
> On Tue, Sep 21, 2021 at 12:14 AM Carsten Bormann <[email-masked]> wrote:
> >
> > Hi Kris,
> >
> > I’m on vacation this week, but I think we have good feedback to go forward with this. More next week…
> >
> > Sent from mobile, sorry for terse
> >
> > On 21. Sep 2021, at 06:29, Kris Zyp <[email-masked]> wrote:
> >
> > Did you ever get a chance to discuss this? Is there a next step for
> > this registry entry submission? By the way, I have been working on
> > implementing cbor-packed as well, which seems to complement the record
> > tag nicely.
> > Thanks,
> > Kris
> >

---

From: Christian Amsüss <[email-masked]>
Date: Thu, 11 Nov 2021 12:43:29 +0100

Hello Kris,

> Just checking on the status of this tag registration, any update on its status?

while the record topic is not scheduled on its own today (because there
were sadly no new reactions or reviews on the list since last IETF),
packed is on the agenda, and not only does that overlap a bit with
records, it is also specifically pointed out in the slides.

Hope to see you there
Christian

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: Kris Zyp <[email-masked]>
Date: Thu, 11 Nov 2021 07:08:44 -0700

Thank you for the update, I appreciate it! And I am happy to hear that
Packed CBOR is continuing to progress. I found the referenced slide,
and appreciate the reference. I see it also mentioned the need for
independent implementations of packed. Hopefully cbor-x is helpful in
regards to having an independent implementation for testing (I believe
it is the only JS implementation of cbor-packed)? If you want any help
testing this against the output or input of other implementations,
would be happy to help with that. Anyway, again, this is great to see;
packing and record structure tags are quite distinct, but definitely
complementary in bringing more efficient encodings!

I certainly understand that any standardization efforts take a lot of
time, and don't want to rush all you who are very busy, I'm sure. I
really just wanted to verify that the record tag registration
submission was on the correct track. I think there are about 100
entries in the tag registry (across a dozen or so specifications), and
I want to make sure the record tag submission (submitted as
https://github.com/kriszyp/cbor-records using the template from RFC
8949 per the registry) is in line with the protocol used for tag
registrations and is on the correct track. Is it fair to say that the
registry status of the record tag/tag 105/26880-27135 is that it has a
submitted registration proposal and under review? And from what I
understand from the registry's explanation of registration procedures,
since this is not a tag in the 0-23 range, it does not involve any
actual IETF standards work, but rather just requires the submitted
specification, presumably for review and approval. Is that a
reasonable understanding of the current status of record tag
registration, or did you have different intentions for this?

> Hope to see you there

Did you want me to attend a meeting for discussion of this?

Anyway, thank you again for all your efforts on CBOR, greatly appreciated!

Thanks,
Kris

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 11 Nov 2021 15:22:25 +0100

Hi Kris,

I’ll come back to testing cbor-packed…

On your proposal:

This is an overview over the tags currently registered:

range  used     %                 free                total
0 1+0    12 50.00                   12                   24
1 1+1    62 26.72                  170                  232
2 1+2    45  0.07                65235                65280
3 1+4     3  0.00           4294901757           4294901760
4 1+8     1  0.00 18446744069414584319 18446744069414584320

In RFC 8949, we have said about the ranges

   New entries in the range 0 to 23 ("1+0") are assigned by Standards
   Action.  New entries in the ranges 24 to 255 ("1+1") and 256 to 32767
   (lower half of "1+2") are assigned by Specification Required.  New
   entries in the range 32768 to 18446744073709551615 (upper half of
   "1+2", "1+4", and "1+8") are assigned by First Come First Served.
   The template for registration requests is:

The idea with 24 to 255 is to be a bit frugal with those tags:  They have to last us a few decades, so we shouldn’t be spending more than a few percent per year.
We mostly want to use these for very common tags that actually can benefit from being very short.

The most recent registration was 25441, which — with respect to the expected usage probably has a larger tag-to-data ration than the record proposal (at least when looking the size over all records).
So I think I’d prefer going for a 1+2 tag — you may want to look for a nice ASCII pair like 25441 did.

With respect to the structure:
In SenML we made the mistake of requiring data in the first record that also usually sets the base configuration.
I’m not sure whether that is a mistake for the record proposal, but I think I would prefer to separate the record setup from the first data item.
That is of course your decision, make that change only if it appeals to you.
(In cbor-packed, we put the setup *around* the data — another variant that you might want to look at.)

Grüße, Carsten

---

From: Kris Zyp <[email-masked]>
Date: Thu, 11 Nov 2021 08:35:32 -0700

Thank you for the explanation of your allocation strategy, really
appreciate it, that makes sense. It does seem a little surprising to
me that some of the 1+1 tags that are explicitly for very specific
technologies (Perl, YANG, Internet of Data things, etc.) would be
considered more "common" than a record structure designation, a
concept that I think is broadly used across many/most programming
languages. And it seems like many of these registrations would not
have a greater need for being short than a tag that specifically
yields significant and broad opportunities for dramatically improving
encoding efficiency/compactness (I don't know if you are interested in
some stats on that?). I would certainly argue that the broad semantic
application and the significant encoding efficiency benefits would
justify a 1+1 tag for a tag record. But if only 1+2 tags are being
permitted or strongly preferred, I can certainly adjust to that
criteria. If I resubmit with a 1+2 tag would that be easier for
registration, and would it be possible to have it submitted with
possible future consideration for upgrade to a 1+1 tag?

> ...requiring data in the first record that also usually sets the base configuration...

It seems like the significant problem with using an "around" type of
structure is that it requires an a priori knowledge of the structure
of the data that will be encoding prior to beginning encoding. IMO, I
don't believe this is a constraint that should be enforced on
encoders. This constraint has implementation implications that can
significantly degrade performance (since encoders that are unaware of
the structures that will potentially be serialized, must "discover"
these prior to beginning encoding). This also can preclude many valid
use cases for encoding data reusable structures that are not fully
known prior to encoding (perhaps data is still being read from a
database, I/O, or other sources and may include various data
structures yet to be known). This seems like a particularly
problematic constraint, considering that it seems like CBOR has a very
smart and prescient design for facilitating the serialization of
"streaming" data with indefinite length arrays which are are extremely
valuable for situations when encoding can begin on data that is
streaming into an encoder (not all known ahead of time). (And
"streaming" is specifically mentioned as an intended use case in RFC
8949)

I had actually also been meaning to ask how this type of streaming of
data might work with the combination of cbor-packed and
cbor-sequences. It seems like a particularly valuable use case for
packed CBOR is in being able to create a socket for sending a sequence
of CBOR messages (between two remote hosts, for example), and sending
a packing table with values that are/will be frequently used in the
stream/sequence of CBOR messages. However, I am not sure if this is
actually possible using standard CBOR sequences and packing tables
because a packing table is wrapped around a single CBOR rump/value,
can't be used to indicate that the packed values can be referenced by
future CBOR items in a sequence? I know CBOR packed explains that
packed values can be defined at an application level, so perhaps this
simply requires an out-of-band or non-standard mechanism for
communicating that a set of packed values can be referenced by
messages in a CBOR sequence?

Anyway, thank you again for the update and explanation and all your
efforts, really appreciate it! And if you believe that resubmitting
this tag registration with a 1+2 tag (probably 29299 based on your
ascii suggestion) would be the best course for getting registered, I
can proceed with that.

Thanks,
Kris

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 11 Nov 2021 18:00:22 +0100

On 11. Nov 2021, at 16:35, Kris Zyp <[email-masked]> wrote:
> technologies (Perl,

We were young when we did this…
Seven years later, RFC 8949 has adjusted the RFC 8126 policies only slightly, but with an understanding that 1+1 will be guarded more tightly.

> YANG, Internet of Data things, etc.) would be
> some stats on that?).

Actually, stats would be very interesting.
I was assuming that the 1+1 setup comes with a number of 1+2 referencing records, the hit from going to 1+2 there as well would be relatively insignificant.
Number are better than assumptions!

> I would certainly argue that the broad semantic
> of the data that will be encoding prior to beginning encoding.

That is a very valid argument.
(We may want to revisit cbor-packed as well with respect whether the sender or the receiver pays that tax.)

> […]
> stream/sequence of CBOR messages.

Yes, table setup that transcends a single CBOR data item in a sequence has been brought up before.
Maybe it’s time to flesh out that design.

> And if you believe that resubmitting

Let’s see the numbers, but I do believe this would streamline the registration.

Grüße, Carsten

---

From: Kris Zyp <[email-masked]>
Date: Thu, 11 Nov 2021 21:15:03 -0700

> Number are better than assumptions!

You are definitely right, using 1+2 tag for defining records is pretty
insignificant (around a quarter of percent in my tests). Anyway, I put
together some tests comparing CBOR packed, record structures, and
combinations with a couple test data structures, with my library, for
the sake of further comparisons:
https://gist.github.com/kriszyp/b623b85d2dc25ac9e3b07d8f39df9307

Anyway, seeing this, I am happy to go ahead and update my proposal to
use a 1+2 tag for defining records. And thinking about this, I don't
think my proposal necessarily even needs to mandate the tag ids used
for referencing records since those are dynamically assigned and
explicitly specified by the encoder itself (encoder obviously must not
conflict and use tag ids that will be used for other purposes), albeit
can encourage a certain range (presumably from the first-come
first-serve range).

Do you have any preference for a tag id to use? It looks like 279 is
the next in the contiguous block, but sounds like choosing aesthetic
characters is the new preference (29299/"rs" perhaps).

Anyway, thanks again for the helpful feedback, really appreciate it!

Thanks,
Kris

---

From: Kris Zyp <[email-masked]>
Date: Wed, 17 Nov 2021 08:16:04 -0700

I have updated my tag registration proposal/submission at
https://github.com/kriszyp/cbor-records to use 1+2 tag entries, which
hopefully makes this a much less intrusive registry entry. I have also
updated the proposed tag definitions to also support up-front
declaration of a set of record structure definitions for a data item
("around" that data item, like the packed approach, as you suggested),
in addition to the current inline record definitions (which can be
scoped with record definitions and should follow a well-specified
order for when they can be referenced by subsequent references). I
hope this offers flexibility for encoders that have all structures
known a-priori and streaming encoders (that do not), while still
maintaining nearly the same mechanics for decoders. Let me know if you
think this looks reasonable.
Thank you!
Kris

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 17 Nov 2021 21:52:03 +0100

Hi Kris,

thank you for the update.
I still need to take a closer look, but I have one immediate reaction:
You probably shouldn’t induce implementations to “hijack” a tag number.
The handling of a tag number that has actually been allocated for something else may be deeply embedded into the CBOR decoder; a record implementation then would only see gobbledygook.
So I think we should allocate some space for tag-allocated tags — I don’t see a big problem with allocating a chunk above 32768 just for the record proposal.

Grüße, Carsten

> On 2021-11-17, at 16:16, Kris Zyp <[email-masked]> wrote:
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Kris Zyp <[email-masked]>
Date: Wed, 17 Nov 2021 22:52:45 -0700

That sounds reasonable to me.

To clarify a little bit of the rationale for this:
The purpose of a global registry, as I understand it, is to define the
global tag ids that need to be known by independent encoders and
decoders prior to their encoding and decoding interaction. However,
ids that can be communicated within a message for the scope of that
message, don't require global allocation, if they are only used within
a specified scope. This is just like in a programming language: a
variable from an inner block may shadow another variable from an outer
block or a global scope temporarily within its block without affecting
the outer or global scope (this doesn't require any coordination with
a registry of globals). And a decoder that really implements this
proposal would, by definition, be implementing this behavior of
temporary tag reassignment/shadowing, if it really is a true record
implementation (i.e. a real record implementation wouldn't see
gobbledygook if it is conformant).

Also allowing dynamic/temporary allocation permits potential use of
shorter/more efficient tag ids based on known use of other tags, and
in the case of record references, these may occur much more frequently
in a data item/document compared record definitions, and be much more
sensitive to size differences. Furthermore, this also sidesteps issues
with having to figure out the appropriate size of a range of ids to
globally allocate. How many ids should be allocated? In terms of
percentage of available space, registering/using 256 ids from the tag
1+2 range seems like it is effectively equivalent as registering 1 id
from the tag 1+1 range (and tag 1+4 would definitely be undesirable
due to the high frequency and sensitivity to size). And I have had
users of my implementations say they need a few hundred structures to
be defined and referenceable, so even 256 may be somewhat limiting to
some users.

But that being said, if you are thinking that it may be too onerous
for (record) implementators to support this type of potentially
dynamic/temporary tag reassignment with possibly conflicting ids (as
opposed to using a more stable table of tags or tag ranges), I can
understand that concern. What size of chunk of tag ids (above 32768)
do you think would be appropriate to allocate? Would a chunk of 256 be
reasonable?

Anyway, thank you for your help and feedback!
Thanks,
Kris

---

From: Kris Zyp <[email-masked]>
Date: Mon, 29 Nov 2021 06:35:00 -0700

To follow up, I did update the proposal/spec to use the tag id range
of 57342-57599. Let me know if you would prefer different ids.
Thanks,
Kris