Subject: [Cbor] Additional data types

From: <[email-masked]>
Date: Mon, 6 Jun 2016 22:08:03 +0000

I'm evaluating use of CBOR as a replacement for a proprietary format that is very similar to CBOR.  With the addition of TypedArrays most of our data types are supported.  I was pleased to see Null and zero length arrays supported as well.  The standard seems to be well thought out!

We have several additional data types which I would appreciate being considered or alternatively educating me on how they may already be included in CBOR.

1.      Multidimensional Arrays - perhaps a tag preceding an array of integer dimensions followed by the array data as a single TypedArray.  Data would be in row major format as is used in C, C++, and Python.

2.      Jagged Arrays<https://msdn.microsoft.com/en-us/library/2s05feca.aspx> or an Array of Arrays.  A tag indicating the dimension of the following jagged array followed by each array.

3.      IEEE1588 Precision Time Protocol Timestamp.  The basic time representation is 48 bits of UTC seconds and 32 bits of nanoseconds.   An additional 16 bits of subnanoseconds can also be present.   Rather than a new tag, I initially thought this could be covered by using a Decimal Fraction with -9 as the exponent.  The mantissa would be seconds*1e9+nanoseconds.  However, it would appear the RFC restricts the semantic tag for epoch-based date/time to ints and floats.   Perhaps if time could be expressed in integer UTC milliseconds, microseconds, nanoseconds, and picoseconds accurately that would fill the need.  UTC milliseconds is a very common timestamp method.

4.      Time Duration or Interval.  This is different from a timestamp in that it measures a difference in time between two events but with similar precision.  This would represent a TimeSpan or Duration as is used in some libraries.

Thanks in advance for your thoughts on these items,

Glenn Engel
Keysight Technologies

---

From: Michel Veillette <[email-masked]>
Date: Tue, 7 Jun 2016 00:34:16 +0000

Hi Glen

Multidimensional Arrays are not directly supported in CBOR but you can encode arrays within arrays.
This should also answer you second question.

For the representation of the is 48 bits of UTC seconds, CBOR don't support directly 48 bits unsigned integer. You can either use a unsigned integer 64 bits (major type 0) or byte string of 6 bytes (major type 2).

All these data types can be tag if such meta information is required. You will need  to request these tags from IANA. The use of tags is not required when the decoder have enough knowledge of the information received to process it correctly. By not transmitting tags, you can minimize the payload size.

Hope this answer some of your questions.

Regards

Michel Veillette
System Architecture Director
Trilliant Inc.
Tel: 450-531-3109

From: CBOR [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, June 7, 2016 12:08 AM
To: [email-masked]
Subject: [Cbor] Additional data types

Glenn Engel
Keysight Technologies

---

From: <[email-masked]>
Date: Wed, 8 Jun 2016 00:18:12 +0000

Thanks for the examples.  cbor.me is a handy tool!

In my case I want to deserialize into a strongly typed language (C++ and C#) so it is desirable to know the type of the array for memory allocation purposes.  Another objective is to be able to round-trip from strongly typed into CBOR and back again resulting in the same types.  This results in somewhat larger encoding in some cases such as a float 1.0 using 4 bytes instead of 2 but it also speeds up serialization.

I think of the 'array'  major type as more of a 'list' which maps nicely in javascript but for analytic data going into C++ for signal processing it's important to keep it as a native array so the type and dimension properties are needed up front for efficient memory allocation of contiguous memory.

If I would like to see a multidimensional array tag that could use the new TypedArrays would that be something I would propose to IANA or would it be a tweak to the existing TypedArray proposal?  I'm thinking a tag that is followed by two items - an array of dimensions followed by a single TypedArray.

What about a TypedArray of strings, another common data type.

Thanks in advance for your advice and insight.

--
Glenn

From: Michel Veillette [mailto:[email-masked]]
Sent: Monday, June 06, 2016 5:34 PM
To: ENGEL,GLENN (K-Labs,ex1) <[email-masked]>; [email-masked]
Subject: RE: Additional data types

Hi Glen

Regards

Michel Veillette

From: CBOR [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Tuesday, June 7, 2016 12:08 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Cbor] Additional data types

Glenn Engel
Keysight Technologies

---

From: Michel Veillette <[email-masked]>
Date: Wed, 8 Jun 2016 00:39:18 +0000

About " so the type and dimension properties are needed up front"

CBOR array already support the "up front" dimension.

For the type part, if your arrays are always homogeneous (array of int, array of string, array of array), your decoder can do a lock up to the next major type following the array tag and dimension to determine the type of this array and perform the proper allocation.

If not, I assume you can register a single tag "TypedArray" and do known if a lock up is necessary.

The alternative method will be to register a tag for each of the TypedArray you need to support but this seem to be an unbounded list of tags. Carsten can probably give you an advise on this.
Regards,
Michel Veillette

From: [email-masked] [mailto:[email-masked]]
Sent: Wednesday, June 8, 2016 2:18 AM
To: Michel Veillette <[email-masked]>; [email-masked]
Subject: RE: Additional data types

--
Glenn

From: Michel Veillette [mailto:[email-masked]]
Sent: Monday, June 06, 2016 5:34 PM
To: ENGEL,GLENN (K-Labs,ex1) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: Additional data types

Hi Glen

Regards

Michel Veillette

Glenn Engel
Keysight Technologies

---

From: <[email-masked]>
Date: Wed, 8 Jun 2016 21:58:01 +0000

I agree that a sea of tags would be a poor approach.  I don't yet see how CBOR would support multi-dimensional arrays without either assumptions on the decoder's part to expect it in places or introducing at least one tag.

Something like:

Tag: XX
Data Item: array
Semantics: Multi-dimensional array

A multi-dimensional array is represented as a tagged array that contains two arrays.  The first array defines the dimensions of the multi-dimensional array while the second array represents the contents of the multi-dimensional array.  If the second array is itself tagged as a TypedArray then the type of the multi-dimensional array is assumed to be the same type as the TypedArray<https://tools.ietf.org/html/draft-jroatch-cbor-tags-03>.   Data in the TypedArray byte string is considered to be in row major order.

I would imagine the encoding as follows.  Hopefully I read the spec right...

uint16_t a[2][3] = {
  {0, 1, 2},   /* row 0 */
  {3, 4, 5},
};

<Tag XX>    # multi-dim array
   82       # array(2)

      82    # array(2)
         02 # unsigned(2) 1st Dimension
        03 # unsigned(3) 2nd Dimension

     d8 41 # uint16 array
         4a    # byte string(12)
            00 00 # unsigned(0)
            00 01 # unsigned(1)
            00 02 # unsigned(2)
            00 03 # unsigned(3)
            00 04 # unsigned(4)
            00 05 # unsigned(5)

Regards,

--
Glenn

From: Michel Veillette [mailto:[email-masked]]
Sent: Tuesday, June 07, 2016 5:39 PM
To: ENGEL,GLENN (K-Labs,ex1) <[email-masked]>; [email-masked]
Subject: RE: Additional data types

Regards,
Michel Veillette

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Wednesday, June 8, 2016 2:18 AM
To: Michel Veillette <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: Additional data types

--
Glenn

Hi Glen

Regards

Michel Veillette

Glenn Engel
Keysight Technologies

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 09 Jun 2016 00:11:32 +0200

> Something like:

Hi Glenn,

this is precisely what I was going to propose as well!
(Except that I wouldn't use the term "row major order", maybe.)

Is this well thought out enough that we include this in the typedarray
draft?
(And is that maybe a good way to use that one reserved tag in the 24-tag
space?)

Grüße, Carsten

---

From: <[email-masked]>
Date: Thu, 9 Jun 2016 15:34:13 +0000

Use of the reserved tag for Multi-dim arrays would be great.   However, my first thought when I saw the reserved tag was it could be used for one of the other major types that are not covered.   For example, text strings.  

The ones missing from my specific application would be TypedArray of 
1. Text string (major 3)
2. Time (tag 1)
3. Bool (Major 7/20-21)

Perhaps these and others could be supported by having a single 'Homogeneous Array' tag:

Tag: XX
Data Item: array
Semantics: Homogeneous Array

This tag provides a hint to decoders that the following array has elements that are all of the same type.   The type of the array is thus determined by the type of the first element.  This can be used by strongly typed languages while decoding  to create native arrays of specific types instead of ordered lists.

While encoding is not optimized for compactness like the numeric types, it does mean that virtually any homogeneous array type can round-trip thru CBOR nicely if the application wants to support specific native array types.

--
Glenn

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]] 
Sent: Wednesday, June 08, 2016 3:12 PM
To: ENGEL,GLENN (K-Labs,ex1) <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [Cbor] Additional data types

Hi Glenn,

Is this well thought out enough that we include this in the typedarray draft?
(And is that maybe a good way to use that one reserved tag in the 24-tag
space?)

Grüße, Carsten