Subject: [Cbor] draft-bormann-cbor-sequence-00 and Big Data

From: Burt Harris <[email-masked]>
Date: Sun, 23 Jun 2019 02:12:10 +0000

draft-borman-cbor-sequence-00 has me thinking, it's a great first draft.

It is unquestionably good to logically distinguish between a single CBOR data item and a (potentially large) stream of CBOR data items.   MIME type makes very good sense, but we might also back it up with by allocating a unique tag (similar to the self-describing CBOR tag number 55799 in RFC7049 section 2.4.5)) as a NO-OP "magic number" at the beginning of a stream.

I think the current name and content of the draft implies the order of the data items in the stream might be logically significant.   I suggest for this media type we should consider overall sequence implementation detail (like the order of keys in a map).   In particular, to facilitate building scalable big-data systems using CBOR, it would be valuable to recognize that if the order of data items is important, a CBOR Array is probably more appropriate.   

Perhaps s/sequence/stream/g might be appropriate in the document because a "sequence" (to me) implies order preservation.

For example, to collect data from a few million IoT devices, I would use a scale-out architecture where several identical front-end machines are load balanced with each one buffering the little cbor messages into a large buffer.   The front-end machines flush their buffers asynchronously, either when the buffer is full or the after expiration of a period.   Due to the asynchronous distributed buffering, the order of the data items may not reflect the actual sequence of events.      This non-sequential logging can be significant even in multicore-processors with sophisticated logging infrastructure.

Similarly, big-data distributed processing architectures, e.g., the map / reduce algorithm; there should be no assumption of global ordering.   But local ordering (within a block) can sometimes be very useful.    For that reason, we might want to consider allocating a single-byte CBOR special (MT 7) value for delimiting portions of a stream where the sequence can be considered meaningful.   For applications not sensitive to the order of the values, it would effectively be a NoOp, but when compression approaches are used in a CBOR stream, it should probably reset the compression state.   A 1-byte NoOp in Major Type 7 can also be useful in padding a CBOR stream into a "splittable" storage format, which aids in distributed analysis processing using frameworks like Hadoop.

Finally, by testing for if a NoOp appears at the last element in a stream, it may be possible to detect if the application writing to the stream paused at that point.    For those familiar with YAML, consider the "..." token.

P.S.  I wanted to thank the members of the WG who participated in moving CDDL forward to become RFC8610, great work!

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 23 Jun 2019 07:43:14 +0200

Hi Burt,

thanks for your kind words on RFC 8610 and CBOR sequences!

Whether the sequence-preserving property of CBOR sequences is useful for your application or not depends on the application.  CBOR sequences do not take a position on this, they just preserve the sequence (because they can, at approximately zero cost).  This is exactly as with CBOR arrays, where it also depends on the application whether the actual sequence means something or not.  For serialization, they need to be in a specific sequence, so we might as well make that available to the application.  There are applications that use CBOR sequences in a way that cannot really be called a stream, so I think we are better off with the “sequence” name.

Tagging a CBOR sequence as such is a bit difficult because logically a CBOR tag would be tagging the first item in the sequence.  CBOR sequences are best used in a context where it is clear what they are (e.g., via a media type); I tend to think of them like CBOR arrays that have been taken out of their package…

Any special split markers can be provided by simply putting CBOR data items for those markers into the CBOR sequence.  And, yes, such an application might be a good reason to allocate a CBOR Simple value or two.  We don’t need an equivalent for YAML “…”, though, because the CBOR data items themselves are self-delimiting.

Grüße, Carsten