Subject: [Cbor] Considerations for converting from other serializations formats like Protocol Buffers and FlatBuffers

From: Jonathan Beri <[email-masked]>
Date: Fri, 10 Apr 2020 08:34:12 -0700

Hello,

I'm looking into a use case that would require converting data from
existing systems that output in well-defined serialization formats
(specifically Protocol Buffers
<https://developers.google.com/protocol-buffers/> and FlatBuffers
<https://google.github.io/flatbuffers/>) to CBOR. The goal is in part to
use other specifications like CoAP with this data as as well an existing
implementation of CoAP.

Has there been any investigation into converting to these formats? Any
lessons or points to consider? I've been reading up on CDDL - might that
help in implementing a conversion process?

Many thanks.

-- 
Jonathan Beri
linkedin.com/in/jonathanberi <https://www.linkedin.com/in/jonathanberi/>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 10 Apr 2020 17:59:23 +0200

Hi Jonathan,

Such a conversion is always “interesting" because of the different generic data models of the different data formats.

CDDL essentially is used to define a subset of the generic data model of CBOR.  It also can “parse” or “construct", so it could be used as a front-end/back-end to such a conversion format, but we would need a Format-X back-end/front-end as well.  And some of this will be idiomatic (e.g., very idiomatic in JSON), so you don’t want to need to say the idiomatic things all the time.  Some conversions will even need computations (say, a JSON enum into a CBOR bit string).

We have done something related to a conversion process (uncompressed vs. compressed) in RFC 4997.  This notation was also saddled with needing to do its own binary decoding/encoding (which still would be useful as another front-end/back-end of the above!), but the idea of having statements that connect on two sides to two different worlds is immediately applicable.
SCHC <https://tools.ietf.org/html/draft-ietf-lpwan-ipv6-static-context-hc-24> is a contemporary attempt to do a bidirectional conversion, but it is somewhat specialized to “packet”-type data.

Note also that T2TRG has done a little work on “YOUPI” <https://tools.ietf.org/html/draft-petrov-t2trg-youpi-01>, which is another way to look at binary data.

As you can see, I think some of this is research, and it borders on the WISHI/One-data-model work that we are doing in T2TRG.  I would really like it if you could bring in your ideas there.

Grüße, Carsten

> On 2020-04-10, at 17:34, Jonathan Beri <[email-masked]> wrote:
> 
> Hello,
> 
> I'm looking into a use case that would require converting data from existing systems that output in well-defined serialization formats (specifically Protocol Buffers and FlatBuffers) to CBOR. The goal is in part to use other specifications like CoAP with this data as as well an existing implementation of CoAP.
> 
> Has there been any investigation into converting to these formats? Any lessons or points to consider? I've been reading up on CDDL - might that help in implementing a conversion process?
> 
> Many thanks.
> 
> -- 
> Jonathan Beri
> linkedin.com/in/jonathanberi
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Jonathan Beri <[email-masked]>
Date: Mon, 13 Apr 2020 08:29:13 -0700

Very helpful, thank you Cartsen! I just posted to the T2TRG.

On computation - I've also noticed formats that weren't originally designed
cross the open internet lack a framework for signing and encryption, like
JOSE & COSE. I don't immediately see challenges when going between X and
CBOR, especially if X doesn't define S&E. Anything that comes to mind?

On Fri, Apr 10, 2020 at 8:59 AM Carsten Bormann <[email-masked]> wrote:

> Hi Jonathan,
>
> Such a conversion is always “interesting" because of the different generic
> data models of the different data formats.
>
> CDDL essentially is used to define a subset of the generic data model of
> CBOR.  It also can “parse” or “construct", so it could be used as a
> front-end/back-end to such a conversion format, but we would need a
> Format-X back-end/front-end as well.  And some of this will be idiomatic
> (e.g., very idiomatic in JSON), so you don’t want to need to say the
> idiomatic things all the time.  Some conversions will even need
> computations (say, a JSON enum into a CBOR bit string).
>
> We have done something related to a conversion process (uncompressed vs.
> compressed) in RFC 4997.  This notation was also saddled with needing to do
> its own binary decoding/encoding (which still would be useful as another
> front-end/back-end of the above!), but the idea of having statements that
> connect on two sides to two different worlds is immediately applicable.
> SCHC <
> https://tools.ietf.org/html/draft-ietf-lpwan-ipv6-static-context-hc-24>
> is a contemporary attempt to do a bidirectional conversion, but it is
> somewhat specialized to “packet”-type data.
>
> Note also that T2TRG has done a little work on “YOUPI” <
> https://tools.ietf.org/html/draft-petrov-t2trg-youpi-01>, which is
> another way to look at binary data.
>
> As you can see, I think some of this is research, and it borders on the
> WISHI/One-data-model work that we are doing in T2TRG.  I would really like
> it if you could bring in your ideas there.
>
> Grüße, Carsten
>
>
>
>
> (specifically Protocol Buffers and FlatBuffers) to CBOR. The goal is in
> part to use other specifications like CoAP with this data as as well an
> existing implementation of CoAP.
> >