Subject: [Cbor] CDDL Codegen in production

From: Sebastien Guillemot <[email-masked]>
Date: Fri, 4 Sep 2020 14:44:31 +0900

Hello all,

Background: most of you are familiar with RFC 8610
<https://tools.ietf.org/html/rfc8610> (CDDL). You may also be familiar with
the CDDL Ruby gem <https://rubygems.org/gems/cddl/versions/0.6.2> which can
be used to parse & validate CDDL. There is also a Rust tool
<https://github.com/anweiss/cddl> by Andrew Weiss that provides equivalent
functionality in Rust.

Our company (EMURGO) leveraged the work by Andrew to build a full Rust
codegen tool for CDDL that we called "cddl-codgen
<https://github.com/Emurgo/cddl-codegen>" where the output Rust code is
WASM-compatible. We then used this codegen tool on a CDDL file created by
one of our partners and generated a Rust library (link if you're curious
<https://github.com/Emurgo/cardano-serialization-lib>). This library
shipped to production last month and it's now being used by multiple
companies with a 6-digit number of end users running the library on their
devices thanks to the free & open source distribution of the whole stack.

There are still rough edges to our codegen library because some features of
CDDL don't work well with codegen (hopefully CDDL can improve this going
forward!), but overall this has been significantly easier for our company
to manage than having to hand-write CBOR parsing code every time our
partner updates their CDDL. We look forward to leveraging our CDDL-codgen
tool for other projects in the future.

I wanted to say thank you to everybody involved in the CDDL specification,
and if you have any questions about our library, don't hesitate to reach
out to me at [email-masked]

<https://twitter.com/sebastiengllmt>
<https://www.youtube.com/channel/UCgFQ0hHuPO1QDcyP6t9KZTQ>
<https://t.me/emurgo>     <https://www.linkedin.com/company/emurgo_io/>
Sebastien Guillemot
Co-VP of Eng & Cardano PM
_______________________
 [email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 4 Sep 2020 11:05:11 +0200

Hi Sebastien,

this is very encouraging news!

Yes, I’m very curious.  We are seeing a lot of pickup of CBOR in the Fintech space, and it is nice to see that CDDL is being embraced, too.

I see that you have open-sourced the code generator at https://github.com/Emurgo/cddl-codegen (with MIT License).
I’m impressed to see that you achieved all that in ~2700 lines of code.

> There are still rough edges to our codegen library because some features of CDDL don't work well with codegen (hopefully CDDL can improve this going forward!), 

We are certainly interested in hearing what CDDL features are troublesome to implement in a code generator.  The github repo lists a number of features that are not implemented, but doesn’t say whether you just didn’t need them or they are simply too much work.

Some of these features could be addressed by CDDL-to-CDDL preprocessors.
E.g., maybe one of the functions I could add to the cddlc tool is a Generics resolver, as well as a topological rule sorter (which might then also do the necessary work for sockets).

Grüße, Carsten

---

From: Robert Aftias <[email-masked]>
Date: Tue, 8 Sep 2020 19:04:30 -0500

Hi Carsten

I'm the dev of our cddl-codegen tool. To address your comments most of the
limitations were simply due to time constraints and priorities. The one
thing that I can think of that wasn't purely a time issue is being able to
define human readable names to types/fields in some contexts: specifically
map keys where the key value is not very informative. I thought about using
comments to do it, but in the cddl-parsing library that Andrew wrote the
comments are stripped out. However, even with comments being preserved,
especially without taking whitespace into account, associating the comments
unambiguously to fields/types would be difficult when for example map keys
are units or other types that don't provide much context.

This I imagine could be a common situation (maps with uint keys) as it
shrinks the resulting CBOR size vs having descriptive key names. Encoding
as an array (so we don't store the keys in cbor and we can have descriptive
names) with optional fields can result in more complicated deserialization
code in some situations due to increased ambiguity during deserialization
(think [?uint, uint, ?text, text] vs { ? 0: uint, 1: uint, ? 2: text, 3:
text}) which is part of the reason why our codegen lib doesn't support
optional fields in deserialization, along with time constraints and it not
being used in our partner's cddl files.

For example to arrive at a meaningful name for key 1 we could comment it as:

foo = {
 ? 0 : bar,
 ; meaningful-name
 ? 1: uint,
 ? 2: baz
}

or

foo = {
 ? 0 : bar,
 ? 1 : uint, ; meaningful-name
 ? 2 : baz
}

but AST-wise those would be in different places - between keys 0 and 1 or 1
and 2 in the two examples, so without taking whitespace into account it's
ambiguous which key the comment is describing. Even with whitespace it's a
little subjective if you formatted the cddl in other ways.

Even if we don't use the comment for the field name it would be very nice
to be able to inject the comment into the resulting rust code.

Our cddl-codgen tool will use the field/key name as the rust field name if
it exists, and use the type otherwise, but in this situation it's not very
helpful to have a bunch of fields named key_0, key_1, key_2, etc.

This also happens when you inline types within other types like
foo = [
    0: [uint, text].
    1: { * uint => text }
]
since we need to define new types here for both due to wasm-bindgen (rust
to wasm auto-conversion) restrictions. However, in this case it's not a big
deal at all since it'd probably be better practice anyway to just define
new types and refer to them in the outer type and this wouldn't impact the
CBOR representation of the CDDL. We did that to get around uninformative
names for variants of group choices too.

Maybe there's another reasonable approach though for the field names where
restructuring without changing the underlying CBOR can't achieve it?

A cddl-to-cddl topological rule sorter would possibly be useful to have for
others. Right now to get around that our lib does a multi-pass over the
cddl for figuring out if some features are unsupported transitively as I
didn't have the time to implement a proper type dependency system. The
multi-pass gets the job done but it's not the most elegant solution. It
does however preserve declaration-order which could be desired in the
resulting codegen. For sockets I just didn't look into how codegen would be
impacted at all since the cddl files we were parsing didn't use them in a
necessary manner so I didn't devote any time to it.

A ccdl-to-cddl generics resolver would get rid of the need for us to
support generics. Andrew's cddl parsing lib does have generic information
that I believe I could use to implement generics in our codegen tool, but I
just had other priorities and didn't have time to try and implement it yet.

Thanks,
Rob

On Fri, Sep 4, 2020 at 4:05 AM Carsten Bormann <[email-masked]> wrote:

> Yes, I’m very curious.  We are seeing a lot of pickup of CBOR in the
> Fintech space, and it is nice to see that CDDL is being embraced, too.
>
> I see that you have open-sourced the code generator at
> https://github.com/Emurgo/cddl-codegen (with MIT License).
> > There are still rough edges to our codegen library because some features
> of CDDL don't work well with codegen (hopefully CDDL can improve this going
> forward!),
>
> We are certainly interested in hearing what CDDL features are troublesome
> to implement in a code generator.  The github repo lists a number of
> features that are not implemented, but doesn’t say whether you just didn’t
> need them or they are simply too much work.
>
> E.g., maybe one of the functions I could add to the cddlc tool is a
> Generics resolver, as well as a topological rule sorter (which might then
> also do the necessary work for sockets).
>
> Grüße, Carsten
>
>