Subject: [Cbor] CDDL Module Design

From: Jim Schaad <[email-masked]>
Date: Tue, 30 Jun 2020 17:52:19 -0700

Here is a topic that we can potentially discuss tomorrow if there is extra
time on the agenda.

When looking at a CDDL module design, there are a set of questions that I
think need to be addressed:

1.  What is the naming scheme to be used?  Both XML Schema and ASN.1 use
unique identifiers for a module.  The problem with this approach is that a
revised module has a new identifier even if all of the changes were
preserving of the old module.  This can also present problems if two
different versions of the same module are used as it can be difficult to
know which version of a module is being provided.   I think therefore it
would be nice to have a naming system that recognizes version numbers, but
can be used with a "latest version" mode.  One possible way to do this would
be to use a query parameter to get a specific version of a module while
without the query parameter the compiler grabs the most recent version it
has.

2.  Do want the naming scheme to be used to access a module name from the
web, and to create a server to host all of the IETF based CDDL modules?

3.  For any number of reasons we need to be able to state that some symbols
are going to be imported from a different module.  There are some issues
around doing a couple of other things:
*  If all symbols are imported then you either need to have a module based
naming system or you need to ensure that there are no collisions.
*  If only specific symbols are imported, then they could either be a) made
to be unique by renaming or b) made to be unique in the current module
simply by how the module is written

4.  ASN.1 has a method to say that only specific symbols can be exported
from a module.  I have never used this capability and I am not clear that it
would be useful to have here.  Should we support it anyway?

5.  How should the system be setup to be able to recognize this capability.
The suggestion in the freezer document currently is to use a pragma system.
Introducing a pragma system can easily be seen as a step to allow for
compilers of CDDL to be written.   The A2C compiler that I wrote used --#
and #-- as the start and stop delimiters for pragmas as "--" is the
start/stop comment character.  (ASN.1 is not line based so there is no
comment till a return.)  A similar approach can easily be used here.

6.  If we use a pragma naming system, do we want to setup in advance for it
to have namespaces so that a single module could be used with different
compilers.  I ended up using a different string  than the OSS compiler did
("--<" ">--") to be able to do this but having a namespace could also
support this.

7.  What set of pragmas are we looking at to start with.  XML Schema has
some attributes that can be used for controlling the compiler.  One of this
is a location hint attribute which can be used for giving alternate
locations to use for finding a module (such as a location on my machine).
There might be some other pragmas that might be useful to specify as well.
It might be easier to specify co-occurrence constraints using pragmas rather
than trying to put it directly into the language.

Jim

---

From: Michael Richardson <[email-masked]>
Date: Tue, 30 Jun 2020 23:54:00 -0400

Jim Schaad <[email-masked]> wrote:
    > know which version of a module is being provided.

Doesn't the version problem also include the "DLL hell" problem?
That is:
     module A includes module B.1 which includes C.1
            A includes module D.1 which includes C.2

and who knows if C.1 and C.2 can be used at the same time?

So I totally agree that this is a problem.

    > I think therefore it

I think that this is probably a very elegant solution.

    > web, and to create a server to host all of the IETF based CDDL modules?

A think that I dislike about the ENTITY stuff in XML is that the URLs are way
too specific.   Do you want avoid that?  Will it be some URN space?

    > simply by how the module is written

Can you give an example of (b)?
I think that Python mostly got this process right.

    > would be useful to have here.  Should we support it anyway?

No. It just leads to stupid work arounds.
_ for things that should not be exported, but not enforced.

    > comment till a return.)  A similar approach can easily be used here.

I prefer that the statement not be a pre-processor-like hack.

    > ("--<" ">--") to be able to do this but having a namespace could also

I don't understand.

    > than trying to put it directly into the language.

This is useful at the top-level file.
Particularly for a group of people who are working together.
For "location on my machine", I think that I'd rather be providing a command
flag, but I don't object to having ../.. paths or
https://my.little.server.example/ in a file.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Jim Schaad <[email-masked]>
Date: Tue, 30 Jun 2020 22:16:35 -0700

> -----Original Message-----
> From: Michael Richardson <[email-masked]>
> Sent: Tuesday, June 30, 2020 8:54 PM
> To: Jim Schaad <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Cbor] CDDL Module Design
> 
> 
>     > would be nice to have a naming system that recognizes version
numbers,
> but
>     > can be used with a "latest version" mode.  One possible way to do
this
> would
>     > 2.  Do want the naming scheme to be used to access a module name
from
> the

I was thinking about the question of doing a URN instead of a URL - hence
the term naming scheme.  The interesting question is how should the naming
be done and what short of mapping is needed.  One of the nice things about
DOIs is that they can be used as an indirect reference with a well known
server to get actual URLs

> I think that Python mostly got this process right.

(b) is just - don't define any conflicts between what I am importing and
what I have written in the text.  If I am going to use COSE_Signed as an
import, then I really should define a different symbol in the current CDDL
file by that name.  

> _ for things that should not be exported, but not enforced.

I tend to agree with this.  One of the interesting questions would be
transitive imports though.  If module A imports COSE_Signed from module B
can module B import COSE_Signed from module B or must it reference module A.
I remember the first time I figured out that ASN.1 allows this and to be
frank I found it somewhat irritating.  It does allow moving symbols between
modules, but that has the potential to break other modules suddenly.

> I prefer that the statement not be a pre-processor-like hack.

The interesting question then, is how much do we want to change the CDDL
language.  If we do pragmas this is a standalone document.  If we do changes
in the grammar then it is harder to make this standalone.

> 
>     > 6.  If we use a pragma naming system, do we want to setup in advance
for
> it

Suppose I write a CDDL compiler - called my_great_cddl_compiler and Carsten
writes one called no_so_good_cddl_compiler.  I could reference pragmas as
ME:UseInt32 and CB:UseInt32 as separate pragmas in the same file and each
compiler would recognize that one that it implements and do the right thing
(which is to use some int sized at least 32 bits?)  They could have
different behavior for the same name.  

>     > It might be easier to specify co-occurrence constraints using
pragmas
> rather
https://my.little.server.example/
> in a file.

Another one at the top level might be to use one of the more interesting
things that the OSS compiler does is to allow one to say - This is the list
of symbols to emit which can be used rather than either emit everything or
emit the first symbol in the file which made me do some really strange
things in the COSE CDDL.

Jim

> 
> --
> Michael Richardson <[email-masked]>, Sandelman Software Works  -
> = IPv6 IoT consulting =-

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 Jul 2020 07:33:24 +0200

Hi Jim,

we have had a long discussion about namespacing in the SDF work, and came up with:
https://www.ietf.org/id/draft-onedm-t2trg-sdf-00.html#name-names-and-namespaces

Now SDF is not CDDL, but I think there are lessons to be learned.

For a while, we were thinking about using URIs to find referenced parts of a specification.  This is comparable to the way Java does its module naming; you essentially need to put the code into the place that is implied by its name.

Instead, we have decided to go for a push model, which is essentially the way C++, C#, and other languages maintain their namespaces.  A module declares into which namespaces its names go.  There is no automatic finding of modules, but there also is no pain about who can contribute to a namespace.

> On 2020-07-01, at 02:52, Jim Schaad <[email-masked]> wrote:
> 
> would be nice to have a naming system that recognizes version numbers,

(I think you are talking about revision numbers.
Version numbers are often not what one wants; instead one should use feature identifiers.)

> web, and to create a server to host all of the IETF based CDDL modules?

That was what we decided we didn’t want to do in SDF.

> are going to be imported from a different module.  

Note how C++ doesn’t really do that.
(Well, if you declare a name and don’t define it, it needs to be external.)

> There are some issues

I would like to separate the external naming system from the internal one we use in a spec.

Note how a spec already today uses symbols from the prelude and from the spec itself.  (This of course works “automagically” only because the prelude is fixed and you simply cannot use the symbols that the prelude squats on.)

> simply by how the module is written

Yes.

> would be useful to have here.  Should we support it anyway?

I think there might be a place for an advisory-level setting (as in “these names will normally be imported if this module is imported whole-sale”).  Experience shows that there are good reasons to violate any “encapsulation” attempt, so I would not disallow importing further names from a module explicitly.

> comment till a return.)  A similar approach can easily be used here.

This is the approach I would prefer.

However, there also is semantics imparted by using a symbol.
In actual CDDL usage, the symbols used become annotations in the post-validation instance.  These annotations will need to be namespaced, and possibly be linked to RDF names (which are URIs not intended for dereferencing).

> ("--<" ">--") to be able to do this but having a namespace could also

Indeed, we could play around with more than one module/namespace management system, as long as there is a way to generate a basic CDDL specification from each.

> locations to use for finding a module (such as a location on my machine).

Tempting, but this leads us back into pull land.

> than trying to put it directly into the language.

I think that all constraints need to be part of the language.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 Jul 2020 08:31:44 +0200

So here are a few terms (and items of thought) that I would like to propose as a ground structure:

Symbols: Rule names as defined in RFC 8610.  This namespace is used within a specification and should stay reasonably clean of considerations on where the name came from.  E.g., a CDDL specification today can use symbols from prelude and from itself.

External names: Names that go into a larger name space, some of which may be global.  A CDDL spec can import definitions associated with an external name (mapping that into symbols local to the spec) and export into (make its symbols available in) the global namespace.  

Global namespace: external names that are curated in such a way that there are no conflicts.  It probably makes sense to allow for “ecosystem namespaces”, i.e., a way to partition off subspaces of the global namespace that can be used in an SDO without the need for coordination with others.

Revision: a specific instant in the progression of a specification.  Revisions may or may not semantically differ (a new revision could have a better copyright statement, a new license, better acknowledgements, reordered rules, etc. without any semantic implications).

Module reference: a name that, at a particular point in time, identifies a set of specifications that make up a module (if that term makes sense).  Module references are intended to allow for evolution of the modules; they may also be frozen to specific revisions (which doesn’t allow for bug fixing).  Module references need to cope with the fact that a specific implementation may be bound to a state of the module that is earlier in time than the one considered most recent by a contributing entity; whether that state is compatible with the desired use of the module will have to be expressed in the module reference (e.g., by using feature identifiers).

Feature name: Name for a (potentially upwardly compatible, potentially not) state of a module that incorporates certain additions/changes.  A feature name can take the form of a version (≠ revision), or it can be more granular.

Semantic name: a name that interfaces with the world outside CDDL, e.g., an RDF class name.  These often will be URIs.  Often, a symbol in a specification will be annotated with a semantic name; this is of no consequence to the structural semantics of CDDL, but might turn up in a post-validation instance.

(It is not clear to me whether there needs to be a shortcut between semantic names and external names — I’d like to keep them separate for now.)

Note that keys (the name part of a name/value pair) in groups are often used as abbreviated semantic names (whether they are ignored at the CDDL level as in arrays or &-choices, or actually used as map keys).  There probably needs to be some way to map these to fully fleshed out semantic names.  (Note that these keys often are written as barewords, so they may look like symbols, but aren’t!)

Grüße, Carsten