Subject: [Cbor] CBOR interop tests?

From: Carsten Bormann <[email-masked]>
Date: Wed, 22 Aug 2018 21:21:51 +0200

This may sound strange five years after the specification was published, with > 50 documented implementations available, but:

During the interim today, when we discussed the CBORbis work, the question came up whether some structured interop testing might be useful at this time to really get to the ground of some of the finer points we have been discussing recently (NaN payloads, anyone?).  It might also help in coming up with a more comprehensive set of test cases for implementers to check their implementations, but the focus will be on interop between different implementations.

To do such an interop, we’d need to write some test descriptions (TD, (*)) so people know what is going to be tested.

We thought it might be possible to have a first set of test descriptions, plus implementations of stubs or other kinds of support on the respective implementation side, at some time in October.  We could do some online testing (say, in a Jitsi-style web conference event) then, and pick up the work with an updated set of test descriptions in Bangkok during the hackathon.

I’m going on a two-week vacation now, but maybe we can start some of the collection of interesting corners to look at, before I return.

First question: Who would be interested in making these interops happen?  I’m pretty sure that much of the work on the TDs could be done by the editors of CBORbis, but we need the experience of implementers (and the stubs that talk to the implementations) to make this a meaningful exercise.

Second question: What were the corners of CBOR encoding/decoding itself, of CBOR c18n/deterministic encoding, of JSON conversion/interop, etc., where you felt like you had to make a decision that wasn’t really supported either way by the existing spec?

Grüße, Carsten

(*) for CoAP, we used https://github.com/cabo/td-coap4; maybe we can use some of that structure.  But then encoding and decoding (and possibly round-tripping) are much simpler than what we needed for CoAP.

---

From: Jim Schaad <[email-masked]>
Date: Wed, 22 Aug 2018 12:50:11 -0700

>From my point of view, this should also cover one more issue.  Part of the process of moving the standard some form of interop report is going to be needed.  Thus I would like to see all of the items which we think need to be covered in that report covered as well.   I don't know what the chairs and ADs are currently looking for in such an interop report.

Jim

> -----Original Message-----
> From: CBOR <[email-masked]> On Behalf Of Carsten Bormann
> Sent: Wednesday, August 22, 2018 12:22 PM
> To: [email-masked]
> Subject: [Cbor] CBOR interop tests?
> 
> This may sound strange five years after the specification was published, with >
> 50 documented implementations available, but:
> 
> During the interim today, when we discussed the CBORbis work, the question
> came up whether some structured interop testing might be useful at this time
> to really get to the ground of some of the finer points we have been discussing
> recently (NaN payloads, anyone?).  It might also help in coming up with a more
> comprehensive set of test cases for implementers to check their
> implementations, but the focus will be on interop between different
> implementations.
> 
> We thought it might be possible to have a first set of test descriptions, plus
> implementations of stubs or other kinds of support on the respective
> implementation side, at some time in October.  We could do some online
> testing (say, in a Jitsi-style web conference event) then, and pick up the work
> with an updated set of test descriptions in Bangkok during the hackathon.
> 
> I’m going on a two-week vacation now, but maybe we can start some of the
> collection of interesting corners to look at, before I return.
> 
> First question: Who would be interested in making these interops happen?  I’m
> pretty sure that much of the work on the TDs could be done by the editors of
> CBORbis, but we need the experience of implementers (and the stubs that talk
> to the implementations) to make this a meaningful exercise.
> 
> Second question: What were the corners of CBOR encoding/decoding itself, of
> CBOR c18n/deterministic encoding, of JSON conversion/interop, etc., where
> you felt like you had to make a decision that wasn’t really supported either way
> by the existing spec?
> 
> Grüße, Carsten
> 
> 
> (*) for CoAP, we used https://github.com/cabo/td-coap4; maybe we can use
> some of that structure.  But then encoding and decoding (and possibly round-
> tripping) are much simpler than what we needed for CoAP.
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Barry Leiba <[email-masked]>
Date: Thu, 23 Aug 2018 09:28:16 -0400

> From my point of view, this should also cover one more issue.  Part of the process of
> moving the standard some form of interop report is going to be needed.

Not really.  We do have to convince the IESG that what we have is
interoperable, but RFC 6410 eliminated the need for an extensive,
formal report (see section 3.2, which says, "The IETF Standards
Process no longer requires a formal interoperability report,
recognizing that deployment and use is sufficient to show
interoperability.").

> I don't know what the chairs and ADs are currently looking for in such an interop report.

What we're looking for now is a reasonable demonstration that there
are a number of interoperable implementations, for which a brief
summary of an interop test would serve nicely, and perhaps a statement
that we are not aware of interoperability issues among compliant
implementations, which will likely fall out of the former anyway.

Barry

---

From: Jim Schaad <[email-masked]>
Date: Thu, 23 Aug 2018 09:52:27 -0700

> From: [email-masked]
> <[email-masked]> On Behalf Of Barry Leiba
> Sent: Thursday, August 23, 2018 6:28 AM
> To: Jim Schaad <[email-masked]>
> Cc: Carsten Bormann <[email-masked]>; [email-masked]
> Subject: Re: [Cbor] CBOR interop tests?
> 
> > From my point of view, this should also cover one more issue.  Part of
> > the process of moving the standard some form of interop report is going to
> be needed.
> 
> interoperable, but RFC 6410 eliminated the need for an extensive, formal
> report (see section 3.2, which says, "The IETF Standards Process no longer
> requires a formal interoperability report, recognizing that deployment and use
> is sufficient to show interoperability.").
> 
> What we're looking for now is a reasonable demonstration that there are a
> number of interoperable implementations, for which a brief summary of an
> interop test would serve nicely, and perhaps a statement that we are not aware
> of interoperability issues among compliant implementations, which will likely
> fall out of the former anyway.

Does that mean that you would think that the interop which is currently on the Wiki is sufficient or would we need to look at covering more of the document that just the basic I can encode/decode this Major.Minor type for some basic types?  As an example, should the issues that have come up with base64 encoding have been on that matrix initially?

Jim