Subject: Re: [Cbor] [core] draft-ietf-cbor-7049bis - Change suggested Canonicalization

From: Jeffrey Yasskin <[email-masked]>
Date: Mon, 08 Jan 2018 18:33:42 +0000

Now that we're past the holidays, are there more comments on
https://github.com/cbor-wg/CBORbis/pull/9?

On Fri, Dec 22, 2017 at 3:56 PM Jeffrey Yasskin <[email-masked]>
wrote:

> I don't think https://github.com/cbor-wg/CBORbis/pull/9 allows anything
> that RFC7049 didn't, which is the meaning I get from "open this up".
>
> We *could* move this (all of section 3?) to a separate document, but I
> haven't seen anyone say that we need to. A downside of moving section 3 to
> another RFC is that it'll make it harder to find. Someone authoritative
> (Francesca?) should just make this call so that we can stop angsting about
> it.
>
> I'm generally happy to change exactly which examples demonstrate that
> protocol designers need to think about canonicalization even if they start
> from the core canonicalization requirements. I'd appreciate a concrete
> statement of which examples to use though.
>
> Once we add floating values, the type of a field matters in defining its
> canonicalization. A float64 field only needs to canonicalize NaN. A
> float16/float32/float64 field needs to canonicalize to either the smallest
> or largest type. An int/float field needs to again canonicalize toward or
> away from int. A bigint field needs to prefer either a fixed-length
> (useful for cryptographic signatures) or the shortest representation. A
> decfrac/bigfloat/number field has an even more complex problem. I don't
> personally have enough examples of existing canonicalized CBOR-based
> protocols to make any confident recommendations here. If the list gives me
> some, along with the field experience that justifies them, I'm happy to
> write them down in my patch.
>
> Jeffrey
>
>
> On Fri, Dec 22, 2017 at 2:23 PM Carsten Bormann <[email-masked]> wrote:
>
>> Hi Jeffrey,
>>
>> quick reactions after a first skim:
>>
>> I’m not sure the direction should be to open this up; I think the
>> recommendations should become more narrow as we learn about the practical
>> issues.
>>
>> We could write a separate document on the preferred c14n so we can keep
>> this out of the main document.
>>
>> I don’t think we necessarily want to encourage cross-over between int and
>> float, so I think the “shortest float” rule should be applied independent
>> of whether that cross-over is desired.
>>
>> Why keep the “shortest int” rule less well defined for protocols that use
>> bignums?  It should apply there as well, i.e., use bignums only for
>> integers too large for the major type 0/1 formats.
>>
>> Grüße, Carsten
>>
>>
>> > On Dec 22, 2017, at 23:13, Jeffrey Yasskin <[email-masked]>
>> wrote:
>> >
>> > On Sun, Dec 3, 2017 at 6:15 AM Carsten Bormann <[email-masked]> wrote:
>> > On Nov 30, 2017, at 23:14, Jeffrey Yasskin <[email-masked]>
>> wrote:
>> > >
>> > > Belatedly, I've discovered a user of "canonical" CBOR who's proposing
>> a different map order than the RFC suggests:
>> https://fidoalliance.org/specs/fido-v2.0-rd-20170927/fido-client-to-authenticator-protocol-v2.0-rd-20170927.html#message-encoding.
>> (Note that this isn't a final standard yet and may change.)
>> >
>> > I looked at the spec referenced.
>> >
>> > So they essentially add
>> >
>> >                 • If the major types are different, the one with the
>> lower value in numerical order sorts earlier.
>> >
>> > as a major sorting rule before the existing RFC 7049 canonicalization
>> rules:
>> >
>> >                 • If two keys have different lengths, the shorter one
>> sorts earlier;
>> >                 • If two keys have the same length, the one with the
>> lower value in (byte-wise) lexical order sorts earlier.
>> >
>> > This is different from simply going for byte-wise lexicographic (memcmp
>> order(*)), which effectively would get us the first rule as the major
>> sorting order already, but get rid of the length-based second rule (first
>> rule in Section 3.9 of RFC 7049).
>> >
>> > I’ve come to see the putting the length comparison rule early in 3.9 as
>> a major regression.
>> > One of the objectives when designing the CBOR serialization was not to
>> repeat one big mistake that ASN.1 BER makes: to make overall lengths of
>> complex composite items visible/important in the encoding of the next
>> higher composite.
>> > Here, we are doing just that.  D’oh.
>> >
>> > > This is justified by the RFC saying that "Those protocols are free to
>> define what they mean by a canonical format and what encoders and decoders
>> are expected to do.  This section lists some suggestions for such
>> protocols." That is (as Jim said), the RFC doesn't specify "canonical"
>> CBOR: it just provides an option for higher-level protocols to do so.
>> >
>> > Right.  So the change would be to mention two options for this, the old
>> canonical, and the saner (memcmp order) canonical.  Now the next step is
>> finding names for legacy canonical/saner canonical.  We then have to decide
>> whether we turn this into a separate document, at Proposed Standard level,
>> or believe that adding another suggestion to 3.9 is essentially a bug fix
>> and can be done in the Standard level document.
>> >
>> > > The use of a different order in CTAP is going to either force its
>> implementers to write custom CBOR encoders and decoders or require the
>> generic encoders to take a configuration option for the map order. If the
>> generic encoders take an option, then it stops being an issue for CBORbis
>> to suggest a different order.
>> >
>> > Right.  So I think you are saying we get to fix this.
>> >
>> > I've tried to implement this in
>> https://github.com/cbor-wg/CBORbis/pull/9. I defined a core set of rules
>> so that other specs can use them by reference, gave several examples of
>> protocols that will need to extend the rules, and defined a second set of
>> rules that match the canonical order that RFC7049 suggested.
>> >
>> > Do folks like this direction?
>> >
>> > I don't have a strong opinion about which document should hold the
>> canonicalization rules. Can we ask the IESG which they'd prefer?
>> >
>> > Jeffrey
>>
>>

---

From: Jim Schaad <[email-masked]>
Date: Mon, 8 Jan 2018 20:12:27 -0800

I did not like the idea of changing the types of items when doing canonicalization.  I want to look just at what is given to me and thus do not like things like a floating point number may be encoded as an any of a number of different formats.  The choice of what an item is encoded as in terms of a floating point number is a function of the document specification and not the canonicalization encoding format.  For things like big numbers, the encoding should not be modified because the leading characters have been chosen by the application for a specific reason.  Consider the leading padding that is part of a public key in cryptography where the length of the value is of equal or greater importance that the fact that is has leading 0 or 0xff bytes.  The rules for mapping between the data model and the encoding are as much up to the protocol specification as they are for the CBOR specification.  It is possible that a specification will only want to use the 64-bit floating point number format because it really simplifies the application even if it makes the encoded format longer.

In sort, I do not believe that the rules should be extended beyond how an encoding works for a single encoded data format.

Jim

From: CBOR [mailto:[email-masked]] On Behalf Of Jeffrey Yasskin
Sent: Monday, January 8, 2018 10:34 AM
To: Jeffrey Yasskin <[email-masked]>
Cc: [email-masked]; Carsten Bormann <[email-masked]>
Subject: Re: [Cbor] [core] draft-ietf-cbor-7049bis - Change suggested Canonicalization

Now that we're past the holidays, are there more comments on  <https://github.com/cbor-wg/CBORbis/pull/9> https://github.com/cbor-wg/CBORbis/pull/9?

On Fri, Dec 22, 2017 at 3:56 PM Jeffrey Yasskin <[email-masked] <mailto:[email-masked]> > wrote:

I don't think https://github.com/cbor-wg/CBORbis/pull/9 allows anything that RFC7049 didn't, which is the meaning I get from "open this up". 

We *could* move this (all of section 3?) to a separate document, but I haven't seen anyone say that we need to. A downside of moving section 3 to another RFC is that it'll make it harder to find. Someone authoritative (Francesca?) should just make this call so that we can stop angsting about it.

I'm generally happy to change exactly which examples demonstrate that protocol designers need to think about canonicalization even if they start from the core canonicalization requirements. I'd appreciate a concrete statement of which examples to use though.

Once we add floating values, the type of a field matters in defining its canonicalization. A float64 field only needs to canonicalize NaN. A float16/float32/float64 field needs to canonicalize to either the smallest or largest type. An int/float field needs to again canonicalize toward or away from int. A bigint field needs to prefer either a fixed-length (useful for cryptographic signatures) or the shortest representation. A decfrac/bigfloat/number field has an even more complex problem. I don't personally have enough examples of existing canonicalized CBOR-based protocols to make any confident recommendations here. If the list gives me some, along with the field experience that justifies them, I'm happy to write them down in my patch.

Jeffrey

On Fri, Dec 22, 2017 at 2:23 PM Carsten Bormann <[email-masked] <mailto:[email-masked]> > wrote:

Hi Jeffrey,

I’m not sure the direction should be to open this up; I think the recommendations should become more narrow as we learn about the practical issues.

We could write a separate document on the preferred c14n so we can keep this out of the main document.

I don’t think we necessarily want to encourage cross-over between int and float, so I think the “shortest float” rule should be applied independent of whether that cross-over is desired.

Why keep the “shortest int” rule less well defined for protocols that use bignums?  It should apply there as well, i.e., use bignums only for integers too large for the major type 0/1 formats.

Grüße, Carsten

> On Dec 22, 2017, at 23:13, Jeffrey Yasskin <[email-masked] <mailto:[email-masked]> > wrote:
>
> On Sun, Dec 3, 2017 at 6:15 AM Carsten Bormann <[email-masked] <mailto:[email-masked]> > wrote:
> On Nov 30, 2017, at 23:14, Jeffrey Yasskin <[email-masked] <mailto:[email-masked]> > wrote:
> >
> > Belatedly, I've discovered a user of "canonical" CBOR who's proposing a different map order than the RFC suggests: https://fidoalliance.org/specs/fido-v2.0-rd-20170927/fido-client-to-authenticator-protocol-v2.0-rd-20170927.html#message-encoding. (Note that this isn't a final standard yet and may change.)
>
>                 • If the major types are different, the one with the lower value in numerical order sorts earlier.
>
> as a major sorting rule before the existing RFC 7049 canonicalization rules:
>
>                 • If two keys have different lengths, the shorter one sorts earlier;
>                 • If two keys have the same length, the one with the lower value in (byte-wise) lexical order sorts earlier.
>
> This is different from simply going for byte-wise lexicographic (memcmp order(*)), which effectively would get us the first rule as the major sorting order already, but get rid of the length-based second rule (first rule in Section 3.9 of RFC 7049).
>
> I’ve come to see the putting the length comparison rule early in 3.9 as a major regression.
> One of the objectives when designing the CBOR serialization was not to repeat one big mistake that ASN.1 BER makes: to make overall lengths of complex composite items visible/important in the encoding of the next higher composite.
> > This is justified by the RFC saying that "Those protocols are free to define what they mean by a canonical format and what encoders and decoders are expected to do.  This section lists some suggestions for such protocols." That is (as Jim said), the RFC doesn't specify "canonical" CBOR: it just provides an option for higher-level protocols to do so.
>
> Right.  So the change would be to mention two options for this, the old canonical, and the saner (memcmp order) canonical.  Now the next step is finding names for legacy canonical/saner canonical.  We then have to decide whether we turn this into a separate document, at Proposed Standard level, or believe that adding another suggestion to 3.9 is essentially a bug fix and can be done in the Standard level document.
>
> > The use of a different order in CTAP is going to either force its implementers to write custom CBOR encoders and decoders or require the generic encoders to take a configuration option for the map order. If the generic encoders take an option, then it stops being an issue for CBORbis to suggest a different order.
>
> I've tried to implement this in https://github.com/cbor-wg/CBORbis/pull/9. I defined a core set of rules so that other specs can use them by reference, gave several examples of protocols that will need to extend the rules, and defined a second set of rules that match the canonical order that RFC7049 suggested.
>
> I don't have a strong opinion about which document should hold the canonicalization rules. Can we ask the IESG which they'd prefer?
>
> Jeffrey

---

From: Mike Jones <[email-masked]>
Date: Tue, 9 Jan 2018 05:45:08 +0000

+1
________________________________
From: CBOR <[email-masked]> on behalf of Jim Schaad <[email-masked]>
Sent: Monday, January 8, 2018 8:12:27 PM
To: 'Jeffrey Yasskin'
Cc: [email-masked]; 'Carsten Bormann'
Subject: Re: [Cbor] [core] draft-ietf-cbor-7049bis - Change suggested Canonicalization

Jim

Now that we're past the holidays, are there more comments on https://github.com/cbor-wg/CBORbis/pull/9?
On Fri, Dec 22, 2017 at 3:56 PM Jeffrey Yasskin <[email-masked]<mailto:[email-masked]>> wrote:
I don't think https://github.com/cbor-wg/CBORbis/pull/9 allows anything that RFC7049 didn't, which is the meaning I get from "open this up".

Jeffrey

On Fri, Dec 22, 2017 at 2:23 PM Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:
Hi Jeffrey,

Grüße, Carsten

> On Dec 22, 2017, at 23:13, Jeffrey Yasskin <[email-masked]<mailto:[email-masked]>> wrote:
>
> On Sun, Dec 3, 2017 at 6:15 AM Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:
> On Nov 30, 2017, at 23:14, Jeffrey Yasskin <[email-masked]<mailto:[email-masked]>> wrote:
> >

---

From: Jeffrey Yasskin <[email-masked]>
Date: Wed, 10 Jan 2018 06:24:45 +0000

Does that mean you support the PR as-is, without the changes Carsten
suggested?

On Mon, Jan 8, 2018 at 8:12 PM Jim Schaad <[email-masked]> wrote:

> I did not like the idea of changing the types of items when doing
> canonicalization.  I want to look just at what is given to me and thus do
> not like things like a floating point number may be encoded as an any of a
> number of different formats.  The choice of what an item is encoded as in
> terms of a floating point number is a function of the document
> specification and not the canonicalization encoding format.  For things
> like big numbers, the encoding should not be modified because the leading
> characters have been chosen by the application for a specific reason.
> Consider the leading padding that is part of a public key in cryptography
> where the length of the value is of equal or greater importance that the
> fact that is has leading 0 or 0xff bytes.  The rules for mapping between
> the data model and the encoding are as much up to the protocol
> specification as they are for the CBOR specification.  It is possible that
> a specification will only want to use the 64-bit floating point number
> format because it really simplifies the application even if it makes the
> encoded format longer.
>
>
>
> In sort, I do not believe that the rules should be extended beyond how an
> encoding works for a single encoded data format.
>
>
>
> Jim
>
>
>
>
>
> *From:* CBOR [mailto:[email-masked]] *On Behalf Of *Jeffrey Yasskin
> *Sent:* Monday, January 8, 2018 10:34 AM
> *To:* Jeffrey Yasskin <[email-masked]>
> *Cc:* [email-masked]; Carsten Bormann <[email-masked]>
> *Subject:* Re: [Cbor] [core] draft-ietf-cbor-7049bis - Change suggested
> Canonicalization
>
>
>

---

From: Jim Schaad <[email-masked]>
Date: Tue, 9 Jan 2018 23:09:53 -0800

No.

Look at what is at line 1328 in the diff and that is exactly what I am talking about as should not be done.

Jim

From: Jeffrey Yasskin [mailto:[email-masked]] 
Sent: Tuesday, January 9, 2018 10:25 PM
To: [email-masked]
Cc: Jeffrey Yasskin <[email-masked]>; [email-masked]; Carsten Bormann <[email-masked]>
Subject: Re: [Cbor] [core] draft-ietf-cbor-7049bis - Change suggested Canonicalization

Does that mean you support the PR as-is, without the changes Carsten suggested?

On Mon, Jan 8, 2018 at 8:12 PM Jim Schaad <[email-masked] <mailto:[email-masked]> > wrote:

Jim

From: CBOR [mailto:[email-masked] <mailto:[email-masked]> ] On Behalf Of Jeffrey Yasskin
Sent: Monday, January 8, 2018 10:34 AM
To: Jeffrey Yasskin <[email-masked] <mailto:[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> ; Carsten Bormann <[email-masked] <mailto:[email-masked]> >
Subject: Re: [Cbor] [core] draft-ietf-cbor-7049bis - Change suggested Canonicalization

Jeffrey

Hi Jeffrey,

Grüße, Carsten

---

From: Jeffrey Yasskin <[email-masked]>
Date: Wed, 10 Jan 2018 15:38:19 +0000

Line 1328 is not part of any of the defined canonicalization rules. It's an
example of a rule a document specification might choose to layer on top, as
your message anticipates.

You might be saying that no document specification should have int/float
fields that would be able to represent the same information two ways?
Otherwise, the document specification absolutely needs to pick one of the
representations for a given data value in order to have a canonical
representation.

Jeffrey

On Tue, Jan 9, 2018 at 11:10 PM Jim Schaad <[email-masked]> wrote:

> *From:* Jeffrey Yasskin [mailto:[email-masked]]
> *Sent:* Tuesday, January 9, 2018 10:25 PM
> *To:* [email-masked]
> *Cc:* Jeffrey Yasskin <[email-masked]>; [email-masked]; Carsten
> Bormann <[email-masked]>

---

From: Jeffrey Yasskin <[email-masked]>
Date: Wed, 31 Jan 2018 19:02:42 +0000

I'd like to get at least the change to canonicalization merged, even if
there's still some disagreement about the examples.

I think I've heard two conflicting pieces of advice w.r.t. the examples
though:

In https://www.ietf.org/mail-archive/web/cbor/current/msg00322.html,
Carsten appears to prefer that the canonicalization rules say to use the
shortest encoding of both floats and bignums, while in
https://www.ietf.org/mail-archive/web/cbor/current/msg00326.html Jim
appears to prefer that cbor-based protocols be able to define their
canonicalization to use only 64-bit floats and fixed-width bignums.

I've uploaded https://github.com/cbor-wg/CBORbis/pull/13 to omit all
changes to the float example and the description of tag canonicalization.
Does anyone object to merging just that part?

Thanks,
Jeffrey

On Wed, Jan 10, 2018 at 7:38 AM Jeffrey Yasskin <[email-masked]>
wrote:

> Line 1328 is not part of any of the defined canonicalization rules. It's
> an example of a rule a document specification might choose to layer on top,
> as your message anticipates.
>