Subject: [Cbor] Representation Variants in CBOR

From: Carsten Bormann <[email-masked]>
Date: Fri, 13 Jul 2018 21:01:39 -0400

There is a discussion going on in the thread about Tag 1 that I'm not
sure I understand.  Since representation variants seem to play a big
role in that discussion, I thought I should write up some background
on representation variants in CBOR.

(Paragraphs in parentheses are additional background information that
can be skipped.)

Grüße, Carsten

# Representation Variants in CBOR

In CBOR, there is often more than one way to serialize a data model
item.

For instance, the unsigned integer value 1 can be represented as 0x01,
0x1801, 0x190001, 0x2000000001, 0x210000000000000001.

(There are other approaches to this problem.  For example, according
to the encoding principles underlying UTF-8, the string "1" could be
represented as 0x31, 0xC0B1, 0xE080B1, or 0xF08080B1.  UTF-8 [RFC3629]
simply outlaws the longer representation variants, so there never is a
choice.  Another approach is to make the encoding non-redundant, as
for instance with the CoAP option length [RFC7252]: The single-nibble
encoding spans the values 0 to 12, the 1.5-byte encoding spans 13 to
268, and the 2.5-byte encoding spans 269 to 65804.  As with the UTF-8
encoding rules, there is never a choice, but here there isn't even a
need for code to check for outlawed values.)

Consistent encoding (called canonicalization in RFC 7049) aims at a
single defined serialization of a data model item.

However, there is a cost to consistent encoding (in particular when
maps are involved), so the basic mode of using CBOR does not employ
consistent encoding.

CBOR has been defined to enable very simple implementations for
applications in very constrained environments.  So it is relatively
modest in its hard and fast requirements.  However, there is an
expectation that an implementation not targeted at very constrained
environments (which we will call a full-function implementation here)
at least:

- accepts all representation variants (unless specifically told not
  to, e.g. when checking consistent encoding), and
- only emits the shortest of multiple alternative representation
  variants for each data item.

Today's implementations pretty much universally fulfill this
expectation for integers, string lengths, container counts, and tags,
but some do fall short for floating point values.

(Even otherwise proficient programmers are often scared of floating
point, or they may just not care about floating point values, or both.
There is example code in the RFC, as well as in many of the 50+ CBOR
implementations available as open source, so there is little excuse
for chickening out on fully implementing floating point.  Some
implementation languages do make it harder than necessary to operate
on floating point values, though -- implementers for those languages
have this writer's sympathy.)

A full-function implementation can decode any representation variant
that a constrained implementation might generate.  In the inverse
direction, there may be some limitations on the constrained side that
a CBOR-based protocol may want to make explicit: E.g., it could
restrict integer values to 32-bit ranges (int32 OR uint32) or even 16
bits where that is possible.  It could also do without 64-bit floating
point where constrained implementations might be limited to 32-bit
floating point libraries.

(This is probably where the discussion of Tag 1 currently going on in
the mailing list was touching representation variants first: 32-bit
floating point is not generally very useful for 1970-epoch based
times, but a full-function implementation sending a floating-point
epoch-based time will generate a 32-bit representation if the time
happens to fall on an integer multiple of currently 128 seconds.)

At the data model level (where tags are defined, and where CDDL
operates), such limitations only limit the range of values; a
full-function generic encoder will then only generate the
representation variants intended.  A constrained encoder theoretically
could be oblivious of this intent and, e.g., generate a 64-bit integer
representation; most likely, these implementations are highly
integrated between data model and serialization layers and thus will
already do the right thing.

(Theoretically, a constrained CBOR implementation could be written to
only ever accept a specific representation variant that is not the
shortest -- e.g. 0x190001 for the example above.  This would limit
interoperability considerably and therefore probably isn't a smart
move.)

CDDL does not have any language features to restrict representation
variants, as it operates at the data model level.  It does allow
restricting the range of a value such that a full-function encoder
would only use a certain subset of representation variants (e.g.,
float32 might be represented in 16-bit or 32-bit serialization, but
never needs 64-bit serialization).  However, there is no way to force
a representation variant that is longer than the shortest one
possible, and it is NOT RECOMMENDED to define CBOR protocols that
would require doing so.

(Yes, this also applies to floating point.  While a high-speed encoder
on a fast network might want to forego the couple of instructions
needed for selection of the shortest encoding, the decoder should
always be able to handle the shorter cases as well.)

---

From: Island Resort <[email-masked]>
Date: Sat, 14 Jul 2018 10:37:22 -0700

Hi Casrten,

My pursuit here is approaches to create interoperable CBOR protocols and how it relates to canonical encoding. Tag 1 is a side exploration and example mostly towards that end.

Neither of these are true, right?
Canonicalization gives interoperability — still need to specify which types your protocol uses and a lot more
Canonicalization is necessary for interoperability — interoperable CBOR protocol can, and in some cases must, use indefinite lengths and unsorted maps because of implementation cost
(It also turns out that canonicalization is not necessary for signing)

Seems like CBOR could use formal conventions for creating interoperable protocols. Something like this:

1) Fully specify protocol messages — This could be done in CDDL or in prose. Probably most protocols can be specified in CDDL. Those that are in the most constrained environments might have to resort to prose. This will (must) result in a clear list of data types that need to be supported. 

2) Specify the encoding rules — This could be by requiring full standard canonical encoding, or by picking and choosing from standard canonical encoding rules or by specifying some custom encoding rules. 

3) Specify max sizes — This may be optional for most protocols, but some need it. Sometimes it will be for individual fields, others for messages.

Once you have this specification, you pick the CBOR libraries that support the types and encodings you need along with the code and heap size constraints and other requirements. It’s less about current CBOR library implementations, more about clear specs for interop.

I’m not up to speed on CDDL yet, so I’m sure what’s possible with it.

To go on a bit further just on canonicalization, there are (only?) four ways encoding can vary:
1) integers — used either as a value or a length
2) indefinite vs definite lengths — for strings, maps and arrays
3) map — sorting order
4) float — encoding of NaN and infinity

Since there are only four, maybe we specify the canonical encoding for each individually and precisely. A protocol can choose which they want à la carte if they need to. Full canonical encoding would still be preferred and a SHOULD.

Each canonical encoding would be described tightly with MUSTs conditional on whether it is providing canonical encoding support. For example, “IF canonical float encoding is implemented, NaN MUST be expressed as 0xf97e00”. The current language for float is “might need to…”.

I’m here in Montreal and will be at the CBOR meeting. But maybe you didn’t want to hear that. :-)

LL

> On Jul 13, 2018, at 6:01 PM, Carsten Bormann <[email-masked]> wrote:
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Jim Schaad <[email-masked]>
Date: Sat, 14 Jul 2018 22:01:51 -0400

See comments in line

From: CBOR <[email-masked]> On Behalf Of Island Resort
Sent: Saturday, July 14, 2018 1:37 PM
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]
Subject: Re: [Cbor] Representation Variants in CBOR

Hi Casrten,

*	Canonicalization gives interoperability — still need to specify which types your protocol uses and a lot more

[JLS] This is true but can come at a cost.  Often times if you have a byte integer always using the two byte encoding of an integer is simpler but not canonical.   (  18XX )

*	Canonicalization is necessary for interoperability — interoperable CBOR protocol can, and in some cases must, use indefinite lengths and unsorted maps because of implementation cost

[JLS] This statement is completely false.   As an example an unsorted map is legal and interoperable but definitely not canonical.

*	(It also turns out that canonicalization is not necessary for signing)

[JLS] Due to the fact that we designed COSE that way, this is a true statement.

[JLS] Would not require a smallest length encoding if you only specified CDDL.  That would always need to be done in prose.  This is also only a true statement if there are no extension points which might have other things put in later.

[JLS] Or just by saying do something reasonable.

 

[JLS] I have no idea of what you mean here.  You can specify a value range either in terms of values or number of bytes for the maximum value.

3) map — sorting order

[JLS] Lengths of encoded float in terms of bytes. 

5) For types with multiple encodings, which one are you going to use.

6) Do you tag explicitly or implicitly.  CWT uses implicit tagging for encoding dates but allows both integer and floating point types.

[JLS] For ASN.1, people almost never managed to get BER encoding messed up.  They got DER encoding messed up on a regular bases.  This maps to ‘Use reasonable encoding’ vs ‘Use canonical encoding’.  It was also far harder to deal with getting DER encoding done on small devices and in very tight code due to the fact that one often needed to encode multiple times to get the DER encoding correct.

Jim

LL

On Jul 13, 2018, at 6:01 PM, Carsten Bormann <[email-masked] <mailto:[email-masked]> > wrote:

Grüße, Carsten

_______________________________________________
CBOR mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/cbor

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 14 Jul 2018 22:18:34 -0400

Comment below

> On Jul 14, 2018, at 10:01 PM, Jim Schaad <[email-masked]> wrote:
> 
> See comments in line
>  
> From: CBOR <[email-masked] <mailto:[email-masked]>> On Behalf Of Island Resort
> To: Carsten Bormann <[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]>

I’m saying that both statements are false to show that canonicalization is neither sufficient or necessary for interoperability. 

LL

---

From: Jim Schaad <[email-masked]>
Date: Sun, 15 Jul 2018 06:20:34 -0400

Sorry, I missed the first line.

From: Laurence Lundblade <[email-masked]> 
Sent: Saturday, July 14, 2018 10:19 PM
To: Jim Schaad <[email-masked]>
Cc: Carsten Bormann <[email-masked]>; [email-masked]
Subject: Re: [Cbor] Representation Variants in CBOR

Comment below

On Jul 14, 2018, at 10:01 PM, Jim Schaad <[email-masked] <mailto:[email-masked]> > wrote:

See comments in line

From: CBOR < <mailto:[email-masked]> [email-masked]> On Behalf Of Island Resort
Sent: Saturday, July 14, 2018 1:37 PM
To: Carsten Bormann < <mailto:[email-masked]> [email-masked]>
Cc:  <mailto:[email-masked]> [email-masked]
Subject: Re: [Cbor] Representation Variants in CBOR

Hi Casrten,

LL

---

From: Sean Leonard <[email-masked]>
Date: Tue, 17 Jul 2018 17:10:53 -0400

> On Jul 14, 2018, at 1:37 PM, Island Resort <[email-masked]> wrote:
> 

There are many other ways that encoding can vary, when tagged data items come into play.

Consider a small positive integer that can be encoded as a big number: it could be an integer (major type 0) or a byte string with tag 2 (bignum). The byte string for 1 could be h'01', h'0001, h'000001', etc. The application may consider the length of the byte string to be an important signal, e.g., of precision.

Naturally, such a number could also be represented as a bigfloat (type 5).

Conversely…if an implementation knows how to process bignums, it knows 2(h'01') is equivalent to 2(h'0001'), but untagged h'01' is not equivalent to h'0001'. A decoder that is tag-agnostic will (likely) treat the tagged version the same as the untagged version, which is to say, not equivalent.

The question is when you want data to be “canonicalized”, what are the range of types that you are going to accept, and what might your application consider to be equivalent representations…which may depend not only on the application protocol, but the implementing platform. Python has no hard limit or range to ints; therefore, the single most natural way to represent integers in CBOR with a Python implementation would be bignums (although, for most real world cases, this would not be very efficient).

> Each canonical encoding would be described tightly with MUSTs conditional on whether it is providing canonical encoding support. For example, “IF canonical float encoding is implemented, NaN MUST b…

Making canonicalization conditional will quickly lead to 2^(tags+4) possible implementation variations, where tags is the number of allocated tags in the IANA registry. :(

Sean

>> On Jul 13, 2018, at 6:01 PM, Carsten Bormann <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 17 Jul 2018 18:02:46 -0400

On Jul 17, 2018, at 17:10, Sean Leonard <[email-masked]> wrote:
> Consider a small positive integer that can be encoded as a big number:

Don’t do that, that’s not “preferred serialization”.

> it could be an integer (major type 0) or a byte string with tag 2 (bignum). The byte string for 1 could be h'01', h'0001, h'000001', etc. The application may consider the length of the byte string to be an important signal, e.g., of precision.
> 

Again, don’t do that, use plain float.

> Conversely…if an implementation knows how to process bignums, it knows 2(h'01') is equivalent to 2(h'0001'), but untagged h'01' is not equivalent to h'0001'. A decoder that is tag-agnostic will (li…

That is something that the consistent encoding spec for an application would need to define.
So the spec should probably say that preferred encoding for bignums does not have leading zeros.

(Right now, it just says

   Decoders
   that understand these tags MUST be able to decode bignums that have
   leading zeroes.

Which implies, but does not spell out, that leading zeroes are a bit weird.)

> The question is when you want data to be “canonicalized”, what are the range of types that you are going to accept, and what might your application consider to be equivalent representations…which m…

The natural way is to use bignums only for numbers that don’t fit in uint/nint.

> Making canonicalization conditional will quickly lead to 2^(tags+4) possible implementation variations, where tags is the number of allocated tags in the IANA registry. :(

Hence the desire to define a preferred serialization.

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 17 Jul 2018 22:10:28 -0400

I like preferred serialization. For me at least, it is getting clear what is a type and what is serialization. Anything to do with type is not serialization and we manage interop of that in the description of the types used by the protocol. 

These are all the preferred serializations, right?

Major type 0 and 1 — encode integers as short as possible

Major type 2 and 3 — encode the length of the string as short as possible; do not use indefinite lengths.

Major type 4 — encode the length of the array as short as possible; do not use the indefinite length.

Major type 5 — same as major type 5; in addition sort by keys

I like stating them by major type so a design like by dumb sensor + smart server can say:
   - The dumb sensor doesn’t have to use any preferred serializations for data it sends
   - The smart server must use the preferred serialization for major type 2 and 3

As Carsten is saying, the variation for things like expression of time or a URL is not a serialization variant because you can be explicit about in the CDDL, in the data model.

I would expect implementations that translate data formats to python, JSON or such would always use all preferred serializations because they really aren’t constrained. Also sending preferred serialization to a constrained device is almost always best so this works out well even when talking to a constrained device with a python implementation.

LL

> On Jul 17, 2018, at 6:02 PM, Carsten Bormann <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 17 Jul 2018 22:26:42 -0400

Hi Laurence,

I’m mostly with you here, except for:

> On Jul 17, 2018, at 22:10, Laurence Lundblade <[email-masked]> wrote:
> 

I don’t think the sorting can be part of the preferred serialization, this would be consistent serialization only.  It is just too expensive.

Major type 6 — encode the tag as short as possible

Major type 7 — encode floating point as short as possible.

So maybe we can factor out that “as short as possible” and just state it as a general rule, because it is a general rule.

# If there is more than one serialization variant available that differ in overall length, the preferred serialization uses the shortest one.

Now, indefinite actually is shorter for, e.g., arrays of more than 255 elements, so the preference for definite should be explicitly *higher priority* than the overall length.  On the other hand, the preference for definite is *weaker* — there are many good reasons to go to indefinite length in particular for arrays and maps (otherwise we wouldn’t have that variant!).

Grüße, Carsten

---

From: Sean Leonard <[email-masked]>
Date: Wed, 18 Jul 2018 01:05:32 -0400

> On Jul 17, 2018, at 10:26 PM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Laurence,
> 

+1. (To me, this validates that there is a canonicalization “ladder”—the more you want to canonicalize, the higher you have to climb up the ladder. And it does seem that this is a ladder to the heavens since there is always “one more thing” that you just might want to canonicalize…)

> So maybe we can factor out that “as short as possible” and just state it as a general rule, because it is a general rule.

+1. I agree with this, and with covering “as short as possible” across all major types.

I also think that major type 3 needs more work in its preferred serialization, namely, that the UTF-8 is “well-formed” as that term is used in the Unicode standard. The current text of RFC7049 does not require that the UTF-8 be well-formed. Shortest encoding is one aspect of being well-formed, but not the only aspect.

> Now, indefinite actually is shorter for, e.g., arrays of more than 255 elements, so the preference for definite should be explicitly *higher priority* than the overall length.  On the other hand, t…

I do not know if being in definite form is desirable for this “preferred serialization” business, for reasons that Carsten mentions. Forcing maps and arrays to be in definite form requires that the number of items be known in advance. (This is better than ASN.1, which requires pre-serializing all of the elements, but is still not the best.) Indefinite form has notable advantages, including having an encoder be able to stream out content infinitely (think…indefinite media stream, indefinite sensor stream, etc.).

Sean

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 18 Jul 2018 03:30:43 -0400

> On Jul 18, 2018, at 01:05, Sean Leonard <[email-masked]> wrote:
> 
> 
> 

Hmm, a ladder with one step (use consistent encoding or don’t) is not very laddery.  Maybe more like stilts :-)

> I also think that major type 3 needs more work in its preferred serialization, namely, that the UTF-8 is “well-formed” as that term is used in the Unicode standard. The current text of RFC7049 does…

I can’t find “well-formed” in RFC 3629.

CBOR does not require any specific form (e.g. normalization form) of the UTF-8 strings, but they do have to be UTF-8 strings.

Consistent encoding requirements of an application may go beyond those of just the CBOR serialization and may include requirements on the data in the data items.  E.g., a case-insensitive application may need to convert strings to lower-case.  Or a Unicode application may need to normalize (in NFC, one might hope).  Neither is the business of CBOR, which is just carrying the text strings.  And constrained nodes usually can’t really solve the LETTER BEH WITH HAMZA ABOVE issue anyway.

> I do not know if being in definite form is desirable for this “preferred serialization” business, for reasons that Carsten mentions. Forcing maps and arrays to be in definite form requires that the…

I agree that the “preferredness” is weaker here.  But in many cases array and map sizes are known, and in that case (i.e., absence of a good reason to go non-preferred) definite is still preferred.

Grüße, Carsten

---

From: Sean Leonard <[email-masked]>
Date: Wed, 18 Jul 2018 09:17:32 -0400

> On Jul 18, 2018, at 3:30 AM, Carsten Bormann <[email-masked]> wrote:
> 
> 
> 

https://www.unicode.org/versions/Unicode11.0.0/ch03.pdf

D85: A Unicode code unit sequence that purports to be in a Unicode encoding form is called well-formed if and only if it does follow the specification of that Unicode encoding form.
Summary of points:

* Must be in shortest encoded form (no C0 5E stuff: identical criteria as CBOR shortest encoded form, therefore not objectionable)
* No “ill-formed subsequences” (example given in D86 is <41 C2 C3 B1 42>: C2 is ill-formed because it needs to have a value in the range 00-7F after it to correspond to a Unicode scalar value)
* UTF-8 must only encode “Unicode scalar values”: this is a little opaque but a “Unicode scalar value” (D76) is any code point (0 - 1FFFFF) *except* surrogate code points. Therefore, well-formed UTF-8 cannot contain U+D800-U+DFFF.

See also WTF-8:
https://simonsapin.github.io/wtf-8/

This requirement for well-formed UTF-8 is both obvious and extremely counterintuitive (and potentially unproductive), depending on how you look at it. It is “obvious” because one expects the shortest encoded form, and if you encode a surrogate pair (one U+D800-U+DB00, followed by one U+DC00-U+DFFF), you get 6 bytes of UTF-8 to represent a Unicode scalar value in the range U+10000-U+1FFFFF, instead of 4 bytes of UTF-8 where the first byte starts with F0-F4. (Also: F5-FF can never appear in UTF-8. It would have been an interesting and perfectly valid design choice to use FF as a terminal marker for an indefinite-length UTF-8 string in CBOR.)

But it is counterintuitive and unproductive, because many software stacks encode Unicode strings in UTF-16 but permit unpaired surrogates, effectively treating “Unicode” strings as arbitrary sequences of 16-bits. These products include JavaScript generally, Windows, Java, Mozilla (somewhat, depends on the boundary interface), etc. See Section 8.2 of RFC 7159.

It could be a security issue to let such exceptional subsequences to go through, but probably the worst that will happen in practice is that emoji get snuck through an emoji scrubber. 😂

The bigger point I guess is that to the extent that CBOR is a strict superset of JSON, JSON does not formally prohibit unpaired surrogates. (It prohibits unpaired surrogates, arguably, by requiring the JSON source text to be in UTF-8, UTF-16, or UTF-32, but it explicitly enables unpaired surrogates in escaped form, see Section 8.2 of RFC 7159.) In fact, to the extent that JSON is used as the serialization format between two implementations that both use 16-bit “UTF-16” without enforcing surrogate pairing, support for unpaired surrogates in JSON is probably necessary for interoperability. One such reason is because if a naive implementation splits a Unicode string right between surrogate pairs, the implementation ought also to reassemble the string properly which will make it good again rather than delete the temporarily offending code points.

If you really want a preferred serialization for CBOR, I do believe that the whole UTF-8 encoding thing has to be addressed. Otherwise, you treat UTF-8 strings exactly like byte strings, but that is not helpful to a relying application higher up in the stack.

Sean

---

From: Laurence Lundblade <[email-masked]>
Date: Wed, 18 Jul 2018 14:13:26 -0400

I am contemplating writing a PR…

Would a new sub section called “Preferred Encoding” be part of "Specification for CBOR Encoding”, rather than "Creating CBOR-Based Protocols”? I’m leaning towards the former since it not about creating a specific protocol.

Feel dumb that I forgot to say tags should be as short as possible.

For the shortest float preference, we could use the steps described in the 7049 canonicalization text. That is definitely some work for implementors and may not be commonly implemented. I’ve checked a few and not found an implementation. 

What about not requiring shortest representation for floats? It would make the preferred encoding for floats not the smallest and not consistent across implementations (not canonical), but might give better interop and might be implemented more commonly.

I think we will need some description for preferred decoders too:
* must decode indefinite and definite lengths
* must decode integers and lengths encoded as 64-bits, 32-bits, 16-bits and 8-bits (even on an 8-bit machine)
* must decode all types of float including half-precision (when I was in Qualcomm using half-precision code in the RFC required talking to a lawyer; writing my own would have been hard).

LL

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 19 Jul 2018 06:57:51 +1200

> If you really want a preferred serialization for CBOR, I do believe that the whole UTF-8 encoding thing has to be addressed. Otherwise, you treat UTF-8 strings exactly like byte strings, but that i…

We hit a very mild form of this problem while designing the GRASP protocol,
which is encoded in CBOR. After seeking advice from the gods of internationalization,
we concluded that the only viable solution was indeed "treat UTF-8 strings
exactly like byte strings." I would **very** strongly suggest keeping the many
horrible issues of UTF-8 canonicalization out of the already complex
issues of CBOR canonicalization. It's a layer 7 problem and should be
left there.

(One result in GRASP is that if someone wants to name a GRASP objective
with a name that includes Cyrillic "с", the "pile-of-poo" symbol and
several line feeds, they are welcome to do so, but it's not a protocol problem.)

Regards
   Brian

---

From: Sean Leonard <[email-masked]>
Date: Wed, 18 Jul 2018 15:09:36 -0400

> On Jul 18, 2018, at 2:57 PM, Brian E Carpenter <[email-masked]> wrote:
> 

The issues I pointed out are wellformedness/definitional issues. A string of bytes that includes 0xFF is not UTF-8 in any way, shape, or form; it is an error on the same level as a definite-length CBOR map or array having 0xFF inside. (No UTF-8 can include 0xFF, not even ill-formed ones, since even in the legacy range of ISO 10646 going to U+7FFFFFFF, the first byte has to be 0b1111110x, which is not 0xFF.) Please also do not confuse “canonicalization” with “Unicode Normalization Forms”, which are a totally different thing (and I agree, normalization is a higher level problem).

If integers are going to have a “preferred serialization”, then UTF-8 strings also ought to have a “preferred serialization”. Symmetry.

Regards,

Sean

---

From: Thiago Macieira <[email-masked]>
Date: Wed, 18 Jul 2018 12:47:20 -0700

On Wednesday, 18 July 2018 12:09:36 PDT Sean Leonard wrote:
> The issues I pointed out are wellformedness/definitional issues. A string of
> bytes that includes 0xFF is not UTF-8 in any way, shape, or form; it is an
> error on the same level as a definite-length CBOR map or array having 0xFF
> inside. (No UTF-8 can include 0xFF, not even ill-formed ones, since even in
> the legacy range of ISO 10646 going to U+7FFFFFFF, the first byte has to be
> 0b1111110x, which is not 0xFF.) Please also do not confuse
> “canonicalization” with “Unicode Normalization Forms”, which are a totally
> different thing (and I agree, normalization is a higher level problem).
> 
> If integers are going to have a “preferred serialization”, then UTF-8
> strings also ought to have a “preferred serialization”. Symmetry.

That's a different thing, though. RFC 7049 currently only places UTF-8 
validation in the Strict Mode, so some implementations treat it as a byte 
array and will not perform any inspection on the content.

Other implementations will, especially those that decode the content into 
their internal string representations. Those cannot cope with a \xFF in the 
UTF-8 content or, for that matter, anything higher than \xF8. They will also 
reject overlong encodings (\xC1 \x81) and UTF-8 decoding into UTF-16 surrogate 
code units.

Therefore, for interoperability, encoders must

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 19 Jul 2018 08:14:02 +1200

> The issues I pointed out are wellformedness/definitional issues. A string of bytes that includes 0xFF is not UTF-8 in any way, shape, or form; it is an error on the same level as a definite-length…

Sorry about that terminology glitch, but it doesn't change what I'm trying to
say. So, if we simply say "treat UTF-8 strings exactly like byte strings",
and if the sender sends an invalid UTF-8 string, that's exactly what the
receiver will get. Are you saying that the CBOR encoder should be looking
for invalid UTF-8?
 
> If integers are going to have a “preferred serialization”, then UTF-8 strings also ought to have a “preferred serialization”. Symmetry.

Sure. And I think it should be "treat UTF-8 strings exactly like byte strings".
Isn't that what running code does? 

Regards
   Brian

---

From: Sean Leonard <[email-masked]>
Date: Wed, 18 Jul 2018 16:23:42 -0400

> On Jul 18, 2018, at 3:47 PM, Thiago Macieira <[email-masked]> wrote:
> 

Did you need to finish that sentence or paragraph? Just checking. :-)

Sean

---

From: Thiago Macieira <[email-masked]>
Date: Wed, 18 Jul 2018 13:29:38 -0700

On Wednesday, 18 July 2018 13:23:42 PDT Sean Leonard wrote:
> Did you need to finish that sentence or paragraph? Just checking. 

... not send malformed UTF-8 strings. That's the same for all Strict Mode 
requirements.

---

From: Thiago Macieira <[email-masked]>
Date: Wed, 18 Jul 2018 13:30:28 -0700

On Wednesday, 18 July 2018 13:14:02 PDT Brian E Carpenter wrote:
> Sorry about that terminology glitch, but it doesn't change what I'm trying
> to say. So, if we simply say "treat UTF-8 strings exactly like byte
> strings", and if the sender sends an invalid UTF-8 string, that's exactly
> what the receiver will get. Are you saying that the CBOR encoder should be
> looking for invalid UTF-8?

It should be actively avoiding them, if possible. If an invalid UTF-8 string 
is sent, the receiver may not get anything.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 19 Jul 2018 08:42:55 +1200

> array and will not perform any inspection on the content.

That seems to be by far the most likely situation in constrained nodes.
 
> Therefore, for interoperability, encoders must

I wish you'd finished that sentence. If you meant "encoders must check for
valid UTF-8" I think that's wrong; the requirement is surely that *senders*
must send valid UTF-8. There is no requirement that the encoder validates it,
and in constrained nodes it won't. If you want to send Недостаточно средств
you'd better encode it correctly in the first place, that's all.

(Also, what happened to the Postel robustness principle here?)

    Brian

---

From: Thiago Macieira <[email-masked]>
Date: Wed, 18 Jul 2018 14:10:07 -0700

> (Also, what happened to the Postel robustness principle here?)

Be conservative in what you send, be liberal in what you accept?

Sure, the encoder should strive to not send invalid UTF-8 anymore than 
duplicated keys in a map. The problem here is that the liberal receiver may do 
one of many different things, so you can't guarantee communication:

 * drop the entire message on the floor (strict decoder) and report error
 * replace the invalid sequence with U+FFFD
 * discard the string, replacing with an undefined
 * ignore the second instance of a key in a map
 * overwrite the first instance of the key in a map
 * decode both, but have unspecified behaviour which one is found when 
    searching

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 18 Jul 2018 19:07:12 -0400

> https://www.unicode.org/versions/Unicode11.0.0/ch03.pdf

Ah.  That is not relevant for RFC 7049.  UTF-8 is defined in RFC 3629.

I believe RFC 3629 is very clear about what is UTF-8 and what is not, without needing to resort to gradations like well-formed but invalid (and most of what could be considered a bit on the side of weasel-words are caused by the abortive idea of carrying a BOM in UTF-8).

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Thu, 19 Jul 2018 13:34:44 -0400

I wrote up a PR straw man for preferred serialization <https://github.com/cbor-wg/CBORbis/pull/28>.

LL

> On Jul 18, 2018, at 2:13 PM, Laurence Lundblade <[email-masked]> wrote:
>

---

From: Jim Schaad <[email-masked]>
Date: Sun, 22 Jul 2018 06:13:01 -0700

+1 - do not go into the i18n canonicalization world.

Jim

> -----Original Message-----
> From: CBOR <[email-masked]> On Behalf Of Brian E Carpenter
> Sent: Wednesday, July 18, 2018 11:58 AM
> To: Sean Leonard <[email-masked]>; Carsten Bormann <[email-masked]>
> Cc: [email-masked]; Laurence Lundblade <[email-masked]>
> > If you really want a preferred serialization for CBOR, I do believe that the
> whole UTF-8 encoding thing has to be addressed. Otherwise, you treat UTF-8
> strings exactly like byte strings, but that is not helpful to a relying application
> higher up in the stack.
> 
> which is encoded in CBOR. After seeking advice from the gods of
> internationalization, we concluded that the only viable solution was indeed
> "treat UTF-8 strings exactly like byte strings." I would **very** strongly suggest
> keeping the many horrible issues of UTF-8 canonicalization out of the already
> complex issues of CBOR canonicalization. It's a layer 7 problem and should be
> left there.
> 
> (One result in GRASP is that if someone wants to name a GRASP objective with
> a name that includes Cyrillic "с", the "pile-of-poo" symbol and several line
> feeds, they are welcome to do so, but it's not a protocol problem.)
> 
> Regards
>    Brian
> 
> >>> I also think that major type 3 needs more work in its preferred
> serialization, namely, that the UTF-8 is “well-formed” as that term is used in
> the Unicode standard. The current text of RFC7049 does not require that the
> UTF-8 be well-formed. Shortest encoding is one aspect of being well-formed,
> but not the only aspect.
> >>
> >> CBOR does not require any specific form (e.g. normalization form) of the
> UTF-8 strings, but they do have to be UTF-8 strings.
> >
> > D85: A Unicode code unit sequence that purports to be in a Unicode encoding
> form is called well-formed if and only if it does follow the specification of that
> Unicode encoding form.
> > Summary of points:
> >
> > * Must be in shortest encoded form (no C0 5E stuff: identical criteria
> > as CBOR shortest encoded form, therefore not objectionable)
> > * No “ill-formed subsequences” (example given in D86 is <41 C2 C3 B1
> > 42>: C2 is ill-formed because it needs to have a value in the range
> > 00-7F after it to correspond to a Unicode scalar value)
> > * UTF-8 must only encode “Unicode scalar values”: this is a little opaque but a
> “Unicode scalar value” (D76) is any code point (0 - 1FFFFF) *except* surrogate
> code points. Therefore, well-formed UTF-8 cannot contain U+D800-U+DFFF.
> >
> > See also WTF-8:
> > This requirement for well-formed UTF-8 is both obvious and extremely
> > counterintuitive (and potentially unproductive), depending on how you
> > look at it. It is “obvious” because one expects the shortest encoded
> > form, and if you encode a surrogate pair (one U+D800-U+DB00, followed
> > by one U+DC00-U+DFFF), you get 6 bytes of UTF-8 to represent a Unicode
> > scalar value in the range U+10000-U+1FFFFF, instead of 4 bytes of
> > UTF-8 where the first byte starts with F0-F4. (Also: F5-FF can never
> > appear in UTF-8. It would have been an interesting and perfectly valid
> > design choice to use FF as a terminal marker for an indefinite-length
> > UTF-8 string in CBOR.)
> >
> > But it is counterintuitive and unproductive, because many software stacks
> encode Unicode strings in UTF-16 but permit unpaired surrogates, effectively
> treating “Unicode” strings as arbitrary sequences of 16-bits. These products
> include JavaScript generally, Windows, Java, Mozilla (somewhat, depends on
> the boundary interface), etc. See Section 8.2 of RFC 7159.
> >
> > It could be a security issue to let such exceptional subsequences to
> > go through, but probably the worst that will happen in practice is
> > that emoji get snuck through an emoji scrubber. 😂
> >
> > The bigger point I guess is that to the extent that CBOR is a strict superset of
> JSON, JSON does not formally prohibit unpaired surrogates. (It prohibits
> unpaired surrogates, arguably, by requiring the JSON source text to be in UTF-
> 8, UTF-16, or UTF-32, but it explicitly enables unpaired surrogates in escaped
> form, see Section 8.2 of RFC 7159.) In fact, to the extent that JSON is used as
> the serialization format between two implementations that both use 16-bit
> “UTF-16” without enforcing surrogate pairing, support for unpaired surrogates
> in JSON is probably necessary for interoperability. One such reason is because
> if a naive implementation splits a Unicode string right between surrogate pairs,
> the implementation ought also to reassemble the string properly which will
> make it good again rather than delete the temporarily offending code points.
> >
> > If you really want a preferred serialization for CBOR, I do believe that the
> whole UTF-8 encoding thing has to be addressed. Otherwise, you treat UTF-8
> strings exactly like byte strings, but that is not helpful to a relying application
> higher up in the stack.
> >
> > Sean
> >
> >>
> >> Consistent encoding requirements of an application may go beyond those of
> just the CBOR serialization and may include requirements on the data in the
> data items.  E.g., a case-insensitive application may need to convert strings to
> lower-case.  Or a Unicode application may need to normalize (in NFC, one
> might hope).  Neither is the business of CBOR, which is just carrying the text
> strings.  And constrained nodes usually can’t really solve the LETTER BEH WITH
> HAMZA ABOVE issue anyway.
> >>
> >>>> # If there is more than one serialization variant available that differ in
> overall length, the preferred serialization uses the shortest one.
> >>>>
> >>>> Now, indefinite actually is shorter for, e.g., arrays of more than 255
> elements, so the preference for definite should be explicitly *higher priority*
> than the overall length.  On the other hand, the preference for definite is
> *weaker* — there are many good reasons to go to indefinite length in
> particular for arrays and maps (otherwise we wouldn’t have that variant!).
> >>>
> >>> I do not know if being in definite form is desirable for this “preferred
> serialization” business, for reasons that Carsten mentions. Forcing maps and
> arrays to be in definite form requires that the number of items be known in
> advance. (This is better than ASN.1, which requires pre-serializing all of the
> elements, but is still not the best.) Indefinite form has notable advantages,
> including having an encoder be able to stream out content infinitely
> (think…indefinite media stream, indefinite sensor stream, etc.).
> >>
> >> I agree that the “preferredness” is weaker here.  But in many cases array
> and map sizes are known, and in that case (i.e., absence of a good reason to go
> non-preferred) definite is still preferred.
> >>
> >> Grüße, Carsten
> >>
> >

---

From: Sean Leonard <[email-masked]>
Date: Sun, 22 Jul 2018 07:07:08 -0700

> On Jul 18, 2018, at 7:07 PM, Carsten Bormann <[email-masked]> wrote:
> 

I do not think anybody is considering putting BOMs in tstr (CBOR UTF-8 text strings). Also, none of this has to do with Unicode internationalization/canonicalization/normalization. That is a totally different subject.

Recourse to RFC 3629 confirms the points that I raised. Here is text from RFC 3629:

   o  The octet values C0, C1, F5 to FF never appear.
   o  The byte-value lexicographic sorting order of UTF-8 strings is the
      same as if ordered by character numbers.  Of course this is of
      limited interest since a sort order based on character numbers is
      almost never culturally valid.

   The definition of UTF-8 prohibits encoding character numbers between
   U+D800 and U+DFFF, which are reserved for use with the UTF-16
   encoding form (as surrogate pairs) and do not directly represent
   characters.  When encoding in UTF-8 from UTF-16 data, it is necessary
   to first decode the UTF-16 data to obtain character numbers, which
   are then encoded in UTF-8 as described above.  This contrasts with
   CESU-8 [...]

   Implementations of the decoding algorithm above MUST protect against
   decoding invalid sequences.  For instance, a naive implementation may
   decode the overlong UTF-8 sequence C0 80 into the character U+0000,
   or the surrogate pair ED A1 8C ED BE B4 into U+233B4.  Decoding
   invalid sequences may have security consequences or cause other
   problems.  See Security Considerations (Section 10) below.

   NOTE -- The authoritative definition of UTF-8 is in [UNICODE].  This
   grammar is believed to describe the same thing Unicode describes, but
   does not claim to be authoritative.  Implementors are urged to rely
   on the authoritative source, rather than on this ABNF.

Security Considerations

   Implementers of UTF-8 need to consider the security aspects of how
   they handle illegal UTF-8 sequences.  It is conceivable that in some
   circumstances an attacker would be able to exploit an incautious
   UTF-8 parser by sending it an octet sequence that is not permitted by
   the UTF-8 syntax.

   A particularly subtle form of this attack can be carried out against
   a parser which performs security-critical validity checks against the
   UTF-8 encoded form of its input, but interprets certain illegal octet
   sequences as characters.  For example, a parser might prohibit the
   NUL character when encoded as the single-octet sequence 00, but
   erroneously allow the illegal two-octet sequence C0 80 and interpret
   it as a NUL character.  Another example might be a parser which
   prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the
   illegal octet sequence 2F C0 AE 2E 2F.  This last exploit has
   actually been used in a widespread virus attacking Web servers in
   2001; thus, the security threat is very real.

   Another security issue occurs when encoding to UTF-8: the ISO/IEC
   10646 description of UTF-8 allows encoding character numbers up to
   U+7FFFFFFF, yielding sequences of up to 6 bytes.  There is therefore
   a risk of buffer overflow if the range of character numbers is not
   explicitly limited to U+10FFFF or if buffer sizing doesn't take into
   account the possibility of 5- and 6-byte sequences.

So RFC 3629 is indeed clear, and it incorporates all of the points in the Unicode standard, which have not actually changed between the publication of RFC 3629 (which references Unicode 4.0 normatively) and the current Unicode 11.0.

The implications for CBOR are clear:

#1: To the extent that CBOR needs to determine whether two tstrs are equal (e.g., for map key comparison), UTF-8 needs to be decoded and checked to Unicode code points. A tstr that contains <C0 80> and a tstr that contains <00> encodes the same code point: U+0000. Therefore, there is a collision. Furthermore, <C0 80> is illegal (“ill-formed”) anyway. Replacing <C0 80> with U+FFFD <EF BF BD> would be bad. A naive memcmp will miss this.

The CBOR decoder can have several possible behaviors in the <C0 80> + <00> case. It could drop one key/value pair. It could drop the other key/value pair. It could signal a fatal error. But it really should not treat the two strings as “different keys” (which it would if they were bstrs), nor should it try to rewrite <C0 80> as U+FFFD or undefined or similar.

The same analysis goes for <FF 3D> and <FE 3D>. Both represent “=”. They are different bstrs, but the same tstrs. A naive bit-for-bit comparison will let both through as unique keys, which is an exploitable problem. Rejecting both as ill-formed UTF-8 is okay; recognizing both as equivalent to “=” is also okay (to the extent that they then are seen as duplicate keys in the map). Even recognizing both as “” (dropping everything after the first ill-formed subsequence) or “U+FFFD=“ (replacing) would be okay since they would still be recognized as duplicates. But letting both through as unique keys would be bad.

Really the analysis is the same for a CBOR map that contains uint keys that are encoded as <08> and <00 08>. The difference is that <00 08> is legal in RFC 7049 CBOR, whereas <C0 80> and <FF 3D> are not legal in UTF-8.

#2: I think that any “preferred serialization” needs to point out that UTF-8 needs to be checked. But a generic decoder that checks for equality of map keys also needs to check UTF-8. I would argue that all generic CBOR decoders SHOULD check UTF-8. The operation is cheap (and actually can be handled in parallel on a multi-threaded CPU). I do not see a performance issue on resource-constrained devices, since I thought that resource-constrained devices won’t be receiving tstrs that often anyway; if they do, ASCII tstrs are the most likely, so the code paths for > U+007F won’t be exercised all that often.

#3: Read the WTF-8 document for dealing with unpaired surrogates. The way I see it is that the industry kind of does a wink and a nod in this regard (at least, many UTF-8 decode implementations do the wink-and-nod). If an unpaired surrogate is encountered (a U+D800-U+DFFF taken in isolation, not talking about the CESU-8 6-byte case vs. proper UTF-8 4-byte case), the robust way to handle it is to let it pass. The reason is because from CBOR’s perspective, it does not impact the security of the equality operation on map keys, because there is no alternative serialization than that one 3-byte sequence. It is probably best to say nothing about that topic (WTF-8) in any CBOR documents, as long as we go into it with eyes open.

#4: I see the equality issue in programming terms: the CBOR decoder ought to say:
 if (key1 == key2) { /* deal with collision */ }
 else { /* add (key2, value2) to in-memory map */ },
where the == operator is overloaded to deal with:
 1) integers (includes CBOR tags),
 2) floats,
 3) binary strings (memcmp),
 4) UTF-8 strings.
With 2) floats, you call the IEEE 754-compliant comparison instruction on the CPU. With 4) UTF-8 strings, you call the “utf8check” and “utf8cmp” functions, however your platform/OS/library implements that. (Note that “utf8cmp” === memcmp only after you know “utf8check” succeeds. This is also why unpaired surrogates are okay, since if the local utf8check lets unpaired surrogates go through, that code point can’t possibly be rewritten to anything else anyway.)

Hope this helps,

Sean

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 22 Jul 2018 21:42:14 +0200

> The implications for CBOR are clear:

Not to me.  I don’t agree that any of these is an implication.

Most of this actually would be an unmotivated deviation from the established existing practice; we don’t do that when moving a Proposed Standard to Internet Standard.

Specifically, the purpose of the reference to RFC 3629 is to have a stable definition of UTF-8.
There is no need to repeat any of this material in the definition of CBOR.

It is up to an application whether checking of the UTF-8 encoding rules actually needs to be performed on incoming data or the UTF-8 strings can just be shipped around unchanged.  If the application specified strict processing, then UTF-8 has to be checked (section 3.10).  That checking has to follow RFC 3629, not any rules we make up in the CBOR specification.

Grüße, Carsten

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 23 Jul 2018 09:01:06 +1200

> Not to me.  I don’t agree that any of these is an implication.

I agree. It's matter of choice. If we choose to define equality
of two UTF-8 strings as bitwise equality that's perfectly
consistent. An *application* of CBOR might wish to do otherwise
but that's before encoding or after decoding.

I do agree that map keys need very clear thinking, but if we
*define* key equality for UTF-8 keys as bitwise equality, we bump
the problem to the application, which is where it should be IMHO.

Specifically:

> resource-constrained devices won’t be receiving tstrs that often anyway; if they do, ASCII tstrs are the most likely, so the code paths for > U+007F won’t be exercised all that often.

That would increase the footprint of the decoder anyway. Memory
counts in constrained nodes, as well as the code path.

In the text of 7049bis, "Specifying Keys for Maps" probably needs
to clarify that UTF-8 keys are compared bitwise. Somewhere we need
to state that UTF-8 normalization is out of scope for encoders/decoders.

(I realise Sean won't agree, but at least we need to be clear.)

    Brian

---

From: Sean Leonard <[email-masked]>
Date: Sun, 22 Jul 2018 18:09:10 -0700

> It is up to an application whether checking of the UTF-8 encoding rules actually needs to be performed on incoming data or the UTF-8 strings can just be shipped around unchanged.  If the applicatio…

Sorry but I am rather confused by this response. Nowhere in my e-mail 
did I state or suggest a change to RFC 7049. The texts of both RFC 3629 
and RFC 7049 are fine as they are. In fact, I directly quoted the text 
of RFC 3629. That is also why I said: "*To the extent* that CBOR needs 
to determine whether two tstrs are equal (e.g., for map key 
comparison)", and, "a generic decoder *that checks for equality of map 
keys* also needs to check UTF-8". I readily acknowledge that there are 
CBOR decoders (for example, small streaming decoders) whose design 
requirements preclude checking for equality of map keys, and therefore, 
that do not operate in Strict Mode.

I did not specifically distinguish Strict Mode by name, however, so 
sorry about not calling that out.

I acknowledge that I also made a stronger statement: "I would argue that 
all generic CBOR decoders SHOULD check UTF-8." I still stand by this 
statement, but did not, and do not, propose any changes to the RFC 7049 
text.

RFC 3629 makes a normative statement: "Implementations of the decoding 
algorithm above MUST protect against decoding invalid sequences." That 
is actually a change from RFC 2279, and is noted as such in Section 12 
RFC 3629.

There is a legitimate question as to whether "UTF-8 decoding" is 
happening at all when dealing with CBOR. To the extent that CBOR is 
ingested by a decoder and tstrs are converted to 16-bit code units, 
i.e., UTF-16 (as, for example, in many JavaScript engines and APIs, 
including but not limited to Windows and Apple/macOS/iOS), the 
UTF-8-to-UTF-16 decoding algorithm is triggered. Similarly, if the 
source representation is in 16-bit code units (UTF-16 or something like 
it), the UTF-16-to-UTF-8 encoding algorithm will always be triggered 
when doing CBOR encoding. A properly implemented UTF-16-to-UTF-8 
encoding algorithm will never emit overlong encodings or out-of-range 
bytes (C0, C1, F5-FF).

However, there are other situations when the tstr content is not changed 
when CBOR is decoded (i.e., remains in 8-bit units, ostensibly UTF-8). 
It is possible that the decoding algorithm is therefore not triggered in 
those implementations.

Alarm bells always sound in my head when data crosses a security 
boundary. CBOR passing between two mutually trusted components is not a 
security boundary. Loading CBOR from disk into memory is *probably* not 
a security boundary, but it depends. However, when stuff is sent and 
received across the Internet between different vendor implementations, 
there is an obvious security boundary. The stuff purporting to be 
"UTF-8" therefore really ought to be checked in most network-connected 
circumstances. It is important that it be checked once. It need not be 
checked repeatedly, but the problem is when the implementation in one 
layer thinks that the checking will be done at a different layer, and 
vice versa, resulting in no checks.

There is a real danger that a CBOR decoder, which has no need to convert 
from UTF-8 to UTF-16 in-memory, will just slap a label on the chunk of 
data as "UTF-8 = good" and pass it along.† What happens, then, when that 
in-memory CBOR needs to be serialized back out? The encoder then needs 
to check whether the UTF-8 really *is* good, especially if it's 
operating in "Strict Mode" (actually defined by RFC 7049) or some 
"preferred serialization" (the subject of this thread).

So, Thiago's comment on July 18: "Sure, the *encoder* should strive to 
not send invalid UTF-8 anymore than duplicated keys in a map." is 
actually correct. But is it the responsibility of the encoder to check 
UTF-8 when the other parts of the software stack have already 
represented that the UTF-8 is good? How many implementers can raise 
their hands today and say, "yes, my CBOR implementation checks that the 
UTF-8 string in memory really is well-formed UTF-8 before it is encoded 
on the wire?"

None of this stuff is new. See Section 3 "Non-Visual Security Issues" of 
Unicode Technical Report #36, Unicode Security Considerations. 
<https://www.unicode.org/reports/tr36/> In particular: Sections 3.1, 
3.5, 3.6, and 3.7. Ignore the sections that talk about normalization 
because they are not relevant for this conversation (nothing in this 
conversation has to do with NFC or whatever).

I still maintain that no text from RFC 7049 needs to be changed in this 
regard. But, proposed text about "bitwise comparison" and the like 
should not be added either.

Regards, Sean

†Malicious UTF-8 (overlong encoding, for example) is not always 
exploitable. If the malicious data is compared bitwise against a fixed 
trusted string, it will fail since the trusted string will be 
well-formed but the malicious string is ill-formed.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 Jul 2018 19:02:18 +0200

Hi Sean,

apparently, I read your message a bit out of context.

> I acknowledge that I also made a stronger statement: "I would argue that all generic CBOR decoders SHOULD check UTF-8." I still stand by this statement, but did not, and do not, propose any changes to the RFC 7049 text.

(For me, this is not an RFC 2119 “SHOULD”, but an RFC6919 “OUGHT TO”, “WOULD PROBABLY”, or maybe even “MAY WISH TO”.)

As a quality of implementation statement, a generic CBOR decoder that delivers invalid UTF-8 to its application and potentially then has the application blow up with an exception in the unrelated place where that UTF-8 is actually being used, is not so nice.  But the trade-offs are hard to weigh from the high vantage point of a data interchange format specification.

> RFC 3629 makes a normative statement: "Implementations of the decoding algorithm above MUST protect against decoding invalid sequences." That is actually a change from RFC 2279, and is noted as such in Section 12 RFC 3629.

Since the decoding is not done by CBOR, that requirement transfers right to the UTF-8 decoder (if any) in the application.

> There is a legitimate question as to whether "UTF-8 decoding" is happening at all when dealing with CBOR. To the extent that CBOR is ingested by a decoder and tstrs are converted to 16-bit code units, i.e., UTF-16 (as, for example, in many JavaScript engines and APIs, including but not limited to Windows and Apple/macOS/iOS), the UTF-8-to-UTF-16 decoding algorithm is triggered.

Right, many 1990’s languages have that problem, and a generic CBOR decoder implementation for such a language probably wants to perform that decoding for the application.  RFC 3629 applies.

> Similarly, if the source representation is in 16-bit code units (UTF-16 or something like it), the UTF-16-to-UTF-8 encoding algorithm will always be triggered when doing CBOR encoding. A properly implemented UTF-16-to-UTF-8 encoding algorithm will never emit overlong encodings or out-of-range bytes (C0, C1, F5-FF).

Correct.

> However, there are other situations when the tstr content is not changed when CBOR is decoded (i.e., remains in 8-bit units, ostensibly UTF-8). It is possible that the decoding algorithm is therefore not triggered in those implementations.

As in many languages designed after the time when “Unicode” meant “16-bit”.

> Alarm bells always sound in my head when data crosses a security boundary. CBOR passing between two mutually trusted components is not a security boundary. Loading CBOR from disk into memory is *probably* not a security boundary, but it depends. However, when stuff is sent and received across the Internet between different vendor implementations, there is an obvious security boundary. The stuff purporting to be "UTF-8" therefore really ought to be checked in most network-connected circumstances. It is important that it be checked once. It need not be checked repeatedly, but the problem is when the implementation in one layer thinks that the checking will be done at a different layer, and vice versa, resulting in no checks.

Indeed, this is a general security consideration of RFC 3629.

> There is a real danger that a CBOR decoder, which has no need to convert from UTF-8 to UTF-16 in-memory, will just slap a label on the chunk of data as "UTF-8 = good" and pass it along.† 

That may be what a performance-oriented generic decoder does.  (It may not even touch the data and just point to it by reference, as in cn-cbor.)

> What happens, then, when that in-memory CBOR needs to be serialized back out? The encoder then needs to check whether the UTF-8 really *is* good,

No, it doesn’t — it does not have that onus.

> especially if it’s operating in “Strict Mode" (actually defined by RFC 7049)

Then it does.

> or some "preferred serialization" (the subject of this thread).

Not sure that has anything to do with UTF-8 validation.

> So, Thiago's comment on July 18: "Sure, the *encoder* should strive to not send invalid UTF-8 anymore than duplicated keys in a map." is actually correct. But is it the responsibility of the encoder to check UTF-8 when the other parts of the software stack have already represented that the UTF-8 is good? How many implementers can raise their hands today and say, “yes, my CBOR implementation checks that the UTF-8 string in memory really is well-formed UTF-8 before it is encoded on the wire?"

That would be an interesting subject of investigation, similar to the floating point investigation you just started.

Fun:
http://cbor.me/?diag=%22%FF%22
http://cbor.me/?bytes=61(FF)

(So much for my “quality of implementation” comment above, but then the problem probably is in the decoders/encoders for diagnostic notation, which was never meant for interchange.)

> None of this stuff is new. See Section 3 "Non-Visual Security Issues" of Unicode Technical Report #36, Unicode Security Considerations. <https://www.unicode.org/reports/tr36/> In particular: Sections 3.1, 3.5, 3.6, and 3.7. Ignore the sections that talk about normalization because they are not relevant for this conversation (nothing in this conversation has to do with NFC or whatever).

No, but the application may actually require NFC…

> I still maintain that no text from RFC 7049 needs to be changed in this regard. But, proposed text about “bitwise comparison" and the like should not be added either.

Because UTF-8 has no representation variants (not considering normalization), bitwise comparison is fine for map key comparison.  Unless the application needs normalized map keys, in which case some additional checking is probably required in the application anyway.

Grüße, Carsten