Subject: Re: [Cbor] Representation Variants in CBOR

From: "Doug Ewell" <[email-masked]>
Date: Wed, 18 Jul 2018 14:44:07 -0700

Sean Leonard wrote:

> But it is counterintuitive and unproductive, because many software
> stacks encode Unicode strings in UTF-16 but permit unpaired
> surrogates, effectively treating “Unicode” strings as arbitrary
> sequences of 16-bits. These products include JavaScript generally,
> Windows, Java, Mozilla (somewhat, depends on the boundary interface),
> etc. See Section 8.2 of RFC 7159.

PMFJI... I remain convinced that virtually nobody who writes software
stacks that permit unpaired surrogates does so because they think
unpaired surrogates are a great thing to permit. I believe they do it
because they are confused about the difference between "code point" and
"Unicode scalar value," or they imagine it's too hard or inefficient to
handle this in code, or they think upstream or downstream systems will
allow them, and want to remain compatible with such systems.

The same goes for 6-byte, CESU-8–style encodings of supplementary
characters, such as <ED A0 80 ED B0 80> for U+10000. This has only been
invalid for the past seventeen years or so.

In the case of a "UTF-8" string that contains one of these, Thiago's
option of dropping the message on the floor and reporting an error seems
like a fine idea.
 
--
Doug Ewell | Thornton, CO, US | ewellic.org