Subject: [Cbor] Self Described CBOR - Little Endian

From: alex thompson <[email-masked]>
Date: Tue, 23 Mar 2021 04:50:29 +0000

​Given the prevalence of little endian hardware and the existence of the LE typed array tags it would be useful to opt-in to LE encoding of the additional information bytes as well. This would simplify decoders that are specific to LE platforms, particularly WebAssembly which is all LE and hasn’t defined a swap instruction.

The opt-in could be self described with tag 55798 (has the same non-unicode magic number characteristic as 55799). The beginning of a stream would default to BE so tag 55798 itself would appear as 0xd9d9f6. Then any following tags and descendant items would be LE.

For completeness, 55798 and 55799 could be nested within one another, switching the endianness as the decoder traverses a tree of items. Nested tags 24 and 63 would pass on their endianness to their embedded items. No effect on typed array tags 64-87.

It would be a breaking change for generic decoders since ignoring 55798 wouldn’t work as expected but I think the impact is very manageable and fits with CBOR’s stated goals of small code size and extensibility.

Alex

---

From: Felipe Gasper <[email-masked]>
Date: Wed, 24 Mar 2021 11:21:36 -0400

> On Mar 23, 2021, at 12:50 AM, alex thompson <[email-masked]> wrote:
> 

I, too, have lamented the use of big-endian byte order in CBOR, FWIW. I *think* my Perl generic decoder works on big-endian machines, but I don’t think I have access to any big-endian machines to actually _test_ it.

-F

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 24 Mar 2021 14:06:38 -0600

Coincidentally, I just got a very small generic WASM decoder working:

https://github.com/hildjj/cbor-wasm

Inspired by https://github.com/quartzjer/cb0r/, but works synchronously in streaming mode, like expat.

— 
Joe Hildebrand

> On Mar 22, 2021, at 10:50 PM, alex thompson <[email-masked]> wrote:
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 24 Mar 2021 14:09:55 -0600

The `s390x/node` docker image emulates big endian (slowly), and includes Perl 5.24.1

I test with "docker run --rm -it -v ${PWD}:/root/cbor -w /root/cbor s390x/node npm test"

— 
Joe Hildebrand

> On Mar 24, 2021, at 9:21 AM, Felipe Gasper <[email-masked]> wrote:
> 
> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 24 Mar 2021 21:27:01 +0100

On 2021-03-24, at 21:09, Joe Hildebrand <[email-masked]> wrote:
> I test with "docker run --rm -it -v ${PWD}:/root/cbor -w /root/cbor s390x/node npm test”

I used to have an iBook (PowerPC CPU, run as big-endian) set up as a CI system…

But to answer the original question, CBOR is big-endian because IETF protocols generally are.  Creating a little-endian fork of CBOR wouldn’t really help that much.  (Turns out my (endian-independent) code is usually shorter for big-endian transfer than it is for little-endian — remember that you'll have to byte-shift the numbers anyway to maintain alignment on the decoded side, so it’s not that much more effort to arrange them.)  And it would hurt a lot; even with a magic number in front of it (see [1] for our current view of how magic numbers for CBOR-based file encodings should look like).

Grüße, Carsten

[1]: https://www.ietf.org/archive/id/draft-ietf-cbor-file-magic-00.html

---

From: Thiago Macieira <[email-masked]>
Date: Wed, 24 Mar 2021 22:38:21 -0700

On Wednesday, 24 March 2021 13:27:01 PDT Carsten Bormann wrote:
> But to answer the original question, CBOR is big-endian because IETF
> protocols generally are.  Creating a little-endian fork of CBOR wouldn’t
> really help that much.  (Turns out my (endian-independent) code is usually
> shorter for big-endian transfer than it is for little-endian — remember
> that you'll have to byte-shift the numbers anyway to maintain alignment on
> the decoded side, so it’s not that much more effort to arrange them.)  And
> it would hurt a lot; even with a magic number in front of it (see [1] for
> our current view of how magic numbers for CBOR-based file encodings should
> look like).

On the dual side, my encoder would have been shorter on little-endian.

Current:
https://github.com/intel/tinycbor/blob/master/src/cborencoder.c#L279-L306

This code writes the 64-bit incoming value and then tries to find the position 
and size of the relevant data.

In little-endian, you don't need to find the position, since it's fixed. You 
just need to calculate the size.

    uint64_t buf[2];
    uint8_t *bufstart = (uint8_t *)buf[1];
    unsigned len = 1;

    buf[1] = ui;
    if (ui >= Value8Bit) {
        uint8_t more = 0;
        if (ui > 0xffU)
            ++more;
        if (ui > 0xffffU)
            ++more;
        if (ui > 0xffffffffU)
            ++more;
        len = 1U + (1U << more);
       --bufstart;
        *bufstart = Value8Bit + more;
    }
    *bufstart += shiftedMajorType;
    return append_to_buffer(encoder, bufstart, len)

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering

---

From: Laurence Lundblade <[email-masked]>
Date: Thu, 25 Mar 2021 10:48:53 -0700

For native implementation, the amount of extra object code for endian conversion seems pretty small. My guess is 100 or 200 bytes. My encoding function that does endian conversion, and preferred encoding and deals with alignment is 150 bytes of object code. When you start encoding and decoding complicated structures like nested maps with optional items and you add in duplicate label detection, the percentage of the overall implementation saved is small.

It might have been better for CBOR to choose little endian because most CPUs are, but at this point any unnecessary divergence would create interoperability issues.

LL

> On Mar 24, 2021, at 1:06 PM, Joe Hildebrand <[email-masked]> wrote:
>