Subject: [COSE] magic number, file format and tags

From: Laurence Lundblade <[email-masked]>
Date: Wed, 20 Jan 2021 09:12:07 -0800

One way to handle CBOR certs in files would be to use the CBOR magic number 55799 (0xd9d9f7) to identify it as CBOR and then define a tag to identify it as a CBOR-cert. Or just rely on the .cbor file extension and a CBOR-cert tag. 

It seems to be common practice to define a tag for CBOR protocol messages. Why not CBOR cert? COSE, CWT, CoSWID and others do it. I opened an issue <https://github.com/EricssonResearch/CBOR-certificates/issues/84> for it.

0xd9d9f7 also definitely distinguishes CBOR from PEM and DER.

LL

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 20 Jan 2021 18:28:40 +0100

On 2021-01-20, at 18:12, Laurence Lundblade <[email-masked]> wrote:
> 
> One way to handle CBOR certs in files would be to use the CBOR magic number 55799 (0xd9d9f7) to identify it as CBOR and then define a tag to identify it as a CBOR-cert.

A single tag will be enough.
(Having both would be useful if there is a meaningful way to handle these by tools that need to recognize this is CBOR, but not that these are CBOR certs.)

> Or just rely on the .cbor file extension and a CBOR-cert tag. 

Right.

> It seems to be common practice to define a tag for CBOR protocol messages. Why not CBOR cert? COSE, CWT, CoSWID and others do it. I opened an issue for it.
> 

Indeed, as would be any smartly chosen tag specifically for CBOR certs.

We may want to make this optional, which means that it wouldn’t go into deterministic encoding, but can easily prefixed (to a data item(!)) when writing the file.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 20 Jan 2021 18:56:01 -0500

Hi, I was thinking about this yesterday too, and after the discussion this
morning at COSE, I wrote:

         https://datatracker.ietf.org/doc/draft-richardson-cbor-file-magic/

which is at:
         https://github.com/mcr/cbor-magic-number

# Introduction

Since very early in computing, operating systems have sought ways to mark
which files could be proposed by which programs.

For instance, the Unix file(1) command, which has existed since 1973
({{file}}), has been able to identify many file formats for decades.

...

As CBOR becomes a more and more common encoding for artifacts, identifying
them as CBOR is probably not useful.

This document provides a way to encode a magic number into the beginning of a CBOR format file.
Two options are presented, with the intention of standardizing only one.

These proposals are invasive to how CBOR protocols are written to disk, but in both cases, the
proposed envelope does not require that the tag be transfered on the wire.

Some protocols may benefit from having such a magic on the wire if they
presently using a different (legacy) encoding scheme, and need to determine
before invoking a CBOR decoder if the sender is using the legacy scheme, or the new CBOR scheme.

# Requirements for a Magic Number

A magic number is ideally a unique fingerprint, present in the first 4 or 8 bytes of the file,
which does not change when the content change, and does not depend upon the length of the file.

Less ideal solutions have a pattern that needs to be matched, but in which some bytes need to be ignored.

# Proposal One

This proposal uses a CBOR Array of size two.
The first byte is therefore 0b100_00010 (0x82).

Array element number one is a CBOR integer in the range 0x80000000 to 0xffffffff.
This number is the magic number described below in {{magictable}}

For a magic number 0x87654321, this results in a total of a six byte sequence:

~~~~
  0b100_00010 0b000_11010 0x87 0x65 0x43 0x21
~~~~

Array element number two is whatever the original CBOR content is supposed to be.
Due the array construct with known size, there is no further syntax required.

# Proposal Two

This proposal uses a CBOR Sequence {{!RFC8742}}.

Array element number one is a CBOR integer in the range 0x80000000 to 0xffffffff.
This number is the magic number described below in {{magictable}}

For a magic number 0x87653412, this results in a total of a five byte sequence:

~~~~
  0b000_11010 0x87 0x65 0x34 0x12
~~~~

This is followed by one or more CBOR data items of whatever type was intended.

... and then some variations.

(I probably had more important work I should have been doing)

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Wed, 20 Jan 2021 20:07:42 -0500

On Wed, Jan 20, 2021 at 18:56:01 -0500, Michael Richardson wrote:
...
...
...
> This proposal uses a CBOR Sequence {{!RFC8742}}.

Recently, I was thinking about the fun fact that the Unicode string "bor"
and the byte string "BOR" end up getting encoded as "cbor" and "CBOR",
respectively.

Maybe that could be somehow (ab)used?  In general, I'm a big fan of
human-readable magic numbers.

Jeff.

---

From: David Waite <[email-masked]>
Date: Wed, 20 Jan 2021 18:30:41 -0700

There are a lot of ways to go here - for instance it could be a tag indicating a two element array, the first being an integer or a mime type

-DW

> On Jan 20, 2021, at 4:56 PM, Michael Richardson <[email-masked]> wrote:
> 
> 
> _______________________________________________
> COSE mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cose

---

From: Michael Richardson <[email-masked]>
Date: Wed, 20 Jan 2021 21:11:49 -0500

Josef 'Jeff' Sipek <[email-masked]> wrote:
    > Recently, I was thinking about the fun fact that the Unicode string
    > "bor" and the byte string "BOR" end up getting encoded as "cbor" and
    > "CBOR", respectively.

obiwan-[work/richardson/cbor-file-magic](2.6.6) mcr 10355 %echo -n cbor | cbor2diag.rb
"bor"

obiwan-[work/richardson/cbor-file-magic](2.6.6) mcr 10356 %echo -n CBOR | cbor2diag.rb
h'424F52'

so, the first case is true, but the second does not seem to be.

---

From: "Doug Ewell" <[email-masked]>
Date: Wed, 20 Jan 2021 19:16:00 -0700

> Since very early in computing, operating systems have sought ways to
> mark which files could be proposed by which programs.

The word "proposed" feels very awkward here. Maybe "opened" or "manipulated," or some recasting of the sentence that talks about files "belonging to" programs, or something similar.

--
Doug Ewell, CC, ALB | Thornton, CO, US | ewellic.org

-----Original Message-----
From: CBOR <[email-masked]> On Behalf Of Michael Richardson
Sent: Wednesday, January 20, 2021 16:56
To: [email-masked]
Cc: cose <[email-masked]>
Subject: [Cbor] CBOR magic number, file format and tags

Hi, I was thinking about this yesterday too, and after the discussion this morning at COSE, I wrote:

which is at:

# Introduction

Since very early in computing, operating systems have sought ways to mark which files could be proposed by which programs.

For instance, the Unix file(1) command, which has existed since 1973 ({{file}}), has been able to identify many file formats for decades.

...

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Wed, 20 Jan 2021 21:19:07 -0500

> obiwan-[work/richardson/cbor-file-magic](2.6.6) mcr 10356 %echo -n CBOR | cbor2diag.rb

Which is hex for 'B' 'O' 'R'. :)

I'm guessing cbor2diag.rb sees the raw data and hexdumps it to be helpful.
Here's what python has to say:

In [1]: import cbor

In [2]: cbor.loads(b'cbor')
Out[2]: 'bor'

In [3]: cbor.loads(b'CBOR')
Out[3]: b'BOR'

Note the difference in return type - the first is unicode string the second
is a byte string.

Jeff.

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 20 Jan 2021 22:18:03 -0500

"Doug Ewell" <[email-masked]> writes:
> The word "proposed" feels very awkward here. Maybe "opened" or
> "manipulated," or some recasting of the sentence that talks about
> files "belonging to" programs, or something similar.

When I first read that, I mis-read it as "processed by".  I suspect that
is what is intended.

But ... Am I missing something?  Isn't a magic number provided by RFC
8949 section 3.4.6?

   3.4.6.  Self-Described CBOR

   [...]
   Tag number 55799 is defined for this purpose, specifically for use at
   the start of a stored encoded CBOR data item as specified by an
   application.  It does not impart any special semantics on the data
   item that it encloses; that is, the semantics of the tag content
   enclosed in tag number 55799 is exactly identical to the semantics of
   the tag content itself.

   The serialization of this tag's head is 0xd9d9f7, which does not
   appear to be in use as a distinguishing mark for any frequently used
   file types.  [...]

Dale

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 21 Jan 2021 04:31:19 +0100

Challenge accepted.
CBOR diagnostic notation is quite versatile, you just have to tell cbor2diag.rb what you want.

cbor2diag.rb -t: bytes_as_text, i.e., use text form for bytes if possible.

$ printf cbor | cbor2diag.rb -t
"bor"
$ printf CBOR | cbor2diag.rb -t
‘BOR'

Oh, and, BTW:

cbor2diag.rb -e: try_decode_embedded, i.e., try embedded CBOR.

$ printf CBOR | cbor2diag.rb -te
<< 'OR' >>

(Do a “gem update” to get cbor2diag.rb version 0.5.9 for -t.)

Back to the magic number issue:

For CBOR data items, I prefer registering a tag and prefixing with that head (*) over prefixing with an array head and a string.  See, we have a magic number registry built right into CBOR...

For CBOR sequences, obviously prefixing the sequence with a CBOR-encoded (text or byte) string sounds quite good.  Making that optional (to save bytes when you don’t need it) requires that the sequence otherwise cannot start with that kind of string.

Grüße, Carsten

(*) E.g., tag 15309736 for the RAINS protocol:

>> CBOR::Tagged.new(15309736, "oooooo").to_cbor
=> "\xDA\x00\xE9\x9B\xA8foooooo"
>> CBOR::Tagged.new(15309736, "oooooo").to_cbor[2..].force_encoding(Encoding::UTF_8)
=> "雨foooooo"

Oh, for those who don’t speak Chinese:

$ trans 雨
雨
(Yǔ)

rain

Definitions of 雨
[ 简体中文 -> English ]

noun
    rain
        雨, 雨天, 霅

adjective
    rainy
        雨, 多雨的

雨
    rain

> On 2021-01-21, at 03:19, Josef 'Jeff' Sipek <[email-masked]> wrote:
>

---

From: Michael Richardson <[email-masked]>
Date: Thu, 21 Jan 2021 07:24:39 -0500

> When I first read that, I mis-read it as "processed by".  I suspect
    > that is what is intended.

    >    [...]  Tag number 55799 is defined for this purpose, specifically

That tells us that it might be CBOR.

It does not distinguish a CBOR encoded certificate from a CBOR encoded
RFC8366 artifact, for instance.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 21 Jan 2021 08:50:17 -0500

> "manipulated," or some recasting of the sentence that talks about files
    > "belonging to" programs, or something similar.

Oops. "processed" was the intended word.

---

From: John Mattsson <[email-masked]>
Date: Thu, 21 Jan 2021 23:19:21 +0000

Hi,

It seems like there is agreement to specify a CBOR tag for a CBOR certificate. I stated specifying this for the next -07 version, se below.

I started to specify a tag based on Laurance suggestion to tag an array wrapping the CBOR Sequence. I don't know how to best specify this data item. It would have been easy before when the CBOR Sequence was specified as a CDDL group. Now when CBORCertificate is specified as an array with human-readable text specifying that it is really a CBOR sequence, both CBORCertificate and [ CBORCertificate ] seems wrong....

Would a tag for a chain/bag or both be needed as well? If a tag for COSE_X509 with CBOR certificates is specified, that could potentially be used to tag a single certificate as well.

Maybe it is confusing to use the COSE_X509 CDDL for CBOR certificates? A new structure for CBOR certificates could then also be more compact by wrapping the certs in arrays instead of bstr.

---------

## CBOR Tags Registry {#tags}

This document registers the following entries in the "CBOR Tags" registry under the "Concise Binary Object Representation (CBOR) Tags" heading.

~~~~~~~~~~~
+------+------------------------------------------------------------+
|  Tag | X.509 Public Key Algorithms                                |
+======+============================================================+
| TDB6 | Data Item: [ CBORCertificate ]                             |
|      | Semantics: The CBORCertificate CBOR sequence wrapped in an |
|      |            array.                                          |
|      | Reference: This document                                   |
+------+------------------------------------------------------------+
| TDB7 | Data Item: COSE_X509                                       |
|      | Semantics: The content of each bstr is the bytes of a CBOR |
|      |            Certificate.                                    |
|      | Reference: This document                                   |
+------+------------------------------------------------------------+
~~~~~~~~~~~

---

From: Michael Richardson <[email-masked]>
Date: Thu, 21 Jan 2021 19:29:02 -0500

John Mattsson <[email-masked]> wrote:
    > It seems like there is agreement to specify a CBOR tag for a CBOR
    > certificate. I stated specifying this for the next -07 version, se
    > below.

I'm not sure what a _CBOR certificate_ is, and I suggest we don't use that
term for your document.  I think it's a CBOR-compressed PKIX certificate.
(CCPC if you like. Maybe someone can figure out how to backronym CCCP, and we
can them Soviet Russians)

    > I started to specify a tag based on Laurance suggestion to tag an array
    > wrapping the CBOR Sequence. I don't know how to best specify this data

I don't know either.

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 21 Jan 2021 21:37:29 -0500

> RFC8366 artifact, for instance.

Uh, yes?  They're both CBOR, right?  It's still the magic number for
CBOR.

Do you mean to say, you want a place in CBOR to place a magic number
that describes the particular kind of CBOR structure this is?  If so,
you should make that clear right at the start of the I-D.  Indeed, in
the Abstract, though it needs other work, too.  Checking again, it looks
like you don't really lay that out until section 6, whereas I only read
it through section 5.  My mistake, but lots of other people will make it
as well.

Dale

---

From: Michael Richardson <[email-masked]>
Date: Fri, 22 Jan 2021 11:17:00 -0500

> that describes the particular kind of CBOR structure this is?

Yes.

    > If so,
    > the Abstract, though it needs other work, too.  Checking again, it
    > looks like you don't really lay that out until section 6, whereas I
    > only read it through section 5.  My mistake, but lots of other people
    > will make it as well.

Better wording suggestions welcome.