Subject: [Cbor] RFC7049bis processing of unknown tags

From: Michael Richardson <[email-masked]>
Date: Wed, 06 May 2020 11:32:39 -0400

After discussion about #176/#181 I submitted #182:

https://github.com/cbor-wg/CBORbis/issues/182

RFC7049 specified that CBOR tags which were not recognized should be ignored.

RFC7049bis wishes to change this behaviour such that unknown tags would not
be ignored, but would at least, be presented to the application for further
determination. This is a change that would render existing CBOR parsers
instantly invalid.

---

The suggestion is that parsers should be in RFC7049 mode by default, and
applications that want RFC7049bis behaviour should initialize the parser with
an option that enables it [or use a new parser with awareness].

Applications that want to make use of tags defined in RFC7049bis need to put
the parser in RFC7049bis mode.

---

I think that Carsten does not agree with my suggested solution, but I'm not
attached to it.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 6 May 2020 18:06:37 +0200

Hi Michael,

Thank you for your interjection at the CBOR interim.
I think this is a good issue to roll up.

> RFC7049 specified that CBOR tags which were not recognized should be ignored.

That is not exactly what it says:

(3.5)
   A decoder that comes across a tag (Section 2.4) that it does not
   recognize, such as a tag that was added to the IANA registry after
   the decoder was deployed or a tag that the decoder chose not to
   implement, might issue a warning, might stop processing altogether,
   might handle the error and present the unknown tag value together
   with the contained data item to the application (as is expected of
   generic decoders), might ignore the tag and simply present the
   contained data item only to the application, or take some other type
   of action.

So there always was a choice.
Note that there always was a preference to present the unknown tag to the application, for “generic decoders”; i.e., the choice is more for application-specific decoders.

> determination. This is a change that would render existing CBOR parsers

A change that would remove the choice is not what was intended so far, just increased emphasis on the options:

— 1 might issue a warning, 
— 2 might stop processing altogether,
— 3 might handle the error and present the unknown tag value together
   with the contained data item to the application (as is expected of
   generic decoders), 

as opposed to

— 4 might ignore the tag and simply present the
   contained data item only to the application, or 
— 5 take some other type
   of action.

Generally, it is a good idea if users of a library know what they will get, so the behavior to be expected needs to be documented.  A generic decoder that only does 3 (as is “expected”) will be the most interoperable one.

There are two pieces of text in 7049bis that may not entirely be aligned:

(3.4:)
   Decoders do not need to understand tags of every tag number, and tags
   may be of little value in applications where the implementation
   creating a particular CBOR data item and the implementation decoding
   that stream know the semantic meaning of each item in the data flow.
   Their primary purpose in this specification is to define common data
   types such as dates.  A secondary purpose is to provide conversion
   hints when it is foreseen that the CBOR data item needs to be
   translated into a different format, requiring hints about the content
   of items.  Understanding the semantics of tags is optional for a
   decoder; it can simply present both the tag number and the tag
   content to the application, without interpreting the additional
   semantics of the tag.

But also:

(7.1:)
   CBOR has three major extension points:

   […]

   *  the "tag" space (values in major type 6).  Again, only a small
      part of the codepoint space has been allocated, and the space is
      abundant (although the early numbers are more efficient than the
      later ones).  Implementations receiving an unknown tag number can
      choose to simply ignore it (process just the enclosed tag content)
      or to process it as an unknown tag number wrapping the tag
      content.  The IANA registry in Section 9.2 is the appropriate way
      to address the extensibility of this codepoint space.

> The suggestion is that parsers should be in RFC7049 mode by default,

But there is no RFC 7049 mode — there is a choice.

> and

It definitely is a good idea to either always get behavior 3, or to provide flags that control the behavior.

> the parser in RFC7049bis mode.

RFC7049bis does not define any new tags (so far).

> I think that Carsten does not agree with my suggested solution, but I'm not

I hope I have explained how the situation is a bit more nuanced than might have come over in the meeting.

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Thu, 7 May 2020 10:57:02 -0700

The upshot seems to be, that if you put a tag into encoded CBOR you better really mean it, because you can’t count on it being ignored by apps and/or decoders that don’t understand it. Tags are not like HTTP or RFC 822 headers where unknowns are explicitly to be ignored.

Looking through the registered tags, most define a new data type. In most cases decoders can’t ignore the new type and successfully implement the protocol using it.

Also, in the registry, just a few tags are conversion hints. These seem like the ones to ignore because 1) they are hints and 2) the decoder might not be converting them and 3) applicability is only in certain operating environments.

However, there is no way to generically know if a tag is a hint, particularly for newly defined tags, so decoders and apps can’t be counted on to ignore even hints and it comes back to the same. One can’t expect tags to be ignored, so don’t put them in unless they are really necessary.  Probably a good thing to discourage rampant use of gratuitous tagging.

LL

> On May 6, 2020, at 9:06 AM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Michael,
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 7 May 2020 15:49:50 -0700

I think at least some of this change was my doing, so I'll send three
opinions.

1) I should have changed the text in 7.1, "Implementations receiving an
unknown tag number can choose to simply ignore it (process just the
enclosed tag content)", and I must have just missed it.
2) I don't think it makes sense for any parser to see a tag and give the
application its contents instead of the tagged contents. That said, this is
a change that does make some RFC7049-compliant parsers non-compliant with
RFC7049-bis, which is a downside. On the third hand, the parsers will still
exist, applications using them don't need to upgrade if it breaks their
protocol.
3) While RFC7049-bis is already somewhat overstepping the bounds of a
standard in specifying error-handling options, it'd be even farther out of
bounds to specify parser API options. :)

Jeffrey

On Wed, May 6, 2020 at 9:07 AM Carsten Bormann <[email-masked]> wrote:

> Note that there always was a preference to present the unknown tag to the
> application, for “generic decoders”; i.e., the choice is more for
> application-specific decoders.
>
> Generally, it is a good idea if users of a library know what they will
> get, so the behavior to be expected needs to be documented.  A generic
> decoder that only does 3 (as is “expected”) will be the most interoperable
> one.
>

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 May 2020 19:42:19 -0400

Jeffrey Yasskin <[email-masked]> wrote:
    > 3) While RFC7049-bis is already
    > somewhat overstepping the bounds of a standard in specifying
    > error-handling options, it'd be even farther out of bounds to specify
    > parser API options. :)

I disagree strongly with this part :-)

It's is a regular and common mistake in IETF/many specifications that they do
not adequately explain what the behaviour is supposed to be when a newer
option shows up.  We probably spend 70% of our time at the IETF making sure
that we don't break old code.
Is that too high?  Well, maybe, because we don't spend the effort up-front.

As an example: IKE (IPsec) has a major and minor version.
We have no idea what it would mean to increment the minor version.
Having a minor version was a waste of time.

(I crafted the text for the VendorID payload in IKEv1.
I believe that I got it right, and made sure that we could make
use of private extensions without conflicts in the number space)

As for API options, yes, we often do need to give advice on providing ways to
put things into compatibility mode, and what that means.  At no point are we
writing down some enum or #define though.

I believe that I am less concerned about having rfc7049bis render some
existing parsers "non-compliant" --- they *ARE* still compliant to RFC7049.

I am a user of parsers, I have occasionally had to write my own conversions,
but mostly I would say that I am not up to speed on some of the details.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 7 May 2020 19:38:47 -0700

> but mostly I would say that I am not up to speed on some of the details.

That's all reasonable. :) Given the discussion, and Lawrence's quick
analysis of the existing tags, how do you currently feel about the state of
RFC7049bis's requirements around error handling?

* I'm not opposed to advice for parsers to have an option to treat a value
tagged with an unknown tag as equivalent to the value itself.
* I dislike the idea of any generic parser doing that by default, I think
based on reasoning like in
https://tools.ietf.org/html/draft-iab-protocol-maintenance-04.
* If a parser passes unknown tags up to the application, a higher-level
protocol can ignore them itself, skip their data item, return an error, or
do something else appropriate to the context. So if the RFC should
recommend a default in generic parsers, I'd vote for that one.
* I don't intend to draft this wording myself. :)

Jeffrey

---

From: Laurence Lundblade <[email-masked]>
Date: Thu, 7 May 2020 21:22:50 -0700

The more core issue for me is how protocols and protocol implementations handle unknown tags. Generic decoders are an implementation detail in a way. It is still useful to discuss then, but what drives the CBOR specification should be the effect on an end-end protocol.

In most use cases where a tag defines a new data type it will not be useful to ignore the tag. An implementation that doesn’t understand the new data type is  out of date and incompatible. However, it is useful to entirely skip the full data item with the unknown tag, but it seems preferable to handle that with map keys/labels or such. If you don’t know the map key/label, skip the data item. An unknown tag shouldn’t break the ability to do that.

In some cases of type definition it might be useful to interpret the data item as if it were untagged, but from a quick look at the registry, it doesn’t seem to be useful that often. For example, if you are tagging something as a positive bignum, you probably really to know that to do anything with the data item. Ignoring a COSE tag isn’t going to be productive in any way. (It is possible that you know it is a positive bignum by other means than tagging, but in that case, you should be tagging in the first place; optional tagging is discouraged).

The analogy doesn’t completely work, but it seems kind of like ignoring type-defining tags is like ignoring  typedefs in C.

It's the case of conversion hints, where ignoring tags can work. But you have to know the purpose of the tag is a conversion hint and that its not a new type definition. Conversion hints don’t seem to the the main purpose of tagging, critically useful, or widely used, so it doesn’t seem critical to change CBOR to accommodate them.

From what Cartsen said it seems it would be an incompatible change from 7049 to require tags be ignored and it doesn’t seem that necessary.

If I’d put anything in CBORbis, it would be a warning that one should not count on unknown tags being ignored in CBOR, unless the protocol specification  says they must be ignored and how to do it and what it means.

LL

Also, it seems fine to define protocol modes that get implemented as API options, but it is the mode that is defined not the API.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 14 May 2020 12:31:39 -0400

Carsten and Paul, my appologies for the length of this email.
I've been working on this an hour or so a day for a week :-(
I will probably reply and emphasis my important points.

This document says it is going to Internet Standard, but *only* in the Contributing section.
Please also forgive me that I haven't re-read RFC7049 in a long while, so I
might be complaining about things that are just matching 7049.

Jeffrey Yasskin <[email-masked]> wrote:
    >> I am a user of parsers, I have occasionally had to write my own
    >> conversions, but mostly I would say that I am not up to speed on some
    >> of the details.

    > analysis of the existing tags, how do you currently feel about the
    > state of RFC7049bis's requirements around error handling?

Maybe time for a top-to-bottom read. Probably appropriate thing to do during WGLC :-)

Intro does say: "It does not create a new version of the format."

I think that I'd like to amend this to say that:

   This document is a revised edition of [RFC7049], with editorial
   improvements, added detail, and fixed errata.  This revision formally
   obsoletes RFC 7049, while keeping full compatibility of the
   interchange format from RFC 7049.  It does not create a new version
   of the format.

->
   This document is a revised edition of [RFC7049], with editorial
   improvements, added detail, and fixed errata.  In clarifying some
   interpretations of [RFC7049] it may in some cases create situations where
   an existing parser may no longer comply to this specification.
   While this revision formally obsoletes RFC 7049, it does not obsolete
   any valid encoders, and thus keeps full compatibility with the
   interchange format from RFC 7049.  It does not create a new version
   of the format.

I note this text (section 2.1):

   While there is a strong expectation that generic encoders and
   decoders can represent "false", "true", and "null" ("undefined" is
   intentionally omitted) in the form appropriate for their programming
   environment, implementation of the data model extensions created by
   tags is truly optional and a matter of implementation quality.

This seems to have something to do with tags.

Also section 2.2 ends with:

   "0.0" as an integer (major type 0, Section 3.1).  However, if a
   specific data model declares that floating-point and integer
   representations of integral values are equivalent, using both map
   keys "0" and "0.0" in a single map would be considered duplicates,
   even while encoded as different major types, and so invalid; and an
   encoder could encode integral-valued floats as integers or vice
   versa, perhaps to save encoded bytes.

To me, this says that all generic encoders that intend to return data in the
native form of their programming environment need to be configured as to the
protocol.  This is supporting my suggestion that a well designed library
would/could have to be configured for the specific data model when it comes
to how unknown tags are treated.

    > * I'm not opposed to advice for parsers to have an option to treat a
    > value tagged with an unknown tag as equivalent to the value itself.

    > https://tools.ietf.org/html/draft-iab-protocol-maintenance-04.  * If a
    > parser passes unknown tags up to the application, a higher-level
    > protocol can ignore them itself, skip their data item, return an error,
    > or do something else appropriate to the context. So if the RFC should
    > recommend a default in generic parsers, I'd vote for that one.  * I
    > don't intend to draft this wording myself. :)

I think that I'm arguing for generic parsers to be in RFC7049 mode, which
might mean ignoring unknown tags (if that's what they did before), passing
the data up using whatever native interpretation there is, until they are
configured otherwise.

That is, if there is a seconds-since-epoch tag (XXX) which a generic parser
did not understand, followed by an integer, that it would return an integer
if it did not have an interface that passed tags.

>   28, 29, 30:  These values are reserved for future additions to the
>      CBOR format.  In the present version of CBOR, the encoded item is
>      not well-formed.

I think that there is a bug here.
What should a parser written today do when it encounters these values?
(forward reference to section 7.2?)
Getting this right is how we deal with future-proofing.
It seems seeing such a thing means a current decoder has to abort/fail.
What we write here has a profound implication, I think, on how easily we
could act on the advice of section 7.2.  Section 10, first paragraph implies
we should say something.

In general, I think that the details in this introductionary encoding section
are too detailed, particularly for 31.  I think that detail belongs later
on. I got no value (I retained nothing) from having that level of detail there.

I wonder if section 3.1, under major type 0 should give clarify that "0"
is encoded as 0b000_00000. (That is no negative 0)

   "A string containing an invalid UTF-8 sequence is well-
    formed but invalid."

I think that this might need clarification.

I guess that RFC8742 include sequences of 7049bis CBOR sequences.
I wonder if Updates 8742 is appropriate.

>   If the break stop code appears after a key in a map, in place of that
>   key's value, the map is not well-formed.

This does mean that the entire map is not well-formed, or just the
key/value pair where this occurs?  I take the first meaning, but I want to be
sure.

3.2.3:
   (Note that zero-length
   chunks, while not particularly useful, are permitted.)

they might be useful in non-TCP/IP situations where it is useful to send a
"keep-alive" on some channel.

I think that it might be cleaner to swap the order of sections 3.2 (infinite
length things), and 3.3 (floating-point and stuff).  This just puts major
type 7 more in context first.

>   As with all other major types, the 5-bit value 24 signifies a single-
>   byte extension: it is followed by an additional byte to represent the
>   simple value.  (To minimize confusion, only the values 32 to 255 are
>   used.)  This maintains the structure of the initial bytes: as for the

Or, to put it another way, 5-bit Values 24->31  in table 3 are also "Simple
Values".    Could future Simple Values (such as 0..19) can, have complex
structure the way that values 24->27 do?

Or to put it another way, can a decoder depend upon unassigned simple values
having the one-or-two byte structure presented and be able to skip unknown
values?  Or does a decoder that encounters undefined values here have to
fail?

>   formed.  (This implies that an encoder cannot encode false, true,
>   null, or undefined in two-byte sequences, only the one-byte variants
>   of these are well-formed.)

I here suggest the text say:

+   formed.  (This implies that an encoder cannot encode false, true,
+   null, floats, undefined-23, reserved-[28..31], or break in two-byte
+   sequences, only the one-byte variants of these are well-formed.)

While it's too late to change, was there a reason "True" didn't get 0b111_00001?
Clearly False, and Null would then compete to be 0b111_00000, and maybe
that's reason enough to not play such games.

Section 3.4 says:

}   Their primary purpose in this specification is to define common data
}   types such as dates.  A secondary purpose is to provide conversion
}   hints when it is foreseen that the CBOR data item needs to be
}   translated into a different format, requiring hints about the content
}   of items.

I don't think that the "primary purpose" is still just dates.
The note about "hints" suggests that tags are always advisory, and I think
that this thread has established that for some protocols, they really are
not.

}   Understanding the semantics of tags is optional for a
}   decoder; it can simply present both the tag number and the tag
}   content to the application, without interpreting the additional
}   semantics of the tag.

I wonder if this text should be stronger.
Maybe:

+   Understanding the semantics of every tag is optional for a decoder;
+   a decoder MAY simply present some or all tags to the
+   application, without interpreting the additional
+   semantics of the tag.

I would then go on:
+   Decoders which translate CBOR values into language specific objects,
+   (e.g., dates, bignum, example3, ...) MAY consume the tags along with
+   the values, returning only the language defined object.

I note that AFAIK, we do not use tag#24 (Encoded CBOR data item) for the
signed object, in COSE.  Should we?
What's the difference between #24 and #55799.
I guess I will read onwards to find out... Got it.

BTW: Tag 25 and 29 are called out after Table 5, but are not listed *in* table 5.
That whole paragraph could use some more periods, and maybe a blank line.
I'm still loss as to why <untagged><null> is better than <epoch><null>.

Why can't we use decimal fractions, or bigfloats for time?
I suppose float64 has enough precision for a millenia or so, even if one
wants microseconds precision.

I think that words "bytewise lexicographic order" used in 4.2.1 may not
survive translations in a meaningful way.  The eight item example might be
clearer if presented in a table so that the bytes can be lined up.
If keys of a map have tags, I assume that the tags are to be included in the
lexicographically order?  Maybe 4.2.3 could cover both ways with a forward reference?

I think that 4.2.2 gets into whether or not a tag is required, and I think
that it might need to be considered more in context of when tags can be
skipped.

I think that the the Introduction should have a section 1.3 that addresses
the concept of "Protocols" on top of CBOR, referencing section 5.
I think that 4.2.2 should forward reference to 5, or maybe sections 4 and 5
I suggest "protocol" be capitalized consistently as Protocol when it is used
in this way.

I don't find that section 5.2 fits into section 5.
I think we already covered this concept.

   "0x62c0ae" does not contain valid UTF-8 and so is not a valid CBOR
   item.  ......
   Generic encoders and decoders are
   expected to forward simple values and tags even if their specific
   codepoints are not registered at the time the encoder/decoder is
   written (Section 5.4).

   Generic decoders provide ways to present well-formed CBOR values,
   both valid and invalid, to an application.  The diagnostic notation
   (Section 8) may be used to present well-formed CBOR values to humans.

I don't personally know enough UTF-8 to know why the above is invalid UTF-8.
Maybe saying that it's not because c0ae is an unsigned code point by because...
(I remember the valid MIME and valid UTF-8 debate we had last time, and I am
not trying to re-open it.)
I think that the second paragraph above should be swapped with the first one.

   1.  Replace the problematic item with an error marker and continue
       with the next item, or

-> this might be a place where that desired "invalid tag" from last week's
discussion fits in!!!

Having read through section 5, I believe even more than two weeks ago, that
the "65535" tag should go into RFC7049bis, not a new document.

5.3.1 "Duplicate keys in a map" seems to suggest that "Stream Encoder" will
be specified/discussed in section 5.6, when in fact that section is about
keys, including duplicate keys.  Maybe 5.3.1/Basic Validity could go later,
or just not be said at all, since the entire section is about this topic?

I think that section 7.1 has a lot of aspirational language ("an attempt
should..."), which might have been appropriate for the ID that let to 7049,
but SHOULD now be definitive.  "An attempt was made to make..."

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 16 May 2020 01:27:50 +0200

Hi Michael,

thank you for taking the time to dive deep!

I’m going to send multiple versions of this reply, with more and more answers filled in; I hope this is not too jarring (you can simply ignore the initial versions if it is).  This is version 1 of my response.

> This document says it is going to Internet Standard, but *only* in the Contributing section.

Fellow https://datatracker.ietf.org/doc/draft-ietf-cose-rfc8152bis-struct/?include_text=1
does not say it at all…
(Well, datatracker knows it, but the document is silent.)

> might be complaining about things that are just matching 7049.

We want to improve on 7049, so if there is a reason to complain, please do!

> (forward reference to section 7.2?)

Give up (not well-formed), as there is no way to know how big the head with these ai values is.

> Getting this right is how we deal with future-proofing.

We do have extension points that have full compatibility; this potential one just doesn’t.

> could act on the advice of section 7.2.  

It would be a CBOR 1.1 (or 2.0), it will not be easy on existing decoders!

> Section 10, first paragraph implies
> we should say something.
> 

You lost me here.

> key/value pair where this occurs?  I take the first meaning, but I want to be

It says that the map is not well-formed, but of course the whole data item is dubious as it is not clear whether the map has ended or not.  So, again, give up.  Should be clear in Appendix C.

> "keep-alive" on some channel.

We haven’t addressed general padding im CBOR (which would again require a 1.1 or 2.0 maybe), and I would hate to suggest this here as the only padding that CBOR already offers.

> Values".    

24, yes, 25 to 27 no (floating point), and 28 to 30 are reserved.
(31 is the break stop code and not a value at all.)

> Could future Simple Values (such as 0..19) can, have complex

No, the general syntax of heads does apply to the unallocated code points as well.

> values?  

Yes.

> Or does a decoder that encounters undefined values here have to
> fail?

No:

5.2.: Generic encoders and decoders are
   expected to forward simple values and tags even if their specific
   codepoints are not registered at the time the encoder/decoder is
   written (Section 5.4).

> +   sequences, only the one-byte variants of these are well-formed.)

The implication we give doesn’t have to be exhaustive.  The sentence you propose is very confusing because it is at a different level: head syntax, but that is not the issue here.  The text here is about the choice that the general head syntax seems to leave for simple values 0..23, which we are saying it actually doesn’t.

> While it's too late to change, was there a reason "True" didn't get 0b111_00001?

Yes.  An earlier version of CBOR used simple values 0..15 for some form of compression.
So we allocated the other simple values at the top, not at the bottom.

> that's reason enough to not play such games.

Right.

> What's the difference between #24 and #55799.

55799 is a tag that can have any CBOR data item as tag content
24 is a tag that can only be on byte strings.
The byte string then *encodes* another CBOR data item.
(The main use here is to keep the decoder from decoding, to provide easy skip-ability or because we need exact bytes as in COSE.)
As often with tags, there is no need for tag 24 on a byte string when it is clear from context that the byte string contains encoded CBOR; this is the case in COSE.

> Why can't we use decimal fractions, or bigfloats for time?

That may have been a mistake (which is one reason we have tag 1001 now).
The WG has generally taken a dim view on extending the domain (allowable syntax for tag content) for a tag, so we can’t “fix” that — note that for the date tag, we have taken the decision not to reuse one tag for two different tag content syntaxes either.

> wants microseconds precision.

It has 53 bits of precision, so it gives ~ microseconds (~ 20 bits after the decimal point) for 2**33 seconds or 272 years from the epoch or until 2242.  Four microseconds until 3058.

> the "65535" tag should go into RFC7049bis, not a new document.

We didn’t want new stuff in 7049bis, so this is now in draft-bormann-cbor-notable-tags with a mention here; RFC7049bis combines with its registries to give the full semantics of CBOR, so this is OK.

> but SHOULD now be definitive.  "An attempt was made to make..."

Section 7 may need another fine combing, indeed.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Fri, 15 May 2020 21:40:22 -0400

Carsten Bormann <[email-masked]> wrote:
    >>> CBOR format.  In the present version of CBOR, the encoded item is not
    >>> well-formed.
    >>
    >> I think that there is a bug here.  What should a parser written today
    >> do when it encounters these values?  (forward reference to section
    >> 7.2?)

    > Give up (not well-formed), as there is no way to know how big the head

Give up, returning the content so-far, or ??

    >> It seems seeing such a thing means a current decoder has to
    >> abort/fail.  What we write here has a profound implication, I think,
    >> on how easily we could act on the advice of section 7.2.

    > It would be a CBOR 1.1 (or 2.0), it will not be easy on existing

Should we plan a tag for version?

    >> Section 10, first paragraph implies we should say something.
    >>
    >> In general, I think that the details in this introductionary encoding
    >> section are too detailed, particularly for 31.  I think that detail
    >> belongs later on. I got no value (I retained nothing) from having that
    >> level of detail there.
    >>
    >> I wonder if section 3.1, under major type 0 should give clarify that
    >> "0" is encoded as 0b000_00000. (That is no negative 0)
    >>
    >> "A string containing an invalid UTF-8 sequence is well- formed but
    >> invalid."
    >>
    >> I guess that RFC8742 include sequences of 7049bis CBOR sequences.  I
    >> wonder if Updates 8742 is appropriate.

    > You lost me here.

Do we need to Update 8742 too?

    >> 3.2.3: (Note that zero-length chunks, while not particularly useful,
    >> are permitted.)
    >>
    >> they might be useful in non-TCP/IP situations where it is useful to
    >> send a "keep-alive" on some channel.

    > We haven’t addressed general padding im CBOR (which would again require
    > a 1.1 or 2.0 maybe), and I would hate to suggest this here as the only
    > padding that CBOR already offers.

Okay, so maybe forward to Protocol.

    >> Could future Simple Values (such as 0..19) can, have complex structure
    >> the way that values 24->27 do?

    > No, the general syntax of heads does apply to the unallocated code

I think we should state this.  It's a extension point that we can reliably skip.

    >> I note that AFAIK, we do not use tag#24 (Encoded CBOR data item) for
    >> the signed object, in COSE.  Should we?  What's the difference between
    >> #24 and #55799.

    > 55799 is a tag that can have any CBOR data item as tag content 24 is a
    > tag that can only be on byte strings.  The byte string then *encodes*
    > another CBOR data item.  (The main use here is to keep the decoder from
    > decoding, to provide easy skip-ability or because we need exact bytes
    > as in COSE.)  As often with tags, there is no need for tag 24 on a byte
    > string when it is clear from context that the byte string contains
    > encoded CBOR; this is the case in COSE.

Understood.

    >> BTW: Tag 25 and 29 are called out after Table 5, but are not listed
    >> *in* table 5.  That whole paragraph could use some more periods, and
    >> maybe a blank line.  I'm still loss as to why <untagged><null> is
    >> better than <epoch><null>.
    >>

    > That may have been a mistake (which is one reason we have tag 1001
    > now).  The WG has generally taken a dim view on extending the domain
    > (allowable syntax for tag content) for a tag, so we can’t “fix” that —
    > note that for the date tag, we have taken the decision not to reuse one
    > tag for two different tag content syntaxes either.

okay.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 16 May 2020 08:49:25 +0200

Good morning Michael,

a few quick answers to your quick answers to my first quick answers.

> Give up, returning the content so-far, or ??

1.2:
      CBOR decoders by definition only return contents from well-formed
      data items.

So the answer is “return an error”.

> Should we plan a tag for version?

No.

First of all, we have no idea whether the room we left here in the encoding space will ever be exercised.  Yes, it is an attractive nuisance (not exercising it leaves one feel strangely unsatisfied), but people have been happy with the other ways to indicated 128-bit etc. values.

Second, the habit to put versions in interchange specifications generally is misguided.  Unless they are needed to prevent false interoperability.  But false interoperability cannot happen here, as the reserved ai values are easily recognizable as such.

(CBOR is designed to be versionless, by defining well-defined extension points.  Version indications also would take significant space in one major area of application of CBOR: Small data items.)

> Do we need to Update 8742 too?

But nothing is updated, and certainly not in 8742?

Yes.

> I think we should state this.  It's a extension point that we can reliably skip.

Well, if we say how heads are decoded, that should be enough — we don’t have to restate that everywhere they are used.  But maybe I don’t understand where you see the extension point here.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Sat, 16 May 2020 18:27:17 -0400

> Well, if we say how heads are decoded, that should be enough — we don’t
    > have to restate that everywhere they are used.  But maybe I don’t
    > understand where you see the extension point here.

I think that maybe we just need to be a bit more regular in how we describe
things.  I like how the document begins, and how it gets a good idea for how
things work without drowning me in detail.

But, I think that we then need to make sure that we are being authoritative
afterwards.   Yes, I think we need to beat people on the head.  In a long
document, people can't read it in one sitting, and so they use it as
reference.  They read only the part that they are working on at that moment.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 17 May 2020 18:30:19 +0200

> On 2020-05-17, at 00:27, Michael Richardson <[email-masked]> wrote:
> 
> 

Now PR #186: https://github.com/cbor-wg/CBORbis/pull/186
Back to your other comments...

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 20 May 2020 15:30:52 +0200

On 2020-05-07, at 19:57, Laurence Lundblade <[email-masked]> wrote:
> 
> However, there is no way to generically know if a tag is a hint,

CoAP options carry a bit that tells the receiver whether processing (and thus understanding) the option is Critical or Elective.  We didn’t do the same for CBOR tags.  But, as you say, most tag usage is not just for a hint, so it is right to emphasize the handling as an “unknown tag”.

I searched for “ignore” in the current draft.
There is some ignoring of unknown tags in the proposed CBOR-to-JSON conversion.
I think this is acceptable.

(There is also some text about ignoring encoding hints in diagnostic notation, unrelated to tags.)

For the occurrence in 7.1 (which I should have caught when handling Jeffrey’s pull request), I propose a slight rewording to stay closer to what we are trying to say.

https://github.com/cbor-wg/CBORbis/pull/187

Note that we also have »Generic encoders and decoders are expected to forward
simple values and tags even if their specific codepoints are not
registered at the time the encoder/decoder is written
({{validity-checking}}).« (Section 5.2) — we could add some exhortations about their API here for the case that they don’t, or we could leave that as is.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 20 May 2020 16:49:49 +0200

Hi Michael,

This is the second version of this reply.
Items that I think we have handled are removed.

>  interchange format from RFC 7049.  It does not create a new version

I don’t think we want to go to that level of detail right there in the intro.
I’m also not sure 7049 was formally defining “compliance” of decoders.

I think we could move some of this thinking into the “changes from 7049” appendix, the fate of which we haven’t really decided as a WG yet.

> This seems to have something to do with tags.

Yes.
Implementation of specific tags was always optional with CBOR; there is no single “required tag”.

> to how unknown tags are treated.

I’d say the implementation of the specific data model has to be configured for the library...

> I think that I'm arguing for generic parsers to be in RFC7049 mode,

(I don’t think that is a thing.)

> which

Generic decoders that want to evolve to be usable with applications that need tag support will need to develop a transition strategy.  Isn’t that obvious to any library developer?

> if it did not have an interface that passed tags.

That generic decoder would not be useful for an application that needs to be able to handle both integers and time tags in the same position.  That doesn’t change.

[Skipping the 28, 29, 30 part, because I think #186 addressed it.]

> on. I got no value (I retained nothing) from having that level of detail there.

I think there was another proposal to move around elements of Section 3.
Sometimes it is necessary to include some detail in an overview for completeness; we can’t really pretend ai=31 does not exist for a few sections and then do a surprise reveal, can we?

> is encoded as 0b000_00000. (That is no negative 0)

Is this really more about major type 1, and the value -1?

> I think that this might need clarification.

Not for a pedantic like me who has already memorized the content of Section 1.2…

> I wonder if Updates 8742 is appropriate.

You still lost me here.
What are 7049 CBOR sequences?

Oh.  We are talking about “Data Streams”.  This probably should mention RFC 8742 (which only happens in 3.1)!

> "keep-alive" on some channel.

We haven’t addressed general padding in CBOR (which would again require a 1.1 or 2.0 maybe), and I would hate to suggest this here as the only padding that CBOR already offers.

Yes.

No:

Right.

> I don't think that the "primary purpose" is still just dates.

Dates were an example here (and we might soon have a data tag :-).

> The note about "hints" suggests that tags are always advisory,

In the second case — does it really suggest that for the data types?

> and I think

each, actually (I prefer not to spell that out).

> +   application, without interpreting the additional

I don’t like interoperability keywords in such places.

> survive translations in a meaningful way.

Deepl turns this into "byteweise lexikographische Ordnung”, na ja.
The next alternative "byteweise lexikographische Reihenfolge” is very close.
No idea about "ordre lexicographique par octet”.
Or "пошаговый лексикографический порядок”, for that matter.
But "字节词序” looks really good :-) (and is completely wrong).

>  The eight item example might be

Well, 4.2.3 is the old (7049) ordering.

> that it might need to be considered more in context of when tags can be

But it is about deterministic encoding, not about that other discussion…

> I think we already covered this concept.

This section reaffirms to concept developed above.  It also discusses the application interface — have we really covered that here?

> I don't personally know enough UTF-8 to know why the above is invalid UTF-8.

Because it uses a long form where there is a shorter form - UTF-8 uses deterministic (shortest) encoding.

> Maybe saying that it's not because c0ae is an unsigned code point by because...

This is a mild reminder that you have to read up on UTF-8 to understand CBOR.

> I think that the second paragraph above should be swapped with the first one.

Well, that would expose that the paragraphs are partially redundant …

Now https://github.com/cbor-wg/CBORbis/issues/189

> -> this might be a place where that desired "invalid tag" from last week's

Well, hmm.  The tag is invalid in interchange to allow the implementation to do with it what it wants, so, yes, you are right, but that wasn’t the intention for its use.

> or just not be said at all, since the entire section is about this topic?

It sure needs to be said (and the term introduced).
The section 5.6 reference needs to be pulled out of the last sentence.

Now https://github.com/cbor-wg/CBORbis/pull/190

> but SHOULD now be definitive.  "An attempt was made to make..."

Section 7 may need another fine combing, indeed (and now has some comb marks in #187).

Grüße, Carsten

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 21 May 2020 10:45:07 -0700

> Not for a pedantic like me who has already memorized the content of

Quick note that I've added a citation to section 1.2 in
https://github.com/cbor-wg/CBORbis/pull/191, which Paul rapidly merged. :)

Jeffrey

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 21 May 2020 22:24:41 +0200

On 2020-05-21, at 19:45, Jeffrey Yasskin <[email-masked]> wrote:
> 
> Quick note that I've added a citation to section 1.2 in https://github.com/cbor-wg/CBORbis/pull/191, which Paul rapidly merged. :)

Thank you!

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Thu, 21 May 2020 19:10:44 -0400

Carsten Bormann <[email-masked]> wrote:
    >> interpretations of [RFC7049] it may in some cases create situations
    >> where an existing parser may no longer comply to this specification.
    >> interchange format from RFC 7049.  It does not create a new version of
    >> the format.

    > I don’t think we want to go to that level of detail right there in the
    > intro.  I’m also not sure 7049 was formally defining “compliance” of
    > decoders.

    > I think we could move some of this thinking into the “changes from
    > 7049” appendix, the fate of which we haven’t really decided as a WG
    > yet.

okay, I can go for that.

    > Yes.  Implementation of specific tags was always optional with CBOR;
    > there is no single “required tag”.

yes, okay, but some Protocols might require them.
So maybe we need some additional BCP14-like terms. Hmm.

    >> which might mean ignoring unknown tags (if that's what they did
    >> before), passing the data up using whatever native interpretation
    >> there is, until they are configured otherwise.

    > Generic decoders that want to evolve to be usable with applications
    > that need tag support will need to develop a transition strategy.
    > Isn’t that obvious to any library developer?

Yes, but sometimes application developers need a hammer.
This happens far less often in open source situations, but in other
situations, the hammer is sometimes necessary to make a supplier spend the
effort.

    > I think there was another proposal to move around elements of Section
    > 3.  Sometimes it is necessary to include some detail in an overview for
    > completeness; we can’t really pretend ai=31 does not exist for a few
    > sections and then do a surprise reveal, can we?

We can just say less in section 3.  ai=31 "Stop-Code, ses section FOO"
That's okay.  It's some of the other detail that seems to accumulated there
which distracted me.

    > Is this really more about major type 1, and the value -1?

maybe.

    > You still lost me here.  What are 7049 CBOR sequences?

Yes.

    >> key/value pair where this occurs?  I take the first meaning, but I
    >> want to be sure.

    > It says that the map is not well-formed, but of course the whole data
    > item is dubious as it is not clear whether the map has ended or not.
    > So, again, give up.  Should be clear in Appendix C.

I am concerned that different kinds of parsers wind up resulting in an error
at a different point, possibly causing some content to be examined, while
other content is not.   This is akin to the duplicate key challenge.

    > No, the general syntax of heads does apply to the unallocated code

I'd like to say this.

    >> Or to put it another way, can a decoder depend upon unassigned simple
    >> values having the one-or-two byte structure presented and be able to
    >> skip unknown values?

okay.

    >> I think that words "bytewise lexicographic order" used in 4.2.1 may
    >> not survive translations in a meaningful way.

    > Deepl turns this into "byteweise lexikographische Ordnung”, na ja.  The
    > next alternative "byteweise lexikographische Reihenfolge” is very
    > close.  No idea about "ordre lexicographique par octet”.  Or "пошаговый
    > лексикографический порядок”, for that matter.  But "字节词序” looks
    > really good :-) (and is completely wrong).

I'm less concerned about translations of the draft by mechanical, but
translations by human who didn't understand the original mathematical
meaning.
Whether that person is a translator, or a non-native english/germanic/latin
speaker writing code.  I'm just asking if you could use a less technical
term.  HIT ME ON THE HEAD.

    >> I think that the the Introduction should have a section 1.3 that
    >> addresses the concept of "Protocols" on top of CBOR, referencing
    >> section 5.  I think that 4.2.2 should forward reference to 5, or maybe
    >> sections 4 and 5 I suggest "protocol" be capitalized consistently as
    >> Protocol when it is used in this way.
    >>
    >> I don't find that section 5.2 fits into section 5.  I think we already
    >> covered this concept.

    > This section reaffirms to concept developed above.  It also discusses
    > the application interface — have we really covered that here?

I don't understand your comments here.

    >> item.  ......  Generic encoders and decoders are expected to forward
    >> registered at the time the encoder/decoder is written (Section 5.4).
    >>
    >> Generic decoders provide ways to present well-formed CBOR values, both
    >> valid and invalid, to an application.  The diagnostic notation

    > deterministic (shortest) encoding.

Can you please say that in the document.

} For instance, "0x62c0ae" uses a long form where there is a shorter form, and
} UTF-8 mandates deterministic (shortest) encoding, therefore, this not a valid
} CBOR item.

    > This is a mild reminder that you have to read up on UTF-8 to understand

Yes, but many will just plop the text into a UTF-8 parser and hope.
We can't all be experts at everything :-)

    >> Having read through section 5, I believe even more than two weeks ago,
    >> that the "65535" tag should go into RFC7049bis, not a new document.

    > We didn’t want new stuff in 7049bis, so this is now in
    > draft-bormann-cbor-notable-tags with a mention here; RFC7049bis
    > combines with its registries to give the full semantics of CBOR, so
    > this is OK.

I feel that this is a reasonable thing to address short-cuming in existng PS,
but I won't fall on this sword.

    > Grüße, Carsten

Thank you for all the work.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Laurence Lundblade <[email-masked]>
Date: Fri, 22 May 2020 12:22:30 -0700

> On May 21, 2020, at 4:10 PM, Michael Richardson <[email-masked]> wrote:
> 

I think the main time type should have solely been an integer. You can express +/- 500 billion years to 1 second accuracy with 64 bits. The earth is only 4.5 billions years old so it is even good enough for most science. It’s only really odd scientific use cases that would need anything else as far as I can imagine.

I would disallow float dates in the main time type. Floats only enable microsecond accuracy. They give no greater range that is of much use. Also, they have less precision and the precision varies by year.

This is about code size and HW support. You don’t want to have to implement decoding of a variety of time formats for 0.0001% of the use cases. It is also a pain for every protocol to say “use the time tag, but disallow floats”. The CDDL type for time has this problem. Maybe should probably have a new CDDL type “simple_time” or that is tag 1, but disallows float.

Then I’d create a second Swiss army knife time format that can use floats, bignums, decimal fractions and so on. I’ve been playing with implementations of conversions of these types and so far it takes 500-1000 bytes of code to convert all of them in some form or other.

I’m kind of in favor of defining yet another time tag that just allows integers, but the current epoch time tag has been around for a long time now so it seems ugly to do so.

In reality, probably everyone will just implement 64-bit only time as sort of an unstated best practice and it will be OK.

LL

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 23 May 2020 03:31:43 +0200

> On 2020-05-22, at 21:22, Laurence Lundblade <[email-masked]> wrote:
> 
> I think the main time type should have solely been an integer.

With hindsight, I almost agree.

> You can express +/- 500 billion years to 1 second accuracy with 64 bits. The earth is only 4.5 billions years old so it is even good enough for most science. It’s only really odd scientific use cases that would need anything else as far as I can imagine.

There is range (of which an integer has plenty for seconds-based time), and there is precision.
Unfortunately, precision isn’t that great with floating point(*) either, which led to the alternative time tag 1001.

> I would disallow float dates in the main time type. Floats only enable microsecond accuracy. They give no greater range that is of much use. Also, they have less precision and the precision varies…

Your CBOR-based protocol is welcome to do that.

> This is about code size and HW support. You don’t want to have to implement decoding of a variety of time formats for 0.0001% of the use cases. It is also a pain for every protocol to say “use the…

Your CDDL definition of the data interchanged by your protocol could usefully say:

itime = #6.1(int)

For many applications, I’d actually go so far as

uitime = #6.1(uint)

(This is most useful for all protocols that describe point in time that lie after the creation of the protocols, such as claims issuance dates etc.)

I think that the notable tags document is a good place to make this discussion accessible as well as CDDL fragments like the above that can be copied (so people don’t invent thousands of names for the same thing).

> Then I’d create a second Swiss army knife time format that can use floats, bignums, decimal fractions and so on. I’ve been playing with implementations of conversions of these types and so far it t…

That’s tag 1001.  But, again, your protocol can restrict that tag to the times you actually need.

> I’m kind of in favor of defining yet another time tag that just allows integers, but the current epoch time tag has been around for a long time now so it seems ugly to do so.

And it is not needed: The tag content is plainly visible, so it is easy to find out if someone sent you a floating point time (which you may not want) or an integer time; as seen above, it is also easy to write the CDDL for restricting this to what you want.

> In reality, probably everyone will just implement 64-bit only time as sort of an unstated best practice and it will be OK.

Hah.  Since (unsigned) 32-bit time takes us to 2106-02-07 06:28:15 UTC, I’m sure lots of people will go for that in their constrained implementations.  Make sure you tell your grandkids not to fly (or otherwise rely on product safety) on that date and time :-)
But note that 32-bit seconds also are already not useful any more for taking out 99-year leases on your neighbor’s garage.

TL;DR: It probably wasn’t too bright to define tag 1 as #6.1(int/float) in RFC 7049, but that’s where we are, and we know how to mitigate any ugliness.

Grüße, Carsten

(Insert discussion of epochs, eras, GPS 1024 week rollovers, and everything else.)

(*) We could have embraced 128-bit numbers (including floating point) earlier.  Seemed preposterous for CBOR in 2013.  But they have entered the picture in RFC 8746 (tags 83 and 87), and there are even CPU architectures for 128-bit processors now.

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 26 May 2020 14:04:11 -0700

> On May 15, 2020, at 6:40 PM, Michael Richardson <[email-masked]> wrote:
> 

My answer on the difference is that you use 55799 when the surrounding data / file / protocol is not CBOR and 24 when it is CBOR. 55799 is intended to work as a magic number, 24 is not because it is not unique enough.

From a decoder point of view, they should be handled exactly the same and should be able to occur legally in exactly the same contexts. Right? Probably this should be stated explicitly. Right now there is no linkage between them in the text.

In addition to skip-ability it seems they could be used when some data item might be JSON or might be CBOR or might be something else and you want to explicit say what it is. Should there be a JSON tag?

This is kind of twisted, but seems legal.
  Encoding: 
    - start with an RFC 3339 date string
    - base 64 encode it and tag it as so with tag 34
    - tag it with tag 24 (55799 ) or to say it is CBOR
    - tag it with tag 0 to say it is a date string
    - tag it with 22 to say it should be b64’d if re-encoded later
 Decoding:
    - remove the base 64 encoding because of the tag 34
    - feed it back to the CBOR encoder because of tag 24
    - interpret it as an RFC 3339 date because of tag 0

Base64 encoding / decoding is not that much code or that difficult, so a generic decoder might actually do this.

LL

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 May 2020 23:43:25 +0200

> On 2020-05-26, at 23:04, Laurence Lundblade <[email-masked]> wrote:
> 
> 
> From a decoder point of view, they should be handled exactly the same

Actually, no.

55799 has any CBOR data item as tag content and essentially has the semantics of that CBOR data item.

24 has a byte string as tag content.  That byte string is identified by the tag as encoded CBOR. 
A byte string with embedded encoded CBOR is a data item, but it is different from the data item that was encoded and embedded. Decoders will differ in their tag 24 handling: they might go ahead and decode that CBOR or just hand the byte string and tag to the application.  In either case, the decoded CBOR is not simply “in place”, making the tag and the byte string vanish.

> and should be able to occur legally in exactly the same contexts. Right? Probably this should be stated explicitly. Right now there is no linkage between them in the text.
> 

Very likely.  That would probably take a UTF-8 string as tag content.
(Tag 262 works for JSON-encoded content like tag 24 does for CBOR-encoded content, requiring a byte string with that content though.)

>     - base 64 encode it and tag it as so with tag 34

Why would one want to do that?

>     - tag it with tag 24 (55799 ) or to say it is CBOR

Ignoring the 55799 case, you are missing one encoding step (tag 24 requires a byte string with encoded CBOR, not a tagged text string).

>     - tag it with tag 0 to say it is a date string

Not allowed.  This only takes major type 3, no tags.

>     - tag it with 22 to say it should be b64’d if re-encoded later

Well, not “it”, but any byte string in the CBOR data item.
So, here it would base-64 encode for JSON conversion the byte string in tag 24 (the tag 24 itself would presumably be stripped in any JSON conversion, but I’m already confused — you can do “JSON in JSON” as well, just not tag it).

>     - remove the base 64 encoding because of the tag 34

That is inside, so you see it last.

>     - interpret it as an RFC 3339 date because of tag 0

Lost track.

> Base64 encoding / decoding is not that much code or that difficult, so a generic decoder might actually do this.

I wouldn’t touch that with a ten-foot pole — it might convert that pole into base64 without asking me.

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Fri, 29 May 2020 12:51:41 -0700

Really appreciate the discussion here (despite my incorrect statements).

> On May 26, 2020, at 2:43 PM, Carsten Bormann <[email-masked]> wrote:
> 

This is well-formed and valid CBOR by RFC 7049 & CBORbis:

D9 D9F7          # tag(55799)
   81            # array(1)
      D9 D9F7    # tag(55799)
         D9 D9F7 # tag(55799)
            01   # unsigned(1)

One is tempted to say that 55799, as a true CBOR no-op, must always be ignored by generic decoders. However, I think most implementations of CBOR protocols (e.g., CWT, COSE) today would not tolerate 55799’s showing up anywhere in the encoded CBOR like in my example above because ignoring it universally has never been the rule.

So you really need to mean it when you put tag 55799 in, the same as any other tag. Its use needs to be mandated as part of the protocol specification so implementors know it will be occurring and are ready to ignore it.

Probably it is not even CBOR protocol specifications that says to use 55799. It is the protocol or file format that carries the encoded CBOR that would indicate it is in use.

Probably 55799 should never occur except at the beginning of some encoded CBOR.

> A byte string with embedded encoded CBOR is a data item, but it is different from the data item that was encoded and embedded. Decoders will differ in their tag 24 handling: they might go ahead and…

Yes of course (I should have known that).

A generic decoder could have a mode that skips the tag24-plus-byte-string the same it skips 55799, but it shouldn’t be the default because protocols usually assign some semantics to it such as having it hashed or distinguishing it from JSON (protocols should never assign such semantics to 55799).

>>    - tag it with 22 to say it should be b64’d if re-encoded later

I’m after a deeper understanding of how base64, tag 24 and 55799 interact or don’t interact. My date string example was poor (sorry).

Base64 is really a content transfer encoding, not a data type.  The original data before base64 encoding is of some type or other. Whom ever processes it after the base64 encoding is removed has to come to know what type it is.

However, CBOR generally treats base64 as a data type itself and as it is has no means for indicating the type of the original data before base 64 encoding was added (by contrast, MIME can describe both type and transfer encoding).

So how can the receiver of base64-tagged data know the type of original data? It pretty much has to be in the definition of the CBOR protocol. It might say it is always an X.509 cert or an elf executable or such. It might say it is looked up by magic number (in which case 55799 might apply). Lots more options...

It is also possible to define tags whose content type can be bstr, base64, base64-url. If it was really common to base64 encode X.509 certs (maybe because of JOSE) and a tag was defined to indicate an X.509 cert, it might be useful to say the X.509 tag content could be bstr, base64 or base64-url.

No suggestion for change and nothing at issue so far, just thinking through how it fits together.

What would really be helpful to me is some detailed example of when you’d use the base64 tags. Why would you ever use them when CBOR carries binary data just fine? Why wouldn’t every CBOR-based protocol just say you always strip the base64 before CBOR encoding? And why wouldn’t every CBOR bstr always be base64 encoded when translating to JSON?

> I wouldn’t touch that with a ten-foot pole — it might convert that pole into base64 without asking me.

I wasn’t suggesting any base64 decoding conversion that wasn’t indicating by a tag in an input stream or any base 64 encoding that wasn’t requested through an explicit API request. Section 3.4.5.3 seems to suggest the decoding is a good thing to do:

3.4.5.3 <https://tools.ietf.org/html/draft-ietf-cbor-7049bis-13#section-3.4.5.3>.  Encoded Text

   Some text strings hold data that have formats widely used on the
   Internet, and sometimes those formats can be validated and presented
   to the application in appropriate form by the decoder.  There are
   tags for some of these formats.  As with tag numbers 21 to 23, if
   these tags are applied to an item other than a text string, they
   apply to all text string data items it contains.

Is there anything wrong with a generic decoder validating and removing bas64 encoding when encountering tag 34?

Also see this issue <https://github.com/cbor-wg/CBORbis/issues/194> I filed against this text and Table 5.

Thanks very much.

LL

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 29 May 2020 23:16:46 +0200

> This is well-formed and valid CBOR by RFC 7049 & CBORbis:

Certainly.  It becomes interesting if you add tags that do have structural requirements:

C1                # tag(1)
   D9 D9F7        # tag(55799)
      1A 5ED17501 # unsigned(1590785281)

is well-formed, but not valid.

It would probably be valid if we had moved from structural to semantic validity for the tags defined in RFC 7049; we decided not to.

> One is tempted to say that 55799, as a true CBOR no-op, must always be ignored by generic decoders.

Tempted, yes, but not supported by 7049 or 7049bis.

> However, I think most implementations of CBOR protocols (e.g., CWT, COSE) today would not tolerate 55799’s showing up anywhere in the encoded CBOR like in my example above because ignoring it universally has never been the rule.

Right.

> So you really need to mean it when you put tag 55799 in, the same as any other tag. Its use needs to be mandated as part of the protocol specification so implementors know it will be occurring and…

You can use it without a protocol specification as well.
But if you have one, that better tell you where 55799 is “expected”.

> Probably it is not even CBOR protocol specifications that says to use 55799. It is the protocol or file format that carries the encoded CBOR that would indicate it is in use.

55799 is a bit like a BOM on UTF-8 (except that the latter is damaged by BOMs because it loses its ASCII compatibility, but let’s ignore that for now).

Definitely not something you want to use unless there is value in determining the format from a “magic number”.

> Probably 55799 should never occur except at the beginning of some encoded CBOR.

Probably.

> A generic decoder could have a mode that skips the tag24-plus-byte-string the same it skips 55799,

The semantics are quite different, though.

> but it shouldn’t be the default because protocols usually assign some semantics to it such as having it hashed or distinguishing it from JSON (protocols should never assign such semantics to 55799).

Yes.

> Base64 is really a content transfer encoding, not a data type.  The original data before base64 encoding is of some type or other. Whom ever processes it after the base64 encoding is removed has to…

We don’t have transfer encodings in CBOR, so we can only model them as types.

The tags 32-36 are really “type tags” — they tell you what’s in there but don’t give that a new meaning.

Why anyone would want to send base64-encoded data in CBOR instead of transferring the byte string (then possibly tagged by 21/22 so it looks the same again after a JSON conversion) is not discussed in RFC 7049.  There may be a need to preserve the actual text string for some reason; base64 is deterministic only if you know all the choices.

> However, CBOR generally treats base64 as a data type itself and as it is has no means for indicating the type of the original data before base 64 encoding was added (by contrast, MIME can describe…

Right, the semantics of a “decoded” 33/34 is that of a byte string.

> So how can the receiver of base64-tagged data know the type of original data? It pretty much has to be in the definition of the CBOR protocol.

Yes, byte string.

If you need a “base 64 encoded MAC-48 address”, you can go ahead and define a tag for that.

> It might say it is always an X.509 cert or an elf executable or such. It might say it is looked up by magic number (in which case 55799 might apply). Lots more options...

The tag, or a data definition (CDDL etc.), are the obvious two approaches.

> It is also possible to define tags whose content type can be bstr, base64, base64-url. If it was really common to base64 encode X.509 certs (maybe because of JOSE) and a tag was defined to indicate…

Yes.  I think we don’t have a tag for X.509 certs at this time; where these occur, they are defined in the protocols (e.g., draft-ietf-cose-x509, which actually defines useful constructions around X.509 certs, which anyway really should be called RFC 5280 certs), ultimately supported by the COSE tags.

> What would really be helpful to me is some detailed example of when you’d use the base64 tags. Why would you ever use them when CBOR carries binary data just fine? Why wouldn’t every CBOR-based pro…

These are all good guidelines; see above for a possible reason, but I don’t know a good one either.

> 3.4.5.3.  Encoded Text

As long as the application knows this is happening — it may no longer be able to validate what it got on the structural level, though.

> Also see this issue I filed against this text and Table 5.

(Where “issue” was a link to https://github.com/cbor-wg/CBORbis/issues/194 .)
Yes, I think we do have a genuine bug there.  The specific text is from pull request #18, with the change dated 2018-01-29.  Thinking about a good resolution now.

Grüße, Carsten