Subject: [Cbor] Tagging requirement

From: Laurence Lundblade <[email-masked]>
Date: Sun, 16 Aug 2020 22:09:42 -0700

My understanding is that the surrounding protocol that uses a tagged type dictates whether the tag should be present or not. For example, CWT dictates that tag 1 (epoch time) must not be used for the “exp”, “nbf” and “iat" claims.

So the surrounding protocol can pick one of these three when it use a tagged type.
 - Tag is FORBIDDEN (e.g., date fields in CWT). Decode error if tag is present.
 - Tag is REQUIRED tags (e.g., COSE type tag when CWT tag is present). Error occurs if tag is absent, probably because item is not of the expected type.
 - Tag is OPTIONAL (e.g., application/cwt and the CWT tag). Maybe an error, maybe not. 

As far as I can see there isn’t a clear rule about which of these it is when the surrounding protocol doesn’t explicitly pick. Deterministic encoding does disallow OPTIONAL. If I were to guess as to which is the default, I would say it is OPTIONAL.

Big floats and decimal fractions make use of the bignum type, but the definition of big floats and decimal fractions doesn’t explicitly say which of the three above it selects, so one might think it is OPTIONAL if one assumes that is the default of the three and that you could distinguish a bignum mantissa because it is a byte string not an integer. One second though, you realize you can’t tell the sign of the bignum unless there is a tag so it has to be REQUIRED to work.

My thought is that OPTIONAL adds no value.  It is a form of “be liberal in what you receive” that seems to be less in favor these days. When you nest tagged type usage with OPTIONAL, you can get a larger fan out.  However, OPTIONAL is allowed in 7049 and things like the application/cwt media type registry use it.

My thought is that it would be helpful to say somewhere that users of tagged types should explicitly say what the tagging requirement is and should avoid OPTIONAL tagging.

(I came to this trying to write code for tag handling for big float, EAT, CWT, COSE and such. I’m trying not to be annoying, but also think this is an issue :-) )

LL

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 17 Aug 2020 09:40:54 +0200

> My understanding is that the surrounding protocol that uses a tagged type dictates whether the tag should be present or not. For example, CWT dictates that tag 1 (epoch time) must not be used for t…

Yes.  More specifically, the CWT protocol does not use Tag 1 for exp, nbf, iat, but instead borrows that Tag’s specification for its internal tag content.

> So the surrounding protocol can pick one of these three when it use a tagged type.

Yes.  The application protocol needs to define the application protocol.

> - Tag is FORBIDDEN (e.g., date fields in CWT). Decode error if tag is present.

That is not what the protocol says — the protocol says it’s a number.  The Tag is FORBIDDEN as much as an array of four byte strings would be FORBIDDEN.

(I’m going with your upper case language here; not sure that adds anything because it makes it seem these words are a menu to choose from.  They aren’t.  FORBIDDEN really means “does not use the Tag”.)

> - Tag is REQUIRED tags (e.g., COSE type tag when CWT tag is present). Error occurs if tag is absent, probably because item is not of the expected type.

This case is similar: The protocol says what it is.  The Tag may not be “REQUIRED" (there may be other choices for that position), but to get the semantics of the Tag, you have to provide the Tag.  (I.e., you only get the Tag if you do use the Tag.)

> - Tag is OPTIONAL (e.g., application/cwt and the CWT tag). Maybe an error, maybe not. 

That is indeed a special case that does not make a lot of sense *within* an application protocol, but is useful at its outer boundary: If you have context (such as a media type), you can use the unadorned data structure for this protocol, otherwise you use the tag.

> As far as I can see there isn’t a clear rule about which of these it is when the surrounding protocol doesn’t explicitly pick.

There is no rule about arrays of four byte strings, either.
The application protocol has to define the application protocol.

> Deterministic encoding does disallow OPTIONAL.

It says that this decision must be deterministic.  There is nothing wrong about OPTIONAL if all senders arrive at the same decision.

> If I were to guess as to which is the default, I would say it is OPTIONAL.

I have no idea what a “default" is here.  When would you get to exercise this default?  With all the language that argues against making Tags optional in an application protocol, why or how would this be the default for anything?

> Big floats and decimal fractions make use of the bignum type, but the definition of big floats and decimal fractions doesn’t explicitly say which of the three above it selects,

No, because Section 3.4.4 defines a couple of Tags.  So for the definition of the Tag, the Tag is “REQUIRED”, because you only have a tag if you have the tag.  Whether other protocols borrow the specification of the tag content (array of e, m) is not relevant for the definition of the Tag.

> so one might think it is OPTIONAL if one assumes that is the default of the three and that you could distinguish a bignum mantissa because it is a byte string not an integer.

Ah, that’s the confusion.

RFC 7049 predates RFC 8610, so we are not formally describing the tag content in RFC 7049.  RFC 7049 is clear that m and e are integers, but does not say which kind of integer (that may be a remnant of semantic validity).  Nowhere it says that if the mantissa is something else (like an array of four byte strings), try to interpret this by guessing tags that could be used to convert the mantissa into an integer.

RFC 8610 actually does the work of fully defining the structure of decfracs and bigfloats:

                  decfrac = #6.4([e10: int, m: integer])
                  bigfloat = #6.5([e2: int, m: integer])

(integer is defined here:

                  integer = int / bigint
                  bigint = biguint / bignint
                  biguint = #6.2(bstr)
                  bignint = #6.3(bstr)
                  int = uint / nint
                  uint = #0
                  nint = #1
)

> One second though, you realize you can’t tell the sign of the bignum unless there is a tag so it has to be REQUIRED to work.

Yes.
(More specifically, there are no Tag 2/3 bignums without tags, just as there are no Tag 1 datetimes without Tag 1.  In both cases, an application protocol could borrow the definition of the structure of the tag content — I could define a protocol that accepts a byte string in a specific position that is then interpreted as an integer by the application like it would be if it carried Tag 2.  Note that this borrowing is common enough that the CDDL operator ~ (unwrap) directly supports this, but it is entirely distinct from using the Tag.)

> My thought is that OPTIONAL adds no value.  It is a form of “be liberal in what you receive” that seems to be less in favor these days. When you nest tagged type usage with OPTIONAL, you can get a larger fan out.  

Don’t do that then [1].

> However, OPTIONAL is allowed in 7049 and things like the application/cwt media type registry use it.

What do you mean by “is allowed”?
The application protocol can indeed define (or use) a Tag that can be used to identify a data item if the identification does not come from the context.
If it doe,s the tag doesn’t add value.  Application/cwt could have been defined such that it never includes the CWT Tag (i.e., the COSE tag is outermost); I don’t have a strong opinion on whether it should have done that.

None of this is something that could be defined for Tags in general; the application protocol (here: RFC 8392) needs to make the call.  The media type application/cwt now includes the unneeded choice; an application might still make further restrictions (always tag 61, never tag 61).

> My thought is that it would be helpful to say somewhere that users of tagged types should explicitly say what the tagging requirement is and should avoid OPTIONAL tagging.

There is nothing wrong with what you call “OPTIONAL”.  You wouldn’t use it on the inside of an application protocol.  You also would specific whether you want a Tag or not in all cases where the application protocol is embedded into another application protocol.

Tag 55799 is the prototype of all these “signature tags” — you only use it if you don’t have other context to detect that the byte sequence you have in hand is an encoded CBOR data item.
An application protocol does not allow or specify this tag unless it allows or specifies it.

Indeed, maybe we should say so in notable-tags.
I’ll draft some text.

> (I came to this trying to write code for tag handling for big float, EAT, CWT, COSE and such. I’m trying not to be annoying, but also think this is an issue :-) )

It becomes an issue if you somehow associate each Tag with your three labels FORBIDDEN, REQUIRED, OPTIONAL.  Don’t to that then...

Grüße, Carsten

[1]: http://www.catb.org/~esr/jargon/html/D/Don-t-do-that-then-.html

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 17 Aug 2020 10:56:21 -0700

I have been thinking of the standardized tag definitions as data type definitions first and the assignment and use of a tag to indicate the type as second.

So a protocol designer decides to use a type, say an epoch date, a CWT or such first. Then, second, the designer decides how to identify it as that type in the protocol. The second step is the choice between FORBIDDEN, REQUIRED or OPTIONAL. The “default" I’m talking about is when the protocol designer says the type is in use, but doesn’t say how its type is indicated.

8392 more or less proceeds like this in the definition of NumericDate. First it references 2.4.1 of 7049. Second, it says the tag MUST be omitted (FORBIDDEN).

You used the term “borrow” below to say what CWT does for NumericDate.  We could describe it like this:
   BORROW means use the data type without the type 6 tag
   USE means a type 6 tag and the content as defined

Then USE maps to my REQUIRED and BORROW maps to my FORBIDDEN and OPTIONAL is when the protocol permits either BORROW or USE.

BORROW and USE seem like an OK way to characterize too, but...

At this point the word “tag” is confusing to me.  When you say “use Tag 1” you seem to mean encode a type 6 integer with value 1 followed by the content which is a number. Unless explicitly allowed, the type 6 integer and number content are inseparable. When you say USE tag 1, it means both the type 6 integer and the number content. By contrast, when CWT refers to the “CWT tag” it is just referring to the type 6 integer.

Maybe this is it?  CWT is defined first as a *protocol* not a tag. It is not necessarily part of a CBOR structure. CWT then it says it can be tagged as one means to identify it. You never need to BORROW the definition of it. By contrast bignum and such are defined as tags first so you have to explicitly BORROW if you want to use them without the type 6 indicating integer. Bignums and friends are assume to be part of a CBOR structure.

With this distinction made manifest, it starts to make sense.

Unless explicitly stated bignums, big floats and such always have the type 6 integer because they are defined in terms of it.

Protocols like CWT each state how they wish to be identified and tagging may be one way.

You have to know the nature of a particular tag definition to know whether the type 6 integer is required or not.

Section 6 of 8392 implies some mutual exclusion between the user of application/cwt and #6.61 to identify a CWT, but it is only implied. In another thread Jim thought it was OK to have both application/cwt and #6.61 on the same CWT even though they are redundant.

A few more comments below.

> On Aug 17, 2020, at 12:40 AM, Carsten Bormann <[email-masked]> wrote:
> 

But the definition of big floats and decimal fractions should stand on their own without 8610.

>> My thought is that OPTIONAL adds no value.  It is a form of “be liberal in what you receive” that seems to be less in favor these days. When you nest tagged type usage with OPTIONAL, you can get a…

The point is to help other not do that.

> It becomes an issue if you somehow associate each Tag with your three labels FORBIDDEN, REQUIRED, OPTIONAL.  Don’t to that then...

The association is not with a tag, it is with an item in a protocol. The APIs are to decode a protocol item as a bignum, an epoch date, a CWT, a COSE and such and the API wants to know if the tag must be present, must be absent or can be either.

LL

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 17 Aug 2020 20:23:01 +0200

On 2020-08-17, at 19:56, Laurence Lundblade <[email-masked]> wrote:
> 
> At this point the word “tag” is confusing to me.  When you say “use Tag 1” you seem to mean encode a type 6 integer

(The argument is not used as an integer, but as a tag number.)

> with value 1 followed by the content which is a number.

Yes.

> Unless explicitly allowed, the type 6 integer and number content are inseparable. 

As long as you are talking about Tags, yes: Tags are a major-type 6 container.
You can also borrow the definition of a Tag content data structure for an application protocol, but then you aren’t using a Tag, you are using the definition of a Tag content data structure (or “an unwrapped tag”, if you must say so).

> When you say USE tag 1, it means both the type 6 integer and the number content. By contrast, when CWT refers to the “CWT tag” it is just referring to the type 6 integer.

We recently revisited that terminology in 7049bis; this is a bit similar (but inverse) to what are tags and elements in XML — what are called elements in XML (except by the 95 % who don’t know the difference between a tag and an element) are tags in CBOR, and what are tags in XML are not really discussed very much (they are briefly referred to as “heads”).

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 17 Aug 2020 13:37:47 -0700

> On Aug 17, 2020, at 11:23 AM, Carsten Bormann <[email-masked]> wrote:
> 

What do you think about defining CBOR protocol items that can be either tag-contained or unwrapped?  In CDDL, “dualistic_bigfloat = bigfloat / ~bigfloat”. 

Making a distinction in the way CWT is defined and named from the way epoch date, bignum and such are defined is what clears it up for me.

The definition of the CBOR structure for CWT is independent of the CWT tag. The CBOR structure for the CWT is defined first. Then the tag definition comes a long and says is contents is a CWT.

For tags epoch date, bignum and such there is no independent definition of the contents. That content definite is wrapped up with the tag definition. If you want to use the content without a tag-container you have to unwrap it.

That a tag is a “container” is really helpful to me.

Thanks for the discussion on this!

LL

---

From: Michael Richardson <[email-masked]>
Date: Tue, 18 Aug 2020 15:03:59 -0400

Carsten, some year ago we had a lot of discussion when doing 7049bis about
what would be described in 7049bis, and what belonged in the Protocol using
document.

I think that many of Laurence's questions fall into this.
I wonder if we need either:

1) a guide to writing CBOR applicability statements.
2) a worked out example.

Probably both.
While a lot of the more normative advice all through 7049bis, it might be too
diffuse to apply quickly.

I think CDDL can be used to clarify things, but it might be that some of
us need the specific ways beat over our heads.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Aug 2020 21:11:30 +0200

Hi Michael,

> 1) a guide to writing CBOR applicability statements.

I’m not sure I’m parsing this correctly — writing CBOR applicability statements?
What do we need an applicability statement for and why would we be writing many of them?

> 2) a worked out example.

I think we have about 30 :-)

https://datatracker.ietf.org/doc/rfc7049/referencedby/
(This is slightly confused — it thinks that RFC 8392 has an informative reference to 7049, and I’m sure the heuristics failed in other places.)

> us need the specific ways beat over our heads.

I think some of the material about tags could go into notable-tags.

But maybe we do need a tutorial document.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Aug 2020 21:18:34 +0200

> What do you think about defining CBOR protocol items that can be either tag-contained or unwrapped?  In CDDL, “dualistic_bigfloat = bigfloat / ~bigfloat”. 

I think that this specific example is the OPTIONAL in your previous message that we would normally try to avoid.

> Making a distinction in the way CWT is defined and named from the way epoch date, bignum and such are defined is what clears it up for me.

Good.

> The definition of the CBOR structure for CWT is independent of the CWT tag. The CBOR structure for the CWT is defined first. Then the tag definition comes a long and says is contents is a CWT.

Right.  Does it even say that a CWT with a tag 61 around it is a CWT?

That is indeed some terminology thing we could try to normalize.

> For tags epoch date, bignum and such there is no independent definition of the contents. That content definite is wrapped up with the tag definition. If you want to use the content without a tag-co…

Indeed.

> That a tag is a “container” is really helpful to me.

We are not using this term in 7049bis.
But if you look at Table 1, there are three (definite-length) major types that have data items in them: 4 (array), 5 (map), 6 (tag).
(Table 2 then reveals that indefinite length strings are represented like containers, but that is not really their semantics.)

> Thanks for the discussion on this!

Certainly!

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 18 Aug 2020 12:39:51 -0700

I went through some of the registered tags and sorted them as to which has an independent definition and which do not. This really helps me get my head straight on this. Pointing out the parallel with maps and arrays is helpful too.

These have tag-independent definitions. If you want to put one of them into a CBOR item without a tag you just reference the independent definition. For example, "the first item in an array is a UUID”. 
  
 Date string — RFC 3339 and RFC 4287
 URI
 Regex
 B64, b64URL 
 MIME
 CWT
COSE_Untagged_Message (there is also COSE_Tagged_Message).

These don’t have tag-independent definitions. If you want to “borrow” the definition, you refer to the tag and then require it be unwrapped. For example, "the first item in the array is an unwrapped CBOR numeric date". Or, like CWT, you say it is a “CBOR numeric date modified to require the tag be omitted”. 

 Epoch Date
 Bignums
 Big float
 Decimal fraction
 Encoded CBOR

If you want the tag for either category above you say “the first item in the array is tag X”, The definition of tag X will tell you want the content is.

At this point, I think of tags more a typed content. 

LL

> On Aug 18, 2020, at 12:18 PM, Carsten Bormann <[email-masked]> wrote:
> 
> On 2020-08-17, at 22:37, Laurence Lundblade <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>>

---

From: Michael Richardson <[email-masked]>
Date: Tue, 18 Aug 2020 17:30:55 -0400

> What do we need an applicability statement for and why would we be writing many of them?

No, it is about how, when doing EAT, or DOTS, or GRASP, or CoSWID, ...
how are we applying CBOR.

    > (This is slightly confused — it thinks that RFC 8392 has an informative
    > reference to 7049, and I’m sure the heuristics failed in other places.)

Yes.
But, if I wanted to tell someone which one to use, which one would I use?
There are many which are not useful.

    > But maybe we do need a tutorial document.

Yes, it's a tutorial about writing a document that references CBOR.