Subject: [Cbor] Invalid tag (issue 176); notable tags draft

From: Carsten Bormann <[email-masked]>
Date: Tue, 12 May 2020 22:28:47 +0200

As we discussed in conjunction with https://github.com/cbor-wg/CBORbis/issues/176, we need a registration for an invalid tag, which is reserved so it can be used in an implementation to store the absence of a tag into an integer that would otherwise carry the number of that tag.

I have put the text for a specification for that registration in:

https://www.ietf.org/id/draft-bormann-cbor-notable-tags-00.html#name-iana-considerations

I have also applied for the tag number at IANA.

I took the occasion to start drafting the “notable tags” draft I have been bandying about in recent CBOR meetings.  Most of the text that would make that draft useful is not yet written.  I also need to shop around for permissions to copy over original tag definition texts into that draft.  But the structure is already a good straw man for how I think this document should look like.

Grüße, Carsten

---

From: Thiago Macieira <[email-masked]>
Date: Tue, 12 May 2020 15:09:30 -0700

On Tuesday, 12 May 2020 13:28:47 PDT Carsten Bormann wrote:
> As we discussed in conjunction with
> https://github.com/cbor-wg/CBORbis/issues/176, we need a registration for
> an invalid tag, which is reserved so it can be used in an implementation to
> store the absence of a tag into an integer that would otherwise carry the
> number of that tag.
> 

The best tag number for this would be 0, since it allows a bzero'ed structure 
to indicate the invalid tag, but that's already taken. The second best would 
be ~uint64_t(0), which would also allow the same property, just by making the 
implementation store the actual tag number plus 1 (0 is -1, 1 is 0, etc.). So 
why limit it to 16-bit?

Anyway, more importantly for me is a bit of text explaining what 
implementations should do when seeing this tag. This is similar to the 
Undefined simple type, which is very concrete and yet has special roles.

For example, in Qt, these two methods exist:

    QJsonObject::insert(QStringView key, const QJsonValue &value)
    QCborMap::insert(QStringView key, const QCborValue &value)

But they differ in behaviour if the value is Undefined. The JSON method, upon 
receiving a value that is undefined, will remove the key from the map, as JSON 
does not support undefined values. QCborMap, on the other hand, stores a CBOR 
value of type Undefined.

If this discussion is of interest, I'll start a new thread to talk about 
Undefined. But for the invalid tag, let's make sure we have some wording to 
indicate its usage. I suggest something along the lines of:

"Implementations MAY choose to discard this tag, as if it were not present. 
Therefore, CBOR interchange may not rely on this tag being processed by the 
recipient in any particular form: the decoder may pass it on to a higher layer 
for processing or it may discard. Similarly, encoders receiving this tag from 
a higher layer may opt not to encode it in the byte stream."

For me, I won't treat it in any special way. If it is present, the decoder 
will see a QCborTag(0xffff) and I will add a QCborKnownTags::Invalid to Qt and 
CborInvalidTag to TinyCBOR (if it fits into 32-bit).

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel System Software Products

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 May 2020 07:13:38 +0200

> to indicate the invalid tag, but that's already taken.

Indeed — we just didn’t think of that implementation approach when we allocated 0 to time/date in 2013.

> The second best would 

This was discussed at the same time we also discussed that 64-bit (i.e., > 32 bit) tags are likely only going to be processed by implementations that do 64-bit numbers as well, and it was pointed out that some implementations will even attempt to get by with 16 bits for the tags they need.  But we could go ahead and reserve ~0 as another invalid tag as that strategy (storing tag number+1) is only visible inside an implementation, so it doesn’t matter whether that tag is processed during interchange.

> implementations should do when seeing this tag.

https://www.ietf.org/id/draft-bormann-cbor-notable-tags-00.html#name-invalid-tag
Is that text sufficient?

> This is similar to the 

(There could be an insert_unless_undefined that inherits the semantics of JSON-oriented method.)

> Undefined.

Yes, please.  Even better if you could include tag 31 https://github.com/svaarala/cbor-specs/blob/master/cbor-absent-tag.rst in that discussion.

> But for the invalid tag, let's make sure we have some wording to 

We are moving away from encouraging discarding tags.
But the point with the invalid tag is that it is always invalid during interchange; we should not give instructions that give the impression there will be uniform processing if the tag actually is encountered during interchange.

> For me, I won't treat it in any special way.

That is certainly a correct way of processing the tag number when it occurs in interchange.
That leaves it to the application to react to it.

> If it is present, the decoder 

Maybe we should go for Invalid1 (0xffff) and Invalid2 (0xffffffffffffffff) then…

Grüße, Carsten

---

From: Thiago Macieira <[email-masked]>
Date: Wed, 13 May 2020 08:41:51 -0700

On terça-feira, 12 de maio de 2020 22:13:38 PDT Carsten Bormann wrote:
> This was discussed at the same time we also discussed that 64-bit (i.e., >
> 32 bit) tags are likely only going to be processed by implementations that
> do 64-bit numbers as well, and it was pointed out that some implementations
> will even attempt to get by with 16 bits for the tags they need.  But we
> could go ahead and reserve ~0 as another invalid tag as that strategy
> (storing tag number+1) is only visible inside an implementation, so it
> doesn’t matter whether that tag is processed during interchange.

Indeed. TinyCBOR's own structure for validating tags could be shrunk to 16-bit 
and save just under half of the memory use. Right now, I'm using some extra 
bits to indicate that tags 21, 22 and 23 apply to arrays, maps and byte 
strings so I need more than 16 bits anyway, but there are other ways of 
indicating the same thing.

> https://www.ietf.org/id/draft-bormann-cbor-notable-tags-00.html#name-invalid
> -tag Is that text sufficient?

It's sufficient, based on the discussion here. But if I hadn't participated, 
I'd read the "are encouraged to" as something I should implement, not just 
something I can ignore.

> Yes, please.  Even better if you could include tag 31
> https://github.com/svaarala/cbor-specs/blob/master/cbor-absent-tag.rst in
> that discussion.

Will do.

> > will see a QCborTag(0xffff) and I will add a QCborKnownTags::Invalid to Qt
> > and CborInvalidTag to TinyCBOR (if it fits into 32-bit).
> 

Then reserve 0xffff, 0xffffffff and 0xffffffffffffffff.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 May 2020 17:43:37 +0200

> Then reserve 0xffff, 0xffffffff and 0xffffffffffffffff.

Makes sense to me.  Any opposing views?

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Wed, 13 May 2020 11:39:05 -0700

+1

LL

> On May 13, 2020, at 8:43 AM, Carsten Bormann <[email-masked]> wrote:
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor