Subject: [Cbor] Encoding Arbitrary Time Ratios

From: Emile Cormier <[email-masked]>
Date: Fri, 2 Apr 2021 22:39:10 -0300

I'd like to be able to encode any arbitrary C++ std::chrono::time_point (
https://en.cppreference.com/w/cpp/chrono/time_point) into CBOR in a
lossless manner. A C++ time_point basically contains three values: a
numerator and denominator to express a ratio in seconds, plus a multiplier
(aka "count"). For example, to store the number of weeks since the epoch,
one could use:

using weeks = std::period<uint64_t, std::ratio<604800, 1>>;
std::chrono::time_point<std::chrono::system_time, weeks> when(123);

In order words, 123 * (604800 / 1) seconds.

I could of course convert 123 weeks to seconds before encoding to CBOR, but
for large numbers of weeks, I would risk arithmetic overflow.

std::ratio<604800, 1> in my example is a compile-time constant, but it
would have to be encoded as a run-time pair of integers over CBOR. Note
that the ratio is arbitrary in C++. Nothing prevents a C++ programmer from
storing a "micro-fortnite" as a std::ratio<14*86400, 1000000>.

To allow the encoding any arbitrary time ratios, I propose a new CBOR tag
wrapping an array of 1 to 3 values:

[Count, Numerator, Denominator]

where Count is an integral or floating point number, and Numerator and
Denominator are integers representing the ratio in seconds.The Numerator
and Denominator would default to 1 if absent.

CBOR tags 1001-1003 would be extended to allow this new CBOR tag as the
value key. This would allow me to encode additional metadata on the
std::chrono::time_point clock type (POSIX, UTC, TAI, GPS).

I can write-up a draft proposal for this new tag on my Github, but I'd like
to hear your thoughts first.

Cheers,
Emile Cormier

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 3 Apr 2021 15:00:11 +0200

On 2021-04-03, at 03:39, Emile Cormier <[email-masked]> wrote:
> 
> To allow the encoding any arbitrary time ratios, I propose a new CBOR tag wrapping an array of 1 to 3 values:
> 

Right.  We already have tag 30 for rational numbers of form n/d.
So this would be c*n/d.
I’d imagine that in many applications, n/d would be a “constant” (i.e., available from the application’s context); prefix packing or simply a convention could handle this.

> where Count is an integral or floating point number, and Numerator and Denominator are integers representing the ratio in seconds.The Numerator and Denominator would default to 1 if absent.

Only the trailing elements really can be “absent” (they could be null as well, but why not simply encode the 1 then).  Not sure that the denominator is likely to be the one that one would leave out.

> CBOR tags 1001-1003 would be extended to allow this new CBOR tag as the value key. This would allow me to encode additional metadata on the std::chrono::time_point clock type (POSIX, UTC, TAI, GPS).

Right, similar to the way we already handle 4 and 5 (and probably should add 30).
Alternatively, there could be separate keys for numerator and denominator, so these could be freely left out and/or combined.

I’m not so familiar with the use cases of this except for a scale factor of 86400 (i.e., days).

Grüße, Carsten

---

From: Emile Cormier <[email-masked]>
Date: Sat, 3 Apr 2021 16:58:44 -0300

On Sat, Apr 3, 2021 at 10:00 AM Carsten Bormann <[email-masked]> wrote:

> I’d imagine that in many applications, n/d would be a “constant” (i.e.,
> available from the application’s context); prefix packing or simply a
> convention could handle this.
>
> Only the trailing elements really can be “absent” (they could be null as
> well, but why not simply encode the 1 then).  Not sure that the denominator
> is likely to be the one that one would leave out.
>

'null' could be the magic value that represents "ratio according to the
application context".

I've just had another idea... a denominator of zero (or possibly null)
could be a magic value used to indicate that the numerator is a base 10
exponent. This could allow a more compact representation when the ratio is
milliseconds, nanoseconds etc. Eg [123, -9, 0] instead of [123, 1,
1000000000] for expressing 123ns.

I'm also unsure about the denominator being more likely to be left out. I
think applications are more likely to transmit times as fractions of
seconds (e.g. Javascript Date in milliseconds, C++ std::chrono::sys_time in
nanoseconds), therefore it might be the numerator that's more likely left
out.

> Alternatively, there could be separate keys for numerator and denominator,
> so these could be freely left out and/or combined.
>

I prefer having a dedicated tag for time expressed as C*N/D so that I can
have a more compact representation when the clock is assumed to be POSIX
and the clock accuracy doesn't matter. This new tag could then be handled
in CBOR extended time the same way it handles tag 4 and 5 (as you hinted).

I'm wondering if a new C*N/D tag could be useful in application domains
outside of timekeeping. It could be called something like "scaled ratio".

> I’m not so familiar with the use cases of this except for a scale factor
Time points based on the 86400/1 ratio now have their own type alias in
C++: std::chrono::sys_date. They are now the recommended way to store and
compute dates in an efficient manner. Breaking them down into
year/month/day components is only recommended for display or data exchange
purposes.

Many thanks for the feedback!

Cheers,
Emile

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 3 Apr 2021 20:00:03 -0700

Here’s a CBOR time FAQ that I wrote a few years ago: https://github.com/laurencelundblade/QCBOR/blob/master/doc/TimeTag1FAQ.md

If designing a protocol that’s to be implemented by lots of folks in many programming languages, then tag 1 epoch time seems like it covers most of the use cases to me. I think it is even good to exclude float from tag 1 because it doesn’t add much above a 64-bit epoch time value for human activity (see FAQ). 

It would be good for the C++ CBOR encoder/decoder to translate from tag 1 epoch time to/from C++ time points since tag 1 epoch is what you’ll find in most CBOR-based protocols.

What you propose seems useful in use cases of one C++ talking to another when the protocol is not intended for broad use.

LL

> On Apr 3, 2021, at 12:58 PM, Emile Cormier <[email-masked]> wrote:
> 
> 
> 
> On Sat, Apr 3, 2021 at 10:00 AM Carsten Bormann <[email-masked] <mailto:[email-masked]>> wrote:
> On 2021-04-03, at 03:39, Emile Cormier <[email-masked] <mailto:[email-masked]>> wrote:
> > 
> 'null' could be the magic value that represents "ratio according to the application context".
> 
> I've just had another idea... a denominator of zero (or possibly null) could be a magic value used to indicate that the numerator is a base 10 exponent. This could allow a more compact representation when the ratio is milliseconds, nanoseconds etc. Eg [123, -9, 0] instead of [123, 1, 1000000000] for expressing 123ns.
> 
> I'm also unsure about the denominator being more likely to be left out. I think applications are more likely to transmit times as fractions of seconds (e.g. Javascript Date in milliseconds, C++ std::chrono::sys_time in nanoseconds), therefore it might be the numerator that's more likely left out.
>  
> 
> I prefer having a dedicated tag for time expressed as C*N/D so that I can have a more compact representation when the clock is assumed to be POSIX and the clock accuracy doesn't matter. This new tag could then be handled in CBOR extended time the same way it handles tag 4 and 5 (as you hinted).
> 
> I'm wondering if a new C*N/D tag could be useful in application domains outside of timekeeping. It could be called something like "scaled ratio".
>  
> 
> Time points based on the 86400/1 ratio now have their own type alias in C++: std::chrono::sys_date. They are now the recommended way to store and compute dates in an efficient manner. Breaking them down into year/month/day components is only recommended for display or data exchange purposes.
> 
> _______________________________________________
> CBOR mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/cbor <https://www.ietf.org/mailman/listinfo/cbor>

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 4 Apr 2021 10:38:00 +0200

> I've just had another idea... a denominator of zero (or possibly null) could be a magic value used to indicate that the numerator is a base 10 exponent. This could allow a more compact representati…

We already have decimal fractions (tag 4), so this is covered.
There is also key -9 etc. for tag 1001, if you want to separate the seconds and their fractional parts.
(I’m not a big fan of encoding magic values in the data items themselves, either.)

> I'm wondering if a new C*N/D tag could be useful in application domains outside of timekeeping. It could be called something like "scaled ratio".

Probably.
Making it amenable to cbor-packed suffix (or circumfix!) compression would make this a quite versatile tool.

> Time points based on the 86400/1 ratio now have their own type alias in C++: std::chrono::sys_date. They are now the recommended way to store and compute dates in an efficient manner. Breaking them…

Right, so seconds (and their decimal fractions) and days are the most important scaling factors.

Grüße, Carsten

---

From: Emile Cormier <[email-masked]>
Date: Sun, 4 Apr 2021 14:05:48 -0300

> We already have decimal fractions (tag 4), so this is covered.

I overlooked tag 4 for that purpose. Thanks for reminding me.

> make this a quite versatile tool.
Are you referring to https://tools.ietf.org/html/draft-ietf-cbor-packed-02
? I wasn't aware of its existence before this.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 4 Apr 2021 20:02:33 +0200

On 2021-04-04, at 19:05, Emile Cormier <[email-masked]> wrote:
> Are you referring to https://tools.ietf.org/html/draft-ietf-cbor-packed-02 ? I wasn't aware of its existence before this.

Indeed.  This doesn’t mean we should design tags with the assumption that we will always have packing available, but we also can assume if there is a strong reason to be very frugal with encoding space, cbor-packed will be available as an alternative to touching every single detail to make it more compact.

Grüße, Carsten

---

From: Thiago Macieira <[email-masked]>
Date: Mon, 5 Apr 2021 08:57:28 -0700

On Saturday, 3 April 2021 20:00:03 PDT Laurence Lundblade wrote:
> What you propose seems useful in use cases of one C++ talking to another
> when the protocol is not intended for broad use.
> 
> LL

Not really,. it's not. It's only useful if both sides already know what the 
other will send, otherwise you can't find a std::chrono::time_point 
specialisation to match the type. Note that the std::ratio specialisations 
that time_point are constructed on are compile-time constants. You can't match 
them unless you know before hand you may need to.

For example, ii I sent a time point specifying a ratio of 14609700 : 864, I 
REALLY do not expect the receiver to find a matching type (that's an average 
century with 36524 and one quarter days). Another "useful" ratio is 1296 : 
1000, the microfortnight.

So, no, let's be very clear that code meant to be interoperable should never 
use one of those. If we think it's useful to allow protocols to specify their 
own time points, sure, make it possible to encode them in a specific tag. I 
don't think it is.

I would say only decimal powers of the second with values 0, -3, -6 and -9 are 
meant to be used in interoperable protocols and therefore we should optimise 
for size for those and only those.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering

---

From: Emile Cormier <[email-masked]>
Date: Mon, 5 Apr 2021 19:09:37 -0300

> them unless you know before hand you may need to.

You are indeed correct that std::ratio is compile-time. But passing along
the ratio via CBOR allows the decoder to compute the values needed to store
it under a different compile-time std::ratio used by the receiving
application (as well as detecting overflow). If the destination ratio
happens to be the same, the decoder can bypass all the intermediate
calculations.

> 1000, the microfortnight.
Now, now, no need to get snarky. ;-)  I concede that ratios other than
86400:1 (days) or 1:10^3N are contrived examples and probably have no
practical use over an interoperable protocol.

> for size for those and only those.
I think dates expressed as a number of days since the epoch would also be
useful. Tag 0 (date/time string) has a numeric counterpart under tag 1. I
think tag 1004 (date string) should also have a numeric counterpart under
one form or another.

If tag 30 (rational number) were allowed to express time in seconds, it
could be used to represent integral days with a range of +/-2^61 / 86400 =
+/-26687997791825 days, which corresponds to +/-73069256156.73 years.
That's 5.3 times the age of the universe.

Of course, seconds since epoch (tag 1) could also be used to exchange a
date as days*86400, but the semantics of the value (a date) would be lost.

How about just a new tag for a date expressed as the number of days since
the epoch? This would definitely be useful (and efficient) in C++ when both
ends use std::chrono::sys_date. Decoders for other languages could simply
multiply the value by 86400 and handle it the same way as tag 1.

---

From: Thiago Macieira <[email-masked]>
Date: Mon, 5 Apr 2021 15:28:52 -0700

> practical use over an interoperable protocol.
[cut]
> think tag 1004 (date string) should also have a numeric counterpart under

I can accept indicating the number of days. I don't find that a particularly 
useful encoding, since it does little to save space, but does add complexity. 
I don't find saving 2 bytes because we can encode the whole number of days 
until June 2149 a compelling value.

> That's 5.3 times the age of the universe.

Please don't require rational numbers or BigFloat to represent time. The only 
valid encodings should be integers and floating point, and maybe not even the 
latter.

> Of course, seconds since epoch (tag 1) could also be used to exchange a date
> as days*86400, but the semantics of the value (a date) would be lost.
>
> the epoch? 

Semantically indicating a date without time would be far more valuable, 
regardless of how it gets encoded. Simply expressing a time point that happens 
to be midnight UTC on a particular date is not that. So I support this idea.

Similarly, it would be useful to have just a time within a day.

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 5 Apr 2021 16:37:26 -0700

> On Apr 5, 2021, at 3:28 PM, Thiago Macieira <[email-masked]> wrote:
> 

It’s been done.:-)  https://tools.ietf.org/html/rfc8943

LL

---

From: Emile Cormier <[email-masked]>
Date: Mon, 5 Apr 2021 20:39:28 -0300

> until June 2149 a compelling value.

My main motivation is not to save space. It's to save the computational
cost of an intermediate ISO8601 date string when both ends simply want to
exchange a std::chrono::sys_date (while preserving the date semantic). It's
the same motivation (presumably) as for tag 1. Not only is the Y/M/D to
days computation complex, you also have to check for invalid dates (e.g.
January 32nd, February 29 on a non-leap-year, etc).

> valid encodings should be integers and floating point, and maybe not even

I can support that (no rational nor BigFloat), as long as there's a
mechanism to exchange dates as the number days since epoch without losing
the 'date' semantic.

I'm afraid the ship has already sailed regarding floating point
representation of time in the existing CBOR time tags. Javascript
'natively' handles time as floating point milliseconds. In Python, it's
floating point seconds.

We _could_ constrain a new, proposed "numeric date" tag to have an
integer-only data item, but it would not be consistent with the other
existing time tags which support both integer and floating point.
Javascript folks might complain about the lack of floating point.

> to be midnight UTC on a particular date is not that. So I support this

I'm also not fond of expressing dates as time-date values that coincide
with midnight.

I'm glad to hear about support for a numeric date tag in the same spirit as
tag 1. I'm happy to drop my "scaled ratio" idea if I can at least
efficiently transfer a std::chrono::sys_date without performing Gregorian
calendar conversions. All other likely use cases for
std::chrono::time_point (in seconds, ms, us, or ns) can be handled by the
existing tags.

That could be expressed with the CBOR duration tag (1002) as the duration
since midnight. Of course, the application would have to know to interpret
this duration as the time of day. std::chrono::duration is the recommended
way to store the time of day efficiently in C++. The new
std::chrono::hh_mm_ss in C++20 is intended more for display purposes.

Perhaps a key could be added in tag 1002 to indicate that the duration
represents a time of day?

Cheers,
Emile Cormier

---

From: Emile Cormier <[email-masked]>
Date: Mon, 5 Apr 2021 20:48:41 -0300

> It’s been done.:-)  https://tools.ietf.org/html/rfc8943

Darn, I missed tag 100 in the CBOR tags list! I would have expected
something like that to be adjacent to tag 1004. I guess the intent was to
save an extra byte.

Tag 100 is "Unsigned or negative integer", no floating point. I can
certainly live with that.

Alright I'd like to drop my original "scaled ratio" idea, as it doesn't
seem that it will gain much traction. At least one thing that came out of
this discussion (from Thago) is there's currently no way to explicitly
express a "time of day" semantic.

Thank you all for indulging me in this "out-of-the-box" idea. :-)

Cheers,
Emile Cormier

---

From: Thiago Macieira <[email-masked]>
Date: Mon, 5 Apr 2021 16:48:51 -0700

> represents a time of day?

My motivation here is to represent the QTime class, which is a component of 
QDateTime, but not a very important motivation for me. We've been asked more 
than once to extend it to allow any duration, not just intra-day.

I'm happy with transferring QDateTime and QDate over a network.