Subject: [calsify] Fwd: New Timestamp Draft

From: Ujjwal Sharma <[email-masked]>
Date: Sat, 23 Jan 2021 02:53:34 +0530

CCing the Calsify ML because I didn't include it in the original email
by mistake.

-------- Forwarded Message --------
Subject: Re: [calsify] New Timestamp Draft
Date: Sat, 23 Jan 2021 02:51:00 +0530
From: Ujjwal Sharma <[email-masked]>
Organization: Igalia S.L.
To: Ned Freed <[email-masked]>

Hi Ned!

I could not agree with you more on the different matters. Here's the
response to the specific points raised by you.

On 23/01/2021 12.17 am, Ned Freed wrote:
> +1. The stability of this specification has been invaluable.

Please refer to my email to Eliot for more clarification on this.
Overall, this is a goal that I share wholeheartedy, and the whole idea
behind the new draft is to design it in a way that is very generalized
and that allows us to avoid any further additions to it and keep a
similar level of stability. Note for example the deferring of key
selection and standardization to bodies like IANA and Unicode consortium.

> +1. And if there's some sort of liason with ISO or whoever is responsible 
> for this, they should be engaged.

My understanding of the situation has been that this liason would be
CalConnect. By all means, we have been collaborating quite closely with
them, thanks to the amazing efforts of Ronald Tse and have plans to work
with them further to enable greater adoption.

Hope this helps, please feel free to ask for further clarifications.

Best,
Ujjwal

-- 
Ujjwal "Ryzokuken" Sharma (he/him)

Compilers Hacker, Node.js Core Collaborator and Speaker

---

From: Ned Freed <[email-masked]>
Date: Sat, 23 Jan 2021 15:17:01 -0800 (PST)

Generality has real costs. To name the obvious example, I don't want to
*ever* see a timezone rule in a log file timestamp, or a field in an
email message.

I wouldn't have a problem with any of this if all you were proposing a new
format for calendar usage. Calendars are unusual in that they routinely deal
with dates in the future, and worse, dates in the future that are subject to
the vagarities of human behavior - so a given value does NOT, in general,
specify a fixed point in time.

But you're proposing a replacement for ISO 8601/RFC 3339, with the implication
that absent per-application profiling you have to accept the gamut. That IMO
goes too far.

				Ned

---

From: Michael Douglass <[email-masked]>
Date: Sat, 23 Jan 2021 18:42:57 -0500

> *ever* see a timezone rule in a log file timestamp, or a field in an

I have had a number of conversations with people tasked with auditing 
log records bemoaning the lack of a timezone id in logs. Without the id 
how do we know how an offset or UTC date was calculated?

Accurately recording the date/time that an event occurred is as 
important for logging as it is for future events.

{and I don't think we're talking about rules - just ids]

> _______________________________________________
> calsify mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/calsify

---

From: Ned Freed <[email-masked]>
Date: Sat, 23 Jan 2021 16:00:53 -0800 (PST)

> how do we know how an offset or UTC date was calculated?

The purpose of a time value in the sorts of logs I'm talking about is to
accurately describe the moment in time when something happened. Nothing more,
nothing less. The RFC 3339/ISO 8601 format gives you this along with the
ability to specify the value as an offset from GMT.

If you really want more information along the lines of what this draft
provides, like what timezone rules were in effect somewhere or what calendar it
should be "projected to" (to use the draft's terminology), then by all
means include that in a separate field in the log.

With computers increasingly going back to being shared resources in some random
data center some place other than anyone involved in generating the log entry,
figuring out the right "somewhere" is a highly nontrivial problem. This
is  true even for time zone offsets, and the further you go the harder it gets.
(Of course this doesn't stop people from using offsets in their logs even when
they have nothing to do with anything other than the location of some random
piece of hardware.)

And of course there can be multiple somewheres, which means these values need
to have additional semantics attached in order to be interpreted properly.

All of which argues strongly for not combining what are in fact separate
pieces of information.

Given all this, my advice to your auditor pals is, "Be careful what you wish
for, because you're likely to end up with useless garbage."

There's also very strong trend in the logging world to try and avoid complex
field values because they hugely complicate aggregation, searching and sorting
operations. In this world the use of RFC 3339 may be profiled to GMT only, or
replaced entirely by a UNIX epoch or milliepoch values.

> it's simply an offset showing what's considered to be local time.

? Offsets are a standard part of ISO 8601/RFC 3339.

> important for logging as it is for future events.

?? None of this has anything to do with accuracy. If you're not putting
in the correct offset for your time values you're doing it wrong.

> {and I don't think we're talking about rules - just ids]

??? We most certainly are talking about rules. If you think otherwise you
need to reread the draft.

				Ned

---

From: "Bron Gondwana" <[email-masked]>
Date: Sun, 24 Jan 2021 14:23:57 +1100

Ned,

Would your concern here be resolved by having this draft define a "point in time" profile which was RFC3339 minus the 2-digit-year option, and a "full date" profile which includes all the rest of the details - such that logging software could be defined to use the "point in time" profile.

I'm really keen for there to be an RFC which can be referenced by both new calendaring drafts and by other IETF documents, which is developed by this group of people who are already involved in both the ISO work (via Ronald and CalConnect) and the ECMA standardisation effort.

Generality has real costs, but so does excess simplicity - if there's not a standard way to describe things that people really need, then they all develop their own extensions.  This effort is to capture the extensions that have been needed in the real world already, and standardise them to encourage everyone to extend the same way!

Cheers,

Bron.

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]

---

From: Ned Freed <[email-masked]>
Date: Sat, 23 Jan 2021 23:32:57 -0800 (PST)

> Would your concern here be resolved by having this draft define a "point in
> time" profile which was RFC3339 minus the 2-digit-year option, and a "full
> date" profile which includes all the rest of the details - such that logging
> software could be defined to use the "point in time" profile.

Maybe. I don't have any objection to deprecating features of RFC 3339 that
experience has shown we don't want or need.

However, I don't think you've really made the case that the full date stuff is
best addressed by packing all kinds of stuff in a single field value. It's not
like JSON, XML, and so on have no mechanism for specifying a group of related
values. As such, I question the need to define an additional syntax to do this,
one that's considerably less flexible.

There's also the issue of what constraints are applied to the additional stuff.
The obvious one is that it MUST NOT be allowed to move the point in time
of the timestamp around - I don't see this stated anywhere.

> I'm really keen for there to be an RFC which can be referenced by both new
> calendaring drafts and by other IETF documents, which is developed by this
> group of people who are already involved in both the ISO work (via Ronald and
> CalConnect) and the ECMA standardisation effort.

> Generality has real costs, but so does excess simplicity - if there's not a
> standard way to describe things that people really need, then they all develop
> their own extensions.

Actualy, it's this draft that engages in excess simplicity - the simplicity,
for example, of assuming that it never makes sense to associate more than
one set of time zone rules with a timestamp.

And there's no missing standard here. We have no shortage of standardized
grouping syntaxes.

> This effort is to capture the extensions that have been needed in the real
> world already, and standardise them to encourage everyone to extend the same
> way!

I'm a very long way from being convinced of this.

				Ned

---

From: "Bron Gondwana" <[email-masked]>
Date: Sun, 24 Jan 2021 19:36:46 +1100

> one that's considerably less flexible.

This idea has been "reinvented" more than once, which suggests that there is demand for it.

> of the timestamp around - I don't see this stated anywhere.

For sure you would then need to have both a [+-] and a zone name.  That's an interesting idea, and it does raise the question of what you do if the zone and the fixed offset no longer match by the time you process the date (or indeed, are incorrect immediately).  If the time is a floating value (no Z and no [+-]) then the point in time is also floating.

> And there's no missing standard here. We have no shortage of standardized

The initial justification for this was very much "this is true but said grouping syntaxes don't tend to survived round trips through different systems very well, so we want to capture all the important details in a serialised form".

> I'm a very long way from being convinced of this.

I do have one more question (I'm going to do a call for adoption anyway, but I'll take this email thread as a voice against adoption from you unless I hear otherwise)...

Would you be happy if this was either "extends RFC3339" or an entirely separate piece of work that's "only for calendaring systems" - or do you object to the entire principle of doing any work in this space at all?

Thanks,

Bron.

---

From: Ned Freed <[email-masked]>
Date: Sun, 24 Jan 2021 05:55:38 -0800 (PST)

Of course the other way to solve all of these problems is to leave RFC 3339 and
ISO 8601 alone, and instead specify a new format that incorporates RFC 3339  as
a component, the same way you're incorporating time zone rule names, calendar
names, and on.

Then implementors can choose on a case by case basis which format best
meets their needs.

Almost all of the issues here revolve around the fact that you're updating
an existing format.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Sun, 24 Jan 2021 06:00:04 -0800 (PST)

> This idea has been "reinvented" more than once, which suggests that there is

No, what it demonstrates is that there is demand for a way to combine a set of
pieces into a composite. It doesn't demonstate any need to revise the format we
use for specifying a point in time in a readable form, nor does it make the
case that a new syntax is needed.

> For sure you would then need to have both a [+-] and a zone name.  That's an
> interesting idea, and it does raise the question of what you do if the zone and
> the fixed offset no longer match by the time you process the date (or indeed,
> are incorrect immediately).  If the time is a floating value (no Z and no [+-])
> then the point in time is also floating.

Well, now we have a problem, because now you have a situation where you have to
understand the meaning of all the pieces in order to be sure you know what time
is being specified. (Saying "this needs further adjustment in some way" is very
different than saying "display the value using this type of calendar".)

This essentially makes the format non-extensible unless you include some sort
of "mandatory/optional" label, never mind what the specification says about
extensibiity.

> The initial justification for this was very much "this is true but said
> grouping syntaxes don't tend to survived round trips through different systems
> very well, so we want to capture all the important details in a serialised
> form".

Really? JSON and XML don't survive round trips through different systems? 

If you have found this to be the case, you are operating in a different
universe than the one I occupy. Mind you, there's no shortage of problems with
JSON, XML, and ASN.1 for sure (and probably CBOR, although I lack experience in
that area), but I don't tnink surviving transport is one of them. (Well, OK,
the fact that JSON has to be treated as binary is sometimes missed, but this is
a generic JSON issue.)

That said, I think you're talking about semantics, not syntax, here. People
uncover a need to specify the timezone rules or display calendar or whatever
and the base format doesn't support that, and rather than agree on a single
mechanism for grouping this stuff they have instead extended the base
format in different and incompatible ways.

This ia valid argument for standardizing a grouping mechanism. But again, it
falls far, far short of making the case for revising or extending the time
stamp format. That's a different layer.

> I do have one more question (I'm going to do a call for adoption anyway, but
> I'll take this email thread as a voice against adoption from you unless I hear
> otherwise)...

> Would you be happy if this was either "extends RFC3339" or an entirely
> separate piece of work that's "only for calendaring systems" - or do you object
> to the entire principle of doing any work in this space at all?

Having this be a separate piece of work that uses RFC 3339 as a component
rather than obsoleting or extening it removes my objection to pursuing the
work. Of course the result still needs to interoperate, and this conversation
has convinced me that there's work to do in that regard.

				Ned

---

From: "Bron Gondwana" <[email-masked]>
Date: Mon, 25 Jan 2021 10:15:02 +1100

ISO8601 had a major update in 2019, due to there being demand to be able to represent additional types of date information.

But this does sound like everyone would be a lot happier with a small update to 3339 to remove 2 digit years, allow 6 digits years, and otherwise leave pretty well enough alone - and then define a new format for everything else.  I expect that would be acceptable.

Cheers,

Bron.

---

From: "Neil Jenkins" <[email-masked]>
Date: Mon, 25 Jan 2021 12:18:28 +1100

I think the disagreement here stems from two viewpoints on what RFC3339 is. Ned is looking at it from the viewpoint that it defines a format for (log) timestamps only. The other side is viewing it as simply a standard serialisation format for dates in internet protocols without restriction on use case.

Having looked at the document's introduction again, Ned is technically correct:

   There are many ways in which date and time values might appear in
   Internet protocols:  this document focuses on just one common usage,
   viz. timestamps for Internet protocol events

However, I would assert that in my experience, many view RFC3339 as just a standardised serialisation format for all dates in contexts that go beyond this. I think there is a clear demand for a serialisation format that *can* incorporate a time zone or calendar identifier, and standardising this within the IETF is more likely to lead to interoperability. I also agree that you would never want to use such a time-zone variant in log files. 

So I think it comes down to two options:
 * A single RFC that replaces 3339 containing all the variants. The benefit is you have one place to reference for how to serialise a date-time.
 * A new adjunct RFC for how to serialise date-times with IANA time zones, or without an absolute time at all (local "wall clock" date-times). This would not overlap 3339, which would continue to define how to serialise UTC and fixed-offset date-times.
I am not too fussed which of these paths we go down.  I was originally advocating for the first option, but looking at it again, I can see good arguments for the second instead.

Neil.

---

From: Ned Freed <[email-masked]>
Date: Sun, 24 Jan 2021 19:50:18 -0800 (PST)

> I think the disagreement here stems from two viewpoints on what RFC3339 is.
> Ned is looking at it from the viewpoint that it defines a format for (log)
> timestamps only.

Not quite. I pointed out that RFC 3339 was and is used for timestamps in logs.
I never said, or even implied, that it doesn't have (many) other uses, either
by itself or in combination with other information.

However, the more important point, as you note below, is that RFC 3339
specifies a timestamp format. It's entirely self-contained, and aside from
allowing specification of a time zone offset, it provides no information about
presentation or anything else.

> The other side is viewing it as simply a standard serialisation format for
> dates in internet protocols without restriction on use case.

That may be true but I question the relevance. There are a myriad of use cases
for RFC 3339. There are also a myriad of use cases for RFC 3339 combined with
other information. I don't think anyone is claiming otherwise.

The issue here isn't that these other cases aren't worthy of consideration -
they are - it's that addressing them by changing the core semantics of a widely
used timestamp format isn't the right way to go about it.

>    viz. timestamps for Internet protocol events

Exactly.

> However, I would assert that in my experience, many view RFC3339 as just a
> standardised serialisation format for all dates in contexts that go beyond
> this.

I don't know what this means. RFC 3339's semantics are clear, and the syntax
only allows for the expression of those semantics. If you "go beyond" those
semantics, you either have to extend the syntax, in which case you're not
talking about RFC 3339 format any more, or you have to create some sort of
grouping of RFC 3339 with additional information. In either case you have
extended RFC 339, and that's not a matter of how you view it.

> I think there is a clear demand for a serialisation format that *can*
> incorporate a time zone or calendar identifier, and standardising this within
> the IETF is more likely to lead to interoperability. I also agree that you
> would never want to use such a time-zone variant in log files.

OK... so it sounds to me like we're in agreement that RFC 3339's
semantics need to be retained. Good.

>  * A single RFC that replaces 3339 containing all the variants. The benefit
>    is you have one place to reference for how to serialise a date-time.

Eliot already explained why this is a really bad idea.

>  * A new adjunct RFC for how to serialise date-times with IANA time zones, or
>    without an absolute time at all (local "wall clock" date-times).

A convention for this is already provided by RFC 3339, actually. Section 4.2.

>    This
>    would not overlap 3339, which would continue to define how to serialise
>    UTC and fixed-offset date-times.

This is a little confusing: Adjunct implies that you're extending RFC 3339, "no
overlap" implies you're not. In any case, I would recommend building on the
existing syntax of RFC 3339 if possible.

> I am not too fussed which of these paths we go down.  I was originally
> advocating for the first option, but looking at it again, I can see good
> arguments for the second instead.

Exactly.

				Ned

---

From: Shane Carr <[email-masked]>
Date: Mon, 25 Jan 2021 02:01:56 -0600

A few things:

- 3339 and 8601-1 are good as a timestamp format.  The extension Ujjwal
proposed generalizes those formats based on the demonstrated needs of the
ecosystem when dealing with "human dates" as opposed to "computer dates".
It's an interesting observation that timestamps (e.g. log files) are
typically used for contemporary events, so the extended 6-digit year format
may not really be necessary.  Therefore, I think it would be a reasonable
option to keep 3339 unmodified and alive as the timestamp format, and
consider the new proposal to be a peer.

- I see a thread with Eliot Lear, but I don't see where the pros and cons
of a combined RFC versus separate RFCs are discussed.  (But this is moot if
we were to decide to keep RFC 3339 alive in its current state.)

- The key question solved by this RFC is the data model.  What Ujjwal is
proposing is to use a timestamp paired with information about the time zone
and calendar system.  Once we agree on the data model (I do), then we can
talk about how to package the data.  Could we use JSON?  Sure, but we still
need to specify the structure of the JSON and what form the keys/values can
take.  Ujjwal is proposing a string syntax with bracketed annotations
because (a) it is already used in the wild, and (b) compared with JSON,
strings are more compact [no keys], more compatible [byte array rather than
struct/map], and easier to deal with [no need to format/stringify].  I hope
not to get into a debate on point (b), but if there are people here who
believe that we should stop defining new string syntaxes and instead let
JSON (ECMA-404) rule all, then we may need to engage on that topic.

Shane

---

From: Eliot Lear <[email-masked]>
Date: Mon, 25 Jan 2021 11:36:23 +0100

Hi Shane,

> On 25 Jan 2021, at 09:01, Shane Carr <sffc=[email-masked]> wrote:
> 
> A few things:
> 
> - 3339 and 8601-1 are good as a timestamp format.  The extension Ujjwal proposed generalizes those formats based on the demonstrated needs of the ecosystem when dealing with "human dates" as opposed to "computer dates".  It's an interesting observation that timestamps (e.g. log files) are typically used for contemporary events, so the extended 6-digit year format may not really be necessary.  Therefore, I think it would be a reasonable option to keep 3339 unmodified and alive as the timestamp format, and consider the new proposal to be a peer.

Yes, though I really have no problem now obsoleting 2 digit dates, and giving advice for applications to warn of their use (for instance).

> 
> - I see a thread with Eliot Lear, but I don't see where the pros and cons of a combined RFC versus separate RFCs are discussed.  (But this is moot if we were to decide to keep RFC 3339 alive in its current state.)

I only have cons, but see below.

> 
> - The key question solved by this RFC is the data model.  What Ujjwal is proposing is to use a timestamp paired with information about the time zone and calendar system.  Once we agree on the data model (I do), then we can talk about how to package the data.

Yes.  When considering this, let’s be sure that we have live use cases that are important for at least one community to address.  Oddly I don’t think that’s the TZ community, because much of what is being offered is already in iCal.  That being said, there are people more expert here than me, and I am quite happy to be proved wrong.

> Could we use JSON?  Sure, but we still need to specify the structure of the JSON and what form the keys/values can take.  Ujjwal is proposing a string syntax with bracketed annotations because (a) it is already used in the wild, and (b) compared with JSON, strings are more compact [no keys], more compatible [byte array rather than struct/map], and easier to deal with [no need to format/stringify].  I hope not to get into a debate on point (b), but if there are people here who believe that we should stop defining new string syntaxes and instead let JSON (ECMA-404) rule all, then we may need to engage on that topic.

The nice thing about defining the data model is that you can serialize it however you like, and there certainly needn’t be only one way.  This having been said, my only suggestion here is that you start in a way that leverages toolsets that are commonly used for at least the applications that are envisioned.  The nice thing about JSON and XML is that you can’t shake a stick without hitting such a thing (though if you have to lift the XML results, be sure to bend your knees so as not to break your back ;-).  FWIW YANG can help, and I would encourage its use as a description language, even if you have others, simply so that the work can be adopted more broadly by the O&M community.  YANG isn’t very popular with apps people, but what we’re talking about here is so small that maybe one could skate by.

Eliot