Subject: Accuracy and Precision in dates

From: Sam Ruby <[email-masked]>
Date: Sat, 14 Aug 2004 13:24:28 -0400

Contemplate, for a moment, realtime publishing.  By that, I mean a world 
in which weblog entries are created, validated, made available, issued, 
modified, accepted, copyrighted, and submitted within seconds.

On top of that, consider the constraints of a user polling once an hour 
or so.  In such a system, knowing precisely in which point in in a 
thirty second process the date was assigned amounts to unnecessary 
precision.

In physics class, many years ago, I learned the difference between 
accuracy and precision.  Consistently hitting the target, it not the 
bullseye, was accuracy.[1]  Consistently hitting the same place, even if 
that place was yards away from the target, is precision.  The 
implication being that if you have a choice between accuracy and 
precision, accuracy is generally preferred.

Now, let's look at pubDate in RSS 2.0.  It very clearly specifies that 
this is to be the date that the entry was published.  And that it is 
optional.

Let's see how this applies to Blogger.  When you create an entry, the 
time you are presented with an entry field, you are also presented with 
a time & date, which you can change.  You can save it as draft and 
publish it later.  The result is a date and time which is *not* the 
publish time.  If you want to be pedantic, the right thing to do is to 
not provide a pubDate, but to define a new element which contains the 
proper semantics.  That would be the precise thing to do.  It would not 
increase interoperability, though.

This is a case where the RSS 2.0 specification is unnecessarily precise. 
  If interoperability is the goal, precisely specifying the radius and 
curvature of a round hole is not the best way to handle square pegs.

Nor is the solution to define more round holes.  Some of the Atom date 
proposals have had as many as five.  The end result of such precision in 
the face of lack of consensus is that such portions of the spec will be 
routinely ignored.

So, how does one increase accuracy?  One way is to enumerate the 
interoperability issues.  Dare's RSSBandit does not treat pubDate the 
same way as NetNewsWire does.  Tim's usage of this element in RSS is 
consistent with the spec, and consistent with the way the tool of his 
choice implements this spec.[2]

How does one resolve this interoperability issue?  Simply to state that 
this value MAY change.  Dare agrees.[3]  Note that this is not a weak 
restriction[4] on producers, but effectively a warning to consumers.

Other such warnings can increase the probability that the date indicated 
more often closely approximates when the entry became available than any 
other date.  An extreme example to illustrate the point:  The Diary of 
Samuel Pepys[5], republished in blog form 340 years later.  How can we 
discourage the "first date of formal issuance" of these entries being 
placed into the atom:d element defined in PaceDateSamRuby?  It would 
seem to me that a note that "Consumers MAY chose to sort based on this 
value." would discourage such usage.

Similarly, how do we stop future dates?  "Consumers MAY chose not to 
display entries containing atom:d elements until the date specified." 
seems like it would be a pretty effective deterrent.

What have we lost?  Well, if it turns out to be important that we 
capture precisely the formal, informative, dates associated with 
professional publications, it may make sense to indicate how such 
information be recorded in a consistent manner.  I can think of nobody 
who has given that particular topic greater thought than the Dublin Core 
Metadata Initiative.  They have conferences and workshops on the topic.

That being said, I wouldn't have a problem with drawing a distinction 
between the definitions the DCMI have come with and the serialization 
syntax that the RSS-DEV Working Group came up for these terms.  Perhaps 
it would make sense to standardize an RFC 3339 version of these dates. 
We can also discuss whether such dates should be in the core or in an 
extension.

Another key element of a number of proposals is a modification or update 
date.  We need to face the very distinct possibility that a number of 
producers will not distinguish between the two.  In fact, some producers 
may not be able to consistently identify that a modification occurred at 
all, let alone when it occurred.  Again, the solution (if this is deemed 
vital to be included in Atom 1.0) may be to identify the consequences of 
various usage patterns for this element.

- Sam Ruby

[1] http://elchem.kaist.ac.kr/vt/chem-ed/data/graphics/acc-prec.gif
[2] http://www.imc.org/atom-syntax/mail-archive/msg07850.html
[3] http://www.imc.org/atom-syntax/mail-archive/msg08419.html
[4] http://www.imc.org/atom-syntax/mail-archive/msg08396.html
[5] http://www.pepysdiary.com/

---

From: Antone Roundy <[email-masked]>
Date: Sat, 14 Aug 2004 13:52:58 -0600

On Saturday, August 14, 2004, at 11:24  AM, Sam Ruby wrote:
> bullseye, was accuracy.[1]  Consistently hitting the same place, even 
> if that place was yards away from the target, is precision.  The 
Here's my thinking on what Sam wrote: we need to decide 1) what the 
target(s) is/are that we want to hit, and 2) how to increase our 
accuracy in hitting that/those target(s).

Here are some possible targets for use of date constructs, some of 
which may not be worth the trouble:

1) "Correct" or at least acceptable sort order (there are a number of 
possibilities for what a user might consider the best sort order)
2) Meaningful display date
3) Signaling that processing needs to be done ("something has changed 
since you saw this entry last")

I think those are listed in order of importance, and that we can 
probably devise a better method for addressing #3 (for example, <id 
version="3">uri...</id>).

So how can we be the most accurate on #1, and can we also be accurate 
on #2?  I don't think it's possible to do both with a single date 
construct.

I'd imagine most users are usually going to want to sort based on 
either the initial publishing date, or the most recent significant 
update.  There are likely also to be cases where people want to sort 
based on the "display date", though those would be more rare.  Can we 
accommodate more than one of these, and how accurately?

> How does one resolve this interoperability issue?  Simply to state 
> that this value MAY change.  Dare agrees.[3]  Note that this is not a 
> weak restriction[4] on producers, but effectively a warning to 
> consumers.
Okay, I see now how this is intended as a warning to consumers.  I'll 
admit that I scanned the example in your proposal pretty quickly, and 
thus missed that point.  I think making the point at least briefly 
before the example would be helpful.  If a small number of implementers 
read specs, the number who read examples in specs carefully is going to 
be even smaller, so depending entirely on them to make important points 
is likely to lead to misunderstanding.

> Other such warnings can increase the probability that the date 
> indicated more often closely approximates when the entry became 
> available than any other date.  An extreme example to illustrate the 
> point:  The Diary of Samuel Pepys[5], republished in blog form 340 
> years later.  How can we discourage the "first date of formal 
> issuance" of these entries being placed into the atom:d element 
> defined in PaceDateSamRuby?  It would seem to me that a note that 
> "Consumers MAY chose to sort based on this value." would discourage 
> such usage.
Explicitly providing a place to put a "display date" would be another, 
and would make the point more directly.  If we DON'T provide a place 
for a display date, I'd like to see spec text more explicitly pointing 
out the implications of putting the "first date of formal issuance" 
there in such cases.

> seems like it would be a pretty effective deterrent.
Again, a "display date" would be another possibility.

> Another key element of a number of proposals is a modification or 
> update date.  We need to face the very distinct possibility that a 
> number of producers will not distinguish between the two.  In fact, 
> some producers may not be able to consistently identify that a 
> modification occurred at all, let alone when it occurred.  Again, the 
> solution (if this is deemed vital to be included in Atom 1.0) may be 
> to identify the consequences of various usage patterns for this 
> element.
There's definitely some distance between what is possible, at least 
with existing systems, and what is desirable.  I'd say knowing the date 
of first issuance and the date of most recent "significant" update 
(where "significant" is subjectively determined by the publisher) and a 
"display date" (ie., a date associated with the content or meaning of 
the entry) would be desirable.  That would enable all the methods of 
sorting I would expect to be most preferred, and the placing of an 
entry in temporal context by displaying a date associated with its 
content.  However, getting all three dates is clearly not always 
possible.

I have a difficult time letting go of the desire for a "display date", 
because if we don't support it explicitly, I can't think of a way to 
express it in a feed other than merging it into the content 
element--except by creating a new extension.  But if the consensus is 
that we can drop it, I'll conceed.

What would be the consequences of having a date with one meaning appear 
in a date construct spec'd for a date with a different meaning?  For 
example, what if we have atom:updated (meaning the last "significatnt" 
update), but somebody puts the date of first issuance (of the ENTRY, 
not the entry's content) in it?  Will that cause us problems in hitting 
our target (a good sort order) accurately?  What if we have atom:issued 
(date of first issuance of the entry) and somebody puts the last 
significant update date in it?  Would it be better to spec a date 
construct with an imprecise meaning ("it could be first issuance, last 
major change, last change of any kind, etc.") so that it is less likely 
to be "wrong"?  I'm afraid that if we relax the meaning of the date 
construct too much, the value of processing it "correctly" will be 
diminished.  I'd rather keep the definition somewhat tight to increase 
the probability that it's meaning will be consistent across feeds, and 
then accept that fact that some people aren't going to follow the spec 
perfectly.  Without thinking too hard about the consequences, I don't 
think they'd be too terrible in most cases.

I'd prefer to spec the date construct to indicate the last significant 
update.  I'd think that would best accommodate:

1) Feeds that never get significant updates (they never have to change 
it)
2) Feeds that DO get significant updates (they CAN change it)
3) People who want to sort by the first issuance date (they can cache 
the first date they see in an entry, and ignore changes to it)
4) People who want to sort by the last significant update date 
(obviously)

People who do make significant updates, but for whatever reason can't 
update the date wouldn't be perfectly served, but I'd say that's their 
own problem--if it causes too much pain, they can improve their 
publishing tool or move to a better one.

People who can't provide any kind of "objective" date are going to 
cause reduced accuracy no matter what solution we pick, unless we 
loosen the spec to make the target difficult to miss.  Since that would 
weaken the format for everyone else, I don't think we should go too far 
just to enable them to claim to be accurate and feel good about 
themselves.

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 16 Aug 2004 00:53:00 -0400

Sam Ruby wrote:
> Contemplate, for a moment, realtime publishing. ...
> consider the constraints of a user polling once an hour 
> or so.  In such a system, knowing precisely in which point in a 
	Realtime publishing[1] is what you get from PubSub.com if you use
our XMPP based service[2] and you are writing to a blog that pings us. The
time delay between when you publish an entry and when we deliver it to
subscribers can be on the order of just a few seconds...
	We should not be using the old-style RSS polling model as the basis
for designing fundamental components of the Atom format or protocol. If
anything, we should be assuming that publishing will be instantaneous -- the
format should work well even if publishing is orders of magnitude faster
than anything we have today. Let's not design in the constraints of old
technology...

		bob wyman

[1] Or, "near realtime" in order to keep the "realtime" purists happy...
[2] http://pubsub.com/developers.php, Note: Soon, we'll be announcing other
news aggregators that provide support for XMPP delivery of Atom entries in
near realtime.