Subject: Identity conundrum

From: Norman Walsh <[email-masked]>
Date: Fri, 30 Jul 2004 14:08:13 -0400

I thought of this while flying home from RDU last night. I haven't
caught up on the mail, so apologies in advance if it's been said
before.

I was thinking about what Dare said about his aggregator and the fact
that if it sees a "republished" entry with the same atom:id, it
continues to sort by the original date, not the new date. (I'm not
suggesting that Dare is alone or that he's wrong, it just happens to
be his message that got me thinking along these lines.)

It seems to me that we have conflicting ideas about what identity
means and that it might help to look directly at that issue, so here
goes.

One idea of identity is that entries have immutable identity. That is,
an atom:id identifies an atom:entry that cannot change. Publishing a
different atom:entry with the same atom:id is logically an error. In
this view, an aggregator that notices an atom:entry with the same
atom:id as one it's already seen can disregard it without another
thought. It's either exactly the same as the one that's already been
seen, or it has changes that are erroneous and it can recover from
this error by ignoring the new entry.

Another view of identity is that entries are snapshots of an
abstraction that has identity. That is, an atom:id identifies an
abstraction that might or might not change. Publishing a different
atom:entry with the same atom:id represents some sort of change in the
underlying abstraction that's being reflected in the Atom world by
posting a new atom:entry. In this view, an aggregator that notices an
atom:entry with the same atom:id as one it's already seen might choose
to merge them, or pick one, or pick the one with the most recent
atom:modified date, or do something to "harmonize" the two entries.
The only thing it would never do is present both the atom:entries to
the user as if they were two different entries.

Choosing the former view suggests that Tim's idea of "a permanent
status page" for a project (like GenX or my own sxpipe) is an abuse of
Atom unless the atom:id is changed everytime the status page is
updated.

Choosing the latter view suggests that Dare's aggregator (and other
aggregators built along the same lines) should be changed. Exactly
what they should do probably need more discussion.

I think I lean towards the latter view, but there's definitely
something appealing about the simplicity of the former view.

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <[email-masked]> | On the other hand, you have different
http://nwalsh.com/            | fingers.

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 30 Jul 2004 11:34:24 -0700 (PDT)

> what they should do probably need more discussion.

RSS Bandit already does what you suggest in the latter
view. I use the most recent version of the content
(title/description/summary/content) I just don't
change the date because to me the issued date should
not change. 

I think you've misinterpreted the discussion and
created two points of view that don't mesh with
reality. 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #25
No matter how well it would perform, I will never construct any sort of machinery which is completely indestructible except for one small and virtually inaccessible vulnerable spot.

__________________________________
Do you Yahoo!?
Y! Messenger - Communicate in real time. Download now. 
http://messenger.yahoo.com

---

From: Norman Walsh <[email-masked]>
Date: Fri, 30 Jul 2004 16:30:30 -0400

/ Dare Obasanjo <[email-masked]> was heard to say:
| I think you've misinterpreted the discussion and
| created two points of view that don't mesh with
| reality. 

Ah well. Best to ignore me then. :-)

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <[email-masked]> | If you run after wit you will succeed
http://nwalsh.com/            | in catching folly.-- Montesquieu

---

From: Elias Torres <[email-masked]>
Date: Sat, 31 Jul 2004 10:09:44 -0400

Although I'm not the god of aggregators, like Dare, I think you did
capture in essence the two major point of views in identity.

I also stand with you on the latter. Every change to an entry, makes
it a new entry. Except that when using LSID the identifier just
changes by a version attribute in the LSID.

urn:torrez.us:2004-blog:entry1 -- First Published
urn:torrez.us:2004-blog:entry1:2 -- Typo Fixed or GenX status changed.

So if you compare the two, they are different unique identifiers, but
we can go a step further and from the LSID spec, we can tell that they
both point to the same concept, just different versions of it.

Anyways, it seems that this is very controversial and most likely the
spec will stay the way it is and hopefully some extra wording for
instructing people. At least it accommodates every publisher, but will
make consumers' life more complicated.

Regards,

Elias

---

From: Sam Ruby <[email-masked]>
Date: Sat, 31 Jul 2004 11:41:56 -0400

> urn:torrez.us:2004-blog:entry1:2 -- Typo Fixed or GenX status changed.

Going back to Norm's original email:

http://www.imc.org/atom-syntax/mail-archive/msg08148.html

     One idea of identity is that entries have immutable identity. That
     is, an atom:id identifies an atom:entry that cannot change...

     Another view of identity is that entries are snapshots of an
     abstraction that has identity. That is, an atom:id identifies an
     abstraction that might or might not change.

It seems to me that Elias, your notions correspond to the former.  If 
so, this reinforces Norm's observation that there are two points of view.

- Sam Ruby

---

From: James M Snell <[email-masked]>
Date: Sat, 31 Jul 2004 10:26:22 -0600

> so, this reinforces Norm's observation that there are two points of 
view.

This has the potential of going even further down a rabbit hole without 
any chance of gaining consensus.  Yes, there are two points of view: 1) a 
single identity regardless of version and 2) a unique identity for each 
version.  Which is the correct view?  Well, both are valid and, IMHO, Atom 
shouldn't have to care which view a publisher decides to adhere to.  I'd 
say that items in an Atom feed MUST have a single unique URI.  A person 
subscribing to the first option will have an atom feed in which a single 
item may change over time.  A person subscribing to the second option will 
have an atom feed that contains a new item for each version of the 
content.

Option 1:
  <feed>
    <entry>
      <id>urn:myentry</id>
      <content>Hello</content>
    </entry>
  </feed>

  <feed>
    <entry>
      <id>urn:myentry</id>
      <content>Hello There</content>
    </entry>
  </feed>

Option 2:

  <feed>
    <entry>
      <id>urn:myentry:2</id>
      <content>Hello There</content>
    </entry>
    <entry>
      <id>urn:myentry:1</id>
      <content>Hello</content>
    </entry>
  </feed>

Unless we either: choose to allow both points of view or force one side to 
compromise, we're not going to achieve consensus on this issue.

> - Sam Ruby

- James M Snell
  [email-masked]
  http://www.ibm.com
  (877) 511-5082 / Office
  930-1979 / Tie Line

---

From: Bill de hÓra <[email-masked]>
Date: Sat, 31 Jul 2004 17:32:50 +0100

> something appealing about the simplicity of the former view.

Neither view is simple. And people tend to want to have both views.

cheers
Bill

---

From: Ken MacLeod <[email-masked]>
Date: 31 Jul 2004 12:12:13 -0500

James M Snell <[email-masked]> writes:

> [Sam Ruby] wrote on 07/31/2004 08:41:56 AM:
> 
> >      One idea of identity is that entries have immutable
> >      identity. That is, an atom:id identifies an atom:entry that
> >      cannot change...
> > 
> >      abstraction that has identity. That is, an atom:id identifies
> >      an abstraction that might or might not change.
> > 
> > It seems to me that Elias, your notions correspond to the former.
> > If so, this reinforces Norm's observation that there are two
> > points of view.
> > 
> 

> compromise, we're not going to achieve consensus on this issue.

There is no reason why both models cannot be satisfied with the
current format.

The crux of the issue appears to be a combination of what <id> is
identifying and whether or not an entry (or resource in general) can
have more than one identifier.

The current model, and that of RSS 1.0 and RSS 2.0, is Norm's latter:
<id> identifies the abstraction of "an entry that may have multiple
versions".  It's not really an abstraction, because aggregators
concretely replace all or portions of their previous contents of the
entry when a new entry with the same <id> arrives.

There *may* exist *another* identifier that can identify a specific
instance or version of "an entry that may have multiple versions".
Atom does not specify where this other identifier is stored, and given
all of the possible means for associating these other identifiers with
"replaces" or "updates", etc., it makes sense for this other
identifier and its corresponding relations to be specified in a
"module" or "extension".  This works exceedingly well over the core
Atom model because the clients not interested in "managed versions"
see only streams of new versions of the same entry.

There is a lesser issue of whether "the identifier" can be "parsed" so
that it can serve both purposes simultaneously, both the identifier
for "an entry that may have multiple versions" and an instance or
version of that entry.  Since only a small number of URI schemes have
explicit support for versioning, it would be preferable for the format
to have two separate locations for the two different identifiers.

Option 3:

  <feed>
    <entry>
      <id>urn:myentry</id>
      <version:id>urn:myentry:2</version:id>
      <content>Hello There</content>
    </entry>
    <entry>
      <id>urn:myentry</id>
      <version:id>urn:myentry:1</version:id>
      <content>Hello</content>
    </entry>
  </feed>

  -- Ken

---

From: Ken MacLeod <[email-masked]>
Date: 31 Jul 2004 12:21:24 -0500

> Neither view is simple. And people tend to want to have both views.

We can do both.

Please take a moment to look at the longer message I just sent.  I
wanted to call attention to it because even though it is longer it
deserves more than a quick skim.  It's a summary of longer discussion
that Asbjørn, Arve, others, and I have had on #atom specifically
addressing this issue.

  -- Ken

---

From: Antone Roundy <[email-masked]>
Date: Sat, 31 Jul 2004 11:38:32 -0600

On Saturday, July 31, 2004, at 10:26  AM, James M Snell wrote:
> This has the potential of going even further down a rabbit hole 
> without any chance of gaining consensus.  Yes, there are two points of 
> view: 1) a single identity regardless of version and 2) a unique 
> identity for each version.  Which is the correct view?  Well, both are 
> valid and, IMHO, Atom shouldn't have to care which view a publisher 
> decides to adhere to.  I'd say that items in an Atom feed MUST have a 
> single unique URI.  A person subscribing to the first option will have 
> an atom feed in which a single item may change over time.  A person 
> subscribing to the second option will have an atom feed that contains 
> a new item for each version of the content.
>
> Option 1:
>   <feed>
>     <entry>
>       <id>urn:myentry</id>
>       <content>Hello</content>
>     </entry>
>   </feed>
>
>   <feed>
>     <entry>
>       <id>urn:myentry</id>

I don't particularly like the idea of overloading the id element with 
both identity and versioning.  If we were to, for example, mandate that 
anything following the trailing colon in an id is a version number, no 
one would be able to use ids with colons in them without including a 
version number--which would take us closer to mandating a single 
scheme.  Another option would be something like <id 
version="2">urn:myentry</id>.  Yet another would be <id 
extension-ns:version="2">urn:myentry</d>.  The problem with either of 
those would be that the id would not be unique (ignoring @*:version), 
which in the case of @atom:version would require everyone to support 
it, even though it seems likely that most people won't be interested in 
doing so.

As you might infer, my preference/POV is that an entry keeps the same 
identity, and the same ID, regardless of version.  This matches the 
current usage of feeds.  People could always tack some kind of 
versioning on to the id, since it wouldn't exactly violate the spec, 
but I would HOPE that it wouldn't become common enough practice that 
aggregators would be expected to support it by not displaying entries 
who's id's version had been bumped.

---

From: Ken MacLeod <[email-masked]>
Date: 31 Jul 2004 12:57:53 -0500

>       <version:id>urn:myentry:1</version:id>

A smaller, follow-on issue.  I checked format-01 for the spec text
that indicates that a feed can only contain one entry with the same
<id>, and notice it doesn't say that (I thought I recalled that it
did).  The spec does say clearly that "same <id> is same entry".

However, to support both models, the spec should clarify:

    Multiple instances or versions of the same entry MAY appear in a
    feed, index, or archive.  Consumers MAY choose to present or
    record only the instance or version of the entry with the most
    recent [[date that indicates change]].

  -- Ken

---

From: James M Snell <[email-masked]>
Date: Sat, 31 Jul 2004 12:45:31 -0600

> > This has the potential of going even further down a rabbit hole 
without 
> There is no reason why both models cannot be satisfied with the

Agreed

>       <version:id>urn:myentry:1</version:id>

This is close. I was thinking about something along these lines:

  <feed>
    <entry>
      <id>urn:myentry2</id>
      <supercedes>urn:myentry1</supercedes>
      <revision>1</revision>
      <content>Hello There</content>
    </entry>
    <entry>
      <id>urn:myentry1</id>
      <revision>1</revision>
      <content>Hello</content>
    </entry>
  </feed>

This assumes every distinct entry in the feed MUST have a unique 
<atom:id>.

This introduces two new elements, <atom:supercedes> and <atom:revision>.

<atom:supercedes> references the ID of another item which this entry 
supercedes/replaces
<atom:revision> specifies an unsigned integer value specifying the 
revision number of this entry.  Every edit to the entry requires the 
revision number to be incremented.  <atom:revision> is not intended to 
provide full featured versioning functions... just the ability to specify 
how many times the entry has been modified.  Any change in content or 
metadata would count as a "revision".

Both elements would be optional. 

This approach would allow us to cleanly support both models.

Again, I know there are folks who would *rather* see us *only* support the 
model where an entry can keep the same identity regardless of changes; and 
there are folks who would *rather* see us *only* support the model where 
each change to an entry requires a change in the id.  This proposal seeks 
to find a compromise between the two by allowing us to support both in a 
rather simple way.

- James M Snell
  [email-masked]
  http://www.ibm.com
  (877) 511-5082 / Office
  930-1979 / Tie Line

---

From: Bill de hÓra <[email-masked]>
Date: Sat, 31 Jul 2004 19:57:57 +0100

> instance or version of "an entry that may have multiple versions".

That would be a tuple that serves to identify (correlate) a stream 
(aka a delayed list) of entries - it's a set/bag membership 
function. If this is what we want I suggest we start with the HTTP 
resource/representation model until it's obviously inadequate.

It's not just another identifier, it's another model of identity 
altogether. See below.

> see only streams of new versions of the same entry.

Fair enough.

> to have two separate locations for the two different identifiers.

Look, the thing that is needed for this identity stuff is a 
evaluation model ('semantics) that ranges over the set of 
identifiers or tuples of indetifiers or whatever -  this will trash 
need functions like equality or order, invariants, and so on so that 
we can implement the neccessary evaluators. Pointers to the literal 
values or tuples of literal values are not sufficient.

cheers
Bill

---

From: Bill de hÓra <[email-masked]>
Date: Sat, 31 Jul 2004 20:05:54 +0100

Bill de hÓra wrote:

> Look, the thing that is needed for this identity stuff is a evaluation 
> model ('semantics) that ranges over the set of identifiers or tuples of 
> indetifiers or whatever -  this will trash need functions like equality 
> or order, invariants, and so on so that we can implement the neccessary 
> evaluators. Pointers to the literal values or tuples of literal values 
> are not sufficient.

Good grief. In English this time:

Look, the thing that is required for any useful notion of identity 
is an evaluation model (a 'semantics') which ranges over the set of 
identifiers (or tuples of identifiers, or whatnot).  This will 
clearly define functions such as equality or order, invariants, and 
so on, so that we can implement the necessary evaluators in code and 
ask important questions like "are these two things the same thing?", 
or "are these things part of that thing?". Pointers to the literal 
values or tuples of literal values are not sufficient in themselves.

cheers
Bill

---

From: Ken MacLeod <[email-masked]>
Date: 31 Jul 2004 15:14:52 -0500

James M Snell <[email-masked]> writes:

> [Ken MacLeod] wrote on 07/31/2004 10:12:13 AM:

> [copied to http://intertwingly.net/wiki/pie/PaceMinimalEntryVersioning]

> <atom:supercedes> references the ID of another item which this entry

The list would have to include all previous versions that this entry
replaces, in the case where the "window of polling" the publisher
misses an "entry instance" in the middle, eg. offline readers.

> Again, I know there are folks who would *rather* see us *only*
> support the model where an entry can keep the same identity
> regardless of changes; and there are folks who would *rather* see us
> *only* support the model where each change to an entry requires a
> change in the id.  This proposal seeks to find a compromise between
> the two by allowing us to support both in a rather simple way.

As Graham put it as an aside once[1],

    (Also the idea that it's meant to be applied to different versions
    of the same entry isn't conveyed very well - some conceptual
    definition of an "entry" elsewhere might do that though)

Your definition of an entry above seems to be more precisely, "entry
instance", and the identifier is of that instance.  My definition, and
current practice (right or wrong), is that entry is, merging Norm and
my definitions, "the abstraction of an entry that has multiple
versions".

So an even simpler form (and I'm not proposing this, because it's
simply changing names for the sake of names), would be:

  <feed>
    <entry-instance>
      <id>urn:myentry:2</id>
      <is-version-of>urn:myentry</is-version-of>
      <content>Hello There</content>
    </entry-instance>
    <entry-instance>
      <id>urn:myentry:1</id>
      <is-version-of>urn:myentry</is-version-of>
      <content>Hello</content>
    </entry-instance>
  </feed>

So all we've done here is changed <entry> to <entry-instance>, and
moved the identifier of "the abstraction of an entry that has multiple
versions" from <id> (where it currently resides in practice) to the
destination URI of <is-version-of>.  A "new" <id> is now what I called
the <version:id> in my examples.

Current aggregators would therefore do a search-and-replace for
atom:entry and change it to atom:entry-instance, change code
references from atom:id to atom:is-version-of, and continue to work
the way they have.  They may ignore <id>, and any form of versioning
model is still left open for further specification (as it would need
to be anyway, versioning is a huge topic in and of itself, and I
suggest outside the scope of core Atom).

Note, this approach doesn't suffer the "window of polling" problem.

I would not want to make this change to the Atom format because:

  1) it is only *slightly* more precise in its terms,

  2) it contains two unique identifiers where we have a hard enough
     problem explaning one, and

  3) we don't need to do it, we can leave the "entry instance"
     identifier to another specification that can more easily apply
     its context above the Atom core context *and* be fully baked.

  -- Ken

[1] http://imc.org/atom-syntax/mail-archive/msg08004.html

---

From: Elias Torres <[email-masked]>
Date: Sat, 31 Jul 2004 22:27:30 -0400

Right, Sam. I think I'm starting to see the issue more clearly and I
accept that the Atom spec must enable people to use it for both cases.
Now, let me just mention a similar issue that we've seen in LSID
use-cases in the field.

LSIDs when resolved contain two different parts of information: data
and metadata. Data must never change for a given LSID (including
version information). On the other hand, metadata has fewer
restrictions. Metadata can be modified as desired by the owner of the
information. One could attach metadata to the main LSID or to a
specific version, etc, etc. Why is this such a neat feature? Because,
we've already have had to solve the same issues we are having in Atom
in LSID deployments.

Basically, people either want to name data and they want to name metadata. 

Let me connect this to Norm's description of the problem. The idea of
an entry being immutable is the same as when the user wants to name a
picture, an output of algorithm, a movie, etc.  The owner always wants
the same data returned when resolving that LSID. So the data returned
by the LSID will always be the raw bytes and these should never
change. Now, the idea that entries are snapshots of an abstraction
that has identity, we think of it as a concept. What we do is simply
not attach data to the LSID, but only metadata. Since metadata can
always change, users can make changes to any field they might have
added to an entry at their leisure, such as title, description,
modified, etc.

In the case you've read this far, then I pose the following question?
What is an entry? Is it data or metadata? Is it both?

How can we help BOTH publishers and consumers understand what the
author intended with this entry? I see further in this thread, people
are starting to make the right suggestions (not that I personally like
the element name entry-instance). I would love to hear people's
opinions on the subject and if there's anything we could recommend
including  as part of the spec. I really think this will help us make
great progress in a few issues, such as dates, versioning and
identification.

Thanks Norm for making the issue clearer to see, so we stop discussing
URIs vs. URNs and HTTP vs. Non-HTTP and focus on the real needs people
have out there and how can Atom solve some of them.

- Elias Torres

---

From: Elias Torres <[email-masked]>
Date: Sat, 31 Jul 2004 22:44:27 -0400

+1 

Ken,

I'm starting to like
http://intertwingly.net/wiki/pie/PaceMinimalEntryVersioning as a
possible solution to my original point of moving away from HTTP URIs
and using much better defined schemes for identification, not location
of objects on the network.

Since, it seems now that votes need to backed up with technical
reasons, here they are: I recommended LSIDs because the IDs themselves
had a well-defined structure, that I find extremely important to avoid
all the super technical discussions on URI comparisons like the ones
MarkP and Tim are having. Basically, I see in this pace a
"well-defined" structure spread over a few atom elements (maybe we can
group into a construct called atom:id :-)). We have an ID (URI), a
version and a date. LSID has these as well, just in different places.
The ID is obvious, the version is optional and the date is usually
found in the metadata. I like the date as part of the ID, because it
gives us a nice way to sort versions for latest w/o having to resolve
for metadata. In LSID, the version can be any format the owner likes,
but that's makes it impossible for people to sort on. I think NewsML
was better in that regard.

Elias Torres