Subject: why is atom:id globally unique?

From: "Michael Stillwell" <[email-masked]>
Date: Fri, 20 Aug 2004 15:40:41 +0100 (BST)

Why does atom:id need to be globally unique?  Maybe I'm missing
something, but I can't imagine any application actually operating under
the assumption that atom:ids were globally unique.  (If an aggregator did
have complete trust in atom:id, I'd be able to publish "updates" to
anyone's blog entry.)  How is this field expected to be used?

Michael

---

From: Henry Story <[email-masked]>
Date: Fri, 20 Aug 2004 16:54:51 +0200

I imagine these could be very useful if you think of moving entries 
into a peer to peer network environment, where for example everybody 
could copy everyone's entries. One would then have a completely 
distributed system where it would be very difficult for anyone to 
destroy the content of the blog entries.

Otherwise it is very helpful if someone writes an entry that is a reply 
to someone else's entry. If you look at the entry 
entry.2004-08-13-1632.n3 [1] in n3 notation
you can see that it is an entry that was written in reply to the 
tag:bblfish.net/20040813/1445/blog1#version1 entry of the blog example 
I have recently developed[2].
In this example I have the entry be in reply to an entry version, which 
I think captures the reality of replying to an entry better.

Further down in the file the location of that id is specified. This 
allows entries to
still refer to what they were in reply to even if a blog by mis-fortune 
gets reorganized. A fully peer-to-peer environment would just be an 
extreme example of such reorganization.

Henry

[1] http://bblfish.net/work/atom-owl/2004-08-12/entry.2004-08-13-1632.n3
[2] http://bblfish.net/work/atom-owl/2004-08-12/blogexample.html

---

From: "Michael Stillwell" <[email-masked]>
Date: Fri, 20 Aug 2004 16:24:01 +0100 (BST)

> destroy the content of the blog entries.

I wasn't thinking of a situation like this.  I was thinking more of a    
                                        case where a service
(technorati?) has a database of all the blog                             
                   entries in the world, where the primary key of each
entry is atom:id,                                             and this is
the only means of identifying entries.  If this is the                   
                            case then the database can easily be poisoned
by a feed pretending to                                             be an
"update."  Because of this, the service would need to use                
                                  something like $FEED_URL + atom:id as
the primary key--in which case                                           
  atom:id only needs to be feed-unique, not globally unique.  (Desktop
aggregators would have the same problem, though it wouldn't be quite so
acute.)

--M.

-- 
http://beebo.org

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 20 Aug 2004 08:19:23 -0700 (PDT)

> anyone's blog entry.)  How is this field expected to

Aggregators are the primary customers for a globally
unique ID. See the post at
http://www.imc.org/atom-syntax/mail-archive/msg07874.html
which describes some of the problems RSS has because
of its lack of globally unique IDs. 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #25
No matter how well it would perform, I will never construct any sort of machinery which is completely indestructible except for one small and virtually inaccessible vulnerable spot.

__________________________________
Do you Yahoo!?
Yahoo! Mail - Helps protect you from nasty viruses.
http://promotions.yahoo.com/new_mail

---

From: "Michael Stillwell" <[email-masked]>
Date: Fri, 20 Aug 2004 16:49:56 +0100 (BST)

> of its lack of globally unique IDs.

Thanks for the link.  What does an aggregator do if I publish a feed and
you publish a feed, and both contain an entry with the same atom:id, but
with entirely different content?  If the entries really are the same (in
the case of (3)) then it doesn't matter.  But if they're different you
have a problem: which one do you throw away?

--M.

-- 
http://beebo.org

---

From: Henry Story <[email-masked]>
Date: Fri, 20 Aug 2004 18:06:16 +0200

That is a good question. Perhaps one should have an extra signature 
that relates the
id with the content and the foaf:Person object. I know nearly nothing 
about digital signatures, so I'd be interested to hear what someone who 
knew on this subject had to say.

Henry

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 20 Aug 2004 12:24:08 -0400

Michael Stillwell wrote:
> Why does atom:id need to be globally unique?
	As Dare points out, the primary motivation is to support the needs
of aggregators that require a means to do cross-feed duplicate detection.
(i.e. if you publish the same item in multiple feeds, we should be able to
recognize the duplication.)
	Unfortunately, as I've pointed out in an earlier message[1], while
the goal is a good one, the mechanism is useless. Given Atom as currently
defined, no respectable aggregator could take advantage of the reputed
"global uniqueness" of atom:ids. Even if you find precisely the same atom:id
in two different feeds, you would only be able to take that as a "hint" that
there might be some similarity between two entries. The similarity in global
ids could be the result of:
	1. A failure to properly understand or implement the spec
	2. A configuration error in an otherwise properly implemented system
	3. An explicitly malicious action. (i.e. spoofing)
	Given that any cross-feed, false-positive matches between atom:ids
can result in data-loss, implementations should not assume perform such
matches. 
	Aggregators should *only* consider atom:ids in single-feed contexts.

	The only way that we could *trust* that cross-feed ids had any
relationship to each other is if we had either:
	1. Mechanisms for doing digital signatures. (Something that we
*really, really* should have anyway...)
	2. Some means by which a-priori trusted statements could be made
about common ownership of feeds. (Note: while this might be useful for
individual aggregators, you still couldn't trust any statements made by an
intermediary aggregator that generated synthetic feeds...)

	The only really good reason to ensure today that atom:ids are
globally unique is in order to anticipate that one day in the future we
will, in fact, add mechanisms to Atom that will allow useful cross-feed
comparison of atom:ids. This is a good thing. The cost of making atom:id
globally unique today is very limited -- even if generating globally unique
ids is not immediately useful. The cost of changing the style and culture of
id generation in the future might be very, very high... So, do it now.
Hopefully, some day it will be useful.

		bob wyman

[1] For my earlier message on spoofing see:
http://www.imc.org/atom-syntax/mail-archive/msg08308.html

---

From: "Roger B." <[email-masked]>
Date: Fri, 20 Aug 2004 11:43:22 -0500

> I was thinking more of a
> entries in the world...

Michael: That's a valid concern, but there's probably not a lot that Atom
1.0 can do about it. Sadly, atom:id spam (and rss:link/guid spam, for that
matter) will be inevitable once the audience reaches critical mass.

It won't be a huge problem for users with desktop aggregators, because they
have to knowingly subscribe to a feed, and can easily unsub at any time.
Server-based aggregators and datastores will be vulnerable, though, and will
find themselves struggling with the same stuff that Google and Company have
endured for years.

The ultimate solution might look a little like a distributed version of
DMoz... thousands of small, focused, and trusted aggregators generating
synthetic feeds that are in turn picked up by 2006's version of Technorati
or Feedster. Instead of letting entries from individual feeds into the
database, everything could be forced through intermediate filters. 

After all, it's a lot easier for one person running a "How To Grow Chiles"
aggregator to police fifty individual feeds than it is for a few dozen
people at a centralized service to keep an eye on that same fifty out of a
pool of millions.

--
Roger Benningfield
JournURL: http://journurl.com/
blog: http://admin.support.journurl.com/

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 20 Aug 2004 09:56:37 -0700

+1

My use case for atom:id is detecting changes between items in the same 
feed, not cross-feed. The security problems Bob points out are very 
real, and I think it highly unlikely that there will be enough 
co-ordination between feeds that aren't published by the same entity, 
or tool support (think about it; how would people use this, 
day-to-day?) to make this scenario practical.

I can see people using heuristics to figure out if feeds are talking 
about the same thing. I can see people coming up with a separate 
extension to do the same thing explicitly. I don't see why it's 
necessary to tie these things to the identity of an entry or feed 
itself.

--
Mark Nottingham     http://www.mnot.net/

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 20 Aug 2004 10:19:00 -0700 (PDT)

> have a problem: which one do you throw away?

The same thing it does if you are subscribed to
multiple RSS feeds where the same entry shows up twice
(e.g. subscribed to http://blogs.msdn.com &
http://blogs.msdn.com/dareobasanjo). No aggregator I
know off will discard one entry. However many users
have expressed the desire to want to be able to mark
items as read if they appear in multiple feeds and are
read once. 

Of course, an implementation may then decide to
discard items with the same atom:id as an
implementation optimization but in my mind that could
lead to problems one of which you've pointed out. 

This problems rears itself even more when you are
subscribed to search feeds like those generated by
PubSub.com or Feedster. 

=====

_______________________________
Do you Yahoo!?
Win 1 of 4,000 free domain names from Yahoo! Enter now.
http://promotions.yahoo.com/goldrush

---

From: Henry Story <[email-masked]>
Date: Fri, 20 Aug 2004 19:37:52 +0200

I think one should focus in the spec not on what will happen in case of 
error or fraud,
but on the mechanics of a well working system. You have to work within 
the legal system which is that you are innocent until proven guilty. 
The structure should be such that security may at any time be added 
orthogonally.

Imagine what a nightmare it would have been if the inventors of tcp/ip 
had also worried about packet fraud. They designed a system on which 
security could be layered. We should do the same. In fact we should 
design this so extensibility is built into the foundation.

Henry

On 20 Aug 2004, at 19:19, Dare Obasanjo wrote:

> No matter how well it would perform, I will never construct any sort 
> of machinery which is completely indestructible except for one small 
> and virtually inaccessible vulnerable spot.
>
>
>

---

From: Walter Underwood <[email-masked]>
Date: Fri, 20 Aug 2004 11:09:13 -0700

--On Friday, August 20, 2004 7:37 PM +0200 Henry Story <[email-masked]> wrote:
> error or fraud, but on the mechanics of a well working system.

With the Spam Army out there, hostile feeds are guaranteed to
show up. A vulnerable system is probably not a working system.

Trusted feeds would be a very clear reason to switch to Atom.

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Robert Sayre <[email-masked]>
Date: Fri, 20 Aug 2004 14:27:09 -0400

Mark Nottingham wrote:

> co-ordination between feeds that aren't published by the same entity, or 
> tool support (think about it; how would people use this, day-to-day?) to 
> make this scenario practical.

You don't think it's good enough to solve the "synthetic feed" problem? 
I'm sick of dealing with duplicate content from, say, Planet Apache.

One of my email clients deals with all the CCing on this list by 
consolidating via Message-ID headers. Seems to work pretty well, but I'm 
sure it's vulnerable.

All of the problems Bob lists will certainly happen, but the solution 
does not need to be 100% effective to make things suck less.

> necessary to tie these things to the identity of an entry or feed itself.
> 

Why not match on ID first, then start the heuristics?

This seems to be an issue for search engines, and not very different 
from the problems they already face. If a synthetic feed is spewing spam 
at me... I'll unsubscribe. It'll work great when I'm getting duplicate 
content from multiple trusted sources.

Robert Sayre

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Fri, 20 Aug 2004 11:53:52 -0700

At 7:37 PM +0200 8/20/04, Henry Story wrote:
>I think one should focus in the spec not on what will happen in case 
>of error or fraud,
>but on the mechanics of a well working system. You have to work 
>within the legal system which is that you are innocent until proven 
>guilty.

ARMCHAIR LAWYER ALERT!

We are talking about a standard that will be used in literally 
thousands of different legal jurisdictions, with at least that many 
different definitions of "error", "fraud", "innocent", and "guilty". 
(Hint: most of those jurisdictions don't use English as their primary 
language.)

We can't fix that; trying to do so is insane. We just shrug and give 
people a standard that says "MUST this" and "MUST that" and let them 
decide locally what to do with it.

>Imagine what a nightmare it would have been if the inventors of 
>tcp/ip had also worried about packet fraud.

They did. And they were smart enough to label it "out of scope".

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Henry Story <[email-masked]>
Date: Fri, 20 Aug 2004 21:14:21 +0200

> (Hint: most of those jurisdictions don't use English as their primary 

I know. I live in France where I am told you are guilty until proven 
innocent.
My point was methodological: when developing a system don't spend all 
your energy imagining up all the ways it could be misused. Concentrate 
on designing the system flexibly so that security can later be layered 
properly.

In another e-mail I even suggested that one could build a signature 
from the content, the Agent and the id. The building of such solutions 
are out of scope, but the architecture has to support the later 
building of them.

> They did. And they were smart enough to label it "out of scope".

That is what I was suggesting we do: label it out of scope, but make 
sure that the basic architecture is not so inflexibly designed as to 
make it impossible to add later.
Hence the importance again of *extensibility*.

Henry Story
http://bblfish.net/

---

From: DJWS <[email-masked]>
Date: Fri, 20 Aug 2004 22:55:44 +0200

At 19:19 20/08/2004, Dare Obasanjo wrote:
>No aggregator I know off will discard one entry. However many users have 
>expressed the desire to want to be able to mark items as read if they 
>appear in multiple feeds and are read once.

Let assume we are in a professional real world. The first demand is to get 
an organized thematic globally unique identification. The need being to 
organize the presentation to remove the duplicates and to identify the real 
source and to be able to exchange with it. If this is not provided by a 
globally unique atom:id, another unique ID must be provided which permits 
to authenticate the signatoree or request more information.

Questions:
1. let call a "guide" a globally unique id with exchange possibility. Can 
it be confirmed that an atom:guide definition is out of scope?
2. in such a case it should be possible to use externally defined atom:id 
where the id semantic would have a meaning to those using it as a "guide" 
and transparent to others. Is this a good place for it? Is there any 
objection to this?

jfc

---

From: Bill de hÓra <[email-masked]>
Date: Fri, 20 Aug 2004 22:23:05 +0100

Mark Nottingham wrote:

> feed, not cross-feed. 

0

My use cases are for both, but primarily the latter. They live 
outside of the scope of blogging/aggregating tools and the problem 
is important enough to ensure a generator and process that will emit 
stable IDs.

> The security problems Bob points out are very 

The argument above leads to the inevitability of using the http: 
scheme for id generation - despite the problems others rightly point 
out, a DPH will find no better bang for the buck. We may coming full 
circle.

[I do this loop roughly every six months in my job.]

> necessary to tie these things to the identity of an entry or feed itself.

Bob's says they're useless; I would would say they're more useful 
than not having them. Except in very specific circumstances, dirty 
searches and composite keys are a last resort. Data quality is out 
of scope but leverage to enable quality is not.

cheers
Bill

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 20 Aug 2004 19:46:10 -0400

Bill de hÓra wrote:
> Bob [Wyman] says they're useless; I would say they're more
> useful than not having them.
	Perhaps a test case would help clarify things. Given the "story"
below, how is atom:id useful in this specific cross-feed example?

	One day, a guy named Farnsworth creates a feed like the following
(very abbreviated...) one:
<feed>
  <author>Farnsworth</author>
  <entry>
    <id>GUID1234</id>
    <content>Today, I invented 3D television!</content>
  <entry>
</feed>

	The folk at "Invention Theft, Inc." use PubSub.com and discover
Farnsworth's feed only seconds after it is published since Farnsworth's CMS
pings http://xping.pubsub.com/ping . They then create the following feed
which, unfortunately, is found by some other blog search or matching service
*before* that service finds Farnsworth's earlier entry. (Different services
will scan feeds or receive updates in different sequences...)
<feed>
  <author>Invention Theft, Inc.</author>
  <entry>
    <id>GUID1234</id>
    <content>Today, I invented 3D television!</content>
  <entry>
</feed>

	What should the second blog search engine do when it processes
Farnsworth's feed? Should it pass the entry through or should they reject it
as an obvious duplicate? If lawyers inspect that database 10 years from now,
to whom will they give credit for inventing the 3D television?
	If atom:id is "useful" in cross-feed situations, it seems logical
that it would be useful in a case where two entries are byte-for-byte
identical.

	I continue to believe that atom:id, as currently defined, is useless
in cross-feed contexts. Nonetheless, as stated before, I do think that we
should require globally unique atom:ids since I think they are cheaply
created and *may* be useful in the future. The spec, however, should
probably warn against any cross-feed comparison of atom:ids.

		bob wyman

---

From: Bill de hÓra <[email-masked]>
Date: Sat, 21 Aug 2004 02:16:11 +0100

Bob Wyman wrote:

> 	Perhaps a test case would help clarify things. 

> Given the "story"

What should you do if this third feed arrives:

<feed>
   <author>Invention Theft, Inc.</author>
   <entry>
     <id>GUID1234</id>
     <content>
        Today, I invented 3D television!
     </content>
   <entry>
</feed>

is it the same as one of the other feeds? How do you *know*?

> in cross-feed contexts. 

That's funny, I'm finding atom:id useful in cross feed contexts. But 
I'm not using Atom for blogging in that case. I'm also mandating 
hierarchal URIs to reduce the chance of clashes - it's not asking 
much to do this, much less than asking for some kind of a bitwise 
comparision on content. You've shown one example no doubt derived 
from real feed data in PubSub; but you haven't show as a result it's 
a 'useless' mechanism.

> Nonetheless, as stated before, I do think that we
> created and *may* be useful in the future. 

They're useful right now.

> The spec, however, should

I disagree, that's a data quality issue. All the spec has to do is 
provide a means to enable data quality.

cheers
Bill

---

From: Phil McCluskey <[email-masked]>
Date: Sat, 21 Aug 2004 12:21:44 +1000

> probably warn against any cross-feed comparison of atom:ids.

If looked at in isolation, the atom:id is probably not useful, but in
the context of both the entry and the link(s), and the uri of the feed
I think it could be usefully used by aggregator authors.

Where the atom:id, the entry text and the link(s) are common, you
could safely assume that the entry has been duplicated across feeds
(as in Dare's scenario above), and mark both entries as read when one
of them is.  I think that's probably more useful behavior for the end
user than discarding an entry that has the same id.

Where the entries and the atom:ids are identical but the links are
different, you could prepend the additional links to both entries as
supplementary.

Where the the atom:ids and links are identical but the entries are
different, it could be possible to thread the additional entry against
each original as if they were part of the same conversation.

---

From: Eric Scheid <[email-masked]>
Date: Sat, 21 Aug 2004 16:24:05 +1000

> My use case for atom:id is detecting changes between items in the same

One of my clients has multiple feeds, one for each section of their site.
They also have a feed for the front page of their site. Not every story they
post ends up on the front page, just the more interesting ones.

Currently, anyone subscribed to the front page feed and one (or more) of the
section feeds will be confronted with cross-feed duplicates. Quite tiresome,
really, I know because I'm subscribed to all their feeds.

e.