Subject: ID - wrong solution to the problem?

From: Ian Davis <[email-masked]>
Date: Thu, 26 Aug 2004 21:10:00 +0100

It seems to me that there's a lot of energy going into solving a 
relatively minor issue: that of duplicate entries. There may be other 
ways to solve the problem.

1. Some way to declare that a feed is an aggregate or filtered version 
of other feeds? List the other feeds so clients can check if the user is 
subcribed to any of those. If so, duplicates should be expected. These 
can be detected by comparing links, titles and descriptions as is done 
in RSS at the moment. The client has some hints about where to look so 
the scale of the problem is reduced.

2. A way to declare that the feed will never contain duplicates. This 
can be true in many situations - most weblogs dont't repeat themselves 
and lots of publishing tools can be configured not to show an entry in 
the feed again if it has just been edited. This would also be suitable 
for event driven feeds such as CVS checkins or log files. In fact I 
believe there are many many feeds that can guarantee not to repeat 
entries, barring system errors.

3. A way to include a version number in an entry. Clients could decide 
not to show an entry if it's not the first version. Publishers would 
change the version number if they change the content of the entry.

Ian

---

From: Antone Roundy <[email-masked]>
Date: Thu, 26 Aug 2004 15:17:39 -0600

> ways to solve the problem.
While it's good to think out of the box, this sounds like a lot more 
trouble than minting unique IDs.

> of other feeds? List the other feeds so clients can check if the user 
> is subcribed to any of those. If so, duplicates should be expected. 
> These can be detected by comparing links, titles and descriptions as 
> is done in RSS at the moment. The client has some hints about where to 
> look so the scale of the problem is reduced.
Aside form this being a lot more work and more prone to error, what do 
you do if a feed is an aggregate or filtered version of a number of 
aggregate or filtered feeds?  You could get to the point where the list 
of source feeds is longer than the feed itself.

> entries, barring system errors.
So what happens if you post an entry, and then before it has made its 
way off the end of the feed, you edit it?  Do you leave the data in the 
feed as originally posted?  Or do you modify the data in the feed?  If 
you do, how should the fact that it's the same entry be detected?  If 
comparing titles, links, and descriptions were fool-proof, IDs may 
never have been suggested at all.

> change the version number if they change the content of the entry.
What if the first version of the entry was posted and then modified 
before you saw it?  You'd miss it completely.

---

From: Dare Obasanjo <[email-masked]>
Date: Thu, 26 Aug 2004 16:30:02 -0700 (PDT)

> ways to solve the problem.

All the solutions you propose are more brittle than
using IDs. In fact on rereading them you don't really
propose any solutions. More importantly there is
consensus on using URIs as IDs. It seems we are down
to Mark and Sam who are still arguing for
canonicalization instead of following the example set
by W3C specs like XML namespaces and RDF. 

I think we should just vote on this and move on. 

> the scale of the problem is reduced.

How exactly is it done in RSS at the moment? As far as
I am aware, it isn't. Perhaps one of the other
aggregator authors can correct me. 

> entries, barring system errors.

The problem isn't knowing which feeds don't have
duplicates but actually dealing with duplicates in the
ones that do.

> change the version number if they change the content

How does this help exactly? How exactly does this
prevent seeing duplicate posts if you are subscribed
to a Feedster feed or reading my blog from both
http://blogs.msdn.com &
http://blogs.msdn.com/dareobasanjo 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #23
I will keep a special cache of low-tech weapons and train my troops in their use. That way -- even if the heroes manage to neutralize my power generator and/or render the standard-issue energy weapons useless -- my troops will not be overrun by a handful of savages armed with spears and rocks.

__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - Send 10MB messages!
http://promotions.yahoo.com/new_mail

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 00:36:14 +0100

> trouble than minting unique IDs.
Several thousand messages discussing globally unique for all time 
identifiers sure seems like a lot of trouble to me. (As an aside, can 
someone give a good example of a globally unique for all time identifier 
  in the real world?)

The subtext of my original message was: how important is the duplicate 
entry problem compared to getting a clean Atom core out of the door?

> of source feeds is longer than the feed itself.

I don't agree that it's more prone to error than the current ID 
proposals. Enumerating the list of aggregated feeds is a publisher 
problem. Comparing IDs is a client problem. There are a lot more clients 
than publishers.

Yes, it's more work for the publisher. But not a lot more work since 
most aggregation is automated and the aggregator has to maintain a list 
of source feeds anyway.

I think this information would be useful to clients. If the user adds a 
subscription to a new feed, the client could inform them that they are 
already subscribed via another aggregate feed.

> comparing titles, links, and descriptions were fool-proof, IDs may never 
> have been suggested at all.

If you're declaring that your feed never contains duplicates then you 
ensure that it doesn't by not blindly including edited entries. If you 
can't ensure that then don't make the declaration.

> before you saw it?  You'd miss it completely.
Yes. Which happens all the time unless you poll insanely fast. It is 
very common to get gaps in feeds. Perhaps while you sleep or go for a 
walk you switch your computer off and it misses a few items. No big deal.

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 01:18:37 +0100

> I think we should just vote on this and move on. 

I've come to the conclusion that requiring every entry to have a 
globally unique ID is just too onerous for the benefit it provides.

How important is the duplicates problem? Is it more annoying than 
getting two copies of this message because I've CC'd the list? Will you 
delete one and get on with your life or complain to your email software 
provider?

It's important to get a handle on the importance of the problem because 
the proposed solution is IMHO going to act a a barrier to innovation 
around Atom. To be clear, the current solution is going to require that 
*every* entry in *every* feed contains a globally and temporally unique 
ID. That includes feeds that consist of CVS checkins, logging 
notifications, editorial diffs to the Atom draft, stock prices, weather 
reports, school closure notices, traffic reports, feed validator 
results, search engine results, POP3 mailbox listings.

Why should a publisher be required to generated an ID for an entry that 
exists only in the feed only for as long as there are no more than x 
more recent entries, e.g. a web server error log?

Fundamentally, the constraints on ID are unenforceable. How is a 
validator going to check whether the IDs in my feed are globally and 
temporally unique? Will it keep a list of every ID ever seen? If so, how 
is it to detect that two different entries that share the same ID are in 
fact the same? This is exactly the same problem you currently face with RSS.

If a validator can't check it then I can be lazy and just hard code my 
IDs in my feed http://example.com/1 through http://example.com/15. You 
can't point at my feed and say it's breaking the RFC because you have to 
  compare against the history, which only exists in archive.org and 
google.com. When your users complain that your aggregator doesn't show 
new items in my feed then what are you going to do?

IDs *will* be reused. I guarantee it. Being the wrong thing to do 
doesn't stop it from happening.

People will change hosting providers and assign new IDs across their 
entire history. They're still globally unique and have never been used 
before. I could even argue that they're new items, since they're now 
part of a new website. These will appear as duplicates to your customers 
and you can't prevent that happening.

Aggregators will aggregate non-Atom content such as RSS and republish as 
Atom. Where will the ID come from? A new one will have to be generated. 
If the original source also happened to have a parallel Atom feed then 
your customers will see the aggregate entries as duplicates and once 
again you can't prevent that. In fact, even if the original source 
doesn't have an Atom feed you're probably going to see duplicates since 
the aggregator will not be able to determine whether two RSS items are 
the same by looking at the link, title and description. This is too hard 
a problem as you have pointed out. Therefore each item will get a new 
Atom ID. More duplicates.

I'm going to speculate here and suggest that a large proportion of 
current duplicates in feeds are generated by linkbloggers. You know the 
type, they quote an entry and add maybe a tiny comment at the end. 
Essentially it's a duplicate of the original. ID is not going to solve 
this either.

Which all leaves me wondering just how useful *is* Atom's ID?

Ian

---

From: Mark Pilgrim <[email-masked]>
Date: Thu, 26 Aug 2004 20:35:41 -0400

> I think we should just vote on this and move on.

Yeah, since that worked so well the last time.

-- 
Cheers,
-Mark

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 26 Aug 2004 18:03:11 -0700

>globally unique ID is just too onerous for the benefit it provides.

Sorry, I haven't seen where in these threads you have shown how it is 
onerous. Many people (including me) have shown how easy it is. So, 
even if the benefit is small or limited to one type of user, you need 
to show how onerous it really is (and thus how those of us who show 
that it is easy are wrong), not how little you consider the price.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 02:19:50 +0100

On 27/08/2004 02:03, Paul Hoffman / IMC wrote:
> onerous. Many people (including me) have shown how easy it is. So, even 
> if the benefit is small or limited to one type of user, you need to show 
> how onerous it really is (and thus how those of us who show that it is 
> easy are wrong), not how little you consider the price.

In my previous message I listed a set of potential feed types where I 
believe generation of an ID is onerous. The reasoning being that these 
feeds consists either of transient information (traffic reports etc) or 
large numbers of tiny entries (CVS checkins etc) or both (search engine 
results, editorial diffs to the Atom draft etc).

Producing IDs for these kinds of feeds wastes resource on the producing 
side because the producer knows that they are never duplicated and that 
they will exist only for a short period of time. This is exacerbated if 
canonicalisation is required.

It wastes resources on the consumer side because these IDs have to be 
stored and checked against every other entry the client ever sees. 
Again, this may mean convoluted URI comparisons.

Perhaps onerous is too strong a word but I certainly feel that requiring 
a globally and temporally unique ID for everything is overkill. That's 
why I'm quetioning the basis for inclusion of ID in the first place. I 
don't think it solves the problem it was intended to solve and it places 
additional responsibilities on both producer and consumer.

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 02:26:51 +0100

Another issue with ID is denial of service.

If I want to suppress a competitor's feed then I just pump my feed full 
of bogus entries using their ID generation scheme. If my feed is fetched 
first then the client will mark my competitors entries as duplicates of 
mine. Just to be sure, I'll make sure the dates are set to some time in 
the past so the bogus entries don't display in the users newsreaders.

Ian

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 26 Aug 2004 21:31:07 -0400

Ian Davis wrote:
> If the user adds a subscription to a new feed, the client could
> inform them that they are already subscribed via another aggregate feed.
	What you suggest could only happen if aggregate feeds contained all
the content of the feeds that they aggregate. However, many feeds are
composed of subsets of feeds. Given two aggregate feeds, even if they have
the same source feeds, it could make sense to subscribe to both since they
may contain non-overlapping subsets of their source feeds.

> Enumerating the list of aggregated feeds is a publisher problem. ...
> There are a lot more clients than publishers.
	PubSub.com aggregates over three million feeds. If we were to
include the list of three million (and growing) in every feed we generate,
this "publisher problem" would turn into a "client problem" very quickly.

> most aggregation is automated and the aggregator has to maintain a
> list of source feeds anyway.
	You seem to speak with some authority here. Can you point us to the
aggregator that you maintain and from which you have learned how easy it is
to build aggregators?

		bob wyman

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 26 Aug 2004 21:39:20 -0400

> Which all leaves me wondering just how useful *is* Atom's ID?
	As defined, the atom:id is only useful in the context of a single
feed. While it is required to be globally unique, no mechanisms are provided
to ensure global uniqueness or verification of uniqueness... 
	People are trying to figure out, without much luck, how to make
atom:id useful in a cross-feed or multi-feed context. This is because the
problem of duplicates is a *very* major one for many users -- even if you
don't see it that way. The debates here are about what is hoped for; we've
yet to deal with reality.

		bob wyman

---

From: Dare Obasanjo <[email-masked]>
Date: Thu, 26 Aug 2004 18:46:47 -0700 (PDT)

> results, editorial diffs to the Atom draft etc).

You still haven't explained how this is onerous. The
amount of code it takes to create a new unique
identifier in C# is basically 

 string identifier = uriScheme + myDomainName +
DateTime.Now.ToString() + Guid.NewGuid().ToString() 

How the heck is that onerous? You've just claimed that
you think these entries are not worth having unique
identifers. 
 
> canonicalisation is required.

Again, I await your explanation as to how the one line
of code above 'wastes resources' on a web server. Is
it running on a 4MHz CPU? 

> Again, this may mean convoluted URI comparisons.

So? I'm an author of an RSS aggregator and I don't
think it is an onerous burden to have to track or
compare IDs.  In fact my users have requested features
and reported a number of issues that are dependent on
having IDs that actually work in the syndication
format. 

Every aggregator author I know of feels the same way.
So exactly who are you speaking for? 

> additional responsibilities on both producer and

As one of person who's actually an aggregator author
instead of playing one on TV, I beg to differ. 

=====

__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail

---

From: Dare Obasanjo <[email-masked]>
Date: Thu, 26 Aug 2004 18:49:06 -0700 (PDT)

> the past so the bogus entries don't display in the

Why do people keep trotting out this hypothetical
problem. If this is such a threat why don't you
demonstrate this in RSS 2.0 by creating a hostile feed
that duplicates the <guid> and <link> elements of some
website then see how much of a malicious hack this is.

I personally have no idea what the impact would be but
I'd rather see some proof instead of theoretical
problems being raised which although possible today
haven't manifested themselves. 

=====

		
__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - 100MB free storage!
http://promotions.yahoo.com/new_mail

---

From: Sam Ruby <[email-masked]>
Date: Thu, 26 Aug 2004 22:06:29 -0400

> Again, this may mean convoluted URI comparisons.

How are these requirements any more onerous than the requirements that 
every item in RSS 1.0 has a mandatory rdf:about attribute? [1]

      {item_uri} must be unique with respect to any other rdf:about
      attributes in the RSS document and is a URI which identifies the
      item.

- Sam Ruby

[1] http://web.resource.org/rss/1.0/spec#s5.5

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 26 Aug 2004 19:05:02 -0700

At 2:19 AM +0100 8/27/04, Ian Davis wrote:
>>Sorry, I haven't seen where in these threads you have shown how it 
>>is onerous. Many people (including me) have shown how easy it is. 
>>So, even if the benefit is small or limited to one type of user, 
>>you need to show how onerous it really is (and thus how those of us 
>>who show that it is easy are wrong), not how little you consider 
>>the price.
>
>In my previous message I listed a set of potential feed types where 
>I believe generation of an ID is onerous. The reasoning being that 
>these feeds consists either of transient information (traffic 
>reports etc) or large numbers of tiny entries (CVS checkins etc) or 
>both (search engine results, editorial diffs to the Atom draft etc).

Are you saying it is onerous to create 
http://hostname/date-down-to-subseconds/hash-of-content for those?

>Producing IDs for these kinds of feeds wastes resource on the 
>producing side because the producer knows that they are never 
>duplicated and that they will exist only for a short period of time. 
>This is exacerbated if canonicalisation is required.

"Waste of resources" is not the same as "onerous". If the creator is 
sure that they will exist only for a short period of time and never 
be duplicated, http://hostname/date-down-to-subseconds is even easier.

>It wastes resources on the consumer side because these IDs have to 
>be stored and checked against every other entry the client ever 
>sees. Again, this may mean convoluted URI comparisons.

The client doesn't have to store and/or compare them if it doesn't 
want to. There is no MUST language for that.

>Perhaps onerous is too strong a word

Agree.

>  but I certainly feel that requiring a globally and temporally 
>unique ID for everything is overkill.

OK, that's different.

>  That's why I'm quetioning the basis for inclusion of ID in the 
>first place. I don't think it solves the problem it was intended to 
>solve and it places additional responsibilities on both producer and 
>consumer.

A number of folks on the list have given a few different reasons why 
they really want it, so saying it doesn't solve "the problem it was 
intended to solve" isn't really the issue. If you feel that it 
doesn't solve the problems that folks have listed, that's fine, but 
you need to show that.

--Paul Hoffman, Director

---

From: Antone Roundy <[email-masked]>
Date: Thu, 26 Aug 2004 20:10:11 -0600

On Thursday, August 26, 2004, at 07:26  PM, Ian Davis wrote:
> If I want to suppress a competitor's feed then I just pump my feed 
> full of bogus entries using their ID generation scheme. If my feed is 
> fetched first then the client will mark my competitors entries as 
> duplicates of mine. Just to be sure, I'll make sure the dates are set 
> to some time in the past so the bogus entries don't display in the 
> users newsreaders.
A solution is to check for duplicate IDs first, and then do some kind 
of comparison between any entries that do have the same ID (though you 
shouldn't need to bother doing this if the cached entries are from the 
same feed--in that case, you can trust that the publisher either isn't 
reusing IDs or will get enough complaints about his non-spec-compliant 
feed that he'll quickly quit reusing IDs).  This would be a MUCH easier 
way to weed out duplicates than checking against EVERY other entry 
you've seen.  Given that duplicates ARE a big issue (I don't have 
references at hand, but this has been said numerous times by aggregator 
developers here), having IDs certainly sounds better than not having 
them.

> Again, this may mean convoluted URI comparisons.
Not if we mandate that IDs be comparable charter-by-character.  Sure, 
some broken publishing tools will get this wrong, but they should be 
educated quickly by their users complaining about duplicates.

<speculation>I wouldn't think you'd necessarily have to store every ID 
you've ever seen.  If you store them for some period of time, and drop 
them as they get stale, you shouldn't run into too much trouble with 
really old entries resurfacing.</speculation>

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 26 Aug 2004 23:00:18 -0400

Antone Roundy wrote:
> A solution is to check for duplicate IDs first, and then do some
> kind of comparison between any entries that do have the same ID
	Would you mind describing what this "some kind of comparison" might
be? 
	What comparison can you do that will allow you to detect a malicious
attempt to replace an existing entry? 
	If such a comparison can be defined and shown to work, this whole
issue would simply evaporate. 

		bob wyman

---

From: Graham <[email-masked]>
Date: Thu, 26 Aug 2004 19:51:31 -0700

You've missed out the main use of IDs, which is matching up current 
entries in the feed with the entries from the last time you loaded the 
feed, so you can carry over metadata (eg whether the user has looked at 
it). It is not possible to do this by "comparing links, titles and 
descriptions".

Graham

---

From: Antone Roundy <[email-masked]>
Date: Thu, 26 Aug 2004 23:14:08 -0600

> issue would simply evaporate.
I didn't have anything detailed in mind, and I don't imagine it would 
be foolproof--just better than throwing IDs out altogether and 
depending ENTIRELY on "some kind of comparison" for duplicate 
detection.  Off the top of my head, pseudocode might look something 
like this:

@existingEntries=lookForIDInDataStore($thisEntry->id);
if (!count(@existingEntries)) ProcessNewEntry($thisEntry);
else {
	$idOldEntry=0;
	foreach $oldEntry (@existingEntries) {
		if (
			($oldEntry->SourceFeed() == $thisEntry->SourceFeed()) ||
			(entrySimilarity($oldEntry, $thisEntry) > 0.8)
		) {
			$isOldEntry=1;
			last;
		}
	}
	if ($isOldEntry) UpdateOldEntry($thisEntry);
	else ProcessNewEntry($thisEntry);
}

function entrySimilarity($e1, $e2) {
	$elementCount=0;
	$similarCount=0;
	foreach $element (@someListOfElements) {
		if (strlen($e1->elementValue($element)) || 
strlen($e1->elementValue($element)) $elementCount++;		if 
(!strcmp($e1->elementValue($element), $e2->elementValue($element))) 
$similarCount++;
			/* some more complex comparison routine that could computer a degree 
of similarity, especially for longer elements like <content> would be 
useful */
	}
	return $elementCount ? ($similarCount/$elementCount) : 0;
}

Perhaps particular similarities or differences would be considered more 
significant than others.  For example, if the domain portion of the 
alternate link is different, that would look more like someone trying 
to grab someone else's traffic than a changed title.  Of course, this 
is turning into wild speculation that would have to be tested against 
real attempts to hijack somebody else's feed, and it's starting to feel 
like sort of like spam filtering.  The point, in the context of the 
discussion, is that most of the extra comparison could be skipped in 
the vast majority of cases by comparing IDs first and moving on if no 
match is found.

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 09:49:23 +0100

> aggregator that you maintain and from which you have learned how easy it is
Do I have to provide credentials to contribute to this group?

myRSS.com is my current activity. theweb.startshere.net was the first 
back in 1999: 
http://web.archive.org/web/19991111061935/http://theweb.startshere.net/

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 09:53:04 +0100

> website then see how much of a malicious hack this is.

It's just as theoretical as the notion that unique IDs are going to 
prevent duplicates. Demonstrate that and then my scenario above becomes 
rather more concrete.

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 09:56:27 +0100

> every item in RSS 1.0 has a mandatory rdf:about attribute? [1]

No more onerous at all. I presume you are making this remark because you 
assume I think RSS 1.0 is the perfect soution. I suggest you check your 
facts first because not all of us agree with that particular restriction:

http://internetalchemy.org/2002/12/rss1Dot0IssuesList#issue3

I woud like to see it removed from RSS 1.0 and not mandatory in Atom.

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 10:05:33 +0100

Care to respond to the technical issues rather than my opinions? I'll 
summarise to make it easier:

How important is duplicate detection?
How many duplicates are due to aggregation rather than quotation?
How is the global and temporal uniqueness of IDs to be checked?
How will clients detect old entries that have been reissued IDs?
How will aggregates of non-Atom sources determine a suitable ID?

Am I missing something? Other than duplicate detection what are the use 
cases for ID?

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 09:59:34 +0100

> it). It is not possible to do this by "comparing links, titles and 

For this you just need a feed-unique ID not one that's unique for all 
time and all places.

Ian

---

From: "Laurent Le Meur" <[email-masked]>
Date: Fri, 27 Aug 2004 11:14:51 +0200

Maybe this issue should be looked from the processing perspective only.
Unique IDs allow for a smart processing, and they don't impose it. 

If Atom adopts the unique ID spec, then the processing model can be
specified as:
- if two IDs are identical (whatever the comparison process is), the
recipient (processor) SHOULD assume that the entries are identical (convey
the same meaning), are a/ replace any previously received instance of the
entry by the last instance or b/ keep both instances (for rollback / piracy
detection or other purpose), but only display/send/process the last
instance.

If versioning is adopted (as an option, with a revision ID and/or a modified
date-time convention), the processing model can also be specified as:
- if two IDs differ only by a revision id (or a modified date-time), the
recipient SHOULD assume that the entry with the higher revision ID (or date)
is a new version of the information, are a/ replace any previous revision of
the information by the last entry or b/ keep all revisions, but only
display/send/process the last revision by default (and display older
revisions when needed/requested by a user or administrator).

>sees. Again, this may mean convoluted URI comparisons. (Ian Davis)

When the ID is a database primary key, comparison (byte-per-byte) is a non
issue. Smart URI comparison is another matter, ok.

> I certainly feel that requiring a globally and temporally 
> unique ID for everything is overkill. (Ian Davis)
> results, editorial diffs to the Atom draft etc). (Ian Davis)

So what ... if people create new information with an existing ID, the new
information will supersede in all cases the previous information. All the
provider has to know is that this is the rule all syndication nodes will
follow.

> of bogus entries using their ID generation scheme.

An optional signature mechanism has been discussed on this list in order to
avoid this threat. High level Atom processor should implement this from the
start.

> How is a validator going to check whether the IDs in my feed are globally
and temporally unique? (Ian Davis)
> feed. While it is required to be globally unique, no mechanisms are
> provided to ensure global uniqueness or verification of uniqueness (Bob
Wyman)

No need to check/validate, as it is only a matter of processing logic.

> As an aside, can someone give a good example of a globally unique for all
time identifier in the real world? (Ian Davis)

NewsML URNs used by press agencies.

regards
Laurent Le Meur

-
		AVERTISSEMENT

Le présent mail et ses pièces jointes sont confidentiels et destinés au seul usage des personnes ou entités auxquelles ils sont adressés. Si vous avez reçu cet e-mail par erreur, veuillez contacter dans les plus brefs délais son expéditeur et effacer le contenu du message de votre système informatique. Toute divulgation, distribution ou copie de cet e-mail est strictement interdite.

-
		DISCLAIMER

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please contact the sender and delete the email from your system. If you are not the named addressee you should not disseminate, distribute or copy this email.

-

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 10:28:07 +0100

> you think these entries are not worth having unique

You need to own a domain name. Most producers of Atom won't and using 
the domain of the generating software has the issues I described in 
relation to the tag scheme.

.NET is blessed with RNGCryptoServiceProvider that can generate the 
required degree of randomness for a GUID. Good luck to all those 
producing Atom feeds via shell scripts, PHP or XSLT.

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 10:33:13 +0100

> http://hostname/date-down-to-subseconds/hash-of-content for those?
Yes. Show me how to get the current time in XSLT.

> intended to solve" isn't really the issue. If you feel that it doesn't 
> solve the problems that folks have listed, that's fine, but you need to 
> show that.
I disagree. Saying it doesn't solve "the problem it was intended to 
solve" *is* the issue. I believe you should be asking the ID proponents 
to demonstrate how it solves the problem that is is designed to address. 
I've provided several arguments to the contrary.

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 10:52:38 +0100

> Unique IDs allow for a smart processing, and they don't impose it. 

You are suggesting that publishers must ensure that all entries have IDs 
that are canonical, globally unique, have never been used before and 
will never be used again no matter what publishing mechanism they use on 
the off-chance that a client might want to compare it with another ID.

> issue. Smart URI comparison is another matter, ok.
As I suggested in another message, most clients will dump these into a 
database and index them. If that's the case, why enforce a URI with all 
the ambiguity that involved. Why not just require a GUID?

> NewsML URNs used by press agencies.

And even here there are ambiguities around canonicalisation and 
comparison. From RFC3085:

"URNs are lexically equivalent if the ProviderId, DateId, NewsItemId, 
and RevisionId are all identical (case-insensitive comparison)."

Are the following the same:

urn:newsml:iptc.org:20001006:NewsMLv1.0:1
urn:newsml:iptc.org:20001006:%4EewsMLv1.0:1

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 12:31:15 +0100

> this "publisher problem" would turn into a "client problem" very quickly.
Why are you contesting this when your own feeds already include the 
necessary structure. From 
http://atom.pubsub.com/c3/cd/7a625435c020f57b2536e64cc6.xml

<entry>
<title>OmniWeb 5.1 Wishlist &amp;#38; RSS News Reading</title>
<ps:source-feed>
   <title>Matt Henderson's Blog</title>
   <link rel="alternate" type="text/html" 
href="http://matt.makalumedia.com/"/>
   <link rel="alternate" type="text/xml" 
href="http://matt.makalumedia.com/index.rdf"/>
</ps:source-feed>

Combine this with a feed-unique ID and you may have a usable duplicate 
detection mechanism.

Ian

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 26 Aug 2004 23:00:18 -0400

bob wyman

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 09:49:23 +0100

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 09:53:04 +0100

Ian

---

From: Antone Roundy <[email-masked]>
Date: Thu, 26 Aug 2004 23:14:08 -0600

---

From: Robert Sayre <[email-masked]>
Date: Fri, 27 Aug 2004 08:25:14 -0400

> Yes. Show me how to get the current time in XSLT.

Pass a parameter to the XSLT Processor.

>> solve the problems that folks have listed, that's fine, but you need 
>> to show that.
> 

Duplication and identification are important. Usenet has IDs. Email has 
IDs. All that "references", and "in-reply-to" stuff sure is useless...

Neither of those technologies are perfect, but yes, IDs help.

 > I've provided several arguments to the contrary.

Yes, you've provided several arguments. They're not very convincing.

Robert Sayre

---

From: Henry Story <[email-masked]>
Date: Fri, 27 Aug 2004 14:59:24 +0200

> attempt to replace an existing entry?

You cannot use the argument that because something can maliciously be 
misused, that it therefore should not be used.

Everything can be misused. A toothbrush can be used to kill someone if 
properly wielded.

A question in return is: would it be impossible to secure this, by eg. 
using a signature?

Henry

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 27 Aug 2004 06:24:01 -0700 (PDT)

> A question in return is: would it be impossible to

Exactly. Every technology can be misused. Should
blogging web servers & HTML never have been invented
because there are hate sites on the Web? Should email
and USENET never have been invented because of spam?
Should operating systems that could connect to the
network never have been envisioned because machines
can be hacked? 

If you start playing this game, then there should be
no software industry because every significant class
of software can be abused. 

=====

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 27 Aug 2004 06:31:36 -0700 (PDT)

> prevent duplicates. Demonstrate that and then my

It's as theoretical as http://www.rssbandit.org 

I currently use links as unique IDs to prevent
duplicates among other things. The problem is that
links are not good enough for a variety of reasons
that have been discussed ad nauseum on this list.
Similarly a combination of link + entry data is not
good enough. 

Practically every aggregator author I am aware of does
something similar.  

=====

_______________________________
Do you Yahoo!?
Win 1 of 4,000 free domain names from Yahoo! Enter now.
http://promotions.yahoo.com/goldrush

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 14:48:44 +0100

> Practically every aggregator author I am aware of does

You still haven't demonstrated that IDs are going to solve the 
duplicates problem. Your response just reiterates what we already know - 
  it's hard to detect duplicates using the content the publisher supplies.

Ian

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 15:00:09 +0100

> of software can be abused. 

This form of argument will not cut it in the RFC process. The RFC will 
have to state the risks and describe how these are avoided or mitigated. 
I suggested a form of denial of service made possible by mandating 
producer generated unique IDs. We need to work out some method of 
mitigating the risk, e.g. "If the normal behaviour of the client is to 
hide duplicate entries then clients SHOULD report the number of 
duplicate entries found in a feed and SHOULD provide the user with a 
mechanism for displaying them. The client MAY display the conflicting 
entries from other feeds."

Ian

---

From: Robert Sayre <[email-masked]>
Date: Fri, 27 Aug 2004 10:07:08 -0400

>  it's hard to detect duplicates using the content the publisher supplies.

If you keep writing this stuff, you may succeed in putting us all to 
sleep. Here are some interesting articles:

How does duplicate detection work?[0]
How does Usenet handle News? [1]

Robert Sayre

[0] http://blogs.msdn.com/exchange/archive/2004/07/14/183132.aspx
[1] http://www.faqs.org/docs/linux_network/x-087-2-news.algorithm.html

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 27 Aug 2004 07:12:59 -0700 (PDT)

>   it's hard to detect duplicates using the content

My mistake, I thought it was painfully obvious. Here
goes 

Scenario 1: A client fetches a feed which contains 10
items on Monday. The client fetches the same feed on
Tuesday and it still has 10 items but 3 of them new
entries. How does the client tell which of the 10
fetched on Tuesday the user has seen and which ones
the user would consider as 'new'? 

How IDs Solve This Problem: The aggregator checks the
ID of items fetched on Tuesday with those on Monday.
That way it determines only 3 are new. EVERY
aggregator that distinguishes between new and
prevuiously seen items has code like this today. 

Scenario 2: Some items have relationships between them
such as the parent-child relationships between USENET
posts or comments in response to an item. How does the
aggregator know which entries are related? 

How IDs Solve This Problem:  The subordinate items can
reference the IDs of items they are related to. As has
been mentioned email and USENET work like this today
and given that there are Atom feeds of USENET and
mailing lists it seems Atom should support this
scenario. 

Scenario 3: Feeds containing results from multiple
feeds such as search engines (Feedster) need to be
able to determine when they are returning the same
result fetched from multiple feeds so they don't show
users duplicate entries. 

How IDs Solve This Problem: If an entry has a unique
ID across the multiple feeds it belongs in then
Feedster can use a combination of the ID + updated
date to determine the single entry to show the user.
There is a potential for malicious hijackings of sites
with this approach but the problem already exists in
RSS today and I've never actually heard of it
happening. 

Scenario 4: An entry can show up in multiple feeds at
once. For example entries from my work blog an be seen
in the feeds for  http://blogs.msdn.com/dareobasanjo,
http://blogs.msdn.com or http://weblogs.asp.net . It
is a conceivable for a user to be subscribed to both
the individual feeds and aggregated feeds in which
case they see duplicate entries [once in the aggregate
feed and once in the individual feed]. The main
problem being when they read an item in one place it
isn't marked as read in the other feed even though it
is the same item

How IDs Solve This Problem: If the item used the same
ID in all feeds it appeared in then the aggregator
could mark it as read in all feeds it appears in once
it is read in any feed. 

There are more but I'll let you chew on these for a
while. :) 

=====

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 15:11:31 +0100

> [0] http://blogs.msdn.com/exchange/archive/2004/07/14/183132.aspx
Thanks, very informative:

"We would have liked to do duplicate detection based solely on the 
Internet Message Id but there are several not-to-be named applications 
out there that use the same Internet Message Id on all their messages."

Hmmm.

Ian

---

From: Robert Sayre <[email-masked]>
Date: Fri, 27 Aug 2004 10:14:17 -0400

> out there that use the same Internet Message Id on all their messages."

Thanks for the out-of-context quote. There's no concept of a duplicate 
without identity. Welcome to my killfile.

Robert Sayre

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 27 Aug 2004 07:18:27 -0700 (PDT)

--- Ian Davis <[email-masked]> wrote:

http://blogs.msdn.com/exchange/archive/2004/07/14/183132.aspx
> out there that use the same Internet Message Id on

Your point? Morons will implement the spec
incorrectly. No matter what technique you come up with
there'll be some broken application that gets it
wrong. The question is how easy is it to get it wrong
and how hard is it to get right? 

Seriously, you are undermining your credibility to
more you continue to post. If you have nothing
constructive to add besides randomization and noise
perhaps you should sit out of this discussion. 

=====

__________________________________
Do you Yahoo!?
Yahoo! Mail Address AutoComplete - You start. We finish.
http://promotions.yahoo.com/new_mail

---

From: Ian Davis <[email-masked]>
Date: Fri, 27 Aug 2004 15:21:23 +0100

What about the problems here:

http://www.imc.org/atom-syntax/mail-archive/msg08986.html

Ian

---

From: Dare Obasanjo <[email-masked]>
Date: Fri, 27 Aug 2004 07:24:05 -0700 (PDT)

I have already responded to that thread. It seems you
are just time wasting troll. 

*kerplunk* - welcome to the killfile. 

There goes 30 minutes of my life I'm not getting back.

> >>You still haven't demonstrated that IDs are going
> to
> >>solve the 
> > I will keep a special cache of low-tech weapons
> and train my troops in their use. That way -- even
> if the heroes manage to neutralize my power
> generator and/or render the standard-issue energy
> weapons useless -- my troops will not be overrun by
> a handful of savages armed with spears and rocks.
> > 
> > 
> > 		

=====

---

From: Henry Story <[email-masked]>
Date: Fri, 27 Aug 2004 16:49:46 +0200

As a side note I have a Atom-OWL model of Atom at:
http://bblfish.net/work/atom-owl/2004-08-12/blogexample.html

I find the ID tag (represented as an EntryID class) very useful, and it 
works very
nicely with an EntryVersion class.  There is a very nice mapping to be 
made from a
simple Entry class to a EntryID that contains all the changes made to 
an entry over time.

Henry

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 27 Aug 2004 12:33:31 -0400

Ian Davis, on noticing that PubSub's feeds contain <ps:source-feed>
information says:
> necessary structure?
	Yes, PubSub.com Atom feeds contain <ps:source-feed> elements which,
when combined with atom:id, can be used to resolve the various atom:id
issues. However, ps:source-feed is a non-standard extension and has not been
received well by the list. (But, I think folk will eventually realize why it
is necessary...)
	When I discuss issues on this list, I try to stick to what is
documented in the drafts and various proposals that have been made. My hope
is we'll end up with an Atom specification that we can use *without*
extensions to fix bugs in the spec. Thus, when people say "atom:id is ok". I
am going to object -- atom:id is *NOT* sufficient, as defined, to do what is
needed.

> Combine this with a feed-unique ID and you may have a usable duplicate
	Yes. But it isn't Atom. It is "Atom + PubSub extensions". That is
not good.

		bob wyman

 -----Original Message-----
From: Ian Davis [mailto:[email-masked]] 
Sent: Friday, August 27, 2004 7:31 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: ID - wrong solution to the problem?

<entry>

Ian

---

From: Henry Story <[email-masked]>
Date: Fri, 27 Aug 2004 19:27:44 +0200

On 27 Aug 2004, at 18:33, Bob Wyman wrote:

> issues.

why is it important to know the source feed in order for the atom:id to 
work correctly?

Henry

> [snip]

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 27 Aug 2004 13:55:46 -0400

Henry Story wrote:
> why is it important to know the source feed in order for the atom:id
> to work correctly?

	A service like PubSub.com's, can use "source-feed" as a means to
verify that some uses of identical atom:id's in multiple feeds are
legitimate.

	Method 1: When an item is received that contains a source-feed
indicator that is not the same as the feed for which the item was read, the
aggregator can treat this as a "ping"[0] and rather than using the new entry
directly, it can scan or re-scan the indicated source-feed to extract the
original copy of the entry. As long as the entry is still in the
source-feed, things will work well. If the entry is not in the source-feed,
local policy would be used to decide if the entry should be discarded or
given a new atom:id.

	A similar, although less reliable, method can be used in the absence
of source-feed data. 

	Method 2: In this method, the aggregator would record all seen
atom:ids along with the source-feeds in which they were discovered. Then, if
another instance of an atom:id is found, the service would determine if the
current instance came from the same feed as the previous instance. If so,
then the new entry would be treated as a replacement for the old one. If
not, the service would re-fetch the feed in which the atom:id was previously
found and attempt to recover the original entry. If the original can be
recovered, it would be used. If not, then local policy would determine if
the more recent entry should be deleted, passed on as legitimate, or
rewritten with a new atom:id. (This method relies on the assumption that the
"authoritative" entry will always be read *before* any "spoof" or
"malicious" entries are read. This may not be a good assumption.)

	Both of these methods require a good bit of work by the aggregator.
While it is completely reasonable for a service like PubSub's to do this
stuff, it might be considered onerous for desktop clients due to the
increased latency in reporting new entries to users as well as the increased
processing requirements and network traffic loads. Nonetheless, the methods
above would allow a service like PubSub's to have greater confidence that
the correct data was being distributed.

	There remains an issue related to "attribution". Even if the
original source of an entry is retrieved, the aggregator probably must
remember the feed in which the entry's existence was discovered. The problem
is that you might have someone who is interested in "all entries from blog
'A'". In this case, even if the original source of an entry was Blog 'B',
you should still deliver a copy to the subscriber. We have no syntax at
present to allow the specification of "delivery via" or "alternate feeds"
(different from rel="alternate") in Atom. Thus, maintaining this data would
require private extensions to Atom.

		bob wyman

[0] "Pinging" is the process by which blog CMS software informs aggregators
that the blog has been updated. See: http://blo.gs , http://pingomatic.com/
etc.

---

From: Graham <[email-masked]>
Date: Fri, 27 Aug 2004 11:32:43 -0700

For God sake people, it's time to stop feeding the troll. We're having 
IDs that are globally unique URIs. There's no need to justify it to 
anyone again.

Graham

---

From: Henry Story <[email-masked]>
Date: Sat, 28 Aug 2004 11:41:24 +0200

> verify that some uses of identical atom:id's in multiple feeds are

I think there are many other ways to do this also.

First of all notice that if someone is maliciously changing id's they 
could also maliciously change the source-feed. That is then up to the 
justice system to stop
those people. We here have defined what is correct behavior on which 
the judicial system can work.

The most protective system would be one that adds a signature for an 
entry that
takes an entry id, a person, a version id, and signs the content 
proving that
all of these were generated together. Perhaps something for a future 
version of Atom.

Finally there are other methods that are based on your very good motto 
"It's the Entries, Stupid!" [1] I have taken that motto to heart in my 
latest N3 model [2].
There a feed is just a pointer to a bunch of entries

<>    a       :Feed ;
       :dynamic <> ;
       :entry  <entry.2004-08-13-1047.n3> , <entry.2004-08-13-1445.n3> ,
                <entry.2004-08-13-1752.n3> , <entry.2004-08-13-1632.n3> ;

with a little info to allow one to decide whether the entries have 
changed:

<entry.2004-08-13-1047.n3>
       :entry-version <tag:bblfish.net/20040813/1047/blog1#version1> ;
       :id     <tag:bblfish.net/20040813/1047/blog1> .

Each entry is a resource (REST), where we can find all the information 
about the entry
itself.

When feeds are first fetched by agregators, the location of the entries 
will in the overwhelming majority of cases not have changed position. 
so if entry.2004-08-13-1047.n3 appears on two of my feeds, because I 
have a "personal" feed, and a general feed. It will be clear to the 
aggregator that the content can be found in the same location, and so 
is the same. That entry file will be the place to look for the 
definitive id, in case of confusion.

Henry

[1] http://www.imc.org/atom-syntax/mail-archive/msg04596.html
[2] http://bblfish.net/work/atom-owl/2004-08-12/blogexample.html

---

From: "Bob Wyman" <[email-masked]>
Date: Sat, 28 Aug 2004 15:17:42 -0400

Henry Story wrote:
> First of all notice that if someone is maliciously changing
> id's they could also maliciously change the source-feed.
	It's not "changing" ids, it's "copying" or "spoofing" ids that's
the problem..
	In any case, the fact that people can copy an id and change the
source feed is one of the reasons that I contend that ID's should be
feed specific -- yet allowed to appear in multiple feeds. If you look
carefully at how we use the "ps:source-feed" element at PubSub.com,
you'll realize that whenever we "import" or "copy" an entry into an
aggregate feed, we copy it's source attribution. One can then use the
combination of id+source is to disambiguate or verify entries. I believe
this is the proper behavior and all processes that construct aggregate
feeds should do the same. Only what they should be doing is including
the feed's atom:head inside atom:entry, instead of ps:source-feed. Given
this practice, it becomes possible for us to do what I suggested in my
earlier mail: i.e. use the appearance of a previously unseen copied
element to trigger a rescan of the source feed. i.e. treat "foreign"
elements in a feed in the same way one would treat a "ping" message.
	Using the procedure above and as described in my earlier
message, it becomes possible to serve a number of needs and varying
levels of "security" concerns. For instance, if you trust a particular
aggregate feed author, you could simply accept that any imported entries
are faithful replications of their originals -- you can save time and
bandwidth by not tracking down the source. On the other hand, if you are
"paranoid" (as a service like PubSub should be), then you would go to
the extra trouble of verifying the entries prior to passing them on. 
	The mechanism I propose has the nice attribute of making it
unnecessary to do what has often been suggested: i.e. providing a means
to mark a feed as "original" or "aggregate." By marking each imported
entry with information about its source, the distinction between an
original entry and an imported entry makes it unnecessary to mark the
feed itself.

>I think there are many other ways to do this also.
	I would appreciate it if you could describe at least a few more
of the "many other ways" that you know of.

You propose:
> a feed is just a pointer to a bunch of entries...
> Each entry is a resource (REST), where we can 
> find all the information about the entry itself.
	This would work, technically, however, I don't think it is
likely to become accepted practice since it would result in quite a few
additional network roundtrips. The feed has the nice property of
gathering entries together so that they can be retrieved with an economy
of network operations. Admittedly, what I'm proposing in my "method" is
that services like PubSub actually do additional work to verify entries.
However, this work is optional and is not required of *all* processors
of feeds -- as would be the case in your proposal. 
	What you're proposing is a good "model" for how the system
should work, however, I'm afraid that a direct implementation of the
model isn't the right thing to do. The "extra-model" operation of
copying entries into the feed, even if they exist as resources on the
web, is necessary as a practicle means of making the reading of
collections of entries more convenient and less complex for the average
client developer.
	Or, have I not properly understood what you propose?

		bob wyman

---

From: Ian Davis <[email-masked]>
Date: Sat, 28 Aug 2004 20:41:56 +0100

On 28/08/2004 20:17, Bob Wyman wrote:
> the feed's atom:head inside atom:entry, instead of ps:source-feed. 

+1 to feed-local ids plus source attribution.

Ian

---

From: Henry Story <[email-masked]>
Date: Sun, 29 Aug 2004 01:01:34 +0200

> 	It's not "changing" ids, it's "copying" or "spoofing" ids that's

That still sounds malicious. If it is malicious then it is in the end a 
matter for the courts.

[Snip very interesting description of pubsub verification mechanism... 
I'd like to
get back to this after clarifying the following point.]

> additional network roundtrips.

I think on the contrary that the RESTful way I propose is going to be 
much more efficient network wise. I was convinced of this by Ken 
MacLeod during a long conversation on Atom irc a couple of months ago. 
He helped me rebuild my model quite dramatically. Here are some things 
to keep in mind.

	(1). With HTTP 1.1 Persistent connections only one tcp connection is 
needed to get all the data from a feed. If the client does not find 
some of the entries in its database it can request them in the same tcp 
connection.
As an example Imagine this is part of your feed file in N3 notation:

<>    a       :Feed ;
       :dynamic <> ;
       :entry  <entry.2004-08-13-1047.n3> , <entry.2004-08-13-1445.n3> ,
                <entry.2004-08-13-1752.n3> , <entry.2004-08-13-1632.n3>.
<entry.2004-08-13-1047.n3>
       :entry-version <tag:bblfish.net/20040813/1047/blog1#version2> ;
       :id     <tag:bblfish.net/20040813/1047/blog1> .

   The client fetches this feed and notices that 
entry.2004-08-13-1047.n3 is at version2. Having parsed this feed file 
the client can fetch the changed entry without having to open a new tcp 
connection.

	(2). The thing that gets polled often is the feed. So we want small 
feed files to minimize the number of packets sent over the internet. No 
need to send the whole entry text. The client will most likely already 
have those from the previous time he polled the feed.

	This is clearly achieved in the example above. All we really need to 
send in the feed is the entry resources and their id and version ids. 
(Additional information, such as the feed title can of course also be 
sent, but I just want to limit myself to what is necessary at first).

    (3) as explained in my previous e-mail this also helps you solve 
your problem of checking the id of an entry. Just download the entry 
itself. In most cases this should be at the location specified by the 
resource.

   (4) "It's the entry, Stupid!" [1]
	Entries should be first class citizens in atom. If they have http 
retrievable urls
then they can also be edited and updated at those urls using PUT, POST 
and DELETE.

> The feed has the nice property of

I think if you look at packets sent over the network you will find that 
the restful method I propose is in fact much better for the network, it 
allows caches on the internet to work properly, it reduces information 
transmission to what is necessary,...

> collections of entries more convenient and less complex for the average

I am going to be developing a client myself to do this, and I really 
don't think that this is going to be much extra work. Also I think one 
should try not to confuse architecture and implementation. If Tim 
Berner's lee had taken this into account, he would have tried to 
incorporate images directly into the html too, and we would not 
separate the styles sheets from the html, ... I think this is just 
basic to the operation of the web.

> 	Or, have I not properly understood what you propose?

[1] title take from 
http://www.imc.org/atom-syntax/mail-archive/msg04596.html

---

From: Antone Roundy <[email-masked]>
Date: Sat, 28 Aug 2004 17:39:01 -0600

On Saturday, August 28, 2004, at 05:01  PM, Henry Story wrote:
> some of the entries in its database it can request them in the same 
> tcp connection.
>
...assuming that all the data is on the same server.

Rather than building pointer-only feeds into the format, thus requiring 
clients that don't support the API to be able to handle them, it might 
be better to do such things via the API. For example, we might define 
methods to query for a list of the most recent IDs (which could be 
checked against your cache to see which are new), and a method to query 
for a set of entries (by ID), which would be returned in the standard 
feed format.

---

From: Henry Story <[email-masked]>
Date: Sun, 29 Aug 2004 06:59:03 +0200

On 29 Aug 2004, at 01:39, Antone Roundy wrote:

>> MacLeod during a long conversation on Atom irc a couple of months 
>> ago. He helped me rebuild my model quite dramatically. Here are some 
>> things to keep in mind.
>>

yes, good point.

>
> Rather than building pointer-only feeds into the format,

The system I propose [2] is quite flexible, and I think Atom as it is 
can also be that
flexible. It does not require pointer only feeds. It allows them. Which 
is really handy
for all feeds in which the feed and the entry are on the same machine, 
which is going to be a large chunk of the cases.

The important thing I ask for is that all entries have their own URIs. 
All entries should be resources.

But feeds can add more information if they wish. The title, the author, 
the summary.... If they want the feed can contain *all* the entry 
information. It will be a much bulkier download of course, and much 
less cache-able. (I even allow for all the entries, with all their 
history to be in one large file [3]. )

It is also possible for the aggregator to place a copy of the Entry, on 
its own machine. That is why we have ids that don't change. It would be 
correct behavior in that case for the aggregator to have that entry 
point to the original entry. Which is not so different from having it 
point to the original feed, except for it being a little more 
consistent with putting the emphasis on Entries rather than on feeds.

I also understand that in that case it would also be good practice to 
have the copy of the entry point to the original feed from which this 
was found, as a further point of interest.

> thus requiring clients that don't support the API to be able to handle 
> them, it might be better to do such things via the API. For example, 
> we might define methods to query for a list of the most recent IDs 
> (which could be checked against your cache to see which are new), and 
> a method to query for a set of entries (by ID), which would be 
> returned in the standard feed format.
>

Henry
http://bblfish.net/

[1] http://www.imc.org/atom-syntax/mail-archive/msg04596.html
[2] http://bblfish.net/work/atom-owl/2004-08-12/blogexample.html
[3] http://bblfish.net/work/atom-owl/2004-08-12/AllInOneDatabase.n3