Subject: Syndication of updates and deletions of content

From: Pier Fumagalli <[email-masked]>
Date: Tue, 17 Aug 2004 18:06:24 +0100

Hi all,

A very short introduction of myself: my name is Pier and I've been 
somehow involved with Apache for the past 7 years... I currently work 
for VNU Business Publications in London, and being in the publishing 
market, we're starting to look at different ways to use syndication 
protocols throughout our systems.

I'd like to start by saying that (for the moment) I am not talking 
about Atom's Publishing Protocol, but rather of the Syndication format, 
which (for us) is at the moment more pressing.

The one thing I find difficult when reading throughout the Syndication 
Format specification is whether it allows the syndication of "deleted" 
resources.

Let's start with a very very very simple example: I have an article 
<http://www.vnunet.com/news/1157398> about Windows XP SP2, and what can 
happen on this resource is that it can either be published or deleted.

Published is easy, yesterday there was no resource, today we have one. 
In Atom, I would syndicate this as:

    <?xml version="1.0" encoding="utf-8"?>
    <feed version="..." xmlns="...">
      <head>
        <title>VNUNET.COM Feed</title>
        <link rel="alternate" type="text/html" 
href="http://www.vnunet.com/atom/index.atom"/>
        <modified>2004-08-17T16:23.52Z</modified>
        <author>
          <name>VNU Business Publications Limited</name>
        </author>
        </head>
      <entry>
        <title>Microsoft lists apps affected by XP SP2</title>
        <id>http://www.vnunet.com/news/1157398</id>
        <link rel="alternate" href="http://www.vnunet.com/news/1157398"/>
        <author><name>Iain Thomson</name></author>
        <issued>2004-08-17T08:29:29Z</issued>
        <modified>2004-08-17T08:29:29Z</modified>
      </entry>
      <entry>
         ....
      </entry>
    </feed>

This means that a resource has been published. In this format I can 
also specify that a resource was modified, if (for example) the 
modified date specified in the new feed is different from the original 
feed in which it first appeared.

I cover, basically, insertions and modifications of resources with no 
problems.

Now, how would I syndicate the fact that a resource is gone away? Of 
course I won't include in the syndication feed all of our content (more 
than 100k articles), but only what happened in the (let's say) past 24 
hours. But how can a subscriber to my Atom feed know whether a resource 
is simply not included, or it was actually removed?

I thought about having an "entry" structured in this way:

      <entry>
        <title>Microsoft lists apps affected by XP SP2</title>
        <id>http://www.vnunet.com/news/1157398</id>
        <link rel="alternate" href="http://www.vnunet.com/news/1157398"/>
        <deleted>2004-08-17T08:29:29Z</deleted>
      </entry>

but I'm not sure whether this makes sense... Probably it might be 
better not to include the <title> tag, as it wouldn't make sense 
anymore (what's the title of a non-existant resource), but I'm just 
shooting out ideas.

What do you guys think? Is there already a way to do this?

Thanks to you all...

	Pier

---

From: Sam Ruby <[email-masked]>
Date: Wed, 18 Aug 2004 07:12:48 -0400

> is simply not included, or it was actually removed?

What do you think that subscribers MAY do with this information? 
SHOULD?  MUST?

Are there ever cases where a publisher MUST or SHOULD include deleted 
information in a feed?

> but I'm not sure whether this makes sense... Probably it might be better 
> not to include the <title> tag, as it wouldn't make sense anymore 
> (what's the title of a non-existant resource), but I'm just shooting out 
> ideas.

By similar reasoning, link doesn't make sense.  Depending on the source, 
not all CMS may know when a delete took place or even *IF* a deletion 
took place.

- Sam Ruby

---

From: Pier Fumagalli <[email-masked]>
Date: Wed, 18 Aug 2004 14:30:31 +0100

On 18 Aug 2004, at 12:12, Sam Ruby wrote:
>> course I won't include in the syndication feed all of our content 
>> (more than 100k articles), but only what happened in the (let's say) 
>> past 24 hours. But how can a subscriber to my Atom feed know whether 
>> a resource is simply not included, or it was actually removed?
>

Imagine a search engine allowing me to search on a bunch of aggregated 
blogs. If I go to (for example) JavaBlogs and search for "Sam Ruby", 
the first entry I'm going to match is Bill de Hora's post of August 
15th 2003.

If you click on that link, and try to access Bill's blog, you'll go to 
a _completely different_ blog post (don't ask me why), if you click on 
"read" on the other hand, you'll see the original content of the 
blog...

Now, something tells me that Bill deleted the "Sam Ruby: Inter-net" 
post, and replaced it, and in no way the search engine was able to pick 
this, and it is serving (now) the wrong content.

Subscribers MAY use this information to know that a specific post has 
expired, and SHOULD delete all local references if they cache the 
original content.

If the feed doesn't allow for a deletion/removal/expiration mechanism, 
the only way in which I can make sure of that, is crawl all the blog 
posts I'm indexing, one by one, and verify if they're still there.

And consider that my employer wants to start syndicating roughly 
115.000 articles using atom... That's quite a challenge! :-P

>>        <link rel="alternate" 
>> href="http://www.vnunet.com/news/1157398"/>
> By similar reasoning, link doesn't make sense.  Depending on the 
> source, not all CMS may know when a delete took place or even *IF* a 
> deletion took place.

Good point... Maybe just <id> and <deleted> are actually pertinent to 
the notification of a removal event.

	Pier

---

From: Eric Scheid <[email-masked]>
Date: Thu, 19 Aug 2004 13:27:22 +1000

> the notification of a removal event.

A delete event fits within the rubric of a full versioning publishing
process, and sits alongside other events like retracted, superceded,
withdrawn, etc.

If I had a situation where I wanted to "delete" some entry I've published,
I'd probably do something like this:

<entry>
    <id>...</id>
    <title>DELETED!</title>
    <content>Sorry, someone threatened me with a kneecapping</content>
    <updated>...</updated>
</entry>

This raises a question for me though ... if version#1 contains some
elements, but version#2 is missing some of those elements, does this mean
that the true entry is in fact missing those elements? If so, will this bork
the SSFF profile of atom?

e.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 19 Aug 2004 12:11:43 -0700

>process, and sits alongside other events like retracted, superceded,

+1

>     <content>Sorry, someone threatened me with a kneecapping</content>

That seems eminently sensible and easy to both write and read.

>the SSFF profile of atom?

It means that the whole versioning publishing process needs to be 
discussed on its own as a well-thought-out extension to Atom. There 
should be a theory document that is fully agreed-to by a group of 
people before any angle brackets are first used.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Pier Fumagalli <[email-masked]>
Date: Fri, 20 Aug 2004 00:58:04 +0100

> process, and sits alongside other events like retracted, superceded,

I'm missing on few core concepts here... What's a rubric, and what is 
the difference between retracted, withdrawn and deleted...

Hmmm... This is getting waaaaay too complicated for me...

	Pier

---

From: Robert Sayre <[email-masked]>
Date: Thu, 19 Aug 2004 22:23:41 -0400

> Hmmm... This is getting waaaaay too complicated for me...

Eric was pointing out that accurate representations of such actions are 
complicated. Did you delete the linked HTML page? You obviously didn't 
delete the Atom Entry, but all the other dates refer to the time the 
Entry was changed in some way.

A meaningful <deleted> element requires a bigger commitment from 
publishers than the WG is comfortable with, IMO.

An alternative strategy would be to consider log file:

time: foo; resource: quux; action: created
time: bar; resource: quux; action: modified
time: baz; resource: quux; action: deleted
...

It would be a lot easier to syndicate that log file (each permalink 
would be a fragment identifier #) than it would be to actually track the 
state of the resource.

If you really, really need to track that resource, we have no 
established practice to draw on. So you (and the market in general) will 
have to figure out an extension for that. Sorry!

Robert Sayre

---

From: Pier Fumagalli <[email-masked]>
Date: Fri, 20 Aug 2004 09:19:10 +0100

On 20 Aug 2004, at 03:23, Robert Sayre wrote:
> Eric was pointing out that accurate representations of such actions 
> are complicated. Did you delete the linked HTML page? You obviously 
> didn't delete the Atom Entry, but all the other dates refer to the 
> time the Entry was changed in some way.

Absolutely... I work for a publication company... NOTHING gets ever 
"deleted" (ever - ever - ever). The "deletion" in a syndication feed 
simply means "if you, who subscribed to this feed, noticed that I 
published this entry, and did something about it, please notice that 
_now_ you won't be able to access that resource anymore" (relatively 
speaking, the entry has been deleted in relation to the subscriber of 
the syndication feed).

> publishers than the WG is comfortable with, IMO.

Fair enough. For our internal use, then, we'll draft up something on 
our own...

You're probably right. It shifts the meaning of a syndication feed from 
"tell me what you've got" to "tell me what you've done", which is a 
pretty substantial change.

	Pier

---

From: Sam Ruby <[email-masked]>
Date: Fri, 20 Aug 2004 08:15:27 -0400

> have to figure out an extension for that. Sorry!

The problem with an extension element within an atom:entry is that 
atom:entry elements may have required children that are may not be 
available or make sense in the context of a deleted entry.

An alternate approach: the current structure of a feed is a head element 
followed by a series of entries.  What if there were a single, optional, 
element at the end, thus:

   <deleted>
     <id>...</id>
     <id>...</id>
     <id>...</id>
   </deleted>

It seems to me that such a proposal would be fairly orthogonal to the 
rest of the syndication uses, so would have very little, if any impact. 
  Those that wish to provide this information, could, and those that can 
make use of such information would be encouraged to do so.

Of course, sites like feedster might want to retain the original data, 
but even there, removing or graying out the link might make sense.

- Sam Ruby

---

From: Walter Underwood <[email-masked]>
Date: Fri, 20 Aug 2004 09:17:53 -0700

--On Friday, August 20, 2004 9:19 AM +0100 Pier Fumagalli <[email-masked]> wrote:
> "deleted" (ever - ever - ever).

This depends on the organization.

When I was working with the search for ABCNews.com, they had permission
to keep wire stories on their site for two weeks. After two weeks,
they were deleted from the site. All the wire stories were deleted
(always - always - always).

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Pier Fumagalli <[email-masked]>
Date: Mon, 23 Aug 2004 11:54:35 +0100

> (always - always - always).

Deleted, or simply "locked away and they cannot go on the site anymore" 
??? Because we do the latter (embargo dates, like any other 
publication), but that's far from getting rid of the content itself...

	Pier

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 23 Aug 2004 09:03:09 -0400

Pier Fumagalli wrote:
>> When I was working with the search for ABCNews.com, they had 
>> permission to keep wire stories on their site for two weeks. After
two 
>> weeks, they were deleted from the site. All the wire stories were 
>> deleted (always - always - always).
	Sites or organizations that work with wire services are often
required to delete all content received after a short period of time.
Contracts will often go to the extent of specifying that even copies on
backup tapes must be erased. 
	This is very traditional in the news business and makes sense if
you consider the economic model for a wire story. Basically, a "fresh"
news story and an "archival" news story are two distinct products and
each has different prices, terms and conditions, etc. If you have an
agreement to receive and distribute "fresh" news, the contract will
often require that you delete any bits you received before those bits
become useful as "archival" stories.
	In the past, many newspapers and even some wire services
themselves had sold the rights to "archival" or "database" access to the
content they created or distributed. Thus, in the past, you couldn't
even get both rights from the same organization. You would typically buy
"fresh" news rights from a syndicator while going to a different company
and writing a second contract in order to buy "archival" access to
exactly the same stories.

		bob wyman

---

From: Robert Sayre <[email-masked]>
Date: Mon, 23 Aug 2004 09:18:23 -0400

> publication), but that's far from getting rid of the content itself...

The hypothetical Atom <deleted> event probably wouldn't distinguish 
between the two cases. It would be the result of an http DELETE method. 
RFC2616 states that a successful DELETE signals that the server 
"...intends to delete the resource or move it to an inaccessible 
location."[0]

Robert Sayre

[0]http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.7

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 23 Aug 2004 11:47:27 -0400

> Format specification is whether it allows the syndication of "deleted"
	I'm guessing that what Pier is looking for is some mechanism similar
to that used in traditional professional syndication formats such as
NewsML[1], NITF[2], etc. Those formats typically allow the communication of
"deleted" as well as a number of other document states.
	For instance, NewsML  provides a "status" element which supports the
following states: 
	* Usable -- The NewsItem and its content may be published without
restriction. An item which is "usable" may transition to either "withheld"
or "canceled" state.
	* Withheld -- Neither the NewsItem nor its content may be published
until further notice. An item which is "withheld" can transition to "usable"
or "cancelled" state.
	* Embargoed --- Neither the NewsItem nor its content may be
published until released for publication by the provider at a certain point
in time. An item which is "embargoed" can transition to "usable" or
"canceled".
	* Canceled -- Neither the NewsItem nor its content may be used under
any circumstances. If the NewsItem or its content has been published the
publisher must take immediate action to withdraw or retract it, as may be
legally necessary.
	Upon creation of a NewsItem, allowed values of the Status element
are "Usable", "Withheld" and "Embargoed" but not "cancelled".
	NITF supports similar "states" with a different syntax.

	NewsML also provides a "StatusWillChange" element which provides
advance notification of a status change that will occur automatically at
some future date and time. For example, an item with a Status of "Embargoed"
might have a StatusWillChange element stating that the status will become
"Usable" at a specified time. This is equivalent to announcing in advance
the time at which the embargo will end and the item will be released.

	For instance, the NewsML fragment below says that the current
revision of the NewsItem, created on Aug 1 and last revised on Aug 24, has a
status of "usable"; however, its state should automatically change to
"cancelled" on 30 August 2004 at 19:12.

<NewsManagement>
  <Status FormalName="usable" Scheme="IptcStatus"/>
  <StatusWillChange 
    FutureStatus="cancelled" Scheme="IptcStatus
    DateAndTime="20040830T191200Z"/>
  <FirstCreated>20040801T171000Z </FirstCreated>
  <ThisRevisionCreated>20040824T113400Z </ThisRevisionCreated>
<NewsManagement/>

Note: The "Scheme" attributes are provided to identify which "vocabulary"
the FormalName comes from.

		bob wyman

[1] http://www.newsml.org/pages/index.php
[2] http://www.nitf.org/

---

From: "Laurent Le Meur" <[email-masked]>
Date: Mon, 23 Aug 2004 19:44:54 +0200

If some of you want to have a better view at NewsML 1 and its "status"
management (plus many existing features relative to some questions raised
here), the IPTC has just released a new documentation (the "NewsML
guidelines v1.0"), available at www.newsml.org (link on the right side,
download the zip package).

It goes deep in details about the standard, but in a more legible fashion
than the original specification did. After reading, some of you may feel
that NewsML is not so complicated after all, and that it could be worth
trying something between Atom and the future NewsML 2  ;-).

Laurent Le Meur
AFP

  > -----Message d'origine-----
  > De : [email-masked] [mailto:owner-atom-
  > [email-masked]] De la part de Bob Wyman
  > Envoyé : lundi 23 août 2004 17:47
  > À : 'Pier Fumagalli'; [email-masked]
  > Objet : RE: Syndication of updates and deletions of content
  > 
  > 
  > Pier Fumagalli wrote:

-
		AVERTISSEMENT

Le présent mail et ses pièces jointes sont confidentiels et destinés au seul usage des personnes ou entités auxquelles ils sont adressés. Si vous avez reçu cet e-mail par erreur, veuillez contacter dans les plus brefs délais son expéditeur et effacer le contenu du message de votre système informatique. Toute divulgation, distribution ou copie de cet e-mail est strictement interdite.

-
		DISCLAIMER

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please contact the sender and delete the email from your system. If you are not the named addressee you should not disseminate, distribute or copy this email.

-

---

From: Mark Pilgrim <[email-masked]>
Date: Mon, 23 Aug 2004 14:34:49 -0400

> download the zip package).

Let's see... an unlinkable announcement (without its own URI) of
guidelines available only as a downloadable ZIP archive of a PDF file.

I think we were kind of hoping for something more... of-the-web.  But
thanks for the pointer... or rather, the description of a pointer.

-- 
Cheers,
-Mark

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 24 Aug 2004 13:39:17 -0600

In article <[email-masked]>,

> "deleted" as well as a number of other document states.

Joe Hildebrand and I have written an individual submission that 
describes methods for sending and receiving Atom notifications via the 
Extensible Messaging and Presence Protocol (XMPP):

http://www.ietf.org/internet-drafts/draft-saintandre-atompub-notify-00.tx
t

TinyURL: http://tinyurl.com/4qlbq

It may be of interest to those looking for notifications related to the 
creation, modification, and deletion of Atom entries.

Peter

---

From: Sam Ruby <[email-masked]>
Date: Tue, 24 Aug 2004 18:27:15 -0400

> creation, modification, and deletion of Atom entries.

Do you have any thoughts on how pubsub:id (a.k.a. ItemIDs) should (or 
should not) relate to atom:id elements in atom:entry elements contained 
within such notifications?

What are the plans for JEP-0060?

   http://www.jabber.org/jeps/jep-0060.html

- Sam Ruby

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 25 Aug 2004 01:37:42 +0100

> creation, modification, and deletion of Atom entries.

This is great :) At work we're building a system that ships rdf/xml 
encoded system/document related 'events' around using Atom for the 
package and XMPP as a protocol (it's an application layer syslog). I 
haven't gotten around to really sitting down and looking at how 
JEP60 would fit with what we're doing yet...

cheers
Bill

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 25 Aug 2004 02:09:43 -0600

Without addressing the formatting issues associated with atom:id's,
yes, I think it would be useful to use the atom:id as the pubsub item
ID.  Item ID's have the nice property that if I publish another item
with the same ID, the old item gets replaced.

Plans for JEP-60 are probably to get some more implementation
experience, and move it to Final status.  It's currently in Draft
status.

http://www.jabber.org/jeps/jep-0001.html

describes the JSF process.  A quote:

<blockquote>
In order for a JEP to advance from Proposed to Draft, it must be
generally stable, must have resolved known holes in Jabber or
deficiencies with existing protocols, must be well-understood by
interested and knowledgeable members of the Jabber community, and must
be formally defined by an XML schema. Elevation to Draft status is a
major advancement for the JEP, indicating a strong belief on the part
of the Jabber Council and Jabber community that the specification will
be of lasting value. Since a Draft standard must be well-understood
and must be known to be reasonably stable, it is relatively safe to
use it as the basis for implementations and production deployments.
However, note that because a Draft standard may still require
additional field experience and may be subject to change based on such
experience, mission-critical or large-scale implementations of the
Draft standard may not be advisable.
</blockquote>

This is a pretty good description of the current state of JEP-60.

-- 
Joe Hildebrand

---

From: Sam Ruby <[email-masked]>
Date: Wed, 25 Aug 2004 09:41:00 -0400

> with the same ID, the old item gets replaced.

What's the process for getting draft-saintandre-atompub-notify updated?

I see three basic approaches, which I will list in my order of preference:

1) decide that the pubsub item ID MUST be the same as the atom:id when 
transporting Atom notifications over the XMLPP.

2) decide that atom:id and pubsub item ID are orthogonal concepts and 
discourage (either implicitly via examples, or explicitly via prose) 
these concepts from being confused.

3) decide that atom:id SHOULD be used as the pubsub item ID, but that 
consumers MUST NOT presume that they are the same.

- Sam Ruby

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 25 Aug 2004 08:09:18 -0600

This is as good a forum as any to discuss it, and we're certainly 
willing to make mods to suit the community on both sides.

On your options:

1) this would work.  I don't see any downside, but it's pretty hard to 
enforce.
2) I don't think this is right.  No need for yet another GUID-y thing 
in the world.
3) This is probably right.  For example, I wouldn't want to have people 
look at the pub/sub ID and not the atom:id in order to do uniqueness 
checks.  That would be a layering problem.

-- 
Joe Hildebrand
Denver, CO, USA

---

From: "Bob Wyman" <[email-masked]>
Date: Wed, 25 Aug 2004 11:27:14 -0400

>What's the process for getting draft-saintandre-atompub-notify updated?
	Before too much energy goes into updating the draft, there should be
a great deal more experience gained with working with Atom over XMPP.
Frankly, I think this is an ID issued "before it's time..." As far as I
know, PubSub.com is the only publicly announced implementer of an Atom over
XMPP system and we have only a handful of clients that read data from us
(they will be announced later this week.) Given this, there is still a great
deal of learning that needs to happen before Atom over XMPP can be properly
defined. We should also get farther along in the definition of Atom itself
before trying to lock too much in stone.
	For instance, as has been noted in other messages, there is concern
that people will be implementing multiple versions of Atom or implementing
features of the "new" Atom before they are well understood. This is one
reason why the PubSub.com implementation of Atom over XMPP relies
exclusively on Atom 0.3 rather than features taken from the current drafts.
However, in response to email exchanges between the authors of the ID and
me, the ID incorporates features of the current draft: Entry is a top-level
object while Atom 0.3 requires that each entry be wrapped by a feed element.
While "new" Atom approach is definitely better, it is premature to encourage
people to implement such a thing -- and they will implement it if there is
an ID describing it.

> 2) decide that atom:id and pubsub item ID are orthogonal concepts
> and discourage (either implicitly via examples, or explicitly via 
> prose) these concepts from being confused.
	+1. atom:id and pubsub item ID are orthogonal for reasons described
in earlier email. i.e. Atom provides no mechanism to *ensure* that atom:id's
are globally unique even though it is "required" that they be globally
unique. Thus, in order to prevent malicious attacks, aggregate feeds would
be forced to create new id's for all entries. This is not good.

		bob wyman

---

From: "Bob Wyman" <[email-masked]>
Date: Wed, 25 Aug 2004 11:02:45 -0400

Joe Hildebrand wrote:
> yes, I think it would be useful to use the atom:id as the pubsub
> item ID.  Item ID's have the nice property that if I publish another
> item with the same ID, the old item gets replaced.

	Using the atom:id as the XMPP/PubSub item id is *not* a good idea.

	1. As has been noted elsewhere, while atom:id's are "required" to be
globally unique, there is no mechanism which ensures that they are, in fact,
globally unique. Thus, there is a significant risk that in aggregate feeds,
items from different source feeds will improperly replace each other simply
because they have the same atom:id. To prevent this, an aggregate feed
producer would have to create new atom:id's for each aggregated entry and
thus break the rule that an atom entry is to have the same atom:id in all
feeds in which it appears.
	2. This is a mixing of architectural layers. Data from the message
or application layer should not be used in the transport layer. In this
case, XMPP and the pubsub elements of JEP-0060 should be considered
transport layer and should be application independent. If the layering is
broken, it becomes difficult to implement transport layer functions without
interfering with application layer semantics.

	Note: As far as I know, the first and only publicly available
implementation of Atom over XMPP is the one that we have built at PubSub.com
and, although we are not even acknowledged in the Internet Draft, most of it
(and substantial portions of JEP-0060) is based on our publicly discussed
work and email/phone discussions with the authors of the ID...
	For those who wish to experiment with Atom over XMPP, I encourage
you to read more about our service at: http://www.pubsub.com/developers.php
. On that page, you'll find pointers to our "tutorial" on how to do Atom
over XMPP as well as pointers to an open source client that we've placed on
sourceforge.
 
		bob wyman

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 25 Aug 2004 08:09:18 -0600

On your options:

-- 
Joe Hildebrand
Denver, CO, USA

---

From: Joe Hildebrand <[email-masked]>
Date: Thu, 26 Aug 2004 17:04:48 -0600

After a chat with Bob Wyman who has already done a ton of work in this
space, I see that my answer below wasn't right.

Option 2 is probably the closest.  What if the pub/sub ID was formed like this:

atom-feed-url?atom:id

where the atom-feed-url is the feed that you read the item from, not
the "original" feed.  This would make it more difficult for people to
overwrite others' entries in pub/sub.  It would be up to receivers to
do cross-feed duplicate detection, using the atom:id's on the items
themselves.

-- 
Joe Hildebrand

---

From: Sam Ruby <[email-masked]>
Date: Thu, 26 Aug 2004 19:28:07 -0400

> do cross-feed duplicate detection, using the atom:id's on the items

It seems to me that the pub/sub ItemID only needs to be unique relative 
to the node, so in general, it will take less characters to make a given 
ItemID unique than an atom:id.

I'm leery of the suggestion of a question mark as feeds URIs may already 
have a question mark in them.  Like some of Dare's category feeds.  Or 
some of the early blosxom feeds.

Finally, I'm not sure what problem we are trying to solve here.  atom:id 
are meant to be globally unique.  Yes, spoofing is a potential problem, 
but I don't see how placing a requirement on the same people who are 
providing the atom:id to provide non-spoofed ItemID solves the problem.

- Sam Ruby

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 26 Aug 2004 22:03:09 -0400

Sam Ruby wrote:
> Finally, I'm not sure what problem we are trying to solve here.
>  atom:id are meant to be globally unique.  Yes, spoofing is a
> potential problem,
	There are a number of issues here. Among the more important ones are
those that deal with proper layering in the protocol stack. If one takes the
atom:id (which is "application" data) and uses it at the channel layer (i.e.
XMPP JEP-0060) you end up with some nasty consequences.
	JEP-0060 says that when two or more messages are passed and have the
same id, all but the last sent message can be discarded. Typically, this
discarding is performed by a client program; however, it can happen in the
channel itself under some circumstances. For instance, if the channel is
configured to store temporarily undeliverable messages or if proxies or
message queues are in the channel, then the intermediaries are free to
discard messages without passing them on to the client. If these messages
are discarded in the channel, then the client will never even know they
existed.
	The standard pubsub example is that of someone subscribing to
messages issued whenever IBM stock is traded. The idea is to notify the user
of changes in the price whenever they occur. In some applications, the user
is only interested in the latest price. Thus, it is ok if the channel
discards any undelivered messages if more recent copies of the message
exist. This would work well if you went on vacation for two weeks... When
you got back and restarted your client, you would only get one initial
message (the most recent price alert) followed by a normal stream of
messages. You would *NOT* receive a copy of every price alert sent during
the last two weeks.
	In many cases, it would be ok for the channel to discard messages
with duplicate pubsub ids even if the pubsub ids were the same as the
atom:id for the entries passed. However, even without dealing with the issue
of spoofing, we can see that for some applications, this is not the correct
behavior. For example: If the channel is filtering out sequential edits of a
message and only allowing the latest version of the message to be delivered,
then you can't build an application that attempts to keep track of the
changing state of an entry over time.  
	In the "stock" example above, if you wanted to track "time and
volume" to detect trends in price over time, you would want to see all the
messages. In a blogging example, you might want to have a system that can
show you "diffs" between various states of messages, similar to a CVS
system, so that you can enable things like roll-back, etc. You might even
just want to write an academic paper about editing/updating habits of
bloggers... In any case, you'd need to have your client all the messages
generated.
	What should be learned from these examples is that sometimes you
want only the most recent message and other times, you want all the
intermediaries. I.e. the correct behavior of the system is not something
that can be wired into the channel -- the decision of what to use as the ID
seen by the channel is application specific -- not something that can be
defined universally. In some cases, the required semantics could be provided
by using the atom:id as the pubsub id. In other cases, this would force the
channel to "break" the application. In the later cases, the program feeding
the channel would have to create new pubsub ids which were different from
the atom:ids.
	In addition to the problems cited above, we must face the fact that
given Atom's non-existent security provisions and the absence of mechanisms
to enforce the uniqueness of atom:ids, it is exceptionally likely that there
*will be* spoofed atom:ids. Someone will get upset by what someone else has
written and will attempt to flush the offending comment from the network by
issuing a new message with the same atom:id as the offending entry. Even if
it isn't intentional, it is inevitable that folk will often make mistakes...
	The problem of spoofing is easily handled if one can assume that all
the atom:ids in a specific feed are created or managed by a single authority
or that atom:ids are feed-specific. Basically, if the producer of some feed
overwrites his own entries, that should be seen as either intentional or the
result of silly behavior for which consequences must be accepted. The
problem comes with feeds that carry content from multiple authors or with
aggregator clients that read from multiple feeds. For instance, the kind of
aggregate feeds that we push over XMPP at PubSub.com or the kind of
multi-feed reader such as RSSBandit, etc.
	Just as aggregate feeds in files cause "special issues", so do
aggregate feeds in XMPP XML streams. At PubSub.com, we now read about 3
million RSS/Atom files multiple times each day. Given the Atom requirement
that atom:id be propagated whenever an entry is copied to a new feed, we
should be creating entries in our XMPP feed which have atom:ids which are
the same as what we found in the source feeds. However, given the rules of
atom, it is very difficult to detect atom:id spoofing without detailed
analysis of each entry. Even with such analysis (which is too expensive to
be reasonable), we can't be sure of doing the right thing in all cases.
Thus, if we use atom:id for pubsub id, we would end up making it pretty easy
for spoofers to attack any entries that get passed through our system. This
is not good -- our vulnerability would become rapidly well known and any
hope of viability for the service would disappear. That would make me very
unhappy...
	So, please, understand that atom:id is application specific data and
should not be confused with XMPP or JEP-0060 channel specific data. The
layering here is important to maintain.
	There may be some cases in which it is reasonable and convenient to
use atom:id for the pubsub item id. However, there are many cases in which
it is not reasonable. Given this, the specification SHOULD NOT require that
the pubsub id be the same as the atom:id.

		bob wyman

---

From: Sam Ruby <[email-masked]>
Date: Fri, 27 Aug 2004 10:43:36 -0400

> layering here is important to maintain.

I am quite content with leaving this as: the spec as currently written
needs to provide some guidance here; otherwise interoperability will
suffer as the consequences of the available choices are not obvious.

- Sam Ruby

---

From: Joe Hildebrand <[email-masked]>
Date: Fri, 27 Aug 2004 09:16:02 -0600

Just to clarify, you're talking about leaving the -00 draft alone? 
I'd be ok with adding a suggestion for how to generate pub/sub id's
that was non-normative.

BTW, I don't care what the separator character is.  Since this ID
isn't a URI, it could even use a space as a separator.  Maybe that
would be the best bet.

-- 
Joe Hildebrand

---

From: Sam Ruby <[email-masked]>
Date: Fri, 27 Aug 2004 11:40:46 -0400

> Just to clarify, you're talking about leaving the -00 draft alone? 

No, I think the spec as currently written is dangerously ambiguous.

If I am reading this correctly, the current consensus is that these two 
concepts are orthogonal.  Just to be clear: this is a clear reversal of 
"No need for yet another GUID-y thing in the world."... which was an 
honest and understandable opinion; the danger is that too many people 
will initially form a similar opinion, and some may act upon it.

> isn't a URI, it could even use a space as a separator.  Maybe that

I'd recommend against issuing a specific syntax recommendation.  People 
would be too tempted to try to parse the id.

  - - -

I don't yet understand the intended mapping between a feed document and 
a Jabber node.  If each feed is intended to be a separate node, then I 
don't see the problem with atom:id being the feed ItemId.

If a given node can represent multiple feeds, then there is a clear 
problem that is easy to spell out: the situation where a given entry is 
syndicated in multiple feeds processed by a single node.  Atom would 
require the atom:id to be the same in each case, Jabber would require 
the ItemId to differ.

As a fan of informative text, examples, and even the occasional 
judicious use of repetition, I would suggest an informative note which 
spells out the scenario above.

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 27 Aug 2004 12:23:50 -0400

Sam Ruby wrote:
> I don't yet understand the intended mapping between a feed document
> and a Jabber node.
	The easiest way to think about this is as though a JEP-0060 "node
id" is the same as the URI for an Atom feed-file. They are both simply
identifiers of a "feed" of entries that will arrive over time. The only real
difference is that with JEP-0060 PubSub, feed metadata isn't passed over the
node itself -- it is retrieved separately using a "metadata retrieval
operation" i.e. an XMPP IQ message. 
	The atom file "http://pubsub.com/foo.xml" could be map directly to:
	The JEP-0060 node: "/pubsub/topics/foo"
	In this case, if you simply logged the entries received over the
"/pubsub/topics/foo" node into a sequential file, you would have precisely
the same result as if you had read the foo.xml file, except that the entries
in the log file wouldn't be wrapped by an atom:feed element. (Note: We
exploit this behavior at PubSub.com. We create Atom files (with atom:feed
elements) that store a history of entries that are sent to the XMPP feeds.
On start-up, clients first read the Atom files once and then receive further
updates via the JEP-0060 PubSub protocol. It is very convenient to ensure
that the syntax between the historical record and the dynamic data is the
same. Also, this means that non-XMPP aggregators can read the "history"
files with no problem. It works wonderfully...)

> If each feed is intended to be a separate [JEP-0060] node, then I 
	In this case, as long as the feed file-feed in question is not an
aggregate feed, then only one of the two problems I outlined in my mail
would exist. We don't have to worry about the intentional or accidental
reuse of atom:ids.
	The remaining problem relates to layering: In the presence of
caching queues in the channel, or any non-instantaneous delivery of
messages, some "old" messages may be discarded by the channel. Thus, clients
who desired to see all messages would not be able to do so. Given the IBM
Stock Price alerting application in my example, this means that you could
always get "current price" of IBM's stock, but you would lose some messages
that would permit "time and volume" analysis or tracking of trends over time
since some messages would be silently dropped by the channel.
	Given that there are two conflicting styles of feed use, we can't or
shouldn't "require" an item identification scheme that makes one of the
styles impossible to implement. Thus, pubsub id and atom:id should be
considered orthogonal.

> problem that is easy to spell out: the situation where a given entry
> is syndicated in multiple feeds processed by a single node.  Atom would
	This is correct. 

		bob wyman

---

From: Joe Hildebrand <[email-masked]>
Date: Fri, 27 Aug 2004 10:34:50 -0600

> will initially form a similar opinion, and some may act upon it.

Yes, this is a complete reversal of  "No need for yet another GUID-y
thing in the world."

I was wrong.  Probably not for the first time.  :)

> would be too tempted to try to parse the id.

What if the syntax recommendation was:

SHA1((feed URL || (subscription jid & subscription node)) & atom:id)

Where & is concatenation.  No way to parse *that*.  (hopefully)

> don't see the problem with atom:id being the feed ItemId.

I see at least 2 cases:
- One node per feed ("normal" case)
- Node contains items from multiple feeds ("synthetic" case)

Imagine a world in which your blog tool does the xmpp publish at the
same time that it wrote things to your web site.  A synthetic agent
could be subscribed to several normal feeds, and republish items it
received.  Another synthetic agent could be subscribed to that
aggregate feed, etc.  That can lead to the case with the same item
being in the same node twice, via a diamond-shaped subscription chain:

   N1    N2
  /   \    /
S1   S2
  \   /
   S3

In N1, you don't care too much what the pub/sub id's are, since they
can only come from a single place.  If someone publishes again with
the same id, the intent is to overwrite the old item.  You would like
the publisher to be able to calculate the pub/sub ID easily, so they
wouldn't have to store one more GUIDy thing.

In S2, you'd like to maintain the property that if an original
publisher wants to overwrite an entry in N1, it would overwrite the
item in the synthetic feed.  You don't want items from N2 to overwrite
items from N1 in S2, since there's no easy way to do end-to-end
authorization.   Using the ID algorithm above, S2 could ensure that
IDs from N1 and N2 are always different, but that N1 could overwrite
items it had originated.

Since items from N1 can come from both S1 and S2 into S3, however,
with different pub/sub IDs, the receiver still has to check atom:id's
for uniqueness, if they want to ensure that the same item doesn't
appear twice.

> require the atom:id to be the same in each case, Jabber would require

Yes.

> spells out the scenario above.

Would a less informal rendering of the scenario above be enough?

-- 
Joe Hildebrand

---

From: Sam Ruby <[email-masked]>
Date: Fri, 27 Aug 2004 13:05:37 -0400

> Where & is concatenation.  No way to parse *that*.  (hopefully)

<Chuckle>

I guess that works.  It could even be an example instead of a 
recommendation.

> [snip]

Yes.

- Sam Ruby

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 30 Aug 2004 10:27:06 -0600

In article <[email-masked]>,

> > Where & is concatenation.  No way to parse *that*.  (hopefully)

http://intertwingly.net/wiki/pie/PaceRecommendIdScheme recommends using 
newsml URNs as described in RFC 3085:

http://www.ietf.org/rfc/rfc3085.txt

If this recommendation is still up to date, then we have a potential 
problem with updated entries. For example, using the examples in 
draft-saintandre-atompub-notify-00, the initial atom:id for the entry 
might be something like:

urn:newsml:example.org:20031213:robots:1

Now when the entry is modified, it might be (note the "U" tacked on the 
end):

urn:newsml:example.org:20031213:robots:1U

The SHA1 hash will then change, resulting in a new XMPP pubsub ItemID. 
However, we want to publish to the same ItemID since doing so overwrites 
the existing pubsub item, thus ensuring that this is understood as an 
update to an existing item rather than as a new item.

Peter

---

From: Sam Ruby <[email-masked]>
Date: Mon, 30 Aug 2004 18:31:13 -0400

> The SHA1 hash will then change, resulting in a new XMPP pubsub ItemID.

The intent is that updates to an entry do not cause the ID to change. 
(Or to put it another way, if the IDs are different, they are to be 
treated as different entries).

Other proposals say this more clearly:

   http://www.intertwingly.net/wiki/pie/PaceIdConstruct
   http://www.intertwingly.net/wiki/pie/PaceIdConstruct2

- Sam Ruby