Subject: Tombstones

From: Mark Nottingham <[email-masked]>
Date: Tue, 4 Dec 2007 18:38:23 -0800

What's the current state of proposals for and interest in what we  
called tombstones a while back?

I'm seeing use cases for managing spam; e.g., if your site accepts  
user-generated-content, and you make a feed of it available, it would  
be nice to be able to be able to have spam control reflected not only  
on the Web page, and in the feed, but also in aggregators' views of  
the data.

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Tue, 04 Dec 2007 20:08:02 -0800

Mark Nottingham wrote:
> 
> 
> What's the current state of proposals for and interest in what we called
> tombstones a while back?
> 

There really isn't a current status.  Msft SSE extensions come the
closest but there are some definite issues that I personally think would
need to be resolved.

What I personally would like to see is an evolved form of SSE produced
on the standards track by a WG.  Tombstones would be covered as part of
that activity.

> user-generated-content, and you make a feed of it available, it would be
> nice to be able to be able to have spam control reflected not only on
> the Web page, and in the feed, but also in aggregators' views of the data.
> 

We have use cases for offline synchronization, bidirectional sync,
seeding search engines, spam control, more efficient subscription models
(e.g. just give me the changes since the last time I checked), and so
on.  After exploring the issue, I strongly believe a single, common
approach would be ideal.

- James

---

From: Steven Lees <[email-masked]>
Date: Tue, 4 Dec 2007 21:03:41 -0800

> closest but there are some definite issues that I personally think

It would be great to hear about those issues. For what it's worth, we released our "v1" spec of SSE today. (Not to say that there isn't room for more discussion.) We also renamed it to "FeedSync", hoping that name might be a little easier to remember/spell/etc. The new spec is on http://feedsync.org/spec/.

> on.  After exploring the issue, I strongly believe a single, common

I definitely agree. We considered a separate spec for tombstones that was independent of the other sync extensions. It didn't seem worth it given the benefit of having all of the sync-related extensions in a single, coherent spec. If you want spam control, chances are pretty good you want one-way sync, and at that point you want more than just tombstones.

Steven

---

From: James M Snell <[email-masked]>
Date: Tue, 04 Dec 2007 21:22:32 -0800

http://www.snellspace.com/wp/?p=818

- James

---

From: "James Holderness" <[email-masked]>
Date: Wed, 5 Dec 2007 06:53:02 -0000

> on the standards track by a WG.  Tombstones would be covered as part of

FWIW I'm not interested in anything more complicated than your tombstone 
proposal from last year. I just want a simple way to mark an item in a feed 
as deleted.

I was considering implementing something based on your last draft, even if 
only for personal use. It would be preferable if there was a real spec to 
work from, but there's no way I would implement anything as complicated as 
SSE when all I want to do is delete spam.

Regards
James

---

From: James M Snell <[email-masked]>
Date: Wed, 05 Dec 2007 01:35:06 -0800

+1. What I want is an approach that scales up well from the simplest
case (just marking tombstones) to more complicated cases.  Just having
an x:deleted element would go a very long ways towards solving the
majority of use cases I've come across.  The rest would seem to be
covered by having a consistent model of being able to request just the
changes that have occurred within a specific period of time.  Everything
else tends to just be variations on the theme.

- James

James Holderness wrote:
> proposal from last year. I just want a simple way to mark an item in a
> feed as deleted.
> 
> I was considering implementing something based on your last draft, even
> if only for personal use. It would be preferable if there was a real
> spec to work from, but there's no way I would implement anything as
> complicated as SSE when all I want to do is delete spam.
> 
> Regards
> James
> 
>

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 6 Dec 2007 00:56:10 +0700

> Steven Lees wrote:
> > 
> > It would be great to hear about those issues. For what it's 
> worth, we released our "v1" spec of SSE today. (Not to say 
> that there isn't room for more discussion.) We also renamed 
> it to "FeedSync", hoping that name might be a little easier 
> to remember/spell/etc. The new spec is on
> http://feedsync.org/spec/.

The main issue I have had with FeedSync is that deleted entries look
like "live" entries for every consumer that doesn't understand FeedSync.
This means that you cannot use FeedSync unless all clients are
FeedSync-aware. James idea of using an extension element solves this
problem nicely.

The "by" endpoint label is supposed to be user-readable but unique; that
seems difficult to achieve. I would like to see some advice on how to
achieve this, especially in the case where one authenticated user has
multiple endpoints (say, two phones, an in-office desktop machine, and a
large pool of laptops). 

> We considered a separate spec for tombstones
> that was independent of the other sync extensions.
> It didn't seem worth it given the benefit of having
> all of the sync-related extensions in a single,
> coherent spec. If you want spam control, chances
> are pretty good you want one-way sync, and at that
> point you want more than just tombstones.

In a single-user blog, where the user is editing the blog in both a
blogging client and in the blog's web interface, the blogging client
should be notified (via the feed) when the user deletes an entry using
the web interface. A simple tombstone mechanism is sufficient for
handling this, because none of the conflict-resolution stuff is
necessary.

- Brian

---

From: Steven Lees <[email-masked]>
Date: Wed, 5 Dec 2007 17:17:14 -0800

> FeedSync-aware. James idea of using an extension element solves this

This issue came up in the August AtomPub interop. Our current solution on http://feedsync.mslivelabs.com is that, by default, the server doesn't give you tombstoned entries. You have to ask for them with a query parameter. I've also been hunting for the right HTTP header to ask for tombstones as an alternate mechanism.

James's suggestion of using a different element is another reasonable solution, but I think it introduces problems as well. More on that a little later...

> multiple endpoints (say, two phones, an in-office desktop machine, and

I discussed that recently here: http://blogs.msdn.com/stevenlees/archive/2007/12/02/about-a-by.aspx. The short answer is that "by" is not really meant to be user readable. For that we'd recommend atom:author or something else in your data model.

> handling this, because none of the conflict-resolution stuff is

I wasn't so much referring to the conflict resolution part as the other sync metadata. Once piece that's really useful is the sx:sync/@updates attribute--it's simpler and more reliable than a timestamp for detecting updates. If you really only care about one-way (aka publish only) sync, then your FeedSync implementation becomes that much simpler.

Steven

---

From: James M Snell <[email-masked]>
Date: Wed, 05 Dec 2007 20:20:20 -0800

Comments below.

Steven Lees wrote:
> This issue came up in the August AtomPub interop. Our current solution 
> on http://feedsync.mslivelabs.com is that, by default, the server doesn't 
> give you tombstoned entries. You have to ask for them with a query parameter. 
> I've also been hunting for the right HTTP header to ask for tombstones as an 
> alternate mechanism.
> 

Such mechanisms are only necessary if you overload atom:entry. Using
something like x:deleted, there's no reason a client would have to do
anything special to get the tombstones.

> James's suggestion of using a different element is another reasonable solution, 
> but I think it introduces problems as well. More on that a little later...
> 

Definitely please do post your thoughts on that.

> [snip]
> I wasn't so much referring to the conflict resolution part as the 
> other sync metadata. Once piece that's really useful is the sx:sync/@updates 
> attribute--it's simpler and more reliable than a timestamp for detecting 
> updates. If you really only care about one-way (aka publish only) sync, 
> then your FeedSync implementation becomes that much simpler.
> 

I'm not convinced about the "more reliable" part.  If app:edited
elements are used properly, they would seem to be more than sufficient.
 I'm curious if you have specific examples of where using the timestamp
breaks down?

- James

---

From: David Powell <[email-masked]>
Date: Thu, 06 Dec 2007 12:45:19 +0000

> 1. Deleted entries (aka Tombstones) are indicated by extension elements
> contained within the atom:entry element. For clients that do not understand
> the SSE extensions, these will appear no different than any other entry
> such clients would have no way of knowing that the entry was deleted.  Also,
> this solution assumes that there is still enough data in the database to
> reconstruct the deleted entry in the feed. The content, title, tags, etc may
> all have been removed from the database.

I disagree.  Having clients that don't understand the extensions
processing them in the same way as updates to the entries seems like a
massive plus-point for FeedSync to me.

I assumed that you aren't supposed to reconstruct those items, after
all you are trying to delete the item.  The tombstone entry would look
something like this:

<entry>
   <id>id of the original entry</id>
   <updated>deletion date</updated>
   <sx:deleted />
   <title>This entry has been removed</title>
   <summary>This entry has been removed</summary>
   ...
</entry>

If the client supports sx:deleted it can remove the entry from its
store, if the client doesn't support sx:deleted, it will replace the
old entry with this low-tech tombstone.  If the client is a framework,
sub-clients will have the opportunity to implement sx:deleted, even if
the framework doesn't.  A publishing client that understands
sx:deleted could delete an entry in this way on any arbitrary atompub
server.  There is very little coordination cost - you can use it today
and wait for the rest of the infrastructure to catch up.

The feed + stream of entries model is the 'protocol' of how feeds
work.  FeedSync is backwards compatible with this protocol; your
alternate proposal seems to require a boil-the-ocean from publishing
clients, atompub servers, intermediaries, feed servers, aggregator
frameworks, and aggregators before these tombstones will work.

Why create a new protocol when we can use an existing one?

-- 
Dave

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 6 Dec 2007 20:27:50 +0700

David Powell
> I disagree.  Having clients that don't understand the 
> extensions processing them in the same way as updates to the 
> entries seems like a massive plus-point for FeedSync to me.

<snip>

> If the client supports sx:deleted it can remove the entry 
> from its store, if the client doesn't support sx:deleted, it 
> will replace the old entry with this low-tech tombstone.  If 
> the client is a framework, sub-clients will have the 
> opportunity to implement sx:deleted, even if the framework 
> doesn't.  A publishing client that understands sx:deleted 
> could delete an entry in this way on any arbitrary atompub 
> server.  There is very little coordination cost - you can use 
> it today and wait for the rest of the infrastructure to catch
> up.

You make a good point. But, if there are frequent deletions then
non-FeedSync-aware clients will have to wade through "This entry has
been removed" entries showing up to replace entries that they have
previously marked as read or otherwise filed as "don't show me this
again unless it is updated".

Consider this alternative formulation:

  <entry>
  <sx:deleted>
     <atom:id>id of the original entry</atom:id>
  </sx:deleted>

This accomplishes your goal, but makes the "low tech tombstone"
optional, for cases where asking clients to navigate "This entry has
been removed" entries is inappropriate. It does require the server to
insert <xs:deleted>, though.

> The feed + stream of entries model is the 'protocol' of how 
> feeds work.  FeedSync is backwards compatible with this 
> protocol; your alternate proposal seems to require a 
> boil-the-ocean from publishing clients, atompub servers, 
> intermediaries, feed servers, aggregator frameworks, and 
> aggregators before these tombstones will work.

The FeedSync mechanism is reasonable if only a few entries are
occasionally deleted. But, it is not reasonable in other situations. In
particular, in WLW I save all my drafts on the server and often delete
drafts without publishing them. I don't want to see a bunch of tombstone
entries in the article list. The Snell mechanism would be more
appropriate in that scenario.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 7 Dec 2007 00:38:43 +0700

Steven Lees wrote:
> This issue came up in the August AtomPub interop. Our current 
> solution on http://feedsync.mslivelabs.com is that, by 
> default, the server doesn't give you tombstoned entries. You 
> have to ask for them with a query parameter. I've also been 
> hunting for the right HTTP header to ask for tombstones as an 
> alternate mechanism.

The most obvious way to do it would be to add a parameter to the
application/atom+xml mime type, and use that parameter along with a
strong "q" parameter in an Accept header:

   Accept: application/atom+xml;type=feed;FeedSync=1.0;q=1,
           application/atom+xml;q=0.9

I don't know what kind of politics are involved in that, though.

> I discussed that recently here: 
> http://blogs.msdn.com/stevenlees/archive/2007/12/02/about-a-by
> .aspx. The short answer is that "by" is not really meant to 
> be user readable. For that we'd recommend atom:author or 
> something else in your data model.

Thanks for the link. I could have sworn that I had read that it was
intended to be user-readable. I guess as long as a (urn:uuid:) URI could
be used, there is no real problem.

> I wasn't so much referring to the conflict resolution part as 
> the other sync metadata. Once piece that's really useful is 
> the sx:sync/@updates attribute--it's simpler and more 
> reliable than a timestamp for detecting updates. If you 
> really only care about one-way (aka publish only) sync, then 
> your FeedSync implementation becomes that much simpler.

Like James, I am not sure how much simpler it is for AtomPub. For other
feeds, I agree that it might be useful, but only if implementations are
more careful about updating it than they have been about updating
atom:updated.

Another question about FeedSync: In the FeedSync model, is it assumed
that the feed contains the complete, canonical representation of each
entry? See the thread "Indicate entry in Collection Feed is full
representation" on the atom-protocol mailing list:

http://www.imc.org/atom-protocol/mail-archive/msg10621.html

If so, then I think that most AtomPub implementations would require a
separate feed for FeedSync anyway (linked from the service document or
the collection feed using some new link relation), and the
backward-compatibility issue would not really be a factor.

- Brian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 6 Dec 2007 09:42:23 -0800

Hmm. Seems like SSE tombstones are good enough for the use case I  
illustrated; it's acceptable that clients who don't understand them  
ignore them (they'll just see more spam), and the rest of the entry  
isn't needed (it's spam, after all).

Are there any aggregators who implement this?

Cheers,

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 7 Dec 2007 02:17:57 +0700

Mark Nottingham wrote:
> Hmm. Seems like SSE tombstones are good enough for the use 
> case I illustrated; it's acceptable that clients who don't 
> understand them ignore them (they'll just see more spam), and 
> the rest of the entry isn't needed (it's spam, after all).

They will see 2X as much spam: once, when they mark the original spam as
read, and then they will see the tombstone show up as an updated entry.
So, effectively, spam is up to twice as bad with the SSE tombstones as
it is without them. 

- Brian

---

From: Steven Lees <[email-masked]>
Date: Thu, 6 Dec 2007 13:23:14 -0800

>  I'm curious if you have specific examples of where using the timestamp

Timestamps are problematic for sync because they assume that you can synchronize clocks across different machines. You also have to do things like convert between timezones which doesn't work well in practice. There are problems for the publish-only case as well as bidirectional sync. On a single machine, reported time values don't always move forward linearly when your clock resyncs against some time authority, and of course because of daylight saving time.  Or your AtomPub "server" is really a server farm, and you make successive edits that are server by different machines who's clocks are offset.

We still have a timestamp in sx:history for use as an approximate time value. We also use it as a tiebreaker in the case of conflict, but only to decide which edit goes on the "outside". We use sx:sync/@updates as the primary way to detect updates and conflicts. I think that makes sense when you consider that our primary design point was around making sync work for many scenarios, and not just deleting previously published items.

Steven

---

From: Steven Lees <[email-masked]>
Date: Thu, 6 Dec 2007 13:41:15 -0800

> again unless it is updated".

As a publisher, you have to consider two kinds of clients: those that don't understand deletions, and those that do.

If you *really* want to make sure that information disappears from any client aggregator, then the most reliable way to do that given the state of the world today is to issue an updated item where the new fields are either empty, or have some message like the one you suggest below.

If you just want to provide deletion as a feature for those who are willing to upgrade their feed client, then you can go with either tombstone method.

> insert <xs:deleted>, though.

This is approximately what we do in FeedSync. You add (or change) sx:sync/@deleted to "true", and any client that is aware of that attribute will recognize the deletion. Publishers can choose to alter the entry, or not.

> appropriate in that scenario.

That doesn't seem like a good example. If your drafts are never published in a feed, then you don't ever need to tombstone them in a feed. But that kind of scenario (having downlevel clients who never care about seeing deletions) is why our default on http://feedsync.mslivelabs.com is to omit tombstones from the feed. Since any client that wants them is already doing some extra work, it's easy enough for that client to opt in to tombstones as well, via a query parameter or HTTP header.

Steven

---

From: Steven Lees <[email-masked]>
Date: Thu, 6 Dec 2007 14:09:40 -0800

>            application/atom+xml;q=0.9

Thanks, seems like a good suggestion. Presumably the client could specify:

        Accept: application/atom+xml;type=feed;FeedSync=1.0;q=1

if it only wants the feed with tombstones. And then on the return trip, the server uses the same Content-Type:

Content-Type: application/atom+xml;type=feed;FeedSync=1.0

One potential drawback is that in both cases, the entity returned is actually a valid Atom feed. It might be nice to have the "tombstone-ness" reflected in a way that doesn't alter the Content-Type.

> be used, there is no real problem.

You're probably not mistaken, the reason that I wrote up the post was that we haven't been very clear with "by". In the past we've often used values that look temptingly user-readable. Any kind of URI is perfectly reasonable to use for by, including urn:uuid.

> more careful about updating it than they have been about updating

You're absolutely right that for sync to work, clients must always correctly update the "updates" attribute. That's what makes the whole thing work. Luckily, it's pretty simple to do, even for someone hand editing the feed. It's a lot harder to generate the UUID by hand. :-)

> backward-compatibility issue would not really be a factor.

FeedSync is no different from plain Atom in this regard. The data that you want to synchronize needs to be in the feed; whether that's partial data or complete is up to the implementer. It's perfectly reasonable to have a separate feed for sync, but our goal is to design the system so that's not at all a requirement.

Steven

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 6 Dec 2007 14:41:28 -0800

Argh, yeah, good point. This depends on the details of how the  
perceive change etc., of course, but I imagine most would show them  
twice.

OTOH, this may be good incentive for aggregators to start respecting  
tombstones :)

Cheers,

---

From: James M Snell <[email-masked]>
Date: Thu, 06 Dec 2007 15:18:23 -0800

Using the x:deleted tombstone element, on the other hand, the spam entry
only shows up once, at a maximum. Readers that do not understand the
tombstone element would act no differently than they currently do and
users should not see any change in current feed reader behavior.

Example: we have a feed with two entries. There are two feed reader
clients, neither of which know about tombstones.  Client A requests the
feed and sees both entries.

  <feed>
    ...
    <entry>
      <id>tag:example.org,2007:1</id>
      ...
    </entry>
    <entry>
      <id>tag:example.org,2007:2</id>
      ...
    </entry>
  </feed>

Entry 1 is then deleted.  Using the tombstone-inside-entry approach, the
feed becomes

  <feed>
    ...
    <entry>
      <id>tag:example.org,2007:1</id>
      <x:deleted />
      ...
    </entry>
    <entry>
      <id>tag:example.org,2007:2</id>
      ...
    </entry>
  </feed>

Client A gets the feed and still sees two entries, one of which is
marked as being updated.

Client B comes along and requests the feed and see two entries, both of
which are new to it but one of which doesn't actually exist.  Why should
Client B have to see that entry at all?  If the entry was deleted, it
shouldn't be in the feed.

Now use the separate tombstone element,

  <feed>
    ...
    <x:deleted>
      <id>tag:example.org,2007:1</id>
    </x:deleted>
    <entry>
      <id>tag:example.org,2007:2</id>
      ...
    </entry>
  </feed>

Client A continues to operate as it always has when an entry that was
once in the feed suddenly isn't and Client B never has to see the
deleted entry at all.  That makes a whole lot more sense to me.

Now, Steven indicates that their implementation will not include the
tombstone by default. Clients that want tombstones would have to ask for
them explicitly.  While that would solve the problem, using the
tombstone element would avoid the need to have that kind of negotiation
mechanism at all.

- James

Mark Nottingham wrote:
> 
> Argh, yeah, good point. This depends on the details of how the perceive
> change etc., of course, but I imagine most would show them twice.
>

---

From: "James Holderness" <[email-masked]>
Date: Thu, 6 Dec 2007 23:23:41 -0000

> So, effectively, spam is up to twice as bad with the SSE tombstones as

That's not the only flaw. Imagine you have a feed full of deleted entries. 
With the SSE method, a new user (that didn't understand tombstones) would 
end up seeing a feed full of entries with blank or "This is deleted" 
content. With James' method, they'd see nothing (which is exactly what you'd 
want).

This was discussed last year when James first proposed tombstones. I suggest 
anyone that wasn't around at the time go and look up the threads from back 
then. Might save some time rehashing old arguments.

Regards
James

---

From: Steven Lees <[email-masked]>
Date: Thu, 6 Dec 2007 17:58:43 -0800

> users should not see any change in current feed reader behavior.

The "spam-shows-up-more-than-once" idea is not correct, regardless of which tombstone mechanism you use. Even if you hand tombstoned entries to a non-tombstone client, the entry only shows up once. If, as we're recommending, you only give tombstones to clients that ask for them, then there's no entry at all.

<snip>

> shouldn't be in the feed.

Agreed. But with the <x:deleted> element, you're still passing unnecessary info to the client. If you go and delete ALL of the entries from the feed, then the client receives a feed with nothing but tombstones, which it didn't want in the first place. In that case, asking the client to opt-in seems like a pretty good idea.

> tombstone element would avoid the need to have that kind of negotiation

Again, the mechanism seems pretty simple to me, and it has the benefit of not passing any tombstones to a client that won't understand them anyway.

Steven

---

From: Steven Lees <[email-masked]>
Date: Thu, 6 Dec 2007 18:02:55 -0800

> then. Might save some time rehashing old arguments.

As I mentioned in my response to James, if clients need to ask for tombstones, then this problem disappears.

As you hinted at above, the opt-in for clients has an added benefit even for clients that DO understand tombstones. On the initial sync of a feed, most sync-aware clients will still want to ask for the feed without tombstones, so that they can just get the non-deleted items. After the initial sync, then they can ask for tombstoned items as well.

Steven

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 6 Dec 2007 18:11:25 -0800

In an ideal world, yes. However, many aggregators I've seen will  
highlight or otherwise draw attention to entries that have changed,  
using various metrics for 'change' -- including, I suspect, the  
introduction of new child elements. YMMV, of course.

Has anyone done a survey of how aggregators behave under different  
conditions like this?

Cheers,

On 06/12/2007, at 5:58 PM, Steven Lees wrote:
> The "spam-shows-up-more-than-once" idea is not correct, regardless  
> of which tombstone mechanism you use. Even if you hand tombstoned  
> entries to a non-tombstone client, the entry only shows up once.  
> If, as we're recommending, you only give tombstones to clients that  
> ask for them, then there's no entry at all.
>
> Agreed. But with the <x:deleted> element, you're still passing  
> unnecessary info to the client. If you go and delete ALL of the  
> entries from the feed, then the client receives a feed with nothing  
> but tombstones, which it didn't want in the first place. In that  
> case, asking the client to opt-in seems like a pretty good idea.
>
> Again, the mechanism seems pretty simple to me, and it has the  
> benefit of not passing any tombstones to a client that won't  
> understand them anyway.
>
> Steven

---

From: "James Holderness" <[email-masked]>
Date: Fri, 7 Dec 2007 02:40:57 -0000

Steven Lees wrote:
> As I mentioned in my response to James, if clients need to ask
> for tombstones, then this problem disappears.

The thing I don't like about that is it requires a more complicated backend. 
There are assumedly caching issues too.

> On the initial sync of a feed, most sync-aware clients will
> still want to ask for the feed without tombstones, so that
> they can just get the non-deleted items.

I wouldn't think that's worth the effort. Tombstones should be small enough 
and infrequent enough that I wouldn't care about getting them on an initial 
sync.

Regards
James

---

From: "James Holderness" <[email-masked]>
Date: Fri, 7 Dec 2007 02:53:16 -0000

Mark Nottingham:
> highlight or otherwise draw attention to entries that have changed,  using 
> various metrics for 'change' -- including, I suspect, the  introduction of 
> new child elements. YMMV, of course.

For me at least, the key issue is whether the atom:updated time has changed 
(what RFC4287 considers a significant update). However there would have to 
be a content change too (replacing the content with "this entry has been 
deleted" would do it). New child elements wouldn't make any difference.

> Has anyone done a survey of how aggregators behave under different 

Well there's the "Updated Conformance Test" on the Atom wiki. It's not a 
very good test, but I suppose it's something.
http://www.intertwingly.net/wiki/pie/UpdatedConformanceTests

Regards
James

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 7 Dec 2007 10:24:32 +0700

Steven Lees wrote:
> if it only wants the feed with tombstones. 

I would expect the presence of the FeedSync parameter to enable the
whole FeedSync mechanism, and the absence of the mechanism to indicate
that none of the FeedSync mechanism is wanted.

> And then on the return trip, the server uses the same Content-Type:
> One potential drawback is that in both cases, the entity 
> returned is actually a valid Atom feed. It might be nice to 
> have the "tombstone-ness" reflected in a way that doesn't 
> alter the Content-Type.

Clients already have to deal with the optional "type" parameter for the
Atom mime type, and if they are using the parameter in the request, then
they should be able to handle it in the response. Further, a
FeedSync-enabled server might require that clients provide the parameter
the Content-Type of PUT and POST requests, so that the client can
indicate that it follows the FeedSync specification.

> You're absolutely right that for sync to work, clients must 
> always correctly update the "updates" attribute. That's what 
> makes the whole thing work. Luckily, it's pretty simple to 
> do, even for someone hand editing the feed. It's a lot harder 
> to generate the UUID by hand. :-)

Every time I open my feed reader, I find at least one feed where all
entries appear updated "today" because the author upgraded to a new
version of WordPress or changed their template or some other thing. The
vast majority of my feeds are authored and admined by top-tier
programmers, and yet these kinds of problems still happen all the time.
To me, this is a good indication that the updates attribute is likely to
not be handled properly either.

> FeedSync is no different from plain Atom in this regard. The 
> data that you want to synchronize needs to be in the feed; 
> whether that's partial data or complete is up to the 
> implementer. It's perfectly reasonable to have a separate 
> feed for sync, but our goal is to design the system so that's 
> not at all a requirement.

Note that this is the opposite of AtomPub; see section 10 of RFC 5023.
In particular, the entries in an AtomPub collection feed are likely to
be only summaries of the actual entries. That is why I think that a
FeedSync feed is likely to be separate from the normal collection feed
(either difference URIs or by using the Accept header or something).

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 7 Dec 2007 11:12:27 +0700

Steven Lees wrote:
> James Snell wrote:
> 
> > Using the x:deleted tombstone element, on the other hand, the spam 
> > entry only shows up once, at a maximum. Readers that do not 
> > understand the tombstone element would act no differently than they 
> > currently do and users should not see any change in current feed
> > reader behavior.
> 
> The "spam-shows-up-more-than-once" idea is not correct, 
> regardless of which tombstone mechanism you use. Even if you 
> hand tombstoned entries to a non-tombstone client, the entry 
> only shows up once. If, as we're recommending, you only give 
> tombstones to clients that ask for them, then there's no entry at all.

What I meant was, first the spam will show up, the user will mark it
read, then the feed reader will see the tombstone, which looks like an
updated version of the original entry, and then mark the entry as
new/unread, and then the user will have to mark it (the tombstone) as
read. The only way to hope to prevent this in FeedSync is to ensure that
the atom:updated value is not updated when the entry is converted into a
tombstone. But, that causes problems when other specifications (e.g. RFC
5005) require that entries be paged/ordered by their atom:updated
property.

> Agreed. But with the <x:deleted> element, you're still 
> passing unnecessary info to the client. If you go and delete 
> ALL of the entries from the feed, then the client receives a 
> feed with nothing but tombstones, which it didn't want in the 
> first place. In that case, asking the client to opt-in seems 
> like a pretty good idea.

But, with the Snell mechanism, the tombstones are invisible to every
client that doesn't understand them. So, they might be bloating up the
feed (slightly), but it is almost guaranteed that they won't be visible
in the UI.

> Again, the mechanism seems pretty simple to me, and it has 
> the benefit of not passing any tombstones to a client that 
> won't understand them anyway.

Sure, but:

(1) Many (most?) AtomPub applications will not significantly benefit
from FeedSync except the tombstone mechanism (because FeedSync and
AtomPub each have their own synchronization protocols). When I look at
the FeedSync spec, I see a lot of stuff I don't really care about. It
feels wrong to implement only the tombstone part of FeedSync when that
isn't even its primary feature, but I'm not going to implement the whole
spec just to feel comfortable using its tombstone mechanism either. When
I look at James' tombstone mechanism, I see exactly what I need.

(2) James Snell's mechanism can be implemented in existing feeds without
causing any problems whatsoever; it is an almost zero-cost solution.

(3) Using a query parameter to control the tombstones does not fit with
the "RESTful" design AtomPub and other Atom-related standards. Unless
you change the opt-in mechanism to one of the following, you will never
be able to win over the hardcore Atom crowd, and another mechanism will
be chosen instead: (a) an HTTP header (perhaps Accept:, like I mentioned
in another post) to request FeedSync annotations within a feed, (b) an
atom link relation to link to a FeedSync-annotated feed, or (c) an
IRI-template-based mechanism for discovering the FeedSync-enabled feed.
(These are listed in order of my personal preference.)

(4) FeedSync could be easily adapted to use James's mechanism [1], or
both mechanisms can coexist. 

Personally, I am willing to implement the FeedSync mechanism if it is
requested via the Accept: header, or if there is otherwise consensus to
use it. Otherwise, I strongly prefer James Snell's solution (the
<x:deleted><atom:id>...</atom:id></x:deleted> variant) because it is so
unobtrusive. In fact, I could implement it in a half hour if we could
agree on a namespace to use for it (preferably one without the word
"tombstone" or other references to death).

- Brian

[1] http://www.snellspace.com/wp/?p=818, starting with "Here's a
modified take"

---

From: Joe Cheng <[email-masked]>
Date: Fri, 7 Dec 2007 10:56:43 -0800

> them explicitly.

This doesn't belong in the implementation, it belongs in the spec.
(Would this discussion even be taking place if opt-in had been in the
spec from the start?) I have advocated HTTP headers for opt-in the
past, but to avoid caching problems it seems like a link relation would
be better.

I like the idea of building tombstone-aware clients on top of client
libraries that are not tombstone aware. The inline opt-in model seems
like the easiest way of achieving that, to me.

By the way, it's worth noting that while we're all still more or less
talking about using tombstones in simple scenarios where AtomPub is
already being used today, like keeping a feed reader store synchronized
with a blog, Steven and his team is thinking of more ambitious
scenarios, such as (but not limited to) multi-master replication with
eventual consistency.

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 8 Dec 2007 03:27:50 +0700

James Holderness wrote:
> James M Snell wrote:
> > What I personally would like to see is an evolved form of 
> > SSE produced on the standards track by a WG.  Tombstones
> > would be covered as part of that activity.
> 
> FWIW I'm not interested in anything more complicated than 
> your tombstone proposal from last year. I just want a simple 
> way to mark an item in a feed as deleted.
> 
> I was considering implementing something based on your last 
> draft, even if only for personal use. It would be preferable 
> if there was a real spec to work from, but there's no way I 
> would implement anything as complicated as SSE when all I 
> want to do is delete spam.

How do you suggest dealing with inappropriate data in the atom:id
element? Examples:

 <x:deleted
id='http://example.org/blog/i-am-so-high-at-work-right-now'/>
 <x:deleted
id='http://example.org/blog/im-getting-divorced-and-marrying-my-secretar
y'/>
 <x:deleted id='http://engizmet.org/{embargoed-iphone-news-item}'/>

(I've seen very similar entries posted and then deleted from actual
feeds.)

This still discloses a significant amount of information that the feed
author doesn't want disclosed after deleting the entry, and this kind of
disclosure is something that a non-technical blogger would not expect.
It would be better to avoid sending the tombstones to clients that
hadn't received the entries.

- Brian

---

From: "James Holderness" <[email-masked]>
Date: Sat, 8 Dec 2007 01:35:22 -0000

Brian Smith wrote:
> element?

Blackmail?

> hadn't received the entries.

Once the post itself has leaked, an inappropriate atom:id is the least of 
your problems. For someone to see your incriminating id they'd have to be 
looking at the source of the feed. And if they're that keen to find 
something incriminating, there's a good chance they would have logged the 
original post.

Anyone that thinks tombstones (in any form) can be used to destroy evidence 
of misposts is going to be sorely disappointed. They're a convenience for 
the reader not the author. The author gets to say: 'I don't think you should 
waste your time reading this post', but it's ultimately the reader's 
decision to follow that recommendation.

Regards
James

---

From: Eric Scheid <[email-masked]>
Date: Sat, 08 Dec 2007 14:52:50 +1100

> decision to follow that recommendation.

/me still amused by "So-n-so would like to recall the message 'x'" emails

e.

---

From: James M Snell <[email-masked]>
Date: Fri, 07 Dec 2007 20:18:48 -0800

Steven Lees wrote:
> [snip]
> Agreed. But with the <x:deleted> element, you're still passing unnecessary 
> info to the client. If you go and delete ALL of the entries from the feed, 
> then the client receives a feed with nothing but tombstones, which it didn't 
> want in the first place. In that case, asking the client to opt-in seems like 
> a pretty good idea.
> [snip]

Perhaps, but there's really not a lot of harm in that; the feed could
end up having lots of information any given client may not find useful.
And if an opt-in mechanism really is needed, one could be used just as
easily with x:deleted as it could with tombstone entries -- that is, the
choice of how to represent the tombstone is orthogonal to the mechanism
clients use to request tombstones.

- James

---

From: James M Snell <[email-masked]>
Date: Fri, 07 Dec 2007 21:25:21 -0800

This could probably help:

  http://tools.ietf.org/html/draft-snell-http-prefer-00

Request:
========
GET /feed
Host: example.org
Prefer: feedsync-tombstones

Response:
=========
HTTP/1.1 200 OK
Vary: Prefer

...

- James

---

From: James M Snell <[email-masked]>
Date: Fri, 07 Dec 2007 21:38:10 -0800

Updated version to fix a few typos in the original....

  http://www.ietf.org/internet-drafts/draft-snell-http-prefer-01.txt

- James

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 8 Dec 2007 22:24:11 +0700

>   http://www.ietf.org/internet-drafts/draft-snell-http-prefer-01.txt

This seems to overlap the functionality of the Opt header from RFC 2774
[1], which is already registered [2], and which I've seen in actual use.

Anyway, I still think that adding a parameter to the
application/atom+xml MIME type makes the most sense, because FeedSync is
effectively a "must-understand" extension, and the "Accept" header is
already used for content negotiation.

Failing that, how about the "Accept-Feature" header from RFC 2295 [3],
which is also already registered [2]?.

[1] http://rfc.net/rfc2774.html#s4.1
[2] http://www.iana.org/assignments/message-headers/perm-headers.html
[3] http://rfc.net/rfc2295.html#s8.2

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 8 Dec 2007 22:30:19 +0700

Joe Cheng wrote:
> > Now, Steven indicates that their implementation will not 
> > include the tombstone by default. Clients that want
> > tombstones would have to ask for them explicitly.
> 
> (Would this discussion even be taking place if opt-in had 
> been in the spec from the start?) I have advocated HTTP 
> headers for opt-in the past, but to avoid caching problems it 
> seems like a link relation would be better.

What kind of caching problems are you thinking of?

- Brian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 10 Dec 2007 22:15:04 +1100

Please, please don't use a media type parameter for this; it works  
fine for the first extension or two that wants to do this, and scales  
horribly after that.

Cheers,

On 07/12/2007, at 9:09 AM, Steven Lees wrote:

> if it only wants the feed with tombstones. And then on the return  
> trip, the server uses the same Content-Type:
>
> One potential drawback is that in both cases, the entity returned  
> is actually a valid Atom feed. It might be nice to have the  
> "tombstone-ness" reflected in a way that doesn't alter the Content- 
> Type.

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 10 Dec 2007 13:58:09 +0100

I realize that there is huge deployed infrastructure where you want  
to introduce this extension. And maybe it is a good idea to tip-toe  
around and show tombstones only to people who ask for it, etc. From  
the technical point of view I think this would do atom a disservice.  
The sooner atom feed readers learn to handle extensions properly, the  
better for the whole protocol suite.

As Mark noted, the trick with Accept headers only works for few  
possible extensions. A good, modern client would some day have to  
supply either a long list of extensions it understands or start some  
sort of negotation/discovery thing with every server (once a day or  
once a week?). "Hey, I know about tombstones and graveyard and  
afterlife extensions. Which is the feed url I should really use?"

The alternative I see is to push extensions to client so that they  
learn how to deal with the unkown ones. The earlier the better,  
future extensions then have it easy. Use a real XML parser, look at  
element names and namespaces.

James second example - <x:deleted> intermixed with <entry> - sounds  
superior to the first style which basically reads to me "here is the  
list of entries, except this one is not, if you know what i mean".

Cheers, Stefan

Am 07.12.2007 um 00:18 schrieb James M Snell:
> entry

>> perceive

--
<green/>bytes GmbH, Hafenweg 16, D-48155 Münster, Germany
Amtsgericht Münster: HRB5782

---

From: Joe Cheng <[email-masked]>
Date: Mon, 10 Dec 2007 09:09:27 -0800

>   http://tools.ietf.org/html/draft-snell-http-prefer-00

Ah, I was not aware of the Vary field. Brian, never mind about caching
issues, as they are addressed by Vary.

(The things I learn about HTTP from this list...)

---

From: "James Holderness" <[email-masked]>
Date: Mon, 10 Dec 2007 18:14:49 -0000

> issues, as they are addressed by Vary.

I thought the problem was that Vary wasn't always well supported by proxies. 
I don't remember the issues exactly, but I'm sure Mark did some research 
into this.

Regards
James

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 11 Dec 2007 03:11:38 +0700

James Holderness wrote:
> Joe Cheng wrote:
> > Ah, I was not aware of the Vary field. Brian, never mind 
> > about caching issues, as they are addressed by Vary.
> 
> I thought the problem was that Vary wasn't always well 
> supported by proxies. I don't remember the issues exactly, but
> I'm sure Mark did some research into this.

If content negotiation is going to be used at all, then "Vary: Accept"
header is more likely to work correctly than anything else besides
"Vary: Content-Encoding".

Regardless of whether a media type parameter is added for FeedSync,
content negotiation should not be required. I think it is just an issue
of whether we want to have:

   <!-- register a media type parameter for FeedSync -->
   <link rel='alternate' href='...'
         type='application/atom+xml;FeedSync=1.0'/>

Or:

   <!-- register an Atom link relation for FeedSync -->
   <link rel='FeedSync' href='...'
         type='application/atom+xml'/>

Or both:
   
   <link rel='FeedSync' href='...'
         type='application/atom+xml;FeedSync=1.0'/>

Or neither:

   <!-- Don't register anything -->
   <link rel='http://feedsync.org/blah' href='...'
         type='application/atom+xml'/>

- Brian

---

From: Steven Lees <[email-masked]>
Date: Mon, 10 Dec 2007 17:09:55 -0800

Mark Nottingham wrote:

> -----Original Message-----
> From: Mark Nottingham [mailto:[email-masked]]
> Sent: Monday, December 10, 2007 3:15 AM
> To: Steven Lees
> Cc: Brian Smith; 'Atom-Syntax'
> Subject: Re: Tombstones
>

Thanks for the guidance. Is there a longer form explanation anywhere of how/why this breaks down? And is there any existing header that is better suited to this purpose?

Steven

---

From: Steven Lees <[email-masked]>
Date: Mon, 10 Dec 2007 17:06:19 -0800

James Snell wrote:

> From: James M Snell [mailto:[email-masked]]
> Sent: Friday, December 07, 2007 9:38 PM
> To: Joe Cheng
> Cc: Mark Nottingham; Brian Smith; 'Atom-Syntax'; Steven Lees
> Subject: Re: Tombstones
>

The Prefer: header looks like it would work. It is a problem, though, that the server isn't required to follow the request? In practice it seems like it could still work, because the server has no reason not to provide tombstones, if they are asked for.

Steven

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 12 Dec 2007 03:51:57 +0100

* James Holderness <[email-masked]> [2007-12-07 03:55]:
> I wouldn't think that's worth the effort. Tombstones should be
> small enough and infrequent enough that I wouldn't care about
> getting them on an initial sync.

It’s pointless either way. You might avoid receiving a bunch of
irrelevant tombstones on the first fetch – only to end up getting
them all on your second fetch.

You could invest somewhat significant extra complexity to make it
actually work… all so you can avoid a handful of tombstones for a
short period after subscribing to a feed. Ahem.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 12 Dec 2007 16:23:33 +1100

Yeah, +1 -- I smell premature optimisation.

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 12 Dec 2007 07:06:05 +0100

* Steven Lees <[email-masked]> [2007-12-06 22:45]:
> Timestamps are problematic for sync because they assume that
> you can synchronize clocks across different machines.

I suppose that’s important when you actually try to do something
where clocks across different machines need to synchronise.

I assume that FeedSync (which I haven’t looked at) tries to cover
some such cases.

But for tombstones, only the server’s clock matters.

Regards,

---

From: Steven Lees <[email-masked]>
Date: Wed, 12 Dec 2007 17:25:45 -0800

> short period after subscribing to a feed. Ahem.

Maybe I've had too much of the ETag cool aid, but I would think that with a reasonably straightforward ETag implementation on the feed, then I would never need to see the tombstones in place before my first sync, regardless of how many times I resync.

I certainly agree that creates a little more work on both the client and server. Whether or not you think it's worth it depends on your scenario. For blog publishing, it's probably not that interesting. For two-way sync of a larger data set, it's a lot more useful.

Steven

---

From: Steven Lees <[email-masked]>
Date: Wed, 12 Dec 2007 17:31:44 -0800

> I assume that FeedSync (which I haven’t looked at) tries to cover

Our design approach for FeedSync is that you're not going to have different clocks in sync, but we want data sync to work anyway. That's why we use version numbers instead of timestamps.

> But for tombstones, only the server’s clock matters.

Agreed. That starts to be not quite as true when your server hardware is upgraded, or your successive edit requests are handled by different machines, and so on, but in practice it will mostly work (for one-way sync).

Steven

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 13 Dec 2007 03:38:43 +0100

* Steven Lees <[email-masked]> [2007-12-13 02:30]:
> Maybe I've had too much of the ETag cool aid, but I would think
> that with a reasonably straightforward ETag implementation on
> the feed, then I would never need to see the tombstones in
> place before my first sync, regardless of how many times I
> resync.

That sounds like RFC 3229+feed, except it applies to tombstones
only and with probably no consideration given yet to how
intermediaries affect the picture. (But I might not giving enough
credit on the second point.)

The difference is that RFC 3229+feed actually offers dramatic
bandwidth savings, whereas a tombstone-specific mechanism would
deliver only negligible ones.

Regards,

---

From: "Joe Gregorio" <[email-masked]>
Date: Mon, 17 Dec 2007 22:50:46 -0500

> massive plus-point for FeedSync to me.

Agreed that putting the deleted entries in the same feed is problematic.
So why not put them in another feed? A feed dedicated to such
deleted entries, ala a trash/tombstone feed?

If I have feed that I want to keep track of deleted entries
for I could have a link to a trash/tombstone feed:

<feed ..>
    <link rel="trash" href="http://example.org/trash/feed/">
    <entry>
       ...
    </entry>
</feed>

And the feed at http://example.org/trash/feed/ would look like:

<feed ..>
    <entry>
      <id></id>
        ...
    </entry>
     ....
</feed>

Now since atom:id does not change for an entry I need only trawl
through the entries
in the trash feed looking for atom:id's I've seen before. Require
that entries be ordered in reverse order by their app:edited value
and that trash feeds are chained together via link/@rel="next" links ala
AtomPub collections and you're all set. All that functionality
and the only thing you have to do is register a new link
relation of 'trash' or 'tombstone'. [Add the atom:link element with a
relation of "trash" to an AtomPub service document
collection as a way for collections to signal that they
support tombstones.]

   -joe

-- 
Joe Gregorio        http://bitworking.org

---

From: James M Snell <[email-masked]>
Date: Mon, 17 Dec 2007 21:57:54 -0800

FWIW, this is the model currently implemented by the Lotus Connections
Activities component.  The primary reason it works is because Activities
uses the notion of a soft-delete; where a deleted entry is moved to a
Trash collection and all of it's metadata is preserved.  The entry still
exists, it's state has just changed. A soft-deleted entry can be
undeleted and restored to it's original collection or purged.  We have
no way of representing a purged entry because the data is completely
removed from the database.

If a system is using soft-deletes and the notion of a trash collection,
then I would say that this is a fairly good approach, mainly because the
atom:entry element are pointing to something that still actually exists.
 However, if the system is doing a hard-delete and is only preserving
the id of items that have been deleted, it does not make any sense to
represent those deleted items using the atom:entry element and doing so
runs the risk of confusing certain clients into thinking that they're
looking at some piece of content that still actually exists on the server.

Bottom line: we should avoid over-overloading atom:entry.  Regardless of
the specific solution required, some invention is required and clients
will have to take specific actions to support tombstones, so we are not
saving anyone any work by reusing atom:entry for this and we run the
risk of confusing clients.  The x:deleted tombstone element avoids that
risk entirely, which for me, makes it the superior solution.

- James

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 18 Dec 2007 10:45:25 +0100

Am 18.12.2007 um 06:57 schrieb James M Snell:
> [...]

+1

Very good summary.

---

From: Keith Fahlgren <[email-masked]>
Date: Tue, 18 Dec 2007 09:00:55 -0800

On 12/17/07 9:57 PM, James M Snell wrote:
> Trash collection and all of it's metadata is preserved.  

This is what we do for Books in our implementations, though we haven't really
thought about the undelete that James has (we just preserve some of the metadata).

HTH,
Keith

---

From: "Joe Gregorio" <[email-masked]>
Date: Tue, 18 Dec 2007 12:39:05 -0500

On Dec 18, 2007 12:57 AM, James M Snell <[email-masked]> wrote:
> Activities component.

Good.

> The primary reason it works is because Activities

Not true, the *only* thing you need is the atom:id, everything else
in the purged entry can be auto-generated.

> looking at some piece of content that still actually exists on the server.

Sorry, that's a straw-man, these are completely different feeds living at
different URIs, there's no chance of "confusing certain clients".

Worst case an aggregator is subscribed to both feeds and
it combines all entries with the same atom:id
and over-writes the old entry with the new 'deleted' entries
auto-generated content. Oh wait, that's *exactly* what you'd want
to happen.This just reinforces the idea of altering the entry with
auto-generated content, like changing the title
to "Tombstone Number {num}".

   -joe

---

From: James M Snell <[email-masked]>
Date: Tue, 18 Dec 2007 10:32:10 -0800

> in the purged entry can be auto-generated.

What are you saying isn't true? I was describing a fact about an
existing implementation. When an entry is purged, we do not keep any
record of it (not even the atom:id) so we have no way of representing
the purged entry. To implement full tombstone support, the most we
should have to do is start tracking the atom:id's of the purged entries
and the date/time of deletion. Why should I have to auto-generate
anything?

That said, what is the value of reusing atom:entry in this case?

Why is this...

<entry>
  <id>tag:example.org,2007:/the/only/useful/thing/here</id>
  <updated>2007-12-12T12:12:12Z</updated>
  <title>useless</title>
  <author><name>useless</name></author>
  <summary>useless</summary>
</entry>

Better than this:

<x:deleted>
  <id>tag:example.org,2007:/the/only/useful/thing/here</id>
  <updated>2007-12-12T12:12:12Z</updated>
</x:deleted>

Or even just this:

<x:deleted>
  <id>tag:example.org,2007:/the/only/useful/thing/here</id>
</x:deleted>

> different URIs, there's no chance of "confusing certain clients".

Are you sure about that?  I've tested the behavior (subscribing to
separate entry and trash feeds) in several reader clients and it's
certainly not ideal, at least not imho.

> to "Tombstone Number {num}".

No. Worst case is that the reader shows two entries.  For instance, in
Liferea, I subscribed to two feeds:

Main...
  <feed xmlns="http://www.w3.org/2005/Atom">
    <id>tag:example.org,2007:foo</id>
    <title>test feed</title>
    <updated>2007-12-12T12:12:12Z</updated>
    <link rel="trash" href="trash.xml" />
    <author><name>James</name></author>
    <entry>
      <id>tag:example.org,2007:foo/1</id>
      <title>An entry</title>
      <summary>A summary</summary>
      <updated>2007-12-12T12:12:12Z</updated>
      <link href="http://example.org" />
    </entry>
  </feed>

Trash...
  <feed xmlns="http://www.w3.org/2005/Atom">
    <id>tag:example.org,2007:foo</id>
    <title>test feed</title>
    <updated>2007-12-12T12:12:12Z</updated>
    <author><name>James</name></author>
    <entry>
      <id>tag:example.org,2007:foo/1</id>
      <title>Deleted</title>
      <updated>2007-12-12T12:13:12Z</updated>
      <content>Deleted</content>
    </entry>
  </feed>

Liferea shows two entries with no obvious relationship to one another,
which is appropriate behavior in order to avoid the risk of spoofing
attacks.  Unfortunately, there is no good way the user can know that the
"Deleted" item listed by one feed is related to the "An entry" item from
the other feed.  The only clue the user has is the fact that when you
click on either one of the entries, the other entries read status is
toggled as well.  It get's worse when there are multiple deleted entries
in the trash folder.

- James

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 19 Dec 2007 02:07:13 +0700

Joe Gregorio wrote:
> > I agree that there are ways to avoid the confusion. But, 
> > the latency from having to sync with the extra feed is 
> > unacceptable,
> 
> To whom? And who is bearing the burden of your trade-off?

If the client is on a high-latency connection, then the last thing you
want to do is double the number of requests and thus double the latency.
For example, if you have 5 second latency, you don't want to make the
user wait five extra seconds to verify that he hasn't deleted any
entries today.

> Most aggregators will ignore the 'deleted' status of an entry 
> and all those deleted entries will just be wasted bandwidth.

That is true if the number of deleted entries is large. But, that will
be rare for most applications I am interested in. Anyway, Atom isn't
really a bandwidth-friendly format to start with.

> > especially when
> > multiplied by dozens or hundreds of feeds.
> 
> There's no reason you couldn't collapse multiple trash feeds 
> into a single feed, all the atom:ids are universally unique.

I am talking about a client subscribed to feeds delivered from unrelated
sites.

- Brian

---

From: "Joe Gregorio" <[email-masked]>
Date: Tue, 18 Dec 2007 14:33:07 -0500

> I am talking about a client subscribed to feeds delivered from unrelated

Those are arguments that actually make sense. I was thinking of
tombstones solely in the case of AtomPub collections, but in the
case of syndication the

<x:deleted>

markup seems like a better fit.

    -joe

---

From: James M Snell <[email-masked]>
Date: Tue, 18 Dec 2007 12:10:55 -0800

Which brings us back to one of my original points: it would make the
most sense to have a single solution that works well across as many use
cases as possible.  Representing tombstones using x:deleted does that.
Representing tombstones using atom:entry does not.

- James

---

From: "David Powell" <[email-masked]>
Date: Tue, 18 Dec 2007 21:19:00 -0000

> risk entirely, which for me, makes it the superior solution.

The x:deleted proposal is overloading atom:feed more inapropriately
than the sx:deleted proposal overloads atom:entry.  

Atom is a great syntax for communicating a single stream of entries
belonging to a feed.  It is not a suitable syntax for communicating
*two* streams of entries and tombstones belonging to a feed.
It is the wrong shape.  Feed metadata elements aren't a good place
to embed a stream of tombstones.  Intermediaries such as
feedburner, yahoo pipes, and the windows feed platform do not support
agregating <x:deleted> feed metadata elements as they do entries.

The practical life-cycle and semantics of a top-level feed element
aren't appropriate for communicating a change in state of an entry.
I think giving up on the feed/entry model is a premature optimization.
If there are practical difficulties with combining tombstones and
normal entries in a feed, then Joe's proposal of using separate feeds
sounds like a better idea.

--
Dave

---

From: "David Powell" <[email-masked]>
Date: Tue, 18 Dec 2007 21:24:57 -0000

> user wait five extra seconds to verify that he hasn't deleted any

Would the requests need to be made synchronously?

-- 
Dave

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Dec 2007 09:28:16 +1100

+1

FWIW, I think a small individual submission on the standards track  
would be the way to go here, defining a single new feed-level element  
(preferably in the Atom namespace; I hope we left ourselves enough  
wiggle room to do that!).

James?

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 19 Dec 2007 08:15:43 +0700

Mark Nottingham wrote:
> FWIW, I think a small individual submission on the standards 
> track would be the way to go here, defining a single new 
> feed-level element (preferably in the Atom namespace; I hope 
> we left ourselves enough wiggle room to do that!).

+1 on stuffing it in the Atom namespace or some other namespace that is
commonly being used for stuff; The current way of having a new namespace
for feature is getting increasingly awkward; we have documents where the
namespace declarations outweigh the rest of the metadata.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 19 Dec 2007 15:18:12 +0700

James M Snell wrote:
> James Holderness wrote:
> > 
> > Brian Smith wrote:
> >> Ha. I already implemented it too, but I used the updated 
> >> syntax that James suggested on his blog
> >> (http://www.snellspace.com/wp/?p=818).
> > 
> > I chose the original syntax because for me it's an order of 
> > magnitude easier to implement. And since I doubt I'm going
> > to see any server support for this either way, I might as
> > well go with whatever is easiest.

I wouldn't be so pessimistic; I think eventually servers will all
implement something like this. 

> FWIW, I have no real preference for either of the two 
> syntaxes so long as we have a way of expressing the atom:id, 
> a timestamp and possibly a source.

Obviously this little detail isn't a deal-breaker, but I think it makes
sense to use the same syntax as <atom:feed>, <atom:entry>, and
<atom:source> (meaning an element instead of an attribute)--especially
if we are going to have <atom:updated> and/or <app:edited> and/or
<atom:source> nested within it.

> >> I think this way makes it clearer how to handle cases 
> >> where an entry was added, we saw it, the entry was deleted,
> >> we saw a tombstone, and then the entry was undeleted, and 
> >> we see the entry again. 
> >> In order to support undelete, we need to be able to order
> >> tombstones and entries by date.
> > 
> > If you're undeleting something, just remove the tombstone from the 
> > feed and add the entry back.
> +1. The timestamp can still be useful when dealing with more complex
> sync use cases.

In particular, without the timestamp(s) it is hard to see how the
tombstones fit in with feed paging and date-range-based queries.

> >> This new syntax also makes it possible to specify a 
> >> <source> of the tombstone, which is needed when dealing
> >> with aggregated feeds,
> > 
> > I was thinking about that too. I don't see how you can 
> > possibly trust a tombstone in an aggregate feed.

If you trust the aggregator, then you can have a policy of matching
tombstones and entries by (source, id). If you don't trust the
aggregator, and you really need to verify the entry was deleted, then
you can use the source metadata to look at the original feed to see if
it contains the tombstones or not.

> > Still, there's nothing stopping you including a source
> > element (or whatever other metadata) in the original syntax.
> > 
> > <x:deleted-entry id="xxx">
> >  <atom:source>
> >     ...
> >  </atom:source>
> > </x:deleted>
> 
> This works. Tombstones in aggregated feeds definitely need to 
> be carefully considered.

I agree the id attribute/element choice doesn't really affect what
atom:source means inside a tombstone.

- Brian

---

From: "James Holderness" <[email-masked]>
Date: Wed, 19 Dec 2007 11:28:09 -0000

> tombstones fit in with feed paging and date-range-based queries.

I don't get this, but if you really need a date, I believe the original 
draft has a "when" attribute that should be suitable. Worst case you can add 
whatever you need as a child element.

> tombstones and entries by (source, id).

Bare in mind that the aggregator needs to propogate source elements from the 
aggregated feeds. So trusting the aggregator means trusting every feed that 
is aggregated, plus every future feed that might be aggregated. That's an 
awful lot of trust. And you're asking that of the user who is subscribing to 
the feed. I'd love to see the text for that checkbox.

> If you don't trust the

Theoretically, sure. Personally there's no way I'm going to download a whole 
new feed for each tombstone just to check if it's real. What if the 
tombstone isn't in the feed anymore? If the feed supports history do you 
start downloading archives to see the tombstone is in one of the archives? 
How many megabytes of data do you download before you give up?

This looks like a potential DOS attack to me. It's just not worth the 
effort, IMHO.

Regards
James

---

From: Mike Macgirvin <[email-masked]>
Date: Wed, 19 Dec 2007 22:54:08 +1100

> tombstones fit in with feed paging and date-range-based queries.

This concerned me as well. Without it, the end result is a server 
spitting out tombstones for every item that's ever been deleted on every 
   feed request - since it has no knowledge of what the client is 
holding and what the client is interested in holding/syncing.

I was also concerned about this during the prev/next discussions but bit 
my tongue when folks suggested that a page they've seen before shouldn't 
ever change. I don't agree. I just edited an entry from 1968. It might 
be buried on either page=279 or weblog/AUG-1968, but either way; it 
changed and a client that really wants to keep in sync should be made 
aware that a change was made. I also recently removed some copyrighted 
material from my site, but couldn't remove said items from the perma 
cache of a certain news reader whose company motto is 'Do No Evil'. 
Looks like a DMCA take-down is the only way to get them to flush the 
nuked entries.

How does one sync this stuff? The only thing that makes sense is for the 
client to tell the server what the client is holding and what it is 
interested in syncing. Then one can look at modification times and 
gather a list of changes to push out. In the case of a file from 1968 
that was modified, a collection of updated entries starting at a client 
specified time could be pushed.

In the case of deletions, there are a few possibilities. It would 
actually be a good idea in theory for the client to tell the server what 
ID's it already has. All of them. Then the server can respond saying 
that id 1234 was deleted. This is hardly efficient, but removes any need 
for the server to maintain state about deleted items.  So I can concede 
that the quick and dirty solution is for the server to keep a tombstone 
list, even if I find it personally disturbing for the server to maintain 
this state. Given these constraints the only thing that makes sense in 
terms of normal 'feeds' is to emit these in the proper 'order' with 
whatever other entries are being emitted. But there's no historical 
starting point. No knowledge of what the client is holding. So you've 
gotta' send out all deletions. All of them. Evey time a client connects. 
Forever. At least if they're in item order, you can limit the deleted 
item list to the range of items currently being emitted. This might work 
in the classic sense of 'working'; at least there's some means of 
communicating deletions, but it doesn't solve the copyright take-down 
problem for items that fall outside of the current requested feed range.

The right way to do it is to emit these via a 'sync' operation where the 
client specifies a start date of interest, and a date of the last 
request it made for changes.  Then you can figure out that the client's 
earliest cared about entry is in 1943, and you can send back the message 
from 1968 that was updated last week, as well as the deletion of a 
copyrighted message from 1982 that happened yesterday. When the client 
connects again next week, you'll have another set of changes that 
occurred since now. This is the most efficient way of doing it (ignoring 
the inefficiencies of protocol chatter).

So we're back to square one. To really do it right, it needs to be a 
client initiated operation because only the client knows what it has, 
what it cares about, and when it last checked. Anything else is a hack 
that will likely not survive.

mike

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 19 Dec 2007 16:01:57 +0100

* James Holderness <[email-masked]> [2007-12-19 12:45]:
> Brian Smith wrote:
>> In particular, without the timestamp(s) it is hard to see how
>> the tombstones fit in with feed paging and date-range-based
>> queries.
>
> I don't get this

Imagine a wiki where you can delete as well as undelete pages.
Atom IDs are associated with pages, not single versions of a
page. You want to represent the history of a page in an Atom
feed. If tombstones lack datetimestamps, you can’t.

Regards,

---

From: "James Holderness" <[email-masked]>
Date: Wed, 19 Dec 2007 15:45:50 -0000

> feed. If tombstones lack datetimestamps, you can’t.

Sure you could do that, but it would be completely useless to a typical feed 
reader, which is all I care about (and I thought that was Brian's interest 
too).

What would be useful (to a feed reader) is the kind of feed that Wikipedia 
already produces. That does a great job of representing wiki history and it 
doesn't require tombstones of any kind.

I understand that there are all sorts of exciting things that people may 
want to do with feeds. However, all I'm interested in is feeds that are of 
use to a general purpose feed reader. And the same goes for what I want in a 
tombstone spec.

If there are people that want to do other things with tombstones related to 
publishing or two-way syncing or whatever, that's great. I just don't see 
why we have to try and share the same protocol if it means making my life 
more difficult.

Regards
James

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 19 Dec 2007 17:16:41 +0100

* James Holderness <[email-masked]> [2007-12-19 17:00]:
> Sure you could do that, but it would be completely useless to a
> typical feed reader, which is all I care about (and I thought
> that was Brian's interest too).

Socialtext produces such feeds. I complained that they’re pretty
useless in Liferea and was told that they work great in
NetNewsWire, which all of the devs use, which is why no one felt
a pressing need to provide diff feeds.

Some feed readers are more typical than others.

> What would be useful (to a feed reader) is the kind of feed
> that Wikipedia already produces. That does a great job of
> representing wiki history and it doesn't require tombstones of
> any kind.

Ever subscribe the history of a page that got merged or renamed?
I wouldn’t say it does a great job, exactly. It conveys the
minimum information necessary to be useful to clients without
local change tracking and not really anything more.

> I just don't see why we have to try and share the same protocol
> if it means making my life more difficult.

The newer proposal uses attribute-less child elements where the
old one used attributes. I don’t see how that possibly makes your
life more difficult.

Regards,

---

From: Steven Lees <[email-masked]>
Date: Wed, 19 Dec 2007 08:33:10 -0800

> why we have to try and share the same protocol if it means making my life

Looking at the discussion over the past day, I've had the same feeling but from a different perspective.

In general, I agree with the principle from James Snell earlier in the thread:

> cases as possible.

But there's definitely a balance to be achieved. It would be helpful to take a few steps back and spend some time discussing the scenarios, not just the technical details. From this thread, I've seen a few (not mutually exclusive) ideas described:

1) "hard" delete: delete an item and it's gone forever
2) "soft" delete: you can delete an item but the data and metadata are preserved so that you can un-delete the item
3) ability to delete comments from a blog for spam control
4) ability to delete posts from a blog
5) full two-way sync between nodes via feeds
6) ability to verify the source of an entry or tombstone

And there are probably more that I missed.

Steven

---

From: James M Snell <[email-masked]>
Date: Wed, 19 Dec 2007 09:07:03 -0800

> And there are probably more that I missed.

7. one way (master-replica) full and incremental sync.
8. ability to notify aggregators that content must be removed for legal
reasons (e.g. dmca... blech)

A common thread across all these scenarios is the need to explicitly
indicate the fact that an entry has been removed from a feed.  Given
that, it would make sense to have a commonly defined tombstone
mechanism.  It also seems clear that the representation of deleted items
is distinct from the ability to perform a sync, so it would make sense
to separate those as well.  Having a separate Tombstones spec is making
a whole lot of sense at this point.

- James

---

From: "James Holderness" <[email-masked]>
Date: Wed, 19 Dec 2007 17:47:36 -0000

> old one used attributes. I don’t see how that possibly makes your

I assure you I'm not making this up. Processing an attribute-less child 
element would require considerably more code for me than the initial 
attribute-based proposal. Blame it on poor design if you will.

Now I have no desire to limit other people's choices because of my problems, 
but until someone is producing feeds that require such a format (and those 
feeds are of some use to me), I'm not going to waste my time writing code to 
deal with them.

I really don't want to spend any more time arguing with anyone about this. 
I've told you what I'm doing so that anyone that wants to do something 
compatible can do so. If nobody does, then that's fine with me too.

Regards
James

---

From: Steven Lees <[email-masked]>
Date: Wed, 19 Dec 2007 13:52:10 -0800

James Snell wrote:

> mechanism.

I agree that this is probably the case, but I'd be more confident about that if I were more clear on exactly what scenarios are most important to people. Things like ability to un-delete affect the tombstone design. I could see two separate designs making sense, although it would be nice if there was a super/subset relationship between them.

Steven

---

From: James M Snell <[email-masked]>
Date: Wed, 19 Dec 2007 14:45:56 -0800

From the use cases I've reviewed here...

> 4) ability to delete posts from a blog

As far as I'm concerned, there's really no difference between these
cases.  Whether it blog entries, bookmarks, documents, etc, if an item
is permanently deleted, we need a way of explicitly indicating that
fact.  The main reason we need this is to support a number of
application scenarios including removing entries from local client
caches, letting search indexers know that entries should be removed from
their indexes, and keeping replicas in sync with a master.

> 2) "soft" delete: you can delete an item but the data and metadata are
>    preserved so that you can un-delete the item

Soft-delete is a special case.  With this, entries are moved to a trash
collection.  We need a way of indicating that the entry has been removed
from it's original feed, while at the same time indicating it's new
location in the trash feed, e.g., something along the lines of:

  <x:deleted>
    <id>tag:example.org,2007:foo</id>
    <link rel="trash"
          type="application/atom+xml"
          href="/trash/foo" />
  </x:deleted>

Entries in the trash collection can be undeleted or purged.  If they are
undeleted, they go back to the original feed and are removed from the
trash.  If they are purged, all of the metadata, save the id and perhaps
a timestamp, will be removed from the database.

> 5) full two-way sync between nodes via feeds

This could change but for now, this is not important to us at the
moment.  All of our sync scenarios follow a master-replica pattern where
sync is always one-way.

> 6) ability to verify the source of an entry or tombstone

This will definitely be important.

> 7) one way (master-replica) full and incremental sync.

This is a significant use case for us.  We must have a reliable and
performant one-way sync model that can support full and incremental
operation.  By incremental I mean we need to have the ability to request
a feed listing only changes that have occurred since the previous sync
operation.

Our internally deployed Atom collections are already working with
hundreds of thousands of entries each.

> 8) ability to notify aggregators that content must be removed for
> legal reasons (e.g. dmca... blech)

I'm not certain it's necessary for us to have the ability to specify why
an entry was deleted so this may fall under the same general category as
items 1, 3 and 4 above.

- James