Subject: PaceFeedState

From: Mark Nottingham <[email-masked]>
Date: Sat, 20 Nov 2004 19:49:19 -0800

I've made a proposal to fill in the "Managing Feed State" section; 
comments / suggestions appreciated.

   http://www.intertwingly.net/wiki/pie/PaceFeedState

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: "Bill Kearney" <[email-masked]>
Date: Sat, 20 Nov 2004 23:56:35 -0500

Are you proposing that something publishing a feed keep track of what
entries were in previous versions?  As in, feed X retrieved just now,
contains entries a,b,c and feed Y was published last containing d,e,f.

What sort of headers are you suggesting for providing info on uninformed
requests?  That is, from an aggregator that's never tried connecting before
and doesn't have any previous knowledge of the feed?  This being how most
RSS aggregators behave currently.

While I find the idea of tracking previous instances interesting I'm not
sure how many content sources are going to capable, let alone interested, in
doing so.

I'd be more inclined to want semantics on what kinds of ranges a source can
provide.  Be they date, numeric index or other forms.  As, informing the
aggregator the source provides date range bounding, dns-like serial number
increments, last ETag or whatever.  But tracking what was the "previous"
feed seems unlikely when something like a dynamic script source is being
used.  There is no "previous" instance only what the script detected as
'relevant' items at that time.

-Bill Kearney

----- Original Message ----- 
From: "Mark Nottingham" <[email-masked]>

---

From: Danny Ayers <[email-masked]>
Date: Sun, 21 Nov 2004 11:11:43 +0100

>    http://www.intertwingly.net/wiki/pie/PaceFeedState

[I typed before seeing Bill's response, I think's there's a bit of overlap]

I'm not sure about some details (below) of what you've got at the
moment but do reckon it would be useful to have some description of
state management. If anything I'd like to see it expanded beyond
desktop readers to describe caching apps, where the state will perhaps
be even more significant. (i.e. put on the REST spectacles). There's
also the well-known hassle of moving a blog from one server to
another.

In "Feed State Model" the assumption is being made that the order of
the entries in the feed representation is significant (most recent at
the top). I could be wrong, but I don't think that's in the spec.

"Reconstructing Feed State", seems to me to be expecting rather a lot
of the client, to remember every change and every feed head. I
wouldn't have though a warning should be needed in its absence. Under
what circumstances would the client actually want to reconstruct feed
state? My guess is that would fall outside 80/20 applications.

If anything like that does need to be preserved, might it not be the
receipt order of the entries? (rel="previous" on entries perhaps)
I think a note to clarify sort order etc would be helpful. 

Is 'wholefeed' feasible/desirable? I wouldn't have thought so outside
of 'control' circumstances. How do you tell the difference between
wholefeed (containing all revisions of entries) and wholefeed
(containing only the latest)?

Delete - yep, separate. As suggested in a recent thread, it would make
most sense to have delete mean "removed from view" rather than
"removed from existence". In which case it's only impact on the state
would be the change of an attribute rather than any structural
revision.

fyi, Henry's done quite a bit of experimentation with state and entry
versioning modelling in OWL (in Bloged), though I haven't any links at
hand.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 21 Nov 2004 08:38:33 -0800

> contains entries a,b,c and feed Y was published last containing d,e,f.

I think you mean 'feed document' instead of 'feed' here, correct?

How you do this is implementation-specific; see below. All I'm really 
suggesting is that Atom Feed Documents are in fact persistent (in that 
there's a URI for the particular instance you're viewing); they can be 
referred to by other Atom Feed Documents to reconstruct an Atom Feed.

> RSS aggregators behave currently.

If they want to try to get the whole state of the feed, they can do so 
by either of the methods described. If they can't reconstruct it, 
they'll inform the user that they can't, but can show the best guess. 
Do you think more is necessary?

> sure how many content sources are going to capable, let alone 

How hard is it to keep the last n feed documents you've published? 
Conceptually, for a feed called myfeed.atom, all you need to do is:

1. extract the 'this' URI from myfeed.atom to $old
2. create a feed document containing the new entries, with a 'this' URI 
of $new
3. publish the new document at $new with a link[@rel='prev']/@this of 
$old
4. make myfeed.atom a symlink or HTTP redirect to $new
5. delete older feed files if necessary.

> 'relevant' items at that time.

I considered a number of approaches like this, where you expose an 
interface that clients can query for specific entries. However, a 
significant number of users will not want to use a script, for both 
performance and deployment considerations.

As such, I wanted to have a system that doesn't require special 
handling on the server side. Nothing in the proposal precludes you from 
layering in a more complex solution. OTOH, a script can easily handle 
this approach if they keep a tiny bit of extra state (which 'this' 
URI(s) an entry is associated with*) around.

Cheers,

* Alternatively, they could keep a list of entry IDs that the last N 
Feed Documents are associated with; this doesn't require per-Entry 
metadata, and wouldn't take as much room.**

** Or they could encode the entries a Feed Document is associated with 
in the 'this' URI if they didn't want to keep this state at all; e.g.,
    link rel='this' href='http://www.example.com/entry;id1;id2;id3;id4'
This approach doesn't require any extra state.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 21 Nov 2004 08:46:00 -0800

> the top). I could be wrong, but I don't think that's in the spec.

Correct. I think we need to disambiguate this in any case, if there is 
an overall concept of a feed that's greater than the current document.

> of the client, to remember every change and every feed head.

It doesn't have to remember every change; it just has to remember the 
last feed 'this' URI that it saw, so it can walk its way backwards to 
it.

> I wouldn't have though a warning should be needed in its absence. Under

Most aggregators do it today; they're just sloppy about it (because 
they don't have a mechanism like this). To me, the most boring 
aggregators are those built into browsers that don't keep state. To me, 
the warning is absolutely critical; it raises the guarantee you get 
with Atom, and differentiates it from RSS substantially.

> I think a note to clarify sort order etc would be helpful.

Yes, it would; I don't mean to preclude the use of lexical order, just 
that it doesn't mean something automatically.

> would be the change of an attribute rather than any structural

Whoa, heavy; I'm not sure it's interesting to talk about what's in the 
set 'existence'; only what's in the set 'Atom Feed'.

> versioning modelling in OWL (in Bloged), though I haven't any links at

Cool; if you come across them, would love to see them.

Thanks!

---

From: "Bill Kearney" <[email-masked]>
Date: Sun, 21 Nov 2004 12:29:28 -0500

> referred to by other Atom Feed Documents to reconstruct an Atom Feed.

What are you suggesting a consuming entity will use as the identifier that
leads to the feed?  If it's a URL, as is done with RSS, then what are you
suggesting subsequent requests using that same URL should use?

> How hard is it to keep the last n feed documents you've published?

How hard isn't as much of a question as is "why should I bother"?

> 2. create a feed document containing the new entries, with a 'this' URI

What's "old" when the stream is immediately extracted from a live database
of entries?  A request of the URL from user A comprises entries a-f, an
instant later it's a-g, seconds pass and it's now b-k.  What's "old" and to
whom?

> > I'd be more inclined to want semantics on what kinds of ranges a
> > source can provide.  Be they date, numeric index or other forms.
>

Ah, so you're differentiating among sites that push data to static files, or
have a construct that could be considered a 'static grouping' versus those
being driven from a live database.  I'm not sure it's a worthwhile
differentiation.

> handling on the server side.

But it does require something server side, doesn't it?  The detection (or
tracking) of a previous 'instance', moving that 'out of the way' and putting
a rippling new ones into place.

I don't fault the notion your proposing but suggest that if you're going to
go through this much work then one might as well consider better negotiation
semantics.

> Nothing in the proposal precludes you from

I'm inclined to think it'd be better to improve the situation overall
instead of a piecemeal approach such as this.

> This approach doesn't require any extra state.

I'm unclear on what you consider to be necessary on the server and client
sides to support your suggestion.  It may be that clarifying the process
some will make it more obvious.

-Bill Kearney
Syndic8.com

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 21 Nov 2004 12:41:12 -0800

> suggesting subsequent requests using that same URL should use?

I'm proposing that there be two kinds of resources that you work with 
to get the state of an abstract Atom feed;

* A "Feed Resource" whose semantic is "I contain the most recent N 
entries of this feed." This is the URI that is commonly thought of as 
being the Atom feed's URI today; its content changes over time (much as 
in the same way that a page about the current weather in Baltimore will 
change). E.g., http://www.example.com/myfeed.atom

* An "Atom Document Resource" whose semantic is "I contain the entries 
present in the feed at a particular time." This is a URI to allow 
people to get previous "snapshots" of the feed to reconstruct them; its 
content does NOT change over time (much as in the same way that a page 
about the weather in Baltimore on November 1st, 2004 doesn't change). 
E.g., http://www.example.com/2004/11/01/myfeed.atom

When you get a Feed Resource, it contains a link[@rel='this']/@href 
that points to the Atom Document Resource for this particular Feed 
Document; that allows aggregators to know where its "permanent" URI is. 
It also contains a link[@rel='prev']/@href that points to the last Atom 
Document Resource published.

So, a consumer can fetch a representation of the Feed Resource and 
compare the 'prev' URI in it to the last 'this' URI it saw. If they 
don't match, it can follow the 'prev' URI; if it hasn't seen *that* 
document's 'prev' URI, it can walk further backwards until it's covered 
all of the entries that have appeared in the feed since it last saw it.

> How hard isn't as much of a question as is "why should I bother"?

You don't have to, if you don't want to. Some people do.

> instant later it's a-g, seconds pass and it's now b-k.  What's "old" 

If the feed content is truly dynamic, the URI for the Atom Document 
Resource could contain this information; see the ** suggestion in my 
last e-mail.

For example, if you have a completely database-driven feed, and your 
exposed Atom feed document contains 10 entries, and you've published 
1000 entries in total, the 'this' URI in your most recent Atom Document 
might be 'http://www.example.com/feeddb?entries=990-1000', and the 
'prev' URI might be 'http://www.example.com/feeddb?entries=979-989'. 
That document, in turn, would be returned with a 'prev' URI of 
'http://www.example.com/feeddb?entries=968-978' (You could, of course, 
overlap them too, if you want to).

Someone else might come along and get a 'this' URI of 
'http://www.example.com/feeddb?entries=991-1001' because you've added 
an entry in the meantime. That's OK; they'll get a 'prev' URI of 
'http://www.example.com/feeddb?entries=980-990'.

Note that this doesn't require you to keep all of the entries around; 
it just requires you to keep a counter of how many entries you've 
produced over time. If someone asks for entries that you no long have, 
you can 404 them (or, more properly, 410).

> being driven from a live database.  I'm not sure it's a worthwhile

Uhh... OK. I think you're making that differentiation; I did not.

> go through this much work then one might as well consider better 

I don't think it's a lot of work; it's quite simple. I'm very wary of 
defining query semantics; the use cases are an open set, and nailing 
them down will take a significant amount of time and effort. Similarly, 
negotiation is a notoriously difficult problem to get right.

> instead of a piecemeal approach such as this.

Can you substantiate that? Why is this a piecemeal approach?

>>     link rel='this' 
>> href='http://www.example.com/entry;id1;id2;id3;id4'

Hopefully, this mail will help.

Cheers,

---

From: Walter Underwood <[email-masked]>
Date: Sun, 21 Nov 2004 13:33:19 -0800

--On Sunday, November 21, 2004 12:41 PM -0800 Mark Nottingham <[email-masked]> wrote:
> * A "Feed Resource" whose semantic is "I contain the most recent N entries
> of this feed." [...]
> 
> present in the feed at a particular time." [...]

We could call these "current state" and "historical state".

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: "Bill Kearney" <[email-masked]>
Date: Sun, 21 Nov 2004 16:49:42 -0500

> E.g., http://www.example.com/2004/11/01/myfeed.atom

Ah, that sheds some light on it.

I'd wonder if calling the latter anything with the word "Feed" in it isn't
more trouble than it's worth?

I'm all for the idea of a way to extract 'wider' ranges of content but am
troubled by the inevitable confusion that calling it a "feed" might bring.
It's a collection of entries, ones that /might/ have been part of a range of
entries that might have been grouped in a previous feed.  But I wouldn't
call them feeds.

> Document;

Being required or optional?

> that allows aggregators to know where its "permanent" URI is.

Well, I'm not sure I see the average 'feed-like' consumption act needing
such a thing.

> all of the entries that have appeared in the feed since it last saw it.

Oh you ARE the optimist aren't you?  I do like the idea you're suggesting
but it sort of seems like a lot to ask.  Just getting developers to break
out of the "one GET to feed it ALL" will be a challenge.  Asking them to
walk over prev links, while a good idea, seems pretty unlikely.  I'd favor
use of some sort of 'range' options over the prev/next idea.  It feels like
more developers would grasp the latter instead of multiple calls traversing
it backwards.  That and I can envision source providers being more inclined
to allow range retrieval much more readily that some sort of manifest of
feed entry 'groupings' that would use prev/next.

> You don't have to, if you don't want to. Some people do.

It's not me so much as the prototypical lazy site and/or client tool
developer.

Now I *could* see using something like what you talk about from Syndic8.
We've got 3 years (nearly 1TB) of RSS data in archives.  I could well
imagine publishing prev/next groupings of the items as we saw them emerge
over time.  But that's only because we took the trouble to actually track
that sort of thing.  And even then the grouping are only based on when we
grabbed the items, not directly on any sort of site grouping.  I'm not sure
I could see individual sites bothering to do this or benefitting from it
even if they did.

> > What's "old" when the stream is immediately extracted from a live
> > database of entries?
>

Hmmm, I start getting worried when I see parameters in URLs and how their
semantics might be 'assumed' by folks.  Just as dates or integers in
namespace URI aren't versions.  Not to mention i18n issues of calling them
'entries' in a URL.  If you want to publish metadata then do so without
requiring special knowledge in a URL.

> 'http://www.example.com/feeddb?entries=980-990'.

No, I just can't see this being a good idea for something called a "feed".
Yes, it'd be good for an archive extractor.  Or local client store 'catch
up'.

What also concerns me is that many feeds aren't linear.  They aren't
comprised of items in any sort of numerical progression.  At least not from
the perspective of something that would lend itself to easy numeric range
indicators.  A feed driven from a category, for example, might have a pretty
wide range of entry 'numbers' from a subset of the main DB such that
catching up on it wouldn't work with a range parameter based on entry items.
There'd be 'holes' in the range.

This is where we get into needing better query semantics.  It seems like
some sort of capabilites statement would HAVE to be made about what sort of
reverse grouping or ranging a site has to offer.  If a site wanted to offer
what you're suggesting then it would seem like a good idea to make it clear.
I could see a degree of extensibility being needed here.

> > Ah, so you're differentiating among sites that push data to static
> > files, or have a construct that could be considered a 'static grouping'
versus
> > those being driven from a live database.  I'm not sure it's a worthwhile
> > differentiation.
>

Well, you've shed more light on the two different things you're calling a
feed, that clarifies things somewhat.  I still think it's less than a good
idea though.

> negotiation is a notoriously difficult problem to get right.

TANSTAAFL.  What's needed is a range query.  Managed on the server side
through your use of 'ranged URLs' and client-side through special knowledge
that it indicates a range.  Although as you suggest it's only whatever the
server considers the "previous" set of entries.  I'd wonder how the
requesting client would know how many it includes?

> Can you substantiate that? Why is this a piecemeal approach?

You made the point yourself, what's needed is a robust querying mechanism.
Your suggestion, while novel, falls short of that.  I'd wonder whether going
halfway is worth it.

> >> This approach doesn't require any extra state.

But just makes for hideous URLs.  Consider the base URL might already be a
mish-mash of parameters to the data.  Tacking stuff like that onto the URL
seems like a bad idea.

> Hopefully, this mail will help.

Yes, somewhat.  I now understand you're talking about two different things.
One being a feed of entries and the other being a way to extract a wider
range of entries.  I can see what you're trying to accomplish.

I'd be more inclined to want some sort of query semantics that let me ask
the site what ways it offers for obtaining 'back content'.  And then using
those semantics to request manifests of the entries for subsequent
retrieval.  If a site has tracked them in groups of previous feed
publications then fine.  But I'd also like to have the ability to do better
range requesting, most probably by timestamp but I'd sure others might want
something different.

Imagine, rather than just http for prev how about torrent or ed2k requests?
Or redirection to an archiving service or 'nearer' data store?

-Bill Kearney

---

From: Robert Sayre <[email-masked]>
Date: Sun, 21 Nov 2004 16:52:42 -0500

>   http://www.intertwingly.net/wiki/pie/PaceFeedState

Versioned Atom Feeds sounds like good practice. +1.

link[@rel="wholefeed"] has a lot of overlap with the protocol. You could 
say it's a representation of a collection of entries that could grow 
extremely large. It's unlikely that clients or servers will want to 
serve that up all at once in public. Take a look at PaceEntryQuery for 
other approaches to this problem.

I don't know where the cutoff is, but, at some point, Atom servers will 
have to "expose the raw data behind the blog to an application."[0]

Robert Sayre

[0] http://bitworking.org/news/Proposed_changes_to_draft_gregorio_07#X3

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 21 Nov 2004 20:15:09 -0800

> more trouble than it's worth?

I'm flexible on terminology, as long as it makes sense (Walter had some 
proposals).

> Being required or optional?

Optional; the proposal makes this a SHOULD.

> feed entry 'groupings' that would use prev/next.

GETting multiple resources' representations didn't stop the IMG tag 
from taking off in HTML, and I don't see how it's a significant 
challenge here. Rather than presupposing what developers are going to 
do, I'd rather get feedback from those people directly.

I don't dispute that this calls for some (manageable) work on the 
client side. I think that is vastly preferable to requiring support for 
query on the server side; there will be many more feeds than 
aggregators, if Atom is successful.

BTW, are you proposing that every request to the server for the feed's 
state is a query? Otherwise, I don't see how you can avoid multiple 
requests when reconstructing feed state (one to get the latest entries, 
at least one to get the entries you missed).

> I'm not sure

If I publish a feed, I want my consumers to see the entire information 
channel, not portions of it. That's a huge benefit.

> requiring special knowledge in a URL.

Perhaps you've misunderstood the proposal. It does not break URI 
opacity; the client is not required or encouraged to interpret the 
semantics of the URI; it's only interpreted by the server (the same 
party that issues it). This has the benefit that from the server side, 
it can be used as a query function if necessary, but the client doesn't 
need to understand the query's semantics, and, more importantly, no 
standardisation of query semantics is necessary.

> I'd wonder how the

Why does it need to?

> range of entries.  I can see what you're trying to accomplish.

I wonder; I'm not sure how to interpret that explanation.

> Or redirection to an archiving service or 'nearer' data store?

You clearly would like to do many ambitious things with Atom. That's 
great, but we need to be realistic about what we can do in this effort 
while still differentiating Atom from RSS.

Both of these approaches can satisfy the requirement to reconstruct a 
feed's state. The design I've proposed is simple, robust, leverages the 
Web and delivers needed functionality quickly; looking at similar 
efforts in the past (SQL, XML Query, RDF Query, conneg, etc.), we can 
reasonably expect that getting query right is going to take a long time 
and a fair amount of iteration, and it's going to be much more complex.

Thanks,

---

From: Eric Scheid <[email-masked]>
Date: Mon, 22 Nov 2004 16:53:42 +1100

> E.g., http://www.example.com/2004/11/01/myfeed.atom

I believe these are good (as in distinct) definitions and deserve to be in
the spec somewhere, possibly refined.

I'm not so sure about Atom Document Resource being "snapshots" ... does this
mean that I'd need separate and distinct ADR's if I edit an entry (but have
published no new entries)?

e.

---

From: Henry Story <[email-masked]>
Date: Mon, 22 Nov 2004 10:07:12 +0100

BlogEd [1] is a thick client blog editor. I am currently writing a link 
from bloged to Roller [2] using the MetaWeblog API. Being able to get 
the complete feed state is very important for a tool such as BlogEd if 
it is to correctly synchronize the state of its entries with those on 
the server.

To illustrate this imagine the following situation:
	1. Joe Blogger writes a few entries E1, E2, E3 in his blog editor
	2. Joe Blogger publishes these on the remote server as RE1, RE2 and RE3
	3. The next day Joe goes on a month long trip around Europe without 
his computer and writes up lots of cool blogs from internet cafés using 
the Roller web interface. These entries called RE4, RE5, RE6, RE7, ...
	4. During the trip Joe may not just add new entries, but update the 
state of already written ones, such as RE1 to RE1version2
	5. Back home after his trip Joe would like to synchronize the state of 
his blog editor with what he wrote in Roller, so that he can continue 
to use the advanced features of his thick client interface.

	I think at stage 5 Joe would be very disappointed if a few of his 
entries went missing because the feed did not keep a full history of 
the changes to the entries. Even more, Joe would probably like to keep 
up with this change history to any of the entries, so that he can 
himself keep an idea on how his entries changed over time.

I think this illustrates very clearly why keeping feed state is very 
important. With the MetaWeblog interface I may be able to get this 
information, but there really is not guarantee that I will be able to 
do it.

Henry Story

[1] https://bloged.dev.java.net/
[2] http://www.rollerweblogger.org/

On 21 Nov 2004, at 17:46, Mark Nottingham wrote:

> aggregators are those built into browsers that don't keep state. To 
> me, the warning is absolutely critical; it raises the guarantee you 
> get with Atom, and differentiates it from RSS substantially.

---

From: "Bill Kearney" <[email-masked]>
Date: Mon, 22 Nov 2004 09:25:36 -0500

> > I'd wonder if calling the latter anything with the word "Feed" in it
> > isn't more trouble than it's worth?
>

Good.  I'm for the idea of making reuse of the code for handling entries, be
they individual, feeds or larger collections.  I just get nervous when
terminologies for something as popular and misunderstood as "feeds" are
mixed up.

> Optional; the proposal makes this a SHOULD.

If the major players at the outset support the idea of publishing a URL in a
feed (in the RSS sense) that would directly retrieve that set of entries
then I'd certainly support the idea.  It would be a tremendous improvement
over the current situation in RSS.

I'm not sure how many tools that currently produce feeds as static documents
will be able to do this (let alone desire doing so).  Some tools, like MT
are database-driven but depend on rendering to static files for delivery.
This is good thing for saving server CPU cycles upon consumption but less
than ideal for finer-grained activities.

> from taking off in HTML,

Ummm, that had much more to do with HTML view controls rather than anything
else.  The examples are not the same and you doubtless know it.  The use of
an HTML control fobs off responsibility from the application.  The idea of
multiple calls for entry data is quite different and requires a much more
active role on the part of the application itself.

> aggregators, if Atom is successful.

So which is it?  Client or Server side work?  It seems to flip back and
forth here. Both sides will (and should) require more effort than they're
doing now.

> at least one to get the entries you missed).

Of course not every request.  Although a GREAT many feeds are dynamically
driven so it really wouldn't be much different for them.  As for multiple
requests, please, what sense of false economy are you talking about here?
ALL of this is going to require multiple requests and getting that behavior
integrated into aggregator programs will be a GOOD THING.  As to how many
requests and what is retrieved is a matter for discussion.

> channel, not portions of it. That's a huge benefit.

I'd suggest "we" represent more of the edge-case group than most.

> standardisation of query semantics is necessary.

Wording to this effect would be a crucial part of any spec that emerged.

> > requesting client would know how many it includes?

Oh, how about cell phones?  Or other resource deprived sitautions?

> while still differentiating Atom from RSS.

Your idea of realistic obviously differs.  I'm not suggesting anything
outrageously complicated.  I'm simply suggesting that if we're going to get
into the idea of retreiving back content for a feed-like instance then it
might benefit from more extensibility.

> feed's state.

Again, what you're talking about is not a feed in the current sense.  It's
perhaps better described as "a site's range of entries previously published
in what may have been multiple feed-like instances".  I stress this to make
the point of avoiding things like RSS for newsfeeds versus RSS for site
indexes.  The former being what most folks consider a "feed" and the latter
being something like what you're suggesting, but in fragmented sections.

> The design I've proposed is simple, robust, leverages the

Quickly now versus lacking in extensibility is what troubles me.

-Bill Kearney

---

From: Henry Story <[email-masked]>
Date: Mon, 22 Nov 2004 15:33:29 +0100

+1
I like the idea.

I had modeled something similar in my August OWL model [1]. There I had 
made a feed a pair pointing to a head Entry and a set of entries.

Thinking about it a bit more now, and incorporating some of Mark's 
points, I would make the Feed a document (foaf:Document perhaps?) that 
can be classified [2] into two exclusive and complimentary groups: 
Dynamic Documents and Static documents
	- Dynamic documents are document resources that do change their 
content a lot over time. These would have very short range http expires 
headers. Documents such as search results, what we usually think of as 
feeds, and newspaper front pages are usually such documents.
	- Static documents are much more RESTful (so to speak): they never 
change and therefore can be cached much more easily: These resource 
usually have an HTTP expires header set way far out in the future. 
Examples of such documents would be archives of news stories. These 
documents try as best as they can to follow the motto "cool URI's don't 
change"

This distinction can then be applied to Feeds. A Feed is a Document  
that has a header
and a list (set, or whatever) of Entries. It also has a number of Link  
objects, represented as association classes [3] in the UML above:
	- a Link that points to the the Dynamic Feed document which is the  
*start* of the
	feed. This document is the one that is usually considered to be the  
feed.
	- a Link that points to the *next* newest feed document, which  
contains the next entry states in increasing age. This is always a  
Static feed.
	- a Link that points to the previous Feed document. This can either be  
a static feed or a dynamic feed document. If the previous document is a  
Static document then it points to the same document as the *start*  
link)

We can now understand some of the confusion. Usually when people speak  
of a Feed, they mean the Dynamic feed document. This is the one that  
has a URL that people drag into their feed reader. The Dynamic Feed  
usually points to a Static Feed document, which in
turn points to another Static Feed document. Each of these static feed  
documents represent the archives of the Feed.

Since static feed documents don't change, it follows that they will  
also represent a history both of the entries, but also of the changes  
to these entries.

Would you say Mark, that the above model is a good representation of  
your proposal?

Henry Story

[1]  
http://bblfish.net/work/atom-owl/2004-08-12/ 
blogexample.html#entry.2004-08-12-2035.n3
[2] Martin Fowler, "UML Distilled, Second Edition" chapter 6, "Multiple  
and Dynamic Classification" p. 82
[3] Martin Fowler, "UML Distilled, Second Edition", chapter 6,  
"Association Class" p. 98

On 21 Nov 2004, at 21:41, Mark Nottingham wrote:

> being the Atom feed's URI today; its content changes over time (much  
> as in the same way that a page about the current weather in Baltimore  
> will change). E.g., http://www.example.com/myfeed.atom
>
> people to get previous "snapshots" of the feed to reconstruct them;  
> its content does NOT change over time (much as in the same way that a  
> page about the weather in Baltimore on November 1st, 2004 doesn't  
> change). E.g., http://www.example.com/2004/11/01/myfeed.atom
>
> Document; that allows aggregators to know where its "permanent" URI  
> is. It also contains a link[@rel='prev']/@href that points to the last  
> Atom Document Resource published.
>
> document's 'prev' URI, it can walk further backwards until it's  
> covered all of the entries that have appeared in the feed since it  
> last saw it.

---

From: Henry Story <[email-masked]>
Date: Mon, 22 Nov 2004 15:42:36 +0100

On 22 Nov 2004, at 15:33, Henry Story wrote:

> 	- a Link that points to the previous Feed document. This can either 
> be a static feed or a dynamic feed document. If the previous document 
> is a Static document then it points to the same document as the 
> *start* link)

sorry that third point should read: "if the previous document is a 
_Dynamic_ document, then it points to the same document as the *start* 
link."

---

From: "Bill Kearney" <[email-masked]>
Date: Mon, 22 Nov 2004 09:48:53 -0500

> an overall concept of a feed that's greater than the current document.

To which end?  Ordering having significance or not?

> last feed 'this' URI that it saw, so it can walk its way backwards to

As in what, provide that as a header of some sort when requesting from the
feed URL?

> with Atom, and differentiates it from RSS substantially.

I certainly agree with the idea of discouraging page-load insanity as seen
in some portal scripts and browser-embedded readers.  I can also see value
in adding semantics that require more than 'blind requests' for "the feed".
I'm not sure, however, that doing it with 'prev/next' is a way that best
solves it given the tools seen in the field.  If we're going to ask folks
(publishers and subscribers) to do something different we might as well ask
them to rise more than just a half-step above the current state of affairs.

-Bill Kearney
Syndic8.com

---

From: Graham <[email-masked]>
Date: Mon, 22 Nov 2004 15:06:39 +0000

I haven't had time to read the Pace properly yet, but browsing this 
thread it seems several people are trying to demand publishers keep 
previous versions of entries, and supply them on request. Why is this 
any of our business?

Huge -1 to Managing Feed State doing anything beyond codifying current 
feed best-practice.

Graham

---

From: "Bill Kearney" <[email-masked]>
Date: Mon, 22 Nov 2004 10:59:42 -0500

> previous versions of entries, and supply them on request. Why is this

I think we mean previous version of FEEDS not entries.  If a site wants to
get into tracking previous versions of the entries themselves, well, that's
a whole different can of worms.

As to whether it's anyone's business that's where optional/required comes
into play.

The most compelling use case I can think of for why it's a "good idea" is a
new user attempting to get up to speed on a feed.  Having their aggregator
be able to know how to traverse backwards through collections of entries
would be tremendously helpful in this regard.  It'd also help a user that's
been disconnected for a period of time find entries that may have been
published in the meanwhile.  It would also provide a way to publish 'chunks'
of entries without getting into full-on querying semantics.

I'm sensing, however, some sort of 'best practices' for publishing such
collections would be interesting.  Well-known 'types' of sets current, last
week, this month, last month, etc would probably be something many users
could identify with using.  I'm not suggesting that here but it's probably
worth considering.

> Huge -1 to Managing Feed State doing anything beyond codifying current

Which, in your estimation, is what?

-Bill Kearney

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 22 Nov 2004 08:12:11 -0800

Hi Graham,

Not demand; publishers aren't required to keep them.

It's also not about previous versions; it's about allowing them to 
catch up with entries (or changes to them) that they've missed.

IMO, this is one of the things that the Atom format can do to 
differentiate itself from RSS; it's really important. Please take a 
look before making a judgement.

Cheers,

---

From: Robert Sayre <[email-masked]>
Date: Mon, 22 Nov 2004 11:13:32 -0500

> As to whether it's anyone's business that's where optional/required comes

SHOULD is too strong, IMHO. We don't need to get into SHOULDs or MAYs 
here. Describe the feature, outline how it's accomplished.

>>Huge -1 to Managing Feed State doing anything beyond codifying current

Sounds like Graham. "Huge -1". Spare me.

People (even some aggregator authors) ask for this all time. A link 
relation that points at a feed with older entries has been in the drafts 
until very recently. It's not a big deal either way.

Robert Sayre

---

From: Walter Underwood <[email-masked]>
Date: Mon, 22 Nov 2004 09:17:52 -0800

It is also a useful model for archiving.

wunder

--On Monday, November 22, 2004 08:12:11 AM -0800 Mark Nottingham <[email-masked]> wrote:

> It's also not about previous versions; it's about allowing them to catch up with entries (or changes to them) that they've missed.
>
> IMO, this is one of the things that the Atom format can do to differentiate itself from RSS; it's really important. Please take a look before making a judgement.
>
> Cheers,
>
>

--
Walter Underwood
Principal Architect
Verity Ultraseek

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 24 Nov 2004 11:08:47 +0000

Mark Nottingham wrote:
> It's also not about previous versions; it's about allowing them to catch 
> up with entries (or changes to them) that they've missed.
> 
> differentiate itself from RSS; it's really important. Please take a look 
> before making a judgement.

I Have This Pattern.

1. I like it,m tho' I don't entirely buy the rationale - not sure if 
we're chartered for provisioning delivery SLAs.

2. My use case is non-standard, ie it's not blogging. It's for 
transmitting events of various sorts to a monitoring sink. Events are 
packed as atom 0.3 feeds. Transmission is done over xmpp.

3. The way it's 'solved' there is not by modelling a linked list with 
hyperlinks. Each feed representation is given a URI which can be 
rerequested or resent. They don't change over time. It's sufficiently 
different from the normal "sliding window" feed usage that I call these 
things "feed blocks" rather than "feeds".

4. I don't see that working out over HTTP, given that HTTP feeds 
entities are idiomatically a window onto a continuous stream of events.

5. Eventually, to programmatically retrieve stuff, there would be a feed 
of feed blocks. That feed would be a sliding window, but sliding at a 
much slower rate.

6. I agree with Walter's ob about archiving.

7. I think you move into rough terrain when you use 'feed' to imply a 
logical stream ("delayed list" in sicp speak), as well as whatever the 
current representation (window) you've been provided width.

8. //atom:head/atom:link[@rel='wholefeed'] strikes me as impractical, 
given sufficient time and monkeys with blogs.

Codifying it would be very useful, but given my use case and the 
protocol issues I won't push for this (Other than archiving, I haven't 
had time to think about the implications for http - there are 
feed/bandwidth ratios to consider, and they made my head hurt last 
time). I won't be surprised if we see a request to embed this construct 
into entries themselves.

My main ob is that how you do this seems to be aligned to the app 
protocol in use, as it holds sway over how you model the resources in 
question. My not so main ob is that over protocols like xmpp and smtp 
you don't seem to need feeds as much as you do over http.

cheers
Bill

[ot: Danny, I'm still in delivery mode, apologies for not looking at the 
dates thing yet.]

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 24 Nov 2004 11:26:09 +0000

Bill de hÓra wrote:
> 1. I like it, [...]
> protocol issues I won't push for this 

Gah 'it' means Mark's Pace.

cheers
Bill

---

From: Joe Gregorio <[email-masked]>
Date: Wed, 24 Nov 2004 09:35:19 -0500

>    http://www.intertwingly.net/wiki/pie/PaceFeedState

I like the concept but I believe the Pace goes too far in proscribing
client behaviour and also in the complexity of the mechanism.

I think a simple <link rel="prev"/> in the head of a feed which points
to the 'previous' feed would be all that is required. The client can
then, at their discretion, keep following 'prev's back until they are
satisfied. Leave it up to the client what to do with duplicate entry
id's if it encounters them (but note in the Pace that it could
happen).

The entire discussion of Feed State Model can be dropped, the heart of
the Pace being:

"To locate previous Atom Feed Documents, consumers can follow the
atom:link element in the atom:head with a @rel of "prev", which
contains a persistent URI for the previous Atom Feed Document
associated with this feed. By following such links progressively
backwards and incorporating the changes in each associated Atom Link
document, until it encounters a link to a document it already has seen
(as identified by the //atom:head/atom:link[@rel='this'] element), a
consumer can reconstruct the state of a feed reliably."

But I would drop the part about "until it encounters a link to a
document it already has seen". That may not be a good metric to go by.
For example, what if I have my top level feed with the last 10 items
in it, and each feeds 'prev' link points back to the previous 10
entries? That means that if I have 100 entries on my site then I've
got 9 'prev' links.
  
    http://example.org/feed.cgi?start=100 
    http://example.org/feed.cgi?start=90
    http://example.org/feed.cgi?start=10

Now what if I add another entry to my site, 101. Then I have 10 *new*
'prev' links:

    http://example.org/feed.cgi?start=101
    http://example.org/feed.cgi?start=91
    http://example.org/feed.cgi?start=11
    http://example.org/feed.cgi?start=1

Not the most efficient mechanism, but certainly plausible and it
causes problems with your spidering heuristic. Best leave it up to the
client.

   Thanks,
    -joe

-- 
Joe Gregorio        http://bitworking.org

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Nov 2004 08:09:26 -0800

Hi Joe,

> The entire discussion of Feed State Model can be dropped, the heart of
[...]
> document it already has seen". That may not be a good metric to go by.

I disagree. If clients have their own criteria for how far back they 
should look, or for how they combine the entries they see into a set, 
they'll act differently, and consistency is important here. One of the 
biggest complaints I have about RSS is that different aggregators have 
different concepts of what my feed is. By having a well-specified model 
of how to reconstruct the feed, as well as a model for what a feed is, 
we can assure that all consumers see the same set of entries.

If we just leave it up to the consumer to decide whether they've seen 
all of the entries, they'll use heuristics to do it, and they'll fall 
into traps in figuring it out. I'd rather have one algorithm that's 
well-tested and known to work.

For example, if a client decides that it's satisfied if the set of 
entries is the same as the last time it saw the feed, it won't go and 
look one further back. However, what if there were a series of 
snapshots that looked like this?

entry1
entry2
entry3
---
entry4
entry5
entry6
---
entry1
entry2
entry3

A client that only saw the first one would look at the last one and 
miss the fact that 4,5 and 6 were in the middle.

Likewise, if we don't say how to combine entries into a set, clients 
will use different rules. I actually think we need more guidance here; 
e.g., how to detect changed entries.

> causes problems with your spidering heuristic.

I agree that this is a problem with the approach I described earlier; 
thanks for pointing it out. Rather than take that approach, a "fully 
dynamic" server will need to keep a table in this form;

[ 'snapshot15': ['entry111','entry112'],
   'snapshot14': ['entry100' ... 'entry110'],
   ...
]

where each snapshot corresponds to a Feed Document Resource (FDR?). 
Once enough entries is added to the most recent snapshot (15 here), 
another is created. So, when someone requests the latest feed, it will 
get a 'this' of http://www.example.com/feeddb?id=snapshot15 and a 
'prev' of http://www.example.com/feeddb?id=snapshot14.

(Once again, a server doesn't have to keep all of the snapshots back in 
time)

>  Best leave it up to the client.

I don't think this follows, for reasons explained above. Only the 
server can determine what a complete set of entries is.

Cheers,

---

From: Joe Gregorio <[email-masked]>
Date: Wed, 24 Nov 2004 11:31:06 -0500

On Wed, 24 Nov 2004 08:09:26 -0800, Mark Nottingham <[email-masked]> wrote:
> different concepts of what my feed is. 

This may be where our point of disagreement hinges on. When I 
say client I am referring to a much larger range of applications
than just 'aggregators'.

> By having a well-specified model

Different aggregators working differently to me isn't such a bad
thing. For example, if an item gets updated does the aggregator
display the updated item as new? suppress displaying it? display diffs
between the versions of the entry?

> snapshots that looked like this?

Ok, that veers wildly from what I thought a series of snapshots would
look like. I was considering a 'prev' hopping back in time by either
week or month. I don't know if the overhead of designing for such a
case as you have outlined above is worth the effort.

> 'prev' of http://www.example.com/feeddb?id=snapshot14.

The part about this that makes me nervous is that this seems to be
veering closer to atom protocol stuff and not just syndication.

    -joe

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Nov 2004 08:40:14 -0800

> say client I am referring to a much larger range of applications

So am I. I'm not talking about differences in how they're displayed in 
a cosmetic sense, or about the facilities they provide (e.g., sorting, 
suppressing seen entries, etc.); it's the underlying data model -- and 
its communication from the server to the client -- that's important.

For example, if I use Atom to track changes in CVS, urgent 
announcements from management, error notifications, system alerts, 
etc., it's really important that I not miss any entries just because 
I'm not online for a little while. Likewise, it's important that my 
aggregator (or other Atom consumer, e.g., a toolkit) not show me 
duplicate entries, or call an entry 'new' when it is in fact just 
changed. Finally, it's super-important that if my tool can't avoid 
missing some entries, it knows that it has, and informs me.

> between the versions of the entry?

I expect that different aggregators will do lots of different things 
here, and that's fine. However, they need to have the same basis of 
understanding of what the feed is.

> case as you have outlined above is worth the effort.

I agree that the case outlined was a corner case, but in practice, it's 
the corner cases that always catch you.

> he part about this that makes me nervous is that this seems to be

Well, it's *a* protocol, for what that's worth; the Atom format is 
already arguably a protocol, because we place requirements on 
consumers. It's not the Atom Publishing Protocol because it's not about 
publishing, it's about the feed.

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Nov 2004 08:47:17 -0800

> given sufficient time and monkeys with blogs.

Agreed; however, it's optional. I suppose the pertinent question is 
whether people will find it useful in some cases; otherwise, it 
probably shouldn't make the cut.

> you don't seem to need feeds as much as you do over http.

Hmm. It's somewhat aligned, in that it assumes something vaguely 
RESTful (e.g., HTTP, FTP, etc.). I don't think it precludes other 
transports, it just isn't as useful on them; this might be clarified by 
adjusting the text or qualifying the section headings.

Cheers,

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 24 Nov 2004 17:04:07 +0000

Mark Nottingham wrote:
> Agreed; however, it's optional. 

I missed that; no objections from me then. i would have no objections to 
@rels that specced a date range or a since date.

> Hmm. It's somewhat aligned, in that it assumes something vaguely RESTful 
> (e.g., HTTP, FTP, etc.). I don't think it precludes other transports, it 
> just isn't as useful on them; this might be clarified by adjusting the 
> text or qualifying the section headings.

That's fair. I've had a chance to let it soak in some more... I'd need 
to write code but I don't seen any big problems with the approach for 
non-http, esp. xmpp - maybe Bob Wyman might have something to say about it.

+1 to this Pace.

cheers
Bill

---

From: "Roger B." <[email-masked]>
Date: Wed, 24 Nov 2004 13:06:37 -0600

> Rather than presupposing what developers are going to

Mark: FWIW, I've always had prev/next traversal enabled. This gets the
latest entries in my blog:

http://support.journurl.com/users/admin/

...and this gets the five previous entries:

http://support.journurl.com/users/admin/?month=11-01-2004&time=20:14:57

...and this gets the five before that:

http://support.journurl.com/users/admin/index.cfm?month=10-27-2004&time=15:32:52

So ignoring any broader implications, I don't have an immediate
problem with the stuff you're describing.

> channel, not portions of it. That's a huge benefit.

Not to those who only want their feed to serve as a source of update
notifications. Of course, those folks are well-served by RSS, so maybe
their usage isn't a big issue.

--
Roger Benningfield
blog: http://admin.support.journurl.com/

---

From: Graham <[email-masked]>
Date: Fri, 26 Nov 2004 01:27:39 +0000

Mark, what's the definition of "whole feed"? You're not seriously 
suggesting aggregators that don't automatically download 10 years of 
archives are in the wrong?

Graham

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 25 Nov 2004 21:30:10 -0800

Graham,

There is nothing in the proposal that requires sites to make all 
entries available, nor is there anything that requires clients to 
download them all.

---

From: "Bill Kearney" <[email-masked]>
Date: Fri, 26 Nov 2004 10:24:48 -0500

> entries available, nor is there anything that requires clients to

FUD aside, there is something to be said for clearing up the confusion about
what's a feed and how tools are expected to use as well as NOT use them.

That a source has a 'current items' docuemt that models RSS is what I think
most people are going to consider a "feed".  That it might contain
indicators on how to retrieve a "previous" instance of itself and/or a range
of additional (if not all) entries seems to be what you're proposing.  The
"prev" examples seem like reasonable ideas.  How a tool should "walk" itself
back over them it certainly something worth establishing some clear
practices on doing.  As in, let's avoid the insanity of feeds without
timestamps and the like.  Let something that has the 'current' feed detect
and retrieve the 'prev' document and perhaps use timestamps to assure that
it is or isn't 'later' that what is known locally.

As an aside, I can see real headaches for many current feed handling tools
in dealing with previous quantities of items.  They'll need to change and it
may not be pleasant for them.  That said, for a reader to be able to walk
back through the entries would be a very good feature.  Should a site want
to provide it, that is.

As to how a consumer of a feed should branch out an use some larger set of
if not all, entries is something worth discussing.  The idea of some
half-baked feed tool mercilessly polling a wholefeed URI every hour, on the
hour, is a nightmare we'd really do well to avoid.

-Bill Kearney
Syndic8.com