Subject: A date-range query mechanism for Atom and RSS (solution to RSS bloat?)

From: "Kevin A. Burton" <[email-masked]>
Date: Sun, 12 Sep 2004 17:56:20 -0700

I posted two long entries over the last two days WRT the current 
blogs.msdn.com issue...

http://www.peerfear.org/rss/permalink/2004/09/12/HTTPDeltasRSSAndDifferentialContentFetching

and

http://www.peerfear.org/rss/permalink/2004/09/11/TheScalabilityOfFullContentFeeds

Quick summary....

HTTP deltas are too complicated (even though I used to be a huge 
advicate of HTTP deltas)...

> A potential solution would be to add additional HTTP headers to 
> specify a date range for a query. This would be similar to 
> If-Modified-Since but would have slightly different semantics.
>
> This is obviously a strawman proposal but we could use two additional 
> HTTP headers of ModifiedAfter and ModifiedBefore. ModifiedAfter would 
> return all posts made after a given date. If there are no feeds we 
> would return HTTP 304 (not modified). (I'll address ModifiedBefore 
> below).
>
> This also supplies uplevel and downlevel compatibility. For example a 
> pre-ModifiedAfter capable client can still use the URL to fetch the 
> feed but the old semantics would be used. A post-ModifiedAfter client 
> could also integrate with existing Atom and RSS feeds that do not 
> support ModifiedAfter|Before.
>
Also this ETag proposal:

    http://www.intertwingly.net/blog/2004/09/11/Vary-ETag

Is along similar lines but I still think is too complicated and doesn't 
offer as much flexibility.  Its better than the delta approach though... 
(IMO).

Kevin

-- 

Please reply using PGP.

    http://peerfear.org/pubkey.asc    
    
    NewsMonster - http://www.newsmonster.org/
    
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
       AIM/YIM - sfburtonator,  Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
  IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster

---

From: "Kevin A. Burton" <[email-masked]>
Date: Sun, 12 Sep 2004 18:21:46 -0700

Kevin A. Burton wrote:

> Is along similar lines but I still think is too complicated and 
> doesn't offer as much flexibility.  Its better than the delta approach 
> though... (IMO).
>

Just had a chance to review this:

http://www.peerfear.org/rss/permalink/2004/09/12/VaryETagExtensions

... sooooo close ;)

Kevin

-- 

Please reply using PGP.

---

From: "S. Mike Dierken" <[email-masked]>
Date: Sun, 12 Sep 2004 18:41:30 -0700

> The use of ModifiedAfter and ModifiedBefore can enable archive query.
> For example say you're subscribing to a feed for the FIRST time. It would
be nice to have the
> last 30 days worth of feeds. The current poll model doesn't support this
but if you specified a
> range between now and 30 days ago you'd be able to pull out an archive.
I started out thinking about date based queries, but then eventually decided
that the actual dates aren't that important & adds more coordination between
client & server (you have to know that date is important & what slot to use
for requests). I think an approach where the client just follows links is
easier - the 'change list' approach. But this hasn't been detailed fully yet
so it's still just theory.

> This enables more efficient and aggressive polling intervals. I could now
enable
> my aggregator to poll at 10 minute intervals with NO additional load
(except 304 not modified) on the
> server. As RSS becomes more of an event model this will become a
requirement.
I agree, but only for polling based event model. I see two approaches,
polling and pushing. We already have push in the form of ping notifications
today (trackback/pingback), they just need to be routed into a multi-layer
system where each layer has high fan-out. I'd like to see this happen and
have participants in each layer be in different domains. But that's for the
future... right now retrievals can be made more efficient as well as more
scalable, and we should work on that. Viewing RSS as an event model is
exactly where I started from & looking for the lightest weight approach for
GETs is where I'm at now.

----- Original Message ----- 
From: "Kevin A. Burton" [email-masked]

---

From: "S. Mike Dierken" <[email-masked]>
Date: Sun, 12 Sep 2004 18:53:32 -0700

The Atom feed definition already supports links to 'next' and 'prev'
collections of entries.
http://bitworking.org/projects/atom/draft-gregorio-09.html#rfc.section.5.4.1

I suggest that somewhere in a weblog there is some auto-discovery of a
resource which is an atom:feed that represents a change list of entries that
have been modified after the originating resource where auto-discovery
started. The change list feed could contain multiple atom:entry elements (as
any feed would) and would be one in a series of feeds - connected via the
<link rel='next'> element - that represent future changes changes.
Traversing this set of change lists feeds would allow a client to 'catch up'
to the state of the server. The very last feed referenced would represent
the 'future' - changes that have not happened yet. Retrieving that resource
would result in a 'no content' response and caching headers that indicate to
the client when new changes are likely to be available.