Subject: How to implement the "Give me all the new & updated entries" use case

From: "Brian Smith" <[email-masked]>
Date: Sat, 14 Jul 2007 09:47:21 +0700

Hello all,
 
I have read through the Atom Syndication, Publishing Protocol, and Feed Paging & Archiving documents. I didn't find anything that allows the client to tell the server what entries it (the client) already has, so that the server can send just the new and/or updated entries. I feel like I must have overlooked something, because that seems like it would be the most frequent use case for APP. Is there a separate effort underway to standardize that? If so, where can I find information about it?
 
Thanks,
Brian

---

From: Steven Lees <[email-masked]>
Date: Fri, 13 Jul 2007 21:21:10 -0700

The APP spec suggests using ETag for this scenario. Take a look at examples 9.2.1 and 9.5.1 in the current draft.

There was some discussion about this on the Atom Protocol mailing list recently: http://www.imc.org/atom-protocol/mail-archive/msg08991.html, and Joe Gregorio wrote up a fairly thorough discussion on his blog: http://bitworking.org/news/150/REST-Tip-Deep-etags-give-you-more-benefits.

Steven

From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Smith
Sent: Friday, July 13, 2007 7:47 PM
To: 'Atom WG'
Subject: How to implement the "Give me all the new & updated entries" use case

Hello all,

Thanks,
Brian

---

From: "A. Pagaltzis" <[email-masked]>
Date: Sat, 14 Jul 2007 06:44:55 +0200

Hi Brian,

* Brian Smith <[email-masked]> [2007-07-14 05:05]:
> I have read through the Atom Syndication, Publishing Protocol,
> and Feed Paging & Archiving documents. I didn't find anything
> that allows the client to tell the server what entries it (the
> client) already has, so that the server can send just the new
> and/or updated entries.

the collection feed is ordered by app:edited, so the first
entries in it are the most recently changed ones, so they’ll
be the ones the client hasn’t seen.

The client can just page backwards through the collection
feed until it finds an entry that it already has.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: Eric Scheid <[email-masked]>
Date: Sat, 14 Jul 2007 14:28:26 +1000

On 14/7/07 12:47 PM, "Brian Smith" <[email-masked]> wrote:

> I have read through the Atom Syndication, Publishing Protocol, and Feed Paging
> & Archiving documents. I didn't find anything that allows the client to tell
> the server what entries it (the client) already has, so that the server can
> send just the new and/or updated entries. I feel like I must have overlooked
> something, because that seems like it would be the most frequent use case for
> APP. Is there a separate effort underway to standardize that? If so, where can
> I find information about it?

generally, we punted on dynamic search parameters, leaving that for
extensions (but we built in a framework for extensibility)

specifically though: with APP, collection feeds are provided with
most-recent edited entries first, so you simply request the collection feed
and you'll receive the first page of most-recently edited entries. You can
then compare atom:id and app:edited of those entries against what you
already have. If they are all more recent that yours, request another page
from the collection. Keep going back through the collection until you come
across a page containing entries which you already have and which were not
more recently edited.

There's also an advanced technique involving etags which keeps track of when
each etag was issued and if an old etag is sent with the request then
everything since then is sent back in the response. Someone will chime in
with the reference.

You wouldn't really want to use the mechanism you suggested though, not once
you've got a few thousand entries at the client (eg. sending a list of all
the atom:ids with their corresponding app:edited values). The above method
involves lots of round trips for the worst case (syncing the whole
collection), but minimises traffic for the more common case (sync relatively
recent changes).

e.

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 17 Jul 2007 06:22:59 +0700

Eric Scheid wrote:

> > and/or updated entries. [snip] Is there a separate 
> > effort underway to standardize that? If so, where can I 
> > find information about it?
> 
> generally, we punted on dynamic search parameters, leaving 
> that for extensions (but we built in a framework for extensibility)

Is there any effort underway on these extensions?

> specifically though: with APP, collection feeds are provided 
> with most-recent edited entries first, so you simply request 
> the collection feed and you'll receive the first page of 
> most-recently edited entries. You can then compare atom:id 
> and app:edited of those entries against what you already 
> have. If they are all more recent that yours, request another 
> page from the collection. Keep going back through the 
> collection until you come across a page containing entries 
> which you already have and which were not more recently edited.

Since APP is for editable feeds, it seems inappropriate for feeds like
mine that are read-only. As far as I can tell, the paging & archiving
internet draft is independent of APP, and can be used without APP. The
consequence is that the ordering of the entries within pages is
undefined, and so is the relative order of the pages themselves, since
there is no app:edited to sort by. (Actually, even with APP, app:edited
is optional according to section 10.2). The client would have to assume
that pages are ordered so that each "next" page contains earlier entries
(as determined by atom:updated, not app:edited).

In any case, the problem with using the paging mechanism is that client
support for paging is itself totally optional--a weblog client can be
Atom-compliant without supporting paging--and there is currently no way
for the server to know if the client has implemented that optional
feature.

In particular, if the paging mechanism is the only thing that is
available, then I would like my weblog feed to send one day's worth of
entries per page if the client supports paging, and a month's worth of
entries if the client does not support paging. Doing so would cut my
feeds bandwidth usage by an order of magnitude.

<Etags part moved to the end>

> You wouldn't really want to use the mechanism you suggested 
> though, not once you've got a few thousand entries at the 
> client (eg. sending a list of all the atom:ids with their 
> corresponding app:edited values). The above method involves 
> lots of round trips for the worst case (syncing the whole 
> collection), but minimises traffic for the more common case 
> (sync relatively recent changes).

I was thinking that the client would just have to send the server the
value of /atom:feed/atom:updated (XPath), and then it would receive a
feed with entries with a later <atom:updated> date. Perhaps it would not
get all of them if there were too many, but it would be guaranteed not
to get any earlier ones.

> There's also an advanced technique involving etags which 
> keeps track of when each etag was issued and if an old etag 
> is sent with the request then everything since then is sent 
> back in the response. Someone will chime in with the reference.

I understand that technique already but I'm not sure it is a good idea.
Has anybody implemented it? What are the drawbacks?

Regards,
Brian