Subject: Order of Entries within a Feed

From: "Becker, Matthew R" <[email-masked]>
Date: Wed, 23 May 2007 14:51:18 -0400

For performance reasons a client paging through a feed may wish for the
feed entries to be ordered in a particular way. Is there a recommended
technique to expose a feed that orders entries different from the
standard?

Thanks,

Matt.

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 23 May 2007 22:11:09 +0200

* Becker, Matthew R <[email-masked]> [2007-05-23 21:45]:
> Is there a recommended technique to expose a feed that orders
> entries different from the standard?

The standard prescribes no ordering, and in fact explicitly
states that the order of entries within a feed is meaningless.

This means, that…

> For performance reasons a client paging through a feed may wish
> for the feed entries to be ordered in a particular way.

… these clients are broken if they can’t handle arbitrary
ordering.

If you want to do such a thing, then you would have to define an
extension element (placed at the feed level, at the top of the
feed, along with the other feed metadata) that describes the
order.

A client could then use this optional information, where it is
present, in order to optimise its processing.

RFC 4287 itself makes no such provision, though.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: "A. Pagaltzis" <[email-masked]>
Date: Wed, 23 May 2007 23:16:22 +0200

* Becker, Matthew R <[email-masked]> [2007-05-23 21:45]:
> for the feed entries to be ordered in a particular way. Is
> there a recommended technique to expose a feed that orders

Btw, if you have so many entries in a feed that you have to use a
specific order in order for clients to be able to achieve
acceptable performance, then maybe what you actually want to do
is reduce the size of your feeds. The Feed History extension
draft would be of interest in that case.

Another option to investigate would be RFC3229+feed:
http://www.google.com/search?q=rfc3229-feed

Regards,

---

From: David Powell <[email-masked]>
Date: Thu, 24 May 2007 00:41:04 +0100

> RFC 4287 itself makes no such provision, though.

... and there is no guarantee that such an extension would work,
because by overriding RFC4287s assertion that a feed is unordered, the
extension would effectively have mustUnderstand semantics, which Atom
doesn't support.

Just because the publisher has added an <ex:ordered /> element,
doesn't mean that services like FeedBurner, Yahoo Pipes, and whatever
else haven't changed the order and passed the <ex:ordered /> element
through. If these services knew of the <ex:ordered /> element, then
they would know to strip it if they change the order, but you can't
expect those services to know anything more than RFC 4287.

... Having said that, Microsoft's Simple List Extensions for RSS [1]
might be useful. They allow you to specify a sort order for display of
a feed, and the default sort order additionally acts as an assertion
that the feed is physically sorted in that order.  The caveats above
still apply though.

[1] http://msdn2.microsoft.com/en-gb/xml/bb190612.aspx

-- 
Dave

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 24 May 2007 02:18:27 +0200

* David Powell <[email-masked]> [2007-05-24 02:05]:
> > A client could then use this optional information, where it
> > is present, in order to optimise its processing.
> 
> ... and there is no guarantee that such an extension would
> work, because by overriding RFC4287s assertion that a feed is
> unordered, the extension would effectively have mustUnderstand
> semantics, which Atom doesn't support.

How so? Pure RFC 4287 implementations would process the entire
feed as usual. Clients which understand the extension could skip
some of the processing based on the order of the entries. I see
no change in feed semantics that would affect oblivious clients.

> Having said that, Microsoft's Simple List Extensions for RSS
> [1] might be useful. They allow you to specify a sort order for
> display of a feed, and the default sort order additionally acts
> as an assertion that the feed is physically sorted in that
> order. The caveats above still apply though.
> 

FWIW, the `cf:treatAs/text()='list'` feature of that spec *does*
change the feed semantics in a way incompatible with RFC 4287.

The `cf:listinfo` feature set OTOH would be what Matthew Becker
is looking for, and can be used independently of the `cf:treatAs`
feature, and if it is, then by the aforementioned argument, the
caveat does *not* apply.

Regards,

---

From: James M Snell <[email-masked]>
Date: Wed, 23 May 2007 18:55:44 -0700

The most reliable approach here would be to create an extension that
allows the appropriate order to be recreated.  The index extension I had
pitched a while back took this approach.  It's most effective if the
feed publisher ensures that the entries are presented in the proper
order, but even if they are not, the client can faithfully recreate the
appropriate ordering.

- James

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 24 May 2007 05:44:47 +0200

* James M Snell <[email-masked]> [2007-05-24 04:15]:
> The most reliable approach here would be to create an extension
> that allows the appropriate order to be recreated.

My reading of Matthew Becker’s request is that he isn’t looking
for a way to sort the entries of a feed into a given order,
rather he wants to be able to make assumptions about the order
the entries appear in the feed, in order to optimise processing
in some way. That doesn’t leave room for sorting after the fact;
either the entries are in the right order for the optimisation to
hold, or they’re not.

Regards,

---

From: James M Snell <[email-masked]>
Date: Wed, 23 May 2007 21:06:11 -0700

Ok, I hadn't read the message that started this thread so I missed that.
 Thanks for making that clear.

- James

---

From: "Becker, Matthew R" <[email-masked]>
Date: Thu, 24 May 2007 07:30:23 -0400

My original message referred to a standard ordering. By that I was
referring to APP Draft 14, section 10 "The Entries in the returned Atom
Feed SHOULD be ordered by their "atom:updated" property, with the most
recently updated Entries coming first in the document order."

We have large feeds so we implemented paging according to section 10.1
of APP draft 14.

Assuming I've interpreted the APP draft correctly, we should be sending
paged feeds where the entries are ordered by atom:updated.

However we have certain clients that would like to see the feed entries
ordered differently (app:edited for example). If this were sorted client
side, the client would have to request every page in the feed and sort
the large number of entries. If the client could request the feed sorted
properly to begin with, then the client could stop paging when
app:edited reached the timestamp of the last request.

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of James M Snell
Sent: Thursday, May 24, 2007 12:06 AM
To: [email-masked]
Subject: Re: Order of Entries within a Feed

- James

A. Pagaltzis wrote:
> My reading of Matthew Becker's request is that he isn't looking
> in some way. That doesn't leave room for sorting after the fact;
> hold, or they're not.

---

From: Eric Scheid <[email-masked]>
Date: Thu, 24 May 2007 23:40:23 +1000

> paged feeds where the entries are ordered by atom:updated.

draft 15 has this: 

    The Entries in the returned Atom Feed SHOULD be ordered by their
    "app:edited" property, with the most recently edited Entries coming
    first in the document order. The app:edited value is not equivalent to
    the HTTP Last-Modified: header and cannot be used to determine the
    freshness of cached responses.

http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-15.html

e.

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 24 May 2007 15:54:01 +0200

> recently updated Entries coming first in the document order."

That has changed in -draft-15:

    The Entries in the returned Atom Feed SHOULD be ordered by
    their "app:edited" property, with the most recently edited
    Entries coming first in the document order. The app:edited
    value is not equivalent to the HTTP Last-Modified: header and
    cannot be used to determine the freshness of cached
    responses.

> Assuming I've interpreted the APP draft correctly, we should be
> sending paged feeds where the entries are ordered by
> atom:updated.

That is the correct interpretation of -draft-14, yes. Again,
though, it is moot, because of the -draft-15 changes.

> However we have certain clients that would like to see the feed
> entries ordered differently (app:edited for example).

If you also need feed ordering by something other than
`app:edited`, then you can’t do that in the collection feed.
There is, however, nothing to stop you from linking other, non-
collection feeds (ie. with a different `rel` value) from the
service document. You’d just need to specify in an extension how
clients can know the ordering of these feeds so they can request
the one they’d like.

> If this were sorted client side, the client would have to
> request every page in the feed and sort the large number of
> entries. If the client could request the feed sorted properly
> to begin with, then the client could stop paging when

That is exactly the rationale that was put forward for the change
in §10.

Regards,

---

From: "Becker, Matthew R" <[email-masked]>
Date: Fri, 25 May 2007 12:43:25 -0400

This approach sounds good. Thanks.
> service document. You'd just need to specify in an extension how
> the one they'd like.

Can anyone elaborate some more on why draft 15 is changing the standard ordering to be based on app:edited instead of atom:updated? Wouldn't feed readers be triggering off of atom:updated? Leaving that as the default sort order would allow for optimization in determining significantly changed content as opposed to ordering on app:edited where changes may or may not have been significant.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of A. Pagaltzis
Sent: Thursday, May 24, 2007 9:54 AM
To: [email-masked]
Subject: Re: Order of Entries within a Feed

If you also need feed ordering by something other than
`app:edited`, then you can't do that in the collection feed.
There is, however, nothing to stop you from linking other, non-
collection feeds (ie. with a different `rel` value) from the
service document. You'd just need to specify in an extension how
clients can know the ordering of these feeds so they can request
the one they'd like.

Regards,

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Fri, 25 May 2007 19:27:17 +0100

> Can anyone elaborate some more on why draft 15 is changing the standard ordering to be based on app:edited instead of atom:updated? Wouldn't feed readers be triggering off of atom:updated? Leaving…

If you search for the discussion you'll find a *very* long one so let's 
not start it again. Basically:

 * the WG felt collections needed at least one default sorting element
 * app:edited is the best one for a collection feed which takes place in 
the context of the AtomPub service itself

In other words, you should not consider the collection feed as the 
"public" feed to which you would like the public to subscribe to a feed. 
Instead consider the collection feed to the limited set of users that 
are interested in editing entries. If you need a "public" feed, create a 
distinct atom feed where you can sort the entries as you please. That 
feed would not be the collection feed, but a simple feed meaningless for 
an APP aware peer.

- Sylvain

---

From: "A. Pagaltzis" <[email-masked]>
Date: Sat, 26 May 2007 00:25:37 +0200

* Becker, Matthew R <[email-masked]> [2007-03-03 01:15]:
> Can anyone elaborate some more on why draft 15 is changing the
> standard ordering to be based on app:edited instead of
> atom:updated? Wouldn't feed readers be triggering off of
> atom:updated?

Aside from the fact that feed readers should not be polling the
collection feed in the first place, they don’t expect any
particular ordering anyway. How to order the feed is the
publisher’s choice, who can decide what entries aggregators
should have the best chance of staying up to date with.

> Leaving that as the default sort order would allow for
> optimization in determining significantly changed content as
> opposed to ordering on app:edited where changes may or may not
> have been significant.

There is no default order. RFC 4287 says no particular meaning
can be derived from the order of entries in a feed.

AtomPP imposes a constraint on that. The rationale for ordering
is to allow efficient synchronisation of local caches by AtomPP
clients.

If app:edited changes, atom:updated doesn’t necessarily.
If atom:updated changes, however, app:edited will also.

Therefore, if the collection feed was ordered by atom:updated,
minor updates could be buried arbitrarily far back in the
collection feed, and clients can that keep a local cache would
have to page through the entire collection to synch with the
server. In contrast, if it’s ordered by app:edited, all edits
(including significant) will bubble to the start of the feed, and
the client can stop paging as soon as it sees an entry that was
last edited before the last client synch.

Regards,

---

From: Bill de hOra <[email-masked]>
Date: Wed, 30 May 2007 20:28:54 +0100

>> semantics, which Atom doesn't support.

mU doesn't begin to cover what's needed to support forward 
compatibility.  mU provides a means to crash already deployed code; 
that's it.

> no change in feed semantics that would affect oblivious clients.

Exactly. Not having to understand random markup is a *feature*.  Atom 
doesn't have the the uniform semantics needed to make as yet 
non-existent "extensions" understandable. *By design* it's not Lisp; 
it's not even RDF.  [This is why I say xmlns provides modularity, not 
extension.]

cheers
Bill