Subject: High volume feeds - one possible solution

From: Danny Ayers <[email-masked]>
Date: Sat, 11 Sep 2004 19:58:24 +0200

The MSDN scenario has been bugging me. I've been going round and round
with thoughts of HTTP chaining and all sorts. But a possible
relatively low-effort approach finally occured to me:

Split the feed into (say) 4 smaller feeds, which are rotated (say) hourly -

Feed A gets all new items between 00:00 and 01:00
Feed B gets all new items between 01:00 and 02:00
Feed C gets all new items between 02:00 and 03:00
Feed D gets all new items between 03:00 and 04:00
Feed A gets all new items between 04:00 and 05:00
Feed B gets all new items between 05:00 and 06:00
and so on...

Each supports conditional GET. So if a client is making requests
hourly, the maximum amount of bandwidth needed would on average be
_at_most_ half the amount + a few bytes compared to that needed for a
single feed (e.g. feeds B and C have changed, A and D give 304s). The
number and size of feeds and the update rate could be tuned to
optimise against the source.

There is a potential problem. Client-side, this as it stands would
still appear as 4 different feeds, depending on the aggregator that
might be less than ideal - "Search MSDN for xxx" might not work as
expected.

But perhaps this is only a problem for RSS. I believe the existing
Atom <id> mechanism offers a way forward. The feed-level atom:id
identifies the *conceptual* feed, right? If this is treated
consistently with the entry-level id then there shouldn't be any
expectation of it returning anything on a GET. So why not have
something like this at top level in each document:

Delivered at http://example.org/msdnA -
 
<feed>
   <id>http://example.org/msdn</id>
   <link rel="previous" href="http://example.org/msdnD" />  
   <link rel="next" href="http://example.org/msdnB" />  
...
</feed>

Delivered at http://example.org/msdnB -

<feed>
   <id>http://example.org/msdn</id>
   <link rel="previous" href="http://example.org/msdnA" />  
   <link rel="next" href="http://example.org/msdnC" />  
...
</feed>

and so on.

This would provide enough information for the aggregator to discover
the rest of the feed data and recombine the entries locally into a
single feed representation. HTML autodiscovery would only need to
point to one of the retrievable docs.

An alternative would be a single master list, but using the same 4
URLs, 1 URI approach.

Shouldn't be difficult to implement server-side. If the current setup
generates static docs, the feed-level shell of each doc is unchanging
and could be templated, just switch template & target filename. If the
feed is generated dynamically from say a SQL DB, then the servlet/asp
or whatever for each doc would use a slightly different query.

Bit trickier to set up client side, but as most aggregators already
support some form of mixing of items from different feeds, shouldn't
be rocket science. In any case, for client-side tools that didn't
support recombination, they just get 4 separate feeds, not a big deal.

Thoughts?

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: "Bob Wyman" <[email-masked]>
Date: Sat, 11 Sep 2004 14:43:02 -0400

Danny Ayers wrote:
> The MSDN scenario has been bugging me. ... Split the feed into
> (say) 4 smaller feeds, which are rotated (say) hourly - ...
> for client-side tools that didn't support recombination, they
> just get 4 separate feeds,
	If you're going to increase the complexity of the server software in
this manner, why just go the tiny further step of supporting RFC3229 delta
feeds with an instance-manipulation method like "feed-delta" which, combined
with Etag, could be used by the server to select the correct file to serve
up? The method you propose pushes a good bit of complexity out to the client
(i.e. the recombination problem and the problems of dealing with multiple
feeds) when RFC3229 already provides a standard method to avoid this
problem. There are far fewer servers then clients. If possible, complexity
should be limited to server -- not pushed to the clients.

		bob wyman

---

From: Danny Ayers <[email-masked]>
Date: Sun, 12 Sep 2004 11:08:10 +0200

> should be limited to server -- not pushed to the clients.

The main advantage of the approach I suggested is that it would be
possible to layer on top of current systems. I think perhaps you
exaggerate the client-side complexity. A client will already combine
data from document A at time X with data from document A at time Y, so
the step needed to combine data from document A with data from
document B (which will be essentially the same internally) can't be
that great, surely? The "problem of dealing with multiple feeds" has
already been solved by most aggregators ;-) Like I said, worst case is
a client sees multiple feeds - that would work *now*.

I've only skimmed RFC3229 so far, though appreciate that sending
deltas is the ideal solution to a problem like MSDN's. However the
impression I get is that implementation would involve more than a
"tiny further step". There appears to be quite a lot of new headers
and corresponding behaviours to take on board. Also I get a very mixed
message about proxy support - if the diff functionality is lost by a
large proportion of intermediaries, then there would be little net
gain.

I haven't yet figured out quite what would be needed to support this
in a typical system - say WordPress (PHP) on Apache serving, with a
desktop client like RssBandit (C#). Are standard libraries available
that could be plugged into such systems to provide delta support?
(Googling got me nowhere).

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 12 Sep 2004 08:09:32 -0700

At 11:08 AM +0200 9/12/04, Danny Ayers wrote:
>A client will already combine
>that great, surely?

Sure it can. The person running the client would need to know which 
feeds are supposed to be multiplexed, the URIs of each of the partial 
feeds, and so on. That is much more complex than "here's the URL for 
the feed to this high-volume list you want".

>  The "problem of dealing with multiple feeds" has

It would only "work" in the sense that you're presenting the user 
with something they don't want.

>"tiny further step".

It's a one-time step taken behind the backs of the users.

>  There appears to be quite a lot of new headers
>and corresponding behaviours to take on board.

Not as many as clients and servers deal with now. :-)

>  Also I get a very mixed

I would be very interested to see hard numbers on HTTP proxies in 
use. I get the impression that it is low, and when used in large 
environments such as major ISPs, the proxies are reasonably 
well-behaved.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Tim Bray <[email-masked]>
Date: Sun, 12 Sep 2004 09:28:28 -0700

http://www.intertwingly.net/blog/2004/09/11/Vary-ETag

  -Tim

---

From: Danny Ayers <[email-masked]>
Date: Sun, 12 Sep 2004 18:46:53 +0200

> the feed to this high-volume list you want".

The person running the client wouldn't need to know anything except a
single starting URI (for autodiscovery). The client software would
have to maintain a set of URIs per source, and instead of checking a
single feed at update time would have to check each on the list. The
multiplexing would be trivial as the order of entries isn't
significant.

> with something they don't want.

Depends on the UI. If it's 3-pane, then sure, instead of :

MSDN (3)

they might see

MSDN A (0) 
MSDN B (1) 
MSDN C (2) 
MSDN D (0) 

If such a UI supported any form of categorization (tree-based
categories are quite common), the multple feeds could be viewed as a
single combination.
If the UI was simply geared towards showing most recent items, there
would be no real difference whether the entries came from a single
feed or many.

> It's a one-time step taken behind the backs of the users.

Ok, but that doesn't make it any easier. The fact that the benefits
only appear after work at both client and server is significant too.
Don't get me wrong, I like the sound of RFC3229, I was just trying to
find a low-effort solution to the immediate problem.
 
> Not as many as clients and servers deal with now. :-)

True enough. But though gzip and conditional GETs can be explained on
one page, still far from everyone implements them, and a major
software company can still screw up their compression thanks to HTTP
version numbers.

> environments such as major ISPs, the proxies are reasonably

I'd like to see some more practical explanation of how to implement
RFC3229 at client and server for Atom. I realise there are potentially
significant benefits, but not only are the costs in terms of
implementation unclear, it isn't even clear how much work will be
needed to specify Atom+RFC3229.

So, Question #1: Which diff algorithm(s)?

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 12 Sep 2004 10:23:12 -0700

At 6:46 PM +0200 9/12/04, Danny Ayers wrote:
>single starting URI (for autodiscovery).

Sorry, I didn't catch that you were using autodiscovery in your 
proposal. This would also require an extension to autodiscovery for 
multiplexing, yes?

>categories are quite common), the multple feeds could be viewed as a

Correct, but you would be forcing another layer in the tree. That's 
user-visible.

>But though gzip and conditional GETs can be explained on

Fully agree. The implementer's guide would need to be very 
dirt-simple about this.

--Paul Hoffman, Director

---

From: Danny Ayers <[email-masked]>
Date: Sun, 12 Sep 2004 19:58:39 +0200

> proposal. This would also require an extension to autodiscovery for

No. I wasn't really proposing *anything* new, just suggesting how the
existing specifications (with a little implementation work) could be
used to alleviate MS's problem.  Autodiscovery would work as currently
specified to identify one feed source URL, the others would be
availabe thanks to the existing definitions in the Atom protocol for
<link rel="prev" and "next".

> Correct, but you would be forcing another layer in the tree. That's

Yes, which is why it would be better for them to support merging of
feeds with the same <id>, but possibly different FeedURIs.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Danny Ayers <[email-masked]>
Date: Sun, 12 Sep 2004 20:04:26 +0200

On Sat, 11 Sep 2004 14:43:02 -0400, Bob Wyman <[email-masked]> wrote:
> this manner, why [not] just go the tiny further step of supporting RFC3229...

A better answer occurred to me -  what I described at the start of
this thread will work with the existing Atom specifications. No spec
extensions needed.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 12 Sep 2004 14:29:20 -0400

> I've only skimmed RFC3229 so far, though appreciate that sending
deltas
> is the ideal solution to a problem like MSDN's. However the impression
> I get is that implementation would involve more than a "tiny further
step".
	Sam Ruby reports that in discussions at Foo Camp, they have come
up with a quite elegant approach to providing RFC3229 support for Atom.
The basic idea is to implement an entry-oriented "instance-manipulation
method" and then simply turn it on by default. This would be possible
with an entry-oriented IM method since the resulting delta would be
syntactically identical to what a client was expecting to receive
anyway! See:
http://www.intertwingly.net/blog/2004/09/11/Vary-ETag

> Are standard libraries available that could be plugged into such
systems
> to provide delta support? (Googling got me nowhere).
	I'm not aware of any standard libaries to do this. But, we're
going to get it done... The savings could be huge once this is
implemented.
	I've already mentioned to a number of folk that PubSub.com is
willing to offer a $5,000 bounty to whoever builds the "best" full
implementation as an open source Apache module by Jan 1, 2005. If you'd
like to help in defining the rules of the contest, help do judging, etc.
let me know...

		bob wyman

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 12 Sep 2004 14:59:39 -0400

Danny Ayers wrote:
>I've been going round and round with thoughts of HTTP chaining and
> all sorts. But a possible relatively low-effort approach finally
> occured to me:
	Danny, the system you are proposing would work... There are some
rough edges, but I don't doubt that they could be smoothed out with a
bit more discussion. However, I'm also sure that I really, really hope
we don't go this way!
	I must admit that my perspective is a bit different from that of
most folks. My system reads well over 3 million RSS and Atom files,
often multiple times, every day and that number is growing rapidly. I
would much prefer an RFC3229 based system since it means that much of
the bookeeping to get proper deltas computed is left to the server --
rather than forcing us (as a reader) to do the link crawling that your
proposal requires. I would also prefer an RFC3229 based solution since
it means that I don't have to open up multiple connections with remote
sites or depend on HTTP persistent connections (they are not always
available...)
	What you are proposing would work, but I think it would make
life much more complex for readers that consume large numbers of feeds.
Can you explain why your proposal is clearly superior to using an
RFC3229 based method? If we accept Sam Ruby's suggestion that
entry-oriented instance-manipulation be turned on by default when
serving Atom, can you still define a clear benefit to your approach?

		bob wyman

---

From: Danny Ayers <[email-masked]>
Date: Sun, 12 Sep 2004 21:07:26 +0200

On Sun, 12 Sep 2004 14:29:20 -0400, Bob Wyman <[email-masked]> wrote:
[snip]

> like to help in defining the rules of the contest, help do judging, etc.

Very cool.

But let me see if I've got the *result* of Sam's proposal straight.

If my aggregator has visited a feed, and got the following:

<feed/><1/><2/><3/></feed>

another entry is added to the feed, then it revisits, the ETag voodoo
will just give :

<feed/><4/></feed>?

and all the ETag management (except for retaining the last one) is
done server side?

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: "S. Mike Dierken" <[email-masked]>
Date: Sun, 12 Sep 2004 12:11:44 -0700

> multiplexing would be trivial as the order of entries isn't
I haven't seen anyone really describe an approach that used URIs rather than
headers - Danny is advocating it, but maybe I missed the details.
I like the approach of a set of resources that model a 'change list' rather
than a single resource for 'the mondo feed'. The single 'mondo feed'
resource approach is the one that seems to be having problems, even though
it is extremely convenient to have just one URI.
The goal of client applications seem to be to 'catch up' on changes - and so
a change list sounds very appropriate. The change list is basically a linked
list using <link rel='prev'> and <link rel='next'> style connections between
a resources that describe a list of changed items. For a personal blog, the
list of changed items likely would have just one entry, but for a service
that aggregates entries or creates a subject-based system, multiple entries
likely would be changing around the same time.

A client would maintain state - just as in an eTag approach - but the state
would be the URI of the most recent change list. As it scans through the
lists (via 'prev' and 'next' tags), when it falls of the end of the list it
has 'caught up'. The last resource would be empty (a 'no content' response
code) and could easily have a cache header indicating when the client should
next get changes. The resources that are change lists would be static -
meaning they are infinitely cacheable and sharable.

---

From: Danny Ayers <[email-masked]>
Date: Sun, 12 Sep 2004 22:21:20 +0200

> bit more discussion. However, I'm also sure that I really, really hope

I'm by no means advocating it as a general solution, just something
relatively low-effort that could be done (I think) within the current
specs.

> the bookeeping to get proper deltas computed is left to the server --

For sure. 

> proposal requires.

What I suggested would need a once-only following of <link rel="next"
and "prev", and those constructs are already in the protocol spec, so
may be something you'd need to consider anyhow.

 I would also prefer an RFC3229 based solution since
> RFC3229 based method? 

I wasn't suggesting it was! The only advantages I can think of are
that it can be implemented now following the current spec, and it'd
probably be easier on top of existing software where an Apache module
wasn't an option. Assuming Sam's suggestion hasn't got any subtle
flaws (I can't see any, but do find the algorithm hard work) it's a
much better fit for the kind of situation you're in.

If we accept Sam Ruby's suggestion that
> serving Atom, can you still define a clear benefit to your approach?

The multiplexed feed approach (lifting Paul's description for coolness
;-) may turn out to be useful in certain circumstances - the way S.
Mike has characterized it is interesting -  but I really wasn't trying
to put it forward as an alternate proposal for the spec, not as an
either/or. Just something that can be done with Atom.

If the deltas are doable, bring 'em on!

Cheers,
Danny.

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 12 Sep 2004 16:37:40 -0400

S. Mike Dierken wrote:
> I like the approach of a set of resources that model a 'change list'
> rather than a single resource for 'the mondo feed'.
	Change lists, either monolithic or linked, are an appealing
concept and could be made to work. Unfortunately, I don't think they buy
us very much.
	We need to recognize that feeds range from the inactive which
might be updated once a week or maybe only once every few months, to the
very active. Very active feeds are those like blogs.msdn.com that is
updated on average every five minutes or the LiveJournal consolidated
feed that has as many as 400 updates every minute...
	1. If the change list is monolithic, i.e. a file with nothing
but pointers to other files, then, eventually the file becomes so large
that you have to start thinking about things like using RFC3229 to
retrieve deltas to the change list or you need to implement the change
list as a linked list. Given 400 updates per minute, the LiveJournal
"changelist" would rapidly grow difficult to handle.
	2. If a chained or linked approach is used, on active feeds, you
would end up having to request potentially dozens or hundreds of
additional entries every time you polled the feed. Even with HTTP
persistent connections, this will result in a great deal of wasted
bandwidth -- if only in passing the names of the entry files to fetch.
Also, such an approach requires quite a few file "opens" which, on many
systems, can be expensive. (Yes, I realize that a variety of in-memory
cache daemons exist and their use would reduce file i/o at the cost of
expensive main memory.)
	While I see that "change list" and "linked file" approaches can
work, I don't see them as superior to implementation of delta feeds ala
RFC3229. In any case, these approaches increase client complexity and do
not deliver benefits until clients actually adopt them. On the other
hand, Sam Ruby's "RFC3229 instance-manipulation on by default" approach
would result in immediate bandwidth utilization whether or not clients
were aware of the mechanism.

		bob wyman

---

From: "S. Mike Dierken" <[email-masked]>
Date: Sun, 12 Sep 2004 16:02:11 -0700

> but pointers to other files, then, eventually the file becomes so large
I'd use multiple chained change lists, not one single unbounded resource.

> Given 400 updates per minute, the LiveJournal "changelist" would rapidly
grow difficult to handle.
With one changelist per minute, with 400 items noted within it (either
references or full item data), there would not be one ever growing resource.

> additional entries every time you polled the feed.
That depends on how many items are batched within a changelist. Active feeds
could put 400 items in a single changelist. The changelist would link to the
future changelist. The client would follow that link - the server would
respond with a 'no content' and a caching header telling the client to back
off for some period of time. This algorithm of following links allows a
client to catch-up regardless of how long they've been offline - one minute,
one day or one week. Using eTag would also work to allow a client to catch
up, but the responses would grow larger as more out of date clients
requested the singular feed resource. It's hard to tell which would work
better without reall data & running the algorithms. A single large resource
might use fewer network round trips than multiple resources, but multiple
resources might be more easily distributed or cached. I'm not sure.

> Even with HTTP persistent connections, this will result in a great deal of
wasted
> bandwidth -- if only in passing the names of the entry files to fetch.
That depends on whether the changelist has references or full (or summary)
item data. And each client only retrieves that data at most one time, and
caching proxies help sharing (to the extent proxies are available and used).

> cache daemons exist and their use would reduce file i/o at the cost of
Main memory isn't that expensive. I know some large ecommerce sites that use
a lot of main memory caching to provide performance.

> were aware of the mechanism.
I really do like the idea of allowing clients to inherit this coolness
without making a change. But I also like the idea of pursuing the idea of
change lists and what a practical approach to them would be.

---

From: Eric Scheid <[email-masked]>
Date: Mon, 13 Sep 2004 09:27:50 +1000

On 13/9/04 6:37 AM, "Bob Wyman" <[email-masked]> wrote:

> RFC3229. 

Bob, it's NOT an either/or proposition. You can have BOTH in effect, which
would deliver even greater efficiencies... and then throw that result
through gzip :-)

You can have a "change list" feed which contains only those entries actually
changed since last requested (via whatever delta mechanism). So, instead of
a SimpleFeedFormat of 25 entries (already very light bandwidth), you might
get back a SimpleFeedFormat of just 5 entries.

> In any case, these approaches increase client complexity and do

I'm not sure if the client I'm using uses if-none-match or
if-modified-since. If it does use ETags, then Sam's approach would break my
client - it forgets entries which are not in the current feed instance. I
would find that really annoying.

e.

---

From: Tim Bray <[email-masked]>
Date: Sun, 12 Sep 2004 17:23:07 -0700

> would find that really annoying.

ETags.  The savings for the big providers are so compelling that 
<strong>this will happen</strong>.  So I suspect that your client's 
behavior is about to become regarded as a bug. -Tim

---

From: Robert Sayre <[email-masked]>
Date: Sun, 12 Sep 2004 20:32:37 -0400

> behavior is about to become regarded as a bug. -Tim

Eric's client is using the server and the server's bandwidth for feed 
storage. Tough luck, you get what you pay for, etc, etc.

Robert Sayre

---

From: Eric Scheid <[email-masked]>
Date: Mon, 13 Sep 2004 10:43:09 +1000

On 13/9/04 10:23 AM, "Tim Bray" <[email-masked]> wrote:

>> I'm not sure if the client I'm using uses if-none-match or if-modified-since.
>> If it does use ETags, then Sam's approach would break my client - it forgets
>> entries which are not in the current feed instance. I would find that really
>> annoying.
>> 
> ETags.  The savings for the big providers are so compelling that <strong>this
> will happen</strong>.  So I suspect that your client's behavior is about to
> become regarded as a bug. -Tim

If a server could use "Vary: in-none-match" to provide the behaviour
outlined by Sam, could it not also use "Vary: if-modified-since"? If so,
that rules out conditional gets for all simple implementations of "show me
the recent changes of this source".

This is a fundamental change in the nature and behaviour of feeds. I think
we should tread carefully here.

e.

---

From: Sam Ruby <[email-masked]>
Date: Mon, 13 Sep 2004 02:06:41 -0400

> we should tread carefully here.

If this were, in fact, a fundamental change, then you should be able to 
find a section, paragraph, sentence, or even a word in at least one of 
the existing specifications that would need to change in order to enable 
this behavior.

Can you?

- Sam Ruby

---

From: Eric Scheid <[email-masked]>
Date: Mon, 13 Sep 2004 16:13:47 +1000

> storage. Tough luck, you get what you pay for, etc, etc.

so much for asking a server for a list of their headlines and, um, what's
the word, ... "syndicating" them, say onto your own website

well, you still can, you just can't rely on using conditional get, not
without maintaining your own per-entry store of previous responses, local
policies for expiration, etc ... something so simple is getting complicated

and this now raises another question: if a publisher pulls an
entry/headline, deleting it from it's feed, how is a syndicator of those
headlines meant to know that? That is, how to tell that non-inclusion in the
conditional get response from the other non-inclusion of the conditional get
response.

thus, if I want to syndicate the headlines from (say) the New York Times,
with links to the nyt.com articles, I would be forced to use non-conditional
requests each time. Now we're going backwards in the bandwidth battle.

Having said all that, it's still an interesting idea. I just think we need
to think this through a bit more carefully.

e.

---

From: Eric Scheid <[email-masked]>
Date: Mon, 13 Sep 2004 16:23:59 +1000

> the existing specifications that would need to change in order to enable

are you saying that RSS is comprehensively specified?

e.

---

From: Sam Ruby <[email-masked]>
Date: Mon, 13 Sep 2004 10:45:30 -0400

> are you saying that RSS is comprehensively specified?

No.  I am not the one making the assertion of a "fundamental change".

In any case, this is either something that is covered by a spec, and 
therefore could conceivably be a fundamental change, or this is 
something that you have assumed.

If the former, then you should be able to cite the relevant portion of 
the spec that will change fundamentally.

If the latter, then that assumption could conceivably turn out to be false.

- Sam Ruby

---

From: Antone Roundy <[email-masked]>
Date: Mon, 13 Sep 2004 09:13:21 -0600

While it would be easy to find the various feeds using prev & next, how 
would you communicate the fact that the client needs to keep polling 
each of the URLs to get all the updates?  My expectation based on the 
spec text

    start: The Atom feed at the URI supplied in the href attribute
       contains the first feed in a linear sequence of entries.
    next: The Atom feed at the URI supplied in the href attribute
       contains the next N entries in a linear sequence of entries.
    prev: The Atom feed at the URI supplied in the href attribute
       contains the previous N entries in a linear sequence of entries.

is that in a linear sequence of entries, new (and perhaps changed) 
entries are going to appear at one end, not at multiple places in the 
middle, and thus I would only continue to poll one URL.  I don't see 
how prev and next can be used both to enable catching up and for 
multiplexing.

BTW, we might want to be more specific about whether new (and changed?) 
entries should go into the "start" URL, or into the last URL pointed to 
be "next".  The spec text leads me to believe that new material is 
going to go into the last thing pointed to by "next", but the 
convention of adding new entries to the top of a feed (which feeds like 
the "start" rather than the "end") might suggest the opposite.

---

From: Danny Ayers <[email-masked]>
Date: Mon, 13 Sep 2004 18:06:46 +0200

> how prev and next can be used both to enable catching up and for

 I take your point, but it doesn't seem to be made in the current
spec. It would be possible (though awkward) to say the "linear
sequence" is that of entries received in each timeslot.

So, what are the use cases for "next" and "prev" in catching up? What
precludes the use of chained "next"s?

> the "start" rather than the "end") might suggest the opposite.

Good point, though I think the ambiguity runs deeper. The notions of
next and previous assume some specific order, without saying what it
is - what is the order of the "linear sequence"? This is also tied
into the 'sliding window' as Bob put it - that's the way things are
usually done, but it doesn't seem to have been made explicit anywhere.
I do hope this doesn' t send us back into the 'natural order'
debate...

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: "S. Mike Dierken" <[email-masked]>
Date: Mon, 13 Sep 2004 22:24:55 -0700

> I do hope this doesn' t send us back into the 'natural order'
I think it can be used for multiple orderings. One is 'recently changed
items', but doesn't rule out other orderings - I don't think baking in or
making visible a time-based ordering is actually required.

---

From: Henry Story <[email-masked]>
Date: Tue, 14 Sep 2004 08:28:47 +0200

Marking a feed as being ordered in a chronological sequence of changes 
is, as I see it, necessary for simple feed readers if they are to be 
able to catch up on changes.
If a feed reader does not know that a feed is chronologically ordered, 
and it has not read the feed for a while, it would logically have to 
read through each of the feed files to see if it had not missed an 
entry.

That would clearly use up a lot of bandwidth.

Henry

---

From: Eric Scheid <[email-masked]>
Date: Wed, 15 Sep 2004 14:18:51 +1000

On 13/9/04 10:32 AM, "Robert Sayre" <[email-masked]> wrote:

> Eric's client is using the server and the server's bandwidth for feed storage.
> Tough luck, you get what you pay for, etc, etc.
> 

On the contrary ... it's an unstated but obvious assumption that a system
that uses *conditional* GET would need to store the last retrieved
representation, just in case it receives 304 NOT CHANGED. If the system
didn't cache the last known retrieval, then it couldn't really use
conditional GET at all, could it?

What we have here is a system that wants to defer to the publisher as to
what the "current set of headlines" is. That is, what it would get if it
were to do an unconditional GET. Unfortunately, when that system wants to
play nice and start doing conditional GET (surely a good thing, right?), it
then gets punished.

Getting something done cheaper is no benefit if it gets "broken" in the
process.

e.

---

From: Dare Obasanjo <[email-masked]>
Date: Tue, 14 Sep 2004 21:51:08 -0700 (PDT)

> Getting something done cheaper is no benefit if it

If your client is fetching a feed repeatedly without
using conditional GET then it is a badly behaved
client and I'd expect it to eventually get banned by
the server. I don't see why we should be designing for
badly behaved and broken clients. 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #23
I will keep a special cache of low-tech weapons and train my troops in their use. That way -- even if the heroes manage to neutralize my power generator and/or render the standard-issue energy weapons useless -- my troops will not be overrun by a handful of savages armed with spears and rocks.

__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - Send 10MB messages!
http://promotions.yahoo.com/new_mail