Subject: RFC3229+feed: Empirical Evidence of significant savings.

From: "Bob Wyman" <[email-masked]>
Date: Sun, 3 Oct 2004 19:25:59 -0400

Given that we've been running support for RFC3229+feed on our servers at
PubSub.com for a few days now, I thought it might make sense to gather some
statistics and see what the savings are. I've documented this analysis on:

http://bobwyman.pubsub.com/main/2004/10/massive_bandwid.html

A summary table of results follows:

Status %Requests %TotalBytes AvgBytes 

200       15%       85%       77,212 
226       10%       14%       18,974 
304       73%     
                              53,443

Clearly, use of RFC3229+feed (status code 226) results in very impressive
bandwidth savings. Given this, I think we should insist that every
aggregator developer support this. 

I believe that we should keep in mind use of RFC3229+feed when considering
how we expect the Atom Syntax to be used in the future. (i.e. The Atom
Documents which are returned by servers are likely to frequently represent
subsets of the actual Atom Documents stored on a server. The result will be
some obfuscation of "editorial" decisions which focus on the significance of
whether or not an entry is actually in any particularly instance of an Atom
Document.)

		bob wyman

---

From: Eric Scheid <[email-masked]>
Date: Mon, 04 Oct 2004 10:19:42 +1000

On 4/10/04 9:25 AM, "Bob Wyman" <[email-masked]> wrote:

> Given this, I think we should insist that every

-1

you've significantly raised the bar for next to minimal implementations of
the most basic utility code.

this is not something we should rush into by only considering one side of
the issue.

e.

---

From: Dare Obasanjo <[email-masked]>
Date: Sun, 3 Oct 2004 18:05:54 -0700 (PDT)

> aggregator developer support this. 

Unless the Atom working group became "the aggregator
licensing board" this sounds out of scope to me. Also
if the Atom working group wants to get in this
business there's a bunch of stuff I'd want to see
insisted upon first (e.g. gzip support,
If-Modified-Since, If-None-Match, etc) before adding
support for RFC 3229 to the list. 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #23
I will keep a special cache of low-tech weapons and train my troops in their use. That way -- even if the heroes manage to neutralize my power generator and/or render the standard-issue energy weapons useless -- my troops will not be overrun by a handful of savages armed with spears and rocks.

__________________________________
Do you Yahoo!?
Read only the mail you want - Yahoo! Mail SpamGuard.
http://promotions.yahoo.com/new_mail

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 04 Oct 2004 03:49:38 +0200

* Dare Obasanjo wrote:
>licensing board" this sounds out of scope to me.

No, it is common to define conformance criteria for specific classes of
products, for example SVG 1.1 defines "SVG Viewers", "SVG Interpreter",
"High-Quality Dynamic SVG Viewer", etc. it is perfectly in scope to de-
fine conformance requirements for Atom implementations.

---

From: Eric Scheid <[email-masked]>
Date: Mon, 04 Oct 2004 12:08:13 +1000

> fine conformance requirements for Atom implementations.

it may be in scope for the generic class of organisations which atom-pub WG
is, but is it in our specific charter?

e.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 04 Oct 2004 04:18:17 +0200

>the most basic utility code.

What do you mean by "significantly" here? Do you have any material on
the implementation cost of RFC 3229? How expensive is it in terms of
lines of code, code footprint, memory consumption, CPU usage, etc. to
implement RFC 3229? Or some other information on how important it is
to allow for "next to minimal implementations" in the specification?
If implementation of RFC 3229 is required, you can still have such
implementations, they would just be non-conforming -- which specific
problems might arise from that? It might yield in less implementations
but that is not neccessarily a problem, it would not contribute to the
success of the Atom format if conforming implementations are considered
misbehaving.

>this is not something we should rush into by only considering one side of

Generally true, but content providers could consider that it is too
expensive for them wrt to traffic fees if a client does not implement
RFC 3229 and thus refuse service to such clients without considering
the implementation cost for these clients either, in which case it
would not really help Atom implementations to be allowed to ignore RFC
3229. Do you think it would be unreasonable to expect that this could
happen?

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 3 Oct 2004 22:37:01 -0400

Dare Obasanjo wrote:
>> Clearly, use of RFC3229+feed (status code 226) results in very
>> impressive bandwidth savings. Given this, I think we should
>> insist that every aggregator developer support this. 
> Unless the Atom working group became "the aggregator licensing
> board" this sounds out of scope to me.
	Yes. I agree. It was not my intent to suggest that the working group
take such as position -- as a working group. In fact, when I wrote that "we
should insist" I meant "we" as individuals, not "we" as the Atom working
group.

> if the Atom working group wants to get in this business there's
> a bunch of stuff I'd want to see insisted upon first (e.g. gzip
> support, If-Modified-Since, If-None-Match, etc) before adding
	For an aggregator, the only incremental work needed to implement
RFC3229+feed beyond that which is needed to implement if-modified-since or
if-none-match is to include "A-IM: feed" in the request headers and be able
to handle a return code of "226 IM Used". Anyone who is implementing if-*
would be well advised to just chuck in support for RFC3229+feed while they
are doing it.

		bob wyman

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 04 Oct 2004 04:40:52 +0200

>is, but is it in our specific charter?

[...]
  The working group will also take steps to ensure interoperability,
  by [...] clearly nominating conformance levels for different types
  of software
[...]

---

From: Eric Scheid <[email-masked]>
Date: Mon, 04 Oct 2004 15:03:38 +1000

> would be well advised to just chuck in support for RFC3229+feed while they

two points: 

 * I see now that you are recommending the "A-IM: feed" approach, and (I
presume) not the "you'll get IM whether you asked for it or not" approach.
It is this latter to which my objection applies. If an implementation sends
"A-IM: feed" then it should know how to handle the response, no sympathy
from me if it doesn't.

 * There's more involved than just inserting "A-IM: feed" in the request
headers. An implementation would need to maintain it's own database of
previously seen entries, it might need to implement it's own policy for the
"sliding window", it would have to guess as to whether a not-present entry
has been retracted/deleted/etc and thus suppress republication, it would
need to have sufficient smarts to identify entries to avoid duplicates (easy
in Atom, not so in RSS). That's quite a bit more than a implementation of a
single version feed cache ... if 304 then use the cache, if 200 then dump
cache, use retrieved resource.

e.

---

From: Eric Scheid <[email-masked]>
Date: Mon, 04 Oct 2004 14:53:45 +1000

On 4/10/04 12:18 PM, "Bjoern Hoehrmann" <[email-masked]> wrote:
> the implementation cost of RFC 3229?

this is my thinking:

A "minimal" implementation of something that fetches atom feeds would simply
do a GET and then process it. Sometimes that simplicity is all that is
required, other times it would be egregiously impolite.

A "next to minimal" implementation would keep track of the last modified
date or maybe the ETag, and use If-Modified-Since or If-None-Match.

With RFC3229+Feed (as default, ie. not asked for), an implementation would
need to build in a database functionality, would need to be smart enough to
identify same/different items (more an issue for RSS than Atom, thankfully),
would need to implement some kind of editorial policy of just how big the
"sliding window" of persistance needs to be, needs to consider how to handle
entries which are not present not because they haven't been updated but
because they have been retracted ... and the sturm und drang goes on.

Simple use case: I want to syndicate the headlines from the NYTimes
technology section on my site. You're asking a hell of a lot of additional
coding to done simply to be able to say "here are the current headlines from
NYT".

> problems might arise from that?

The current non-beta version of one popular aggregator doesn't support
persistence, and thus RFC3229+Feed results in truncated lists of headlines.
That's a problem.

(thankfully the beta of the next version does implement persistence,
although it then introduces the problem of ever increasing counts of unread
items of which most are getting older and older and more irrelevant by the
day).

> Generally true, 

Indeed. My main objection was the one-sided form of the argument as
presented by Bob: "Here are the benefits, no mention of the costs, therefore
lets do it" (I'm paraphrasing, of course).

> but content providers could consider that it is too

I don't believe it's an all or nothing choice. For example, a publisher
could provide a static feed with just headlines or summaries, and a fuller
feed for clients that know how to handle truncated feeds. I also know
publishers that don't provide any feeds at all, and wouldn't even with
3229+Feed.

e.

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 4 Oct 2004 02:36:22 -0400

Eric Scheid wrote:
> I see now that you are recommending the "A-IM: feed" approach, 
> and (I presume) not the "you'll get IM whether you asked for it
> or not" approach. It is this latter to which my objection applies.
	You are correct. I am advocating that we (as individuals) encourage
aggregator developers to send "A-IM: feed" headers with their conditional
GETs. The additional development effort is trivial for that class of
aggregator that already maintains a local copy of the feed state and also
implements conditional GETs. Given that the additional effort is so minimal
and the benefits so great, it is hard to imagine any argument for members of
that class to refuse or neglect to implement RFC3229+feed.

> * There's more involved than just inserting "A-IM: feed" in the
> request headers.
	Please believe me when I say that I am well aware of the
complexities involved. However, there exist a substantial number of
aggregators that have already addressed these complexities. For that class
of aggregator, the additional work is trivial. Given that we're seeing 73%
of requests to PubSub.com result in "304 Not Modified", my guess is that
this class of aggregators is a very large one.

> it might need to implement it's own policy for the "sliding window",
> it would have to guess as to whether a not-present entry has been
> retracted/deleted/etc and thus suppress republication,
	Here you touch on a point that I think has not been well discussed
or documented to date. Neither RSS nor Atom provides any means for
indicating that something has been "retracted/deleted/etc." The model which
appears to be assumed is that you can publish entries to the "feed" but you
can't remove them. Feeds are "write only." Certainly, entries can be removed
from Atom Documents but there is no way to indicate whether they were
removed simply because they "fell off the edge" when the sliding window
moved forward or because of some explicit decision to withdraw them. It
would be useful to have a discussion of Atom "Architecture" somewhere that
made explicit statements concerning the model here...

		bob wyman

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Tue, 05 Oct 2004 08:17:08 +0200

> aggregator developer support this.

I agree. Insisting on it might be a bit hard, but recommending it to  
everyone is definately a good idea.

-- 
Asbjørn Ulsberg    -=|=-     http://virtuelvis.com/quark/
«He's a loathsome offensive brute, yet I can't look away»