Subject: Re: Atom feed refresh rates

From: Graham <[email-masked]>
Date: Wed, 4 May 2005 15:44:08 +0100

On 4 May 2005, at 2:11 pm, Andy Henderson wrote:

>> But it's the only example I know of, of a large scale site that's
>> complained. Note they've never (as far as I know) said that it was  
>> costing
>> them money or causing them other problems, they've just arbitrarily
>> complained about "poorly written" programs and decreed how often  
>> people may
>> check their feed. There's no evidence of a technical problem here,  
>> just a
>> bunch of uptight twunts telling other people how to write their  
>> software.<<<
>
> I'm no expert on the provider side, but I believe it to be a wider  
> issue.
> There was some discussion at http://regsched.bookinfo.info/ and some
> attempts to implement bandwidth throttling but it petered out  
> because, it
> seemed to me, the problem was very difficult to solve from the  
> provider's
> end.

A quick look at that site turned up only one other site actually  
complaining, MSDN, and they changed their minds:

Scoble's post was widely debated by bloggers before being corrected  
by Microsoft's Sara Williams. "In a nutshell, our RSS traffic is  
neglible compared to all the traffic generated by Windows Update,  
MSN, downloads, and the rest of microsoft.com," wrote Williams. "We  
were motivated to reduce the size of the blogs.msdn.com home page  
primarily for operational efficiency's sake."

According to that site, Boing Boing's RSS feed (one of the most  
subscribed to) uses 3GB a day, or about $1.20 at current prices.  
Bandwidth is only going to get cheaper, and designing a system that  
might save a few gigs is going to be about as relevant in the future  
as the LOWSRC IMG tag. I think the forward looking thing to do would  
be for Atom to ignore this issue.

Graham

---

From: Eric Scheid <[email-masked]>
Date: Thu, 05 May 2005 01:15:19 +1000

On 5/5/05 12:44 AM, "Graham" <[email-masked]> wrote:

> uses 3GB a day, or about $1.20 at current prices.

only in some parts of the world.

over here I'm paying 13.2 cents per K and reading from a recent bill
2,982.61 Kbytes cost me $393.79 AUD.

e.

---

From: Tim Bray <[email-masked]>
Date: Wed, 04 May 2005 09:35:16 -0700

> complaining, MSDN, and they changed their minds:

Actually, as I recall, last time this came up (proposed by Walter 
Underwood), someone pointed out accurately that RSS2 has had this 
functionality for a long time and that nobody ever really implemented 
it; thus there was a strong vote from experience against such a 
feature. -Tim

---

From: "Andy Henderson" <[email-masked]>
Date: Thu, 5 May 2005 08:15:10 +0100

>>>feature. -Tim<<<

There is no RSS2 feature I can see that allows feed providers to tell
aggregators the minimum refresh period.  There's the ttl tag.  That was, I
believe, introduced for a different purpose and determines the Maximum time
a feed should be cached in a certain situation.  The tag's usage has morphed
over time and, if more than 60 minutes, I assume it's a Minimum time; but
it's not surprising if feed providers are wary of using the tag in this way.

What has yet to be tried is a specific tag in the core feed standard that
promotes and determines good behaviour for aggregators refreshing their
feeds.  Even if it were to prove only a limited benefit, it would still be a
benefit.

Andy

---

From: Walter Underwood <[email-masked]>
Date: Thu, 05 May 2005 06:16:26 -0700

--On May 5, 2005 8:15:10 AM +0100 Andy Henderson <[email-masked]> wrote:
>
> here is no RSS2 feature I can see that allows feed providers to tell
> a feed should be cached in a certain situation. 

We need both a ttl (max-age) and expires. One or the other is appropriate
for different publishing needs. We also need to specify what you do with
those values, or you end up with a mess, like the RSS2 ttl meaning reversing
over an undocumented value (Yikes!).

> feeds.  Even if it were to prove only a limited benefit, it would still be a

It has been tried several ways, originally in robots.txt extensions and
also in RSS. It doesn't work. The model is not rich enough for publishers
or for spiders/aggregators.

Max-age/expires is already designed and proven. By page count, 20% of the
HTTP 1.1 spec is about caching. If we want to write a new caching/scheduling
approach, we can expect it to be a 20 page spec, plus an additional 10
pages on how to work with the HTTP model.

See the Notes section here for details on when to use max-age or expires,
and on the problems with calendar-based schemes.

  <http://www.intertwingly.net/wiki/pie/PaceCaching>

wunder
--
Walter Underwood
Principal Architect, Verity