Subject: Re: PaceDatePublished

From: Dare Obasanjo <[email-masked]>
Date: Thu, 16 Sep 2004 05:34:39 -0700 (PDT)

--- Eric Scheid <[email-masked]> wrote:

> More likely there are plenty of publishing tools
> that don't have a column
> for storing that value in their database tables.

Then they can add one. I fail to understand why
various people on this list seem obsessed with
ensuring that content producers don't have to change
any code besides renaming the names of generated
elements from their RSS feeds when producing Atom
feeds. 

> Surely you didn't mean that atom:published on every
> entry would be updated
> to DateTime.Now every time the feed is
> re-serialised?

No.  

> > They can keep the user entered date on the weblog
> but
> > they should use the creation date as the
> publishing
> > date for the Atom feed.
> 
> yes, if their publishing system stores that date.

Then they can change their publishing system. Do we
expect people to implement a new standard without
changing any code or database design? 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #23
I will keep a special cache of low-tech weapons and train my troops in their use. That way -- even if the heroes manage to neutralize my power generator and/or render the standard-issue energy weapons useless -- my troops will not be overrun by a handful of savages armed with spears and rocks.

		
__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - 100MB free storage!
http://promotions.yahoo.com/new_mail

---

From: Eric Scheid <[email-masked]>
Date: Thu, 16 Sep 2004 23:55:11 +1000

> elements from their RSS feeds when producing Atom

please don't count me in that group.

I will however acknowledge that the less hurdles (and no, that *doesn't*
mean *no* hurdles) publishers need to jump will result in quicker and wider
take-up. I fail to understand why various people on this list seem obsessed
with ignoring the ugly, brutal, and messy truth which is reality ;-)

e.

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Thu, 16 Sep 2004 21:07:37 +0200

On Thu, 16 Sep 2004 05:34:39 -0700 (PDT), Dare Obasanjo <[email-masked]>  
wrote:

> I fail to understand why various people on this list seem obsessed
> with ensuring that content producers don't have to change any code
> besides renaming the names of generated elements from their RSS
> feeds when producing Atom feeds.

+1. Nonetheless, this has been, and still is, the biggest problem we have  
with nailing stuff like dates. Current tools have mutable dates that could  
either be authorative, subjective or objective, some of them have  
timezones, some don't.

If current tools needs to be rewritten or extended, that should be very  
clear from the specification. One should think that <pubDate> in RSS 2.0  
was good enough specified with the words «Its value is a date, indicating  
when the item was published», but still, implementors use it for all kinds  
of things.

If Atom can break this blind «what I think I know is probably correct»  
implementation practice, that would be a good thing. But as Eric wrote,  
making implementation simpler by not breaking too much from current  
practice, is probably a good idea too. So we need to find a balance.

Defining a good and clear semantic with details on how the element should  
or must be used, plus making too demanding elements optional, might be a  
good way to go.

> Do we expect people to implement a new standard without changing any
> code or database design?

That seems to be, if not a goal, then at least a very strong wish, yes.

-- 
Asbjørn Ulsberg    -=|=-     http://virtuelvis.com/quark/
«He's a loathsome offensive brute, yet I can't look away»

---

From: "Roger B." <[email-masked]>
Date: Sat, 18 Sep 2004 10:41:31 -0500

I've been fiddling with something over the last couple days, and I've
finally installed some working code. What I'm wondering is if any aggregator
authors foresee my approach causing trouble for them.

NOTE: No, I'm not arguing that others should adopt this. It's just something
that achieves a degree of feed minimization with my existing app + ten lines
of new code.

Basically, this is it:

* IF I know you're requesting a feed
* AND you're sending If-Modified-Since
* AND you're not requesting a subset of the feed

* THEN I grab your I-M-S and round it down to the nearest hour
* AND I return Last-Modified == Now()
* AND I return Location == a feed subset containing posts newer than your
rounded I-M-S
* AND I return 302 

Anthing obvious that I'm overlooking? This won't be default behavior... it's
something that can be used in specific cases where posting- and
polling-frequency has created a problem. 

--
Roger Benningfield
JournURL: http://journurl.com/
blog: http://admin.support.journurl.com/

---

From: "Bob Wyman" <[email-masked]>
Date: Sat, 18 Sep 2004 13:35:38 -0400

Roger Benningfield wrote: [See message below]
	I think there are some serious issues with what you propose:
	It appears that you're saying that you would normally respond to
an "if-modified-since" in the same inefficient and wasteful manner that
feed servers have traditionally responded. I.e. You will return all
available items or entries for the feed if any item in the feed has been
created or modified since the if-modified-since time. However,  from
time to time, in "special cases", you will cause the server to force
temporary redirection to feeds that only contain those items or entries
that have been modified since the if-modified-since time. Although not
stated, it would seem that if the "special cases" end (for instance, if
request load goes down) you would revert to the traditional behavior
	What this means is that during times of heavy load, you will
reduce the size of any single transfer; however, you will double the
total number of requests that much be handled. (302 forces a second
request.) This could cause problems.
	Also, since you aren't indicating in any header or other means
that the feed has been modified for the duration of "special cases",
aggregators, caches, etc. are likely to get confused by the sudden
reappearance of the full feed once the "special case" is terminated.
	It seems to me that it would be much easier to simply implement
RFC3229[1] with the "feed"[2] instance-manipulation method that I define
on my blog and will soon write up as an Internet Draft. If you use the
RFC3229 approach, you would eliminate the need for the second request
required by 302, you would be explicitly telling requestors that the
"instance-manipulation" (i.e. abbreviation of the feed) had been
performed, and you would be making it possible for caches to do their
jobs.
	RFC3229 support may take more than "10 lines of code," however,
there are already a number of sample implementations[3] to use as a
model. From an implementation point of view, it would seem that you
already have most of what is needed to implement RFC3229 support. Since
you have abbreviated versions of the file already generated, simply
return the abbreviated version when you get a request. Set the response
code to "226 IM Used", insert a response header that says: "IM: feed"
and another with "Cache-Control: no-store, im" and you're done. It
couldn't be much easier.

	bob wyman

[1] http://www.ietf.org/rfc/rfc3229.txt
[2] http://bobwyman.pubsub.com/main/2004/09/using_rfc3229_w.html
[3] http://bobwyman.pubsub.com/main/2004/09/implementations.html

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Roger B.
Sent: Saturday, September 18, 2004 11:42 AM
To: 'Atom-Syntax'
Subject: Re: High volume feeds - one possible solution

I've been fiddling with something over the last couple days, and I've
finally installed some working code. What I'm wondering is if any
aggregator authors foresee my approach causing trouble for them.

NOTE: No, I'm not arguing that others should adopt this. It's just
something that achieves a degree of feed minimization with my existing
app + ten lines of new code.

Basically, this is it:

* THEN I grab your I-M-S and round it down to the nearest hour
* AND I return Last-Modified == Now()
* AND I return Location == a feed subset containing posts newer than
your rounded I-M-S
* AND I return 302 

Anthing obvious that I'm overlooking? This won't be default behavior...
it's something that can be used in specific cases where posting- and
polling-frequency has created a problem.

---

From: "Roger B." <[email-masked]>
Date: Sat, 18 Sep 2004 14:06:16 -0500

> feed servers have traditionally responded.

Bob: If you consider plain ol' conditional GET inefficient, then yeah. As
with a number of other folks, I don't see a problem with the 'inefficient
and wasteful" status quo... it's yet to come even close to causing a
problem. And if it does, I still have other steps (Gzip) that can be taken.

So this is strictly a "just in case" thing, not a pressing concern.

> Although not

That's all up to the individual user. I mean, sure, I can force the
redirect, but it's really just an option for individuals who think they need
it. If they want to make it the default behavior, they can.

> however, you will double the
> total number of requests that much be handled.
> This could cause problems.

It's possible, but the DB will break down long before IIS dies from an
overabundance of requests.

> reappearance of the full feed once the "special case" is terminated.

Neither my own aggregator nor Newzcrawler seem to have any particular
problem. They both see the 302, follow it, and pull the resulting feed items
into their databases. All of the same "have I seen this item yet?"
mechanisms apply.

So what I'm really looking for here are real-world examples of apps that are
going to catch fire if they get redirected to a truncated feed.

> RFC3229[1]...

Nope, it wouldn't. It would require a ground-up rewrite of my database
caching code, something that's not going to happen any time soon.

---

From: "Roger B." <[email-masked]>
Date: Sat, 18 Sep 2004 14:23:35 -0500

> request load goes down) you would revert to the traditional behavior

Bob: Let me elaborate here.

The behavior I've specified is triggered by the addition of a new key/value
pair to the query string. The idea is that if a user starts seeing a surge
in RSS traffic, she can change her autodiscovery and other links to include
the key/value pair, and future subscribers will see the new behavior across
the board. Existing subscribers won't see a change.

It's certainly *possible* for me to send a 301 and force compliant apps that
are subscribed to the standard feed to switch, and it's also possible to
force them to switch back the same way. But that's not what I'm talking
about at this time.

---

From: Walter Underwood <[email-masked]>
Date: Mon, 20 Sep 2004 14:01:20 -0700

> * AND I return Location == a feed subset containing posts newer than your

This is just plain weird. I go to fetch something that was a 200
last time, and now I get a 302. What? I don't want some other URL
with a delta, I want this resource, and I want to know if it is
up to date.

The 302 would also destroy cached copies in intermediate caches,
so it would increase the load on the destination server.

Please, please do not do this.

wunder
--
Walter Underwood
Principal Architect
Verity Ultraseek

---

From: "Roger B." <[email-masked]>
Date: Mon, 20 Sep 2004 17:28:46 -0500

> last time, and now I get a 302. What?

Walter: I'll cop to weird. :D I spent some more time fiddling with it today,
and the previously described behavior no longer applies. I found (pending
further testing) that I can get it pretty close to Sam's proposal, but with
time-based deltas instead of item-based... no 302 required.

---

From: Sam Ruby <[email-masked]>
Date: Mon, 20 Sep 2004 19:32:44 -0400

> time-based deltas instead of item-based... no 302 required.

If you have any sort of threading, or your server support multiple 
processes, then you can get into trouble.

FYI: my current implementation is different than than the hashing that I 
originally spec'ed.  What I do is capture the feeds and etags, and 
produce deltas from these known captures.  If you provide me an etag, I 
know what version you previously got, so can confidently provide you 
with a delta.  If for some reason, you provide me an etag of something I 
didn't happen to capture, I provide you with the whole feed.  I don't 
keep etagged' versions forever, so if you wait long enough, again you 
get a whole feed.

In short, while imperfect, it errs on the side of caution.  In most 
cases, there will be a performance benefit.  In all cases, you will get 
at least as much new information as you did before this support.

The key open question as I see it is whether or not to do this without 
the client's consent.  For clients that already support ETags and 
Last-Modified headers, the additional support required is as small as 
possible: one fixed header ("A-IM: feed").  To date, I have seen two 
clients (FeedDemon/1.2 Beta 1 and Bloglines/2.0) supporting this.

- Sam Ruby