Subject: Thought experiment: Using a synthetic feed to find out about other feeds

From: "John Panzer" <[email-masked]>
Date: Wed, 7 Jul 2004 18:11:00 -0700

All,

Note: I'm not attempting to continue the discussion about introspection
and feeds in this post, or at least I want to focus on a different
angle.  Let's ignore introspection completely for the moment (I pick it
up again in the P.S. below).

Suppose I want to offer a service in which people can subscribe to a
collection of pointers to Atom feeds.  Given that we already have RSS
feeds for doing things like tracking FedEx packages, this doesn't
seem too outlandish.  Sample use cases:

o List of blogs that I own and publish for others to read.

o List of blogs that I recommend/link to via an "other blogs" section,
that my friends can use to see what I'm reading.

o List of blogs that have linked to me via an "other blogs" section.

o List of "top 10" blogs by some metric.

Now, I can of course just compose a feed with regular entries, each of
which has a hyperlink off to the relevant web page.  But what if I want
to allow automatic processing of the feed into something else useful?

This sounds like another use for synthetic feeds; so, I'm lazily
borrowing from Bob Wyman's example synthetic feed below to see what
this might look like.  Imagine this is a feed which you can subscribe
to via http://example.com/bugsbunny/atom-feeds:

<?xml version='1.0'encoding="utf-8"?>
<feed version='0.3' xmlns='http://purl.org/atom/ns#' xmlns:ps='http://www.pubsub.com/xmlns'>
<link rel='alternate' type='text/html' href='http://example.com/bugsbunny'/>
<modified>2004-07-07T18:36:11-04:00</modified>
<author>
<name>Bugs Bunny</name>
</author>
<title><![CDATA[Bugs Bunny's Blogs]]></title>

<entry>
<ps:source-feed>
<title><![CDATA[Title of my First Blog]]></title>
<link rel='alternate' type='text/html' href='http://example.com/bugsbunny/MyFirstBlog'/>
<link rel="service.post" type="application/atom+xml" href="http:/example.com/bugsbunny/MyFirstBlog/atompost.cgi"/>
</ps:source-feed>
<modified>2004-07-07T19:34:10-04:00</modified>
<issued>2004-07-07T19:34:10-04:00</issued>
</entry>

<entry>
<ps:source-feed>
<title><![CDATA[Title of my Second Blog]]></title>
<link rel='alternate' type='text/html' href='http://example.com/bugsbunny/MySecondBlog'/>
<link rel="service.post" type="application/atom+xml" href="http:/example.com/bugsbunny/MySecondBlog/atompost.cgi"/>
</ps:source-feed>
<modified>2004-07-05T04:44:00-04:00</modified>
<issued>2004-07-05T04:44:00-04:00</issued>
</entry>
</feed>

o Note that this doesn't
make much use of real entry fields, except atom:modified and
atom:issued, which I'm imagining would be calculated in the obvious way
from the originating feeds' entries.

o Most of the interesting information is in the ps:source-feed
element.  Which can contain HTML URIs as well as endpoints -- taken
from the originating feeds -- giving URIs for, say, posting.  Or other
feed-level pieces of information.

The only oddity about this 'feed' is that, of course, the 'entries' are
themselves synthesized out of non-entry data.  But this is exactly what
happens in the FedEx tracking example.  So it's a synthetic feed of
synthetic entries, which doesn't seem unreasonable.

Note, of course, that it's perfectly possible to add atom:content
elements to the above with nicely formatted human readable HTML content
pointing at the relevant HTML web pages, if someone subscribes to this
feed in a 'regular' RSS reader.  This would be useful.

It would be even more useful, though, if someone used a 'feed tracker'
feature pointing at http://example.com/bugsbunny/atom-feeds;
this could do interesting things with the results, given the knowledge
that what it's pointing at is really a synthesized feed of other
(indirect) feeds.  For example, tracking new feeds for possible
subscription in a special area of the UI.  (Though actually that seems
like a generally useful feature for _all_ synthetic feeds...)

So, let me ask this:  Does this seem like a reasonable use of synthetic
feeds?  Does anyone see problems with this sort of usage?

-John Panzer

P.S.: If you look at the feed above closely, you might notice that it
contains elements that you might consider to be 'introspective'.  But
that's just a consequence of the synthetic feed format; it falls out
naturally, though of course the choice of precisely which elements to
include is up to the feed generator.  So, the question here is:  If
this is a reasonable use of the synthetic feed format, would it meet
most or all needs that would be met by a 'pure' introspection API?  I
dunno, honestly.  What do you think?

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 8 Jul 2004 12:14:54 -0400

> Suppose I want to offer a service in which people can subscribe to a
collection of pointers to Atom feeds.
	Let me add a few additional applications to John's list:
1. List of "new blogs" on a site. i.e. Whenever AOL Journals creates a new
journal (blog), it could add it to the list of "new journals" so that
services like Feedster, Technorati, PubSub, etc. that wish to discover and
then monitor these things could discover the new journals immediately. The
usage would be somewhat like what is done today with "new entry" feeds at
places like weblogs.com, blo.gs, livejournal, etc.
2. Cooperating blog searching or matching services could send "new atom
file" feeds to each other from time to time to propogate information about
the new feeds that they had discovered.
3. Search engines could return lists of blogs that tend to discuss certain
subjects.
4. FOAF applications could return lists of blogs "within three links" of
someone's entry.
5. Various services could build lists of "feeds updated during the last X
minutes."

       This proposal has some similarities to Ken MacLeod's recent proposal
for a <site/> element that would be linked to by entries. The similarity is
in making the <feed/> data stand on it's own as a first-class object
independent of any entries. Thus, it moves us toward RDF-in-spirit, if not
syntax: independent, reusable, re-combinable elements that can stand on
their own or be composed into aggregates that convey what we expect from a
feed today.
       
       	bob wyman

---

From: Ken MacLeod <[email-masked]>
Date: 08 Jul 2004 13:06:40 -0500

"Bob Wyman" <[email-masked]> writes:

> John Panzer wrote:
> > Suppose I want to offer a service in which people can subscribe to
> > a collection of pointers to Atom feeds.

> 1. List of "new blogs" on a site. i.e. Whenever AOL Journals creates
> a new journal (blog), it could add it to the list of "new journals"
> so that services like Feedster, Technorati, PubSub, etc. that wish
> to discover and then monitor these things could discover the new
> journals immediately. The usage would be somewhat like what is done
> today with "new entry" feeds at places like weblogs.com, blo.gs,
> livejournal, etc.
> 2. Cooperating blog searching or matching services could send "new
> atom file" feeds to each other from time to time to propogate
> information about the new feeds that they had discovered.
> 3. Search engines could return lists of blogs that tend to discuss
> certain subjects.
> 4. FOAF applications could return lists of blogs "within three
> links" of someone's entry.
> 5. Various services could build lists of "feeds updated during the
> last X minutes."

I think these cut right to the core of Beecher's suggestion of
defining how "a set of entries" differ from "lists", particularly "a
set of recently changed entries", and how an "entry" is different from
a "generic metadata record".

An aggregator works on the principle of displaying new entries when
they appear, and sometimes holding "old" entries for perusal, and
re-showing updated entries if they're changed.

I understand the interest in thinking of a "feed" as a "list" in
contrast to a "a set of recently changed entries", but to fit that
perception I always expect the justification for why that "list"
should be presented as "a set of recently changed entries" as a
default.

In this case, "a set of feeds" (which is really an "index", a complete
set of "sites" that are not merely "recently changed") and "a set of
recently changed feeds" don't match my concept of "a set of recently
changed entries".

I'm good with a "list of sites", that is either an "index" or a "set
of recently changed sites", and allowing a client to properly display
that to a user, such as, "site Foo just added feed Bar, subscribe?"

But when a "list of sites" is co-mingled conceptually with a "list of
entries", I can see an aggregator a) hiding all the feeds you already
know about, and showing you a new feed when it appears as a new entry,
but I'm perplexed by b) what the concept of "modified" should cause
the aggregator to do.  Should the aggregator re-display the "modified
feed" like it would a modified entry?  or should it know that it's a
feed and that feeds get modified all the time?  The current *default*
behavior of aggregators would be re-displaying "feed entries" every
time those feeds themselves had new entries!

  -- Ken

---

From: Eric Scheid <[email-masked]>
Date: Fri, 09 Jul 2004 10:27:41 +1000

On 9/7/04 4:06 AM, "Ken MacLeod" <[email-masked]> wrote:

> I'm perplexed by b) what the concept of "modified" should cause
> feed and that feeds get modified all the time?

and perhaps deliberately retrieve those feeds which are (a) modified, and
(b) already in your subscription list?

sounds a bit like a ping interface, but one which can penetrate firewalls
because it is pulled, not pushed; and one which could be tailored to only
reference feeds you are interested in.

e.

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 8 Jul 2004 21:38:24 -0400

Eric Scheid wrote:
> sounds a bit like a ping interface, but one which can penetrate 
> firewalls because it is pulled, not pushed; and one which could be 
> tailored to only reference feeds you are interested in.
	Right. But, if that is what you want, you could subscribe to
pings[1] using the Jabber/XMPP-based PubSub system that we provide at
PubSub.com.[2,3,4] The pings would be "pushed" to you, so you wouldn't
suffer the latency inherent to a polling solution. These pings would
penetrate the firewall since Jabber/XMPP opens a persistent connection from
the client within the firewall to the server. 
	By leveraging broadly accepted and deployed instant messaging
technology, we can do all sorts of things with Atom and syndication that
were much more difficult before. Atom over Jabber via PubSub offers a new
set of interesting ways to work with blogs and feeds...

		bob wyman

[1] You can get the effect of "pings" from a blog named
"http://example.com/foo.xml" by creating a PubSub subscription to
"SOURCE:example.com/foo.xml" (i.e. "Show me new items whose source is:
"example.com/foo.xml".) Using the Jabber/XMPP PubSub Interface, if you only
want the "ping" data, not the actual content of the new item/entry, just set
the subscription option "show-content" to the value "exclude" (actually,
that's the default so you don't need to set it...). (See the tutorial at
[2]).
[2] http://developers.pubsub.com
[3] http://developers.pubsub.com/pubsub_xmpp_draft.html
[4] http://sourceforge.net/projects/pubsubxmpp (open source C# code and
binaries for a really simple (and very "beta") client that uses the PubSub
Jabber/XMPP interface).

---

From: "Bob Wyman" <[email-masked]>
Date: Thu, 8 Jul 2004 21:37:09 -0400

Eric Scheid wrote:
> firewalls because it is pulled, not pushed; and one which could
> be tailored to only reference feeds you are interested in.

		bob wyman

---

From: Sam Ruby <[email-masked]>
Date: Fri, 09 Jul 2004 07:20:56 -0400

> re-showing updated entries if they're changed.

Some form of that statement should make its way into AtomTerminology[1].

- Sam Ruby

[1] http://www.intertwingly.net/wiki/pie/AtomTerminology

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Fri, 09 Jul 2004 15:15:50 +0200

> Some form of that statement should make its way into AtomTerminology[1].

Done. I've also sorted the definition terms alphabetically, to make the  
list more readable.

-- 
Asbjørn Ulsberg         -=|=-        [email-masked]
«He's a loathsome offensive brute, yet I can't look away»

---

From: "John Panzer" <[email-masked]>
Date: Wed, 14 Jul 2004 13:29:11 -0700

All,

I've been thinking about this a bit more, in between what my company 
considers my 'real' work. Reading the comments from Eric Scheid,  Bob 
Wyman, and Ken MacLeod, I don't see violent opposition to the general 
concept, but some concern about what "modified" means for feeds whose 
subject matter is sites, and why a general "list of things" is different 
from a "feed".

Ken seems to object to the use of a "feed of sites" as a synonym for 
"index of sites".  I agree that the two are different, and I think 
they'd serve different purposes to some extent, but that both will be 
useful.

A "feed of sites" is fuzzy, though -- what does that mean?  Let me 
suggest a more concrete underlying concept:  A "feed of site events". 
That is, a feed which consists of interesting events happening to sites 
separate from the addition/updating/deletion of entries.

The most important event that happens to a site, of course, it its 
creation.  In the "feed of site events" worldview, this would correspond 
to a new entry.  (Today, this new entry appears in places like 
http://bloglines.com/newblogs .)

(The "birth notice" event might also provides a convenient place to put 
some information about the timestamp of the latest entry inside the 
site, but that's really just an optimization for going to the feed URI 
and doing a HEAD request.  So let's say the "birth notice" is a frozen 
historical record.)

A second event that can happen to a site is deletion or quiescence.  In 
the "feed of site events" worldview, this would also correspond to a new 
entry.  (Today, this new entry typically is appended to the end of the 
feed itself, with human readable text.)

A third event is renaming or movement of the site.  This could provide a 
historical record of what happened, in addition to any redirection or 
final notices put on the original site.  (The original site will 
probably go away at some point.)  There are other possible events too.

Looking just at "birth" and "death" notices, a "feed of site events" 
would then give you a chronological journal of what's happening to a 
specific set of sites:

   Site1 born (metadata about site1)
   Site2 born (...)
   Site1 dies (URI relates this event to birth notice above)
   Site3 born

(One could also imagine corrections to the original birth notice and all 
the other edge cases that apply to entries.)

How does this relate to an "index of sites"?  The index gives you a 
snapshot in time (the current state of the world), while the "feed of 
site events" gives you the backstory.  Obviously, given the full 
backstory, you can reconstitute the current state of the world.  The 
index after all the events above would be:

   Site1
   Site3

It would clearly be easier for clients who are interested only in the 
state of the world today to deal with the site index.  However, there 
are use cases for the feed of site events too.  For example, knowing for 
_certain_ that a feed is going away or transmorgifying into something 
else would allow aggregators to help manage feeds more intelligently. 
There's an existing use case (http://bloglines.com/newblogs) for getting 
notified about new feeds, and I think we can all imagine variations of 
that theme.

Please note, I'm not at all arguing against having an "index of feeds" 
capability, whether it's via WebDAV (which seems like a nice, easy, 
standards based approach) or some other way.  I really have two points:

1) There are reasonable use cases for wanting "feeds of site events".
2) An "index of feeds" can be derived from "feeds of site events" but 
not the other way around.
3) Aggregators could use either or both for autodiscovery.

-John Panzer

P.S: I think that a "feed of site events" fits exactly into the

---

From: Ken MacLeod <[email-masked]>
Date: 14 Jul 2004 17:51:37 -0500

"John Panzer" <[email-masked]> writes:

> That is, a feed which consists of interesting events happening to
> sites separate from the addition/updating/deletion of entries.

This has been discussed before, on the wiki look around
RevisedConceptualModel, ConceptualModelRevisited.  Some head messages
in the list archive:

    http://imc.org/atom-syntax/mail-archive/msg01029.html
    http://imc.org/atom-syntax/mail-archive/msg01046.html
    http://imc.org/atom-syntax/mail-archive/msg01068.html
    http://imc.org/atom-syntax/mail-archive/msg01075.html

These refer specifically to the model of "event streams", or feed
elements that represent changes of state of resources.  (Note: not
talking about events in the sense of "football game" or "conference".)

The distinction is subtle but obvious in hindsight:

  * The current model of "feed of entries" is copies of or references
    to entry resources that have changed state, rather than any
    indication of "why" they have changed or why they are in the feed.

  * A "feed of site events" models a set of "change notices" telling
    you what or why something changed, and then pointing to the entry
    or other site resource.

I wasn't too particular the first time about which model we choose,
but it should be clear that these models *are* distinct, and current
Atom and syndication practice follow the "feed of entries" model.

 > >> An aggregator works on the principle of displaying new entries
 > >> when they appear, and sometimes holding "old" entries for
 > >> perusal, and re-showing updated entries if they're changed.

A "feed of site events" would fulfill that requirement, but one thing
I want to be clear in the "Atom is only generic containers" thread is
that each of the resource types really does model different things and
Atom is speccing how those models are represented and how they are
used.  One can propose a change to the model, but one can't say the
models are generic, all-purpose containers.  WebDAV has a much more
generic model of resource containers, resource metadata, and
notification that would fulfill the requirements as well, yet it too
has a model of representation and usage that is different.

  -- Ken

---

From: "John Panzer" <[email-masked]>
Date: Wed, 14 Jul 2004 18:37:35 -0700

> Atom and syndication practice follow the "feed of entries" model.

Hmm.  I think one of us isn't understanding the other here.

The "feed of site events" I'm talking about has absolutely nothing to do 
with the entries _inside_ the site.

In other words, if I post an entry, my "feed of site events" does not 
change in any way.

If I create a new site, however, my "feed of site events" for (say) my 
username, or for the host provider, produces a new entry, which 
represents the creation event.

I'm not trying to say that _every_ Atom feed is a "feed of events".

Am I confused?

-John

---

From: Ken MacLeod <[email-masked]>
Date: 15 Jul 2004 10:11:27 -0500

"John Panzer" <[email-masked]> writes:

> The "feed of site events" I'm talking about has absolutely nothing
> to do with the entries _inside_ the site.
> 
> In other words, if I post an entry, my "feed of site events" does
> not change in any way.
> 
> If I create a new site, however, my "feed of site events" for (say)
> my username, or for the host provider, produces a new entry, which

I'm quite ok with opening up <atom:feed> to contain other resources
that change, besides <atom:entry> resources.  We'd need to be clear
about that in the spec, so when clients come across
application/atom+xml resources with an {http://purl.org/atom/ns#}feed
element type that that element type may contain sets of changed
resources that aren't {http://purl.org/atom/ns#}entry element types.

Separate from that concern would be which pattern, or model, to
follow.  atom:entry are resource representations of entries, not
events describing changes in those representations.  The pattern to
follow would be to have a "feed if sites" be atom:site resource
representations, not atom:site-event elements describing changes in
those representations.

Or not.  I'm not strongly against "events" in a "notification system".
A "notification system" is a function I see syndication feeds
performing in addition to actual replication of representation.  HTTP
Events and Apache's mod_pubsub, for example, provide "events" that are
"change notifications of resources".  Those events tend to map HTTP
methods, "created" (PUT/POST), "updated" (PUT), "deleted" (DELETE).
WebDAV has a "search" function that can return "recent changed items
[of specified types]", in WebDAV's case you'd get metadata (partial
representation) of resources, not "events" in that sense.

I'm good with the exploration we're doing here.  I'm interested in
seeing it clearly modeled when we write the Paces to incorporate the
ideas, because we have to be clear and thorough about specifying these
features.

  -- Ken

---

From: Robert Sayre <[email-masked]>
Date: Thu, 15 Jul 2004 11:25:07 -0400

>resources that aren't {http://purl.org/atom/ns#}entry element types.

I am probably missing the point here, but why is this better than making 
a normal feed? People already do this kind of thing with CVS checkins 
and whatnot. This is syndicating a logfile--that already works pretty well.

Robert Sayre

---

From: Ken MacLeod <[email-masked]>
Date: 15 Jul 2004 11:18:01 -0500

Robert Sayre <[email-masked]> writes:

> I am probably missing the point here, but why is this better than
> making a normal feed? People already do this kind of thing with CVS
> checkins and whatnot. This is syndicating a logfile--that already
> works pretty well.

If the augmented "normal feed" is for the intended syndication client
audience, I'm fine with that, that's what extensions are for.  If the
augmented "normal feed" is a hack to "re-use" the Atom format, that
seems less than optimal for both the producer and the consumer.  It's
simpler to use a separate resource type (which may be as simple as
different namespace).

If this is simply an augmented normal feed, then I'm not sure why
we're discussing it, because it presumes we all agree on what the Atom
publishing format and protocol model is.

On the other hand, it seems we're discussing it because we want to
extend, open up, or at least clarify the model.  As I said, I'm fine
with exploring that.

Beecher and John, if I understand John's proposal, have asserted that
the Atom model supports this kind of usage, I don't agree because I
don't see these alternate models as something Atom is speccing
properly.

What do we need to do to get in synch?

  -- Ken

PS.  There may be a class of "utility feeds" that it doesn't make
sense for human readers to subscribe to but may fit the model.  That's
a stretch, but I'm probably ok with those, too, as long as we
understand how it affects the model with respect to auto-subscription,
feed discovery, etc.