Subject: Selfish Feeds...

From: "Bob Wyman" <[email-masked]>
Date: Fri, 6 May 2005 15:41:00 -0400

I've got another example of a selfish feed which is producing dynamic
content which will cause many duplicate entries to float around the
blogosphere. The feed in question here is an RSS feed. Nonetheless, I think
we must expect people will do the same stupid tricks with Atom feeds. Check
out:

http://www.b-eye-network.com/xml/articles.php

What you'll get is a feed with entries that look something like the one at
the bottom of this page. The interesting thing to note is that the item has
a <link> element with the url:

  <link>http://www.b-eye-network.com/view/index.php?
   cid=836&fc=0&frss=1&ua=Mozilla/4.0 (compatible; 
    MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; Alexa Toolbar)</link>

What's happened here is that the site has appended my User Agent to the URL
in the link. I assume that this is to allow some kind of tracking. However,
the impact is that the contents of the feed depend on what tool you use to
read the feed. If you access the feed, you will undoubtedly get different
content then I did... For instance, if PubSub's crawler had read the feed,
the value of the "ua" attribute in the URL would have been different and the
URL would have read:
   <link>http://www.b-eye-network.com/view/index.php?
    cid=836&amp;fc=0&amp;frss=1&amp;ua=PubSub.com RSS reader - 
    http://www.pubsub.com/</link>

If this feed is read by more than one synthetic feed generator or if items
from the feed are copied from this feed to another, it is inevitable that
we'll have multiple copies of the item floating around and we'll have very
little means for determining which one is authoritative -- essentially they
all are. It would be handy to have a "dynamic content flag" that allows us
to ignore this stuff...

This business of people including dynamic content in their feeds for selfish
purposes is making it very difficult to build a decent infrastructure for
distributing and caching RSS/Atom entries... We've got a "tragedy of the
commons" situation going on here. The much-too-respectable SEO crowd is
trying to seek profit at the expense of the network at large... Because they
can.

	bob wyman

====== Full example entry from the feed =============
<item>
  <title>Nanotechnology Basics Defined</title> 
 <description>
 <![CDATA[ ADVERTISEMENT - <a
href='http://www.b-eye-network.com/adserver-new/adclick.php?bannerid=199&amp
;zoneid=27&amp;source=&amp;dest=http%3A%2F%2Finfo7.net%2Fs%2Fhyperion%2F53w%
2F2wp' target='_blank'>Find out how Hyperion can help: An Intelligent
Approach to Business Intelligence</a><div id="beacon_199" style="position:
absolute; left: 0px; top: 0px; visibility: hidden;"><img
src='http://www.b-eye-network.com/adserver-new/adlog.php?bannerid=199&amp;cl
ientid=178&amp;zoneid=27&amp;source=&amp;block=0&amp;capping=0&amp;cb=818aa5
08d0a2ebeec0c37d74b10535d6' width='0' height='0' alt='' style='width: 0px;
height: 0px;'></div><br/><br/>Nanotechnology is the Next Big Thing.
  ]]> 
  </description>
 
<link>http://www.b-eye-network.com/view/index.php?cid=836&fc=0&frss=1&ua=Moz
illa/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; Alexa
Toolbar)</link> 
  <pubDate>Thu, 5 May 2005 00:00:00 MST</pubDate> 
</item>

---

From: Sam Ruby <[email-masked]>
Date: Fri, 06 May 2005 16:13:23 -0400

> all are. It would be handy to have a "dynamic content flag" that allows us

It seems to me that instead of adding a dynamic content flag, having a 
separate id element (or in RSS 2.0's case, utilizing the guid element) 
would be more to the point.

- Sam Ruby

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 6 May 2005 16:37:23 -0400

Sam Ruby wrote:
> It seems to me that instead of adding a dynamic content flag, having
> a separate id element (or in RSS 2.0's case, utilizing the guid
> element) would be more to the point.
	Relying on a GUID alone only works if you implement a policy that
says that you are only interested in seeing content with "new" GUIDs and you
are willing to ignore any "updates" to previously seen entries/items.
Similarly, relying on atom:id + atom:updated implies a policy of only being
interested in content changes which are explicitly flagged by the publisher
as being "worthy of notice." These are certainly appropriate policies for
*some* aggregators addressing *some* user needs. However, other aggregators
implement different policies which address other user needs. For instance,
many aggregators will update their content stores whenever *any* change
occurs in an item whether or not the GUID or Atom:id has changed. Some of
these aggregators will flag any change as a "new or unread" entry. (which I
think is a really stupid policy...) Others will, like Gush, distinguish
between "new" items and "updated" items. (I think this is much more
sensible, others will say it is overly complex and unnecessary.)
Conceivably, once Atom is released, some aggregators will wish to record
three states for an entry: "new", "major update" and "minor update." (I
would support anyone doing this, others would not.)
	To understand this issue and many other syndication issues, it is
vital that you try to consider the full range of policies that are
implemented by aggregators and that you try to look beyond your personal
preferences. Please try to understand that this isn't a simple issue -- at
least not from the point of view of a channel intermediary like PubSub. As
was recently pointed out, a very large percentage of the HTTP specification
covers issues related to proxies (which is very much the role that PubSub
plays.) The same is true of the State Management (Cookie) RFC. I remember
that when we were working on that RFC, proxy issues were just about the
*only* thing we discussed... Problems which are "simple" in point to point
networks become much more complex when you introduce intermediaries.
	Frankly, I really wish that we had done the "blog architecture" work
many months ago so that we would all have a shared understanding of the
system-wide issues and components rather than the widely divergent personal
and partial views that are obvious in many our conversations today...

		bob wyman

---

From: Walter Underwood <[email-masked]>
Date: Fri, 06 May 2005 14:34:30 -0700

> and partial views that are obvious in many our conversations today...

Agreed. "A conceptual model of a resource" is up there at the front of
our charter, and if we don't have that, it doesn't seem like the WG is done.

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Graham <[email-masked]>
Date: Fri, 6 May 2005 22:49:06 +0100

On 6 May 2005, at 9:37 pm, Bob Wyman wrote:

> as being "worthy of notice."

Well if you won't trust info in the feed, there's nothing any format  
can do for you. Get thee some venture capital and hire a few  
linguistics/statistics undergrads to work out a real "significant  
update" algorithm for yourself. If you're complaining because people  
are putting tracking info in links, you're obviously not doing a very  
good job at this at the moment.

Graham

---

From: Graham <[email-masked]>
Date: Fri, 6 May 2005 20:54:02 +0100

> we must expect people will do the same stupid tricks with Atom  

Bob, what the fuck do you think ids are for?

Graham