Subject: Re: feedsets and atom:source

From: Thomas Broyer <[email-masked]>
Date: Mon, 16 Feb 2009 09:59:02 +0100

AFAICT, this has nothing to do with Atom-the-Protocol but only with
Atom-the-Syntax, so CC'ing atom-syntax (and please follow-up there).

On Mon, Feb 16, 2009 at 9:25 AM, Erik Wilde wrote:
>
> hello again.
>
> RTFM, i guess. atom:source is expected to contain the *feed*'s metadata, not
> the *entry*'s metadata from the originating feed. this almost solves my
> problem, the exception being the question of how to identify the feeds that
> are aggregated in a feed, and how to relate the entries to these aggregated
> feeds. the <link id="" rel="via"/> and <entry via:source="" /> scheme might
> still be one solution for that, allowing consumers to learn more about *all*
> aggregated feeds, while pure atom clients could use atom:source to at least
> understand where each individual entry came from. according to the extension
> rules, via:source would have to be an element rather than an attribute, but
> then it would be ok, i guess. and maybe xml:id for the link/@id? but it
> seems like atom does not allow link/@xml:id, but any local attribute (but
> only if a future version of atom actually defined it).

My reading is backwards: undefinedAttribute in the RNC explicitly
prohibits local:* attributes while allowing xml:id (and any "foreign"
attribute besides xml:base and xml:lang)

> so if link/@id and
> link/@xml:id both are not allowed by the current spec, what other method
> would there be to make an association between entries and via-links?

<feed xmlns="http://www.w3.org/2005/Atom">
<link rel="via" href="http://example.com/feed-A" />
...
<entry>
  <source>
    <link rel="self" href="http://example.com/feed-A" />
...

And relate entries to link[@rel='via'] using the @href value.

-- 
Thomas Broyer

---

From: Erik Wilde <[email-masked]>
Date: Mon, 16 Feb 2009 10:35:51 -0800

hello thomas.

> Atom-the-Syntax, so CC'ing atom-syntax (and please follow-up there).

ok, follow-up to atom-syntax, cc'ed to atom-protocol and will remove 
that list from future posts.

> attribute besides xml:base and xml:lang)

you're correct. which means xml:id would be allowed; that's nice.

> And relate entries to link[@rel='via'] using the @href value.

that's almost what i want, but not quite, because this way, clients can 
only find out entries from which feeds did show up in the consumed feed, 
but not which feeds have been aggregated by the consumed feed. as a 
side-effect, i's also like the aggregated feed to be able to create and 
advertise IDs for the aggregated feeds, so that clients can query the 
aggregated feed by specifically asking only for certain entries from 
certain feeds. this is the bigger picture of the question i am asking 
here; how to create a query language for feeds. as one goal, we want it 
to be able to work across feeds, and a nice way to do this would be to 
specify it as working on an aggregated feed.

<feed xmlns="http://www.w3.org/2005/Atom" xml:via="...">
<link xml:id="feed1" rel="via" href="http://example.com/feed-A" />
<link xml:id="feed2" rel="via" href="http://example.com/feed-B" />
...
<entry>
   <source via:via="feed1">
     <link rel="self" href="http://example.com/feed-A" />

that would be the idea, i guess. it does duplicate the feed URIs, but it 
is usable for pure atom clients, which is good. but even though entries 
from feed-B might not show up in the aggregated feed too often (because 
the aggregated feed is limited to some number on entries and feed-B is a 
low-volume feed, perhaps), it would still be visible that feed-B is 
contributing to the aggregated feed, and clients could specify filters 
and query the aggregated feed asking for entries only from feed-B (by 
using the "feed2" id in the query).

cheers,

dret.

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Tue, 17 Feb 2009 04:46:07 +0100

>     <link rel="self" href="http://example.com/feed-A" />

Under what circumstances would the entry self link ever differ
from the feed via link? What’s the use of those IDs and via:via
attributes?

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: Erik Wilde <[email-masked]>
Date: Mon, 16 Feb 2009 21:13:33 -0800

> from the feed via link? What’s the use of those IDs and via:via

i don't think that these two URIs would ever differ. based on my 
scenario (providing aggregated feeds as a way to specify queries across 
a number of feeds), there are are reasons for the IDs:

- even if the aggregated does not contain any entry of a feed it is 
aggregating, the <link /> still specifies that this feed is contributing 
to the entries available through the aggregated feed.

- in the query on the aggregated link, the IDs are used in the query 
itself to specify the entries a client is requesting, such as "i am 
interested in all entries from 'feed1' where the title contains the 
string 'atom'."

cheers,

dret.

---

From: Eric Scheid <[email-masked]>
Date: Tue, 17 Feb 2009 17:35:21 +1100

On 17/2/09 4:13 PM, "Erik Wilde" <[email-masked]> wrote:

>> from the feed via link? What¹s the use of those IDs and via:via
>> attributes?
> 

this would be communicated via the @rel attribute, surely?

> interested in all entries from 'feed1' where the title contains the

Hmm ... interesting. So you're not necessarily aggregating *all* entries
from http://example.com/feed-A but instead possibly just a subset. So
another aggregator who wants to carry this further shouldn't munge your id
values?

This raises a further wrinkle in that your ids, even in your own application
space, would need to be globally unique. Eg: if your application is
providing a "top movie reviews" feed-set and also a "geeky web talk"
feed-set, you generally wouldn't find the same id values used in both
(unless it's a top movie review found in some geeky web talk forum).

I would suggest something else to carry your feed-criteria data ..

    <link rel="via"
        href="http://example.com/feed-A"
        via:criteria="http://via.example.com/feed-A/criteria/1"
        type="application/atom+xml" />

That way your criteria idents won't get stomped on by any further
aggregation.

e.

---

From: "James Holderness" <[email-masked]>
Date: Tue, 17 Feb 2009 11:17:04 -0000

Erik Wilde wrote:
> good point. i have not hey thought about how aggregation of aggregated 
> feeds would work. i guess that for the second aggregator, the fact that it 
> is aggregating an agggregated feed should not matter. but then the 
> question is how to treat the <atom:source> and more specifically its 
> via:via="" attribute. so i guess aggregation of aggregated feeds does need 
> some special treatment. how about retaining all the <atom:source> 
> children, but replacing the via:via="" attribute's value with the ID of 
> the aggregated feed?

There should be one and only one atom:source element per entry, reflecting 
the original source of the entry. If you're aggregating an entry that 
already has an atom:source, that atom:source element should be left 
unchanged.

See section 4.1.2 of RFC4287: "atom:entry elements MUST NOT contain more 
than one atom:source element." And 4.2.11: "...by adding an atom:source 
child element, if it is not already present in the entry".

Regards
James

---

From: Erik Wilde <[email-masked]>
Date: Tue, 17 Feb 2009 11:51:08 -0800

James Holderness wrote:
>> feeds would work. i guess that for the second aggregator, the fact 
>> that it is aggregating an aggregated feed should not matter. but then 
>> the question is how to treat the <atom:source> and more specifically 
>> its via:via="" attribute. so i guess aggregation of aggregated feeds 
>> does need some special treatment. how about retaining all the 
>> <atom:source> children, but replacing the via:via="" attribute's value 
>> with the ID of the aggregated feed?
> There should be one and only one atom:source element per entry, 
> reflecting the original source of the entry. If you're aggregating an 
> entry that already has an atom:source, that atom:source element should 
> be left unchanged.

thanks for pointing that out. i was (almost) proposing to leave 
atom:source unchanged, with the exception of the foreign markup of the 
via:via attribute. to not violate the rules you are citing, i could move 
this information to an extension element as a child of the entry, 
instead of an attribute of atom:source. so the revised version of the 
markup would look like this:

   <via:via ref="feed1"/>

in the first-level aggregation, and the second aggregator would create 
something like this:

<feed xmlns="http://www.w3.org/2005/Atom" xml:via="...">
<link xml:id="id1" rel="via" href="http://example.com/feed-C" />
<link xml:id="id2" rel="via" href="http://example.com/feed-D" />
...
<entry>
   <via:via ref="id1"/>
   <source>
     <link rel="self" href="http://example.com/feed-A" />

which would make clear that (a) the entry has been originally published 
in feed-A, but has been aggregated through feed-C ("id1"). this pattern 
could be repeated through any number of aggregators. or would the 
via:via element no simply be a repetition of <link> and it should be 
like this:

<feed xmlns="http://www.w3.org/2005/Atom" xml:via="...">
<link xml:id="id1" rel="via" href="http://example.com/feed-C" />
<link xml:id="id2" rel="via" href="http://example.com/feed-D" />
...
<entry>
   <link rel="via" href="http://example.com/feed-C"/>
   <source>
     <link rel="self" href="http://example.com/feed-A" />

if i use this design, the only new markup i need is xml:id on the feed's 
  link[@rel = 'via'] children, but in the end my application would have 
more strictly defined rules on how to handle <link> when aggregating 
than what is defined by atom itself. the rules would be:

1. if you aggregate a feed, you must list that feed as a <link 
rel="via"/> and you must add an @xml:id for it. this id must be stable 
over time.

2. for each aggregated entry, you must add the feed from which you 
aggregated it as a <link rel="via"/> on the entry.

these rules ensure that whenever i read an entry from an aggregated 
feed, there will be a feed/link[@rel = 'via'][exists(@xml:id)] for the 
entry/link[@rel = 'via'].

cheers,

dret.

---

From: "James Holderness" <[email-masked]>
Date: Tue, 17 Feb 2009 20:52:48 -0000

Erik Wilde wrote:
> via:via attribute.

Sorry, I misunderstood you. I thought you had intended to add multiple 
source elements to each entry. The foreign markup doesn't bother me that 
much.

> or would the via:via element no simply be a repetition of <link> and it 
> should be like this:
>

The spec is a bit vague in its description, but the intended purpose of the 
via link, as I understood it, was so that the entry author could credit the 
source of their "story". For example, if you read about a new iPhone release 
on Engadget and decided to blog about it, your blog entry could include a 
via link back to Engadget.

Since you're not the author of these entries, you have no right to makes 
claims about the source of the story. In general you need to be very careful 
about making *any* changes to an entry when you're republishing someone 
else's content verbatim in an aggregated feed.

As for the use of xml:id, I think that's unwise considering you can't 
guarantee uniqueness across the whole feed (which could make the document 
invalid). You have no way of knowing what other extensions might be using 
xml:id attributes which could conflict with yours (they would also be unwise 
to do so, but if you're considering it, you have to assume that someone else 
is too).

It wouldn't be so bad if you were using something more universally unique in 
your ids, like a URI of some sort. But in that case, you might just as well 
reference the link href and avoid the xml:id altogether.

Regards
James

---

From: Bob Wyman <[email-masked]>
Date: Tue, 17 Feb 2009 18:50:54 -0500

On Mon, Feb 16, 2009 at 1:35 PM, Erik Wilde <[email-masked]> wrote:
> how to create a query language for feed

At least three aspects of Atom serve to facilitate precisely this
application (if I understand you correctly...).
1. Aggregate feeds are facilitated by having the atom.source element that
allows feed metadata to be copied into the entry. If an entry contains an
atom.source, then that entry, when read in an aggregated feed, can have the
same semantics as if it had been read from its source feed.
2. Atom prohibits assigning any semantics to an entry's order within a feed.
Thus, an atom.entry's semantics depend *only* on what is in the entry and
the feed's metadata if the entry has no atom.source.
3. Atom entries are "top-level" objects (unlike RSS items) and thus can be
read as self contained, complete documents within any feed or even outside a
feed.

Given these three aspects of Atom, it is quite possible to build a robust
feed query language that allows statements like: "FROM feeda, feedb, feedn
SELECT all entries that contain 'foobar'".

It seems, however, that the thing that is hanging you up here is that you
want to be able to encode a listing of the set of feeds which were scanned
in the process of building the result feed for a query. I think this is what
you're trying to accomplish with the link="via" stuff.

The problem, of course, is that this business of listing source feeds
doesn't scale -- thus, it isn't something that we should be trying to
standardize. For queries that only span a small number of feeds, it might
seem reasonable to explicitly list the source feeds, however, most "feed
search" engines these days work with many millions of feeds. Thus, an
aggregate result feed that contained only a handful of entries might need to
have a listing of 10's of millions of feeds -- whether or not the result
feed contained any of their entries.

It has been pointed out by others that you are misreading the defintion of
link="via". Hopefully, you won't try to overload it. However, you could
define a custom relationship, perhaps link="http://example.com/queried_feed"
for applications with small numbers of queried feeds. Alternatively, you
might create some custom XML file that served as a "query specfication" and
then link to that via some relationship... Of course, this relationship
should be associated with the feed and not with any of the entries within
the feed. Each entry in the feed would still state it's "origin" via the
atom.source element. (Note: The list of queried feeds really is an attribute
of the result feed, not the entries in it, thus, this shouldn't be a big
problem.)

Unfortunately, there is no way to encode an entry that shows both the
entry's source as well as the reason why it has been included in an
aggregate feed. Doing this would require an ability to "add data" to an
entry after it is originally published. But, Atom provides no mechanisms for
adding data to entries once they are published -- except for the business
about inserting atom.source if not present. During the process of defining
the atom format, we discussed at length a number of alternatives that would
have permitted adding more "history" to entries but we couldn't get any
agreement on what to do. The discussions typically included ideas for
describing the "provenance" of an entry...

Are you sure you *really* want to include a listing of the queried feeds?
What would you do if you had millions of feeds to query?

bob wyman

---

From: "Lindsley Brett-ABL001" <[email-masked]>
Date: Wed, 18 Feb 2009 07:05:43 -0500

Tracking the aggregation chain is an interesting problem that I played
with in the past. One comment in this note grabbed my attention: "In
general you need to be very careful about making *any* changes to an
entry when you're republishing someone else's content verbatim in an
aggregated feed." One issue in particular is handling *signed* entries.
Any modification to the entry breaks the signature. One thought I had is
to use foreign markup to indicate the aggregation chain, but to exclude
the foreign markup in computing the signature. This way the original
signature authenticates. Of course, this then begs the issue of
authenticating the aggregation information...

Brett Lindsley, Motorola Applied Research & Technology Center

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of James Holderness
Sent: Tuesday, February 17, 2009 2:53 PM
To: [email-masked]
Subject: Re: feedsets and atom:source

much.

>     <link rel="self" href="http://example.com/feed-A" />

The spec is a bit vague in its description, but the intended purpose of
the 
via link, as I understood it, was so that the entry author could credit
the 
source of their "story". For example, if you read about a new iPhone
release 
on Engadget and decided to blog about it, your blog entry could include
a 
via link back to Engadget.

claims about the source of the story. In general you need to be very
careful 
about making *any* changes to an entry when you're republishing someone 
else's content verbatim in an aggregated feed.

As for the use of xml:id, I think that's unwise considering you can't 
guarantee uniqueness across the whole feed (which could make the
document 
invalid). You have no way of knowing what other extensions might be
using 
xml:id attributes which could conflict with yours (they would also be
unwise 
to do so, but if you're considering it, you have to assume that someone
else 
is too).

It wouldn't be so bad if you were using something more universally
unique in 
your ids, like a URI of some sort. But in that case, you might just as
well 
reference the link href and avoid the xml:id altogether.

Regards
James

---

From: Peter Keane <[email-masked]>
Date: Wed, 18 Feb 2009 06:57:41 -0600

On Tue, Feb 17, 2009 at 5:50 PM, Bob Wyman <[email-masked]> wrote:

...

> about inserting atom.source if not present.

There was recently some interesting discussion on just this issue on
the activity streams mailng list.  It came up in the context of
"bookmarking" or "favorite-ing" or "rating" someone elses activity
(activities are represented by atom entries).  The idea being that you
include that entry but, essentially, add some metadata saying why you
are including it.  And that act is itself and activity, also
represented by an atom entry.

It's actually an interesting issue that won't go away anytime soon.
It'd be nice to have some best practices established ( a  pattern
language for Atom anyone?) .

--peter keane

 During the process of defining

---

From: Bob Wyman <[email-masked]>
Date: Wed, 18 Feb 2009 11:44:57 -0500

On Wed, Feb 18, 2009 at 7:05 AM, Lindsley Brett-ABL001 <
[email-masked]> wrote:
> One issue in particular is handling *signed*
> entries. Any modification to the entry breaks
> the signature. One thought I had is to use
> foreign markup to indicate the aggregation chain,
> but to exclude the foreign markup in computing
> the signature.
This is essentially the approach that I argued for when this was debated
years ago on this list. The idea is to define some new element that is used
to encapsulate annotations. Additionally, the entry canonicalization rules
for Atom would be modified to require that the encapsulated data should be
removed from the entry prior to signing or signature verification.
Unfortunately, at the time this was discussed on the list, there was little
understanding of why it was even useful or reasonable to sign Atom entries.
Thus, proposals that made signature processing any more complex than it
already was were not received well. Perhaps now that we've got more people
using Atom and more folk understanding the utility of signatures, we can get
such an extension accepted.

> Of course, this then begs the issue of
Clearly, if one goes to the trouble of defining a new element and extended
canonicalization procedures, one should also ensure that:
1. There is a way to sign third-party annotations
2. The annotations can themselves be annotated and such annotations would be
excluded via canonicalization from the signing of their parent.

The remaining problem comes in defining a means to allow an annotation to be
composed of several signed annotations which are themselves included in the
signature of a parent... This gets complicated....

bob wyman

---

From: Peter Keane <[email-masked]>
Date: Wed, 18 Feb 2009 11:05:36 -0600

On Wed, Feb 18, 2009 at 10:44 AM, Bob Wyman <[email-masked]> wrote:
> On Wed, Feb 18, 2009 at 7:05 AM, Lindsley Brett-ABL001
> <[email-masked]> wrote:

Hmm.  Makes me think that Git has in fact addresses many/all of the
issues we'll be struggling with the next few years (i.e., identifying
unique content w/ in other content).  I wonder how that might apply ot
the world of documents...

--peter keane

---

From: Richard Salz <[email-masked]>
Date: Wed, 18 Feb 2009 12:30:31 -0500

If you define an element to hold the annotation, then the XPath Filter 2.0 
(http://www.w3.org/TR/xmldsig-filter2/) transform, which is probably 
supported by almost every XML DSig implementation can do what you want. 
The drawback is that the entity signing the entry would have to generate 
the right signature (using the filter) ahead of time.  Or a recipient, 
upon failing a verification, would have to go and remove the annotation 
elements -- and the "correct" whitespace around them.

The alternative is to define a new canonicalization scheme.  I doubt this 
would get much traction.

        /r$

--
Visiting Member, IBM Academy
STSM, DataPower Chief Programmer
WebSphere DataPower SOA Appliances
http://www.ibm.com/software/integration/datapower/

---

From: Erik Wilde <[email-masked]>
Date: Thu, 19 Feb 2009 10:32:37 -0800

> represented by an atom entry.

that sounds a lot like RDF. making assertions about atom entries by 
creating atom entries about atom entries. you quickly run into the 
problem of "how do i create rules for how this is done in a given 
scenario", and in the end maybe you come up with the idea of "metafeeds" 
which are carrying metadata about a feed. creating standards around this 
pattern is a pretty challenging topic, i guess, but it is true that in a 
way, it is simply a generalization of the aggregation pattern i am 
trying to deal with. it also has the same issues of recursion: it should 
be possible to "favorite-ing" someone's "favorites", right?

cheers,

dret.