Subject: PaceDuplicateIDWithSource2

From: Antone Roundy <[email-masked]>
Date: Mon, 2 May 2005 15:03:26 -0600

http://www.intertwingly.net/wiki/pie/PaceDuplicateIDWithSource2

This is what I thought had been suggested.  Hopefully I didn't err in 
putting two changes into one proposal, but the first seemed a little 
odd without the acknowledgment provided by the second.

Coming up with wording for the first change that makes the point 
without getting horrendously verbose and convoluted isn't as easy as I 
thought!

Antone

== Abstract ==

Allow multiple entries with the same ID in a feed document if their 
source feeds are demonstrably different. Adjust the wording regarding 
the uniquess of atom:id values to reflect the recognition that 
publishers might duplicate each others' atom:ids, even if accidentally.

== Status ==

Proposed.

== Rationale ==

The current spec does not allow multiple entries in a feed document 
with the same ID. This can cause problems if, whether accidentally or 
maliciously, different feeds use the same IDs for their entries. This 
proposal alleviates this problem by allowing a feed document to contain 
multiple entries with the same ID as long as their source feed's ID is 
different, enabling consumers to determine that they are not duplicates 
of the same entry.

The current spec requires atom:id values to be minted in a way that 
"assures uniqueness". This is impossible, since no one can prevent 
others from, whether accidentally or maliciously, duplicating one's 
ids. This proposal rewords that requirement to reflect recognition of 
this fact.

== Proposal ==

In relation to atompub-format-08:

In section 4.1.1, replace:

  atom:feed elements MUST NOT contain atom:entry elements with identical 
atom:id values

With:

  atom:feed elements MUST NOT contain atom:entry elements with identical 
atom:id values unless the values of the atom:id elements for each of 
their source feeds is different. If the value of atom:id for the source 
feed cannot be determined by one of the two following rules, then the 
value of atom:id for the atom:entry MUST be unique within its enclosing 
atom:feed element.

  * If an atom:entry element contains an atom:source element, and that 
element contains an atom:id element, use its value.
  * If an atom:entry does not contain an atom:source element, and the 
atom:entry element's parent atom:feed element contains an atom:id 
element, use its value.

In 4.2.7, replace:

  The content of an atom:id element MUST be created in a way that 
assures uniqueness.

With:

  The content of an atom:id element MUST be created in a way that, as 
nearly as possible, assures uniqueness. An atom:id value that has been 
used with one entry in a particular feed MUST NOT ever be used with a 
different entry in the same feed, and SHOULD NOT ever be used with an 
entry in another feed.

== Impacts ==

Allows duplication ids within feeds, so atom:source/atom:id or 
atom:feed/atom:id must be examined when looking for duplicates.

---

From: Graham <[email-masked]>
Date: Tue, 3 May 2005 00:23:24 +0100

On 2 May 2005, at 10:03 pm, Antone Roundy wrote:

>  The content of an atom:id element MUST be created in a way that,  
> as nearly as possible, assures uniqueness. An atom:id value that  
> has been used with one entry in a particular feed MUST NOT ever be  
> used with a different entry in the same feed, and SHOULD NOT ever  
> be used with an entry in another feed.

That last part: Are you against category feeds? Are you against  
aggregated feeds? Are you against search results feeds?

Graham

---

From: Antone Roundy <[email-masked]>
Date: Mon, 2 May 2005 17:38:28 -0600

On Monday, May 2, 2005, at 05:23  PM, Graham wrote:
>> nearly as possible, assures uniqueness. An atom:id value that has 
>> been used with one entry in a particular feed MUST NOT ever be used 
>> with a different entry in the same feed, and SHOULD NOT ever be used 
>> with an entry in another feed.
>
No, my wording is just imperfect.  It should not be use with an entry 
originating in a different feed.

Category feeds: that's something we haven't dealt with well yet.  
Should they, or feeds that combine category feeds present the entries 
like aggregated feeds?  Should we add an "authoritative" link type, as 
recently discussed, and change this wording to work with that?

Search results feeds: more difficult.  If the search criteria don't 
change, then it's the same feed, so the prior sentence applies.  If the 
search criteria do change...is it the same "entry"?  I'd imagine that 
in some cases it might be, but in some I'd think it isn't.  Perhaps 
that's exception to the SHOULD NOT.  But it does sound like a good 
argument for fixing this sentence.

It's time for dinner, so I'll have to wording to express the idea that 
an ID should correspond to exactly one "entry resource".  Feel free to 
beat me to it.

---

From: "Bob Wyman" <[email-masked]>
Date: Mon, 2 May 2005 21:22:16 -0400

Antone Roundy asked:
> Category feeds: ... Should they, or feeds that combine category 
> feeds present the entries like aggregated feeds?
	Yes, at least normally. An entry should have only one "source."
	On a site that has a "master" feed and also provides copies of
entries in a variety of category feeds, I feel that the "master" feed should
be the source and things in the category feeds should be copies which
contain atom:source elements pointing to the master. Of course, you could
reasonably do it the other way around, however, for a variety of reasons I
think that would be less then optimal. 
	One thing you should not do with category feeds is "tag" things
simply by putting them in one feed or another. If you have a feed for things
that are in category "foobar" then each entry which is in the "foobar" feed
should also have an atom:category element with the value "foobar." The entry
should standalone and be associated with the same category or categories no
matter which feed it is found in.

> Should we add an "authoritative" link type, as recently discussed, 
> and change this wording to work with that?
	It would be great if we could permit one feed to subordinate itself
to another, however, I don't that would really change the fact that there
really should be only one source for an entry.

> change, then it's the same feed, so the prior sentence applies. 
	I'm not sure what you're getting at here. However, it seems to me
that a search results feed is almost always going to be an aggregate feed
and thus should be filled with entries that have source elements identifying
other feeds.
	But, I suspect that I haven't understood properly what you were
trying to say. Please try rephrasing.

		bob wyman

---

From: Antone Roundy <[email-masked]>
Date: Mon, 2 May 2005 20:35:29 -0600

> think that would be less then optimal.

I agree with all of the above in thoery, but getting people to do this 
could be hampered by the fact that category feeds would have to 
duplicate the master feed's metadata in every entry...or at least 
should...since the WG decided against creating a method of doing it 
only once in aggregate feeds.  That duplication makes me hesitant to 
embrace this idea wholeheartedly.  But I don't know of another solution 
that the WG hasn't already rejected.

> really should be only one source for an entry.

Agreed on both points.

> trying to say. Please try rephrasing.

Oops, my mistake--I was thinking of searches of non-entry resources 
(such as web page searches) being published as a feed.  Defining a 
method for generating ids for such things is a little too far outside 
of the core of what we're doing to both with.  You're right, of course.

Anyway, I've added "originating" three times in that last sentence, 
making it "An atom:id value that has been used with one entry 
originating in a particular feed MUST NOT ever be used with a different 
entry originating in the same feed, and SHOULD NOT ever be used with an 
entry originating in another feed."  Does that address the question 
sufficiently (within the scope of the Pace at least)?

---

From: Martin Duerst <[email-masked]>
Date: Tue, 03 May 2005 17:21:13 +0900

I'm not really happy with this. Conceptually, it seems to replace
an ID for an entry with a pair (ID,feed). As IDs are URIs/IRIs
(remember what they expand to), that doesn't make sense.
What guarantee do we have that two feeds will be different?
(yes, these days, we get feeds over http, but there are other
ways to get feeds, where things may not be that easy).

If we don't have a solution for the malicious case, and we
think we need one, we should work on some security stuff.

If we think that accidential ID duplication is a problem,
then let's look at how we can improve the explanation.
After that, there may still be an occasional accident,
but the spec should be worded to catch that, not to
provide a loophole.

If we have to allow duplicate IDs, I'd rather prefer we
do it without all this feed/source/... stuff: I.e. if
you are an aggregator and can't manage to do duplicate
elimination, you can just delegate the problem to the
next place in the feeding chain.

Regards,    Martin.

 >This is what I thought had been suggested.  Hopefully I didn't err in 
putting two changes into one proposal, but the first seemed a little odd 
without the acknowledgment provided by the second.
 >
 >Coming up with wording for the first change that makes the point without 
getting horrendously verbose and convoluted isn't as easy as I thought!
 >Allow multiple entries with the same ID in a feed document if their 
source feeds are demonstrably different. Adjust the wording regarding the 
uniquess of atom:id values to reflect the recognition that publishers might 
duplicate each others' atom:ids, even if accidentally.
 >
 >== Status ==
 >
 >Proposed.
 >
 >== Rationale ==
 >
 >The current spec does not allow multiple entries in a feed document with 
the same ID. This can cause problems if, whether accidentally or 
maliciously, different feeds use the same IDs for their entries. This 
proposal alleviates this problem by allowing a feed document to contain 
multiple entries with the same ID as long as their source feed's ID is 
different, enabling consumers to determine that they are not duplicates of 
the same entry.
 >The current spec requires atom:id values to be minted in a way that 
"assures uniqueness". This is impossible, since no one can prevent others 
from, whether accidentally or maliciously, duplicating one's ids. This 
proposal rewords that requirement to reflect recognition of this fact.
atom:id values
 >  atom:feed elements MUST NOT contain atom:entry elements with identical 
atom:id values unless the values of the atom:id elements for each of their 
source feeds is different. If the value of atom:id for the source feed 
cannot be determined by one of the two following rules, then the value of 
atom:id for the atom:entry MUST be unique within its enclosing atom:feed element.
 >  * If an atom:entry does not contain an atom:source element, and the 
atom:entry element's parent atom:feed element contains an atom:id element, 
use its value.
 >
 >In 4.2.7, replace:
 >
 >  The content of an atom:id element MUST be created in a way that assures 
uniqueness.
 >  The content of an atom:id element MUST be created in a way that, as 
nearly as possible, assures uniqueness. An atom:id value that has been used 
with one entry in a particular feed MUST NOT ever be used with a different 
entry in the same feed, and SHOULD NOT ever be used with an entry in 
another feed.

---

From: Thomas Broyer <[email-masked]>
Date: Wed, 04 May 2005 00:41:23 +0200

> next place in the feeding chain.

OK, so, something like this (conceptually).

First, start with the "originating publisher":
  1. *You* MUST NOT use an atom:id for different entries (entries as
     resource, not as representations) in any of *your* feeds.
  2. *You* MUST NOT include the same entry (resource, not representation)
     more than once in a feed.

-> atom:id should be reworded to clarify that they identify an "entry 
resource", not each of its representations.

Next, what about aggregation? First thoughts:
As atom:id are not guaranteed to be "globally unique", aggregated feeds 
might contain different entries (entry resources) with the same atom:id. 
As they are aggregated entries, they should (I'd also wish it were a 
MUST) have an atom:source.
So we can say that a feed is allowed to include different entries 
sharing the same atom:id as long as at most one of them doesn't have an 
atom:source). This is also a way to enforce the second item of the above 
list in the "publishers' camp".

-> please, change the SHOULD on atom:source to a MUST

One step further: aggregation of aggregated feeds.
We didn't handled the case where a single entry can appear more than 
once in an aggregated feed because of different aggregation routes.
There, we need to identify and distinguish feeds from each other, so yes 
we're moving to an identifier pair (feed+entry). So we have to look at 
the atom:source and its content.
However, we didn't solve the problem as, IMO, we shouldn't require an 
entry to have a single source feed. Publishers should be allowed to 
publish an entry in several feeds (main + category) without the 
atom:source hassle. atom:id is also not a required metadata of a feed.
Suggestion: "delegate the problem to the next place in the feeding 
chain". This will push delegation 'til the end-user's aggregator where 
he might have settings on how to treat entries with the same atom:id but 
different feed's atom:id.

Finally, what about atom:id at the feed level?
IMO, different feed "flavors" (full-content, summary-only, title-only) 
should use the same atom:id as a mean to say "I guaranty that these are 
the same (e.g. they are updated simultaneously and one is not a 
subset/superset of another), with different levels of richness". It 
should be the same for last-10-entries vs. last-50-entries feeds, as 
long as their [insert lesser denominator here, e.g. 10] most recent 
entries (based on their atom:updated) are the same (based on their 
atom:id), it'd then be a mean to say "I guaranty that these are the same 
(e.g. they are updated simultaneously), with different history lengths".

Oh, and last but not least: these considerations on "entry 
identification" should IMO appear in an informative appendix. The 
normative part of the spec would just say a feed SHOULD NOT include more 
than one entry with a given atom:id and then point to the appendix for 
more about some ways of identifying entry duplicates.

-- 
Thomas Broyer

---

From: Robert Sayre <[email-masked]>
Date: Wed, 4 May 2005 20:31:53 -0400

http://www.intertwingly.net/wiki/pie/PaceOriginalAttribute

On 5/3/05, Martin Duerst <[email-masked]> wrote:
> 
> I'm not really happy with this.

I found Martin's comments (copied in full below) to be accurate. So, I
thought I would try another approach. Comments, suggestions, and
alterations are welcome.

Robert Sayre

== Abstract ==

Preserve the original ID elsewhere, and require republishers to mint
new IDs for *their entries*.

== Status ==

Open

== Rationale ==

Duplicate entry ids in feeds are too easy to create unintentionally,
and the legitimate uses can't be verified as updates unless they come
from the originating feed.

== Proposal ==

Add an 'original' attribute to atom:source and reword as follows:

{{{

   If an atom:entry is copied from one feed into another feed, then the
   source atom:feed's metadata (all child elements of atom:feed other
   than the atom:entry elements) MAY be preserved within the copied
   entry by adding an atom:source child element, if it is not already
   present in the entry, and including some or all of the source feed's
   metadata elements as the atom:source element's children.  Such
   metadata SHOULD be preserved if the source atom:feed contains any of
   the child elements atom:author, atom:contributor, atom:copyright, or
   atom:category and those child elements are not present in the source
   atom:entry.

 4.2.11.1 The 'original' Attribute
 
   Atom entries can be republished and altered by intermediaries, but 
   Atom feeds MUST NOT contain duplicate atom:id values. The 'original' 
   attribute contains the entry's initial atom:id value. atom:source 
   elements MUST have an 'original' attribute.

}}}

== Impacts ==

== Notes ==

----

CategoryProposals