Subject: atom:id - comparison and generation

From: Bill de hÓra <[email-masked]>
Date: Sun, 29 Aug 2004 15:01:24 +0100

Having watched this thread on URIs v Strings,

  http://www.imc.org/atom-syntax/mail-archive/msg08886.html

go on for many posts and decline into an intellectually slothful 
shouting match, I suspect the debate is not framed in a manner that 
can produce a satisfactory outcome, something I indicated in that 
thread.

Here's why URIs v Strings will go no-where. It's a false dilemma. 
Arguing to URIs v Strings is like arguing to Waves v Particles but 
worse, without any rigor for either position. The truth is we 
require IDs to be both URIs and Strings depending on context.

I'd like to suggest any further such discussion on atom identifiers 
be done in terms of Comparison and Generation. Here is what I 
perceive as consensus.

Generation:

Use of functions which return URIs for use as atom:id.

Comparison

Use of functions which compare pairs of atom:id constructs as 
case-sensitive Strings.

So, what are we arguing about?

The remaining issues seems to be in concern about confusing 
implementors or users, and ascertaining either post processing rules 
on generators or pre-processing rules on comparators to reduce the 
likelihood of mismatches.

One issue seems to be that treating URIs as not-URIs (Strings) is 
confusing. I think this might be true, but I think it is mostly true 
because we start threads focusing on URIs and Strings themselves and 
not on how identifiers are to be /used/. It seems that people have 
generally accepted that URIs generators are to be used, but it's my 
belief that much confusion evaporates when you explain atom:id in 
use contexts rather than variant formats.

The primary issue appears to be canonicalization - some people feel 
it will be probably be helpful, some people feel it probably won't 
be helpful and there was at least one view that it would possibly be 
detrimental. IMHO, that comes down to:

  1. Say generation SHOULD post-process
    to canonical form before emitting

  2. Say generation MUST post-process
    to canonical form before emitting

  3. Say nothing about canonical forms
     for generation

I did not see corresponding positions around comparison 
pre-processing.

My opinion is that 1 is a sufficiently appeasing non-decision that 
allows us to move on.

I would not be inclined to keep Postel's law in mind when trying to 
establish an optimal tradeoff, allocation of burden, or just trying 
to think clearly about this issue. If that's heresy so be it, but my 
opinion is that that law can be used here to support conflicting 
arguments and thus isn't specifically helpful.

FYI:

This was the summary of consensus roughly 10 days ago,

  http://www.imc.org/atom-syntax/mail-archive/msg08572.html

I have not included views that suggests atom:id is not a useful 
construct as designed since it is a separate matter to the above and 
is mostly being discussed in a separate thread. I haven't seen any 
alternative proposals to atom:id, but in not keeping up with the 
wiki lately I might have missed them. [I have seen an alternative 
approach from Bob Wyman but I don't think he's proposed it for Atom 
use.]

cheers
Bill

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Sun, 29 Aug 2004 17:59:20 +0200

On Sun, 29 Aug 2004 15:01:24 +0100, Bill de hÓra <[email-masked]> wrote:

>   1. Say generation SHOULD post-process to canonical form before
>      emitting
>
>  [....]
>

I agree with this. I actually think everyone who wants c14n agrees with  
this. The ones that have been silent about this option are the ones that  
are against c14n. If they aren't comfortable with «SHOULD post-process to  
c14n», could they at least say so? That is, just '-1' on Bill's proposal  
here.

Would it be useful to have a survey on this, just to see how the rough  
consensus is? I have really no idea what everyone on atom-syntax feels  
about this, since there are only about six people that have been active in  
this discussion. A survey would not conclude anything, but make people's  
stand clearer.

-- 
Asbjørn Ulsberg         -=|=-        [email-masked]
«He's a loathsome offensive brute, yet I can't look away»

---

From: Julian Reschke <[email-masked]>
Date: Sun, 29 Aug 2004 18:28:23 +0200

Asbjørn Ulsberg wrote:
> this. The ones that have been silent about this option are the ones 
> that  are against c14n. If they aren't comfortable with «SHOULD 
> post-process to  c14n», could they at least say so? That is, just '-1' 
> on Bill's proposal  here.

OK,

-1.

Reason: it makes the spec more complicated without having any effect 
when clients indeed implement string comparison.

> ...

Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: [email-masked] (John Panzer)
Date: Sun, 29 Aug 2004 09:49:18 -0700

Bill de hÓra wrote:

>
> I'd like to suggest any further such discussion on atom identifiers be 
> done in terms of Comparison and Generation...
>
+1.

>    to canonical form before emitting

+1.

(I have followed this thread, though not contributed to the discussion 
as there seemed to be plenty of discussion already.)

-John
http://journals.aol.com/panzerjohn/abstractioneer

---

From: Danny Ayers <[email-masked]>
Date: Sun, 29 Aug 2004 19:41:47 +0200

> when clients indeed implement string comparison.

There is a significant effect. For example, entries with the URIs :

http://EXAMPLE.org/post1 
and
http://example.org/post1

will incorrectly show up as different if they aren't canonicalized
before string comparison.

This isn't a particularly forced example either, it's easy to imagine
one publisher's convention being one form and a republishers being the
other.

So I say +1 to publishers/generators SHOULD canonicalize. Take your
pick for comparison, SHOULD use char-by-char is probably the easiest
for all concerned.

Cheers,
Danny.

---

From: Dare Obasanjo <[email-masked]>
Date: Sun, 29 Aug 2004 11:06:38 -0700 (PDT)

> one publisher's convention being one form and a

Really? Then show me examples of where it has happened
in the past. The examples I've seen in the wild based
on customer reports in RSS Bandit are 

1.) http://www.example.com/post1

and 

    http://example.com/post1

AND 

2.) http://www.example.com/GUID-uses-ABCDEF

and 

  http://www.example.com/guid-uses-abcdef

Neither of which is helped by canonicalization.
Canonicalization helps contrived edge cases while not
applying to the actual problems that occur in the
wild.  However like I've said I see nothing wrong in
telling people canonicalization is a good idea.
Requiring it on the other hand...  

> pick for comparison, SHOULD use char-by-char is

Surely you mean MUST? Without a MUST requirement then
comparison of IDs will be an interoperability fiasco. 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #23
I will keep a special cache of low-tech weapons and train my troops in their use. That way -- even if the heroes manage to neutralize my power generator and/or render the standard-issue energy weapons useless -- my troops will not be overrun by a handful of savages armed with spears and rocks.

__________________________________
Do you Yahoo!?
Take Yahoo! Mail with you! Get it on your mobile phone.
http://mobile.yahoo.com/maildemo

---

From: Julian Reschke <[email-masked]>
Date: Sun, 29 Aug 2004 20:14:27 +0200

> before string comparison.

Why are you saying "incorrectly"? If we spec string comparison, this is 
*exactly* what the producer should expect, it's easy to test and will 
quickly be discovered.

> one publisher's convention being one form and a republishers being the

In which case the republisher broke the spec. Let's have good test 
cases, and this shouldn't be an issue.

> pick for comparison, SHOULD use char-by-char is probably the easiest

Unless we make string comparison a MUST, I fail to see how IDs can work 
as expected.

Julian

---

From: Danny Ayers <[email-masked]>
Date: Sun, 29 Aug 2004 20:51:56 +0200

On Sun, 29 Aug 2004 20:14:27 +0200, Julian Reschke
<[email-masked]> wrote:
> Why are you saying "incorrectly"? 

Because we're talking about URIs. They both identify the same resource
according to rfc2396bis.

If we spec string comparison, this is
> *exactly* what the producer should expect, it's easy to test and will

...and will be inconsistent with the rest of the web.

> Unless we make string comparison a MUST, I fail to see how IDs can work

Ok, sure, make it a MUST - Dare seems to like that too. 

Following the discussion surrounding client handling of ill-formed
XML, I'm starting to suspect it's a waste of time trying to argue
about spec'ing /anything/ for the consumer. The lowest common
denominator holds all the aces. But all the more reason to try and get
things right for the producers.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Julian Reschke <[email-masked]>
Date: Sun, 29 Aug 2004 21:30:13 +0200

> Because we're talking about URIs. They both identify the same resource

No, we are talking about Atom IDs, which are strings taking the 
syntactical form of a URI.

> ...and will be inconsistent with the rest of the web.

Such as RDF and XML namespaces?

Best regards, Julian

---

From: Bill de hÓra <[email-masked]>
Date: Sun, 29 Aug 2004 21:02:16 +0100

> when clients indeed implement string comparison.

Julian, when you say complicated, do you mean it makes the spec more 
complicated for those generating IDs to implement in the absence of 
any benefit?

cheers
Bill

---

From: "Bob Wyman" <[email-masked]>
Date: Sun, 29 Aug 2004 16:45:14 -0400

> 1. Say generation SHOULD post-process to canonical form before emitting
	This would be acceptable but unfortunate.	

> 2. Say generation MUST post-process to canonical form before emitting
	+1. This would be best and does the most for interop.

> 3. Say nothing about canonical forms for generation
	-1. This would be very bad.

String comparision should be a MUST.

		bob wyman

---

From: Martin Duerst <[email-masked]>
Date: Mon, 30 Aug 2004 10:00:04 +0900

Hello Bill,

Many thanks for your refraiming of the debate.

At 15:01 04/08/29 +0100, Bill de hモra wrote:

>The primary issue appears to be canonicalization - some people feel it 
>will be probably be helpful, some people feel it probably won't be helpful 
>and there was at least one view that it would possibly be detrimental. 
>IMHO, that comes down to:
>
>I did not see corresponding positions around comparison pre-processing.
>
>My opinion is that 1 is a sufficiently appeasing non-decision that allows 
>us to move on.

I have to say I don't like number 1. I have nothing against
in some way or another recommending that IDs be canonical
where possible, but I think explicitly talking about post-processing
is wrong. ID generation SHOULD produce IDs that are canonical
where possible, it should not rely on a post-processing step.
As an example (explicitly taking one that in practice should
be a non-issue), if a generator produces http://www.example.org:80/foo
and I have to post-normalize that to http://www.example.org/foo,
then something is wrong.

So I'd propose to either rewrite 1. above, or to add 4., as follows:

1.(new)/4. Say generation SHOULD produce IDs in as canonical
    a form as possible

I hope that this will help Julian a bit, because it removes
the additional step of post-canonicalization. Generating
something canonical is much easier than post-processing, in
particular also because the generator has much more knowledge
(e.g. about the scheme,...). I would even go as far as claiming
that most generators out there already produce something that
is very close to canonical.

Please also note that I explicitly use the wording "as canonical
a form as possible", because it may not always be clear what
the canonical form is in a specific case.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 30 Aug 2004 09:48:20 +0900

>...and will be inconsistent with the rest of the web.

Are you saying that XML Namespaces and RDF, which both use
string comparison, are not part of the Web?

Regards,    Martin.

---

From: Walter Underwood <[email-masked]>
Date: Sun, 29 Aug 2004 20:50:20 -0700

> string comparison, are not part of the Web?

Please, that wasn't the statement at all. It is about consistency
with the recommended URI handling in another major spec. Much more
central to the web than Atom, I might add.

You can also argue that both of those are designed to be used
independently of the web. Atom is not.

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Julian Reschke <[email-masked]>
Date: Mon, 30 Aug 2004 09:19:47 +0200

Bill de hÓra wrote:

> complicated for those generating IDs to implement in the absence of any 
> benefit?

- it makes the spec more complicated (measured in words, or requirements 
imposed on implementors)

- it also makes ID generation more complicated

As far as I can tell, a spec that doesn't say anything about 
canonicalization in this place still defines the same protocol; and the 
risk that people get it wrong is actually *smaller*, because client 
implementors will not possibly get the idea that 
canonicalization-upon-receipt may be ok.

Julian

---

From: Danny Ayers <[email-masked]>
Date: Mon, 30 Aug 2004 13:48:19 +0200

On Sun, 29 Aug 2004 20:50:20 -0700, Walter Underwood <[email-masked]> wrote
> independently of the web. Atom is not.

Thanks Walter.

I'd also add that the use of URIs as ids in XML namespaces is very
different from that planned for Atom. I can't speak for anyone else,
but I find I rarely need more than maybe half-a-dozen namespaces in a
specific kind of document. So for doc generation my usual approach is
to hard-code the URIs as string constants, and reuse those constants.
I think it unlikely that anyone will be hardcoding and reusing
individual entry ids.

The point of RDF being used outside the web is well made. But
following what Roy Fielding said on the matter, I've a feeling it
might be worth trying to encourage canonicalization-for-generators
into any future RDF spec revisions.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Graham <[email-masked]>
Date: Mon, 30 Aug 2004 10:36:39 -0700

I'm OK with the spec saying people MAY canonicalize if that's what they 
want to. SHOULD is way too strong (it means basically MUST unless you 
have a really good excuse).

Graham

---

From: "Laurent Le Meur" <[email-masked]>
Date: Mon, 30 Aug 2004 20:38:11 +0200

Laurent Le Meur

  > From Martin Duerst

+1 for the idea; post-processing is a fuzzy concept in the spec of a format.

  > the canonical form is in a specific case.

-1 for the form; "as xx as possible" has no place in a spec. 
I would rather read "Generation SHOULD produce ID as a URI, generated in a
canonical form" + a reference to the rules for canonicalization (eg
rfc2396bis section 6.3 "Canonical Form", as soon as the rfc is ratified) + a
note in the guidelines (ie out of the specs) about how to deal with special
cases (as exposed in previous mails).

Regards,   Laurent Le Meur.

-
		AVERTISSEMENT

Le prיsent mail et ses piטces jointes sont confidentiels et destinיs au seul usage des personnes ou entitיs auxquelles ils sont adressיs. Si vous avez reחu cet e-mail par erreur, veuillez contacter dans les plus brefs dיlais son expיditeur et effacer le contenu du message de votre systטme informatique. Toute divulgation, distribution ou copie de cet e-mail est strictement interdite.

-
		DISCLAIMER

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please contact the sender and delete the email from your system. If you are not the named addressee you should not disseminate, distribute or copy this email.

-

---

From: Martin Duerst <[email-masked]>
Date: Tue, 31 Aug 2004 15:50:31 +0900

At 10:36 04/08/30 -0700, Graham wrote:
>want to. SHOULD is way too strong (it means basically MUST unless you have 
>a really good excuse).

I was thinking about this, too. I agree that SHOULD, as defined in
RFC 2119, is too strong. But MAY is usually used in cases where
it describes a protocol option, so it also doesn't look like exactly
what we want. The IETF doesn't seem to have a formal way of labeling
something as just a good idea.

Regards,   Martin.

---

From: Sam Ruby <[email-masked]>
Date: Tue, 31 Aug 2004 08:13:04 -0400

Martin Duerst wrote:
>> I'm OK with the spec saying people MAY canonicalize if that's what 
>> they want to. SHOULD is way too strong (it means basically MUST unless 
>> you have a really good excuse).
> 

The way Namespaces in XML 1.1 accomplishes this is to avoid the usage of 
any capital words in the expressing of this requirement whatsoever. 
This exact wording is picked up in

http://www.intertwingly.net/wiki/pie/PaceIdConstruct

Note: this is also exactly in accordance with the compromise proposed by 
Paul:

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Tue, 31 Aug 2004 14:22:26 +0200

> something as just a good idea.

If it's "just a good idea", it shouldn't be part of the normative 
protocol description. Put it into a best practices appendix.

Julian

---

From: Dare Obasanjo <[email-masked]>
Date: Tue, 31 Aug 2004 07:23:51 -0700 (PDT)

I was +1 for that compromise then and I still like it
now [as long as relative URIs are explicitly
disallowed which PaceIdConstruct does]. 

=====

_______________________________
Do you Yahoo!?
Win 1 of 4,000 free domain names from Yahoo! Enter now.
http://promotions.yahoo.com/goldrush

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Wed, 01 Sep 2004 00:05:10 +0200

On Mon, 30 Aug 2004 13:48:19 +0200, Danny Ayers <[email-masked]>  
wrote:

> different from that planned for Atom.

Indeed.

> I can't speak for anyone else, but I find I rarely need more than
> maybe half-a-dozen namespaces in a specific kind of document.

Same here.

> So for doc generation my usual approach is to hard-code the URIs as
> string constants, and reuse those constants.

Same here. :-)

> I think it unlikely that anyone will be hardcoding and reusing

That would be clear breakage of the specification, so I hope not. Another  
thing is the amount of atom:id's and XML namespaces that are generated,  
how they are generated, and by whom.

Bob will generate atom:id. Bob seldom generates XML namespaces. atom:id  
will mostly be created automatically. XML namespaces are mostly created  
manually. I would state that there are already more atom:id's out there  
than XML namespaces.

These differences make the two URI usages rather incomparable, imho.

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Wed, 01 Sep 2004 00:11:30 +0200

On Mon, 30 Aug 2004 20:38:11 +0200, Laurent Le Meur  
<[email-masked]> wrote:

> I would rather read "Generation SHOULD produce ID as a URI, generated in  
> a canonical form"

I read this as optional URI syntax for atom:id -- that atom:id SHOULD be a  
URI, but may be something else. I'm sure that's not what you're saying,  
but separating the sentence like that makes me read that into it.  
Something like this might be better:

   The generation of an ID SHOULD produce a URI in canonical form

Not sure if that's perfect either, but not parting up the sentence makes  
it clearer that the SHOULD applies to «canonical», not «URI». Imho, at  
least.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 31 Aug 2004 17:19:04 -0700

>comparison of IDs will be an interoperability fiasco.

This is an interesting avenue to explore. When does comparison of IDs 
cause an interop problem? From my limited understanding, comparing 
IDs is always for display or creating meta-feeds. While both of these 
are helped by clear and unambiguous IDs, I don't see the interop 
problem with either false positives or false negatives from a bad 
comparison.

What are the interoperability issues here?

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Dare Obasanjo <[email-masked]>
Date: Tue, 31 Aug 2004 17:51:33 -0700 (PDT)

> What are the interoperability issues here?

IDs are used for determining identity. If I'm
synchronizing the state of two aggregators [something
aggregators like RSS Bandit & Newsgator already do]
then I want to be a 100% sure they use the same ID
comparison semantics so I don't lose entries or have
the state similarly screwed up. This was a big problem
when myself and a bunch of aggregator authors where
trying to spec SIAM[0] and something that ate up weeks
of discussion then eventually went nowhere because we
couldn't solve the identity problem for RSS. 

The fact that Atom will require an ID attribute is a
step in the right direction. But a bigger step is
ensuring that all clients use THE SAME ALGORITHM for
comparing IDs. 

[0] http://www.25hoursaday.com/draft-obasanjo-siam-01.html

=====