Subject: Re: Compromise on canonicalization

From: Graham Parks <[email-masked]>
Date: Wed, 01 Sep 2004 01:27:27 -0400

On Tuesday, August 31, 2004, at 08:20AM, Sam Ruby <[email-masked]> wrote: 

>The way Namespaces in XML 1.1 accomplishes this is to avoid the usage of 
>any capital words in the expressing of this requirement whatsoever. 
>This exact wording is picked up in 
> 
>http://www.intertwingly.net/wiki/pie/PaceIdConstruct 
> 
>Note: this is also exactly in accordance with the compromise proposed by 
>Paul: 
> 
>http://www.imc.org/atom-syntax/mail-archive/msg08572.html 

Remove the word "strongly" and I'm on board. 

Graham

---

From: Danny Ayers <[email-masked]>
Date: Wed, 1 Sep 2004 19:04:36 +0200

-1.

The compromise seems to have shifted towards effectively dropping any
real encouragement for canonicalization. Will the next compromise be
to drop the URI requirement?

Given that id-reliability is a significant issue in syndication, I
think the suggestions of RFC 2396bis are worth taking on board, and so
believe canonicalisation ought to be a SHOULD (at id generation). The
cost is minimal, and it prevents a whole range of potential problems.

On a metapoint, this is a specification we're talking about, not an
individual application and the usual design practices don't
necessarily apply *.  The "simplest thing that will possibly work" now
may be totally borked in 3 months time.

Cheers,
Danny.

* "The hallmarks of a good design is when you have nothing left to
take out" - how about the  seatbelts and airbags of your car design,
like, when did you last use them?

-- 

http://dannyayers.com

---

From: Bill de hÓra <[email-masked]>
Date: Wed, 01 Sep 2004 19:20:17 +0100

> to drop the URI requirement?

Although they're separate, you could indeed drop URIs and use 
soemthing else; they're just syntax for one half of the atom:id coin 
after all. What's important are the functions and invariants 
operating over both halves of that coin and whether canonicalization 
belongs as a step.

I'm not being a smartass; I want to stay focused and wrap this thing 
  before we go into another week. My opinion is that compromise 
(consensus?) is shifting due to design by exhaustion.

> cost is minimal, and it prevents a whole range of potential problems.

When we say the cost of canonicalization is minimal, what are we 
saying? That someone can implement it properly in an afternoon?

> may be totally borked in 3 months time.

What I would like to see are clear simple explanations as to why 
canonicalization is or is not a neccessary(must)/useful(should) part 
of the generation function. That means no prognostications or 
speculations or other crystal ball-gazing on the relative burdens or 
outcomes, which we seem to be not able to get past in terms of 
argument.  This is where we're stuck - if something has a cost it 
can be measured; anything else is economics.

[I thought about summarizing the threads again, but if I don't have 
time enough to go down to Galway town to play foaf drinking games, 
there's no chance of me doing that.]

I've just finished an 'i12n' of xmpp jids; canonicalization can't be 
possibly be more of a pita than that (if it is, that might just sway 
me).  Do we need code samples to get past this?

cheers
Bill

---

From: Dare Obasanjo <[email-masked]>
Date: Wed, 1 Sep 2004 11:44:30 -0700 (PDT)

> to drop the URI requirement?

http://www.datanation.com/fallacies/distract/ss.htm

Oh look, the slippery slope logical fallacy. 

Anyway, no one has suggested dropping encouragement.
However SHOULD in IETF terminology is effectively a
MUST unless there are extenuating circumstances. Many
on these thread believe that canonicalization is not 
significant enough to make it basically a requirement.

> cost is minimal, and it prevents a whole range of

No it doesn't. As an aggregator author, the most main
issues I see in with ID generation in the wild are not
helped much [if at all] by canonicalization. 

=====
THINGS TO DO IF I BECOME AN EVIL OVERLORD #23
I will keep a special cache of low-tech weapons and train my troops in their use. That way -- even if the heroes manage to neutralize my power generator and/or render the standard-issue energy weapons useless -- my troops will not be overrun by a handful of savages armed with spears and rocks.

_______________________________
Do you Yahoo!?
Win 1 of 4,000 free domain names from Yahoo! Enter now.
http://promotions.yahoo.com/goldrush

---

From: Dare Obasanjo <[email-masked]>
Date: Wed, 1 Sep 2004 12:10:41 -0700 (PDT)

The perils of rewriting. Reposting with superflous
text removed. 

--- Dare Obasanjo <[email-masked]> wrote:
> 
> No it doesn't. As an aggregator author, the 
> main
> issues I see with ID generation in the wild are
> not

=====

__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - Send 10MB messages!
http://promotions.yahoo.com/new_mail

---

From: Jeremy Smith <[email-masked]>
Date: Wed, 01 Sep 2004 17:46:19 -0400

> of the generation function.

    Imagine there are two web sites: Planet Aquarium 
(http://planetaquarium.example.com) and Planet Tadpole 
(http://planettadpole.example.com).  Both are sites that aggregate 
content from other sites' atom feeds, much like the current "Planet *" 
sites do.  Both happen to aggregate the content from the blog "I Farm 
Frogs" (http://ifarmfrogs.example.com) and redisplay it on their sites 
and offer it on their respective atom feeds, 
http://planetaquarium.example.com/feed.atom and 
http://planettadpole.example.com/feed.atom.
    The kicker is that ifarmfrogs.example.com only has an RSS 2.0 feed.  
So, each of the Planet sites must transform the RSS 2.0 items into atom 
entries.  The original RSS 2.0 feed looks like:

<item>
<title>Tadpoles Just Hatched in Aquarium</title>
<link>http://iFarmFrogs.example.com/2004/04/04/index.html</link>
<description>Some tadpoles just hatched today</description>
<guid>http://iFarmFrogs.example.com/2004/04/04/index.html</guid>
</item>

    Both sites use the guid element as the atom ID; however, Planet 
Aquarium lowercases the host name while Planet Tadpole does not.
    Joe User is subscribed to both Planet feeds and is constantly 
getting duplicates.  But, who does Joe User email to let them know of 
their error?  Does he email Planet Aquarium for lowercasing the hostname 
when they convert an RSS 2.0 guid to an atom id?  Does he email Planet 
Tadpole for not lowercasing the hostname?  Does he email his aggregator 
software's company/author for not canonicalizing the ids before comparison?
    I strongly believe that the specification should weigh in one way or 
another on this matter so that matters like this can be cleared up 
according to the spec.  And, I believe that the clearest way to specify 
a resolution to the above problem is by saying atom producers SHOULD 
canonicalize ids.
    Granted, the above example is contrived, but the whole world is just 
the agglomeration of all possible contrivances.  The spec should strive 
to be complete.

Jeremy

---

From: Robert Sayre <[email-masked]>
Date: Wed, 01 Sep 2004 17:58:16 -0400

Jeremy Smith wrote:

> entries.  

The problem with this one is that it assumes Atom feeds must preserve 
RSS GUIDs. There is no defined method of translating RSS2 to Atom, so 
there is no reason to assume that sites will come up with identical 
atom:id values when performing such translations. Garbage in, garbage out :)

Performing such translations may be a good topic for a separate 
standards document when Atom is closer to completion.

Robert Sayre

---

From: "jt" <[email-masked]>
Date: Wed, 1 Sep 2004 18:16:11 -0400

I haven't been keeping up that much.  Without regard to this specific use
case, the below is, in actual fact, extremely incorrect.

It'd be a good topic up-front for (to me anyhoo) obvious reasons:  That is,
IF you're interested in Atom being adopted, besides being spec'd, I mean.
Yeah, garbage in regards to what is important leads to garbage out, so I
agree with that much...;-D

Bowing out, 'gain...

| -----Original Message-----
| From: [email-masked]
| [mailto:[email-masked]]On Behalf Of Robert Sayre
| Sent: Wednesday, September 01, 2004 5:58 PM
| To: Jeremy Smith
| Cc: [email-masked]
| Subject: Re: Compromise on canonicalization
|
|
|
| Jeremy Smith wrote:

| >    Both sites use the guid element as the atom ID; however, Planet
| > Aquarium lowercases the host name while Planet Tadpole does not.
|
| The problem with this one is that it assumes Atom feeds must preserve
| RSS GUIDs. There is no defined method of translating RSS2 to Atom, so
| there is no reason to assume that sites will come up with identical
| atom:id values when performing such translations. Garbage in,
| garbage out :)
|
| Performing such translations may be a good topic for a separate
| standards document when Atom is closer to completion.
|
| Robert Sayre
|

---

From: Jeremy Smith <[email-masked]>
Date: Wed, 01 Sep 2004 18:20:58 -0400

> atom:id values when performing such translations. Garbage in, garbage 

    Good point.
    The example I was trying to hit is two producers of atom feeds both 
operating from the same "data source"; but one canonicalizing URIs while 
the other was not.  And, with the spec not weighing in either way on the 
issue, who do you email?

---

From: Dare Obasanjo <[email-masked]>
Date: Wed, 1 Sep 2004 16:46:58 -0700 (PDT)

> Aquarium lowercases the host name while Planet

I really hate contrived examples. Why would Planet
Aquarium lowercase the host name but not the entire
URI? Sounds pretty arbitrary to me.  What if Planet
Aquarium also strips out the 'http://' scheme but
preserves the case? 

> according to the spec.

The Atom spec should weigh in on how to convert RSS
2.0 to Atom? Will it also do this for RSS 1.0 as well?
As an aggregator author, the only thing I need from
the spec is a well defined way to tell if two entries
are identical or not. This basically calls for a well
defined comparison algorithm. Everything else is just
attempts to second guess what buggy implementations
MIGHT do if written  by bozos who didn't read the
spec. 

=====

---

From: Martin Duerst <[email-masked]>
Date: Thu, 02 Sep 2004 10:05:37 +0900

Hello Bill,

At 19:20 04/09/01 +0100, Bill de hモra wrote:

>canonicalization is or is not a neccessary(must)/useful(should) part of 
>the generation function. That means no prognostications or speculations or 
>other crystal ball-gazing on the relative burdens or outcomes, which we 
>seem to be not able to get past in terms of argument.  This is where we're 
>stuck - if something has a cost it can be measured; anything else is economics.

Earlier, I commented that we shouldn't think about canonicalization
about a post-generation step. But I think I have to be even clearer:
I don't think we should think (or write) about canonicalization about
a step in the generation process, or as a part of the generation
function. Ideally, it should be a *property* of the generation function.

Ideally, this property is achieved without any extra code, just by
having the right code. To take a not-so-relevant example (because
I don't think IDs should contain a lot of %-encodings), RFC 2396bis
says that hex letters in %-encoding should be upper case. The generation
code can be written to just produce upper case; no need to actually
have an upper-casing function.

In some cases, there may be some extra code (e.g. lower-case a domain
name, only output port if not equal to default port), but there should
not be any section in the generation function that says 'and now we
canonicalize'.

As with many other cases, building something into the system by design
can be extremely cheap if you think about it early enough.

Regards,    Martin.

---

From: "jt" <[email-masked]>
Date: Wed, 1 Sep 2004 21:12:52 -0400

| -----Original Message-----
| From: [email-masked]
| [mailto:[email-masked]]On Behalf Of Dare Obasanjo
| Sent: Wednesday, September 01, 2004 7:47 PM
| To: Jeremy Smith; [email-masked]
| Subject: Re: Compromise on canonicalization

To those for whom this is too wordy, AIA.  And to those who do not
understand any or all of my comments, sorry to be unable to explain any
further, at this time.  Questions can be considered rhetorical, as far as
I'm concerned with this 'discussion'.

| I really hate contrived examples.

Notwithstanding the rest of your comments, I would note that some people
work better with concrete examples, and some prefer theoretical models.
Most, in some varying proportions.

| The Atom spec should weigh in on how to convert RSS
| 2.0 to Atom? Will it also do this for RSS 1.0 as well?

"Weigh in" may be too heavy-handed to some...  But as a practical matter,
"make conducive for"...?  I guess even "keep in mind" in spec'ing, *OR
"allow for and better yet optimize" may seem like too much "weighing in", to
some.

Again, as a practical matter adoption could be easier accomplished by
leverage, in my experience, and I've noticed some fair bit of RSS2.

| As an aggregator author,

How many paid users do you support??  And what's the per seat price, ie a
very concrete example of the users' expectation levels, if I may ask??  And
if you (any-a you) think users of freebies and/or aggregator authors MUST
dictate, let alone SHOULD dictate, what the market will do *AND/*OR won't
do, you might find out otherwise.  At least there is a possibility that
ANYone might find out some things that surprise, at some stage in this
what-you-play-as 'game'.

YmMV, of course.  I'm outta gas for this stuff above and below, snipped or
left in...

Consider me weapon-less and neutralized, if you will...  IF not, then
consider me however you consider me (like this is a multiple-choice
possibility...)-;.

| =====
| THINGS TO DO IF I BECOME AN EVIL OVERLORD #23
| I will keep a special cache of low-tech weapons and train my
| troops in their use. That way -- even if the heroes manage to
| neutralize my power generator and/or render the standard-issue
| energy weapons useless -- my troops will not be overrun by a
| handful of savages armed with spears and rocks.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 02 Sep 2004 09:57:05 +0900

At 16:46 04/09/01 -0700, Dare Obasanjo wrote:

>I really hate contrived examples.

I agree. There are currently no rules of how to
come up with ids for things from outside Atom (RSS,...),
and I agree we should not have any specific rules for this.
[we might think about more general rules, maybe to help
people decide whether they need a new ID or they can
reuse something]

However, if we extend the thinking for Atom
(id generator generates hopefully canonical ids, then
it becomes very clear that taking some distinguishing
URI from some other technology and using it as an ID in
atom is not ID generation, but just ID propagation.
And for this, the rules for atom should apply: Don't
change a single character! So for this case,
Planet Aquarium is wrong, and Planet Tadpole is
right.

Maybe the atom spec should say something along the
lines that if identifiers are taken from other
protocols, they should not be canonicalized?

>Why would Planet
>URI?

This is a minor detail, but anyway: because the host
name in http can be lowercased without changing resolution,
but the rest of the URI can't. If it were a href on
an <a> or a src on an <img>, it would work. But of
course, it's not.

Regards,    Martin.

>Sounds pretty arbitrary to me.  What if Planet
>I will keep a special cache of low-tech weapons and train my troops in 
>their use. That way -- even if the heroes manage to neutralize my power 
>generator and/or render the standard-issue energy weapons useless -- my 
>troops will not be overrun by a handful of savages armed with spears and rocks.
>
>
>

---

From: Mark Pilgrim <[email-masked]>
Date: Wed, 1 Sep 2004 21:20:36 -0400

On Wed, 1 Sep 2004 16:46:58 -0700 (PDT), Dare Obasanjo <[email-masked]> wrote:
> Everything else is just

Surely you mean morons.  Bozos are people who can't generate well-formed XML.

http://diveintomark.org/archives/2004/08/16/specs
http://www.tbray.org/ongoing/When/200x/2004/01/11/PostelPilgrim

-- 
Cheers,
-Mark

---

From: "jt" <[email-masked]>
Date: Wed, 1 Sep 2004 21:30:27 -0400

Dag, I forgot to ask these 'rhetorical' questions.

| the only thing I need from
| the spec is a well defined way to tell if two entries
| are identical or not.

Compromise tends to obliviate well-defined, or is that not your all's design
experience...??

| This basically calls for a well
| defined comparison algorithm.

Two (or some reasonable number > 1) well-defined algorithms being different
than compromise, (and, for example, I assume you 'morons' and 'bozos' have
System APIs to make date formats a moot point, right?,) then what exactly is
the holdup??  It may have been answered already [or re-answered, as in
Martin Duerst's recent post?] but I've not followed close enuf, like I said.
Then again, this seems to be a permathread, so it may not be real obvious
yet to some other 'lamers' besides me. ?

<bowing>
out

---

From: Bill de hÓra <[email-masked]>
Date: Thu, 02 Sep 2004 13:28:52 +0100

> not be any section in the generation function that says 'and now we

That's a fair point, I'll try to stop calling it a step and think of 
it as intrisinc. Tho' I am thinking about situations where we've 
decided to have it happen (assuming we can make a decision) and one 
has to bolt something on to get the right thing emitted.

cheers
Bill

---

From: Graham Parks <[email-masked]>
Date: Thu, 02 Sep 2004 10:08:28 -0700

>has to bolt something on to get the right thing emitted.

What happens for systems that are passing through ids? Do they have to validate? What if they blindly canonicalize, possibly changing the id?.

Graham

---

From: Graham Parks <[email-masked]>
Date: Thu, 02 Sep 2004 10:12:36 -0700

>of the generation function.

I maybe regret the tone of this one, but it's still a fairly accurate listing of what the benefits of c14n amount to:

http://www.imc.org/atom-syntax/mail-archive/msg09142.html

(ie It may not be expensive, but you get what you pay for)

Graham

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Sep 2004 19:19:00 +0200

> What happens for systems that are passing through ids? Do they have to validate? What if they blindly canonicalize, possibly changing the id?.

That would be a bug. Just like canonicalization-before-comparison.

Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Danny Ayers <[email-masked]>
Date: Thu, 2 Sep 2004 20:27:20 +0200

On Mon, 30 Aug 2004 10:36:39 -0700, Graham <[email-masked]> wrote:
> I'm OK with the spec saying people MAY canonicalize if that's what they
> want to. SHOULD is way too strong (it means basically MUST unless you
> have a really good excuse).

Ok, this thread does seem to be stuck. In the interests of moving on:

+1 to MUST use URIs as ids
+1 to consumers/republishers MUST maintain the ids char-by-char
+1 to id generators (publishers) MAY canonicalize URIs
+1 to id comparators (consumers/republishers) MAY use char-by-char comparison

Just for the record, the first two are my first choice, I think the
3rd is too weak, I've not heard any good excuse for not
canonicalizing. The 4th follows because it won't make a deal of
difference to the number of false comparisons if nothing is mandated
for the generators of ids.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Sep 2004 20:38:51 +0200

> for the generators of ids.

So a recipient that compares IDs after uppercasing all characters in the 
ID would be compliant? How is this supposed to work?

+1 to id comparators (consumers/republishers) MUST use char-by-char 
comparison

Those who disagree please explain how the ID is supposed to actually do 
what it's for otherwise.

Best regards, Julian

---

From: Eric Scheid <[email-masked]>
Date: Fri, 03 Sep 2004 04:42:48 +1000

> +1 to MUST use URIs as ids

remembering: absolute (not relative) URIs, and allowing #fragments ... so
whatever special name that construct has, let it be.

e.

---

From: Antone Roundy <[email-masked]>
Date: Thu, 2 Sep 2004 12:59:51 -0600

> for the generators of ids.
How about:

+1 to SHOULD use URIs as ids (to enable carry-forward of non-URI ids 
from other formats)
+1 to EVERYONE MUST maintain the ids char-by-char (including 
publishers, which you didn't mention)
+1 to id generators (publishers) might consider canonicalizing URIs to 
help them avoid accidentally altering ids
+1 to comparators (consumers/republishers) MUST use char-by-char 
comparison

On MUST vs SHOULD for the first, what would each lose:
MUST:
* Can't compare non-Atom ids to Atom ids (not such a big deal, but...)
* Can't convert from non-Atom sources to Atom and keep the same ids--so 
if a non-Atom resource with a non-URI id is converted to Atom by 
different systems, they are likely to have different ids, and thus show 
up as duplicates.

SHOULD:
* Publishers are more likely to produce non-unique IDs

The second is the whole point of canonicalization, if I'm not mistaken 
(and thus the third is just to help with the second).

The fourth follows from mandating the second--there would be no reason 
to do otherwise.

---

From: Dare Obasanjo <[email-masked]>
Date: Thu, 2 Sep 2004 12:39:10 -0700 (PDT)

> use char-by-char comparison

-10

Why the heck are you dragging this thread out? Paul
gave a great description of the consensus on the list.
You're above description is a giant leap back. 

"id comparators (consumers/republishers) MAY
 use char-by-char comparison" ?

MAY? WTF? Have you misread everything that has been
posted about why existing specs like RDF and XML
namespaces use char by char comparison? Have you
missed statements by different aggregator authors
pointing out that the main issue is not the comparison
algorithm? 

Canonicalization buys you jack. The compromise was to
have some wording in the spec saying its a good idea
so stonewallers like you who seem determined to impede
progress of the working group can let us move on to
other issues. Instead you have not only stuck to your
obsession with requiring canonicalization despite a
number of examples showing its ineffectiveness in the
real world as opposed to your contrived examples[0]
but now are trying to open discussion on another
consensus issue. 

Hell, no. 

[0] http://www.imc.org/atom-syntax/mail-archive/msg09211.html

=====

---

From: Danny Ayers <[email-masked]>
Date: Thu, 2 Sep 2004 22:11:26 +0200

> You're above description is a giant leap back.

My estimate of opinion on each is:

URIs as id  - strongly pro
maintain ids - strongly pro
publishers canonicalize - locked
char-by-char at client - inconclusive

Hence my stepping back on the third. Based on the technical arguments
put forward the position I suggested on the last point seems
reasonable.

> namespaces use char by char comparison? 

What I've seen on that specifically is 1. that they use it, and 2.
that not everyone's convinced it's done well, quoting Roy Fielding:
[[
It is just poorly phrased in the RDF specification.  RDF can define
how RDF does comparisons of URI references in the processing of
assertions.  However, that definition does not change the fact that
two URI that are equivalent (by their very nature) may be falsely
determined to be different by the RDF algorithm -- that is the nature
of lossy optimizations.  This is not a problem when the URI references
are provided in a reasonably canonical form.
]]

Have you
> pointing out that the main issue is not the comparison

If the main issue is not the comparison algorithm, then why are you
bothered about char-by-char comparison being a MAY?

> but now are trying to open discussion on another

1. You will note that I changed to MAY on canonicalization above.
Hardly stonewalling.
2. Your examples [0] deal with *different* URIs

> Hell, no.

If you have convincing arguments why comparison at client ought to be
a SHOULD, let's hear them.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Julian Reschke <[email-masked]>
Date: Thu, 02 Sep 2004 22:23:30 +0200

Danny Ayers wrote:

> ...

It should be a MUST...

Anyway, the reason is so that people can rely on different clients to do 
the same thing with a given set of legal IDs.

Julian

---

From: Dare Obasanjo <[email-masked]>
Date: Thu, 2 Sep 2004 15:23:27 -0700 (PDT)

> 2. Your examples [0] deal with *different* URIs

Most developers [and in fact many web servers] do not
see the difference between 

1.) http://www.example.com/GUID-uses-ABCDEF
2.) http://example.com/GUID-uses-ABCDEF
3.) http://www.example.com/guid-uses-abcdef
4.) http://WWW.EXAMPLE.COM/GUID-uses-ABCDEF

Canonicalization says (4) is equivalent to (1). That's
it. In practice I've seen producers generate
variations of (1), (2) and (3) in situations where
they expect them to be equivalent but not (4). 

> a SHOULD, let's hear them.

Comparison on the client should be a MUST. If an Atom
feed uses the above 4 IDs over time and different
clients consider different entries to be identical
exactly how will the spec resolve who is right and who
is wrong? How about when the URI scheme is not HTTP
but instead file, nntp, news, tag URIs, etc. Are
clients supposed to know every scheme specific
comparison algorithm?  

Go read

"If your spec isn’t good enough, morons have no chance
of ever getting things right. For everyone who
complains that their software is broken, there will be
two assholes who claim that it’s not. The spec, whose
primary purpose is to arbitrate disputes between
morons and assholes, will fail to resolve anything,
and the arguments will smolder for years."

=====

---

From: Danny Ayers <[email-masked]>
Date: Fri, 3 Sep 2004 00:26:33 +0200

> the same thing with a given set of legal IDs.

I don't find that very compelling in itself, as the primary
communication path is from server to client, not client to client. At
best it means producers will have to be designed to target consumer
behaviour, rather than following specifications of their own.
Regarding the MUST, what if I want to use all the comparison ladder as
described in the URI RFC?

Whatever, without consistency in what's being served I don't think it
matters very much whether it's a MAY, SHOULD or MUST client side.

At least the morons will get along with the bozos.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Bill de hÓra <[email-masked]>
Date: Fri, 03 Sep 2004 00:49:30 +0100

Graham Parks wrote:
> What happens for systems that are passing through ids? 

I'm not sure what kinds of systems you mean. Proxies at least 
ideally shouldn't be goofing with content. Are you talking about 
server-sided aggregators?

>Do they have to validate? 

I won't be able to answer that without an example of a system.

>What if they blindly canonicalize, possibly changing the id?.

As above, but the idea of intermediaries not goofing with content 
seems a good one.

cheers
Bill

---

From: Mark Pilgrim <[email-masked]>
Date: Thu, 2 Sep 2004 21:22:25 -0400

> 4.) http://WWW.EXAMPLE.COM/GUID-uses-ABCDEF

You forgot the obvious one: http://example.com/ and
http://example.com/index.html

> they expect them to be equivalent but not (4).

So your argument is that canonicalization solves some problems but not
others, so we shouldn't use it at all?  That makes no sense.

Canonicalization solves all the problems you can solve without
actually dereferencing the URI to check for server-side redirects or
duplicate content returned.

(I feel compelled to note *ONCE AGAIN* that this entire mess is only
our problem because *some* people insist on allowing http:// URIs for
atom:id.  Other URI schemes have well-defined comparison rules that
are not ambiguous, not onerous, and not open to profiling by
individual formats.)

-- 
Cheers,
-Mark

---

From: Dare Obasanjo <[email-masked]>
Date: Thu, 2 Sep 2004 18:51:46 -0700 (PDT)

> others, so we shouldn't use it at all?  That makes

Nope, I've said several times that I'm fine with
canonicalization being mentioned in the spec. I don't
think it should be required since it doesn't buy much
in the long run. It is basically optimizing for an
edge case. 

> duplicate content returned.

Which as pointed out above isn't a whole lot. 

> are not ambiguous, not onerous, and not open to

Agreed. :) 

=====

---

From: Julian Reschke <[email-masked]>
Date: Fri, 03 Sep 2004 09:14:36 +0200

> behaviour, rather than following specifications of their own.

Sorry? I thought that the whole point of a new spec is that we get 
*predictable* behaviour of clients?

> described in the URI RFC?

Then that's a bug.

> matters very much whether it's a MAY, SHOULD or MUST client side.

Sorry, can't follow. The purpose of the spec is to define a feed format 
that will behave the same in different clients. If this isn't what 
you're looking for, I'm not sure what we're supposed to discuss here.

Julian

---

From: Danny Ayers <[email-masked]>
Date: Fri, 3 Sep 2004 09:55:47 +0200

> are not ambiguous, not onerous, and not open to profiling by

I'm not entirely happy with giving resources on the web with
http-retrievable representations non-http URIs. This will be the most
common case from what can be gleaned from the arguments re.
'alternate'. But given the way things are heading it is beginning to
sound like the best way of avoiding ambiguity in the definition of
ids. Ah well, roll on tag: URIs.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Danny Ayers <[email-masked]>
Date: Fri, 3 Sep 2004 10:19:11 +0200

> *predictable* behaviour of clients?

Unless I'm misinterpreting the charter, the aim to define the Atom
format, not Atom agents. Surely reliable communication is the aim,
consistent interpretation by the client only being a part of that.

People aiming to publish Atom feeds read the spec definitions to
determine what should go in to their outbound data, consumer writers
read the definitions and figure out how to get the data they want into
their application. The way things have been heading is that things are
being said about what a client should *do*, rather than saying what
Atom *is*. The producer writer has to reverse-engineer from client
behaviour to find out what is expected of their applications. That
doesn't smell good to me.

> > described in the URI RFC?

Fair enough. I have my own opinion of where the bug would lie.

> you're looking for, I'm not sure what we're supposed to discuss here.

The format doesn't behave, it contains a set of data. It's the format
that should be unambiguous, not the behaviour. Clients should know
what to expect. Definitions of what they do with the data are
orthogonal.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 3 Sep 2004 12:03:23 -0400

Dare Obasanjo wrote:
>Canonicalization says (4) is equivalent to (1). That's it. In practice
> I've seen producers generate variations of (1), (2) and (3) in 
> situations where they expect them to be equivalent but not (4).

In order to introduce some uncontrived "evidence" to this discussion, I've
put a tiny subset of the domain names that we see at PubSub.com while
monitoring the Blogosphere. You can find a selection of about 3,000 domain
names that use at least one uppercase character at:

http://bobwyman.pubsub.com/main/files/CaseInDomainName.txt

I find that about 5% of domain names that we read have at least one upper
case character in them. Please note that it actually appears fairly rare for
people to use all upper case in domain names. The normal use of uppercase
characters in domain names involves either just the "WWW." portion being
capitalized or mixed case. Mixed case, not involving the WWW is by far the
most common use of upper case characters in the domain names we see.

Character-by-character comparison MUST be required.

		bob wyman

---

From: Graham Parks <[email-masked]>
Date: Fri, 03 Sep 2004 10:10:52 -0700

>doesn't smell good to me.

Yeah actually I think we have been heading down the wrong path with how we've been discussing this. The definition of sameness should only describe what publishers have to do to comply with the spec where it says "same id", which it seems means emmitting the same characters every time. Consumers meanwhile can use any comparison method they like (how will anyone find out?). The upshot of this is that those using looser methods is vulnerable to collisions if people choose identifiers that aren't very distinct.

a) I think this unlikely - it would mean one entry being example.com and a different id having Example.com, but being identical otherwise
b) Canonicalization only solves this in a roundabout way. Much better would be to say "Make sure your ids are reasonably distinct" - It would be easy enough to come up with a list of the minimal differences required between two ids (which would mirror the c14n list).

Graham

---

From: Martin Duerst <[email-masked]>
Date: Sat, 04 Sep 2004 15:17:45 +0900

At 10:10 04/09/03 -0700, Graham Parks wrote:

>b) Canonicalization only solves this in a roundabout way. Much better 
>would be to say "Make sure your ids are reasonably distinct" - It would be 
>easy enough to come up with a list of the minimal differences required 
>between two ids (which would mirror the c14n list).

+1.

Regards,    Martin.

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Sat, 04 Sep 2004 16:43:17 +0200

On Wed, 1 Sep 2004 11:44:30 -0700 (PDT), Dare Obasanjo <[email-masked]>  
wrote:

> Anyway, no one has suggested dropping encouragement. However SHOULD
> in IETF terminology is effectively a MUST unless there are
> extenuating circumstances.

So, what do the anti-c14n-people feel about spelling «should» in lowercase  
letters? It would give almost the same kind of encouragement to the  
publishers to do c14n, but won't use interoperability as an «excuse». E.g.:

   The generated ID should be in canonical form.

-- 
Asbjørn Ulsberg         -=|=-        [email-masked]
«He's a loathsome offensive brute, yet I can't look away»

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Sat, 04 Sep 2004 17:16:50 +0200

> are not ambiguous, not onerous, and not open to profiling by

And I feel compelled to once again give my +1 to this.

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Sat, 04 Sep 2004 17:19:15 +0200

On Fri, 3 Sep 2004 09:55:47 +0200, Danny Ayers <[email-masked]>  
wrote:

> http-retrievable representations non-http URIs.

It's only the non-retrievable URI that is non-HTTP. Since you can't expect  
atom:id to be retrievable, why do you need it to be an HTTP URI?

> But given the way things are heading it is beginning to sound
> like the best way of avoiding ambiguity in the definition of

Ah, great. How many tag URI opponents are there out there?

---

From: Danny Ayers <[email-masked]>
Date: Sat, 4 Sep 2004 17:41:43 +0200

> atom:id to be retrievable, why do you need it to be an HTTP URI?

I've explained why in a document with the URI:

tag:[email-masked],2004-09-04:FindMeIfYouCan

> Ah, great. How many tag URI opponents are there out there?

Undoubtedly quite a few.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Sat, 04 Sep 2004 18:01:34 +0200

On Sat, 4 Sep 2004 17:41:43 +0200, Danny Ayers <[email-masked]>  
wrote:

>> It's only the non-retrievable URI that is non-HTTP. Since you can't  
>> expect atom:id to be retrievable, why do you need it to be an HTTP URI?
>

Funny. But inaccurate. What you've given me there is an identifier. I'm  
not an Atom consumer; at least not at the moment. What am I supposed to do  
with it? If your document has an (alternate) HTTP URI, I (as a human)  
would of course prefer that one.

>> Ah, great. How many tag URI opponents are there out there?

Yes, probably. It would be great if they spoke.

---

From: Jeremy Smith <[email-masked]>
Date: Sat, 04 Sep 2004 12:25:15 -0400

> So, what do the anti-c14n-people feel about spelling «should» in 
> lowercase  letters? It would give almost the same kind of 
> encouragement to the  publishers to do c14n, but won't use 
> interoperability as an «excuse». E.g.:
>

    Rereading "Key words for use in RFCs to Indicate Requirement Levels" 
(http://www.ietf.org/rfc/rfc2119.txt), I think MAY is appropriate.  From 
BCP 14:

>5. MAY   This word, or the adjective "OPTIONAL", mean that an item is
>   truly optional.  One vendor may choose to include the item because a
>   particular marketplace requires it or because the vendor feels that
>   it enhances the product while another vendor may omit the same item.
>   An implementation which does not include a particular option MUST be
>   prepared to interoperate with another implementation which does
>   include the option, though perhaps with reduced functionality. In the
>   same vein an implementation which does include a particular option
>   MUST be prepared to interoperate with another implementation which
>   does not include the option (except, of course, for the feature the
>   option provides.)
>
    I would +1 a "Producers of atom ids MAY use canonicalized URIs."

---

From: Julian Reschke <[email-masked]>
Date: Sat, 04 Sep 2004 18:39:58 +0200

>    I would +1 a "Producers of atom ids MAY use canonicalized URIs."

Technically that's correct.

But again; exactly how does this make the spec better? Both feeds with 
canonicalized and non-canonicalized IDs well be conformant; and any 
recipient will have to be able to deal with both. Not saying anything 
would mean the same thing.

Best regards, Julian

---

From: Tim Bray <[email-masked]>
Date: Sat, 04 Sep 2004 10:26:36 -0700

>    I would +1 a "Producers of atom ids MAY use canonicalized URIs."

Irrespective of whether we're talking about atom:id or something else, 
2396bis specifically recommends the use of canonicalized URIs for all 
the reasons that have bubbled up here, and I would see no problem if 
the WG wanted our drafts to echo that recommendation or at least point 
to it, for *all* URI usages. -Tim

---

From: Jeremy Smith <[email-masked]>
Date: Sat, 04 Sep 2004 14:05:35 -0400

> Both feeds with canonicalized and non-canonicalized IDs well be 
> conformant; and any recipient will have to be able to deal with both.

    I would be happiest with a "MUST canonicalize" in the spec, but 
there is a severe lack of consensus with that.  So, I would be 
"compromisingly happy" with the spec weighing in with a distinct 
preference for canonicalized URIs, which is what "MAY" seems to 
suggest.  MAY, also, seems to allude to the fact that implementors (and, 
by "implementors," I am referring to both producers and consumers of 
atom feeds) who do not auto-canonicalize MUST interoperate with those 
that do:

>   include the option

    But, if a consensus can be built around SHOULD or MUST, I am fine 
with that.

---

From: Martin Duerst <[email-masked]>
Date: Sun, 05 Sep 2004 11:04:30 +0900

>Ah, great. How many tag URI opponents are there out there?

I'm not a tag opponent. But I'm also not an HTTP opponent.
I want people to have a choice to use whatever URI scheme
they think fits their needs (and the needs of the Atom spec)
best. URIs have been specifically designed the way they are
so that specs do NOT have to say: "Here we only allow this
kind of identifier, and over there only that kind of identifier."
In a short-term view, restricting things may look appealing.
But the long term benefits of not doing so are huge.

Regards,    Martin

---

From: Martin Duerst <[email-masked]>
Date: Sun, 05 Sep 2004 10:56:55 +0900

>are not ambiguous, not onerous, and not open to profiling by

All other URI schemes? Or just some?

Regards,    Martin.

---

From: Dare Obasanjo <[email-masked]>
Date: Sat, 4 Sep 2004 19:36:49 -0700 (PDT)

> kind of identifier, and over there only that kind of

That sounds incorrect. So every Web application
supports every URI scheme? 

=====

---

From: Martin Duerst <[email-masked]>
Date: Sun, 05 Sep 2004 11:07:34 +0900

Hello Danny,

You say below "the format doesn't behave". But the format isn't
just syntax, it's also semantics. And sometimes it helps enormously
to describe these semantics as a behavior of the sender or recipient,
because such a description increases the chance of implementers
creating implementations that respect the semantics of the format
enormously.

Regards,     Martin.

---

From: Danny Ayers <[email-masked]>
Date: Sun, 5 Sep 2004 11:45:23 +0200

> supports every URI scheme?

That's a good question. I think the answer is a qualified "yes". In
the abstract of rfc2396bis it says:

[[
The URI syntax defines a grammar that is a superset of all valid URIs,
such that an implementation can parse the common components of a URI
reference without knowing the scheme-specific requirements of every
possible identifier.
]]

then right up the ladder in 6.2.3 :

[[
Software may use scheme-specific rules, at further processing cost, to
reduce the probability of false negatives.
]]

The spec could insist on a single scheme, but this may be
unnecessarily restrictive. An Atom application won't be interested in
what's retrievable from a http: URI, but still may use it as an id. So
generally speaking I think Martin's probably right - everyone should
have *some* level of support for all URIs (maybe just string
comparison) but then if their application uses something more specific
elsewhere (e.g. http: or nntp: ) then that's their prerogative. Atom
may reasonably say the id is for id purposes alone (within Atom),
there doesn't have to be a conflict.

Cheers,
Danny.

-- 

http://dannyayers.com

---

From: Martin Duerst <[email-masked]>
Date: Mon, 06 Sep 2004 06:58:59 +0900

>supports every URI scheme?

There is a difference between "allow" (what I said)
and "support" (what you said).

For IDs, that's of course the same (if we agree, and
I think the two of us as well as many others do, that
IDs get compared character-by-character). So for IDs,
Atom easily would indeed support every URI scheme.

For other uses of URIs, it's not the same. There could
be Web browsers that don't implement ftp:, and they
wouldn't violate any spec. There could even be Web
browsers that don't implement http:, and they still
would not violate any spec (think about a browser
in a standalone 'information kiosk' situation).

Regards,   Martin.

---

From: Walter Underwood <[email-masked]>
Date: Tue, 07 Sep 2004 08:09:50 -0700

--On Saturday, September 4, 2004 10:26 AM -0700 Tim Bray <[email-masked]> wrote:
> Irrespective of whether we're talking about atom:id or something else, 2396bis specifically recommends the use of canonicalized URIs for all the reasons that have bubbled up here, and I would see no problem if the WG wanted our drafts to echo that recommendation or at least point to it, for *all* URI usages. -Tim
>

+1 on recommending canonicalization for all URIs in Atom.

Lowercase "should" is fragile, since people might think it is a
mistake and treat it as SHOULD. Instead, use "is strongly recommended"
or something similar.

wunder
--
Walter Underwood
Principal Architect, Verity