Subject: URI canonicalization

From: Graham <[email-masked]>
Date: Sun, 30 Jan 2005 17:17:54 +0000

This controversial text is still in:

    Because of the risk of confusion between URIs that would be
    equivalent if dereferenced, the following normalization strategy is
    strongly encouraged when generating Identity constructs:

    o  Provide the scheme in lowercase characters.
    o  Provide the host, if any, in lowercase characters.
    o  Only perform percent-encoding where it is essential.
    o  Use uppercase A-through-F characters when percent-encoding.
    o  Prevent dot-segments appearing in paths.
    o  For schemes that define a default authority, use an empty
       authority if the default is desired.
    o  For schemes that define an empty path to be equivalent to a path
       of "/", use "/".
    o  For schemes that define a port, use an empty port if the default
       is desired.
    o  Preserve empty fragment identifiers and queries.
    o  Ensure that all portions of the URI are UTF-8 encoded NFC form
       Unicode strings.

For starters its in the "Dereferecing" section for some reason. 
Secondly, no consensus was reached to include it. Tim shrugged when no 
proposal gained consensus, and included it anyway. Thirdly, the 
rationale in the spec doesn't match any of the even-vaguely-valid ones 
given on the list. Fourthly, none of those rationales were valid. 
Fifthly, it's micromanaging. Of all the things we could go into great 
detail telling people how to do, this doesn't even rate. I've never 
seen a feed that has any of the problems this might solve.

Please delete it.

Graham

---

From: Graham <[email-masked]>
Date: Sun, 30 Jan 2005 17:50:59 +0000

I suppose I should offer an alternative solution. Two scenarios were 
given to justify canonicalization:

1) A publisher accidentally uses a different, though very similar, URI 
for their id. They then apply the canonicalization rules and the error 
is erased. This will only work if they remember to apply the rules, if 
they accidentally chose a similar enough id for the rules to apply, and 
more bizarrely if they change the code that generates ids regularly, or 
fill out the id field themselves by hand from memory each time it is 
requested (the mark of a true blogger). This is not a problem that 
needs solving or that c14n solves.

2) An intermediary automatically c14nizes all URIs it processes. If 
URIs come pre-c14nized from the publisher, this won't do any damage. 
This is valid, but the problem is that these intermediaries are 
currently imaginary. A much better way to solve it would be to write, 
at the end of the comparison section:

	For this reason, intermediaries should be careful not to canonicalize 
or otherwise modify Identity constructs they
      process.

Problem solved, much more efficiently, much more reliably (since the 
current method breaks if one blogger don't remember to c14 their feed), 
and much less wordily.

Graham

---

From: Tim Bray <[email-masked]>
Date: Sun, 30 Jan 2005 10:20:41 -0800

On Jan 30, 2005, at 9:50 AM, Graham wrote:

> currently imaginary.

I may be moving toward agreement with Graham, but the statement above 
is incorrect.  Sam (or was it Mark) demonstrated clearly that if you 
use the URI.equals() (or equivalent) method in many modern programming 
languages, you will get all sorts of canonicalization done without 
asking, and it is very likely that lots of programmers will do this. 
-Tim

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 13:36:30 -0500

> asking, and it is very likely that lots of programmers will do this. 

How about this:

   "The only comparison method Atom Processors MUST support is 
character-by-character comparison [RFC3986].
   Atom Processors MAY perform additional scheme-specific comparisions."

If you do this:

   http://Example.org/thing
   http://example.org/thing

You cannot count on a positive or negative, and you are sloppy.

Robert Sayre

---

From: Anne van Kesteren <[email-masked]>
Date: Sun, 30 Jan 2005 19:45:28 +0100

> You cannot count on a positive or negative, and you are sloppy.

That would mean that the implementation could differ between Atom 
processors, right? And that in your example one would return true and 
the other false when the URIs are compared?

Tim: <http://www.intertwingly.net/blog/2004/07/31/URI-Equivalence>

-- 
  Anne van Kesteren
  <http://annevankesteren.nl/>

---

From: Julian Reschke <[email-masked]>
Date: Sun, 30 Jan 2005 19:54:50 +0100

>   Atom Processors MAY perform additional scheme-specific comparisions."

If an Atom processor MUST support char-by-char, what would be the point 
of supporting anything else? One comparison method should be enough, 
shouldn't it?

> You cannot count on a positive or negative, and you are sloppy.

Why can't I count on the result, if the spec says I can? (not that I 
would recommend to do that in the real world)

Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 14:04:36 -0500

Julian Reschke wrote:

> If an Atom processor MUST support char-by-char, what would be the 
> point of supporting anything else? One comparison method should be 
> enough, shouldn't it?

> would recommend to do that in the real world)

Um, the spec doesn't say you can. If the comparision is done with 
URI.equals(), it will be positive. If it is done with String.equals(), 
it will be negative. That text is a refelection of reality.

Robert Sayre

---

From: Sam Ruby <[email-masked]>
Date: Sun, 30 Jan 2005 14:06:49 -0500

Paraphrasing Tim [1] "I'm definitely -1 on losing 3.5.1, the 
canonicalization warning is a hard-won compromise and seems to cause 
no-one any pain."

We discussed this at extreme length, and no new arguments have been 
brought forward.  Rough consensus does not mean absolute consensus.

- Sam Ruby

[1] http://www.imc.org/atom-syntax/mail-archive/msg12672.html

---

From: Julian Reschke <[email-masked]>
Date: Sun, 30 Jan 2005 20:14:37 +0100

> it will be negative. That text is a refelection of reality.

<http://atompub.org/2005/01/27/draft-ietf-atompub-format-05.html#rfc.section.3.5.2>:

"Instances of Identity constructs can be compared to determine whether 
an entry or feed is the same as one seen before. Processors MUST compare 
Identity constructs on a character-by-character basis in a 
case-sensitive fashion."

Best regards, Julian

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 14:17:01 -0500

Julian Reschke wrote:

>> URI.equals(), it will be positive. If it is done with 
>> String.equals(), it will be negative. That text is a refelection of 
>> reality.
>
>
> an entry or feed is the same as one seen before. Processors MUST 
> compare Identity constructs on a character-by-character basis in a 
> case-sensitive fashion."

Right. I am proposing text to replace that, because my text reflects the 
reality that implementors will face.

Robert Sayre

---

From: Graham <[email-masked]>
Date: Sun, 30 Jan 2005 19:30:31 +0000

> brought forward.  Rough consensus does not mean absolute consensus.

This is how PaceIdConstruct was accepted:

"Since then, we've had another flurry of messages, but I still think 
letting MNot take a crack at working PaceIdConstruct into a format-02 
draft, then suggesting improvements to that, is the most constructive 
way forward. -Tim"

http://www.imc.org/atom-syntax/mail-archive/msg08984.html

No rough consensus even claimed.

Graham

---

From: Eric Scheid <[email-masked]>
Date: Mon, 31 Jan 2005 09:54:30 +1100

On 31/1/05 6:17 AM, "Robert Sayre" <[email-masked]> wrote:

>   character-by-character comparison [RFC3986]. Atom Processors
>   MAY perform additional scheme-specific comparisions."

-1. The case-sensitive char-by-char comparison produces correct results
every time, while other forms of comparisons can vary from platform to
platform. Just because an implementer has the option of comparing the URI to
a banana or some other piece of fruit doesn't mean it's sensible and must
therefore be allowed for in the spec.

e.

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 18:16:39 -0500

>therefore be allowed for in the spec.

I agree with you in principle, but I find the current text unrealistic. 
It's just fodder for stupid arguments.

Robert Sayre

---

From: Eric Scheid <[email-masked]>
Date: Mon, 31 Jan 2005 10:43:14 +1100

> It's just fodder for stupid arguments.

what? and "Atom Processors MAY perform additional scheme-specific
comparisions" won't lead to stupid arguments? Here's one for a starter: if
the case-sensitive char-by-char comparison produces one result, and the
scheme specific comparison produces a different result, which one takes
precedence? You've left that wide open.

Another: "The spec allows for additional comparisons. Therefore the results
of those additional comparisons must be significant, otherwise why mention
them in the first place. Therefore scheme specific comparisons have
precedence over case-sensistive-char-by-char comparisons."

The spec should (nay, MUST) mandate a method of comparing Identity
Constructs which will always produce the *same* result, and *not* allow
comparisons which can produce differing results. Draconian perhaps, but the
essential nature of <id> requires it.

e.

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 18:53:35 -0500

>essential nature of <id> requires it.

Nonsense. URI scheme specs define their own comparison methods. Know 
what they are before you use them. That's all there is to it.

Robert Sayre

---

From: Graham <[email-masked]>
Date: Mon, 31 Jan 2005 00:06:18 +0000

On 30 Jan 2005, at 11:43 pm, Eric Scheid wrote:

> comparisions" won't lead to stupid arguments?

Yeah, that's a horrible loose end to leave hanging.

> essential nature of <id> requires it.

Yes. It would be prudent of developers to choose significantly 
different ids for different entries, to avoid false positives, and it 
may even be worth mentioning that in the spec.

Graham

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 19:16:20 -0500

> Yeah, that's a horrible loose end to leave hanging.

No, it isn't. URI comparison is not our problem, and what our spec says 
about it doesn't matter a bit.

Robert Sayre

---

From: Graham <[email-masked]>
Date: Mon, 31 Jan 2005 00:26:58 +0000

On 31 Jan 2005, at 12:16 am, Robert Sayre wrote:

> No, it isn't. URI comparison is not our problem, and what our spec 
> says about it doesn't matter a bit.

Yes it is times one million. Ha ha I win et cetera

Defining rules for ID comparison is our problem if we're even going to 
pretend there'll be interop amongst Atom software. The spec currently 
says "Use these rules: [states rules]. Or use some other rules. We 
don't really care."

Graham

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 19:43:52 -0500

> says "Use these rules: [states rules]. Or use some other rules. We 

How about "Make sure your id is unique from a character-by-character 
perspective, but also unique in the face of scheme-specific 
comparisons". That is, don't lean on scheme-specific comparisons to 
match URIs, but they don't have to be canonical either.

a) http://Example.com/thing
b) http://example.com/thing

a) is ok, but don't expect clients to match it to b).

Make sense?

Robert Sayre

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Mon, 31 Jan 2005 02:16:24 +0100

> brought forward.  Rough consensus does not mean absolute consensus.

Thank you. We've discussed this too much already. Please let's leave this  
horse; it's already beaten to death and won't get any more dead no matter  
how much we keep on beating.

-- 
Asbjørn Ulsberg     -=|=-    http://virtuelvis.com/quark/
«He's a loathsome offensive brute, yet I can't look away»

---

From: Graham <[email-masked]>
Date: Mon, 31 Jan 2005 01:21:46 +0000

> match URIs, but they don't have to be canonical either.

Yes, changing the wording to that is fine, as long as it also says 
processors that do scheme-specific comparison are bad.

> a) is ok, but don't expect clients to match it to b).

"Expect a subset of clients to match b to a", surely?

Graham

---

From: Robert Sayre <[email-masked]>
Date: Sun, 30 Jan 2005 20:31:35 -0500

> processors that do scheme-specific comparison are bad.

Suppose your user is subscribed to a feed containing 1000 entries. One 
day, the host name is no longer capitalized. Are you going to put 1000 
new, duplicate entries in front of the user?

> "Expect a subset of clients to match b to a", surely?

Sure.

Robert Sayre

---

From: Graham <[email-masked]>
Date: Mon, 31 Jan 2005 01:50:16 +0000

> new, duplicate entries in front of the user?

Yes. They have different ids as defined by the spec. If you think 
character-by-character is too fragile, write a pace proposing an 
alternative.

Graham

---

From: Joe Gregorio <[email-masked]>
Date: Sun, 30 Jan 2005 22:53:06 -0500

> brought forward.  Rough consensus does not mean absolute consensus.

+1

    -joe

-- 
Joe Gregorio        http://bitworking.org

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 31 Jan 2005 13:24:46 +0100

>new, duplicate entries in front of the user?

It seems the Working Group is split on the requirements for duplicate
detection, we have participants who want to require a deterministic
algorithm to gurantee interoperable detection, participants who want
implementation-defined algorithms to implement best-effort strategies,
and participants who want to encourage interoperable detection while
allowing implementation-defined algorithms for various reasons. Maybe
we should first find some form of consensus on the requirements before
we proceed with writing new or removing old specification text.

I do not think it will contribute to Atom's success if some Atom im-
plementations are better than other implementations when it comes to
duplicate detection.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Antone Roundy <[email-masked]>
Date: Mon, 31 Jan 2005 10:42:53 -0700

> match URIs, but they don't have to be canonical either.

We could augment this:

3.5.2  Comparing Identity Constructs

    Instances of Identity constructs can be compared to determine whether
    an entry or feed is the same as one seen before.  Processors MUST
    compare Identity constructs on a character-by-character basis in a
    case-sensitive fashion.

...with something like this (...but not exactly, of course):

Because some dopes might not read the preceding sentence, and might use 
URI comparison routines instead of string comparison routines, it is 
prudent to ensure that your Identity Constructs are both 
character-by-character unique as well as unique according to the rules 
of the chosen URI scheme.  For example, because some consumers might 
incorrectly consider the following to be the same, it would be prudent 
not to use more than one of them as an identifier:

http://Example.com/1024
http://example.com/1024
http://example.com/102%34

---

From: DJWS <[email-masked]>
Date: Mon, 31 Jan 2005 14:55:26 +0100

I am not sure this is relevant but all this is supporting IRI?
jfc

---

From: Sam Ruby <[email-masked]>
Date: Mon, 31 Jan 2005 13:20:23 -0500

+1

It will be actively harmful to Atom's success if the standard is defined 
in such a way that is no realistic expectation that it will be followed.

- Sam Ruby

---

From: Martin Duerst <[email-masked]>
Date: Tue, 01 Feb 2005 12:10:24 +0900

I have just looked at the text in question in -05.txt,
and read through the discussion. I'll give my comments
here, but they are not specifically on this mail.

First, for me, the goal of having reproducible id comparison
is most important; this is the basic requirement.

Second, given that there are an amazing number of things
that can be compared differently once you start (the
URI/IRI specs, the scheme-specific specs, and Unicode
give you a lot of rope), character-
by-character comparison is the only way to go.

Third, I don't think that the normalization advice in
"3.5.1  Dereferencing Identity Constructs" is extremely
important, but I don't mind if it's there.

I have the following actual editing proposals to hopefully
make this part of the spec a bit clearer:

1) switch sections 3.5.1 and 3.5.2 to make clear that
    comparison of IDs is the most important operation.

2) At the start of "Comparing Identity Constructs", change the
    sentence "Instances of Identity constructs can be compared to
    determine whether an entry or feed is the same as one seen before."
    to "To determine whether an entry or feed is the same as one
    seen before, their Identity Constructs are compared."
    This makes it clear that we are talking about "here is how
    you do it", rather than "here's one way to do it".

3) Change "Processors MUST compare Identity constructs on a
    character-by-character basis in a case-sensitive fashion."
    to "Processors MUST compare Identity constructs on a
    character-by-character basis. For details, see section
    5.3.1.,  Simple String Comparison, of [RFC3987].
    We may want to add something about case-sensitivity as
    a note, but it should not be in the main text. There are
    way to many other ways that this could go wrong, in particular
    in an Unicode context.

4) Add a sentence saying something like "Feeds or Entries
    are identical if their IDs compare identical.".
    Seems obvious, but isn't stated anywhere.

5) Add a note saying something like "Comparison functions
    provided by many URI classes/implementations make additional
    assumptions about equality that are not true for Identity
    Constructs. Atom processors therefore should use simple
    string functions for comparing Identity Constructs."
    I think such a note could be a good balance to the normalization
    advice.

I understand that in general, we have tried to reduce
implementation advice in the spec as much as possible.
But in my experience, adding such advice or notes is
often a good way to reach better consensus.

Regards,    Martin.

At 02:17 05/01/31, Graham wrote:
 >For starters its in the "Dereferecing" section for some reason. Secondly, 
no consensus was reached to include it. Tim shrugged when no proposal 
gained consensus, and included it anyway. Thirdly, the rationale in the 
spec doesn't match any of the even-vaguely-valid ones given on the list. 
Fourthly, none of those rationales were valid. Fifthly, it's micromanaging. 
Of all the things we could go into great detail telling people how to do, 
this doesn't even rate. I've never seen a feed that has any of the problems 
this might solve.

---

From: Graham <[email-masked]>
Date: Tue, 1 Feb 2005 04:20:23 +0000

>    comparison of IDs is the most important operation.

Yes.

>    you do it", rather than "here's one way to do it".

We might be treading on toes making that assertion.

>    Seems obvious, but isn't stated anywhere.

No. Feeds/entries with the same id are different versions or instances 
of a common ancestor. They are not "the same".

>    I think such a note could be a good balance to the normalization

Yes, though I don't like that wording.

Graham

---

From: Tim Bray <[email-masked]>
Date: Mon, 31 Jan 2005 20:40:32 -0800

> We might be treading on toes making that assertion.

Yes, but it's not only correct, it's good advice, so we should put it 
in.

> of a common ancestor. They are not "the same".

Graham's right, the word "identical" is wrong, because in fact you will 
commonly encounter two instances of the same entry which aren't 
identical (e.g. the one in your cache and the one you just fetched).  I 
suggest "Software MUST treat any two entries which have the same ID as 
instances of the same entry, regardless of how they were obtained or 
any differences in their content."  Or if you want it less prescriptive 
"By definition, any two entries which have the same ID are instances of 
the same entry, regardless of how they were obtained or any differences 
in their content." -Tim