Subject: Helping out with canonicalization of URIs

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 8 Aug 2004 20:56:10 -0700

Greetings again. In the discussion of PaceCanonicalIds, some 
questions were brought up about what draft-fielding-uri-rfc2396bis 
really says about canonicalization. Section 6 of that draft says a 
few different things. At the URI BOF at the IETF meeting last week, I 
volunteered the Atompub WG to be reviewers for that document. :-)

So, all you canonicalization folks: please review the document, 
particularly section 6, and send comments to [email-masked] (archived at 
<http://lists.w3.org/Archives/Public/uri/>). Just like on this list, 
if you see something you consider wrong, suggest new text. Your 
comments will be considered for the soon-to-happen IETF last call on 
the document.

It is likely that we will have to change the wording in 
PaceCanonicalIds to not say "the rules defined in rfc 2396bis" unless 
the rules we list exactly match the rules that eventually go into 
2369bis.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Sam Ruby <[email-masked]>
Date: Mon, 09 Aug 2004 08:54:33 -0400

Paul Hoffman / IMC wrote:
> 
> Greetings again. In the discussion of PaceCanonicalIds, some questions 
> were brought up about what draft-fielding-uri-rfc2396bis really says 
> about canonicalization. Section 6 of that draft says a few different 
> things. At the URI BOF at the IETF meeting last week, I volunteered the 
> Atompub WG to be reviewers for that document. :-)
> 
> <http://lists.w3.org/Archives/Public/uri/>). Just like on this list, if 
> you see something you consider wrong, suggest new text. Your comments 
> will be considered for the soon-to-happen IETF last call on the document.

Excerpts from sections 3 "Syntax Components":

       foo://example.com:8042/over/there?name=ferret#nose
       \_/   \______________/\_________/ \_________/ \__/
        |           |            |            |        |
     scheme     authority       path        query   fragment

    authority   = [ userinfo "@" ] host [ ":" port ]

    userinfo    = *( unreserved / pct-encoded / sub-delims / ":" )

Excerpt from section 6.3 "Canonical Form":

   # Always provide the URI scheme in lowercase characters.
   # Always provide the host, if any, in lowercase characters.
   # Only perform percent-encoding where it is essential.
   # Always use uppercase A-through-F characters when percent-encoding.
   # Prevent dot-segments appearing in non-relative URI paths.
   # For schemes that define a default authority, use an empty authority
     if the default is desired.
   # For schemes that define an empty path to be equivalent to a path of
    "/", use "/".

These rules completely cover scheme, path, and partially cover 
authority.  Here are some URIs that I can't determine if they are in 
canonical form based solely on the rules listed in rfc2396-bis:

   http://:@example.com/
   http://example.com:80/
   http://example.com/gateway.cgi?
   http://www.w3.org/2000/01/rdf-schema#

My initial inclination would be to declare all of these as 
non-canonical, but there is enough common practice of the last example 
that it probably should be an exception.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Mon, 09 Aug 2004 08:59:55 -0400

Paul Hoffman / IMC wrote:
> 
> It is likely that we will have to change the wording in PaceCanonicalIds 
> to not say "the rules defined in rfc 2396bis" unless the rules we list 
> exactly match the rules that eventually go into 2369bis.

The proposal contained in PaceCanonicalIds does not mention 2369bis. 
The only mention of 2369bis in the Pace is a mention in the abstract of 
the Pace itself.

- Sam Ruby

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Mon, 9 Aug 2004 07:16:55 -0700

At 8:59 AM -0400 8/9/04, Sam Ruby wrote:
>>PaceCanonicalIds to not say "the rules defined in rfc 2396bis" 
>>unless the rules we list exactly match the rules that eventually go 
>>into 2369bis.
>
>The only mention of 2369bis in the Pace is a mention in the abstract 
>of the Pace itself.

Exactly right. If we decide to list the rules for canonicalizing, and 
those rules don't match whatever comes in 2396bis, we're going to 
have to note that in our spec. Some developers may be using 
libraries, and those libraries are likely to be derived from 2396bis.

Given the discussion at the BOF on Friday, I don't think that 2396bis 
will meet our needs, and that we should do our own list. That, plus a 
note that our list doesn't match 2396bis, should suffice.

--Paul Hoffman, Director

---

From: Graham Klyne <[email-masked]>
Date: Mon, 09 Aug 2004 16:39:14 +0100

At 08:54 09/08/04 -0400, Sam Ruby wrote:
>These rules completely cover scheme, path, and partially cover authority.

>Here are some URIs that I can't determine if they are in canonical form 
>based solely on the rules listed in rfc2396-bis:

I'll offer here my opinions based on my understanding gained from 
implementing a parser directly from a slightly earlier version of 
RFC2396bis.  (That is, having read and worked with the specification, but 
without recalling specific assertions in each case.)

>   http://:@example.com/

I'd say that's different from http://example.com/, in that it contains 
empty username/password values, which the latter does not.  For example, 
following the exhortation not to expose passwords, my software would (by 
default) display this as:
   http://:[email-masked]/
whereas the other would be displayed unchanged.

(I'm not claiming this is a *useful* distinction, but lacking any text that 
says a null username/password is the same as having no username/password, 
I'd say that it does exist.)

>   http://example.com:80/

I think this is the same as http://example.com/ according to RFC2396bis, 
but that you have to climb someway up the equivalence ladder 
(protocol-specific equivalence) to recognize this.  It must be expected 
that many software packages would not recognize this equivalence.

(I'm not a great fan of "the ladder" approach, but I don't have anything 
better to offer...)

>   http://example.com/gateway.cgi?

I'd say this is distinct from http://example.com/gateway.cgi? -- an empty 
query is not the same as no query at all.

>   http://www.w3.org/2000/01/rdf-schema#

I'd say this is distinct from http://www.w3.org/2000/01/rdf-schema -- an 
empty fragment is not the same as no fragment at all (note, when used as 
namespace URI in an RDF document, they certainly would not give rise to the 
same resource identifiers according to the RDF specifications -- see RDF 
syntax spec (10 Feb 2004), section 6.1.2, URI accessor)

>My initial inclination would be to declare all of these as non-canonical, 
>but there is enough common practice of the last example that it probably 
>should be an exception.

As you see, I come to different conclusions in most cases.

#g

------------
Graham Klyne
For email:
http://www.ninebynine.org/#Contact

---

From: Walter Underwood <[email-masked]>
Date: Mon, 09 Aug 2004 12:02:03 -0700

> note that our list doesn't match 2396bis, should suffice.

More custom Atom-only code. Sigh.

How do they not meet Atom's needs?

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Sam Ruby <[email-masked]>
Date: Mon, 09 Aug 2004 15:48:23 -0400

> More custom Atom-only code. Sigh.

There certainly will be custom Atom-only code in the feedvalidator, but 
I would be surprised if you were planning on generating ids with 
uppercase schemes, mixed case hosts, etc.

> How do they not meet Atom's needs?

At the moment rfc 2396bis is not as clear as it could be on whether or 
not empty fragments, queries, passwords (or possibly userinfo) are to be 
preserved by canonicalization.  I'm seeking to get this resolved as a 
part of last-call, but failing that, I'm soliciting inputs by the spec 
authors on how such cases should be handled.

PaceCanonicalIds will be made consistent with these inputs once they are 
available.

The one case where PaceCanonicalIds goes beyond rfc 2396bis is in 
requiring utf-8 encoding of non-ASCII characters.

- Sam Ruby

---

From: Walter Underwood <[email-masked]>
Date: Mon, 09 Aug 2004 13:30:34 -0700

--On Monday, August 09, 2004 03:48:23 PM -0400 Sam Ruby 
<[email-masked]> wrote:
>
> There certainly will be custom Atom-only code in the feedvalidator, but I
> would be surprised if you were planning on generating ids with uppercase
> schemes, mixed case hosts, etc.

I'm thinking about canonical URI throughout Atom, not just IDs.

Given the ignorecase filenames on Windows, we see a lot of variant
case in the path portion of HTTP URLs. The situation is bad enough
that spiders get a decent speedup by ignoring case in all URL paths,
even though that is clearly wrong behavior. When every element of
a pathname can be UPPER, lower, or Capitalized, a spider can see a
large number of URLs for the same resource.

Using canonical URIs throughout Atom makes it work better with
the rest of the web: caches, bookmarks, search engines, etc.

> requiring utf-8 encoding of non-ASCII characters.

This is consistent with the (non-normative) recommendation in
HTML 4.01: http://www.w3.org/TR/html401/appendix/notes.html#h-B.2.1

Since that is widely implemented, 2396bis really should address it.

wunder
--
Walter Underwood
Principal Architect
Verity Ultraseek

---

From: John Cowan <[email-masked]>
Date: Mon, 9 Aug 2004 17:51:33 -0400

Clive D.W. Feather scripsit:

> HTTP doesn't distinguish them that I can see. Can you explain the RDF bit
> in words of one syllable?

RDF maps QNames like foo:bar to URI references by appending "bar" to the
namespace name currently bound to the prefix foo.  It's useful for
such namespace names to end in # or /, usually #, so that the local name
is clearly separated from the namespace name.

-- 
"But the next day there came no dawn,           John Cowan
and the Grey Company passed on into the         [email-masked]
darkness of the Storm of Mordor and were        http://www.ccil.org/~cowan
lost to mortal sight; but the Dead              http://reutershealth.com
followed them.          --"The Passing of the Grey Company"

---

From: Martin Duerst <[email-masked]>
Date: Tue, 10 Aug 2004 17:15:16 +0900

>the rest of the web: caches, bookmarks, search engines, etc.

Are you suggesting that case should be ignored and everything
should be canonicalized to lower case in the path part of the
URI?

>Since that is widely implemented, 2396bis really should address it.

No, this is not consistent with that specific recommendation.
Using IRIs rather than just URIs is consistent with that recommendation.
Always requiring %-encoded octets to be in UTF-8 has only partially
to do with that recommendation, because that recommendation applies
to non-ASCII characters in an attribute, whereas the pace requires
%-encoding.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 10 Aug 2004 17:07:37 +0900

>requiring utf-8 encoding of non-ASCII characters.

Why is this required? How is this part of canonicalization?
What about schemes that e.g. would explicitly require another encoding?

[Note that I'm of course very much for using UTF-8, because that
makes things compatible with IRIs, and is a good idea for many other
reasons, but forcing this where it is not appropriate does no good.]

Regards,   Martin.

---

From: Walter Underwood <[email-masked]>
Date: Tue, 10 Aug 2004 08:42:58 -0700

> should be canonicalized to lower case in the path part of the

Huh? How can you read what I said to mean that? I was talking only
about spiders, not Atom, and I said it is "clearly wrong behavior".
I am describing what spiders currently do, because they must deal
with non-canonical URIs. I was *not* proposing that Atom should
lowercase URIs. Yuk.

My point is that canonical URIs have big advantages where Atom
touches the rest of the web, independent of the issues within Atom.

Requiring canonicalization of URIs throughout Atom would make
Atom a trusted source of URIs for spiders, and would mean that
they would not need to do drastic, risky things like lowercasing
paths.

wunder

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 10 Aug 2004 20:50:50 +0200

>Since that is widely implemented, 2396bis really should address it.

It is most certainly not widely implemented, Mozilla Firefox 0.9 does
not do anything to this effect at all, MSIE/Win does this only for the
path component, etc.