Subject: Special characters in URIs

From: "Martin J. Duerst" <[email-masked]>
Date: Wed, 26 May 1999 14:38:33 +0900

This is a question for background information from the URI/URL
community.

The fact that URIs (RFC 2396) don't define the character semantics
of the byte values they encode has been discussed on various occasions.

To alleviate the problem, various URL schemes have started to base
themselves on UTF-8, and some formats that carry URIs have defined
error behaviour based on UTF-8.

The second case basically works by saying that if in these formats
(e.g. HTML), an URI contains a non-ASCII character, this character
is converted to a byte sequence using UTF-8 and then %-encoded to
produce a legal URI.

The question now has come up whether this behaviour can be extended
to characters in the ASCII range, i.e. any of:

 control     = <US-ASCII coded characters 00-1F and 7F hexadecimal>
 space       = <US-ASCII coded character 20 hexadecimal>
 delims      = "<" | ">" | "#" | "%" | <">
 unwise      = "{" | "}" | "|" | "\" | "^" | "[" | "]" | "`"

'#' and '%' of course have to stay excluded. For the formats in
question (mostly XML), control characters are not allowed anyway.
"<", <">, ... would appear as &lt;, &amp;,... only anyway.
Space would have to be used with caution because collapsing
rules might apply.

So the question is mainly about the rest:
"{", "}", "|", "\", "^", "[", "]", "`"

The reasons given in RFC 2396 for excluding them don't apply
in the relevant context, and before leaving that context,
these would be escaped anyway.

This is as far as my argumentation goes. I would very much like
to know if there is any problem with this.

Regards,   Martin. 

#-#-#  Martin J. Du"rst, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org

---

From: "Larry Masinter" <[email-masked]>
Date: Thu, 27 May 1999 14:28:06 PDT

URL character escaping normally should only be done at the
time the URL is constructed from its component pieces, and
normally should only be undone (unescaped) when the URL
is decomposed into its internal pieces.  Your description
of the process of either applying or removing %XX escaping
seems to be based on having the escapes applied or removed
when the URL is removed from or embedded in some context
such as XML. In general, you cannot change an arbitrary
%XX into the character the XX byte sequence represents in
ASCII without some risk of changing the meaning of the URL,
and so you should not recommend this process at all.

Larry
-- 
http://www.parc.xerox.com/masinter

> is converted to a byte sequence using UTF-8 and then %-encoded to

I think "works" is ambitious. It "works" because most
HTTP servers are forgiving about this kind of transliteration
and most URLs are HTTP.

---

From: Dan Connolly <[email-masked]>
Date: Thu, 27 May 1999 17:41:59 -0500

Larry Masinter wrote:
> is decomposed into its internal pieces.

True.

>  Your description
> such as XML.

only when it's removed

> In general, you cannot change an arbitrary

true.

> and so you should not recommend this process at all.

The excerpt below doesn't mention unescaping. Only how
to take an XML attribute value and turn it into a URL
in the case that it's not already a URL (because it
has non-URL characters).

It's probably worth warning folks that the inverse operation
is not licensed, but that doesn't mean the operation
itself is a problem.

> Larry

It "works" in the case that, for example, a user copies
a filename from a desktop filebrowser into an XML document
	href="xyz__"
where __ is some non-URL character.

Meanwhile, the HTTP server, when it exports the xyz__ file,
uses the same convention: UTF-8 encoding, %XX escaped.

That doesn't mean the HTTP server should grab xyz%XX%XX off
the tcp socket and unescape it; it means the HTTP server
should (do something equivalent to) enumerate each file
in the directory and escape it, and compare the resultin URI path
to xyz%XX%XX.

It's a bit of a kludge; the cleaner thing to do would
be to say "don't put things other than URIs in those
XML attribute values." But we haven't had any luck doing that.
And this "kludge" just so happens to be consistent with
the existing specs (though subtly) and consistent with
a fair amount of acutal practice (or at least so I
gather from Martin; I haven't seen the evidence 1st hand).

And it provides a global convention for interoperability
between HTTP servers exporting filesystems that use
iso-latin-1 to encode filenames and those that
export filesystems that use shift-jis or UCS-2.

-- 
Dan Connolly, W3C
http://www.w3.org/People/Connolly/

---

From: "Larry Masinter" <[email-masked]>
Date: Fri, 28 May 1999 15:17:16 PDT

(I'm hoping that [email-masked] will migrate to [email-masked],
although I've not gotten an acknowledgement. I suppose people
should look for news at http://www.ics.uci.edu/pub/ietf/uri )

> where __ is some non-URL character.

This works for me if you say that what's in the XML document
attribute isn't really a "URI" but rather something else.
For example, we could use the "IURI" draft to define what
appears in XML, and note that in order to turn it into a URI,
it needs to be escaped. I don't have a problem with that.

> in the directory and escape it, and compare the resultin URI path

Right.

> gather from Martin; I haven't seen the evidence 1st hand).

This works for me too, I'd just like to get this into the
specs.
 
> export filesystems that use shift-jis or UCS-2.

I'm not sure how that works (the shift-jis part), and I wonder
if this deserves a fuller explanation.

The URL internationalizationd raft has been sitting around
for a long time; maybe it's time to move it forward now?