Subject: location uri, ucs and the http scheme definition.

From: Robert Collins <[email-masked]>
Date: Wed, 03 Aug 2005 21:51:33 +0000

Hi,
	trying to determine the correct method to encode unicode characters
into URI's in the Location: header....

std66 says that *new* URI Schemes should allow require UCS characters to
be encoded as utf8, and then those that fall outside the unreserved set
to be percent escaped.

The http:// scheme is as far as I can determine (maybe my google juice
is failing me) specified by rfc2616 which doesn't mention UCS at all -
it predate std66 by quite some time.

So: what is the correct encoding approach for non US-ASCII characters in
HTTP URI's in Location: ?

Rob

-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Aug 2005 12:19:48 +0200

> HTTP URI's in Location: ?

What does this have to do with "Location"? A URI never ever contains 
non-ASCII characters, no matter where it appears.

That being said; the *best* way (if you control the server) to embed non 
ASCII data into a URI usually is to UTF-8 encode, then percent-escape.

Best regards, Julian

---

From: Robert Collins <[email-masked]>
Date: Mon, 08 Aug 2005 03:21:29 +0000

> non-ASCII characters, no matter where it appears.

Uhm. URIs do contain non-ASCII characters quite commonly. Such as
http://example.com/~usérname

Note that that cannot be unambiguously percent escaped from the rules
contained in rfc2616 and std66. The missing link is the encoding to use
before percent escaping, which you suggest a heuristic ..

> ASCII data into a URI usually is to UTF-8 encode, then percent-escape.

What I'm asking for is current RFC or std document that specifies this.
The reference for Location is because Location is the header I currently
need to ensure correct encoding of, and its definition is 'absoluteURI'
in rfc2616, which is incorporated from ... and thus we trace through to
std66 which passes the buck for the canonical pre-percent-escape
encoding back to the standard defining the URI scheme, which is rfc2616.
Full circle and no stance taken.

Rob

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Aug 2005 09:20:24 +0200

> http://example.com/~usérname

No, by definition that is not a URI. Non-ASCII characters are not 
allowed in URIs, that's why you need to escape them.

> before percent escaping, which you suggest a heuristic ..

Yes. Many servers use UTF-8, but in general a client cannot assume this 
is the case.

> in rfc2616, which is incorporated from ... and thus we trace through to

The rules for the URI in the Location header are exactly the same as for 
the HTTP request itself, so I'm still not sure where there would be a 
difference.

> Full circle and no stance taken.

There's no single encoding that will work for any server. There may be 
RFCs that recommend UTF-8 (possibly RFC3987), but these do not 
normatively effect RFC2616-compliant servers.

Best regards, Julian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 08 Aug 2005 05:15:40 -0500

>>> That being said; the *best* way (if you control the server) to embed 
>>> non ASCII data into a URI usually is to UTF-8 encode, then 
>>> percent-escape.
>>

First, rfc2616 predates std66, which doesn't override the conclusions of
that rfc; section 3.2.3 spells it out, to a webserver the URI is an
opaque series of octets with a few specific exceptions.  The webserver
has no opinion.

That said; for example WinNT's filesystem is truly unicode, which Apache
2.0, for example, treats as a utf-8 filesystem for resource names.  The
typical *nix system today may in fact use utf-8 file names, but does
not enforce them (they remain opaque octets to the posix layer).  It's
entirely up to the implementor what to serve based on a URI.

Convention is developing around utf-8 URI names, because no header in
the http spec further defines how to interpret the URI (the request or
response body, yes; the URI, no.)  But again, in HTTP/1.1 do as you
like as long as you escape the non-ascii and reserved codes.  Std66
didn't add an explicit charset for us, and it's unlikely until yet
another rfc for HTTP.

Bill

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Aug 2005 12:24:22 +0200

William A. Rowe, Jr. wrote:
>> The rules for the URI in the Location header are exactly the same as 
>> for the HTTP request itself, so I'm still not sure where there would 
>> be a difference.
>>

RFC2616 normatively refers to RFC2396 for the definitions of URI 
components 
(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.2.1>). And 
RFC2396 did not allow non-ASCII characters in URIs, either.

> entirely up to the implementor what to serve based on a URI.

Yes. That's a problem. See 
<http://greenbytes.de/tech/webdav/draft-reschke-webdav-url-constraints-latest.html> 
for a work-in-progress attempt to fix things at least for WebDAV.

> ...

Best regards, Julian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 08 Aug 2005 05:56:01 -0500

At 05:24 AM 8/8/2005, Julian Reschke wrote:
>>>There's no single encoding that will work for any server. There may be RFCs that recommend UTF-8 (possibly RFC3987), but these do not normatively effect RFC2616-compliant servers.
>>
>RFC2616 normatively refers to RFC2396 for the definitions of URI components (<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.2.1>). And RFC2396 did not allow non-ASCII characters in URIs, either.

Julian stop it already; we -know- that only non-reserved, 
ASCII characters can be transmitted over the wire.  The
question Robert raises is what charset is normative prior
to it being encoded for transmission over the wire.  2396
says nothing that the %-escape decoded value must be be ASCII
for all components, quoting RFC 2396 section 2;

  "

Within a URI, characters are either used as delimiters, or to
   represent strings of data (octets) within the delimited portions.
   Octets are either represented directly by a character (using the US-
   ASCII character for that octet [ASCII]) or by an escape encoding."

ahhh... "or by an escape encoding" - which is all Robert has asked
for since the conversation started ;-)  Yet - we still don't have
a definitive charset, it's opaque octet data.

>Yes. That's a problem. See <http://greenbytes.de/tech/webdav/draft-reschke-webdav-url-constraints-latest.html> for a work-in-progress attempt to fix things at least for WebDAV.

Ack :)  The more comprehensive solution of course, HTTP/1.2, 
although I know some have their hearts set on HTTP-NG first.

Bill

---

From: Joe Gregorio <[email-masked]>
Date: Mon, 8 Aug 2005 09:14:27 -0400

> Full circle and no stance taken.

RFC 3986.

  -joe

-- 
Joe Gregorio        http://bitworking.org

---

From: Robert Collins <[email-masked]>
Date: Mon, 22 Aug 2005 01:47:23 +0000

On Mon, 2005-08-08 at 05:56 -0500, William A. Rowe, Jr. wrote:
..
...
> a definitive charset, it's opaque octet data.

Yes!

> although I know some have their hearts set on HTTP-NG first.

I'd be happy with a HTTP/1.1 errata that updates the http:// scheme to
declare it as utf8 before the escape encoding is done.

Cheers,
Rob

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 22 Aug 2005 14:20:08 +0100

> declare it as utf8 before the escape encoding is done.

Not reasonable.

There are a significant number of HTTP/1.1-compliant servers which
work with URLs that are derived from text in other encodings, and
there are servers where the encoding depends on the URL (because the
server's job is to pass along the URL unmodified to individual
resource handlers).

Because of that, proxies must continue to work with URIs that contain
arbitrary %-escaped sequences, without filtering or changing them
(even if they don't represent valid UTF-8), servers must continue to be
able to serve documents containing such URIs, and clients must
continue to be able to retrieve documents using those URIs.

In principle, the escape-encoding represents an application-specific
opaque octet stream, and it need not represent "characters" at all.

An appropriate place to define UTF-8 as the encoding to use would be
in document standards, such as XML and HTML, as this question really
is about how to convert character sequences (in documents and user
interfaces) that feature non-ASCII characters and purport to be URIs
(but aren't really URIs) into well-formed URIs for network operations.

The place where it's useful to specify a character encoding are:

    - How non-ASCII characters in documents in places such as an
      "href" attribute are converted into proper URIs for HTTP.

    - How non-ASCII characters in forms are converted into proper
      URI query parts.  (This is covered somewhat already in HTTP 4).

    - How non-ASCII characters in other parts of a typical client's
      user interface such as the "location bar", are converted into
      proper URLs for HTTP document retrieval.

-- Jamie

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 22 Aug 2005 10:03:24 -0500

> declare it as utf8 before the escape encoding is done.

You cannot change the declaration.  The best thing that can be done
within the HTTP/1.1 errata is to note that utf8 is one accepted and
common mapping, and suggest it as a preferred presentation format.

Your server needs to only accept URI's that it is willing to serve,
so if every resource it serves was %-encoded utf8, this is entirely
legitimate.  But your client applications must be willing to send any
arbitrary octet stream %-encoded, and treat href's that it will handle
as an unknown/arbitrary mapping.  That is, unless your client has the
limited scope of connecting to your server.

Bill

---

From: Robert Collins <[email-masked]>
Date: Tue, 23 Aug 2005 01:23:27 +0000

> common mapping, and suggest it as a preferred presentation format.

So - a note that that utf8 is an accepted and common mapping would be a
good improvement over the current unguided situation.

Ack on the roundtripping and opaqueness - that is the base levels uris
work from.

Cheers,
Rob

---

From: Robert Collins <[email-masked]>
Date: Tue, 23 Aug 2005 01:36:24 +0000

> server's job is to pass along the URL unmodified to individual

I put it to you that this has occured because of the lack of guidance in
rfc2616. Even though we can't retroactively change the standard, adding
in the std66 recommendation as a wg recommendation would be a positive
step IMO.

> continue to be able to retrieve documents using those URIs.

Anything compliant with any of the uri standards must continue to work
with any % escape uri representation. Sure - but it would be nice to
document what *should* work.

> opaque octet stream, and it need not represent "characters" at all.

For URIs in general, yes. but std66 section 2.5 does provide guidance
for this...
>       proper URLs for HTTP document retrieval.

Which, given we started this thread on the Location header in http,
which sets the user interface location bar ... seems relevant to me.

Anyway, what I'd like to see is some reference suggesting a best
practice for http uris, if that is able to be defined. Using whatever
guidelines are present for the next http protocol would be ideal ;0.

Rob

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 23 Aug 2005 16:53:57 +0100

> in the std66 recommendation as a wg recommendation would be a positive

Those web servers _far_ predate RFC2616.  Whatever guidance goes into
an HTTP URI standard, it must remain backward compatible with what's
widely deployed, which is precisely why the RFCs don't mandate it yet,
even as they suggest further work is needed on it.

> document what *should* work.

It is documented.  According to documentation, any %-escaped octet
sequence should work :)  Converting them to *characters for visual
presentation* is outside the scope of HTTP.

> which sets the user interface location bar ... seems relevant to me.

The Location header only has that effect in _web browsers_.

There are lots of other programs which use HTTP for which the
"characters" encoded in a URL are irrelevant.

Increasingly, we may find that non-web-browser HTTP agents see
non-ASCII characters in parts of a document that claim to be URIs, and
must follow them.  Or, they see URIs containing %-encoded characters
and need to convert those to presentable text in documents.

Broadly, the UTF-8-ness affects programs which relate documents
containing non-ASCII characters with URLs.  For example, a spider
which indexes pages that happen to contain non-ASCII characters in the
URLs in "href" attributes... those are actually not valid URLs, but
the spider has to make a heuristic decision if it's to follow them.

Unfortunately, if we mandate that non-ASCII characters found in "href"
URL attributes should be %-escaped as UTF-8 to follow them, we'll find
that this *breaks* some existing deployed sites.  Maybe this is for
the best...

In the other direction, a program which is generating index pages of
links may wish to present the links visually as text, converting
%-escape sequences into good looking text.  However, this may look
nice but it's prone to causing security problems...

> guidelines are present for the next http protocol would be ideal ;0.

It still has to be backward-compatible, if it's HTTP/1.2.

So guidance for future applications is sensible.  Making it a
requirement has to done much more carefully.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Tue, 23 Aug 2005 18:04:49 +0200

> even as they suggest further work is needed on it.

Right!

> The Location header only has that effect in _web browsers_.

Sorry? It's for instance used in 3xx (redirect) responses.

> and need to convert those to presentable text in documents.

Yes. This is a common issue in WebDAV. See 
<http://greenbytes.de/tech/webdav/draft-reschke-webdav-url-constraints-latest.html> 
(work in progress).

> that this *breaks* some existing deployed sites.  Maybe this is for

It's ugly, but probably still the best approach.

Best regards, Julian

---

From: "Max Clark" <[email-masked]>
Date: Tue, 23 Aug 2005 09:04:54 -0700

Read the whole thing. This is an actual letter sent to a man
>  >>>     named Ryan DeVries by the Michigan Department of Environmental
>  >>>     Quality, State of Michigan. This guy's response is hilarious,
>  >>>     but read the State's letter before you get to the response letter
>  >>>     (This is the State's Letter!)
>  >>>     SUBJECT: DEQ File No.97-59-0023; T11N; R10W, Sec. 20; Montcalm
>  >>>     County
>  >>>
>  >>>     Dear Mr. DeVries:
>  >>>
>  >>>     It has come to the attention of the Department of Environmental
>  >>>     Quality that there has been recent unauthorized activity on the
>  >>>     above referenced parcel of property. You have been certified as
>  >>>     the legal landowner and/or contractor who did the following
>  >>>     unauthorized activity:
>  >>>
>  >>>     Construction and maintenance of two wood debris dams across the
>  >>>     outlet stream of Spring Pond.
>  >>>
>  >>>     A permit must be issued prior to the start of this type of
>  >>>     activity. A review of the department's files shows that no
>  >>>     permits have been issued. Therefore, the Department has
>  >>>     determined that this activity is in violation of Part 301,
>  >>>     Inland Lakes and Streams, of the Natural Resource and
>  >>>     Environmental Protection Act, Act 451 of the Public Acts of
>  >>>     1994, being sections 324.30101 to 324.30113 of the Michigan
>  >>>     Compiled Laws, annotated.
>  >>>
>  >>>     The Department has been informed that one or both of the dams
>  >>>     partially failed during a recent rain event, causing debris and
>  >>>     flooding at downstream locations. We find that dams of this
>  >>>     nature are inherently hazardous and cannot be permitted. The
>  >>>     Department therefore orders you to cease and desist all
>  >>>     activities at this location, and to restore the stream to a
>  >>>     free-flow condition by removing all wood and brush forming the
>  >>>     dams from the stream channel. All restoration work shall be
>  >>>     completed no later than January 31, 2005.
>  >>>
>  >>>     Please notify this office when the restoration has been
>  >>>     completed so that a follow-up site inspection may be scheduled
>  >>>     by our staff. Failure to comply with this request or any further
>  >>>     unauthorized activity on the site may result in this case being
>  >>>     referred for elevated enforcement action. We anticipate and
>  >>>     would appreciate your full cooperation in this matter.
>  >>>     Please feel free to contact me at this office if you have any
>  >>>     questions.
>  >>>
>  >>>     Sincerely,
>  >>>
>  >>>     David L. Price, District Representative
>  >>>     Land and Water Management Division
>  >>>
>  >>>     ** Here is the actual response sent back by Mr. DeVries: **
>  >>>     Re: DEQ File No. 97-59-0023; T11N; R10W, Sec. 20; Montcalm
County.
>  >>>
>  >>>     Dear Mr. Price,
>  >>>
>  >>>     Your certified letter dated 12/17/02 has been handed to me to
>  >>>     respond to. I am the legal landowner but not the Contractor at
>  >>>     2088 Dagget, Pierson, Michigan. A couple of beavers are in the
>  >>>     process of constructing and maintaining two wood "debris" dams
>  >>>     across the outlet stream of my Spring Pond.
>  >>>
>  >>>     While I did not pay for, authorize, nor supervise their dam
>  >>>     project, I think they would be highly offended that you call
>  >>>     their skillful use of natures building materials "debris." I
>  >>>     would like to challenge your department to attempt to emulate
>  >>>     their dam project any time and/or any place you choose.
>  >>>
>  >>>     I believe I can safely state there is no way you could ever
>  >>>     match their dam skills, their dam resourcefulness, their dam
>  >>>     ingenuity, their dam persistence, their dam determination and/or
>  >>>     their dam work ethic.
>  >>>
>  >>>     As to your request, I do not think the beavers are aware that
>  >>>     they must first fill out a dam permit prior to the start of this
>  >>>     type of dam activity.
>  >>>
>  >>>     My first dam question to you is: (1) Are you trying to
>  >>>     discriminate against my Spring Pond Beavers, or (2) do you
>  >>>     require all beavers throughout this state to conform to said dam
>  >>>     request? If you are not discriminating against these particular
>  >>>     beavers, through the Freedom of Information Act, I request
>  >>>     completed copies of all those other applicable beaver dam
>  >>>     permits that have been issued. Perhaps we will see if there
>  >>>     really is a dam violation of Part 301, Inland Lakes and Streams,
>  >>>     of the Natural Resource and Environmental Protection Act, Act
>  >>>     451 of the Public Acts of 1994, being sections 324.30101to
>  >>>     324.30113 of the Michigan Compiled Laws, annotated.
>  >>>
>  >>>     I have several concerns. My first concern is; aren't the beavers
>  >>>     entitled to legal representation? The Spring Pond Beavers are
>  >>>     financially destitute and are unable to pay for said
>  >>>     representation -- so the State will have to provide them with a
>  >>>     dam lawyer. The Department's dam concern that either one or both
>  >>>     of the dams failed during a recent rain event, causing flooding,
>  >>>     is proof that this is a natural occurrence, which the Department
>  >>>     is required to protect. In other words, we should leave the
>  >>>     Spring Pond Beavers alone rather than harassing them and calling
>  >>>     their dam names.
>  >>>
>  >>>     If you want the stream "restored" to a dam free-flow condition
>  >>>     please contact the beavers -- but if you are going to arrest
>  >>>     them, they obviously did not pay any attention to your dam
>  >>>     letter, they being unable to read English.
>  >>>
>  >>>     In my humble opinion, the Spring Pond Beavers have a right to
>  >>>     build their unauthorized dams as long as the sky is blue, the
>  >>>     grass is green and water flows downstream. They have more dam
>  >>>     rights than I do to live and enjoy Spring Pond. If the
>  >>>     Department of Natural Resources and Environmental Protection
>  >>>     lives up to its name, it should protect the natural
>  >>>     resources(Beavers) and the environment (Beavers' Dams).
>  >>>
>  >>>     So, as far as the beavers and I are concerned, this dam case can
>  >>>     be referred for more elevated enforcement action right now. Why
>  >>>     wait until 1/31/2005? The Spring Pond Beavers may be under the
>  >>>     dam ice then and there will be no way for you or your dam staff
>  >>>     to contact/harass them then.
>  >>>
>  >>>     In conclusion, I would like to bring to your attention to a real
>  >>>     environmental quality (health) problem in the area. It is the
>  >>>     bears! Bears are actually defecating in our woods. I definitely
>  >>>     believe you should be persecuting the defecating bears and leave
>  >>>     the beavers alone. If you are going to investigate the beaver
>  >>>     dam, watch your step! (The bears are not careful where they
dump!)
>  >>>     Being unable to comply with your dam request, and being unable
>  >>>     to contact you on your dam answering machine, I am sending this
>  >>>     response to your dam office.
>  >>>
>  >>>     THANK YOU.
>  >>>
>  >>>     RYAN DEVRIES & THE DAM BEAVERS
>  >>>
>  >>
>
>
>
>

----- Original Message ----- 
From: "Jamie Lokier" <[email-masked]>
To: "Robert Collins" <[email-masked]>
Cc: "William A. Rowe, Jr." <[email-masked]>; "Julian Reschke" 
<[email-masked]>; "HTTP Working Group" <[email-masked]>
Sent: Tuesday, August 23, 2005 8:53 AM
Subject: Re: location uri, ucs and the http scheme definition.

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 23 Aug 2005 17:36:27 +0100

> Sorry? It's for instance used in 3xx (redirect) responses.

The effect the above refers to is "display of the URL in the user

-- Jamie