Subject: UTF-8 in URIs

From: Gabriel Montenegro <[email-masked]>
Date: Wed, 15 Jan 2014 19:55:30 +0000

Hi folks,

Some of us (cc line) have been discussing the unfortunate lack of determinism with respect to URI encoding in HTTP/1.1 and would like HTTP/2.0 to improve upon the situation.

I just opened this issue: https://github.com/http2/http2-spec/issues/342 (enable determinism for URI encoding in HTTP/2.0).

The "http" and "https" URI schemes don't have a fixed encoding. The URI RFC (http://tools.ietf.org/html/rfc3986#section-2.5) talks about the generic syntax for URI components:

  *   Legacy URI components (before 2005) tend to use UTF-8 "or some other superset of the US-ASCII character encoding"
  *   New schemes (after 2005) have to use UTF-8 with percent encoding for reserved characters.
The first bullet explains why we currently have non-determinism for "http" and "https" URIs. This is particularly problematic when parsing URIs at the server side or at intermediate proxies (e.g., when looking for a cache hit).

Proposed goal: enable determinism for URI encoding in HTTP/2.0.

Either (1)  a SETTING "SETTINGS_URI_ENCODING" or (2) an ":encoding" header. We favor option (2). In either case, the value to denote the charset would be a 32-bit integer equivalent to the "MIBenum" value in the IANA registry (http://www.iana.org/assignments/character-sets/character-sets.xhtml). Hence, the value would be 106 for UTF-8. The legacy behavior of non-determinism is indicated via the value 0. Notice that this is a reserved value for MIBenum.

Note: We could use the charset name, but there are actually two "name" columns in the IANA table (with the "name" value possibly having multiple values, and the "preferred MIME name" not always being present). We would also have to define a name to denote the legacy behavior.

Some use cases:

1.     A legacy client behind an HTTP/2.0 capable proxy, talking to an HTTP/2.0 capable server.

The client will use HTTP/1.1 to talk to the proxy.  Without special out-of-band knowledge, the proxy will not know the encoding for sure so would have to turn off the assumption when talking to the server by setting the value to 0 to denote legacy behavior.

2.      A HTTP/2.0 capable client behind an HTTP/2.0 capable proxy, talking to a legacy server.

The client will use HTTP/2.0 to talk to the proxy.  The never-standardized (and now expired) 3987bis added text about using the encoding of the containing HTML, e.g. iso-8859-1. If the server had that assumption, then the proxy has no way to know what the encoding was.
Thus to get correct behavior, either the client has to turn the SETTING off (indicating legacy behavior) when going via a proxy and get no benefit, or else have a way for the client to pass via HTTP/2.0 what the encoding was of the containing document or protocol.  The SETTING allows the latter, in this case by using the value 4 for iso-8859-1. This information allows the proxy's interpretation, e.g., for purposes of looking for a cache hit, whereas today it would get a cache miss.

Some pros and cons of these two mechanisms:

1.     SETTINGS would naturally allow to specify a general encoding like UTF-8 across all requests
o    Pro: This uses HTTP/2.0 as defined.
o    Cons: per-request changes are a bit hackish, requiring constantly sending this SETTING

2.     :encoding header would allow to specify the encoding on a per-request basis, but there would be no way to specify it in general.
o    Pro: clean per-request scope for such use-cases
o    Pro: benefits from header compression
o    Cons: This implies sending it on every request per current rules in HTTP/2.0, or another exception to those rules (similar to :authority).

Comments?

Thanks,

Gabriel

---

From: Michael Sweet <[email-masked]>
Date: Wed, 15 Jan 2014 15:07:16 -0500

Gabriel,

Encoding is only one aspect of this; the normalization form varies between OS's, so you'd need to say something (at least) about the normalization form to use with UTF-8 and other Unicode encodings - probably as a server requirement to do normalization to the local form that is used for files, etc.

On Jan 15, 2014, at 2:55 PM, Gabriel Montenegro <[email-masked]> wrote:

> The “http” and “https” URI schemes don’t have a fixed encoding. The URI RFC (http://tools.ietf.org/html/rfc3986#section-2.5) talks about the generic syntax for URI components:
> Legacy URI components (before 2005) tend to use UTF-8 “or some other superset of the US-ASCII character encoding”
> New schemes (after 2005) have to use UTF-8 with percent encoding for reserved characters.
> The first bullet explains why we currently have non-determinism for “http” and “https” URIs. This is particularly problematic when parsing URIs at the server side or at intermediate proxies (e.g., when looking for a cache hit).
>           
> Either (1)  a SETTING “SETTINGS_URI_ENCODING” or (2) an ":encoding" header. We favor option (2). In either case, the value to denote the charset would be a 32-bit integer equivalent to the “MIBenum” value in the IANA registry (http://www.iana.org/assignments/character-sets/character-sets.xhtml). Hence, the value would be 106 for UTF-8. The legacy behavior of non-determinism is indicated via the value 0. Notice that this is a reserved value for MIBenum.
>  

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Dave Thaler <[email-masked]>
Date: Wed, 15 Jan 2014 20:14:43 +0000

As long as the normalization form is NFC or NFD (i.e. not NFK*) then you don't need to say anything more,
since conversion between NFC and NFD is lossless.  The receiver can convert to its local form without additional
information.

-Dave

From: Michael Sweet [mailto:[email-masked]]
Sent: Wednesday, January 15, 2014 12:07 PM
To: Gabriel Montenegro
Cc: [email-masked]; Osama Mazahir; Dave Thaler; Mike Bishop; Matthew Cox
Subject: Re: UTF-8 in URIs

Gabriel,

On Jan 15, 2014, at 2:55 PM, Gabriel Montenegro <[email-masked]<mailto:[email-masked]>> wrote:

Hi folks,

Some use cases:

  1.   A HTTP/2.0 capable client behind an HTTP/2.0 capable proxy, talking to a legacy server.

     *   Pro: This uses HTTP/2.0 as defined.
     *   Cons: per-request changes are a bit hackish, requiring constantly sending this SETTING

  1.  :encoding header would allow to specify the encoding on a per-request basis, but there would be no way to specify it in general.

     *   Pro: clean per-request scope for such use-cases
     *   Pro: benefits from header compression
     *   Cons: This implies sending it on every request per current rules in HTTP/2.0, or another exception to those rules (similar to :authority).

Comments?

Thanks,

Gabriel

---

From: Zhong Yu <[email-masked]>
Date: Wed, 15 Jan 2014 14:46:39 -0600

Can you give an example where an intermediary benefits from decoding
URI octets into unicodes?

On Wed, Jan 15, 2014 at 1:55 PM, Gabriel Montenegro
<[email-masked]> wrote:
> Hi folks,
>
>
>
> Some of us (cc line) have been discussing the unfortunate lack of
> determinism with respect to URI encoding in HTTP/1.1 and would like HTTP/2.0
> to improve upon the situation.
>
>
>
> I just opened this issue: https://github.com/http2/http2-spec/issues/342
> (enable determinism for URI encoding in HTTP/2.0).
>
>
>
> The “http” and “https” URI schemes don’t have a fixed encoding. The URI RFC
> (http://tools.ietf.org/html/rfc3986#section-2.5) talks about the generic
> syntax for URI components:
>
> Legacy URI components (before 2005) tend to use UTF-8 “or some other
> superset of the US-ASCII character encoding”
> The first bullet explains why we currently have non-determinism for “http”
> and “https” URIs. This is particularly problematic when parsing URIs at the
> server side or at intermediate proxies (e.g., when looking for a cache hit).
>
>
>
> Either (1)  a SETTING “SETTINGS_URI_ENCODING” or (2) an ":encoding" header.
> We favor option (2). In either case, the value to denote the charset would
> be a 32-bit integer equivalent to the “MIBenum” value in the IANA registry
> (http://www.iana.org/assignments/character-sets/character-sets.xhtml).
> Hence, the value would be 106 for UTF-8. The legacy behavior of
> non-determinism is indicated via the value 0. Notice that this is a reserved
> value for MIBenum.
>
>
>
> Note: We could use the charset name, but there are actually two "name"
> columns in the IANA table (with the "name" value possibly having multiple
> values, and the "preferred MIME name" not always being present). We would
> also have to define a name to denote the legacy behavior.
>
>
>
> Some use cases:
>
>
>
> The client will use HTTP/1.1 to talk to the proxy.  Without special
> out-of-band knowledge, the proxy will not know the encoding for sure so
> would have to turn off the assumption when talking to the server by setting
> the value to 0 to denote legacy behavior.
>
>
>
> The client will use HTTP/2.0 to talk to the proxy.  The never-standardized
> (and now expired) 3987bis added text about using the encoding of the
> containing HTML, e.g. iso-8859-1. If the server had that assumption, then
> the proxy has no way to know what the encoding was.
>
> Thus to get correct behavior, either the client has to turn the SETTING off
> (indicating legacy behavior) when going via a proxy and get no benefit, or
> else have a way for the client to pass via HTTP/2.0 what the encoding was of
> the containing document or protocol.  The SETTING allows the latter, in this
> case by using the value 4 for iso-8859-1. This information allows the
> proxy's interpretation, e.g., for purposes of looking for a cache hit,
> whereas today it would get a cache miss.
>
>
>
> 2.     :encoding header would allow to specify the encoding on a per-request
> basis, but there would be no way to specify it in general.
>
> o    Cons: This implies sending it on every request per current rules in
> HTTP/2.0, or another exception to those rules (similar to :authority).
>
>
>
> Comments?
>
>
>
> Thanks,
>
>
>
> Gabriel

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 15 Jan 2014 20:54:10 +0000

>URI octets into unicodes?

Not necessarily from converting them into unicodes, but normalising
them using whatever rule we might prefer, so that cache-lookups
will always find the same object, no matter how the URI was mangled
with encodings.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 10:08:04 +0100

> will always find the same object, no matter how the URI was mangled

Why do you need more normalization than you have right now?

/me confused

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 16 Jan 2014 09:14:35 +0000

>Why do you need more normalization than you have right now?

The point is I need normalization, that slows down the lookup.

Remember:  Some of us have a goal of 1TB/s HTTP traffic :-)

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 10:22:52 +0100

> Remember:  Some of us have a goal of 1TB/s HTTP traffic :-)

I'm still confused.

Could you provide an example?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 16 Jan 2014 09:33:04 +0000

>Could you provide an example?

The current spec makes it legal to send %41 instead of 'A', normalizing
that before cache-lookup takes time.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Thu, 16 Jan 2014 10:52:26 +0100

> URI octets into unicodes?

Intermediaries can not perform URL-based filtering it they can not decode
URLS reliably. Intermediaries need to normalise URLs to a single encoding
if they log them (for debugging or policy purposes). unix-like "just a
bunch of bytes with no encoding indication" is an i18n disaster supported
only by users of ASCII scripts

I favour making URLs UTF-8 by default in HTTP/2 (just as it was in XML,
that's one part of the XML spec that worked very well) and require http/1
to 2 bridges to translate to the canonical form. Helping clients push
local 8bits encodings will just perpetuate pre-2000 legacy mess.

Whenever someone specifies a new better encoding it will be time for
HTTP/3. Unicode specs are way more complex than http, changes won't happen
quicker than http revisions.

Regards,

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 11:04:51 +0100

> that before cache-lookup takes time.

But you don't *have* to normalize it.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 11:06:45 +0100

> only by users of ASCII scripts

Well, you could log what you got on the wire. It's ASCII.

> local 8bits encodings will just perpetuate pre-2000 legacy mess.

How do you translate a URI with unknown URI encoding to UTF-8?

> quicker than http revisions.

The problem here is that HTTP URIs are octet sequences, not character 
sequences. There is no simple way to get from a) to b) without breaking 
a significant number of sites.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 16 Jan 2014 10:19:34 +0000

>But you don't *have* to normalize it.

And by default I don't.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Thu, 16 Jan 2014 11:24:01 +0100

> Well, you could log what you got on the wire. It's ASCII.

And it's useless if you can't interpret it reliably. May as well log the
output of /dev/random at the time. Don't have time to get humans comb
millions of log lines to fix encoding errors.

> How do you translate a URI with unknown URI encoding to UTF-8?

You treat it as UTF-8. If it fails UTF-8 sanity rules you fail with an
error. That will make people fix their encodings quickly.

> sequences.

The problem is that octet sequences are useless by themselves if you can
not decode them.

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 11:28:49 +0100

> millions of log lines to fix encoding errors.

Define "encoding error" in the context of a URI.

> error. That will make people fix their encodings quickly.

This is not going to work:

a) People may have chosen a non-UTF8 encoding by accident (system locale 
etc) and can't change it retroactively,

b) There might be actual *binary* data in the URI.

> The problem is that octet sequences are useless by themselves if you can

Hm, no. They just happen to work in a way different from your 
preference, but they do just work fine.

Best regards, Julian

(Yes, I'd prefer that there'd be more UTF-8 in HTTP, but there are 
problems that are hard to solve without breaking things)

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 16 Jan 2014 12:00:03 +0100

* Gabriel Montenegro wrote:
>determinism with respect to URI encoding in HTTP/1.1 and would like 
>HTTP/2.0 to improve upon the situation.

The practise of encoding character data in `http:` addresses using
anything other than UTF-8 is dying out fast and it is rather unclear
what practical benefit there is in discriminating between addresses
that use only character data and all character data is UTF-8-encoded
and addresses that include non-character data or use some legacy en-
coding.

Note that it is perfectly normal to run a service like

  http://example.org/transcode?from=iso-8859-1&to=utf-8&bytes=%C3%B6

Also note that a client cannot possibly know `%C3%B6` can be inter-
preted as UTF-8 bytes without the server telling it as much. This does
not change when it's instead

  http://example.org/transcode/from/iso-8859-1/to/utf-8/bytes/%C3%B6

Further note that some clients, for display purposes, treat at least
one of the two examples as though the `%C3%B6` were UTF-8.

>In either case, the value to denote the charset would be a 32-bit 
>integer equivalent to the "MIBenum" value in the IANA registry 
>non-determinism is indicated via the value 0. Notice that this is a 
>reserved value for MIBenum.

Allowing arbitrary encodings needs an exceedingly good reason.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Zhong Yu <[email-masked]>
Date: Thu, 16 Jan 2014 05:18:45 -0600

> anything other than UTF-8 is dying out fast and it is rather unclear

UTF-8 is not very good for CJK charsets. It may not be a big deal in
general, however, URLs are often displayed verbatim on user
interfaces, the length matters.

---

From: Zhong Yu <[email-masked]>
Date: Thu, 16 Jan 2014 05:25:04 -0600

There is no way to enforce UTF-8 on URIs; we cannot even enforce
%-encoding, the server can always build proprietary encoding on top of
ASCII chars (for its own convenience, not for being cryptic to others)

URIs have never been supposed to be understandable by anyone other
than the original server. I don't see how we can change that, unless
we turn URI into a full blow language with structures, semantics, and
a huge vocabulary.

Zhong Yu

---

From: Michael Sweet <[email-masked]>
Date: Thu, 16 Jan 2014 09:29:19 -0500

Julian,

Consider a file named "exposé.html", served by www.example.com. This URI can be encoded in many different ways depending on the character set and (for Unicode) normalization form used, for example:

    ISO-8859-1      http://www.exmaple.com/expos%E9.html

    UTF-8 NFD       http://www.exmaple.com/expose%CC%81.html

    UTF-8 NFC       http://www.exmaple.com/expos%C3%A9.html

Today, you have no guarantee that typing "http://www.example.com/exposé.html" in your web browser will work since the browser's choice of character set and normalization form may not match the server's, and it may not be possible for the server to correctly guess.  And an intermediate proxy will have difficulty efficiently/correctly caching the content as well.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Thu, 16 Jan 2014 15:33:30 +0100

> we turn URI into a full blow language with structures, semantics, and

Look, that is all nonsense.

URLs are treated as text in html documents. URL are treated as text in
logs and traffic consoles. URL are treated as text by web site designers
(otherwise all accesses would be in the form mywebsite.com/opaquenumber
and how many sites actually do that?). Web traffic is not direct
end-to-end it goes through intermediaries that need to decode part of the
http envelope and besides web sites are more and more inter penetrated
(URL soup aka mashup and clouds) so decoding has not been a private web
site affair for a long time

All those elements do not manipulate chains of bytes but text and the
difference between chains of bytes and text is clear encoding rules (I
know it is a huge understanding leap for most developers that didn't have
to deal extensively with encoding problem fallouts)

There is a difference between semantics (which are the business of web
sites) and technical encoding. I don't care a fig about what encoding a
web server uses on its filesystem or the encoding of web pages. What I
want is that the on-wire representation, that needs to be decoded by all
kinds of third parties for things to work smoothly, to be clearly defined
without the usual "chain of bytes" cop-out.

Software writers that love exotic encodings just have to translate URLs to
UTF-8 representation before http emission. Their choice. Interoperability
problems should be pushed to end-nodes not embedded in the protocol
itself.

I'd rather have clear interop rules and simple to understand breakage at
http2 adoption time rather than diffuse heisenbugs and sites that work by
sheer chance as long as you chain components with the same unwritten
assumptions. See the python 2 unicode debacle and how much fuzzy encoding
rules cost them.

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 15:38:34 +0100

>      UTF-8 NFC       http://www.exmaple.com/expos%C3%A9.html

Yes. I'm painfully aware of that. It's a big issue with WebDAV as 
Windows and OsX picked different normalization forms.

> Today, you have no guarantee that typing "http://www.example.com/exposé.html" in your web browser will work since the browser's choice of character set and normalization form may not match the server's, and it may not be possible for the server to correctly guess.

The best way to fix this is to educate people to use UTF-8. One way to 
do this is to let the UAs do what they do already (default to UTF-8).

This is an evangelism issue; changing the protocol doesn't help.

> And an intermediate proxy will have difficulty efficiently/correctly caching the content as well.

It can just treat them as different URIs.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 15:41:52 +0100

> Look, that is all nonsense.

Um, no.

> and how many sites actually do that?). Web traffic is not direct

Yes. So?

> site affair for a long time

I still don't understand why intermediaries "need" to "decode" request URIs.

> to deal extensively with encoding problem fallouts)

The URI on the wire is indeed a sequence of ASCII characters (well, a 
legal one). The fact that non-ASCII characters and delimiters can be 
embedded using percent-escaping doesn't change that fact.

> without the usual "chain of bytes" cop-out.

Again: please clarify why it needs to be "decoded" by anybody except the 
origin server.

> ...

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Thu, 16 Jan 2014 15:57:06 +0100

> Define "encoding error" in the context of a URI.

Any URI that can not be reliably decoded in the textual representation the
URL creator preferred by a random http processor (web site, intermediary,
web client) without outside help.

And there *is* a preferred textual representation because you know, people
do not enter URLs in binary editors.

> etc) and can't change it retroactively,

They can add an UTF-8 translator gateway at http/2 adoption time. No
different and much easier than the mass of documents that needed to be
fixed once people started exchanging them in binary not dead wood form.
Some past mistakes need correction you can't grandfather them eternally at
the cost of eternal future interop problems.

> b) There might be actual *binary* data in the URI.

So just define the canonical binary-to-utf8 mapping. If you don't your URL
will crash as soon as it needs to be displayed in an address bar, network
console or activity log. Again, much easier to define a single
binary-to-utf8 mapping than random encoding to random display encoding
rules (hint: it is not possible and that's the core problem).

> preference, but they do just work fine.

No, they don't work. Working is not "avoid any automated processing and
for christsakes never use anything but ASCII because the state is
undefined and things will randomly break"

There is a reason url bars do not display binary sequences. You need
reliable text decoding for humans to get a grip on the result. %foo
punicode approach is predicated on browsers being able to display a text
form, otherwise the result is just binary soup and humans can not
distinguish correct binary soup from hijacked binary soup.

All the talk about binary http2 being ok because someone will just write a
wireshark plugin is just wind if the wireshark plugin does not have
reliable rules to translate the on-wire representation to text humans
understand.

Best regards,

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Thu, 16 Jan 2014 16:08:32 +0100

> I still don't understand why intermediaries "need" to "decode" request

Because you want to write intermediary processing rules in text form, just
like server sites write their rules in text form, and the web browser user
writes his request in text form, and nobody wants to write his rules in
binary because the encoding of the processed objects is undefined

Because traffic consoles that displays octet value chains are useless in
practical terms.

Because web objects are identified by urls and the identifier changing
depending on random client/server encoding choices increases the
complexity level way over just telling everyone "write your urls in http2
in utf-8".

Because there *are* semantics in web site organisation but they are only
apparent in the text encoding form the site creator used.

Because all the systems that tried to jungle multiple implicit encodings
instead of imposing a single rule have been pathetic failures (they "work"
as long as all the actors to not use the multiple encoding freedom but add
the encoding convention the designer forgot to provide)

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 16:11:47 +0100

> web client) without outside help.

A valid URI is all US-ASCII. There's nothing that needs to be decoded at 
all.

> fixed once people started exchanging them in binary not dead wood form.

And that translator rewrites all URIs that appear in payloads?

> the cost of eternal future interop problems.

The only interop problem I'm aware of is when clients construct *new* 
URIs, such as is common in WebDAV. The way to fix this is to *advocate* 
UTF-8. But even if everybody agrees on UTF-8 there's still the NFC/NFD 
mismatch between OSX and the rest of the world.

> rules (hint: it is not possible and that's the core problem).

I still have no clue what problem you are trying to solve. Sorry.

> undefined and things will randomly break"

The state is fully defined. It's just that you don't like that state.

It seems we aren't getting anywhere.

Can somebody else help me understanding what this is all about? :-)

Best regards, Julian

---

From: Gabriel Montenegro <[email-masked]>
Date: Thu, 16 Jan 2014 18:48:47 +0000

To clarify: The proposal is *NOT* to impose a default of UTF-8 in URIs for HTTP/2.0. As some have mentioned, there are too many legacy issues.

The issue is that since http and https are legacy scheme's from the point of view of rfc3986, they don't have a fixed encoding. If somebody were to define a new scheme, say, "http2" that would benefit from rfc3986 rules so the encoding would be known: UTF-8 with percent encoding.

Unfortunately, for http and http2 URI handling at either the proxy (to check for a cache hit) or at the origin server is non-deterministic. Several encodings are tried until one works. Such non-determinism is also a potential security issue, as a URI could decode in more than one way as several encoding are tried. 

However, iff an HTTP/2.0 client knows for sure the encoding (e.g., UTF-8), per the proposal it could indicate it so at the receiving side there are no guessing games: in the presence of such an explicit indication, either it is valid UTF-8, or it is an error, no further processing is done.

-----Original Message-----
From: Julian Reschke [mailto:[email-masked]] 
Sent: Thursday, January 16, 2014 7:12
To: Nicolas Mailhot
Cc: Zhong Yu; Gabriel Montenegro; [email-masked]; Osama Mazahir; Dave Thaler; Mike Bishop; Matthew Cox
Subject: Re: *** GMX Spamverdacht *** Re: UTF-8 in URIs

On 2014-01-16 15:57, Nicolas Mailhot wrote:
> ...
>>> And it's useless if you can't interpret it reliably. May as well log 
>>> the output of /dev/random at the time. Don't have time to get humans 
>>> comb millions of log lines to fix encoding errors.
>>
> Any URI that can not be reliably decoded in the textual representation 
> the URL creator preferred by a random http processor (web site, 
> intermediary, web client) without outside help.

A valid URI is all US-ASCII. There's nothing that needs to be decoded at all.

> And there *is* a preferred textual representation because you know, 
> people do not enter URLs in binary editors.
>
>>>>> I favour making URLs UTF-8 by default in HTTP/2 (just as it was in 
>>>>> XML, that's one part of the XML spec that worked very well) and 
>>>>> require
>>>>> http/1
>>>>> to 2 bridges to translate to the canonical form. Helping clients 
>>>>> push local 8bits encodings will just perpetuate pre-2000 legacy mess.
>>>>
>>> You treat it as UTF-8. If it fails UTF-8 sanity rules you fail with 
>>> an error. That will make people fix their encodings quickly.
>>

> Some past mistakes need correction you can't grandfather them 
> eternally at the cost of eternal future interop problems.

The only interop problem I'm aware of is when clients construct *new* URIs, such as is common in WebDAV. The way to fix this is to *advocate* UTF-8. But even if everybody agrees on UTF-8 there's still the NFC/NFD mismatch between OSX and the rest of the world.

> So just define the canonical binary-to-utf8 mapping. If you don't your 
> URL will crash as soon as it needs to be displayed in an address bar, 
> network console or activity log. Again, much easier to define a single

> No, they don't work. Working is not "avoid any automated processing 
> and for christsakes never use anything but ASCII because the state is 

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Thu, 16 Jan 2014 10:59:31 -0800

> However, iff an HTTP/2.0 client knows for sure the encoding (e.g., UTF-8), per the proposal it could indicate it so at the receiving side there are no guessing games: in the presence of such an exp…

What you are proposing, perhaps, is that HTTP/2.0 support the carriage
of IRIs.  Noting that this is effectively what browsers and HTML
already do, perhaps that's not a terrible thing.

Would this apply to just :path, or would you extend this to :authority
and allow IDNs there?  That does mean that the proxy/server is
potentially exposed to unicode normalization rules and so forth.

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 17 Jan 2014 09:52:06 +1300

On 2014-01-17 07:59, Martin Thomson wrote:
> On 16 January 2014 10:48, Gabriel Montenegro wrote:
>> However, iff an HTTP/2.0 client knows for sure the encoding (e.g., 
>> UTF-8), per the proposal it could indicate it so at the receiving side 
>> there are no guessing games: in the presence of such an explicit 
>> indication, either it is valid UTF-8, or it is an error, no further 
>> processing is done.
> 

A little nastiness exists for caches like PHK indicated. But this is no 
more trouble than Vary and can be handled in the same ways if necessary. 
It is optional and just tradeoff between CPU and memory in the end.

A slightly bigger issue is what mapping 2.0->1.1 gateways do with 
random-encoded URI.

As mentioned above the URI representation in HTTP/2 is already being 
split into components, so it does not exactly fit with the RFC 2616 
textual format to begin with. It seems simple enough to reference the 
new :foo components as containing RFC 3986 compliant UTF-8 pieces, to be 
reassembled or decoded by the implementations as needed.

I wonder if the whole situation would not be improved by making HTTP/2 
transport UTF-8 on a SHOULD requirement on emitting, with a note that 
senders may have limited themselves to the ASCII compatible range with 
%-encode of other bytes, and 2.0->1.1 gateways MUST %-encode the 
characters outside the ASCII range when delivering into HTTP/1 ?

Amos

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Jan 2014 22:32:28 +0100

> To clarify: The proposal is *NOT* to impose a default of UTF-8 in URIs for HTTP/2.0. As some have mentioned, there are too many legacy issues.

Yes.

> Unfortunately, for http and http2 URI handling at either the proxy (to check for a cache hit) or at the origin server is non-deterministic. Several encodings are tried until one works. Such non-det…

A proxy does not need to normalize. Full stop. There is no issue here, IMHO.

The URIs are minted by the origin server, and IMHO it's not a problem if 
a client that decodes and re-encodes using a different encoding gets 
punished by a cache miss.

An origin server doesn't need to try different encoding. It's sufficient 
if it uses the encoding it used when generating the URI. And if that 
encoding happens to be UTF-8, it will also benefit from IRIs typed into 
a browser bar.

> However, iff an HTTP/2.0 client knows for sure the encoding (e.g., UTF-8), per the proposal it could indicate it so at the receiving side there are no guessing games: in the presence of such an exp…

Again; nobody *needs* to guess right now. So I'm still confused about 
what problem you're trying to solve.

And even *if* this helps this still leaves the related issue of NFC vs 
NFD unsolved.

Best regards, Julian

---

From: Larry Masinter <[email-masked]>
Date: Fri, 17 Jan 2014 07:50:40 +0000

It's a little hard to wade through the rhetoric ("dead trees" ?) but I don't see a problem with saying in HTTP/2.0 that a request can be an IRI or an IRI-path encoded in UTF-8. Also a "Host" header can have a UTF-8 encoded Unicode for IDN.

Gateways from HTTP/1 can leave the URI path decoded; probably should not change any values.
Gateways from HTTP/2 to HTTP/1 should percent-hex-encode any non-ASCII character in both host and path.

This is a new HTTP/2 feature, though, and is it worth it? It's more complexity and a small saving in space, all things considered.

Larry
--
http://larry.masinter.net

-----Original Message-----
From: Nicolas Mailhot [mailto:[email-masked]] 
Sent: Thursday, January 16, 2014 7:09 AM
To: [email-masked]
Cc: Nicolas Mailhot; Zhong Yu; Gabriel Montenegro; [email-masked]; Osama Mazahir; Dave Thaler; Mike Bishop; Matthew Cox
Subject: Re: UTF-8 in URIs

-- 
Nicolas Mailhot

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 17 Jan 2014 16:59:52 +0900

Hello Gabriel,

Responding to your original post after having read through the thread.

>    *   New schemes (after 2005) have to use UTF-8 with percent encoding for reserved characters.

Please note that there are a lot of schemes that indeed do so, even 
before 2005, at least if implemented per spec. For examples, see 
http://tools.ietf.org/search/rfc3987#section-1.2, point c).

On the other hand, there are schemes that are clearly post 2005, but 
which seem to work exactly as http/https. As far as I understand, ws/wss 
works that way (using the same codepaths in browsers as http/https), 
although I haven't tested it and would of course be pleasantly surprised 
if I was wrong.

> Either (1)  a SETTING "SETTINGS_URI_ENCODING" or (2) an ":encoding" header. We favor option (2). In either case, the value to denote the charset would be a 32-bit integer equivalent to the "MIBenum…

This might work in theory, but it would be a very late fix to a 
long-standing problem that would either work out but push things in the 
wrong direction, or fail anyway.

I think a less ambiguous thing could do more good and push the Web more 
in the right direction: Just have two values, one for UTF-8 and one for 
"unknown". Actually, because UTF-8 is easily detected heuristically, we 
are already close to that.

> Note: We could use the charset name, but there are actually two "name" columns in the IANA table (with the "name" value possibly having multiple values, and the "preferred MIME name" not always bei…

I agree that the number is easier to handle than the name. It also fits 
better with the binary nature of HTTP2. I don't think we need a 32-bit 
value, 16 bits would be enough, in case we want to save bits. There are 
a few corner cases that would need careful examination. As an example, 
on the Web, the label 'iso-8859-1' in practice often means 
'windows-1252', so we would need to specify what to do in such cases.

> The client will use HTTP/2.0 to talk to the proxy.  The never-standardized (and now expired) 3987bis added text about using the encoding of the containing HTML, e.g. iso-8859-1.

That was only for the query part, and it was added because that's what 
browsers do in practice, for the query part. The path part still is 
(supposed to be) UTF-8, as again that's what browsers do in practice.

So you can have two encodings in the same URI. How would your proposal 
handle such cases?

Regards,    Martin.

If the server had that assumption, then the proxy has no way to know 
what the encoding was.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 17 Jan 2014 17:19:52 +0900

> interfaces, the length matters.

The idea that UTF-8 is not very efficient for CJK text is to the most 
part just a myth. It is true that UTF-8 requires 3 bytes per character 
for CJK characters. However, because CJK characters are ideographic or 
syllabic, the number of characters is much lower than for alphabetic 
scripts.

Languages with alphabetic scripts that require two bytes for each 
character in UTF-8 (e.g. Greek, Russian, Arabic, Hebrew,...) easily 
require more memory for the same text than CJK languages. Languages with 
alphabetic scripts that require three bytes per character in UTF-8 (e.g. 
Hindi and all the other Indian languages, Thai, Lao, Khmer,...) most 
surely require more memory for the same text than CJK languages.

When UTF-8 in URIs is transported or displayed with %-encoding, an 
additional factor of 3 is needed, but that's the same for all the 
languages mentioned above. This is indeed quite inconvenient when 
displayed, but the main inconvenience is not the length, but the fact 
that it cannot be read. Displaying the actual characters, and using 
UTF-8 directly in transport (i.e. essentially using IRIs) solves this 
problem.

Regards,   Martin.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 17 Jan 2014 09:02:50 +0000

In message <[email-masked]>, "Nicol

>assumptions. See the python 2 unicode debacle and how much fuzzy encoding

Seconded.

This is one clear way HTTP/2 can improve over HTTP/1.1:  Eliminate
unnecessary and pointless fuzziness.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 17 Jan 2014 10:10:12 +0100

> unnecessary and pointless fuzziness.

There is no fuzziness here.

If you really want to improve over HTTP/1.1, there are more interesting 
issues, such as character encoding in header field values.

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 17 Jan 2014 11:18:16 +0100

Le Jeu 16 janvier 2014 22:32, Julian Reschke a écrit :

> A proxy does not need to normalize. Full stop. There is no issue here,

A security proxy does need to normalize. Full stop. Otherwise malware can
trivially bypass security blocks by fuzzing encoding enough the proxy does
not realize anymore the block needs to be applied.

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Fri, 17 Jan 2014 11:28:48 +0100

> not realize anymore the block needs to be applied.

Are you talking about normalization beyond removing unneeded 
percent-escapes?

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Fri, 17 Jan 2014 11:55:15 +0100

> Are you talking about normalization beyond removing unneeded

I'm taking about the very common case when a botnet or malware stain
signature is an URL fragment it tries to communicate with on random zombie
hosts on the Internet. It is very common to configure proxy gateways to
block any access to an url that includes this fragment as first level
defence while more accurate and complete cleanup measures are
investigated.

(malware is the worst case, sometimes it's just misbehaving browser
plugins or other web clients that need blocking to keep the network
operational)

Obviously that only works if the gateway can recognize the URL fragment
without being confused by encoding games. So the gateway does need a
reliable way to map byte chains to the text signature (and there is a text
signature because the app writer did use text stings and not random
constants in his code). Unspecified text encoding conventions in URLs make
reliability go away.

Again, I would like http/2 to specify that URLs are transported as UTF-8
text in http2 metadata (ideally not %-escaped), with the endpoints being
responsible to converting their local representation to this form before
emission, or baring that
1. add encoding info somewhere
2. require the web client and server to fill this info.

But I really would prefer if the wire representation was unambiguous and
encoding conversions pushed to endpoints. That's the model python people
settled on after years of failing to make the "push everything as chain of
bytes, whatever needs text will manage to convert by itself" work. And
http nodes are way less flexible than a python program.

Regards,

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Fri, 17 Jan 2014 12:02:56 +0100

> http nodes are way less flexible than a python program.

Nicolas,

please provide a concrete example.

Best regards, Julian

---

From: Zhong Yu <[email-masked]>
Date: Fri, 17 Jan 2014 06:53:19 -0600

On Fri, Jan 17, 2014 at 2:19 AM, "Martin J. Dürst"
<[email-masked]> wrote:
> The idea that UTF-8 is not very efficient for CJK text is to the most part
> just a myth. It is true that UTF-8 requires 3 bytes per character for CJK
> characters. However, because CJK characters are ideographic or syllabic, the
> number of characters is much lower than for alphabetic scripts.
>
> Languages with alphabetic scripts that require two bytes for each character
> in UTF-8 (e.g. Greek, Russian, Arabic, Hebrew,...) easily require more
> memory for the same text than CJK languages. Languages with alphabetic
> scripts that require three bytes per character in UTF-8 (e.g. Hindi and all
> the other Indian languages, Thai, Lao, Khmer,...) most surely require more
> memory for the same text than CJK languages.

So there are languages that suffer even more from UTF-8. That doesn't
make me feel any better about UTF-8:)

> additional factor of 3 is needed, but that's the same for all the languages
> mentioned above. This is indeed quite inconvenient when displayed, but the
> main inconvenience is not the length, but the fact that it cannot be read.
> Displaying the actual characters, and using UTF-8 directly in transport
> (i.e. essentially using IRIs) solves this problem.

An UTF-16 option would be nice. Let's be honest, UTF-8 is
English-centric. It may be necessary to interoprate with previous
ASCII based systems. But going forward, UTF-8 should not be favored
just because it is the best option for the English language.

Zhong Yu

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 17 Jan 2014 13:03:43 +0000

>just because it is the best option for the English language.

You're kidding, right ?

You want to make all HTTP software much more complex and prone
to software bugs and exploitation, just because you think UTF-8
is "english-centric" ?

Really ?

---

From: Michael Sweet <[email-masked]>
Date: Fri, 17 Jan 2014 08:10:34 -0500

With all due respect, all of the protocols we use on the Internet use octets as the basis of text strings, and in particular most strings passed over the Internet (headers, header values, URIs, hostnames, etc.) do not even need support beyond US ASCII.  This is a huge benefit for interoperability at the tiny expense of expansion for some languages.

The place where UTF-16 has the most benefit is in the content that is transferred, not in the fractional protocol overhead used to transfer that content.  (and even there, I would compare the size of your content encoded as UTF-8 and as UTF-16 before making a decision - web pages often are better off as UTF-8 due to the HTML markup, while email messages are better as UTF-16, for example...)

---

From: Zhong Yu <[email-masked]>
Date: Fri, 17 Jan 2014 19:55:36 -0600

On Fri, Jan 17, 2014 at 7:10 AM, Michael Sweet <[email-masked]> wrote:
> With all due respect, all of the protocols we use on the Internet use octets as the basis of text strings, and in particular most strings passed over the Internet (headers, header values, URIs, hostnames, etc.) do not even need support beyond US ASCII.  This is a huge benefit for interoperability at the

It is interesting that a lot of non-English resources have URLs with
English words. However, technical limitations may be a contributing
factor. It probably should not be taken as a basic assumption when
designing new technical specs.

tiny expense of expansion for some languages.

---

From: Tim Bray <[email-masked]>
Date: Sat, 18 Jan 2014 09:34:21 -0800

> just because it is the best option for the English language.

UTF-16 is a horrorshow, what with its surrogates, the inability to handle
it in C code as either char* or wchar_t *, and so on.  Yes, I agree that
UTF-8 is sort of bigoted.  But it has a lot of advantages, and actually I
find it had to worry too much about the somewhat-less-than-50% overhead
(less than 50% due to ASCII markup), because when I am having trouble with
network congestion, the congestion is always due to media files... you need
a *lot* of text to match a few seconds of music or video.

Without expressing an opinion on exactly what to say about URIs, I
definitely think everything should be UTF-8 wherever possible, and note
with pleasure that the Internet is moving steadily in that direction.

---

From: Martin Thomson <[email-masked]>
Date: Sun, 19 Jan 2014 12:46:26 -0800

On 18 January 2014 09:34, Tim Bray <[email-masked]> wrote:
> definitely think everything should be UTF-8 wherever possible, and note with
> pleasure that the Internet is moving steadily in that direction.

That's definitely the trend, and a good one.

If size is an issue, then that is a task for compression.  Not that
I'm recommending we customize hpack to account for this now - that
would be premature - but the option remains open.

---

From: Larry Masinter <[email-masked]>
Date: Wed, 22 Jan 2014 19:39:29 +0000

A URI is a sequence of ASCII characters, with a subset chosen such that even EBCDIC can represent it. (Yes, EBCDIC was a concern.)

Yes, some URI schemes can define other uses for %xx-encoded octets, but this belongs in the scheme definition and not generic  processing.

An IRI is a sequence of Unicode characters.  It can be generally translated to a URI for processing by encoding in UTF-8 and %xx percent-hex-encoding the results.

There are a couple of exceptions recommended for browsers (only), but they shouldn't change the general rule:
  Exception 1: For interoperability with IDN, an IRI with a non-ASCII host name in the  authority, and the scheme is one
             of a small number of schemes (ftp, http), the IRI might also be translated into a URI where the host name is punicode encoded.

This exception is proposed because IDN servers only recognize the punicode representation of domain names
and not %xx hex-encoded.

   Exception 2: For IRIs embedded (only) in HTML (only) where the scheme is http (only),where the query portion of the IRI has non-ascii characters 
         and the HTML document character set is not ASCII, then the Query portion of the IRI (only) should be the %xx percent-hex-encoding
         of the query portion alone should proceed by representing the query string in the document charset and then hex-encoding that.

This exception is proposed as implementation-compatibility measures for browsers, because one
browser was released with a bug, web sites were coded to match the browser's bugs, and now
all the browsers need to mimic the broken behavior.

I see no justification for any additional use of non-UTF8 encodings used in the IRI -> URI translation.

If anyone has any evidence of any implementations using other encodings, could they please describe it?

We want to update the IRI spec under the apps area working group.e

Thanks,

Larry