Subject: Unicode sucks, get over it (Re: Delta Compression and UTF-8 Header Values)

From: Nico Williams <[email-masked]>
Date: Sun, 10 Feb 2013 16:45:01 -0600

On Sun, Feb 10, 2013 at 3:04 PM, Roberto Peon <[email-masked]> wrote:
> Another place where we may need to know about normalization is for caching.
> Does the lookup, etc. occur on the normalized form, or on the given data?
>
> All in all, utf-8 without addendum sucks for protocol work.

Normalization is not a UTF-8 thing, it's a Unicode thing, and it's not
really a Unicode thing either, but a result of our stupid, human
scripts and their stupid collation and other rules.

There is *nothing* that we can do for dealing with text that would do
both of: a) meet the needs of our users, and b) not suck for string
comparison, collation, and other such operations.

In other words: all these arguments about how it sucks to deal with
UTF-8 or Unicode are not useful arguments.  We have to deal with text
in at least some parts of our protocols, and that means we have to
deal with I18N.

Worse, much worse than the problems Unicode brings with it, are the
problems of having either no clue what codeset some text is in
(interop failures result), or having to support many, many codesets
(trade one set of complexities for a bigger one).  Clearly it is
better to just use Unicode for text in Internet protocols.

It is also clear that we can't really have a decent one-size-fits-all
static Huffman coding table for text that may be written in any of
tens of scripts spanning ~100k codepoints.  Now, perhaps we can
encourage the world to use URIs not IRIs and so on, but really, that
would be a step backwards.

My proposal:

 - All text values in HTTP/2.0 that are also present in HTTP/1.1
should be sent as either UTF-8 or ISO8859-1, with a one-bit tag to
indicate which it is.

   This pushes re-encoding to the ends, but it lets middle boxes
re-encode as well where they want or need to, and it gives us a nice
upgrade path.

 - All text values in HTTP/2.0 that are NOT also present in HTTP/1.1
should be sent *only* as UTF-8.

Why UTF-8 and not some other encoding of Unicode?  Because I don't see
how UTF-16 or UTF-32 could help us here.  Other encodings seem even
less likely to be useful: sure, punycode would be all ASCII, but it
wouldn't actually cause static Huffman coding to be useful.  At best
one can argue that UTF-8 penalizes some scripts with a 50% or 100%
expansion relative to script-specific codesets, so that we should
prefer UTF-16 or -32 for fairness reasons; let's not.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 10 Feb 2013 20:19:40 -0500

+1

A lot of the 'security issues' folk come up with relating to homographs
etc. turn out to be complete busts in practice.

URIs have always been case sensitive. Why should the encoding of accents
not be sensitive as well?

Yes, someone might confuse two similar glyphs from different character
sets. But what is the probability that someone would type the weird
encoding in at the keyboard? I think its zero.

Sure you can come up with corner cases and there are some security issues.
But I can't see a HTTP issue here still less a reason to be concerned about
encoding

-- 
Website: http://hallambaker.com/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Feb 2013 09:35:45 +0100

On 2013-02-10 23:45, Nico Williams wrote:
> ...

Why do we need two options?

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Mon, 11 Feb 2013 09:17:11 -0600

> Why do we need two options?

We probably don't.  The idea was that if you have a client and server
speaking HTTP/1.1 and using ISO8559-1 (including non-ASCII
codepoints), *and* HTTP/2.0 proxies were involved that wanted to
rewrite the HTTP/1.1 as 2.0, well, they could do it and avoid
re-encoding those ISO8859-1 strings.  Probably not worth it; better go
with UTF-8 alone, period.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 11 Feb 2013 10:56:57 -0500

+1

I can't see a good reason why a HTTP2 proxy would not speak HTTP/1.1 for a
long time to come. If it is rewriting requests as HTTP2 then it probably
has a reason for doing so that would make UTF8 desirable as well.