Subject: UTF-8 or ASCII Header Names?

From: James M Snell <[email-masked]>
Date: Tue, 13 Aug 2013 15:08:13 -0700

https://github.com/http2/http2-spec/issues/231

The current header compression spec allows for UTF-8 encoded header
names without any character restrictions. The main HTTP/2 spec,
however, states that header names are "strings of ASCII characters"
(also without specifying any character restrictions). We need to be
clearer.

Recommend that we specify in both the HTTP/2 and Header Compression
spec that header names MUST conform to:

    LOWERALPHA = %x61-7A
    header-name = "!" / "#" / "$" / "%" / "&" / "'" /
                  "*" / "+" / "-" / "." / "^" / "_" /
                  "`" / "|" / "~" / DIGIT / LOWERALPHA

Which is the all-lower-case equivalent to the header-name definition
currently in httpbis.

---

From: Michael Sweet <[email-masked]>
Date: Tue, 13 Aug 2013 18:12:18 -0400

+1

>    header-name = "!" / "#" / "$" / "%" / "&" / "'" /

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Martin Thomson <[email-masked]>
Date: Tue, 13 Aug 2013 23:20:47 +0100

>     header-name = "!" / "#" / "$" / "%" / "&" / "'" /

Actually, it's:
    LOWERALPHA = %x61-7A
    header-char = "!" / "#" / "$" / "%" / "&" / "'" /
                  "*" / "+" / "-" / "." / "^" / "_" /
                  "`" / "|" / "~" / DIGIT / LOWERALPHA
    header-name = (":" / header-char) *header-char

though this might be better:
    header-name = [":"] 1*header-char

and if we're feeling especially generous:
    header-name = 1*(":" / header-char)

This sounds reasonable - though I think that this needs to be a little
more nuanced.  Header compression might describe a transformation that
produces the limited set of values as described above, but the *input*
to header compression needs to be a valid HTTP header (or a special
HTTP/2.0 :-header).

---

From: James M Snell <[email-masked]>
Date: Tue, 13 Aug 2013 15:26:46 -0700

Thanks for catching the missing ":" ... and yes, [":"] 1*header-char
is a much better choice.

-1 to adding any "nuance" or transformations, however. Let's be clear
and strict about this: an HTTP/2 header field name ought to always
match... period.

    LOWERALPHA = %x61-7A

We don't need any other options or "nuance" here.

- James

---

From: Fred Akalin <[email-masked]>
Date: Tue, 13 Aug 2013 15:36:06 -0700

I'm definitely for removing any reference to UTF-8 in the header
compression spec, if only to avoid the giant can of worms it introduces
with lower-casing.

---

From: James M Snell <[email-masked]>
Date: Tue, 13 Aug 2013 15:39:53 -0700

-1, that's certainly not the recommendation I was making.

Header field names ought not be UTF-8... Allowing UTF-8 in header
field values is extremely valuable.

- James

On Tue, Aug 13, 2013 at 3:36 PM, Fred Akalin <[email-masked]> wrote:
> I'm definitely for removing any reference to UTF-8 in the header compression
> spec, if only to avoid the giant can of worms it introduces with
> lower-casing.
>
>

---

From: Fred Akalin <[email-masked]>
Date: Tue, 13 Aug 2013 16:01:39 -0700

That opens another can of worms, which is Unicode string comparison.

If not ASCII, I'd rather have header values be arbitrary octet strings and
for string equality to be byte-wise; then you can put UTF-8 in there if you
wish.

On Tue, Aug 13, 2013 at 3:39 PM, James M Snell <[email-masked]> wrote:

> compression

---

From: Roberto Peon <[email-masked]>
Date: Tue, 13 Aug 2013 19:31:09 -0700

I thought we agreed about this already at the interim?

Header keys are a subset of us-ascii as defined in the 1.1 spec, and are
case insensitive as defined in the 1.1 spec.

The compressor spec will/would then include a MUST transform keys to
lowercase.

-=R

---

From: James M Snell <[email-masked]>
Date: Tue, 13 Aug 2013 19:34:18 -0700

It may have been agreed to at the interim, but the spec has not yet
been updated to reflect that. The header compression draft, for
instance, only says that header field names are encoded as "literal
strings" which are defined as being UTF-8.  The drafts need to be
fixed to reflect what has been decided.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 14 Aug 2013 07:15:30 +0200

>     header-name = "!" / "#" / "$" / "%" / "&" / "'" /

Agreed.

Willy

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 14 Aug 2013 22:23:40 +0900

For header names, I think ASCII is enough. Please note that in e-mail, 
headers are ASCII but can contain virtually any kind of symbol character 
(see http://tools.ietf.org/html/rfc5322#section-3.6.8). But as this is 
more limited in HTTP 1.1, this is fine.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 16 Aug 2013 21:28:51 +0900

> for string equality to be byte-wise; then you can put UTF-8 in there if you

The semantics of comparisons of header field values will depend on the 
specific header field. A field with a date is compared differently from 
a field with e.g. a domain name or an URI or whatever else.

And using UTF-8 doesn't exclude using bitwise equality, if that makes 
sense for some header.

But saying data is binary and then having each header have to define how 
to use charaters on top of that is really a bad idea. Just make it 
UTF-8, and be done with it.

Regards,   Martin.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 16 Aug 2013 08:44:46 -0700

The compressor could be aware or the content of fields, but right now, with
the sole exception of the cookie field, it has no need to do so, so why
impose what is unnecessary, and potentially harmful?

Why should the compressor be required to do utf-8 normalization?

Such requirements belong at the http/1 spec level, or perhaps in the http2
spec, but not in the compressor.

The keys should be ASCII, and the values bytes. This allows for utf-8 to be
compressed as well, though compression efficiency will go down if the input
isn't normalized AND if there is a lot of variation in the encodings of a
particular string.

-=R
On Aug 16, 2013 5:31 AM, Martin J. Dürst <[email-masked]> wrote:

> specific header field. A field with a date is compared differently from a
> field with e.g. a domain name or an URI or whatever else.
>
> to use charaters on top of that is really a bad idea. Just make it UTF-8,
> and be done with it.
>
> Regards,   Martin.
>
>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 16 Aug 2013 09:19:00 -0700

On 16 August 2013 08:44, Roberto Peon <[email-masked]> wrote:
> The keys should be ASCII, and the values bytes.

That's a fairly narrow view.  If the values were (for example) ASCII,
then you'd have an opportunity to compress better.  At worst, you can
wipe the high order bit from every octet.

At some level you are going to need to either make assumptions about
the properties of values, or rely on specific knowledge about them if
you are going to compress effectively.  Even if it were the case that
the bytes were UTF-8, you could still make some gains over pure bytes
(even just by exploiting the fact that certain byte sequences are not
possible in UTF-8).

---

From: Roberto Peon <[email-masked]>
Date: Fri, 16 Aug 2013 09:29:57 -0700

I view it as liberating-- as the compressor is now freed from worrying
about normalization, etc. which, if done, should be done at a higher layer.

There is currently exactly one field that the compressor makes assumptions
about and we could change that by requiring that the HTTP-layer do the
transformation of cookie into cookie-crumbs instead of having the
compressor do it. The compressor knows zero about anything else,
semantically right now.

The huffman encoder that we had and will likely add back worked on bytes.
It mostly encountered ASCII, and thus the frequency table was skewed to
compress ASCII better than other things, but it could still handle UTF-8,
raw binary, whatever.

I could certainly see an eventual future where some values are just raw
binary.
Sure, the huffman-based encoder would not compress that very well, but that
is OK-- the binary rep should already be fairly small in comparison to the
B64 encoding we do today (I'd rather have the data remain the same size
than getting a 30% decrease after a 4X expansion, which is what would
happen today...), and an escape valve of not having to use the huffman
encoding has always been the plan.

We could still allow for compressors to do things with semantic knowledge,
but there is no need to *require* it by declaring the type of all values a
prior.
Simply require that any transformation the compressor does must not change
the semantic meaning of the value. Problem solved, I think.

-=R

-=R

---

From: James M Snell <[email-masked]>
Date: Fri, 16 Aug 2013 09:49:15 -0700

On Fri, Aug 16, 2013 at 9:29 AM, Roberto Peon <[email-masked]> wrote:
> I view it as liberating-- as the compressor is now freed from worrying about
> normalization, etc. which, if done, should be done at a higher layer.
>

FWIW, I don't believe anyone had said anything about normalization...
valid UTF-8 octets, yes, but not normalization. The compression
mechanism is really not affected by whether or not we say UTF-8
here...

- James

> transformation of cookie into cookie-crumbs instead of having the compressor
> do it. The compressor knows zero about anything else, semantically right
> now.
>
> The huffman encoder that we had and will likely add back worked on bytes. It
> mostly encountered ASCII, and thus the frequency table was skewed to
> B64 encoding we do today (I'd rather have the data remain the same size than
> getting a 30% decrease after a 4X expansion, which is what would happen
> today...), and an escape valve of not having to use the huffman encoding has
> always been the plan.
>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 16 Aug 2013 09:57:38 -0700

In addition to compressing the bytestrings, the compressor will have to
validate utf-8. Nearly the same complexity as normalization (which was
proposed earlier) to me-- I now get to scan things yet another time,
increasing CPU utilization.. for what? Basically nothing in return if the
upper-level doesn't care about it.

If the upper-level cares about it, then it should be a prereq of feeding
something into the compressor. If not, then it shouldn't be. Either way,
these concerns belong outside the compressor.

-=R

On Fri, Aug 16, 2013 at 9:49 AM, James M Snell <[email-masked]> wrote:

> about
> compressor
> than
> > On Fri, Aug 16, 2013 at 9:19 AM, Martin Thomson <
> [email-masked]>
> > wrote:
> >>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sat, 17 Aug 2013 22:12:17 +0900

On 2013/08/17 1:49, James M Snell wrote:
> On Fri, Aug 16, 2013 at 9:29 AM, Roberto Peon<[email-masked]>  wrote:

Yes indeed, this should be clearly layered. Compression can be done 
binary (making sure we have a compression method that works well for 
UTF-8, and even better for ASCII).

Checking for valid UTF-8 can be done at the end. And please note that 
checking for valid UTF-8 can be really fast, I'd guess something like 
two to three machine instructions per byte. But still we don't need to 
do it when a proxy just moves the headers from left to right.

Unicode Normalization then is again one level higher, and depends on the 
header in question, as I already wrote.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sat, 17 Aug 2013 22:17:50 +0900

> proposed earlier) to me-- I now get to scan things yet another time,

Sorry, but that's not true. You can get close to it being true for 
ASCII-only data, but that's about it. Checking for UTF-8 validity is a 
very small state machine (around 10 states) looking at one byte a time, 
and it can only succeed or fail. Normalization needs lots of data (a few 
10K bytes) for lookup, may need a buffer of indefinite length, may 
lengthen or shorten the data, and so on.

> these concerns belong outside the compressor.

I agree that this should be outside the compressor.

Regards,    Martin.

---

From: James M Snell <[email-masked]>
Date: Sat, 17 Aug 2013 06:23:07 -0700

On Aug 17, 2013 6:18 AM, Martin J. Dürst <[email-masked]> wrote:
[Snip]
> I agree that this should be outside the compressor.

Agreed... But that doesn't mean we can't put minimal type info in the
compressed encoding.

- james