Subject: Re: h2 header field names

From: Mark Nottingham <[email-masked]>
Date: Wed, 3 Sep 2014 09:45:10 +0300

I’ve raised this as editorial - see <https://github.com/http2/http2-spec/issues/604>.

Thanks,

On 1 Sep 2014, at 5:37 am, Roy T. Fielding <[email-masked]> wrote:

> This comment is in reference to section 10.3 of
> 
> http://tools.ietf.org/id/draft-ietf-httpbis-http2-14.txt
> 
>> 10.3.  Intermediary Encapsulation Attacks
>> 
>>   HTTP/2 header field names and values are encoded as sequences of
>>   octets with a length prefix.  This enables HTTP/2 to carry any string
>>   of octets as the name or value of a header field.  An intermediary
>>   that translates HTTP/2 requests or responses into HTTP/1.1 directly
>>   could permit the creation of corrupted HTTP/1.1 messages.  An
>>   attacker might exploit this behavior to cause the intermediary to
>>   create HTTP/1.1 messages with illegal header fields, extra header
>>   fields, or even new messages that are entirely falsified.
>> 
>>   Header field names or values that contain characters not permitted by
>>   HTTP/1.1, including carriage return (ASCII 0xd) or line feed (ASCII
>>   0xa) MUST NOT be translated verbatim by an intermediary, as
>>   stipulated in [RFC7230], Section 3.2.4.
>> 
>>   Translation from HTTP/1.x to HTTP/2 does not produce the same
>>   opportunity to an attacker.  Intermediaries that perform translation
>>   to HTTP/2 MUST remove any instances of the "obs-fold" production from
>>   header field values.
> 
> This section is incorrectly stated.  While it is worthwhile to warn
> implementors of the difference between header field name syntax,
> the actual sending of an HTTP/1.1 message is governed by HTTP/1.1
> (not this spec). The second paragraph is already forbidden by RFC7230,
> as is the third.
> 
> What this should say:
> 
> HTTP/2 allows header field names that are not valid header fields in
> the Internet Message Syntax used by HTTP/1.1 and cannot be registered
> as such with IANA.  An intermediary that is attempting to translate an
> HTTP/2 request or response containing such an invalid field name into
> an HTTP/1.1 message ought to ...
> 
>   {discard the field because only someone who wanted it to be discarded
>    would be stupid enough to use an invalid field-name} |
>   {do some magic undefined thing that has never been defined because
>    it is a stupid idea for any Internet protocol to suggest a change
>    to field name syntax} |
>   {send a STREAM_ERROR because they deserve it}.
> 
> Alternatively, restrict the HTTP/2 header name syntax (not including
> pseudo-headers) to the same syntax as HTTP/1.1, because that is the
> sane thing to do.
> 
> HTTP/2 allows header field values that are not valid header field values
> in the Internet Message Syntax used by HTTP/1.1.  An intermediary that
> is attempting to translate an HTTP/2 request or response containing such
> an invalid field name into an HTTP/1.1 message MUST perform the following
> encoding of the octets that are not allowed in field-content: {pick one}.
> 
> 
> Cheers,
> 
> Roy T. Fielding                     <http://roy.gbiv.com/>
> Senior Principal Scientist, Adobe   <http://www.adobe.com/>
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Wed, 3 Sep 2014 12:36:12 -0700

I think that this might be more substantial, so I'm going to confirm
my choices here before proceeding.

> an HTTP/1.1 message ought to ...

I'm currently thinking that the correct choice is to treat the request
or response as malformed
(http://http2.github.io/http2-spec/#malformed), which would force a
request or response to be treated as an error and dropped or ignored.

> encoding of the octets that are not allowed in field-content: {pick one}.

I think that we can be stricter here than that.  The encapsulation
attack we're talking about can be more effectively avoided if we treat
the request or response as malformed (as above) if it contains octets
that we don't want.

The problem is that we're basically committed to taking pretty much
any old crap in values, including not just VCHAR, SP and TAB, but also
obs-text and even some of the ASCII control characters (BEL anyone?),
which HTTP/1.1 doesn't expressly permit, but we've evidence to suggest
that they are used, even widely.

This is why I think that the requirement to remove CR/LF from values
is still valuable.  HTTP/1.1 doesn't say anything about these because
they either form part of the parsing logic or obs-fold.  Roy's text
would force them to be encoded, which we could do by relying on the
translation to a single SP character mandated by RFC 7230.  That is
almost good enough, but we've generally avoided hiding errors that
way.  I'd rather be consistent with a zero tolerance error handling
philosophy and say that the request is malformed.

Here's what I have:

   The HTTP/2 header field encoding allows the expression of names that
   are not valid field names in the Internet Message Syntax used by
   HTTP/1.1.  Requests or responses containing invalid header field
   names MUST be treated as malformed (Section 8.1.2.6).  An
   intermediary therefore cannot translate an HTTP/2 request or response
   containing an invalid field name into an HTTP/1.1 message.

   Similarly, HTTP/2 allows header field values that are not valid.
   While most of the values that can be encoded will not alter header
   field parsing, carriage return (CR, ASCII 0xd), line feed (LF, ASCII
   0xa), and the zero character (NUL, ASCII 0x0) MUST NOT be translated
   verbatim by an intermediary.  An attacker might exploit direct
   translation of these octets to cause an intermediary to create
   HTTP/1.1 messages with illegal header fields, extra header fields, or
   wholly falsified messages.  Any request or response that contains a
   CR, LF or NUL character in a header field value MUST be treated as
   malformed (Section 8.1.2.6).

   Characters in header field values that are not valid according to the
   "field-content" rule (see [RFC7230], Section 3.2), SHOULD be percent-
   encoded before translated into HTTP/1.1 header field values.

I hate using SHOULD, but it seems like if I made that a MUST, it would
be immediately ignored.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Sep 2014 20:07:08 +0000

--------
In message <[email-masked]>

>that they are used, even widely.

This is one of the "default" decisions I really don't understand.

Just because somebody made a stupid programming mistake (the
most common case I've seen is \0 before or after CRNL on headerlines)
doesn't mean that we should go out of our way to propagate
their mistake through the web for all possible future.

There is a valid architectual question: are headers ASCII or UTF-8?

But the lack of decision on that doesn't mean that we have to
let through BEL, BS and ESC while we make up our mind.

And why don't we just make the architectural decision in the
first place ?

	In HTTP/2 header names are [_A-Z0-9-]* - this ensures that
	everybody has a chance to figure out what the header
	may or may not do, but the header value can be UTF-8 so we don't
	have to encode everything into BASE64 all the time.

	If Your HTTP/1 application goes beyond that ?  Stay off HTTP/2

Now we've put a stake in the ground which web-app designers can
navigate after, rather than the current "maybe or maybe not..."
confusion.

In particular I find the "all HTTP/1, valid or not, must tunnel
through HTTP/2" position deeply confusing, when it comes from the
same people who say they are only going to allow HTTP/2 on TLS,
since most of the crap is on the plain-text traffic.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Sep 2014 09:18:31 +0300

> And why don't we just make the architectural decision in the

The time for doing that was in 2616bis. We talked about it a lot, and decided we couldn’t retroactively change the allowed values — and that doing so might not be a good idea anyway.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 4 Sep 2014 08:37:02 +0200

> request or response to be treated as an error and dropped or ignored.

I agree. In fact I'm thinking that if we slice the gatewaying process into
multiple stages, we'd see :
  - HTTP/2 demux
  - HPACK decoding
  - HTTP/1 message emission
  - HTTP/1 parsing

The last step would result in a 400/badreq or whatever variations implementors
decide upon. If we simplify the stages, we can imagine a number of things
including closing the stream during decoding if the decoding stage is
dedicated to 2->1 conversion only here.

So I think that indeed we must mandate that 2->1 gateways actively reject
messages that cannot be emitted over HTTP/1 according to the spec.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Sep 2014 08:53:16 +0200

On 2014-09-03 21:36, Martin Thomson wrote:
> ...
> ...

Wait! Percent-Encoded?

If we can't transport the field value, we really shouldn't alter it in 
a way the recipient isn't going to understand (reminder: percent 
encoding isn't really a feature of field values!).

SO I think it would be much better to reject these, optimally on the 
HTTP/2 layer as well.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Sep 2014 10:27:56 +0300

On 4 Sep 2014, at 9:53 am, Julian Reschke <[email-masked]> wrote:

> If we can't transport the field value, we really shouldn't alter it in a way the recipient isn't going to understand (reminder: percent encoding isn't really a feature of field values!).
> 
> SO I think it would be much better to reject these, optimally on the HTTP/2 layer as well.

+1 on “What… what?”

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 08:21:43 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>The time for doing that was in 2616bis. We talked about it a lot, and 
>decided we couldn't retroactively change the allowed values - and 
>that doing so might not be a good idea anyway.

...and that's water under the bridge now.

And if this had been HTTP/1.2 I would also agree that all HTTP/1.1
traffic must be able to pass through.

But we're doing HTTP/2.0.

The general interpretation of a new major version number bump in
the IT industry is "You'd better read the manual to see what's
changed".

People will be expecting some things not to work, and they'll expect
the changes to be for the better.

Firming up things like charset for headers would therefore make
people think more and better of HTTP/2.0 than if we just grandfather
all the HTTP/1.1 warts in and add new ones.

What actual trouble do you expect to see if we firm up the definition
of header character sets in HTTP/2.0 ?

HTTP/1 through HTTP/2 tunneling is going to be mostly between a
proxy/balancer and web-severes, and if that doesn't work because
the web-service emits NUL in headers, they'll figure out and
postpone HTTP/2 deployment until they fix their pointer-errors.

I also don't understand the arbitrainess of these decisions.

We have jettisoned retaining HTTP/1.1 chunking when tunneling through
HTTP/2.0, but we have to retain their NUL characters in headers ?

Far more applications rely on chunk-boundaries than on NUL headers,
so what exactly is our compatibility criteria here ?

I really don't get it...

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Sep 2014 11:26:46 +0300

I was responding to your question about the “architectural decision” of a character set.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 08:30:53 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>I was responding to your question about the 'architectural decision'
>of a character set.

Yes, and I'm asking why that that particular decision is out of
bounds for HTTP/2.0, when we can kill other mis-uses of HTTP/1.1
without trouble ?

How is restricting the charset in a way which is compatible with
what the HTTP/1.1 spec says different from throwing out chunked
boundaries in a way which is compatible with what HTTP/1.1 says ?

Both a architectural decisions which means that some tiny subset of
HTTP/1.1 traffic won't tunnel through HTTP/2.0.

But one decision could be made with no trouble, the other is out of bounds ?

Why ?

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Sep 2014 14:42:23 +0300

Because chunk boundaries have no semantics in HTTP/2, whilst headers do.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 12:02:15 +0000

--------
In message <[email-masked]>, Mark Nottingham wri

>Because chunk boundaries have no semantics in HTTP/2, whilst headers do. 

The argument for not tightening up header definitionis that "weird
stuff" have semantics in practical HTTP/1.1 so we can't do away with
it, in case we need to tunnel it.

Chunk boundaries also have semantics in practical HTTP/1.1 and we
just did away with them, even for tunneled HTTP/1.1.

What is the difference ?

There is a valid architectural point to decide ASCII vs. UTF-8, and
we can and should debate that.

But I havn't seen *anybody* say that need to be able to put NUL,
STX or ANSI-escape sequences in HTTTP headers, so I don't understand
why can't we outlaw them in HTTP/2.0, even if we don't settle the
ASCII/UTF-8 question yet ?

IMO nothing *in* the headers should contain 0x00-0x1f or 0x7f.

What makes that decision impossible ?

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Sep 2014 15:07:40 +0300

> What makes that decision impossible ?

Didn’t say it was. What do people think?

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 4 Sep 2014 23:11:06 +1000

On 4 September 2014 22:07, Mark Nottingham <[email-masked]> wrote:

> What do people think?

I can't speak authoritatively on what the impact would be, but I do think
that reducing ambiguity is a good thing. More over, reducing ambiguity
about charsets is of significant benefit for implementations in terms of
code simplicity, security vulnerabilities and efficiency.

I think 2.0 is definitely an opportunity to fix this and I'd be happy to
take some heat from the few users likely to be affected.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Sep 2014 15:18:56 +0200

> Didn’t say it was. What do people think?

They are disallowed in HTTP/1.1 (MUST NOT send...), so I think it would 
be good to disallow them in HTTP/2 (hard fail?) as well.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 4 Sep 2014 15:23:04 +0200

On Thu, Sep 04, 2014 at 03:18:56PM +0200, Julian Reschke wrote:
> >Didn?t say it was. What do people think?
> 

+1

Willy

---

From: "Martin Nilsson" <[email-masked]>
Date: Thu, 04 Sep 2014 15:25:20 +0200

> What makes that decision impossible ?

I want to be able to send binary header values as soon as possible, to get  
rid of all layers of base-n encoding. The names makes perfect sense as  
ASCII only.

/Martin Nilsson

-- 
Using Opera's mail client: http://www.opera.com/mail/

---

From: Michael Sweet <[email-masked]>
Date: Thu, 04 Sep 2014 09:38:12 -0400

Martin,

Seems like binary header values would require some indication in the HPACK encoding that those headers contain binary data, so that 2.0-to-1.1 proxy can convert the binary data to Base64.  And *that* seems fraught with interoperability issues since then a 1.1-to-2.0 proxy will either a) need to know which headers want binary encoding or b) make some guess based on the value being all Base64.

Regardless, what are you trying to accomplish with binary header values?

On Sep 4, 2014, at 9:25 AM, Martin Nilsson <[email-masked]> wrote:

> On Thu, 04 Sep 2014 14:02:15 +0200, Poul-Henning Kamp <[email-masked]> wrote:
> 
>> 
> I want to be able to send binary header values as soon as possible, to get rid of all layers of base-n encoding. The names makes perfect sense as ASCII only.
> 
> /Martin Nilsson
> 
> -- 

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 05 Sep 2014 02:28:20 +1200

On 5/09/2014 1:23 a.m., Willy Tarreau wrote:
>>>> But I havn't seen *anybody* say that need to be able to put
>>>> NUL, STX or ANSI-escape sequences in HTTTP headers, so I
>>>> don't understand why can't we outlaw them in HTTP/2.0, even
>>>> if we don't settle the ASCII/UTF-8 question yet ?
>>>> 
>> They are disallowed in HTTP/1.1 (MUST NOT send...), so I think it
>> would be good to disallow them in HTTP/2 (hard fail?) as well.
> 
> +1

+1

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 05 Sep 2014 02:40:53 +1200

On 5/09/2014 1:38 a.m., Michael Sweet wrote:
> Martin,
> 
> Seems like binary header values would require some indication in
> the HPACK encoding that those headers contain binary data, so that 
> 2.0-to-1.1 proxy can convert the binary data to Base64.  And
> *that* seems fraught with interoperability issues since then a
> 1.1-to-2.0 proxy will either a) need to know which headers want
> binary encoding or b) make some guess based on the value being all
> Base64.

I dont see why the gateway would need to know which headers want
binary encoding in particular. To trasmit over 1.1 the headers already
"want" a base-64 encoded form which should be able to relay over 2.0
without decoding.

* a 2.0->1.1 gateway gets the binary indicator.

* a 1.1->2.0 gateway gets base-64 and sends as non-binary.

* a 2.0->1.1->2.0 chain the second 2.0 gateway gets base-64 and sends
either non-binary storing base-64 or binary forms.

The end recipient "just" needs to handle:
 * optionally base-64 encoded in non-binary form for both 2.0 or 1.1, or
 * UTF-8 in binary form for 2.0 traffic.

If the header sent with non-ASCII does not define a suitable base-64
representation for 2.0->1.1 gateways to emit then the content is
invalid 1.1 and invalid message rejection should not be a problem.

> Regardless, what are you trying to accomplish with binary header 

Good question. In a nutshell, the gain is simplicity for
implementations no longer having to include base-64 encoder/decoders
or spend CPU cycles doing the coding.

Amos

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Sep 2014 17:03:02 +0200

> or spend CPU cycles doing the coding.

I think the question is: why do you need binary data in header field 
values in the first place?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 15:17:19 +0000

--------

>IMO nothing *in* the headers should contain 0x00-0x1f or 0x7f.

Actually... 0x7f is valid UTF-8 isn't it ?

But 0x00-0x1f should be banned outright.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 15:31:50 +0000

--------

>rid of all layers of base-n encoding. The names makes perfect sense as  

I'm not at all warm on unrestricted binary, it invites too much trouble.

I'm also not convinced there is a real upside:  Binary data in this
context usually means cryptographic output which again means
uncompressable.

Also the current HPACK huffman code doesn't do too badly on base64.

I did some experiments with an "algorithmic sub-mode" to HPACK which
spotted BASE64 and encoded it as a binary length + string, but the gains
over the huffman were very slight and certainly not worth the complexity.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Sep 2014 17:42:28 +0200

> Actually... 0x7f is valid UTF-8 isn't it ?

It's the UTF-8 encoding of DEL, so no, we shouldn't allow it in a header 
field.
Best regards, Julian

---

From: Jason Greene <[email-masked]>
Date: Thu, 4 Sep 2014 10:43:24 -0500

On Sep 4, 2014, at 10:03 AM, Julian Reschke <[email-masked]> wrote:

> I think the question is: why do you need binary data in header field values in the first place?

Security tokens, cryptographic signatures, complex encoded types (e.g. ASN.1), specialty compressed values, precise serialization of IEEE754 floating point values, middleware intermediary tracking/routing information, unix timestamps etc

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: James M Snell <[email-masked]>
Date: Thu, 4 Sep 2014 08:46:15 -0700

> I think the question is: why do you need binary data in header field values

Just some background... This was been debated back and forth quite a
while ago. In a range of experiments I conducted a while back, I found
that by adopting efficient binary encodings for certain types of
header field values (dates, numbers, cookies, etc), we could actually
realize a significant reduction in bytes-on-the-wire. Dates, for
instance, could be encoded using as few as 4 or 5 bytes rather than
the current average of 29. At the time I was conducting those
experiments, I posted several iterations of a "binary optimized header
encoding" that defined one approach to handling the binary encoding in
a more-or-less backwards compatible way.

At the same time, the WG was starting to consider HPACK. At one of the
face to face get togethers (I forget which one), a decision was made
that HPACK could be simplified if it simply ignored the actual
encoding of header field values and instead just treated them as a
sequence of opaque bytes. It was also decided that efficient binary
encodings like what I had proposed would not be considered. This meant
that the existing header field value definitions remain untouched, but
that -- as far as the http/2 framing and header processing is
concerned -- header field values are just bags of bits, and it's up to
the application layer to figure it out from there.  The side effect of
this is that it's perfectly legal for me, at the framing layer, to
send any random sequence of bytes as the value of any header field. It
would be at least theoretically possible for a malicious actor to take
advantage of this fact by attempting to send an invalid sequence of
bytes to an intermediary/server that performs h2->h1 translation. That
translation code would need to be written defensively to ensure that
invalid or dangerous sequences are rejected.

While I'm certainly not going to argue it any further, I believe the
approach I proposed in the bohe drafts provided the most sensible and
measured approach as far as allowing binary header field values is
concerned -- particularly in that it did not attempt to do too much
and provided a clear approach to translating back to h1. I'm certainly
not a fan of what's currently documented in the h2 spec, but the time
for that debate has long since passed.

- James

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 15:47:57 +0000

--------
>It's the UTF-8 encoding of DEL, so no, we shouldn't allow it in a header 

Makes sense.

---

From: Michael Sweet <[email-masked]>
Date: Thu, 04 Sep 2014 11:51:49 -0400

Amos,

> or spend CPU cycles doing the coding.

OK, to restate my question: why do you feel that passing binary data is the hammer you need to use for your particular use case (whatever that is)?  Are you really wanting to pass arbitrary BLOBs, or do you just want to be able to pass UTF-8 or other non-ASCII text?

(not touching the CPU argument...)

---

From: Julian Reschke <[email-masked]>
Date: Thu, 04 Sep 2014 17:57:06 +0200

> Security tokens, cryptographic signatures, complex encoded types (e.g. ASN.1), specialty compressed values, precise serialization of IEEE754 floating point values, middleware intermediary tracking/…

And all of these can be converted in some sensible way to ASCII, right?

The main reason why I'm concerned about arbitrary binary data is that 
all of the HTTP APIs I'm aware essentially work on character sequences, 
not octet sequences (in header fields).

The lack of a standard encoding in HTTP/1.1 is already a problem; but 
having a mix of both character and octet based fields with no generic 
and reliable way to distinguish these concerns me a lot.

VCHAR works reliably, and gives you 94 characters, so you might be able 
to squeeze a few more bits into these than using BASE64. But is it worth 
the trouble?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 17:07:30 +0000

--------

>not octet sequences (in header fields).

That is my concern too:  Most code handling HTTP today don't expect
to see embedded NUL's, and a lot of it will happily send through
"poison" characters like ESC.

>The lack of a standard encoding in HTTP/1.1 is already a problem;

But we could apply gentle persuasion here:  If we make HPACK work
really well with baser64 (it does, but could be better) a lot of
people will take the hint.  Not everybody, but a lot of them.

---

From: Kinkie <[email-masked]>
Date: Thu, 4 Sep 2014 19:22:27 +0200

> people will take the hint.  Not everybody, but a lot of them.

You are an optimist, aren't you?
People will happily choose whatever is easiest to implement and works,
especially in enterprise "intranet" environments.

So either allowed character sets are made very strict, or it's just as well
that it is not considered it a protocol issue,  and is better managed at
the application framework level,  where the protocol should stay as much as
possible out of the way. I'm convinced halfway measures will not achieve
anything worthy.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Sep 2014 19:34:50 +0000

--------
In message <CA+Y8hcNpEwQr5A1d=TyBuwTetqWevo=[email-masked]>

>You are an optimist, aren't you?

Not really.

But I am a firm beliver of supplying good tools to qualified people,
knowing they will get the most out of them, whereas no matter what
you do, the unqualified will make a mess of things.

I used to say that I'd be happy if FreeBSD had 1% market share,
provided it was the 1% smartest users.

So I think we should make it a goal, duely prioritized, that HPACK
work well with base64 (it already does, but mostly by accident)
and advertise this decision.

The competent will take the clue and use base64, and that's enough
for me.

---

From: "Martin Nilsson" <[email-masked]>
Date: Thu, 04 Sep 2014 22:05:24 +0200

On Thu, 04 Sep 2014 17:57:06 +0200, Julian Reschke <[email-masked]>  
wrote:

>>
>> Security tokens, cryptographic signatures, complex encoded types (e.g.  
>> ASN.1), specialty compressed values, precise serialization of IEEE754  
>> floating point values, middleware intermediary tracking/routing  
>> information, unix timestamps etc
>

It's just such a big waste to build this nice, 8-bit clean tunnel and then  
intentionally ding it up because there are APIs for previous versions of  
the protocol that would otherwise have to be extended to allow for future  
extensions of the protocol.

I'm not proposing that the currently defined HTTP headers should allow  
binary content, although I am very sympathetic to more efficient value  
encodings of the type James Snell have suggested (and it is still possible  
to add them later as a processing step before and after HPACK, when  
negotiated). I do want my new, binary headers, when I invent them, to not  
be mandatory thrown away by intermediaries.

/Martin Nilsson

---

From: "Martin Nilsson" <[email-masked]>
Date: Thu, 04 Sep 2014 22:06:40 +0200

> and advertise this decision.

It's not an accident. It was trained with real headers, and a substantial  
part of headers are base64-encoded binary data.

/Martin Nilsson

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 5 Sep 2014 08:55:12 +1000

On 4 September 2014 23:25, Martin Nilsson <[email-masked]> wrote:

> I want to be able to send binary header values as soon as possible

I'd like to remind everybody that headers are not flow controlled, cannot
be  segmented to allow multiplexing (can only be segmented for streaming)
and can be very large.

To allow binary headers is going to invite significant protocol misuse.
Why send your data in a message body if you can put it cheaply into the
header and not have to contend with sharing the connection with any other
requests/streams etc.!

cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Sep 2014 07:08:22 +0000

--------
In message <[email-masked]>, "Martin Nilsson" 
writes:

>It's just such a big waste to build this nice, 8-bit clean tunnel [...]

I can certainly understand and share that frustration.

But there are two ways to exploit the 8-bit tunnel:  End-to-end or
using path compression.

HPACK does the later with pretty good job on the latter as far as
base64 is concerned, so we wouldn't gain much in efficiency for all
the pain it would be to get PHP programmers to understand binary
headers.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Sep 2014 07:52:56 +0000

--------
In message <[email-masked]>, "Martin Nilsson" 
>part of headers are base64-encoded binary data.

Correct, but precisly what ratio of base64 that training data
contains relative to other traffic is unknown, the training set was
a very random (in the statistical sense) sample of all HTTP traffic.

Also I don't recall ever seeing the actual process used to derive
the huffman table documented, and suspect that it should be documented
and reviewed before we finalize the huffman table.

For instance I'd like to know if "Transfer-Encoding: chunked",
"Connection: {closed|keep-alive}" and similar HTTP/1-only
headers were eliminated from the set ?

If, as it looks to me, more and more traffic gets infected by cookies
with crypto-blobs, artificially boosting the base64 charset in the
huffman table could make sense:  It would amount to a gradual
improvement of HPACK as usage changes over time.

Right now the huffman codes for the base64 set has an average length
of 6.92 bits (min=4, max=10).

This means that if we posit a hypothetical encoding of 2 bytes overhead
and 8-bit binary encoding, it would beat our current huffman table
once you have more than 104 bits in your binary blob.

Improving the average to 6.75 would move the cut-over to 128 bits

Improving the average to 6.38 would move the cut-over to 256 bits

But in both cases it would come at an expense of compression
efficiency of other headers.

If on the other hand we want to discourage huge cookies, going
binary would be the worst thing we could do.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Sep 2014 09:10:22 -0700

> and reviewed before we finalize the huffman table.

It should be in the minutes of the NY interim.  Chrome deployed an
experiment that performed HPACK compression on all requests and
responses seen by their browser over a significant data set.  The
resulting counts for each octet were collected.

That was before removing the reference set removal so results will be
dominated slightly more by header fields that change frequently and
there will be instances of '\0' (used at that time to ensure ordering
of values was maintained) that are no longer valid.  I can speculate
that a new experiment along the same lines would shift the bias
slightly in favour of base64 characters, but only a similar experiment
will decide that.

(I'll also note that the results fortuitously produce sequences that
don't have unique mappings from length to character, which reduces
information leakage from the resulting compressed length. The research
Google sponsored indicates that this is minimal.)

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Sep 2014 16:25:15 +0000

--------
In message <CABkgnnVErb7s=[email-masked]>
, Martin Thomson writes:
>It should be in the minutes of the NY interim.

Can I propose that the origin of the data and the procedure used
to derive the table be added to the draft as an appendix.

These are rather paranoid times and being open about it costs us
nothing.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Sep 2014 16:49:23 +0000

--------
>It should be in the minutes of the NY interim.

I can't seem to find it there, got a link ?

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Sep 2014 09:48:48 -0700

> to derive the table be added to the draft as an appendix.

You certainly can propose pretty much anything you like.  I personally
don't see the point.

I'd be interested in learning why you might think that tampering would
produce a benefit for anyone.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Sep 2014 09:55:05 -0700

> I can't seem to find it there, got a link ?

I guess it missed the minutes, but it hit the mailing list:

http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1196.html

Includes the source code and results.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Sep 2014 17:05:43 +0000

--------
In message <CABkgnnVEyR2BV7gf_BANXN=cF3UbdV=[email-masked]>
>produce a benefit for anyone.

There are no accusations of tampering anywhere I can see, and it's
not about what I think.

This table is a key element in HPACK and therefore their provenance
should be documented, if nothing else for historys sake.

I found this email:

Which seems to be for a check-run to see if the current able should
be updated ?

It doesn't answer my question if HTTP/1 headers which cannot ever
be sent through HTTP/2 were filtered out before estimating the PDFs.

I'm thinking specifically of:

	"HTTP/1.1" (HTTP-Version in the status line)
	Reason-Phrases in the status line
	Connection: close
	Connection: keepalive
	Transfer-Encoding: chunked

None of these will pass through HTTP/2.0 and should therefore not
be used as training material for HPACK

Does anybody know if these headers were removed from the data-set ?

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Sep 2014 10:18:27 -0700

> be sent through HTTP/2 were filtered out before estimating the PDFs.

https://code.google.com/p/chromium/codesearch#chromium/src/net/spdy/spdy_http_utils.cc&sq=package:chromium&rcl=1409711490&l=109

...indicates that those were removed.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Sep 2014 18:10:20 +0000

--------
In message <[email-masked]>
>...indicates that those were removed.

How is that code related to the data that went into the huffman table ?

Also: Entries from the static table should also be removed.

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 5 Sep 2014 21:46:04 +0300

> Improving the average to 6.38 would move the cut-over to 256 bits

One can do a bit better by optimizing the character set.

Over token characters, the optimal 64-chacter set is:
0123456789ABCDEFGHIJKLMNOPQRSTUVWYabcdefghijklmnopqrstuvwxyz-_.%

~6.33 b/c => ~1B/146.3b. (~6.36 and ~133.6 if % -> Z).

The theoretical limit for expansion is ~1B/457.9b.

Over quoted characters, the optimal set is:
0123456789ABCDEFGHIJKLMNOPQRSTUVWYabcdefghijklmnopqrstuvw /=-_.%

~6.28 b/c => ~1B/170.7b (~6.30 and ~161.7 if % -> z).

But due to having the fixed +5 bytes from token encoding, this
needs great amounts of data to break even.

The theoretical limit for expansion is ~1B/29530b.

-Ilari

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Sep 2014 12:00:52 -0700

You can look at the code yourself to answer those questions. I have
satisfied myself that Johnny did the right thing.
On Sep 5, 2014 11:10 AM, "Poul-Henning Kamp" <[email-masked]> wrote:

> --------
> In message <
> [email-masked]>
> , Martin Thomson writes:

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Sep 2014 19:37:33 +0000

--------

>0123456789ABCDEFGHIJKLMNOPQRSTUVWYabcdefghijklmnopqrstuvwxyz-_.%

I doubt we have the clout to redefine base64

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Sep 2014 14:02:18 -0700

> I doubt we have the clout to redefine base64

That's not the point.  The point is that if you are using private,
opaque tokens (cookies are one example), and you can choose the
mapping between binary and what gets put in the header field, then the
choice of optimal character set allows for higher efficiency.

I built a tool that does this for you based on arbitrary character
availability and the HPACK table
(https://github.com/martinthomson/bhpack).  The gains over base64 are
marginal enough that it's really worth using in practice.

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Fri, 5 Sep 2014 22:45:59 +0000

On 9/5/14, 2:10 PM, "Poul-Henning Kamp" <[email-masked]> wrote:

>--------
>In message 
><[email-masked]>
>, Martin Thomson writes:
>>https://code.google.com/p/chromium/codesearch#chromium/src/net/spdy/spdy_
>>http_utils.cc&sq=package:chromium&rcl=1409711490&l=109
>>

And all (remaining) header field names should be converted to lowercase.
This appears to have occurred in the code above.

Were all of these aspects documented anywhere or is this code the only
documentation we have?  I agree with PHK - we are likely to see folks
wishing to understand the process and I believe transparency and clear
documentation are the best answer.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Sep 2014 16:33:41 -0700

> documentation are the best answer.

In general, participation in the process achieves that goal.  There is
no obligation on us to document the reasoning behind any and all
decisions, only to capture those decisions.  Where we do provide
explanations, there can be a range of reasons.  Transparency alone
isn't enough of a reason.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 06 Sep 2014 06:26:57 +0000

--------
In message <[email-masked]>
, Martin Thomson writes:

>decisions, only to capture those decisions.

There may not be an obligation, but it's still a good idea.

---

From: "Martin Nilsson" <[email-masked]>
Date: Sat, 06 Sep 2014 11:06:36 +0200

> documentation are the best answer.

I think the HTTP/2 FAQ page would be the most appropriate place.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/