Subject: HPACK opcode bit patterns

From: Osama Mazahir <[email-masked]>
Date: Wed, 6 Aug 2014 00:13:04 +0000

1) 1xxxxxxx: "Indexed Header Field Representation"
2) 01xxxxxx: "Literal Header Field With Incremental Indexing"
3) 001xxxxx: "Maximum Header Table Size Change"
4) 0001xxxx: "Literal Header Field Never Indexed"
5) 0000xxxx: "Literal Header Field Without Indexing"

Do we really expect tablesize (3) to be more common than literal_without_indexing (5)?

I don't have a strong opinion to change the masks, but I am curious if the bit pattern selection was deliberate.

---

From: Alvaro Lopez Ortega <[email-masked]>
Date: Wed, 6 Aug 2014 12:32:31 +0200

> literal_without_indexing (5)?

Excellent point. Unless I'm missing something here I believe swapping the
third and fifth patterns would be the right thing to do.

I don’t have a strong opinion to change the masks, but I am curious if the
> bit pattern selection was deliberate.
>
>
I don't know. It doesn't look like it was deliberate. I'd say it's a
micro-optimization worth doing.

Regards,
Alvaro

---

From: David Krauss <[email-masked]>
Date: Wed, 6 Aug 2014 18:58:15 +0800

On 2014–08–06, at 8:13 AM, Osama Mazahir <[email-masked]> wrote:

> 5) 0000xxxx: "Literal Header Field Without Indexing”

> Do we really expect tablesize (3) to be more common than literal_without_indexing (5)?

The XXXX in 4-5 is only used to refer to something that’s already been indexed, without bumping it to the front of the index table. Therefore it is unusual for XXXX to be nonzero, regardless of how often cases 4-5 happen overall. (Also, note that the overhead of overflow is just one byte.)

> I don’t have a strong opinion to change the masks, but I am curious if the bit pattern selection was deliberate.

Also, an implementation can treat “without indexing” as “never indexed” and use a common code path, since the encodings are identical modulo a don’t-care bit. Wouldn’t be surprised if that was deliberate.

---

From: Michael Sweet <[email-masked]>
Date: Wed, 06 Aug 2014 08:12:09 -0400

David,

On Aug 6, 2014, at 6:58 AM, David Krauss <[email-masked]> wrote:
> ...

That's the approach I am taking, and frankly I wouldn't cry if we just did away with the semantic difference and just had "never indexed".  While I don't think the extra bit will make a huge difference in compression ratio, it would simplify all implementations and it puts the onus of efficient encoding on the sender (vs. intermediaries like proxies).

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Jason Greene <[email-masked]>
Date: Wed, 6 Aug 2014 09:30:20 -0500

> That's the approach I am taking, and frankly I wouldn't cry if we just did away with the semantic difference and just had "never indexed".  While I don't think the extra bit will make a huge differ…

If a proxy coalesces connections its encoding context can be radically different than the clients. A field sent by one client might already be in the state table from another client, and so an optimal hpack encoder would notice that and reuse that index, even if it was sent without indexing (no reason not to). So its important for a proxy to know the difference between a client which was simply optimizing its encoding context, and a header that should never be indexed.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Michael Sweet <[email-masked]>
Date: Wed, 06 Aug 2014 11:12:30 -0400

Jason,

> If a proxy coalesces connections its encoding context can be radically different than the clients. A field sent by one client might already be in the state table from another client, and so an opti…

With my current experience in implementing HTTP/2, I'm currently encoding the following headers using one of the non-indexed forms:

- Authorization (never indexed)
- Content-Length (without indexing)
- Content-Location (without indexing)
- Content-MD5 (without indexing)
- Content-Range (without indexing)
- Content-Version (without indexing)
- Date (without indexing)
- If-Modified-Since (without indexing)
- If-Unmodified-Since (without indexing)
- Last-Modified (without indexing)
- Link (without indexing)
- Location (without indexing)
- Range (without indexing)
- Retry-After (without indexing)

My general rule for "without indexing" is "anything that will likely change between requests/responses", which includes date/time fields, hashes, locations, ranges, byte counts, etc.  And of course authorization data is "never indexed".

While I agree that an intermediary like a proxy MIGHT be able to more efficiently re-encode a client's provided headers, and that it is already re-encoding when coalescing connections to a server, I am *not* convinced that it will save more over the wire than adding a bit for the index into the header table will.  And certainly if you want to make the "right" choices for that re-encoding you need to adjust those choices based on the trends you see in the requests you proxy and not on a simplistic "index everything I can" strategy since that WILL cause header table thrashing for all but the simplest clients.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 7 Aug 2014 08:22:14 +1000

On 7 August 2014 01:12, Michael Sweet <[email-masked]> wrote:

> With my current experience in implementing HTTP/2, I'm currently encoding
> the following headers using one of the non-indexed forms:
>
> My general rule for "without indexing" is "anything that will likely
> change between requests/responses", which includes date/time fields,
> hashes, locations, ranges, byte counts, etc.  And of course authorization
> data is "never indexed".
>

Jetty pretty much has the same.   However I'm thinking that we will index
Date as we often see batches of messages sent within the same second.

We are also currently never indexing set-cookie, but I admit to be confused
about the need to do this or if it can just be without index?

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Michael Sweet <[email-masked]>
Date: Wed, 06 Aug 2014 20:41:48 -0400

Greg,

On Aug 6, 2014, at 6:22 PM, Greg Wilkins <[email-masked]> wrote:
> ...
> Jetty pretty much has the same.   However I'm thinking that we will index Date as we often see batches of messages sent within the same second.

I thought the same, but when I index Date the header table gradually fills with Date headers and you eventually see (well, sort of) all of the headers "marching out" as new Date headers are added. That leads to re-indexing of the lost non-Date headers, etc.

> We are also currently never indexing set-cookie, but I admit to be confused about the need to do this or if it can just be without index?

You could decide based upon the use/requirement of https for the cookie - at least then you are protecting (never index) data that is normally encrypted vs. allowing indexing for cleartext...

---

From: Martin Thomson <[email-masked]>
Date: Wed, 6 Aug 2014 21:09:50 -0700

> about the need to do this or if it can just be without index?

Never index is a protection for downstream intermediaries that might
mix your set-cookie in with attacker-sourced guesses about the value
of the header field.  If you have lots of entropy in the field, it is
probably safe to allow it to be indexed.  Or, if the header field
doesn't contain any confidential information, you don't need it.  That
said, a) you can't really tell mechanically whether those conditions
are true, and b) repeating set-cookie with the same value is probably
more likely to be a bug than a real thing.

---

From: RUELLAN Herve <[email-masked]>
Date: Fri, 29 Aug 2014 16:19:03 +0000

> -----Original Message-----
> From: David Krauss [mailto:[email-masked]]
> Sent: mercredi 6 août 2014 12:58
> To: HTTP Working Group
> Subject: Re: HPACK opcode bit patterns
> 
> 
> 
> 	I don't have a strong opinion to change the masks, but I am curious if
> the bit pattern selection was deliberate.
> 
> 
> Also, an implementation can treat "without indexing" as "never indexed" and
> use a common code path, since the encodings are identical modulo a don't-
> care bit. Wouldn't be surprised if that was deliberate.

Yes it was deliberate: not only for sharing a common code path, but also from a security perspective, so that the two representations are as close as possible.

Hervé.