Subject: Call for Consensus: Remove "reference set" from HPACK (to address #552)

From: Mark Nottingham <[email-masked]>
Date: Sat, 12 Jul 2014 16:46:36 +1000

It seems like there's broad agreement to remove the reference set from HPACK. 

Jeff proposed this in <https://github.com/http2/http2-spec/issues/552> to address other issues (#549-551); while it may be that it does at least help mitigate them, we'll consider those issues separately. We can also consider further modifications (e.g., requiring :-headers to be first).

Does anyone disagree with taking this first step?

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 12 Jul 2014 09:57:51 +0200

> Does anyone disagree with taking this first step?

I don't have experience in HPACK implementation yet, but I feel that
it could simplify the sender's job. Also, my past experience working
on our proposal 2 years ago involved a very similar "common" header
set which was hard to manage for the sender and would need to be
reset in many cases and not provide that much a gain. I have the
non-scientific feeling that it will be the same here.

So that's a very small +1 from me if that counts.

Willy

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Sat, 12 Jul 2014 17:44:24 +0900

On Sat, Jul 12, 2014 at 3:46 PM, Mark Nottingham <[email-masked]> wrote:

> Jeff proposed this in <https://github.com/http2/http2-spec/issues/552> to
> address other issues (#549-551); while it may be that it does at least help
> mitigate them, we'll consider those issues separately. We can also consider
> further modifications (e.g., requiring :-headers to be first).
>
​Agreed.
Reference set is tricky, especially when eviction from header table is
involved.  I believe everyone will encounter this kind of bug.  While it is
still manageable and we can get it right, we have no data to show its
advantages, and it now becomes obstacle to the resolutions for the issues
we have, so remove it.

​Best regards,
Tatsuhiro Tsujikawa​

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 12 Jul 2014 09:11:32 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

>Jeff proposed this in <https://github.com/http2/http2-spec/issues/552> 
>to address other issues (#549-551); while it may be that it does at 
>least help mitigate them, we'll consider those issues separately. We can 
>also consider further modifications (e.g., requiring :-headers to be 
>first).
>

+1

(I also still think we should have a way to extend/improve/vary
compression algorithms.)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Moto Ishizawa <[email-masked]>
Date: Sat, 12 Jul 2014 21:33:48 +0900

+1
I think that the eviction of reference set is so tricky.

--
Moto Ishizawa

---

From: <[email-masked]>
Date: Sat, 12 Jul 2014 12:37:53 +0000

>> Does anyone disagree with taking this first step?

Yes

> On 12 Jul 2014, at 11:14, "[email-masked]" <[email-masked]> wrote:

+1

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Cory Benfield <[email-masked]>
Date: Sat, 12 Jul 2014 14:57:03 +0100

> Does anyone disagree with taking this first step?

As mentioned previously, I'm +1.

---

From: James M Snell <[email-masked]>
Date: Sat, 12 Jul 2014 07:40:05 -0700

+1

---

From: Kazu Yamamoto (山本和彦) <[email-masked]>
Date: Sun, 13 Jul 2014 08:05:03 +0900 (JST)

Hi,

> Jeff proposed this in <https://github.com/http2/http2-spec/issues/552> to address other issues (#549-551); while it may be that it does at least help mitigate them, we'll consider those issues sepa…

I agree to remove reference set.

--Kazu

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 14 Jul 2014 11:02:59 +1000

I agree to remove the reference set... but .....

I would like to see a bit more discussion of an actual draft change to do
that as I think there were a few features of the design that were driven by
the existence of the reference set.

Specifically, do we still need to copy static entries into the header set?

My understanding of that aspect feature was to control the maximum size of
the reference set for zero state implementations.  Now that there is no
reference set and the header table has it's own explicit size limit, can we
just let static references stay in the static table.

There may be some other changes needed because of this - such as putting
the static indexes below the header table indexes?

Eitherway, can we see a pull request?

cheers

On 13 July 2014 09:05, Kazu Yamamoto <[email-masked]> wrote:

> to address other issues (#549-551); while it may be that it does at least
> help mitigate them, we'll consider those issues separately. We can also
> consider further modifications (e.g., requiring :-headers to be first).
>

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Kazu Yamamoto (山本和彦) <[email-masked]>
Date: Mon, 14 Jul 2014 10:41:18 +0900 (JST)

Hi,

> just let static references stay in the static table.

I think that the index for static table and one for dynamic table can
be separated. If we take such a design, HPACK implementation would get
simpler.

--Kazu

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Mon, 14 Jul 2014 11:54:49 +0900

2014/07/14 10:45 "Kazu Yamamoto
> be separated. If we take such a design, HPACK implementation would get

Could you elaborate your idea?
I guess you have new index opcode for static table but it is not sufficient.

Best regards,
Tatsuhiro Tsujikawa

---

From: Kazu Yamamoto (山本和彦) <[email-masked]>
Date: Mon, 14 Jul 2014 16:35:56 +0900 (JST)

> I guess you have new index opcode for static table but it is not sufficient.

What I thinking is to use two tables separately.

          <-- Header  Table -->  <-- Static  Table -->
          +---+-----------+---+  +---+-----------+---+
          | 1 |    ...    | k |  | 1 |    ...    | 61|
          +---+-----------+---+  +---+-----------+---+
          ^                   |
          |                   V
   Insertion Point      Dropping Point

Each table has its own index. Prepare necessary operations and their
binary format.

--Kazu

---

From: Roberto Peon <[email-masked]>
Date: Mon, 14 Jul 2014 01:41:17 -0700

The reference set has little or nothing to do with any of the other
protocol changes or issues on the table right now. It has been incorrectly
conflated with other issues.

The reference set does not decrease efficiency.
The reference set can substantially increase efficiency in some cases, and
is likely to do so in the common case if senders adapt to the compression.

   - To make interleaving possible, *all* state referring or state
   modifying opcodes must not be permitted to cross a frame boundary.
   - Interleaving of headers would create a huge DoS hole, and would be a
   poor design decision in the first place as it creates an obvious and easily
   exploitable DoS surface area which is proportional to the maximum allowed
   header size times the number of allowed concurrent streams.
   - Ordering of ':' headers first is possible using the current spec,
   unchanged.-- we had chosen not to require it solely as it could decrease
   efficiency.
   - Maintaining the order of headers is possible with HPACK as it exists
   today. Again, we had chosen not to require it solely as it could decrease
   efficiency.

Are there any other technical problems we have with the reference set?

If we're stepping back and redesigning the compressor, there are other
things I'd change that would increase efficiency, and make the use of the
reference set optional for encoders (and decoders have the easy part
anyway).

It doesn't make sense to me to remove it unless we believe that the
use-cases it potentially solves are not going to happen.

The reference set makes server-push more efficient (and it can be
substantially more efficient with its use).
It makes non-browser workloads more efficient.
It makes proxy communication more efficient when demuxing/muxing is done
properly by the loadbalancing proxy (i.e. it creates connections properly
to its targets).

-=R

---

From: <[email-masked]>
Date: Mon, 14 Jul 2014 09:49:57 +0000

On Monday,14 July 2014 10:41, [email-masked] wrote:
> The reference set has little or nothing to do with any of the other protocol
> changes or issues on the table right now. It has been incorrectly conflated with
> other issues.
>

I thought Mark made it a separate issue so as to _purposely_ not to conflate it with other issues.

> The reference set does not decrease efficiency.

Kazu didn't say it decreases efficiency.  His data just showed that it was *equivalent* to Linear+Huffman for the available data set (admittedly small).

Furthermore, in all of the replies to this thread, nobody has said anything about efficiency.  The words that have been associated with the reference set have been "tricky" or "difficult".

Making a "tricky" feature mandatory with no immediately obvious benefit doesn't seem like a good idea.

- "Do not add new functionality unless you know of some real application that will require it." [1]
- "If you can get 90 percent of the desired effect for 10 percent of the work, use the simpler solution." [1]

> The reference set can substantially increase efficiency in some cases

Data please.

> , and is
> likely to do so in the common case if senders adapt to the compression.
>

Speculative (though I agree).

>
> *     To make interleaving possible, *all* state referring or state modifying
> opcodes must not be permitted to cross a frame boundary.
> *     Interleaving of headers would create a huge DoS hole, and would be a

I think I'm convinced of this point, which is why I support the 'Greg et al' proposal that requires the header block to be in a single frame.

> *     Ordering of ':' headers first is possible using the current spec,
> *     Maintaining the order of headers is possible with HPACK as it exists
> If we're stepping back and redesigning the compressor, there are other things I'd
> change that would increase efficiency,
>

I think everybody would love to hear your ideas.

> and make the use of the reference set
> optional for encoders (and decoders have the easy part anyway).
>

I think optional would be OK, but I would like to see it optional from the peer's point of view (which is how the rest of the settings already are)  i.e. a client can't use it until the server declares support

[1] http://en.wikipedia.org/wiki/X_Window_System_protocols_and_architecture#Design_principles

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Mon, 14 Jul 2014 23:04:25 +0900

>           <-- Header  Table -->  <-- Static  Table -->
​More binary format means we have to reduce prefix bits, which reduces
compression efficiency. Meanwhile, static table gets smaller number, so
there is a trade off.  Actually this is orthogonal to removal of reference
set.
Personally, I don't think current scheme (static table at the end of
dynamic one) is very hard to implement or tricky. So I'm not so sure this
change is worth its cost.  If dynamic header table index gets screwed up,
we still valid index in static table but it is corrupted anyway as a whole.​

​Best regards,
Tatsuhiro Tsujikawa​

---

From: Johnny Graettinger <[email-masked]>
Date: Mon, 14 Jul 2014 12:02:22 -0400

> protocol

It's a separable decision, but not a separable concern.

> Kazu didn't say it decreases efficiency.  His data just showed that it was
> *equivalent* to Linear+Huffman for the available data set (admittedly
> small).
>

Does anybody truly believe the data is representative of the broad web, and
the usage patterns that occur there? I don't. For example, the effect of
cookie crumbs is tremendously under-represented.
I also don't believe the data represents the easy, low-hanging fruit that
will likely be optimized upon HPACK deployment (eg, quantize the Date
header with courser granularity).

Furthermore, in all of the replies to this thread, nobody has said anything
> about efficiency.  The words that have been associated with the reference
> set have been "tricky" or "difficult".

I'm unsympathetic to this. SPDY uses deflate for headers, which is far more
complicated under the hood. The choice of compressor is orthogonal to
whether each implementer must go and implement it. Multiple implementations
are available under permissive licenses, as has been pointed out on this
list.

Meanwhile, we have actual operational experience with multiple implementers
who have built out the reference set. We've already turning it on for real
users. I have yet to get a bug report for it. Granted I may, but my
experience to date is that it's not the landmine it's being painted as.

> The reference set can substantially increase efficiency in some cases

Chromium has offered to do a study on this. That offer received no response
or endorsements.

It would still be interesting to do, but only if there's honest engagement
and buy-in from the list. Setting up something like this takes time, and is
not compatible with the rate-of-change in arguments we're seeing on the
list lately.

> things I'd

Let's *not* open this can of worms. Let's finish.

---

From: Cory Benfield <[email-masked]>
Date: Mon, 14 Jul 2014 17:32:43 +0100

On 14 July 2014 17:02, Johnny Graettinger <[email-masked]> wrote:
> not compatible with the rate-of-change in arguments we're seeing on the list
> lately.

I've expressed interest in this data at least twice: once directly
from you, and once in a more coded form when I suggested that it would
be nice to have even more test data to test HPACK interop on. If you
can get this data I'll happily evaluate it. I'd also love it if we
could broaden our corpus of sample headers (as used by the HTTP2JP
group's HPACK test cases) to reflect what you believe to be a more
reasonable sample of web traffic. Having that large corpus of data
will be useful for evaluating future compression proposals (assuming
there will be some).

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Tue, 15 Jul 2014 01:59:03 +0900

I am also interested in this.  If we have data to demonstrate the advantage
of reference set, I can get behind that.

Best regards,
Tatsuhiro Tsujikawa

> > not compatible with the rate-of-change in arguments we're seeing on the
list

---

From: Johnny Graettinger <[email-masked]>
Date: Mon, 14 Jul 2014 16:19:04 -0400

On Mon, Jul 14, 2014 at 12:32 PM, Cory Benfield <[email-masked]> wrote:

>  I'd also love it if we could broaden our corpus of sample headers (as
> used by the HTTP2JP

Agreed, it would be. There are major privacy concerns baked into any
collection of this kind, though, that I simply don't know how to reasonably
solve. So instead, we're instrumenting the browser itself to aggregate the
metric of interest, and definitely not collecting any raw headers.

The core metric of interest seems to be the compression performance
(aggregated byte-length ratio) of a delta encoder + Huffman. We can provide
separate ratios for an encoder with and without the reference set.

This should hopefully inform the question before us, but is of little
utility for future experimentation with compressors. That's unfortunate,
but I think it's just the trade-off for getting good coverage.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 15 Jul 2014 10:46:36 +1000

> we still valid index in static table but it is corrupted anyway as a whole.​

I don't think it is so much an argument about being difficult to implement
- it's is pretty simple.  I think the argument is about  what is most
efficient.

I think it is self evident that copying static entries into the dynamic
table is less efficient.  It causes 2 entries for the same field (or more
for dumb encoders), which wastes space in the index space.  It also causes
churn in the header table which will result in retransmits.   So without
the need to limit the reference set size, I think it is a no brainer to
stop doing this copy.

Then references to the  static table are going to be a lot more frequently
used.  If the static table is indexed first, then those frequent entries
will only be 1 byte.  If they are indexed 2nd, then they are going to be at
least 2 bytes for any non trivial sized header table.

If we can get some good example headers, then it should be pretty simple to
work out which is best for a few sample header table sizes.

Failing that, a dedicated static table bit is a good guess at a compromise
for different sized header tables:

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 1 | S |    Index (6+)         |
   +---+---------------------------+

This gives 62 static entries (excluding 0 and 64) that fit into 1 byte,
plus the first 62 dynamic entries would also fit into 1 byte.     I think
this allows use-cases with lots of common headers and ones with lots of
custom headers to both have a good number of 1 byte field encodings.

cheers

---

From: Martin Thomson <[email-masked]>
Date: Mon, 14 Jul 2014 20:43:11 -0700

On Jul 14, 2014 5:51 PM, "Greg Wilkins" <[email-masked]> wrote:
> Failing that, a dedicated static table bit is a good guess at a

The testing done thus far suggests two things: that the static table is
rarely used after an initial phase; that a six bit index is somewhat,
though not significantly, worse than a seven bit one at avoiding a need for
two byte references.

If you have good data, please share. Otherwise, I think that we need to be
a little more respectful of the work that had already been done. Yes, more
data could likely suggest an alternative course, that is always true. But
knowing when to stop second guessing, that is the trick. If you plan to
actually do thing, that is.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 15 Jul 2014 13:59:05 +1000

On 15 July 2014 13:43, Martin Thomson <[email-masked]> wrote:

> rarely used after an initial phase;

I assume that is with the current design of copying static index's fields
to the header table. With that design, it is natural for the static
reference to be used less over time.   But as I have argued, that is an
unnecessary copy if there is no longer a reference table.

I will do some experiments with the limited header data that I have.

regards

---

From: Roberto Peon <[email-masked]>
Date: Mon, 14 Jul 2014 21:13:34 -0700

Note that there the spec says to copy, but a smart implementation needs not
actually do that. It is an accounting trick.

-=R

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 15 Jul 2014 14:26:14 +1000

On 15 July 2014 14:13, Roberto Peon <[email-masked]> wrote:

> Note that there the spec says to copy, but a smart implementation needs
> not actually do that. It is an accounting trick.

Yes I understand reference counting!

As previously discussed, I'm not concerned about the actual copy of the
strings of the header.   I'm concerned that by copying a static entry into
the header table you have to evict another entry.     This makes
inefficient use of the header table allowance as it can contain many (and
duplicate) copies of static entries.  The resulting churn in the header
table will mean that all entries will need to be periodically resent.

Note also that on an aggregate connection this copy can be used as part of
a DoS, as a request could contain many and duplicated static indexes, which
will cause the header table to be flushed and all shared connections to
have to resend fields.

It is also needless complexity for the encoder, which should just be able
to know the index of :status:200 and drop that byte into the buffer.
Instead it now has to do a lookup to see if and where the :status:200 field
is in the dynamic table and use that index.

I know it is not all that difficult to implement, but the only reason given
for this copy was to limit the size of the reference set.    If there is no
reference set, then the copy has no purpose what so ever and it has some
cost.    We can't seriously be proposing to leave it in there as some kind
of legacy kludge?

regards

---

From: Martin Thomson <[email-masked]>
Date: Mon, 14 Jul 2014 21:28:41 -0700

On 14 July 2014 21:26, Greg Wilkins <[email-masked]> wrote:
> We can't seriously be proposing to leave it in there as some kind of legacy
> kludge?

The reasons I cited are still relevant considering that most entries
don't have associated values.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 15 Jul 2014 14:39:59 +1000

> don't have associated values.

Well of course literal fields with index'd names will be copied into the
header table.  That is an entirely different matter than copying an indexed
static field.

To be precise I am concerned with the text in 4.1 that says:

   An _indexed representation_ corresponding to an entry _not present_
   in the reference set entails the following actions:

   o  If referencing an element of the static table:

      *  The header field corresponding to the referenced entry is
         emitted.

      *  The referenced static entry is inserted at the beginning of the
         header table.

      *  A reference to this new header table entry is added to the
         reference set, unless this new entry didn't fit in the header
         table.

I believe that if we do not have a reference set, then this should just
become:

   An _indexed representation_ entails the following actions:

   o  If referencing an element of the static table:

      *  The header field corresponding to the referenced entry is
         emitted.

I just cannot understand why you wish to create an entry in the dynamic
table whenever the static table index is used for something like
:status:200 ?    I only just accepted the reason that it limited the
reference set size to 0 if the header table size was set to 0.  Now there
is no reference set, there is no reason for this.

---

From: Roberto Peon <[email-masked]>
Date: Mon, 14 Jul 2014 22:33:57 -0700

If there is no reference set, there would be no reason to copy (or pretend
to copy) items from the static table to the dynamic header table.

Then again one could choose to have opcodes for ephemeral actions vs state
changing actions, or change how one emits the reference set by making it
explicit, or make the static table entries negative or put them at the
start of the space instead of the end, or have several reference sets (good
for proxies), or require references to show up before literal opcodes, or
have different opcodes for static table entries, or have opcodes which
modify ranges of entries, or have arrays of opcodes of the same type, or
use 7 bit encoding or fields or use sentinels instead of using length
prefixing or have differing Huffman tables for different directions on
different frames, or encoder lengths differently, or typed fields, or have
different limits for header entry count vs size, etc.

I've tried all of the above, incidentally.

The current design was the winner as it best balanced speed and compression
efficiency.

---

From: <[email-masked]>
Date: Tue, 15 Jul 2014 09:37:05 +0000

Hi Roberto-

On Tuesday,15 July 2014 07:34, [email-masked] wrote:
> ... or have opcodes which modify ranges of entries, ...

Did you also look at "having opcodes which *reference* ranges of entries"?

A week or so ago Herve mentioned this as an alternative to the reference set.  During my research of header compression I came to the same idea and found it to improve the compression factor much in the same way as a reference set.

As you have mentioned before, and I concur, applications will eventually adapt to the compression mechanism.  I believe this would also be true for referencing ranges of entries from a header table. While it couldn't achieve the same compression as a reference set, it could come really close as applications start shaping the header set to contain the largest possible contiguous range of headers which are repeated from request to request.

-keith

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 15 Jul 2014 11:52:34 +0200

Hi Greg,

> is no reference set, there is no reason for this.

The benefit I found in doing so was to lower the static table indexes. In
fact, I'm seeing the dynamic table as a small cache and the static table
as the second level cache. Accessing recent data requires only a few bits,
accessing less recently used data (static table) requires a few more bits
sometimes leading to an extra byte.

I tend to find this design quite smart, because it manages to shrink the
working set representation simply by changing the indexes. I'm not sure
you'll be able to reach the same compression level without adding an
extra bit to the index number. That's a pure guess of course, but that's
what I find smart an efficient in this algorithm.

Cheers,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 15 Jul 2014 10:50:39 +0000

>> is no reference set, there is no reason for this.

I really don't like that its indicies vary from connection to
connection, (based on the size of the dynamic table used.  That's
going to make debugging and packet-dumping require a lot more state
before you can see any kind of pattern in the data.

Has anybody thought about putting algorithmic compressions in the static table?

If we make 33.Value and 36.Value mean "Integer time_t follows"
(using 4.1.1's integer format), we will probably shave 10 bytes of
each request and response.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Jul 2014 10:15:03 -0700

> Has anybody thought about putting algorithmic compressions in the static table?

Yes.

---

From: James M Snell <[email-masked]>
Date: Tue, 15 Jul 2014 10:31:54 -0700

Yup. algorithmic compressed were deemed to be "uninteresting" to the
working group at this time... although, the current stateful
compression doesn't really do anything to make transmission of
timestamps and other frequently changing values more efficient, which
is unfortunate.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Jul 2014 10:35:34 -0700

On 15 July 2014 10:31, James M Snell <[email-masked]> wrote:
> the current stateful

Yes, Huffman only helps a little.

---

From: Jason Greene <[email-masked]>
Date: Tue, 15 Jul 2014 14:14:25 -0500

> each request and response.

Whatever happened with this idea? Searching my archives I couldn’t find it getting rejected. I also don’t see an issue filed for it. 

It would be a nice perf boost to be able to do calculations against time_t vs parsing.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Jul 2014 13:21:05 -0700

> Whatever happened with this idea? Searching my archives I couldn’t find it getting rejected. I also don’t see an issue filed for it.

Search for "typed codecs" and variations on the same.

Also: http://tools.ietf.org/html/draft-snell-httpbis-bohe-13  (note
the version number)

---

From: James M Snell <[email-masked]>
Date: Tue, 15 Jul 2014 18:01:55 -0700

> Also: http://tools.ietf.org/html/draft-snell-httpbis-bohe-13  (note

Note also that the version number is reflective only of my own
editing/publishing style and not of any particular amount of
consideration from the WG

- James

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 12:13:48 +1000

I ran the hpack test data through the various options being discussed for
several different header table sizes:

HPACK/HEADER TABLE01024409616384h2-1264.22%47.06%42.18%41.82%h2-12-noRefSet
64.22%47.39%42.57%42.44%h2-12-noRefSet-noCopy64.22%47.16%42.50%42.67%
h2-12-noRefSet-noCopy-static164.22%47.18%42.36%40.97%

There is actually very little variation in results +/- 1% which is likely
to be below the variation due to different data sets.       So on that
basis we should go for the simplest mechanism, which is either
h2-12-noRefSet-noCopy  or h2-12-noRefSet-noCopy-static1

But if we trust that data set, then the best compression was achieved with
h2-12-noRefSet-noCopy-static1 and a large header table.    Also
h2-12-noRefSet-noCopy-static1 was better than h2-12-noRefSet-noCopy for all
but the 1024 table size.

So on this data we should go with h2-12-noRefSet-noCopy-static1

h2-12 == the HPACK used by draft 12 of HTTP
noRefSet == no Reference Set
noCopy == Do not copy static fields to the header table
static1 == Static indexes are 1-61, dynamic are 62-...

cheers

On 16 July 2014 11:01, James M Snell <[email-masked]> wrote:

> >> Whatever happened with this idea? Searching my archives I couldn’t find
> it getting rejected. I also don’t see an issue filed for it.
> >

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jul 2014 12:31:33 +1000

What do people think of this proposal?

So far, my reading of the WG is that we want to get rid of the reference set, and are just talking about the details of how to do so.

Regards,

On 16 Jul 2014, at 12:13 pm, Greg Wilkins <[email-masked]> wrote:

> I ran the hpack test data through the various options being discussed for several different header table sizes:
> 
> HPACK/HEADER TABLE	0	1024	4096	16384
> h2-12	 64.22%	47.06%	42.18%	 41.82%
> h2-12-noRefSet	64.22%	 47.39%	42.57%	42.44%
> h2-12-noRefSet-noCopy	64.22%	47.16%	 42.50%	42.67%
> h2-12-noRefSet-noCopy-static1	 64.22%	47.18%	42.36%	 40.97%
> 
> 
> There is actually very little variation in results +/- 1% which is likely to be below the variation due to different data sets.       So on that basis we should go for the simplest mechanism, which is either h2-12-noRefSet-noCopy  or h2-12-noRefSet-noCopy-static1
> 
> But if we trust that data set, then the best compression was achieved with h2-12-noRefSet-noCopy-static1 and a large header table.    Also h2-12-noRefSet-noCopy-static1 was better than h2-12-noRefSet-noCopy for all but the 1024 table size.
>

---

From: Rob Trace <[email-masked]>
Date: Wed, 16 Jul 2014 03:05:49 +0000

We are OK with removing the reference set.  The marginal compression benefit does not justify the added complexity.

Thanks!!

-Rob

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Tuesday, July 15, 2014 7:32 PM
To: Greg Wilkins
Cc: HTTP Working Group
Subject: Re: Call for Consensus: Remove "reference set" from HPACK (to address #552)

Regards,

On 16 Jul 2014, at 12:13 pm, Greg Wilkins <[email-masked]> wrote:

> h2-12 == the HPACK used by draft 12 of HTTP noRefSet == no Reference 
> Set noCopy == Do not copy static fields to the header table
> >> Whatever happened with this idea? Searching my archives I couldn't find it getting rejected. I also don't see an issue filed for it.
> >
> http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that 
> scales http://www.webtide.com  advice and support for jetty and cometd.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 16 Jul 2014 13:18:27 +1000

On 16 July 2014 12:31, Mark Nottingham <[email-masked]> wrote:

> So far, my reading of the WG is that we want to get rid of the reference
> set, and are just talking about the details of how to do so.
>
> Regards,
>
>
I'm happy to drop the reference set. I don't care overly beyond that,
either way.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Jul 2014 20:25:31 -0700

> static1 == Static indexes are 1-61, dynamic are 62-...

Does this mean two different ways of referencing the static table?  Or are
you adding the flag for name references in literals as well?

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 13:52:35 +1000

Martin,

static1 was simply switching the indexes so static indexes are 1-61,
dynamic are 62-.   It was not the proposal to create an extra static bit,
but is much the same as 7+ allows 126 one byte values, so the first 64
dynamic entries are 1 byte only.

No change to the literals. If they use a static name then they index the
static table as 1-62.  If they are an indexed literal, then a header table
entry is created and the field will have a field index >61 when next used.

cheers

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Jul 2014 22:25:39 -0700

On 15 July 2014 20:52, Greg Wilkins <[email-masked]> wrote:
>
>
> static1 was simply switching the indexes so static indexes are 1-61, dynamic
> are 62-.   It was not the proposal to create an extra static bit, but is
> much the same as 7+ allows 126 one byte values, so the first 64 dynamic
> entries are 1 byte only.

Sorry, gotcha.  So references to the header table for name-only
require two bytes most times.  Seems like that's rare enough that
there is a gain.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jul 2014 07:45:20 +0200

> entry is created and the field will have a field index >61 when next used.

The problem I'm seeing with this change is that you need one extra-byte to
send litteral headers fields that are not present in the static table. Maybe
on the other hand it rarely happens that they're not in the static table ?
It may be nice to re-run a check if we change that to see if the static table
is still optimal then, because before this change, it's only used as a
fallback to look up common values. Now we want it to be used for almost
all litteral lookups, and we might want to add one or two fields there
if we realize that a common one is missing.

I agree however that since we have 7 full bits to send an indexed header,
this will make no change here so that's fine.

Willy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 15:55:58 +1000

> require two bytes most times.  Seems like that's rare enough that

You still get the first 64 entries of the header table in 1 byte, so not
all non static names indexes will be 2 bytes.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 16:00:33 +1000

On 16 July 2014 15:45, Willy Tarreau <[email-masked]> wrote:

> send litteral headers fields that are not present in the static table.
>

The first 64 entries can still be sent with 1 byte.   So I think in many
common use-cases almost all the indexes will just be 1 byte.

> Maybe

Well the static table has few name+value fields, but the ones it does have
are common.

> if we realize that a common one is missing.

I don't think it is correct to say for 'almost all literal lookups'.   Any
field with a custom value (with or without an indexed name) will be put
into the header table.  The first 64 of those (ie the last 64) will be
indexed with a single byte.   You only need go to a 2 byte index if the
header table is large enough to hold more than 64 entries and your entry
was added more that 64 additions ago.    I think 2 byte indexes are going
to be rare.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jul 2014 08:08:25 +0200

> was added more that 64 additions ago.    I think 2 byte indexes are going

I mean, the current design allows you to have a header field whose value
changes between consecutive requests (eg: a proxy aggregating multiple
users' requests), and still have that changing value referenced with
little overhead. Here to achieve the same result we'll need to have
that header field present in the static table in order for the index to
fit in a 6-bit quantity. I *think* that we have most common fields in the
static table so that should not be an issue. But that's something we might
want to double-check. Note that I'm clearly speaking about changing a
header value, not reusing the same value (which needs only one byte for
all cases).

Willy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 16:19:41 +1000

On 16 July 2014 16:08, Willy Tarreau <[email-masked]> wrote:

> little overhead.

That is still possible.

If one user sends a field

   wibble: now is the time for all good men to come to the aid of the party

that will be sent literal name and value and will go into the header
table.  Then when another user sends

  wibble: the moon is blue to a fish in love

That can be sent as a literal, but with indexed name - pointing to the
previous entry (but the value is ignored).

Now both fields are in the table and both can be used with a single index
reference.  If they are in the first 64, then it is only a single byte.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jul 2014 08:18:44 +0200

> header value, not reusing the same value (which needs only one byte for

After checking in a number of traces, the only field I'm seeing subject to
changing between requests is x-forwarded-for, and that's clearly one which
is sent where we don't care about saving one byte per request over the wire.
So your proposal might be reasonably efficient.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jul 2014 08:40:36 +0200

> reference.  If they are in the first 64, then it is only a single byte.

Then I must be missing something because in 7.2.1 I'm seeing "Index (6+)".
If those 6 bits are used for the static table, I don't see how encode the
header table on 6 bits :-/

Willy

---

From: Cory Benfield <[email-masked]>
Date: Wed, 16 Jul 2014 07:54:23 +0100

> So far, my reading of the WG is that we want to get rid of the reference set, and are just talking about the details of how to do so.

Myself and a few others are hesitating here. Roberto and Johnny have
both suggested that they believe the reference set _is_ helpful, and I
believe Johnny is collecting data to prove the point. I'd like to be
forward-looking where we can, so I'm inclined to want to wait a week
or so to see what data Johnny can collect.

Is that problematic from the perspective of our timetable?

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jul 2014 16:59:22 +1000

Last we heard from him on this AFAIK was:

> It would still be interesting to do, but only if there's honest engagement and buy-in from the list. Setting up something like this takes time, and is not compatible with the rate-of-change in arguments we're seeing on the list lately.

and then:

> The core metric of interest seems to be the compression performance (aggregated byte-length ratio) of a delta encoder + Huffman. We can provide separate ratios for an encoder with and without the reference set.
> 
> This should hopefully inform the question before us, but is of little utility for future experimentation with compressors. That's unfortunate, but I think it's just the trade-off for getting good coverage.

Johnny, would you be able to provide any indication in a short time frame, or are we talking about something longer-term?

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 17:21:21 +1000

On 16 July 2014 16:54, Cory Benfield <[email-masked]> wrote:

> I believe Johnny is collecting data to prove the point
>

I'm happy to run any data set through my tests if you can get it to me.

cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 16 Jul 2014 06:43:34 +0000

In message <CAH_y2NFna2Gh9=[email-masked]>, Greg Wilkins wri

>So on this data we should go with h2-12-noRefSet-noCopy-static1

Sounds good to me.

Can I persuade you do a run of h2-12-noRefSet-noCopy-static1 where headers
which contain only a timstamp compresses to 5 bytes, so we can see if 
"algorithmic compressors" are worth looking at ?

As I remember, it's these five headers:

	Date:
	Expires:
	Last-Modified:
	If-Modified-Since:
	If-Unmodified-Since:

A more comprehensive algorithm would also reach for timestamps in cookies etc.
but don't worry about that for now.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 16 Jul 2014 06:45:26 +0000

>on the other hand it rarely happens that they're not in the static table ?

Sounds like a good way to persuade people to not create new headers all
the time :-)

Which reminds me:

Should we fold "X-Forwarded-For" to "Forwarded-For" as part of the
compression ?

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jul 2014 11:25:49 +0200

> Sounds like a good way to persuade people to not create new headers all

Good point :-)

> Should we fold "X-Forwarded-For" to "Forwarded-For" as part of the

No we must not do that, because X-whatever as its name implies is not
standardized and can be used any other way. Also there are a very large
number of systems around which make use of it and which will not yet be
able to parse Forwarded, which has a different representation.

Willy

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Wed, 16 Jul 2014 22:30:38 +0900

On Wed, Jul 16, 2014 at 11:13 AM, Greg Wilkins <[email-masked]> wrote:

> HPACK/HEADER TABLE 010244096 16384h2-12 64.22%47.06%42.18% 41.82%
> h2-12-noRefSet64.22% 47.39%42.57%42.44% h2-12-noRefSet-noCopy64.22%47.16%
> 42.50%42.67%h2-12-noRefSet-noCopy-static1 64.22%47.18%42.36% 40.97%
>
​Thank you for the experimentation.​

​From the table, h2-12 shows best efficiency to me.
Also the result depends on the encoder algorithm and data set.
If someone does the same tests with their own implementation, the result
may vary.

Best regards,
Tatsuhiro Tsujikawa
​

>> >> Whatever happened with this idea? Searching my archives I couldn’t
>> find it getting rejected. I also don’t see an issue filed for it.
>> >
> scales

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 23:41:25 +1000

> "algorithmic compressors" are worth looking at ?

I just did a quit hack to use the literal with indexed name encoding, but
that detects the five date headers.  The value is then just sent as 5
bytes.

But the savings were good:
HPACK/HEADER TABLE01024409616384h2-1264.22%47.06%42.18%41.82%h2-12-noRefSet
64.22%47.39%42.57%42.44%h2-12-noRefSet-noCopy64.22%47.16%42.50%42.67%
h2-12-noRefSet-noCopy-static164.22%47.18%42.36%40.97%
h2-12-noRefSet-noCopy-static1-date553.62%39.48%34.99%33.62%
So that's a 6-10% saving, which is lots more than any other feature we are
discussing here. So worth looking into!

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 23:58:22 +1000

> ​From the table, h2-12 shows best efficiency to me.

Well to be precise h2-12-noRefSet-noCopy-static1 showed the best
efficiency, but now h2-12-noRefSet-noCopy-static1-date5 does.

However, I think the data set is way too small to be claiming any of these
(other than date5) gives the best result.   Looking at the results from the
individual stories in the test data, they are each +/- 5%. I have simply
combined the results from all stories and there is no rigour in that as
some are probably more frequent that others.

So reporting the figures to two decimal places was not a valid thing for me
to do and the following table probably more accurately represents the
inaccuracies of this test:

HPACK/HEADER TABLE01024409616384h2-1264%47%42%42%h2-12-noRefSet64%47%43%42%
h2-12-noRefSet-noCopy64%47%42%43%h2-12-noRefSet-noCopy-static164%47%42%41%
h2-12-noRefSet-noCopy-static1-date554%39%35%34%

Ie, none of these variations (except date5) give any significant savings on
this data set.   Hence I think the simplest is the best:
h2-12-noRefSet-noCopy-static1

But I would be very interested to see a concrete proposal for
h2-12-noRefSet-noCopy-static1-date5

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 17 Jul 2014 00:22:00 +1000

Note also that those numbers are from a Jetty encoder that had a encoding
strategy designed with a RefSet in mind.   It makes some choices about
which encoding to use - eg it doesn't put set-cookie in the header table as
it is unlikely to be sent again.

So I'm making no representation about the absolute numbers as the choice of
encoding may not be optimal, and for things like Date it will depend on the
speed of a server if it is worthwhile putting them in the header table or
not.  Although I note my numbers are similar to others previously published.

The only representation I'm making with this data is a very limited one
about the relative values with/without various features - that being that
they don't make a lot of difference.

cheers

On 16 July 2014 23:58, Greg Wilkins <[email-masked]> wrote:

> So reporting the figures to two decimal places was not a valid thing for
> me to do and the following table probably more accurately represents the
> HPACK/HEADER TABLE0 1024409616384 h2-1264%47% 42%42% h2-12-noRefSet64%47%
> 43%42%h2-12-noRefSet-noCopy 64%47%42% 43%h2-12-noRefSet-noCopy-static164%
> 47%42%41% h2-12-noRefSet-noCopy-static1-date554%39% 35%34%
>
> Ie, none of these variations (except date5) give any significant savings
> on this data set.   Hence I think the simplest is the best:

---

From: James M Snell <[email-masked]>
Date: Wed, 16 Jul 2014 07:50:30 -0700

On Jul 16, 2014 7:01 AM, "Greg Wilkins" <[email-masked]> wrote:
[snip]

My proposal would be to adopt the the typed codec encodings from section
3.1 of bohe-13*. The WG ruled out typed encodings long ago, however. So I'm
not sure if it's something we can really put back on the table now.

- James

* the section 3.1 encodings can be used independently of the rest of the
bohe-13 draft.

scales

---

From: "Martin Nilsson" <[email-masked]>
Date: Wed, 16 Jul 2014 17:03:19 +0200

On Wed, 16 Jul 2014 16:50:30 +0200, James M Snell <[email-masked]>  
wrote:

> 3.1 of bohe-13*. The WG ruled out typed >encodings long ago, however. So  
> I'm not sure if it's something we can really put back on the table now.

They way I recalled it was that the WG was interested in typed encodings,  
but saw that as an independent work item that could be deferred. We've  
certainly used typed encoding in Operas protocol with great result, and  
fully independent of the application layer and the compression layer.

/Martin Nilsson

-- 
Using Opera's mail client: http://www.opera.com/mail/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 16 Jul 2014 15:02:54 +0000

In message <CAH_y2NHvFhN75KXmFLxKkgwjSB+vG=[email-masked]>, Greg Wilkins wri
tes:

> |-----------------------------------+---+----+----+-----|
> |h2-12-noRefSet-noCopy-static1      |64%|47% |42% |41%  |
> |-----------------------------------+---+----+----+-----|
> |h2-12-noRefSet-noCopy-static1-date5|54%|39% |35% |34%  |
> +-------------------------------------------------------+

>h2-12-noRefSet-noCopy-static1-date5

17% compression over the baseline is more than I expected.

My proposal is:

Static table:
          +-------+-----------------------------+--------------+
          | Index | Header Name                 | Header Value |
          +-------+-----------------------------+--------------+
          | 1     | :authority                  |              |
          | 2     | :method                     | GET          |
	  [...]
          | 33    | date                        | time_t       |
	  [...]
          | 36    | expires                     | time_t       |
	  [...]
          | 40    | if-modified-since           | time_t       |
	  [...]
          | 43    | if-unmodified-since         | time_t       |
          | 44    | last-modified               | time_t       |
	  [...]

Where the Header Value is "time_t", the encoding is:

     0   1   2   3   4   5   6   7
   +---+---+---+---+---+---+---+---+
   | 1 |        Index (7+)         |
   +---+---------------------------+------//------------+
   | 32 bit unsigned network byte order time value      |
   +-------------------------------+------//------------+

The time value is the number of seconds since 1970-01-01 00:00 UTC
without accounting for leap seconds.  (ie:  The UNIX/POSIX time_t
definition)

If the value cannot be encoded in a 32 bit unsigned
integer it SHALL be sent as a string.

Notice that I'm proposing a unsigned 32 bit, to be Y2038
compliant.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 16 Jul 2014 10:58:16 -0700

> Notice that I'm proposing a unsigned 32 bit, to be Y2038

I realize that it's going to be less efficient, but the regular HPACK
numeric encoding with a 0-bit prefix will be more regular (and not
need the string exception).  At a cost to regularity.

This is a slippery slope; are we really going there?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 16 Jul 2014 18:03:06 +0000

In message <CABkgnnXAjoj70HSpbqXHz7cskTc9YBCob9WPkSD9-nyVtFYg=[email-masked]>, Martin Thomson w

>need the string exception).

The major exception is negative numbers, it's not uncommon to see
dates being 1969-12-31 23:59:59Z and I don't belive the HPACK numeric
encoding handles that ?

Also, the string exception will still be needed for the case where
the date is misformatted on arrival at h1->h2 proxies.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 17 Jul 2014 14:38:56 +1000

> This is a slippery slope; are we really going there?

No, we already had consensus here, and I don't see any new information.

Regards,

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 17 Jul 2014 17:45:02 +0200

Hi Greg,

> Well the static table has few name+value fields, but the ones it does have

I'm noticing that we need only 4 bits to send any possible static header+value
(assuming we swap accept-charset and accept-encoding), since all other values
are empty and will never be used as-is but as litterals.

So I think that based on your suggestion to split indexing of static and
dynamic, I'm suspecting we could systematically split indexes between static
and dynamic in opcodes. Indeed, some index fields are already small (6+ for
litterals and 4+ for litterals not indexed/never indexed), so having the
static first will force one extra byte for all dynamic headers. However,
stealing one bit in unused value ranges could help make that situation
better. Something along these lines seems possible (please forgive me for
errors, I'm encoding as I write) :

  11 XXXXXX => send 6+indexed header from *dynamic* table as-is  (0xC0 reserved)
  10 01XXXX => send 4+indexed header from *static* table as-is   (0x80 reserved)

  00 00XXXX => send 4+indexed *dynamic* header without indexing (unchanged)
  10 00XXXX => send 4+indexed *static* header without indexing

  10 1XXXXX => send litteral based on *dynamic* header table (0=new name or reserved)
  01 XXXXXX => send litteral based on *static* header table (0=new name)

The idea is that even when we have few bits to encode the index, we're not
forced to add an extra byte to encode the values from the other table, whatever
table we pick first as first.

I don't know if you see what I mean of if that's even more confuse.

Cheers,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 18 Jul 2014 09:47:38 +1000

I've heard that Johnny is out of the office, so we shouldn't expect an immediate response. My understanding is that a week (or even a few weeks) would be asking for a lot here. Furthermore, it looks like we wouldn't get access to the raw data, just his summary.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 18 Jul 2014 10:13:09 +1000

Willy,

I see what you are saying.  However my experience with hpack is that
fiddling the bit encodings is going to give you +/- <1% difference.

I can make +/- >5% differences by picking different encoding strategies and
see similar differences over different stories over the test data.

So I think tweaking the encodings at this stage is really just operating
within the noise of the different headers.   So I don't think we should
change hpack in this way.  We just don't have the data to optimise the last
1% nor do we know if natural variation is such that it is pointless to try
to find a one-size-pefectly-fit-all solution.

Removing RefSet is definitely a good thing to do.

Removing the copy of static entries to the dynamic table make measurable
impact on compression, so I think we should remove it for simplicity and to
save CPU - it only being added because of the RefSet.

With those removed,  I'm 90% happy with hpack

Once you remove the static copy, I do think there is an argument to be made
to revert the static table to below the dynamic table. This is for
simplicity and because it will allow the common field indexes to be
precomputed as a single byte, saving CPU.      It does not appear to affect
the compression efficiency in any significant way.

With this I'd be 95% happy.

Encoding dates as integers does appear to give some additional
compression.  But apparently it has been proposed and rejected before by
the WG.     So while I think it would be good, I'm not going to advocate
that we revisit at this stage.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 18 Jul 2014 07:49:10 +0200

Hi Greg,

> fiddling the bit encodings is going to give you +/- <1% difference.

In fact I'm not trying to save much, I feel concerned by the fact that with
static table below the dynamic one, we'll always require one extra byte for
any litteral whose name isn't in the static table. And switching the two
very likely changes the importance of the header fields that were put in
the static table, because initially it was not *that* important, mostly
for the first request, now it's for all requests. Since most header fields
appear to be there in the static table, it's not a big deal, but I'd prefer
that we ensure we don't miss any.

For example, some browsers send "TE". Here it's not present in the static
table, probably because it was not worth consuming an entry for a 2-byte
name which would end up in the dynamic table after the first reqeust. But
now, referencing it from the dynamic table will systematically require one
extra byte if the value changes. I gave the example of the XFF header which
should not be an issue over the links where byte count matters though.

> to find a one-size-pefectly-fit-all solution.

I'm really not fond of changing it either because I think it's properly
designed. But if we change the fondamental principle of the dynamic table
containing any recently emitted header and with a low index, this changes
the index distribution.

> precomputed as a single byte, saving CPU.

I know and am not against this, quite the opposite in fact. You may remember
that two years ago I was discussing if we shouldn't encode common headers as
a single byte or so :-)

> It does not appear to affect the compression efficiency in any significant
> way.

That's the point where I think we should be more careful and ensure we have
enough relevant data (eg: some mobile browsers requests captured before the
operators' transparent proxies).

If at least we had the ability to encode both static table and most recent
dynamic entries with a single index, I would feel better. For example you
can have another approach :
  - positive indexes = static table index
  - negative indexes = - dynamic table index

You encode (index + 10), so that you can encode up to the last 10 emitted
fields, and you can encode up to 54 static headers in a single byte (it's
just a matter of proper sorting but it's already reasonably clean). I think
you get the idea.

> that we revisit at this stage.

Same here, it's a desirable optimization but if we go down that route, we'll
continue with specific integer encoding for content-length and accept-ranges,
then we'll start to suggest that we encode all common content-type tokens as
a few bits, etc... And we're back to redefining the whole encoder. So I'd
rather avoid touching it in this regard.

BTW, I think that with the data you currently have and with your encoder, you
could check how often a literal refers to the dynamic table in your data set,
which will be a good indication of the relevance or not of my concern above.

Thanks,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 18 Jul 2014 16:28:07 +1000

Willy,

Below is the frequency distribution of the various field encodings uses and
their size, with some annotations. This is for my preferred option of the
noCopy static1 and a default 4K table.   You can see that from 38037
fields, only 690(1.8%) of them needed 2 byte encoding, whilst 3594(9.4%)
used a 1 byte static index and 18884 (49.6%) used a 1 byte index to the
dynamic header table.

  18884 IdxField 1                # 1B index to header table
   3958 LitIdxName1HuffVal 23     # 1B name index, 22 byte huffman value
(date!)
   3594 StaticIndexed 1           # 1B index to static table
   1492 LitIdxName1HuffValIdxd 5  # 1B index, 4 byte huffman value
    827 LitIdxName1HuffValIdxd 6
    690 IdxField 2                # 2B index to static table
    622 LitIdxName1HuffValIdxd 22
    591 LitIdxName1HuffValIdxd 4
    486 LitIdxName1HuffValIdxd 13
    337 LitIdxName1HuffVal 22
    324 LitIdxName1HuffValIdxd 7
    312 LitIdxName1HuffVal 24
    307 LitIdxName1HuffValIdxd 19
    264 LitIdxName1HuffValIdxd 18
    250 LitIdxName1HuffValIdxd 9
    245 LitIdxName1HuffValIdxd 14
    241 LitIdxName1HuffValIdxd 8
    179 LitHuffNameHuffValIdxd 14
    175 LitIdxName1HuffValIdxd 10
    173 LitIdxName1HuffValIdxd 20
    132 LitIdxName1HuffVal 17
    124 LitIdxNameXHuffValIdxd 34
    124 LitIdxName1HuffValIdxd 12
    116 LitIdxName1HuffValIdxd 48
    115 LitIdxName1HuffValIdxd 21
    106 LitIdxNameXHuffValIdxd 11
    105 LitIdxNameXHuffValIdxd 36
     91 LitIdxName1HuffValIdxd 15
     86 LitHuffNameHuffValIdxd 18
     75 LitIdxName1HuffValIdxd 3
     74 LitHuffNameHuffValIdxd 25
     69 LitIdxName1HuffValIdxd 11
     64 LitIdxNameXHuffValIdxd 52
     64 LitIdxNameXHuffValIdxd 51
     64 LitIdxName1HuffVal 18
     62 LitIdxName1HuffVal 5
     60 LitHuffNameHuffValIdxd 20
     57 LitIdxName1LitValNeverIdx 104
     55 LitIdxNameXHuffValIdxd 7
     52 LitIdxName1HuffValIdxd 47
     49 LitIdxName1HuffValIdxd 44
     48 LitIdxName1HuffValIdxd 24
     48 LitIdxName1HuffVal 16
     47 LitIdxName1HuffVal 12
     45 LitIdxName1HuffValIdxd 50
     44 LitIdxName1HuffValIdxd 23
     44 LitIdxName1HuffValIdxd 17
     42 LitIdxNameXHuffValIdxd 50
     42 LitIdxName1HuffValIdxd 16
     41 LitIdxNameXHuffValIdxd 53
     41 LitIdxName1HuffValIdxd 27
     41 LitIdxName1HuffVal 4
     39 LitIdxNameXHuffValIdxd 37
     39 LitIdxNameXHuffValIdxd 14
     37 LitHuffNameHuffValIdxd 21
     35 LitIdxName1HuffValIdxd 28
     34 LitHuffNameHuffValIdxd 16
     33 LitIdxName1HuffVal 19
     31 LitIdxName1HuffValIdxd 29
     30 LitIdxName1LitValNeverIdx 100
     29 LitIdxNameXHuffValIdxd 41
     28 LitHuffNameHuffValIdxd 15
     26 LitHuffNameHuffValIdxd 22
     24 LitIdxName1LitValNeverIdx 91
     24 LitIdxName1HuffValIdxd 35
     23 LitIdxName1HuffValIdxd 39
     20 LitIdxNameXHuffValIdxd 42
     20 LitIdxNameXHuffValIdxd 10
     20 LitIdxName1HuffValIdxd 32
     20 LitHuffNameHuffValIdxd 24
     19 LitHuffNameHuffValIdxd 26
     18 LitIdxName1HuffValIdxd 43
     17 LitIdxNameXHuffValIdxd 16
     16 LitIdxName1HuffValIdxd 41
     15 LitIdxNameXHuffValIdxd 54
     15 LitIdxNameXHuffValIdxd 24
     15 LitIdxName1LitValNeverIdx 98
     15 LitIdxName1LitValNeverIdx 96
     15 LitIdxName1HuffValIdxd 40
     15 LitIdxName1HuffValIdxd 33
     15 LitIdxName1HuffValIdxd 31
     15 LitIdxName1HuffValIdxd 30
     15 LitIdxName1HuffVal 28
     15 LitIdxName1HuffVal 11
     14 LitIdxName1LitValNeverIdx 81
     14 LitIdxName1HuffValIdxd 25
     14 LitHuffNameHuffValIdxd 183
     13 LitIdxNameXHuffValIdxd 6
     13 LitIdxNameXHuffValIdxd 132
     13 LitIdxName1LitValNeverIdx 82
     13 LitIdxName1HuffValIdxd 26
     13 LitHuffNameHuffValIdxd 17
     12 LitIdxNameXHuffValIdxd 85
     12 LitIdxNameXHuffValIdxd 60
     12 LitIdxNameXHuffValIdxd 59
     12 LitIdxNameXHuffValIdxd 22
     12 LitIdxName1LitValNeverIdx 78
     12 LitIdxName1HuffValIdxd 106
     12 LitHuffNameHuffValIdxd 27
     11 LitIdxNameXHuffValIdxd 74
     11 LitIdxNameXHuffValIdxd 46
     11 LitIdxNameXHuffValIdxd 40
     11 LitIdxNameXHuffValIdxd 109
     11 LitIdxName1HuffValIdxd 59
     11 LitHuffNameHuffValIdxd 13
     11 LitHuffNameHuffValIdxd 11
     10 LitIdxNameXHuffValIdxd 43
     10 LitIdxNameXHuffValIdxd 39
     10 LitIdxNameXHuffValIdxd 19
     10 LitIdxNameXHuffValIdxd 12
     10 LitIdxName1LitValNeverIdx 112
     10 LitIdxName1HuffVal 26
     10 LitHuffNameHuffValIdxd 30
     10 LitHuffNameHuffValIdxd 157
      9 LitIdxNameXHuffValIdxd 49
      9 LitIdxNameXHuffValIdxd 25
      9 LitIdxName1HuffValIdxd 46
      9 LitIdxName1HuffValIdxd 38
      9 LitIdxName1HuffValIdxd 34
      9 LitIdxName1HuffValIdxd 109
      9 LitHuffNameHuffValIdxd 28
      9 LitHuffNameHuffValIdxd 23
      8 LitIdxNameXHuffValIdxd 30
      8 LitIdxNameXHuffValIdxd 26
      8 LitIdxNameXHuffValIdxd 23
      8 LitIdxNameXHuffValIdxd 21
      8 LitIdxNameXHuffValIdxd 110
      8 LitIdxName1LitValNeverIdx 14
      8 LitIdxName1LitValNeverIdx 103
      8 LitIdxName1HuffValIdxd 42
      8 LitHuffNameHuffValIdxd 34
      8 LitHuffNameHuffValIdxd 32
      8 LitHuffNameHuffValIdxd 29
      7 LitIdxNameXHuffValIdxd 35
      7 LitIdxName1LitValNeverIdx 107
      7 LitIdxName1HuffValIdxd 49
      7 LitIdxName1HuffVal 15
      7 LitHuffNameHuffValIdxd 12
      6 LitIdxNameXHuffValIdxd 62
      6 LitIdxNameXHuffValIdxd 58
      6 LitIdxNameXHuffValIdxd 5
      6 LitIdxNameXHuffValIdxd 44
      6 LitIdxNameXHuffValIdxd 106
      6 LitIdxName1LitValNeverIdx 86
      6 LitIdxName1LitValNeverIdx 150
      6 LitIdxName1LitValIdxd 26
      6 LitIdxName1HuffValIdxd 45
      6 LitIdxName1HuffVal 37
      6 LitHuffNameHuffValIdxd 66
      6 LitHuffNameHuffValIdxd 41
      6 LitHuffNameHuffValIdxd 40
      6 LitHuffNameHuffValIdxd 19
      5 LitIdxNameXHuffValIdxd 9
      5 LitIdxNameXHuffValIdxd 78
      5 LitIdxNameXHuffValIdxd 38
      5 LitIdxNameXHuffValIdxd 29
      5 LitIdxNameXHuffValIdxd 17
      5 LitIdxNameXHuffValIdxd 15
      5 LitIdxNameXHuffValIdxd 13
      5 LitIdxName1LitValNeverIdx 84
      5 LitIdxName1LitValNeverIdx 76
      5 LitIdxName1LitValNeverIdx 356
      5 LitIdxName1LitValNeverIdx 143
      5 LitIdxName1HuffValIdxd 64
      5 LitIdxName1HuffValIdxd 61
      5 LitIdxName1HuffValIdxd 55
      5 LitIdxName1HuffValIdxd 36
      5 LitIdxName1HuffVal 35
      5 LitIdxName1HuffVal 27
      5 LitHuffNameHuffValIdxd 80
      5 LitHuffNameHuffValIdxd 50
      4 LitIdxNameXHuffValIdxd 91
      4 LitIdxNameXHuffValIdxd 86
      4 LitIdxNameXHuffValIdxd 57
      4 LitIdxNameXHuffValIdxd 20
      4 LitIdxNameXHuffValIdxd 18
      4 LitIdxNameXHuffValIdxd 154
      4 LitIdxNameXHuffValIdxd 125
      4 LitIdxName1LitValNeverIdx 88
      4 LitIdxName1LitValNeverIdx 64
      4 LitIdxName1LitValNeverIdx 199
      4 LitIdxName1LitValNeverIdx 169
      4 LitIdxName1LitValNeverIdx 142
      4 LitIdxName1LitValNeverIdx 118
      4 LitIdxName1LitValNeverIdx 113
      4 LitIdxName1HuffValIdxd 53
      4 LitIdxName1HuffVal 34
      4 LitIdxName1HuffVal 33
      4 LitIdxName1HuffVal 14
      4 LitIdxName1HuffVal 13
      4 LitHuffNameHuffValIdxd 49
      4 LitHuffNameHuffValIdxd 43
      4 LitHuffNameHuffValIdxd 33
      4 LitHuffNameHuffValIdxd 31
      3 LitIdxNameXHuffValIdxd 77
      3 LitIdxNameXHuffValIdxd 75
      3 LitIdxNameXHuffValIdxd 71
      3 LitIdxNameXHuffValIdxd 61
      3 LitIdxNameXHuffValIdxd 48
      3 LitIdxNameXHuffValIdxd 33
      3 LitIdxNameXHuffValIdxd 28
      3 LitIdxNameXHuffValIdxd 180
      3 LitIdxName1LitValNeverIdx 97
      3 LitIdxName1LitValNeverIdx 90
      3 LitIdxName1LitValNeverIdx 63
      3 LitIdxName1LitValNeverIdx 215
      3 LitIdxName1LitValNeverIdx 200
      3 LitIdxName1LitValNeverIdx 167
      3 LitIdxName1LitValNeverIdx 160
      3 LitIdxName1LitValNeverIdx 120
      3 LitIdxName1LitValNeverIdx 109
      3 LitIdxName1LitValNeverIdx 105
      3 LitIdxName1HuffValIdxd 82
      3 LitIdxName1HuffValIdxd 78
      3 LitIdxName1HuffVal 36
      3 LitIdxName1HuffVal 20
      3 LitHuffNameHuffValIdxd 9
      3 LitHuffNameHuffValIdxd 86
      3 LitHuffNameHuffValIdxd 8
      3 LitHuffNameHuffValIdxd 68
      3 LitHuffNameHuffValIdxd 67
      3 LitHuffNameHuffValIdxd 65
      3 LitHuffNameHuffValIdxd 51
      3 LitHuffNameHuffValIdxd 42
      2 LitIdxNameXHuffValIdxd 81
      2 LitIdxNameXHuffValIdxd 8
      2 LitIdxNameXHuffValIdxd 73
      2 LitIdxNameXHuffValIdxd 68
      2 LitIdxNameXHuffValIdxd 64
      2 LitIdxNameXHuffValIdxd 56
      2 LitIdxNameXHuffValIdxd 55
      2 LitIdxNameXHuffValIdxd 27
      2 LitIdxNameXHuffValIdxd 105
      2 LitIdxNameXHuffValIdxd 104
      2 LitIdxName1LitValNeverIdx 92
      2 LitIdxName1LitValNeverIdx 85
      2 LitIdxName1LitValNeverIdx 79
      2 LitIdxName1LitValNeverIdx 71
      2 LitIdxName1LitValNeverIdx 54
      2 LitIdxName1LitValNeverIdx 340
      2 LitIdxName1LitValNeverIdx 305
      2 LitIdxName1LitValNeverIdx 197
      2 LitIdxName1LitValNeverIdx 186
      2 LitIdxName1LitValNeverIdx 183
      2 LitIdxName1LitValNeverIdx 149
      2 LitIdxName1LitValNeverIdx 144
      2 LitIdxName1LitValNeverIdx 131
      2 LitIdxName1LitValNeverIdx 127
      2 LitIdxName1LitValNeverIdx 117
      2 LitIdxName1LitValNeverIdx 116
      2 LitIdxName1LitValNeverIdx 110
      2 LitIdxName1LitValNeverIdx 108
      2 LitIdxName1LitValNeverIdx 102
      2 LitIdxName1LitValIdxd 47
      2 LitIdxName1LitValIdxd 278
      2 LitIdxName1LitValIdxd 125
      2 LitIdxName1LitValIdxd 10
      2 LitIdxName1HuffValIdxd 92
      2 LitIdxName1HuffValIdxd 72
      2 LitIdxName1HuffValIdxd 68
      2 LitIdxName1HuffValIdxd 52
      2 LitIdxName1HuffValIdxd 37
      2 LitIdxName1HuffValIdxd 2
      2 LitIdxName1HuffVal 51
      2 LitIdxName1HuffVal 31
      2 LitIdxName1HuffVal 29
      2 LitHuffNameHuffValIdxd 89
      2 LitHuffNameHuffValIdxd 88
      2 LitHuffNameHuffValIdxd 71
      2 LitHuffNameHuffValIdxd 61
      2 LitHuffNameHuffValIdxd 60
      2 LitHuffNameHuffValIdxd 39
      2 LitHuffNameHuffValIdxd 37
      2 LitHuffNameHuffValIdxd 10
      1 LitIdxNameXHuffValIdxd 95
      1 LitIdxNameXHuffValIdxd 92
      1 LitIdxNameXHuffValIdxd 89
      1 LitIdxNameXHuffValIdxd 80
      1 LitIdxNameXHuffValIdxd 79
      1 LitIdxNameXHuffValIdxd 76
      1 LitIdxNameXHuffValIdxd 67
      1 LitIdxNameXHuffValIdxd 4
      1 LitIdxNameXHuffValIdxd 209
      1 LitIdxNameXHuffValIdxd 175
      1 LitIdxNameXHuffValIdxd 120
      1 LitIdxNameXHuffValIdxd 107
      1 LitIdxNameXHuffValIdxd 100
      1 LitIdxName1LitValNeverIdx 99
      1 LitIdxName1LitValNeverIdx 94
      1 LitIdxName1LitValNeverIdx 80
      1 LitIdxName1LitValNeverIdx 74
      1 LitIdxName1LitValNeverIdx 59
      1 LitIdxName1LitValNeverIdx 45
      1 LitIdxName1LitValNeverIdx 44
      1 LitIdxName1LitValNeverIdx 397
      1 LitIdxName1LitValNeverIdx 39
      1 LitIdxName1LitValNeverIdx 372
      1 LitIdxName1LitValNeverIdx 324
      1 LitIdxName1LitValNeverIdx 301
      1 LitIdxName1LitValNeverIdx 293
      1 LitIdxName1LitValNeverIdx 289
      1 LitIdxName1LitValNeverIdx 285
      1 LitIdxName1LitValNeverIdx 281
      1 LitIdxName1LitValNeverIdx 277
      1 LitIdxName1LitValNeverIdx 273
      1 LitIdxName1LitValNeverIdx 269
      1 LitIdxName1LitValNeverIdx 266
      1 LitIdxName1LitValNeverIdx 23
      1 LitIdxName1LitValNeverIdx 225
      1 LitIdxName1LitValNeverIdx 219
      1 LitIdxName1LitValNeverIdx 214
      1 LitIdxName1LitValNeverIdx 209
      1 LitIdxName1LitValNeverIdx 189
      1 LitIdxName1LitValNeverIdx 184
      1 LitIdxName1LitValNeverIdx 179
      1 LitIdxName1LitValNeverIdx 176
      1 LitIdxName1LitValNeverIdx 174
      1 LitIdxName1LitValNeverIdx 172
      1 LitIdxName1LitValNeverIdx 162
      1 LitIdxName1LitValNeverIdx 159
      1 LitIdxName1LitValNeverIdx 154
      1 LitIdxName1LitValNeverIdx 138
      1 LitIdxName1LitValNeverIdx 134
      1 LitIdxName1LitValNeverIdx 133
      1 LitIdxName1LitValNeverIdx 129
      1 LitIdxName1LitValNeverIdx 126
      1 LitIdxName1LitValNeverIdx 124
      1 LitIdxName1LitValNeverIdx 121
      1 LitIdxName1LitValNeverIdx 106
      1 LitIdxName1LitValNeverIdx 101
      1 LitIdxName1LitValIdxd 89
      1 LitIdxName1LitValIdxd 68
      1 LitIdxName1LitValIdxd 57
      1 LitIdxName1LitValIdxd 53
      1 LitIdxName1LitValIdxd 40
      1 LitIdxName1LitValIdxd 366
      1 LitIdxName1LitValIdxd 36
      1 LitIdxName1LitValIdxd 347
      1 LitIdxName1LitValIdxd 34
      1 LitIdxName1LitValIdxd 299
      1 LitIdxName1LitValIdxd 262
      1 LitIdxName1LitValIdxd 249
      1 LitIdxName1LitValIdxd 177
      1 LitIdxName1LitValIdxd 134
      1 LitIdxName1LitValIdxd 102
      1 LitIdxName1HuffValIdxd 960
      1 LitIdxName1HuffValIdxd 87
      1 LitIdxName1HuffValIdxd 84
      1 LitIdxName1HuffValIdxd 81
      1 LitIdxName1HuffValIdxd 80
      1 LitIdxName1HuffValIdxd 77
      1 LitIdxName1HuffValIdxd 76
      1 LitIdxName1HuffValIdxd 75
      1 LitIdxName1HuffValIdxd 74
      1 LitIdxName1HuffValIdxd 73
      1 LitIdxName1HuffValIdxd 718
      1 LitIdxName1HuffValIdxd 70
      1 LitIdxName1HuffValIdxd 69
      1 LitIdxName1HuffValIdxd 674
      1 LitIdxName1HuffValIdxd 67
      1 LitIdxName1HuffValIdxd 65
      1 LitIdxName1HuffValIdxd 639
      1 LitIdxName1HuffValIdxd 63
      1 LitIdxName1HuffValIdxd 62
      1 LitIdxName1HuffValIdxd 60
      1 LitIdxName1HuffValIdxd 58
      1 LitIdxName1HuffValIdxd 567
      1 LitIdxName1HuffValIdxd 54
      1 LitIdxName1HuffValIdxd 522
      1 LitIdxName1HuffValIdxd 51
      1 LitIdxName1HuffValIdxd 493
      1 LitIdxName1HuffValIdxd 466
      1 LitIdxName1HuffValIdxd 445
      1 LitIdxName1HuffValIdxd 420
      1 LitIdxName1HuffValIdxd 377
      1 LitIdxName1HuffValIdxd 370
      1 LitIdxName1HuffValIdxd 342
      1 LitIdxName1HuffValIdxd 299
      1 LitIdxName1HuffValIdxd 255
      1 LitIdxName1HuffValIdxd 180
      1 LitIdxName1HuffValIdxd 178
      1 LitIdxName1HuffValIdxd 177
      1 LitIdxName1HuffValIdxd 141
      1 LitIdxName1HuffVal 3
      1 LitIdxName1HuffVal 25
      1 LitIdxName1HuffVal 21
      1 LitHuffNameHuffValIdxd 96
      1 LitHuffNameHuffValIdxd 93
      1 LitHuffNameHuffValIdxd 81
      1 LitHuffNameHuffValIdxd 72
      1 LitHuffNameHuffValIdxd 47
      1 LitHuffNameHuffValIdxd 44
      1 LitHuffNameHuffValIdxd 36
      1 LitHuffNameHuffValIdxd 35
      1 LitHuffNameHuffValIdxd 135
      1 LitHuffNameHuffValIdxd 128
      1 LitHuffNameHuffValIdxd 109

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 18 Jul 2014 16:31:29 +1000

>     690 IdxField 2                # 2B index to static table

s/static/dynamic/

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 18 Jul 2014 16:37:38 +1000

>     124 LitIdxNameXHuffValIdxd 34

Sorry to drip feed comments....

NameX here means more than 1 byte needed to encode the name index.   So the
most frequent occurrence of
that is 124 and in total there were 1168(3%) of fields that required a >1
byte name index lookup.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 18 Jul 2014 16:56:28 +1000

Willy,

and just some more data for you,  here is the frequencies broken up by
field name and encoding type.   Kind of suggests that I shouldn't be
putting content-length into the header table... at least not for general
values:

   2852 :status StaticIndexed
   2457 expires LitIdxName1HuffVal
   2267 content-length LitIdxName1HuffValIdxd
   2215 connection IdxField1
   2213 last-modified LitIdxName1HuffVal
   2197 date IdxField1
   2181 content-type IdxField1
   1983 server IdxField1
   1469 cache-control LitIdxName1HuffValIdxd
   1250 cache-control IdxField1
   1131 content-encoding IdxField1
   1056 accept-ranges IdxField1
    939 vary IdxField1
    676 date LitIdxName1HuffValIdxd
    657 content-type LitIdxName1HuffValIdxd
    552 age LitIdxName1HuffValIdxd
    456 p3p IdxField1
    448 etag LitIdxName1HuffVal
    428 server LitIdxName1HuffValIdxd
    416 nncoection IdxField1
    405 pragma IdxField1
    395 transfer-encoding IdxField1
    384 set-cookie LitIdxName1LitValNeverIdx
    379 cneonction IdxField1
    349 :scheme StaticIndexed
    349 :method StaticIndexed
    335 user-agent IdxField1
    334 accept-language IdxField1
    334 accept-encoding IdxField1
    310 accept IdxField1
    291 content-length IdxField1
    279 :path LitIdxName1HuffValIdxd
    261 referer IdxField1
    249 x-cache IdxField1
    240 via IdxField1
    235 :authority IdxField1
    231 x-amz-cf-id LitIdxNameXHuffValIdxd
    219 x-powered-by IdxField1
    176 p3p LitIdxNameXHuffValIdxd
    175 via LitIdxName1HuffValIdxd
    150 access-control-allow-origin IdxField1
    149 connection LitIdxNameXHuffValIdxd
    143 vary LitIdxName1HuffValIdxd
    131 x-content-type-options IdxField1
    130 x-cache LitIdxNameXHuffValIdxd
    127 x-cache-lookup LitIdxNameXHuffValIdxd
    114 :authority LitIdxName1HuffValIdxd
    111 content-encoding LitIdxName1HuffValIdxd
    102 age IdxField1
    100 server IdxField2
     93 content-type IdxField2
     93 accept-ranges LitIdxName1HuffValIdxd
     90 x-aspnet-version IdxField1
     87 pragma LitHuffNameHuffValIdxd
     85 connection IdxField2
     84 content-encoding IdxField2
     73 transfer-encoding LitIdxName1HuffValIdxd
     70 connection LitHuffNameHuffValIdxd
     69 s IdxField1
     69 location LitIdxName1HuffValIdxd
     64 cookie IdxField1
     60 x-fb-debug LitIdxNameXHuffValIdxd
     59 vary IdxField2
     57 x-pad IdxField1
     55 accept-ranges IdxField2
     46 x-frame-options IdxField1
     46 p3p LitHuffNameHuffValIdxd
     44 :path StaticIndexed
     43 :status LitIdxName1HuffValIdxd
     42 x-xss-protection IdxField1
     39 x-content-type-options LitHuffNameHuffValIdxd
     39 s LitIdxNameXHuffValIdxd
     39 referer LitIdxName1HuffValIdxd
     39 access-control-allow-origin LitIdxName1HuffValIdxd
     36 transfer-encoding IdxField2
     36 cneonction LitHuffNameHuffValIdxd
     34 date IdxField2
     34 accept LitIdxName1HuffValIdxd
     33 x-powered-by LitHuffNameHuffValIdxd
     33 x-cache-lookup IdxField1
     33 cache-control IdxField2
     32 nncoection LitHuffNameHuffValIdxd
     31 x-cache LitHuffNameHuffValIdxd
     31 x-amz-id-2 LitIdxNameXHuffValIdxd
     30 x-xss-protection LitHuffNameHuffValIdxd
     30 content-language IdxField1
     29 keep-alive LitHuffNameHuffValIdxd
     29 cookie LitIdxName1LitValIdxd
     28 rlogid LitIdxNameXHuffValIdxd
     27 location IdxField1
     26 nncoection IdxField2
     26 :path IdxField1
     23 x-cdn IdxField1
     23 cneonction IdxField2
     23 :status IdxField1
     22 x-cnection IdxField1
     22 ntcoent-length IdxField1
     21 x-aspnet-version LitHuffNameHuffValIdxd
     21 pragma IdxField2
     20 x-li-uuid LitIdxNameXHuffValIdxd
     19 x-powered-by LitIdxNameXHuffValIdxd
     19 x-frame-options LitHuffNameHuffValIdxd
     17 x-served-by LitHuffNameHuffValIdxd
     17 x-amz-id-1 LitIdxNameXHuffValIdxd
     16 keep-alive LitIdxNameXHuffValIdxd
     15 x-proc-data LitHuffNameHuffValIdxd
     15 x-amz-id-2 LitHuffNameHuffValIdxd
     15 cteonnt-length LitHuffNameHuffValIdxd
     13 x-pad LitHuffNameHuffValIdxd
     13 cteonnt-length LitIdxNameXHuffValIdxd
     13 content-language LitIdxName1HuffValIdxd
     12 x-fb-debug LitIdxName1HuffValIdxd
     12 p3p LitIdxName1HuffValIdxd
     12 p3p IdxField2
     11 x-served-by LitIdxNameXHuffValIdxd
     11 x-amz-request-id LitIdxNameXHuffValIdxd
     11 user-agent LitIdxName1HuffValIdxd
     11 content-disposition LitIdxName1HuffValIdxd
     10 x-amz-id-1 LitHuffNameHuffValIdxd
     10 ntcoent-length LitHuffNameHuffValIdxd
     10 accept-language LitIdxName1HuffValIdxd
     10 accept-encoding LitIdxName1HuffValIdxd
      9 x-fs-uuid LitIdxNameXHuffValIdxd
      9 x-flickr-static LitHuffNameHuffValIdxd
      9 ts LitHuffNameHuffValIdxd
      9 tracking-status LitHuffNameHuffValIdxd
      8 x-radid LitHuffNameHuffValIdxd
      8 x-lb-nocache LitHuffNameHuffValIdxd
      8 x-amz-request-id LitHuffNameHuffValIdxd
      8 ntcoent-length LitIdxNameXHuffValIdxd
      8 keep-alive IdxField1
      8 if-modified-since LitIdxName1HuffVal
      8 content-disposition IdxField1
      7 x-request-time LitHuffNameHuffValIdxd
      7 x-powered-by IdxField2
      7 x-li-uuid IdxField1
      7 x-fs-uuid IdxField1
      7 x-flickr-static IdxField1
      7 x-fb-server LitIdxNameXHuffValIdxd
      7 x-fb-metrics LitIdxNameXHuffValIdxd
      7 x-cache-age LitHuffNameHuffValIdxd
      7 x-cache-action LitHuffNameHuffValIdxd
      7 cteonnt-length IdxField1
      7 content-length IdxField2
      6 x-via LitIdxNameXHuffValIdxd
      6 x-cnection LitHuffNameHuffValIdxd
      6 x-amzn-requestid LitHuffNameHuffValIdxd
      6 tracecode IdxField1
      6 s LitHuffNameHuffValIdxd
      6 pragma-directive LitHuffNameHuffValIdxd
      6 cache-directive LitHuffNameHuffValIdxd
      5 x-ua-compatible LitHuffNameHuffValIdxd
      5 x-aspnetmvc-version LitHuffNameHuffValIdxd
      5 transaction IdxField1
      5 rlogid IdxField1
      4 x-xss-protection LitIdxNameXHuffValIdxd
      4 x-radid LitIdxNameXHuffValIdxd
      4 x-msadid LitHuffNameHuffValIdxd
      4 x-lb-nocache IdxField1
      4 x-host LitHuffNameHuffValIdxd
      4 x-cdn LitHuffNameHuffValIdxd
      4 x-cache-lookup LitHuffNameHuffValIdxd
      4 x-cache-age IdxField1
      4 x-cache-action IdxField1
      4 x-amz-meta-md5-hash LitIdxNameXHuffValIdxd
      4 x-amz-meta-jets3t-original-file-date-iso8601 LitIdxNameXHuffValIdxd
      4 x-amz-cf-id LitIdxName1HuffValIdxd
      4 x-amz-cf-id LitHuffNameHuffValIdxd
      4 ts LitIdxNameXHuffValIdxd
      4 tracecode LitHuffNameHuffValIdxd
      4 status LitHuffNameHuffValIdxd
      4 rendertime IdxField1
      4 content-transfer-encoding LitHuffNameHuffValIdxd
      4 allow LitIdxName1HuffValIdxd
      4 allow IdxField1
      4 access-control-allow-origin IdxField2
      3 xserver LitHuffNameHuffValIdxd
      3 x-rightmedia-hostname LitHuffNameHuffValIdxd
      3 x-request-backend LitHuffNameHuffValIdxd
      3 x-radid IdxField1
      3 x-ocsp-reponder-id LitHuffNameHuffValIdxd
      3 x-msadid LitIdxNameXHuffValIdxd
      3 x-li-uuid LitIdxName1HuffValIdxd
      3 x-cache-hits LitHuffNameHuffValIdxd
      3 x-c LitHuffNameHuffValIdxd
      3 x-aspnet-version IdxField2
      3 x-age LitHuffNameHuffValIdxd
      3 transaction LitHuffNameHuffValIdxd
      3 tracking-status IdxField1
      3 rlogid LitHuffNameHuffValIdxd
      3 pragma LitIdxNameXHuffValIdxd
      3 machine LitIdxNameXHuffValIdxd
      3 machine IdxField1
      3 errorcodecount LitHuffNameHuffValIdxd
      3 edge-control LitHuffNameHuffValIdxd
      3 content-location LitIdxName1HuffValIdxd
      3 access-control-allow-methods LitHuffNameHuffValIdxd
      3 access-control-allow-headers LitHuffNameHuffValIdxd
      2 xserver LitIdxNameXHuffValIdxd
      2 x-ua-compatible IdxField1
      2 x-served-by IdxField1
      2 x-request-time LitIdxNameXHuffValIdxd
      2 x-rendertime LitIdxNameXHuffValIdxd
      2 x-proc-data LitIdxNameXHuffValIdxd
      2 x-ocsp-reponder-id IdxField1
      2 x-mii-cache-hit LitHuffNameHuffValIdxd
      2 x-kuid LitHuffNameHuffValIdxd
      2 x-host LitIdxNameXHuffValIdxd
      2 x-fb-server LitHuffNameHuffValIdxd
      2 x-fb-server IdxField1
      2 x-fb-metrics LitHuffNameHuffValIdxd
      2 x-fb-metrics IdxField1
      2 x-fb-debug LitHuffNameHuffValIdxd
      2 x-aspnetmvc-version IdxField1
      2 x-aspnet-version LitIdxNameXHuffValIdxd
      2 x-amzn-requestid LitIdxNameXHuffValIdxd
      2 x-amzn-requestid IdxField1
      2 x-amz-meta-md5-hash LitHuffNameHuffValIdxd
      2 x-amz-meta-jets3t-original-file-date-iso8601 LitHuffNameHuffValIdxd
      2 x-amz-id-2 IdxField1
      2 x-amz-id-1 IdxField1
      2 transaction LitIdxNameXHuffValIdxd
      2 tracecode IdxField2
      2 sts LitHuffNameHuffValIdxd
      2 s LitIdxName1HuffValIdxd
      2 rlogid LitIdxName1HuffValIdxd
      2 rendertime LitIdxNameXHuffValIdxd
      2 media LitHuffNameHuffValIdxd
      2 if-none-match LitIdxName1HuffVal
      2 dcs LitIdxNameXHuffValIdxd
      2 dcs LitHuffNameHuffValIdxd
      2 cteonnt-length LitIdxName1HuffValIdxd
      2 bk-server LitHuffNameHuffValIdxd
      1 x-xss-protection IdxField2
      1 x-via LitHuffNameHuffValIdxd
      1 x-transaction LitHuffNameHuffValIdxd
      1 x-sap-pg LitHuffNameHuffValIdxd
      1 x-rightmedia-hostname LitIdxNameXHuffValIdxd
      1 x-requested-with LitHuffNameHuffValIdxd
      1 x-rendertime LitHuffNameHuffValIdxd
      1 x-radid LitIdxName1HuffValIdxd
      1 x-permitted-cross-domain-policies LitHuffNameHuffValIdxd
      1 x-pal-host LitHuffNameHuffValIdxd
      1 x-pal-host IdxField1
      1 x-pad IdxField2
      1 x-msadid IdxField1
      1 x-li-uuid LitHuffNameHuffValIdxd
      1 x-hello LitHuffNameHuffValIdxd
      1 x-fs-uuid LitHuffNameHuffValIdxd
      1 x-fb-debug IdxField1
      1 x-ebay-request-id LitHuffNameHuffValIdxd
      1 x-config-age LitHuffNameHuffValIdxd
      1 x-cnection IdxField2
      1 x-cdn IdxField2
      1 x-c IdxField1
      1 x-amzn-requestid LitIdxName1HuffValIdxd
      1 x-amz-id-2 LitIdxName1HuffValIdxd
      1 x-amz-id-1 LitIdxName1HuffValIdxd
      1 vtag LitHuffNameHuffValIdxd
      1 trackingid LitHuffNameHuffValIdxd
      1 tracking-status LitIdxNameXHuffValIdxd
      1 tracecode LitIdxNameXHuffValIdxd
      1 tracecode LitIdxName1HuffValIdxd
      1 tcn LitHuffNameHuffValIdxd
      1 sts IdxField1
      1 status IdxField1
      1 rlog LitHuffNameHuffValIdxd
      1 rendertime LitHuffNameHuffValIdxd
      1 machine LitHuffNameHuffValIdxd
      1 location IdxField2
      1 dl_s LitIdxNameXHuffValIdxd
      1 dl_s LitHuffNameHuffValIdxd
      1 dl_s IdxField1
      1 cw-server LitHuffNameHuffValIdxd
      1 cteonnt-length IdxField2
      1 content-transfer-encoding IdxField1
      1 content-location IdxField1
      1 connection LitIdxName1HuffValIdxd
      1 access-control-allow-credentials LitHuffNameHuffValIdxd

On 18 July 2014 16:37, Greg Wilkins <[email-masked]> wrote:

> NameX here means more than 1 byte needed to encode the name index.   So
> the most frequent occurrence of

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 18 Jul 2014 09:36:44 +0200

On Fri, Jul 18, 2014 at 04:56:28PM +1000, Greg Wilkins wrote:
> field name and encoding type.

Thanks for this break-down and the other one. You convinced me :-)

> Kind of suggests that I shouldn't be
> putting content-length into the header table...
> at least not for general values:
(...)

That's a good point indeed since there's little chance it will remain
identical across multiple messages, so that would needlessly pollute
the header table after several responses. In practice, if responses
are looking similar and all headers fit into the dynamic table, it
should have no impact since each name:value pair will not be reused
and will be automatically evicted.

Best regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jul 2014 15:00:01 -0400

Based upon what we know now, we appear to have consensus to remove the reference set from HPACK.

If new data shows up that convinces the WG to change its mind, we can consider that then.

I’m marking this issue as editor-ready, with the expectation that the editors will work out the exact way to do this; if there are issues with how they go about it, we can discuss that afterwards.

Regards,

--
Mark Nottingham   http://www.mnot.net/