Subject: Streaming Header Compression

From: Mike Bishop <[email-masked]>
Date: Tue, 27 Feb 2018 18:31:05 +0000

During the HTTP/QUIC header compression discussions, we've been discussing whether there's a requirement to have streaming / single-pass header compressors or if it's reasonable to assume that an encoder has access to the full header set at once.  I know during the HPACK design process, streaming decoders were a priority, but I don't recall anything about streaming encoders.  So I'd like to consult the HTTP community to see whether implementations have requirements around this.

Because HEADERS frames are length-prefixed, you have to know the full content of the frame before you can send any of it.  In HTTP/2, there is a way around this - CONTINUATION frames can be used for subsequent chunks as you generate them.  However, I don't know that I've heard of anyone using CONTINUATION for this purpose.

In HTTP/QUIC, we've removed CONTINUATION frames in favor of much larger frame lengths, and to general approval because CONTINUATION was a source of bugs and angst.  However, that means it's no longer possible to write the beginning of the frame without having seen and processed the entire header set.  Is that a change we need to revisit?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 27 Feb 2018 21:06:32 +0000

--------
In message <[email-masked]>, Mike Bishop writes:

>However, that means it's no longer possible to write the beginning
>of the frame without having seen and processed the entire header
>set.

Speaking for the Varnish Cache project:  We are fine with that.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Feb 2018 05:30:17 +0100

> Speaking for the Varnish Cache project:  We are fine with that.

Same for haproxy, we always process all headers at once.

And BTW, we have still not implemented support for CONTINUATION frames
because in the current model they would cause too many problems,
precisely because we process headers at once. No negative feedback for
this yet, only h2spec noticed, possibly indicating that they are not
that much used in field.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 28 Feb 2018 23:14:56 +1300

> this yet, only h2spec noticed, possibly indicating that they are not

Squid is the same in this regard. Always wait for the entire header
block before any processing beyond parse happens. That is true
regardless of HTTP version being used.

Amos