Subject: RE: CONTINUATION proposal w/ minimum change

From: <[email-masked]>
Date: Tue, 1 Jul 2014 10:38:29 +0000

Responding to everybody at once...

> Martin said:

> I think that there needs to be some text regarding the crossover between

> HEADERS and CONTINUATION. Particularly the case where you have an opcode

> that is followed by a literal that crosses into a CONTINUATION. This needs

> to be clarified. That is, the opcode and all of the literal values

> associated with the opcode MUST fit within the initial frame.

I think this is the easiest solution (i.e. not allowing splits across HEADERS to CONTINUATION boundary). But as Michael points out, you have to at least allow splits across the CONTINUATION to CONTINUATION boundaries in case of a single header >64KB.

What is preferred?

1. No splits across the first boundary; splits allowed on CONTINUATION to CONTINUATION boundaries.

2. Splits allowed on any boundary, but splits across the first boundary must be a non-state-changing opcode.

> Martin said:

> ... the reference set is emitted at the end of the HEADERS/PUSH_PROMISE

> frame, even if there is a partially completed opcode in flight.

Yes, that was a mistake. See above regarding “opcode in flight”.

> Mike & Jason & Herve said:

> Did you intend to allow static table references? Those do not affect

> compression state when the effective table size is 0...

Yes, I completely forgot about static references.  I will definitely add that.

> Herve said:

> ...CONTINUATION frames do not affect compression state, but may use it:

> in HPACK a literal header representation can either use a literal or a

> reference for the header name.

But only static references right? If you reference something that's not static it could be changed by a HEADERS frame on another stream or am I missing something?

> Michael said:

> It feels very awkward to make the compressor aware of framing.  Previously,

> we had said that you take your header set, run it through HPACK, then frame

> up and send the output using HEADERS/PUSH_PROMISE and CONTINUATION.  Now,

> the HPACK compressor has to know where the boundary is and change its

> behavior when it passes that boundary.  I assume that means on the

> decompression side that the reference set is emitted when the frame boundary

> is hit, which means the decompressor also needs to be aware of framing.

These are really good points and trade-offs of this proposal. Jason had reasonable ideas for dealing with them, but in general my preference is to also have loose coupling between layers.

I will point out though that having CONTINUATION frames at all is essentially a coupling between the layers.  If it weren't for the shared hpack state, we would just have HEADERS and DATA frames.

> Michael said:

> With this proposal, I can use flow control to control how much header data

> I receive at once, but I can't drain that filled space by sending it to the

> app, because I'm still waiting for the rest of the header.  That means a

> server can't use "application read" as the trigger to send a WINDOW_UPDATE,

> because it's not passing anything to the app, just realizing that there's

> more to read in before it can, and RTTs are burning while that happens.

Also a really good point.  However, I'll point out again that CONTINUATIONS are supposed to be the exception rather than the rule. According to Richard's data, only 0.02% of requests and 0.006% of requests have headers >16KB. If the performance is worse for large header sets, that might be a good thing - it acts as a disincentive to use large headers.

> Jason said:

> I do think it was unintentional that the wording in Keith's proposal allows

> muxing on the same stream. I don't think that makes a lot of sense to do.

> & Martin said:

> I think that you still need text here to point out that the current stream

> cannot be continued until the header block is complete. That is, you can't

> send PUSH_PROMISE [-END_HEADERS] then send

> DATA/HEADERS/PUSH_PROMISE, you have to send

> PUSH_PROMISE [-END_HEADERS], CONTINUATION...[+END_HEADERS]

> before sending DATA.

Yes, that was a mistake.  On that topic though, I'll point out that one of the more confusing parts of the h2 draft for me is the sequence of frames.

For HTTP it only makes sense to have:

HEADERS [CONTINUATION] DATA [HEADERS [CONTINUATION]]

...but it seems unclear to me if HEADERS and CONTINUATION are allowed to be randomly interspersed with the DATA frames. If so, for what? Metadata? Must the metadata be formatted in name-value pairs and encoded with hpack?

(Also a question regarding what you said Martin. Isn’t there another HEADERS [CONTINUATION] sequence for the response headers after PUSH_PROMISE?)

> Matthew said:

> I suspect it would be less disruptive to completely replace CONTINUATIONS

> with a whole new frame with its own stand-alone internal encoding. Hell,

> if the concatenated payloads of the CONTINUATION frames was just a

> traditional "h1: v1\r\nh2: v2\r\n\r\n" string, that'd get the data out

I really like the length-prefix of hpack.  I'd rather not go back to this.

**Is it worth continuing with this proposal?**

I think Michael brought up some really valid points.

Assuming the following changes:

+ opcodes and their associated literal values MUST fit within the initial HEADERS frame

+ opcodes and their associated literal values MAY span CONTINUATION frames

+ static table references are OK in CONTINUATION

+ same-stream muxing between HEADERS and CONTINUATION is disallowed

+ reference set emitted at the end of HEADERS/PUSH_PROMISE

Does anyone thing this proposal is still worth pursuing?

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: RUELLAN Herve <[email-masked]>
Date: Tue, 1 Jul 2014 13:12:54 +0000

> But only static references right? If you reference something that's not static it
> could be changed by a HEADERS frame on another stream or am I missing
> something?

My point is that CONTINUATION frames may depend on the compression state, for example by using a reference to an entry of the dynamic header table for the header name. This dependency enforce an order between CONTINUATION frames and HEADER frames from other streams.

I see two possibilities for removing this dependency:
- Allow only literal header representations, not indexed, and using a literal for the header name.
- Use a specific compression state for the CONTINUATION frames where the header table size is 0: this would allow referencing entries of the static table (either as a whole, or only for the header name).

Hervé.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Tue, 1 Jul 2014 22:24:09 +0900

On Tue, Jul 1, 2014 at 7:38 PM, <[email-masked]> wrote:
>
>   ***Is it worth continuing with this proposal?***
>
>
>

​Personally I prefer the current CONTINUATION spec to the proposed one.
The proposed solution removes some restrictions, but introduces lots of
complexities.
And those complexities are just for "​only 0.02% of requests and 0.006% of
requests" in the world.
I think it probably does not worth the cost.

The servers always have the power to terminate connection if header size is
too large for them.
It is already a good incentive and pressure for peer not to abuse HEADERS,
because large request will result in connection lost with higher
probability.

We already know headers > 16K (e.g., Kerberos), so we need CONTINUATION for
them in anyway.
Flow controlling CONTINUATION unfairly penalizes those valid HTTP requests
by arbitrarily delaying transmission because of flow control and
scheduling.  In addition to this, as already discussed, headers are more
likely kept in memory while reading entire request, so applying flow
control for them just complicates both specification and implementation
without good value.

Best regards,

Tatsuhiro Tsujikawa

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 14:16:19 +0000

In message <CAPyZ6=[email-masked]>, Tatsuhiro Tsujikawa write

>We already know headers > 16K (e.g., Kerberos), so we need CONTINUATION for

... or jumbo frames, which would be so much simpler in the first place.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Jason Greene <[email-masked]>
Date: Tue, 1 Jul 2014 09:48:48 -0500

On Jul 1, 2014, at 8:24 AM, Tatsuhiro Tsujikawa <[email-masked]> wrote:

> The proposed solution removes some restrictions, but introduces lots of complexities.
> And those complexities are just for "​only 0.02% of requests and 0.006% of requests" in the world.
> The servers always have the power to terminate connection if header size is too large for them.

Can they? The blacklist proposal that was recently suggested today suggests that browsers won’t talk to them if they actually do.

> It is already a good incentive and pressure for peer not to abuse HEADERS, because large request will result in connection lost with higher probability.
> 
> We already know headers > 16K (e.g., Kerberos), so we need CONTINUATION for them in anyway.
> Flow controlling CONTINUATION unfairly penalizes those valid HTTP requests by arbitrarily delaying transmission because of flow control and scheduling.  In addition to this, as already discussed, headers are more likely kept in memory while reading entire request, so applying flow control for them just complicates both specification and implementation without good value.

How does it arbitrarily delayed. Senders have pressure to send them as fast as they can. The only delay flow control has is allowing other streams to progress, which is exactly what should happen in a multiplexed protocol.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Wed, 2 Jul 2014 00:10:38 +0900

On Tue, Jul 1, 2014 at 11:48 PM, Jason Greene <[email-masked]>
wrote:

> Can they? The blacklist proposal that was recently suggested today
> suggests that browsers won’t talk to them if they actually do.
>
>
​I don't remind the blacklist proposal, but
today, nginx, apache​ and other servers have their own limit regarding
header fields.
HTTP/2 enabled servers are no exception here.

>
> > It is already a good incentive and pressure for peer not to abuse
> HEADERS, because large request will result in connection lost with higher
> probability.
> >
> > Flow controlling CONTINUATION unfairly penalizes those valid HTTP
> requests by arbitrarily delaying transmission because of flow control and
> How does it arbitrarily delayed. Senders have pressure to send them as
> fast as they can. The only delay flow control has is allowing other streams
> to progress, which is exactly what should happen in a multiplexed protocol.
>
>
​Connection-level flow control fully blocks CONTINUATION and DATA in a
connection.  Transmission of CONTINUATION​ is delayed until WINDOW_UPDATE
is received.​  Also, in client scheduler, it may get very small send window
in contention.  If we can send entire request headers without flow control,
we have less block and server can use their time more useful.

Best regards,

Tatsuhiro Tsujikawa

---

From: Jason Greene <[email-masked]>
Date: Tue, 1 Jul 2014 11:07:20 -0500

On Jul 1, 2014, at 10:10 AM, Tatsuhiro Tsujikawa <[email-masked]> wrote:

> 
> 
> 
> On Tue, Jul 1, 2014 at 11:48 PM, Jason Greene <[email-masked]> wrote:
> 
> today, nginx, apache​ and other servers have their own limit regarding header fields.
> ​Connection-level flow control fully blocks CONTINUATION and DATA in a connection.  Transmission of CONTINUATION​ is delayed until WINDOW_UPDATE is received.​  Also, in client scheduler, it may get very small send window in contention.  If we can send entire request headers without flow control, we have less block and server can use their time more useful.

That’s not arbitrary delay, thats fair controlled delay. If a receiver wants more data it simply sets the window appropriately. As you mention, we are talking this 0.02% large cases here. Making the .02% stand in line like everyone else is *good*. I do agree with you that scheduling problems are bad, and there is a huge one with the current draft which allows one stream to starve the rest. So for the purpose of argument, lets say the flow control aspect is dropped. Do you still prefer a HOL blocking multiplexed protocol?

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Wed, 2 Jul 2014 02:12:55 +0900

On Wed, Jul 2, 2014 at 1:07 AM, Jason Greene <[email-masked]>
wrote:

> That’s not arbitrary delay, thats fair controlled delay. If a receiver
> wants more data it simply sets the window appropriately. As you mention, we
> are talking this 0.02% large cases here. Making the .02% stand in line like
> everyone else is *good*. I do agree with you that scheduling problems are
> bad, and there is a huge one with the current draft which allows one stream
> to starve the rest. So for the purpose of argument, lets say the flow
> control aspect is dropped. Do you still prefer a HOL blocking multiplexed
> protocol?
>
>
I'm sure that everyone ​wants to avoid HOL blocking.  But the thing is how
much it costs, at this stage.
As I stated earlier, this proposal adds more complexities than it removes
and it serves just for treating 0.02%.  I have to say the current
CONTINUATION spec is well balanced compromise if you ask.
I believe good intermediary implementation does not allow hugely abused
headers.
I'd like to see wide deployment of h2-13 to see whether or not CONTINUATION
really causes serious issue.

​Best regards,

Tatsuhiro Tsujikawa​

---

From: Jason Greene <[email-masked]>
Date: Tue, 1 Jul 2014 14:37:40 -0500

On Jul 1, 2014, at 12:12 PM, Tatsuhiro Tsujikawa <[email-masked]> wrote:
> I'm sure that everyone ​wants to avoid HOL blocking.  But the thing is how much it costs, at this stage.

Not much. A few lines in the spec, and some more intelligence in header splitting. 

> As I stated earlier, this proposal adds more complexities than it removes and it serves just for treating 0.02%.

It’s not about the .02%, it’s about get 99.8% not getting trumped by the .02%.

>  I have to say the current CONTINUATION spec is well balanced compromise if you ask.

There is no balance, it’s .02% rules all. I also have seen no compromise. There are a lot of proposals to fix it, and I think those that have repeatedly highlighted this issue are open to many different options. We just need the will to fix it.

> I believe good intermediary implementation does not allow hugely abused headers.

The intermediar's hands are tied. It has no idea what the origin will accept, and it has to forward what the content provided by the client.

Even if you shrug at the traffic prioritization problem, consider this situation. A proxy accepts 128K headers (that’s reasonable right?), while one of the origins only accepts 20K. If it coalesces connections (which is a big advantage to a multiplexed protocol), the following situation can happen

 1. Client A sends 128K request
 2. Client B sends 64K request
 3. Client C sends 48K request
 4. Client D sends 32K request
 5. All of the above send 16K requests
 6. Proxy relays 128K request to Origin
 7. Origin rejects and has to hang up the connection
 8. Proxy reestablishes the connection
 9. Proxy relays 64K request to Origin
10. Origin rejects and has to hang up the connection
11. Proxy reestablishes the connection
12. Proxy relays 32K request to Origin
13. Origin rejects and has to hang up the connection
14. Proxy reestablishes the connection
15. Finally the rest of the traffic goes through after potentially getting resent due to all of the drops

This can become more chaotic with multiple hops.

> I'd like to see wide deployment of h2-13 to see whether or not CONTINUATION really causes serious issue.

If you paint your car’s windshield black, do you need to drive it to know if it will be a problem?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 19:54:41 +0000

In message <[email-masked]>, Jason Greene writes:

>> As I stated earlier, this proposal adds more complexities than it
>> removes and it serves just for treating 0.02%.

Given the amount of complexity huge HTTP headers and the proposed
handing with CONTINUATION frames cause, I think it is time the WG
puts the horse in front of the cart:

Remove CONTINUATION from the draft.

Declare that the compressed headers can be no longer than 16383 bytes
in "vanilla" HTTP/2.

Applications needing longer headers can either remain in HTTP/1
(where it presumably already works for them), they can redesign
their application to use smaller headers or they can push a negotiable
HTTP/2.0 extension for sending bigger requests through the IETF.

Loosing 0.02% of the potential HTTP/2 traffic to that decision
is utterly trivial compared to the amount of traffic lost to
browers which will not support HTTP/1 upgrade.