Subject: Getting to Consensus: CONTINUATION-related issues

From: Mark Nottingham <[email-masked]>
Date: Fri, 18 Jul 2014 10:44:34 +1000

We've had a rollicking discussion about the design tradeoffs in CONTINUATION, especially regarding HOL blocking and DoS considerations.

I see very little new information entering that discussion, and I think everyone has come to understand the tradeoffs. For a refresher, please see the wiki:
  https://github.com/http2/http2-spec/wiki/ContinuationProposals

I proposed two options the other day:

a) Remove CONTINUATION from the specification and add a new setting that dictates the maximum HEADERS/PUSH_PROMISE frame size (as distinct from max_frame_size) a peer is willing to receive. I.e., the setting refers to the compressed header size.

b) Keep CONTINUATION in the specification, and add a new setting that advises the maximum header set size (i.e,. uncompressed) a peer is willing to receive (but might not imply PROTOCOL_ERROR or STREAM_ERROR on receipt).

Although there have been some tentative proposals for additional options since, I haven't heard a clamour for support for them, so I think these are realistically the ways we can go.

As stated before, there will no doubt be tweaking and adjustments made to these, but I think we're in a place where we can choose a general direction.

I'd like to hear:

1) Your preferred outcome (if any)
2) Whether you can live with the other option, and if not, why

"I have no preference" is useful information too.

If you indicate you can't live with one (or both) of the options, you MUST give a detailed, relevant reason as to why; omitting the reason means your "can't live with" will be ignored.

Thanks,

P.S. Please state *your* preference, not what you think the WG can live with.

P.P.S. This is not a call for more discussion; please resist replying to others' preferences.

--
Mark Nottingham   https://www.mnot.net/

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 18 Jul 2014 14:53:26 +1200

> 1) Your preferred outcome (if any)

(A).

> 2) Whether you can live with the other option, and if not, why

"CONTINUATION without a length limit" adds a great complexity to the
intermediary code and opens us to unacceptible levels of resource
commitments for the rare[1] chance of being useful. In 1.1 this is only
just coped with, in h2 with multiplexing it grows unacceptible.

"CONTINUATION with a length limit" is simpler both for spec and
implementation when written in the form of a Large Frame
unlimited-length value for use only by 1:1 "TCP tunnel" type proxying
where h2 as a whole ceases relevance.

[1] Specifically the unlimited length need is rare. As opposed to
knowable but large lengths, which are far more common.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 18 Jul 2014 13:15:40 +1000

> knowable but large lengths, which are far more common.

Hi Amos -- please be explicit -- is this the reasoning for your preference, or a statement of why you can't live with it?

(I ask because "can't live with it" is a big statement either way, and we need to make sure it's unambiguous when it's said).

Thanks,

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 18 Jul 2014 14:43:21 +1000

Prefer a)
Can live with b)

On 18 July 2014 10:44, Mark Nottingham <[email-masked]> wrote:

> We've had a rollicking discussion about the design tradeoffs in
> CONTINUATION, especially regarding HOL blocking and DoS considerations.
>
> I see very little new information entering that discussion, and I think
> everyone has come to understand the tradeoffs. For a refresher, please see
> the wiki:
> a) Remove CONTINUATION from the specification and add a new setting that
> dictates the maximum HEADERS/PUSH_PROMISE frame size (as distinct from
> max_frame_size) a peer is willing to receive. I.e., the setting refers to
> the compressed header size.
>
> b) Keep CONTINUATION in the specification, and add a new setting that
> advises the maximum header set size (i.e,. uncompressed) a peer is willing
> to receive (but might not imply PROTOCOL_ERROR or STREAM_ERROR on receipt).
>
> Although there have been some tentative proposals for additional options
> since, I haven't heard a clamour for support for them, so I think these are
> realistically the ways we can go.
>
> As stated before, there will no doubt be tweaking and adjustments made to
> these, but I think we're in a place where we can choose a general direction.
>
> I'd like to hear:
>
> If you indicate you can't live with one (or both) of the options, you MUST
> give a detailed, relevant reason as to why; omitting the reason means your
> "can't live with" will be ignored.
>
> Thanks,
>

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 18 Jul 2014 16:55:40 +1200

> (I ask because "can't live with it" is a big statement either way, and we need to make sure it's unambiguous when it's said).

Reasoning for a "prefer A". As you say, "cant live with" is hard to
justify when implementations obviously exist. It is "just" a massive
amount of trouble for an edge case that is easily avoidable via simpler
means.

Amos

---

From: Rob Trace <[email-masked]>
Date: Fri, 18 Jul 2014 05:04:22 +0000

We prefer outcome B as this will allow us to continue to support existing 1.1 applications with HTTP/2.

While we can live with option a, it will cause us a lot of complexity both in terms of implementation and in terms of interoperability with existing 1.1 applications.  In option a, the receiver controls the maximum frame size not the sender, so we cannot “ask” for an allowance and we have to live with what the receiver offers.  To make this work realistically, the sender will either have to create a new platform limit for existing applications or will have to create potentially complicated compression “unwind” or “preview” functionalities.

Thanks!!

-Rob

From: Greg Wilkins [mailto:[email-masked]]
Sent: Thursday, July 17, 2014 9:43 PM
To: HTTP Working Group
Subject: Re: Getting to Consensus: CONTINUATION-related issues

Prefer a)
Can live with b)

On 18 July 2014 10:44, Mark Nottingham <[email-masked]<mailto:[email-masked]>> wrote:
We've had a rollicking discussion about the design tradeoffs in CONTINUATION, especially regarding HOL blocking and DoS considerations.

I'd like to hear:

Thanks,

--
Greg Wilkins <[email-masked]<mailto:[email-masked]>>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Jason Greene <[email-masked]>
Date: Fri, 18 Jul 2014 00:10:29 -0500

> 1) Your preferred outcome (if any)

A: It is the cleanest solution, with the least complexity, and addresses all but a very limited use case. 

> 2) Whether you can live with the other option, and if not, why

I can live with B

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: <[email-masked]>
Date: Fri, 18 Jul 2014 05:27:01 +0000

Preference order:
- Z: hybrid of A+B*
- A
- B (can live with)

*Option Z (hybrid A+B): Remove CONTINUATION. No frame size limit on HEADERS/PUSH_PROMISE up to 2^24-1 (ie "unlimited headers" so no rewind necessary).  Compressed length is declared up front (ie buffer at sender). Advisory setting for uncompressed limit a peer is willing to receive (but might not imply PROTOCOL_ERROR or STREAM_ERROR on receipt)

-Keith

On Jul 18, 2014, at 7:09, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

Thanks!!

-Rob

Prefer a)
Can live with b)

I'd like to hear:

Thanks,

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Robert Collins <[email-masked]>
Date: Fri, 18 Jul 2014 17:30:29 +1200

> 1) Your preferred outcome (if any)

A for ease of implementation. Also because unlimited size requests are
already de facto broken in 1.1 due to intermediaries like squid having
fixed upper limits on parsing.

> 2) Whether you can live with the other option, and if not, why

I can live with b

> If you indicate you can't live with one (or both) of the options, you
MUST give a detailed, relevant reason as to why; omitting the reason means
your "can't live with" will be ignored.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 17 Jul 2014 22:37:41 -0700

Option a- I find this option to be an extremely poor option. I hate it, it
should not happen.

It increases aggregate complexity while increasing the DoS surface as it
would require state rewind in the compressor. This would impact scalability
in a very negative way, in terms of memory, and in terms of cpu (copying
the whole compression context would be expensive, and would need to be done
for every response unless one can accurately predict the compressed size).
Compressor rewind would often result in a doubling of the memory
requirements for the compressor while compressing, as the least error-prone
rewind is to make a copy of the complete compressor state. While more
efficient 'rewinds' are possible, the complexity becomes significantly
higher, and when done improperly, could result in state committments of
more than double the compression size (it could be as large as message-size
+ state size)

Option b-

 I could live with it. It acts against the right thing-- uncompressed size,
which implies no state rewind from the compressor, and, so long as the
setting can be set large, it can retain compatibility with HTTP/1.

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 18 Jul 2014 05:59:01 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

1)  Prefer a)

2)  No to b).

The lack of upfront indication of compressed header size puts the
DoS and memory management trouble where we least can afford it from
a performance point of view.

The only quasi-sensible argument for fragmented header transmission I
have heard pertains to server -> client communication at one particular
very well-funded website.

I have not seen a shred of evidence that "header-streaming" is
relevant to any request in client->server direction where our main
DoS/performance problems lie.

Imposing performance and DoS resistance costs on everybody else for
the sake of this one website makes no sense.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 18 Jul 2014 08:11:33 +0200

Hi Mark,

> 1) Your preferred outcome (if any)

I'd prefer a) but I think I can possibly live with b) (would be better
with the requirement that CONT are only allowed after full header frames).

I'm saying "I think" because I'm seeing comments from people who seem
violently against either option that I don't understand so I'm wondering
if I'm missing something.

To be clear: I want my receiver to be able to announce its limits (ideally
the uncompressed set, but also the max frame size), and not to have to deal
with these CONT frames which can block everything, or can force the receiver
to decompress one byte at a time. My understanding of your proposal is that
a) covers the second part and partially the fist one, and that b) covers the
first one only.

Willy

---

From: Shigeki Ohtsu <[email-masked]>
Date: Fri, 18 Jul 2014 15:31:16 +0900

I prefer b).

I can live with a) if we can avoid complexity to unwind compression 
contexts. That's too hard to manage HPACK.

Regards,

---

From: David Krauss <[email-masked]>
Date: Fri, 18 Jul 2014 15:30:28 +0800

On 2014–07–18, at 1:37 PM, Roberto Peon <[email-masked]> wrote:

> Option a- I find this option to be an extremely poor option. I hate it, it should not happen.
> 
> It increases aggregate complexity while increasing the DoS surface as it would require state rewind in the compressor.
> 
Mark, may limit violation in option A still allow recipients to either streaming-decode the rejected block, or GOAWAY? Always-GOAWAY or attempting to proceed as if the HEADERS never happened simply are not workable. But, I think the real vote here is about CONTINUATION, and nothing more detailed. (I tried to get details from the Wiki page, but can’t find apples-to-apples in any of it.)

I favor option A because it provides a better sweet spot for simpler implementations. However, the spec should also be clear that the preferred intermediary behavior is to ignore MAX_FRAME/MAX_HEADERS and forward headers indefinitely. This works without CONTINUATION, since now we have jumbo frames.

I can live with option B because I’ll eventually do streaming anyway. But, given the discussions of the past few months, I don’t believe that header streaming is a reasonable implementation requirement. Some folks just won’t be able, or motivated, to do it. In practice, CONTINUATION will be poorly supported, and B just devolves to A with a slightly more complicated binary format.

TL;DR: We added jumbo frames already, now let’s require them to fix what they were supposed to fix.

---

From: Mike Belshe <[email-masked]>
Date: Fri, 18 Jul 2014 01:16:20 -0700

> 2) Whether you can live with the other option, and if not, why

Desired outcome:  neither.

a) Cannot live with it.  As written here, this breaks compat with HTTP/1.1
and is a non-starter.
b) Can live with it, but recommend against the new setting proposed.

Desired outcome would remove continuations and have big frames.
 (continuations are required to be adjacent anyway, so instead just
increase the frame size - its simpler yet the same effect).

Mike

---

From: Cory Benfield <[email-masked]>
Date: Fri, 18 Jul 2014 09:27:36 +0100

> "I have no preference" is useful information too.

I have no meaningful preference. I've seen good arguments for both
proposals, and I can live with either. No substantial implementation
cost for me either way. I'm happy to let those in the WG to are
genuinely affected lead the way here.

---

From: <[email-masked]>
Date: Fri, 18 Jul 2014 08:26:48 +0000

On Friday,18 July 2014 10:16, [email-masked] wrote:
> (continuations are required to be adjacent anyway, so instead just increase the
> frame size - its simpler yet the same effect).

This is essentially the "Option Z" I also stated as my preference.

I added "Option Z" to the wiki: https://github.com/http2/http2-spec/wiki/ContinuationProposals#option-z

Feel free to edit/comment the wiki.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 18 Jul 2014 07:36:49 -0400

Prefer A, can't live with B because with large frames it would mean sifting through potentially 2^24-1 per frame of extra compressed header data, assuming you even wanted to leave the connection up at that point.  We already place a tremendous burden on the endpoints to support HPACK (compared to sending headers the HTTP/1.1 way) - with a single limit upfront we can control the size of that burden for everyone.

> P.P.S. This is not a call for more discussion; please resist replying to others' preferences.

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Fri, 18 Jul 2014 21:08:27 +0900

> 2) Whether you can live with the other option, and if not, why
My preference is b)​: Keep CONTINUATION.

​Cannot live with a) since it suffers from interop issue with http/1.1,
which is non-starter to me.
​b) is far more flexible in this regard.

Best regards,
Tatsuhiro Tsujikawa

---

From: Patrick McManus <[email-masked]>
Date: Fri, 18 Jul 2014 09:03:10 -0400

My top preference is to close the issue with no action.

My next preference is option [b]. In the best of all worlds it just adds
new information about an existing problem (the existing implementation
header limit) which is helpful.

I very strongly dislike [a] for reasons already stated in this thread:
http/1 tail compat and rewind.

---

From: Nicholas Hurley <[email-masked]>
Date: Fri, 18 Jul 2014 09:34:43 -0700

My ultimate preference is to keep the status quo.

Failing that, I could live with (B).

I can not live with (A).

For one, it drastically increases the complexity of implementation for
no discernible gain. This complexity comes from two sources - (1)
special-casing the implementation of one particular frame type, by
allowing for it to have a different max size from any other frame, and
(2) requiring a rollback capability in the HPACK encoder. Down both of
those roads lies the madness of subtle interop issues. Given that we've
already shown good interop with CONTINUATION, I see no need to introduce
two brand new sources of problems.

The required rollback capability that is implicit in (A) also gives us
yet another issue - the memory and/or CPU requirements to implement it.
The requirements aren't exactly pretty on desktop or server, and they're
downright nasty on mobile or other low-powered devices.

Finally, (A) explicitly breaks backwards compatibility with HTTP/1.1,
which is a horrible position to put ourselves in.
-- 
Peace,
  -Nick

---

From: Michael Sweet <[email-masked]>
Date: Fri, 18 Jul 2014 13:07:34 -0400

Nicholas,

> a) Remove CONTINUATION from the specification and add a new setting that dictates the maximum HEADERS/PUSH_PROMISE frame size (as distinct from max_frame_size) a peer is willing to receive. I.e., t…

I believe all that is being proposed here is to require headers to be sent in a single frame, whose maximum size is advertised by the recipient with a new setting.  HEADER and PUSH_PROMISE frames already require special handling, and the maximum header frame size would be a limit for the HPACK code generating the data for the frame - where is the increased complexity?

Nothing has been said about a required rollback capability, and I can't find any mention of rollback on the linked wiki page either.  The only proposal I've read recently that might even come close is a "reset the header table" mechanism to recover from a failed request without shutting down the connection, and that would not increase an implementation's memory requirements.

As for breaking backwards compatibility with HTTP/1.1, how?  HTTP/1.1 makes no guarantees that arbitrarily large headers can be sent, and in fact section 3.3.35 of RFC 7230 specifically says that the server can reject requests whose headers are too big.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 18 Jul 2014 10:17:28 -0700

On 18 July 2014 10:07, Michael Sweet <[email-masked]> wrote:
> Nothing has been said about a required rollback capability, and I can't find any mention of rollback on the linked wiki page either.

Say you have a block of headers that you need to encode for sending.
You discover that you exceed the limit only after performing the
encoding up to the point that you exceed the limit.  At that point,
you can't send the block.  And all the changes that have been applied
to the header table need to be backed out.

---

From: Jason Greene <[email-masked]>
Date: Fri, 18 Jul 2014 12:57:57 -0500

> to the header table need to be backed out.

Right so to address this you would need to check if the uncompressed size > compressed limit, and if so either do a first pass length compute or a temporary copy of the state table. 

It’s extra complexity, but the implementation isn’t difficult (a cake walk compared to other aspects of the spec). I can certainly appreciate the perspective from implementations that don’t want to touch their code though.
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Nicholas Hurley <[email-masked]>
Date: Fri, 18 Jul 2014 11:01:03 -0700

On Fri, Jul 18, 2014, at 10:57, Jason Greene wrote:
> Right so to address this you would need to check if the uncompressed size
> > compressed limit, and if so either do a first pass length compute or a
> temporary copy of the state table. 

Yep, and as I indicated in my email, that is an unacceptable tradeoff on
mobile (the tradeoff is the same as implementing a rollback mechanism).
That's effectively asking me to DOUBLE my state commitment; not a sane
thing to ask.

> It’s extra complexity, but the implementation isn’t difficult (a cake
> walk compared to other aspects of the spec). I can certainly appreciate
> the perspective from implementations that don’t want to touch their code
> though.
> --
> Jason T. Greene

-- 
Peace,
  -Nick

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 18 Jul 2014 18:20:50 +0000

In message <[email-masked]>
>to the header table need to be backed out.

I don't think it says anywhere that you cant *send* the block, but if
you do, you should expect it to get rejected.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 18 Jul 2014 11:28:50 -0700

> you do, you should expect it to get rejected.

I hadn't considered that interpretation at all.

I can't see that working out well.  I'd have thought that if you set a
limit of 10k, then treating anything that exceeds that as a connection
error is the logical conclusion.  Otherwise, you have to process more
than you said you wanted to receive, just to maintain a consistent
compression state.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 18 Jul 2014 18:40:20 +0000

In message <[email-masked]>
>than you said you wanted to receive, just to maintain a consistent

I'd expect behaviour in this area to develop and depend on field
experience.

I really don't think requests (or responses!) exceeding the limit
are going to happen outside attack-scenarios.

And setting or no setting, receivers will have limits, the only
difference is that we tell senders about them.

---

From: Glen Knowles <[email-masked]>
Date: Fri, 18 Jul 2014 18:27:14 -0700

I prefer B, by knowing the uncompressed limit the application layer - where
the messages are built - adjustments can be made if it's going over the
limit.

If A is selected I'll most likely either throw away the advertised limit
and hope the remote likes what I send, or try to somehow derive a sensible
uncompressed limit from the compressed limit to pass up to the application
layer. It's unreasonable to expect the application layer to stay under a
limit that depends on the order of delivery.

I'm not really concerned about continuations per se.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jul 2014 15:00:54 -0400

My reading of the WG is that the best we’re going to do for consensus here is option (b) — keeping continuation, and adding an advisory setting for the largest uncompressed header set that a recipient is willing to accept. 

We can split these up into separate decisions if people violently disagree; if not, I think we can stop discussing CONTINUATION.

Does anyone have further information to addd? If not, will mark as editor-ready.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 20 Jul 2014 21:33:01 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

>My reading of the WG is that the best we're going to do for consensus 
>here is option (b) - keeping continuation, and adding an advisory 
>setting for the largest uncompressed header set that a recipient is 
>willing to accept. 
>

If we're having to live with the CONTINUATION DOS-vector, can we at
least have the END-STREAM flag appear on the last CONTINUATION
frame ?

---

From: David Krauss <[email-masked]>
Date: Mon, 21 Jul 2014 05:47:26 +0800

> Does anyone have further information to addd? If not, will mark as editor-ready.

Why were the decisions balloted together in the first place? I never saw how compressed vs. uncompressed implied CONTINUATION vs. jumbo-only. There is little support for a declared, post-compression limit, so the results are skewed.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jul 2014 23:13:23 -0400

> Why were the decisions balloted together in the first place? I never saw how compressed vs. uncompressed implied CONTINUATION vs. jumbo-only. There is little support for a declared, post-compressio…

It’s not a matter of one implying the other; these were the two combinations that had the most support. It may be that we could close these issues with no changes at all, but some people’s concerns seemed to be mitigated by it, and there wasn’t significant pushback on it (if not universal enthusiasm).