Subject: Options for CONTINUATION-related issues

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jul 2014 17:08:49 +1000

Assuming that we can separate the removal of the reference set from HPACK from how we deal with CONTINUATION altogether (which may be optimistic, but let's try), I currently see two realistic options for (largely) resolving #550 and #551 -

a) Remove CONTINUATION from the specification and add a new setting that dictates the maximum HEADERS/PUSH_PROMISE frame size (as distinct from max_frame_size) a peer is willing to receive. I.e., the setting refers to the compressed header size.

b) Keep CONTINUATION in the specification, and add a new setting that advises the maximum header set size (i.e,. uncompressed) a peer is willing to receive (but might not imply PROTOCOL_ERROR or STREAM_ERROR on receipt).

There are a lot of details to discuss here, default values, possible tweaks, etc. but ignore that for the time being. Looking through the wiki and watching the discussion, I think these are the two frontrunners.

Are there any other realistic (i.e., capable of achieving consensus, NOT just your favourite approach) options that we should be considering?

Thanks,

P.S. This is NOT an invitation to debate merits or express preferences; we'll get there soon enough.

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jul 2014 09:40:22 +0200

On Wed, Jul 16, 2014 at 05:08:49PM +1000, Mark Nottingham wrote:
> a) Remove CONTINUATION from the specification and add a new setting that
> dictates the maximum HEADERS/PUSH_PROMISE frame size (as distinct from
> max_frame_size) a peer is willing to receive. I.e., the setting refers to the
> compressed header size.

I think that if we're going with a 24-bit max_frame_size, we'll indeed need to
have a separate max_header_size.

> b) Keep CONTINUATION in the specification, and add a new setting that advises
> the maximum header set size (i.e,. uncompressed) a peer is willing to receive
> (but might not imply PROTOCOL_ERROR or STREAM_ERROR on receipt).

I suspect that your proposal about only hinting in the spec that header sets
larger than XYZ kB are very likely going to receive a 431 should cover this.

Just my 2 cents,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 17 Jul 2014 00:53:56 +1000

On 16 July 2014 17:08, Mark Nottingham <[email-masked]> wrote:

> Are there any other realistic (i.e., capable of achieving consensus, NOT
> just your favourite approach) options that we should be considering?

hmmmm I am probably being unrealistic.... but let's tilt at this windmill

c) Remove CONTINUATION from the specification, allow HEADERS to be
fragmented and add a new setting that advises the maximum header set size
(i.e,. uncompressed) a peer is willing to receive (but might not imply
PROTOCOL_ERROR or STREAM_ERROR on receipt).

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Jason Greene <[email-masked]>
Date: Wed, 16 Jul 2014 10:42:46 -0500

On Jul 16, 2014, at 9:53 AM, Greg Wilkins <[email-masked]> wrote:

> c) Remove CONTINUATION from the specification, allow HEADERS to be fragmented and add a new setting that advises the maximum header set size (i.e,. uncompressed) a peer is willing to receive (but might not imply PROTOCOL_ERROR or STREAM_ERROR on receipt).

I have a fourth option to add to the mix which is based on all of the feedback I have seen.

AFAICT there is only one limited use-case that can not be covered by the recent move to allow large frames. That is the ability to split a frame into chunks so that a 1:1 proxy doesn’t have to buffer. 

We could have a more narrow form of CONTINUATION: 

- The sum of all continuation frames can not exceed max_frame_size.

Since this use case is avoiding buffering, they may not know the exact size has exceeded the max until the Nth continuation is sent. Therefore:

- Add a discard flag on the last HEADER/CONTINUATION frame, which instructs the endpoint to discard the request and all buffered data.

Pros:
- Addresses 551
- There is only one max setting
- Encourages reduction of gigantic headers
- As a side-effect of the discard flag, it provides a client the ability to inform the server that a request was too large and would have generated a 431 (not really significant but it came up in a thread and worth mentioning)
 
Cons:
- As with all other length limited proposals, sender has to rewind encoder state to that of last sent frame.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 17 Jul 2014 14:44:40 +1000

> - The sum of all continuation frames can not exceed max_frame_size.

The default of max_frame_size is 16k; that means that a server can't ever send more than 16k (or 32k, depending on whether you intend to include HEADERS in the max) of response headers without some adjustment by the browser...

> - Add a discard flag on the last HEADER/CONTINUATION frame, which instructs the endpoint to discard the request and all buffered data.

Why not just reset the stream?

> - As with all other length limited proposals, sender has to rewind encoder state to that of last sent frame.

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 17 Jul 2014 14:45:01 +1000

> c) Remove CONTINUATION from the specification, allow HEADERS to be fragmented and add a new setting that advises the maximum header set size (i.e,. uncompressed) a peer is willing to receive (but m…

For such an invasive change, I think we need more than a sentence. I also see getting consensus around that being very problematic.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 17 Jul 2014 15:48:05 +1000

On 17 July 2014 14:45, Mark Nottingham <[email-masked]> wrote:

> For such an invasive change,

Well it is not all that invasive.  It is essentially continuations redone
without the issues forced on it by hpack's RefSet.   But either way
"invasive" is subjective not technical reason, just like "ugly".

I think we need more than a sentence.

The details have been put in the wiki as requested: "Remove CONTINUATIONS,
Fragment HEADERS"
https://github.com/http2/http2-spec/wiki/ContinuationProposals

I also see getting consensus around that being very problematic.

Where?  I thought you asked us to put the proposals into the wiki with pros
and cons?  The only con listed against this proposal is the suggestion that
it may increase the DoS attack surface but that is disputed if it is a
significant increase or not.      Can we have some more detail about what
is "very problematic" otherwise it is impossible to refute arguments that
are only given as meta-reasons.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 17 Jul 2014 15:54:39 +1000

On 17 Jul 2014, at 3:48 pm, Greg Wilkins <[email-masked]> wrote:

> Well it is not all that invasive.  It is essentially continuations redone without the issues forced on it by hpack's RefSet.   But either way "invasive" is subjective not technical reason, just like "ugly".
> 
> The details have been put in the wiki as requested: "Remove CONTINUATIONS, Fragment HEADERS"

From there:

"""
	• Remove the CONTINUATION frames.
	• Remove priority fields from the HEADERS frame, as these can be sent in a separate PRIORITY frame without concerns of fragmentation.
	• Deduct the HEADER frame sizes from the flow control window sizes, but do not block header frames due to flow control.
	• If it is desired to declare a maximum header size (for 551), then add a SETTINGS_MAX_HEADER_SIZE expressed in uncompressed bytes
	• Remove the header block fragment from the PUSH_PROMISE frame. Instead send the headers in a HEADERS+ frame sequence following the PUSH_PROMISE frame.
"""

That touches many parts of the spec, introducing the opportunity for potentially many new interoperability problems.

But let's cut to the chase. Would any of our current implementers consider trying this out?

> Where?  I thought you asked us to put the proposals into the wiki with pros and cons?  The only con listed against this proposal is the suggestion that it may increase the DoS attack surface but that is disputed if it is a significant increase or not.      Can we have some more detail about what is "very problematic" otherwise it is impossible to refute arguments that are only given as meta-reasons.

That was my gut feeling, if the WG comes out in support of it, I'm happy to be proven wrong.

After all, you did say it was "unrealistic" and "tilt[ing] at this windmill" yourself, Greg.

Regards,

---

From: Jason Greene <[email-masked]>
Date: Thu, 17 Jul 2014 12:04:33 -0500

On Jul 16, 2014, at 11:44 PM, Mark Nottingham <[email-masked]> wrote:

> The default of max_frame_size is 16k; that means that a server can't ever send more than 16k (or 32k, depending on whether you intend to include HEADERS in the max) of response headers without some adjustment by the browser…

Right. The thinking there is that 99.8% of headers are < 16K, so this is more than enough, and when you have a .2% you can just bump the frame size.

If you compare CONTINUATION to Large Frames, they are actually very similar.

They both:
- Can send very large headers 
- Block other streams until completion
- Support writing to the wire in chunks

Continuations adds:
- 16MB -> Unlimited headers
- Ability to write in chunks without knowing the total compressed length

The former is not really desirable and causes #551. 
The latter offers very limited value, in that a 1:1 proxy can re-encode the frame using different hpack state and send the data across in piecemeal. 

So this option is simply saying we can address that particular use-case, if deemed important, by supporting fragmentation of the HEADERS frame. We don’t have to span frames to accomplish the need. By not spanning frames we solve #551, and also eliminate the DOS exposure concern which has been brought up against a header size limit solution. 

> Why not just reset the stream?

The only benefit in not reseting the stream is just notifying the server that a request wasn’t sent because it was too big. Although I agree its of limited value. It was a concern brought up against size limits, that a server does not know when a request is too big. I personally disagree with the concern, I just was describing one way it could be handled.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 17 Jul 2014 10:20:25 -0700

> Right. The thinking there is that 99.8% of headers are < 16K, so this is more than enough, and when you have a .2% you can just bump the frame size.

Here my only concern is that you don't have any way of triggering this
bump.   Say we got rid of CONTINUATION...

What does a client do if it prefers to have 16K frames?  How does a
server indicate that it wants to send headers that exceed this limit?

Flip that around: how does a client indicate that it wants to send
headers that exceed this limit?  At least a server can send a
response... and it has a stream upon which to send it.

Browsers, which compete on soul tolerance (thanks Carsten), now have a
strong incentive to advertise > 16K frame sizes.  I think that you
will find that this is an outcome that browser will be especially
unhappy with.

>> Why not just reset the stream?

I think that resetting a stream is the only good response (with an
optional 431).  I don't know what you would invent something else that
has the same net effect.

---

From: Nicholas Hurley <[email-masked]>
Date: Thu, 17 Jul 2014 11:01:50 -0700

On Wed, Jul 16, 2014 at 10:54 PM, Mark Nottingham <[email-masked]> wrote:

>         • Remove priority fields from the HEADERS frame, as these can be
> sent in a separate PRIORITY frame without concerns of fragmentation.
>         • Deduct the HEADER frame sizes from the flow control window
> sizes, but do not block header frames due to flow control.
>         • If it is desired to declare a maximum header size (for 551),
> then add a SETTINGS_MAX_HEADER_SIZE expressed in uncompressed bytes
>         • Remove the header block fragment from the PUSH_PROMISE frame.
> Instead send the headers in a HEADERS+ frame sequence following the
> PUSH_PROMISE frame.
> """
>
> That touches many parts of the spec, introducing the opportunity for
> potentially many new interoperability problems.
>

I'm not at all interested in trying this out. As I see it, it solves no
issues, and in fact introduces new ones (the ability to unintentionally
starve oneself of flow control when something has caused you to send a lot
of headers, quite likely out of your own control). The arguments keep seem
to be boiling down to "I don't like CONTINUATION, so let's find some reason
to get rid of it".

Another issue is this: I have large headers to send. How do I inform my
peer of that, assuming they have advertised a smaller frame size? I can't.
Even worse: someone has large headers to send to me (lots of set-cookie, or
a kerberos token). I don't want frames larger than 16k in any case, as it
hurts the muxing and prioritization. I'm either stuck not being able to
talk to that peer (bad) or I have to open myself up to really bad muxing
and prioritization (also bad).

Nope, none of those eventualities sound good. In fact, it sounds more like
we're either (1) ensuring a lot of situations where interop won't exist, or
(2) forcing everyone to accept large frames (which plenty of us have
already said we won't do). This is not the way to make a working,
interoperable protocol.
--
Peace,
  -Nick

---

From: Roberto Peon <[email-masked]>
Date: Thu, 17 Jul 2014 11:09:04 -0700

The list you have here didn't mention that any length requirement also
requires the compressor to be able to rewind state, which is most decidedly
non-trivial, and is a great way to require memory over-commitment.
These are considered with pros/cons/requirements in the wiki link that Mark
sent out a few days back-- it would be good to refer to that doc (and/or
modify it), as it is likely to better capture all pros/cons.
Certainly it is more reliable than my memory :)

-=R

On Thu, Jul 17, 2014 at 10:04 AM, Jason Greene <[email-masked]>
wrote:

> >>> Are there any other realistic (i.e., capable of achieving consensus,
> NOT just your favourite approach) options that we should be considering?
> >>>
> >> AFAICT there is only one limited use-case that can not be covered by
> the recent move to allow large frames. That is the ability to split a frame
> into chunks so that a 1:1 proxy doesn’t have to buffer.
> >>
> > The default of max_frame_size is 16k; that means that a server can't
> ever send more than 16k (or 32k, depending on whether you intend to include
> HEADERS in the max) of response headers without some adjustment by the
> browser…
>
> Right. The thinking there is that 99.8% of headers are < 16K, so this is
> more than enough, and when you have a .2% you can just bump the frame size.
>
> The latter offers very limited value, in that a 1:1 proxy can re-encode
> the frame using different hpack state and send the data across in piecemeal.
>
> So this option is simply saying we can address that particular use-case,
> if deemed important, by supporting fragmentation of the HEADERS frame. We
> don’t have to span frames to accomplish the need. By not spanning frames we
> solve #551, and also eliminate the DOS exposure concern which has been
> brought up against a header size limit solution.
>
> >
> >
> >> Since this use case is avoiding buffering, they may not know the exact
> size has exceeded the max until the Nth continuation is sent. Therefore:
> >>
> >> - Add a discard flag on the last HEADER/CONTINUATION frame, which
> instructs the endpoint to discard the request and all buffered data.
> >
> The only benefit in not reseting the stream is just notifying the server
> that a request wasn’t sent because it was too big. Although I agree its of
> limited value. It was a concern brought up against size limits, that a
> server does not know when a request is too big. I personally disagree with
> the concern, I just was describing one way it could be handled.
>
> >
> >
> >> Pros:
> >> - Addresses 551
> >> - As a side-effect of the discard flag, it provides a client the
> ability to inform the server that a request was too large and would have
> generated a 431 (not really significant but it came up in a thread and
> worth mentioning)
> >>
> >> Cons:
> >> - As with all other length limited proposals, sender has to rewind
> encoder state to that of last sent frame.
> >
> > Cheers,
> >
> >
> > --

---

From: Jason Greene <[email-masked]>
Date: Thu, 17 Jul 2014 13:23:35 -0500

> >> - As with all other length limited proposals, sender has to rewind encoder state to that of last sent frame.

I was just focusing on the differentiation of the pros though. 

I didn’t bother to add it to the wiki, just because I wasn’t sure there would be any interest to go this way. I can definitely do that though just in case.

On Jul 17, 2014, at 1:09 PM, Roberto Peon <[email-masked]> wrote:

> The list you have here didn't mention that any length requirement also requires the compressor to be able to rewind state, which is most decidedly non-trivial, and is a great way to require memory over-commitment.
> These are considered with pros/cons/requirements in the wiki link that Mark sent out a few days back-- it would be good to refer to that doc (and/or modify it), as it is likely to better capture all pros/cons.
> On Thu, Jul 17, 2014 at 10:04 AM, Jason Greene <[email-masked]> wrote:
>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 17 Jul 2014 11:29:11 -0700

If it isn't there already, it certainly deserves to be!

I suspect that that wiki will end up being an interesting work product
itself-- recording why one doesn't go down a particular path, or why
another path is chosen is useful.
-=R

On Thu, Jul 17, 2014 at 11:23 AM, Jason Greene <[email-masked]>
wrote:

> I didn’t bother to add it to the wiki, just because I wasn’t sure there
> would be any interest to go this way. I can definitely do that though just
> in case.
>
> > These are considered with pros/cons/requirements in the wiki link that
> Mark sent out a few days back-- it would be good to refer to that doc
> (and/or modify it), as it is likely to better capture all pros/cons.
> > >> Since this use case is avoiding buffering, they may not know the
> exact size has exceeded the max until the Nth continuation is sent.
> Therefore:
> > >>

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 18 Jul 2014 10:36:42 +1000

On 18 July 2014 04:01, Nicholas Hurley <[email-masked]> wrote:

> The arguments keep seem to be boiling down to "I don't like CONTINUATION,
> so let's find some reason to get rid of it".

Nicholas,

You are under no obligation to support any proposal, but please do not
denigrate the motives of those who are making them in good faith.

The proposals to remove continuations are motivated by genuine concerns
that they will harm the future of the web.   Effort has been made to listen
to all the issues that continuations were designed to address and the
proposals have been prepared to those multiple requirements in mind.

I understand it is tedious to go over this again and again, but I genuinely
believe that the apparent willingness of the WG to remove the refset from
HPACK has removed one of the key drivers for the continuation design, and
thus it is worthwhile to discuss alternatives at this point.

If the consensus comes out that the WG would prefer to keep continuations
(and I suspect Mark is about to call it that way), then fine.  But please
don't suggest that those contributing to the process are doing so for
anything but good in good faith.