Subject: Striving for Compromise (Consensus?)

From: Jeff Pinner <[email-masked]>
Date: Fri, 11 Jul 2014 00:32:26 -0700

How do people feel about the following compromise:

1) Increase frame size to 16-bits.
2) Remove reference set from HPACK allowing for "streaming" decoding.
3) Requiring that all ":"-headers appear first.
4) Only allowing CONTINUATION if the previous frame is 2^16-1 bytes.
5) Allowing interleaving of CONTINUATION frames with other frames.

For implementors that know that they will never accept more than 64kb
of headers, they don't have to implement CONTINUATION frames.

For implementors that are concerned about not receiving routing
headers and having to buffer full frames, they will be the first
things that show up and the rest of the headers can be proxied without
any buffering.

For implementors that need to support headers > 64kb they can do so
using CONTINUATION frames.

For implementors concerned about interleaving headers between multiple
streams, that is now allowed because the reference set does not
coincide with the end of the header block.

For implementors that are concerned with spec complexity, the entire
notion of "header block fragments" is removed.

Thoughts?

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 11 Jul 2014 09:43:36 +0200

> notion of "header block fragments" is removed.

I could live with that. At I consider this as a significant improvement
over what we currently have.

Willy

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 11 Jul 2014 17:48:18 +1000

Well I thought we were close to consensus on the large frame proposal until
the reduction <24 bits.    If 24bits is so objectionable that we want to
continue circling on this one .....

So let's look at your alternative.

> 5) Allowing interleaving of CONTINUATION frames with other frames.
I could almost live with that.   Problem is that can't fragment headers to
less than 64KB, which if I understand the current arguments against the
"Greg et al" is a must have.

So how about:

1) Increase frame size to 16-bits
1a) Add a settings for max_frame_size
2) Remove reference set from HPACK allowing for "streaming" decoding.
3) Requiring that all ":"-headers appear first.
4) Only allowing CONTINUATION if the previous frame is max_frame_size.
5) Allowing interleaving of CONTINUATION frames with other frames.
5b) The size of the HEADERS and CONTINUATION frames are removed from the
flow control window, but the they are never flow controlled.

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Jul 2014 18:03:48 +1000

Hi Jeff,

Thanks. Based on the discussion today, I'm open to this.

From where I sit, the only one that seems controversial is #4 - and we've already seen a proposed modification from Greg. 

Personally, I wonder if we can just ditch #4 and still have reasonable properties; an implementation that receives too many CONTINUATIONS can reset the stream and continue to process other streams, correct?

Let's talk it through. 

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 11 Jul 2014 10:26:22 +0200

On Fri, Jul 11, 2014 at 06:03:48PM +1000, Mark Nottingham wrote:
> From where I sit, the only one that seems controversial is #4 - and we've
> already seen a proposed modification from Greg. 
> 
> Personally, I wonder if we can just ditch #4 and still have reasonable
> properties; an implementation that receives too many CONTINUATIONS can reset
> the stream and continue to process other streams, correct?

> > of headers, they don't have to implement CONTINUATION frames.

so that's not really an option here.

Willy

---

From: <[email-masked]>
Date: Fri, 11 Jul 2014 08:53:24 +0000

I'm a bit confused (definitely not the first time). Will somebody please help me understand why we still need CONTINUATION frames if we remove the hpack reference set?

In other words, why can't a request look like:
1*HEADERS [1*DATA]

and a response:
1*HEADERS [1*DATA] [1*HEADERS]

with the constraint that all frames may be interleaved, but HEADERS frames must be processed in order.

-confused in Vienna

> On Jul 11, 2014, at 10:30, "[email-masked]" <[email-masked]> wrote:
>
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Jul 2014 19:04:53 +1000

Willy,

> so that's not really an option here.

I am not interested in catering to people who just don’t want to implement a particular frame type.

As far as I can tell, the underlying issues — HOL blocking, buffering, etc. — are the same here whether or not CONTINUATION is used, under this proposal.

If that’s not true, express your objection in those terms.

Thanks,

--
Mark Nottingham   http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 11 Jul 2014 11:26:37 +0200

On Fri, Jul 11, 2014 at 07:04:53PM +1000, Mark Nottingham wrote:
> I am not interested in catering to people who just don?t want to implement a
> particular frame type.
> 
> As far as I can tell, the underlying issues ? HOL blocking, buffering, etc. ?
> are the same here whether or not CONTINUATION is used, under this proposal.
> 
> If that?s not true, express your objection in those terms.

Maybe I missed something, but the underlying issues that led to the proposal
to remove CONTINUATION were complexity, processing cost and sensitivity to
DoS.  So while the HOL blocking and buffering are probably not impacted by
the presence or not of CONTINUATION, these other issues definitely are. Or
am I off-topic ?

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Jul 2014 19:30:32 +1000

> the presence or not of CONTINUATION, these other issues definitely are. Or

There are lots of pathological ways to increase processing cost — e.g., 1-byte DATA frames. We’re not talking about limiting those (and I will be EXTREMELY unsympathetic to anyone who brings it up after this).

Complexity has many facets, as we’ve heard; it’s not a clean-cut decision. 

AIUI the DoS scenario had more to do with HOL blocking and cost of teardown; if there’s a DoS vector that’s specific to this scenario, let’s hear it.

Thanks,

---

From: <[email-masked]>
Date: Fri, 11 Jul 2014 09:41:31 +0000

> How do people feel about the following compromise:

-1
It eliminates both purposes of the 'Greg et al' proposal:
  a) Eliminate the CONTINUATION ugliness (complexity, processing, etc.), and
  b) add bits & settings for tuning frame lengths.

> 1) Increase frame size to 16-bits.

I don't know how this could be claimed as a good idea without also having the SETTINGS_HEADER_SIZE & SETTINGS_FRAME_SIZE setting.  Roberto, Patrick, etc. have repeatedly said that 16K is the 'optimal' frame size for multiplexing on today's networks.  Moving to 16 bits without a setting means everybody will use 64K.  Remember, in the 'Greg et al' proposal the default frame size is still 16K (actually it's literally one better - it's 16K instead of 16K-1!).

It's also terribly short-sighted to not have any space for expansion.  At the very least you should keep 1 reserved bit so you can do the ugly "jumbo frame" hack later when you figure out you want bigger frame sizes.

> 2) Remove reference set from HPACK allowing for "streaming" decoding.

This is a good idea.

> 3) Requiring that all ":"-headers appear first.

This is also a good idea.

> 4) Only allowing CONTINUATION if the previous frame is 2^16-1 bytes.

See Roberto's thread "Fragmentation for headers: why jumbo != continuation" [1] for reasons why he thinks this is a bad idea.

> 5) Allowing interleaving of CONTINUATION frames with other frames.

Admittedly better than the status quo, but still not entirely clear why this is necessary. i.e. why not 1*HEADERS instead?

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/0687.html

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 10:04:51 +0000

In message <CAH_y2NFa3x6Gn4A_OpP+coRR1UE5VdO6SgE6y4NoAreZ=y=[email-masked]>, Greg Wilkins wri

>1) Increase frame size to 16-bits

I still think this is horribly near-sighted and first rate false economy.

>1a) Add a settings for max_frame_size

For reasons of interop I would add: 

1b)  This setting must be configurable to 16384 bytes.
1c)  Settings below 256 bytes are illegal.

This establishes a configurable "we know this will work" and
ensures that it is always possible to get /robots.txt.

>3) Requiring that all ":"-headers appear first.

YES!

>4) Only allowing CONTINUATION if the previous frame is max_frame_size.

So what exactly happens if a complete header doesn't fit into the
frame ?

Does HPACK keep per-stream state ?

>5) Allowing interleaving of CONTINUATION frames with other frames.

"with other frames" or "with all other frames" ?

Can we please also move the END_STREAM bit to the last CONTINUATION ?

>flow control window, but the they are never flow controlled.

I think they should be flow controlled on the connection window, that way
the connection window can be used to balance competing connections for
a contested resource.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Jul 2014 20:53:16 +1000

>  b) add bits & settings for tuning frame lengths.

See my previous message to Willy. These are not issues, they’re a wish list. 

To be clear — the time to argue over the aesthetics of the protocol has long passed; I’m not willing to hold this effort up because there’s disagreement about how it looks. If there are concrete technical issues that aren’t on the list, bring them up. If you think that this proposal doesn’t address the relevant issues, say so. “It’s too complex” isn’t a concrete technical issue; it’s a judgment call with many facets, as discussed. I pulled Roberto up for justifying a position with “bleh” earlier today, and I expect no less from everyone else.

Likewise, asking for the result to look in a particular way (such as particular settings) is an unreasonable request at this stage in this protocol’s development; you need to show concrete problems that are caused by the design or proposal.

Be aware that the even then, the WG may decide not to address a particular issue.

> I don't know how this could be claimed as a good idea without also having the SETTINGS_HEADER_SIZE & SETTINGS_FRAME_SIZE setting.  Roberto, Patrick, etc. have repeatedly said that 16K is the 'optim…

Let other people make their own assessments of the proposal, please. If Roberto, Patrick et al like this proposal, all the better.

> It's also terribly short-sighted to not have any space for expansion.  At the very least you should keep 1 reserved bit so you can do the ugly "jumbo frame" hack later when you figure out you want…

I’m getting unsympathetic to arguments of the “we might need that” variety. The WG has long held that if major changes are necessary, the protocol can be versioned much more easily than from /1 to /2.

Let’s focus on what we need now. If we need 31 bits to address our issues, so be it — but this is largely separable from the current discussion about header sizes, buffering, etc.

> See Roberto's thread "Fragmentation for headers: why jumbo != continuation" [1] for reasons why he thinks this is a bad idea.

See also my follow-up (in anticipation of that).

> Admittedly better than the status quo, but still not entirely clear why this is necessary. i.e. why not 1*HEADERS instead?

Let’s agree on the general approach before we go into tweaking.

Thanks,

---

From: <[email-masked]>
Date: Fri, 11 Jul 2014 11:33:47 +0000

Hi Mark-

On Friday,11 July 2014 12:53, [email-masked] wrote:
> See my previous message to Willy. These are not issues, they're a wish list.

You lost me on this one. We (the Greg et al group) spent a considerable amount of time on our proposal with a rather detailed explanation of the issues and our proposed solutions, see [1].  There has been a lot of discussion on this proposal, IMO mostly positive from the WG, *including yourself.*

To say now that the two main goals of our proposal is a wish list totally baffles me.

I agree that since this has turned into a full-scale discussion of non-editorial changes there may be some "wishes" creeping in.  But I take issue with you calling this a wish list just because I didn't copy-and-paste each point from Greg's e-mail [1] when we introduced the proposal.

> Let other people make their own assessments of the proposal, please. If Roberto, Patrick et al like this proposal, all the better.

Gladly! But they better have some pretty sound justification as to why the "optimal" frame length for efficient and responsive multiplexing magically changed from 16K to 64K!!!

> I'm getting unsympathetic to arguments of the "we might need that" variety. The WG has long held that if major changes are necessary, the protocol can be versioned much more easily than from /1 to /2.
> Let's focus on what we need now. If we need 31 bits to address our issues, so be it - but this is largely separable from the current discussion about header sizes, buffering, etc.

I think you're more optimistic than me :)  I seriously doubt it will be so easy to version so quickly. In any case, it will be much easier to go from /2 /3 if we have reserved bits so we don't have to modify the base framing header in an incompatible way or with an ugly hack. On the other hand, I'm also sympathetic to your idea to "focus on what we need now".  There are plenty of protocols that have reserved bits that will never be used.

> Let's agree on the general approach before we go into tweaking.

This is NOT a tweak!  If header fragmentation can be accomplished with 1*HEADERS instead of HEADERS+CONTINUATION*, then keeping CONTINUATION is just keeping complexity for the sake of being too bothered to remove it. I keep bringing it up because nobody has provided a satisfactory answer as to why we still need CONTINUATION of we remove the reference set from HPACK.

-keith

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/0427.html

---

From: Jeff Pinner <[email-masked]>
Date: Fri, 11 Jul 2014 05:17:49 -0700

> This is NOT a tweak!  If header fragmentation can be accomplished with 1*HEADERS instead of HEADERS+CONTINUATION*, then keeping CONTINUATION is just keeping complexity for the sake of being too bot…

CONTINUATION frames simplify the processing of multiple header blocks
since the HEADERS / PUSH_PROMISE frames contain other fields that
CONTINUATION cannot have. As long as HEADERS contains a priority field
and PUSH_PROMISE a promised stream ID, CONTINUATION frames make error
handling simpler.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 12:52:34 +0000

In message <[email-masked]>, Mark Nottingham writes:
>To be clear the time to argue over the aesthetics of the protocol 
>has long passed; 

This is not a matter of aesthetics Mark.

As currently specified CONTINUATION is a giant invitation to DoS attacks.

The missing "this many headers will arrive" early notice is a surefire
way to force all receivers to implement complex and slow memory management
schemes, also prone to DoS exploitation.

The crucial core of our proposal is that header-sets go into a single
frame, so the length is announced up front.  16K being not long enough
for that, is another good reason for large frames.

Not wanting to trade the "ulimited CONTINUATIONS" for "unlimited frame
sizes, the SETTINGS for MAX frame size follows naturally.

---

From: <[email-masked]>
Date: Fri, 11 Jul 2014 13:03:33 +0000

Hi Jeff-

On Friday,11 July 2014 14:18, [email-masked] wrote:
>> why [do] we still need CONTINUATION [if] we remove the reference set from HPACK.
>
> CONTINUATION frames simplify the processing of multiple header blocks since
> the HEADERS / PUSH_PROMISE frames contain other fields that CONTINUATION
> cannot have. As long as HEADERS contains a priority field and PUSH_PROMISE a
> promised stream ID, CONTINUATION frames make error handling simpler.

Thanks.  Now your eariler proposal make more sense (to me), particularly the reason for adding SYN_STREAM...

> PUSH_PROMISE 1*HEADERS SYN_STREAM 1*HEADERS 0*DATA [DATA 0*HEADERS]

Presumably a non-pushed request would look like this...?

SYN_STREAM 1*HEADERS 0*DATA [DATA 0*HEADERS]

I'm with Greg, though, what is the reason for needing a distinction between HEADERS and DATA frames?  The same thing could easily be accomplished with just DATA frames logically divided into a http request of "headers data [trailers]" just by the END_SEGMENT flag.

-keith

---

From: Jeff Pinner <[email-masked]>
Date: Fri, 11 Jul 2014 06:18:28 -0700

> I'm with Greg, though, what is the reason for needing a distinction between HEADERS and DATA frames?  The same thing could easily be accomplished with just DATA frames logically divided into a http…

The distinction is due to two things:

1) Having different opcodes for DATA and HEADERS frames allows us to
know up front if the frame needs to be decompressed by HPACK.

Using END_SEGMENT requires us to remember which segment in the HTTP
message is currently being received to decode the data properly. It
also prevents experimenting with the type of inter-message HEADERS
frames that some people wanted (streaming checksums and the like) that
are currently permitted to be sent but have no "semantic" mapping to
HTTP/1.1

2) Unlike DATA frames, HEADERS contain content that is not arbitrarily
divisible which makes flow controlling them difficult
"out-of-the-box."

For example, if I want to send 2 bytes of DATA on a stream that has a
window_size of 1, I can break it up and send a single byte in the
frame. The receiver then knows the window_size has gone to 0 and that
will cause the stream to stall.

If we try to flow control HEADERS, we run into a situation where we
could deadlock: if I compress a header to a 2 byte HPACK opcode, and I
want to send it on a stream with that has a window_size of 1, I can't.
I could try to split it into two frames using CONTINUATIONS and send
one, but now the HPACK opcode crosses a frame boundary where the
second frame needs an RTT to be sent, stalling all header processing.

In order to flow control HEADERS we'd have to add an expanded BLOCKED
frame that lets the sender inform the receiver that it is blocked on a
stream with a non-0 window_size and include how many extra bytes the
window_size needs to be increased in order to fit the frame.

---

From: Johnny Graettinger <[email-masked]>
Date: Fri, 11 Jul 2014 13:08:28 -0400

> window_size needs to be increased in order to fit the frame.

Yep. Or allow the non-indexed-literal-with-a-literal-name opcode (only) to
span HEADER/CONTINUATION frame boundaries (which would also allow single
large header values to be fragmented across smaller frames).

Addressing the larger proposal: If we flow control HEADERS, or even just
allow the interleaving of header block fragments, we probably also want to
re-work the stream concurrency limit mechanism of the protocol. Requiring
:headers up-front is nice, but not really sufficient (consider a proxy
routing on a session cookie). It also doesn't play well with flow-control
(what if the session window is too small to send all :headers?).

I think we'd want a "has the remote endpoint accepted this stream?" bit of
stream state, and a protocol setting limiting the number of unaccepted
streams an endpoint may create. That would allow a connection to multiplex
large numbers of accepted streams, while limiting the memory exhaustion
attack surface a proxy must present. A proxy would only "accept" a stream
once it has an established upstream route through which it can immediately
forward header fragments & data. A straw-man mechanism might be that a
stream is accepted upon receipt of a HEADERS or PRIORITY frame.

Such a setting would likely replace the current
SETTINGS_MAX_CONCURRENT_STREAMS, as an endpoint can implicitly enforce that
setting by deferring stream acceptance.

---

From: Johnny Graettinger <[email-masked]>
Date: Fri, 11 Jul 2014 13:21:50 -0400

To be clear, I'm not actually suggesting we do this (there are parts I
like, but it's more complexity), but I do think something like it is needed
if header fragments may be interleaved. Otherwise proxies must use max
stream concurrency as their only lever to bound memory exhaustion attack
surface.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 10:47:34 -0700

This seems to be the set that people are talking about most seriously:

On 11 July 2014 00:48, Greg Wilkins <[email-masked]> wrote:
> 1b) ...that defaults to 16k
> 1c) ...with some minimum (which might be 256, though a 256 byte frame could be all padding...)

I think that 4 and 5 might be problematic.

4 doesn't seem to be well understood, but the interaction between TCP
congestion window and something like the proposed 5b could mean some
serious stalling/HOL issues.  More serious than the issues it purports
to address.

I think that 5 is a non-starter.  Roberto's analysis on this has
convinced me that this is an undesirable feature.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 11 Jul 2014 20:08:02 +0200

Hi Martin,

> convinced me that this is an undesirable feature.

I think that in Roberto's case, it's the opposite in fact. Assuming he
runs a customer-facing gateway that aggregates streams to origin servers
using less connections, *and* that he forwards headers on the fly without
buffering, then a slow client sending a small header frame and slowly
pushing continuation frames would simply block all traffic from all other
clients to that same server for the same connection.

This problem does not happen if headers are buffered, and if they are
buffered, there's no need for CONTINUATION unless headers are larger
than max_frame_size, which is exactly point 4.

Regards,
Willy

---

From: Nicholas Hurley <[email-masked]>
Date: Fri, 11 Jul 2014 11:13:37 -0700

> > 1) Increase frame size to 16-bits

This seems OK in conjunction with 1a and 1b. But not without 1a + 1b.

> > 1c) ...with some minimum (which might be 256, though a 256 byte frame

This minimum needs to be bigger, like I've said before. A laughable size
like 256 just provides too much opportunity for someone to shoot themselves
in the face. 4k is still my suggestion here, based on telemetry on
compressed header sizes in Firefox.

> > 2) Remove reference set from HPACK allowing for "streaming" decoding.

Yes, please!

>  > 3) Requiring that all ":"-headers appear first.

Sounds fine by me.

>  > 4) Only allowing CONTINUATION if the previous frame is max_frame_size.

I'm not totally against this at first glance, but as you point out below,
it's not well understood, and that makes me opposed to this. We're trying
to nail down the protocol, not introduce more uncertainty or opportunities
for footguns.

> > flow control window, but the they are never flow controlled.

No on both counts. Interleaving is, as Roberto pointed out, bad. Modifying
the flow control window without flow controlling that which modifies the
window is disgustingly asymmetric, and another great opportunity for a
footgun. And flow controlling headers means we can easily get ourselves in
a state where we can't start new streams that are absolutely necessary. So,
no.

---

From: Jason Greene <[email-masked]>
Date: Fri, 11 Jul 2014 13:23:58 -0500

> convinced me that this is an undesirable feature.

Just to clarify you mean 5b not 5, correct?

Fragmenting’s justification is proxies. However proxies can’t have HOL blocking occurring, so without 5 the feature is of limited value.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 11:30:37 -0700

> clients to that same server for the same connection.

That's true.  So don't do that.

Perhaps we are talking past each other.  Here's the matrix:

0 can and do stream (1-3), no need for max_size frames (!4), no
interleaving (!5)

   - No state commitment DoS concern
   - May spend extra cycles on framing
   - May have HOL
   - No stalling

1 can and do stream (1-3), need to fill frames before CONTINUATION
(4), no interleaving (!5)

   - No state commitment DoS concern
   - No extra cycles on unnecessary framing
   - May have HOL
   - May stall

2 don't stream, no need for max_size frames (!4), no interleaving (!5)

   - This is the status quo
   - No state commitment DoS concern
   - May spend extra cycles on framing
   - May have HOL
   - No stalling

3 don't stream, need to fill frames before CONTINUATION (4), no
interleaving (!5)

   - No state commitment DoS concern
   - No extra cycles on unnecessary framing
   - May have HOL
   - May stall

4 can and do stream (1-3), no need for max_size frames (!4), interleaving (5)

   - No state commitment DoS concern
   - May spend extra cycles on framing
   - Can avoid HOL
   - May stall

5 can and do stream (1-3), need to fill frames before CONTINUATION
(4), interleaving (5)

   - State commitment DoS concern
   - No extra cycles on unnecessary framing
   - No HOL
   - May stall
   - Streaming not possible where max_size doesn't match up

6 don't stream, no need for max_size frames (!4), interleaving (5)

   - State commitment DoS concern
   - May spend extra cycles on framing
   - Can avoid HOL
   - No stalling

7 don't stream, need to fill frames before CONTINUATION (4), interleaving (5)

   - State commitment DoS concern
   - No extra cycles on unnecessary framing
   - Can avoid HOL
   - May stall

I think that's good enough for our purposes.

Given that my priority order here is approximately: state commitment >
stalling > HOL > CPU, I think that status quo is looking pretty good.

Note that option 5, which is the complete combination of all proposed
changes, seems to actually be the worst in this analysis.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 11:32:04 -0700

> Just to clarify you mean 5b not 5, correct?

No, I meant 5.
http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/0760.html

---

From: Jason Greene <[email-masked]>
Date: Fri, 11 Jul 2014 13:35:17 -0500

> http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/0760.html

Ok in that case, Roberto’s analysis does not prove what you say it does.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 11:38:35 -0700

> Ok in that case, Roberto’s analysis does not prove what you say it does.

I'm pretty sure that it does.

This point:
> The current design handles this fairly well, at most one set of headers can
> be incomplete at any point in time (sending a large number of incomplete
> headers and keeping most of them incomplete most of the time is an
> excellent attack vector, which the design currently precludes).

---

From: Jason Greene <[email-masked]>
Date: Fri, 11 Jul 2014 13:45:49 -0500

>> excellent attack vector, which the design currently precludes).

This is the flaw:

"1) Stalling a connection by never finishing the sending of a full set of headers.

I don't find #1 interesting, since the attacker is mostly just attacking
themselves"

If you coalesce connections there are N users per connection. Thats a real problem you can’t just wave away.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 12:03:21 -0700

> If you coalesce connections there are N users per connection. Thats a real problem you can’t just wave away.

No, that is still right.  You are mounting the DoS on yourself.  Your
N users can blame you, not the protocol.

If it were possible to stream small bits of messages (as we can do for
DATA) and the real subject of Roberto's analysis weren't a problem (it
is), then you might be able to stream onto a shared resource and get
away with it.  As a practical matter, I don't believe that streaming
into a shared connection is advisable.

Still, the real issue is the one I quoted: forcing an implementation
to hold multiple partially complete header sets open is a superb DoS
vector.  Better than the HOL blocking we've been talking about, which
can be easily contained.

---

From: Johnny Graettinger <[email-masked]>
Date: Fri, 11 Jul 2014 15:06:31 -0400

> problem you can’t just wave away.

Without interleaving, such a proxy coalescing un-trusted users must buffer
at most *one* set of incomplete request headers per-user before forwarding.
As you note, it can't forward fragments. This prevents an attacker from
impacting other coalesced clients.

But with interleaving, such a proxy must buffer M different sets of
incomplete request headers for each of N different users. Requiring that
:headers appear in the first frame increases the likelihood that the proxy
is able to incrementally forward, but it doesn't eliminate it (again,
consider a reverse proxy routing on a session cookie). The only bound on M
right now is the stream concurrency limit, which is large.

---

From: Jason Greene <[email-masked]>
Date: Fri, 11 Jul 2014 14:40:53 -0500

> vector.  Better than the HOL blocking we've been talking about, which

You can do that the same (and prevent it the same) either way. I can create N connections with incomplete headers just as easily as I can create N streams. The server can prevent N streams just as easily as it can prevent N connections.

Roberto’s analysis is really just an argument that streaming is problematic for receivers, and to that I agree. If it’s benefits are to be limited to 1:1 connection proxies then I don’t think the benefits of the feature justify its complexity.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 12:46:43 -0700

> You can do that the same (and prevent it the same) either way. I can create N connections with incomplete headers just as easily as I can create N streams. The server can prevent N streams just as…

The limit that a server might permit is very different for streams and
connections, so it's not the same thing at all.

> Roberto’s analysis is really just an argument that streaming is problematic for receivers, and to that I agree. If it’s benefits are to be limited to 1:1 connection proxies then I don’t think the b…

It's more than that, it's an observation that concurrency limits
matter.  The current design limits this particular point to 1, not
SETTINGS_MAX_CONCURRENT_STREAMS.  That bounds state commitments in a
predictable fashion.

---

From: Jason Greene <[email-masked]>
Date: Fri, 11 Jul 2014 14:54:53 -0500

> connections, so it's not the same thing at all.

I can have 300k requests delivered over any ratio of connections to streams. Granted connects are a bit more throttled, and make a bit more noise.

> SETTINGS_MAX_CONCURRENT_STREAMS.  That bounds state commitments in a

The DOS attack is the amount of memory allocated per incomplete request. The server can track that, and it can easily RST_STREAM when it detects there is too much.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 11 Jul 2014 21:53:55 +0200

On Fri, Jul 11, 2014 at 02:40:53PM -0500, Jason Greene wrote:
> You can do that the same (and prevent it the same) either way. I can create N
> connections with incomplete headers just as easily as I can create N streams.
> The server can prevent N streams just as easily as it can prevent N
> connections.

And with streams, you're limited to MAX_CONCURRENT_STREAMS per connection.

That said, it's not a reason for dismissing a DoS risk for one's typical
use case. I think it favors the single-framed headers with no continuation
at all.

Currently continuation are a way to build a super-large atomic header frame
which prevents interleaving and breaks interactivity just as if an equally
large frame was set at once instead. So if we can all agree on 64kB max
frame size for the rare case where such large headers are required and get
rid of continuation, there's no more HOL blocking than with continuation
and we avoid the DoS caused by concurrent users sharing the same connection.

Regards,
Willy

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 13:00:55 -0700

> The DOS attack is the amount of memory allocated per incomplete request. The server can track that, and it can easily RST_STREAM when it detects there is too much.

That's true for your implementation, but I know that others may prefer
a simpler formulation:  work out available resources, work out what a
single connection can use, divide.  That way, you can do things like
better isolate bad behaviour on one connection from others (though you
lose some scaling advantage, sure).  Having the transitory header
processing costs for each connection be based on a multiple of the
stream concurrency limit - even if it's worst case - will make that
harder.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 20:07:03 +0000

In message <CANV5PPXaPUGEJWvSgAo2ftX82AL5Gqch1NnYVYXm7pP9N=[email-masked]>

>compressed header sizes in Firefox.

There are a lot of HTTP traffic never seen by Firefox, for instance
"internet of things" gadgets reporting to home-ships.

For these kind of applications 256 will be plenty.

The reason to have a minimum size in the first place, is to make
sure that you can actually do a "GET /" or "GET /robots.txt" and
similar "convention" requests to all servers, all the time.

If people configure there server for a 256 size, they get what
the deserve.  This is not a error-42 protection measure.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 20:09:00 +0000

In message <CABkgnnUeH-nVXmsC0GYd=UO6aSj+cbWk4a=[email-masked]>

>> excellent attack vector, which the design currently precludes).

This would be even more the case if we insist, as proposed, that all
headers go into a single frame.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 13:11:10 -0700

> headers go into a single frame.

You mean that it would help avoid having multiple incomplete header
blocks outstanding.  If so, then yes.  Knowing size up front means
that you can RST streams that you know will blow your limits (though
with compression, you can't be sure that a smaller frame won't).

---

From: <[email-masked]>
Date: Fri, 11 Jul 2014 20:12:58 +0000

Hi Martin-

What does...
5) Allowing interleaving of CONTINUATION frames with other frames.
...have to do with Roberto's analysis? Are you referring to his #1, unfinished header sets?

That is solved by removing the reference set. Each HEADERS/PUSH_PROMISE/CONTINUATION frame can be interleaved with other frames from other streams so long as the aforementioned frame types are processed in order.

(I don't buy that preventing state-changing hpack opcodes from crossing frame boundaries is so hard. If the *uncompressed* size of the next header is too big to fit in the remaining space, then finish up the current frame and start a new one.)

-Keith

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 20:23:55 +0000

In message <CABkgnnV=[email-masked]>, Martin Thomson w

>with compression, you can't be sure that a smaller frame won't).

I mean that if we insist the entire header-set goes into a single
frame at most *zero* set of headers can be incomplete at any point
in time.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 13:27:52 -0700

> frame at most *zero* set of headers can be incomplete at any point

That's a valid point, though I was considering the fact that you have
to read/buffer/process that frame, which takes non-zero time.  The
point being that whether it's 0 or [0,1), this is very much different
to N.

---

From: Johnny Graettinger <[email-masked]>
Date: Fri, 11 Jul 2014 16:30:44 -0400

> frame at most *zero* set of headers can be incomplete at any point

False. Just because I declare a frame size doesn't mean I have to send all
of it.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 20:38:10 +0000

In message <[email-masked]>, Johnny Graetting

>False. Just because I declare a frame size doesn't mean I have to send all

If you ever want to use the connection for anything else, you'll have to.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 20:37:09 +0000

In message <[email-masked]>, Martin Thomson w

>point being that whether it's 0 or [0,1), this is very much different

I'll agree that 1 is much easier to manage than N, but zero better still
by a significant margin.

The main difference between 0 and 1 is that we make it the senders job
to preannounce the memory requirement (ie: frame length).

This is the BIG thing in both DoS detection/mitigation and in efficiency.

---

From: Nicholas Hurley <[email-masked]>
Date: Fri, 11 Jul 2014 14:27:15 -0700

> "internet of things" gadgets reporting to home-ships.

Please, tell me more about how I know nothing about anything outside
Firefox. We've always been looking for hard, technical arguments in these
discussions, so I made one. I'm still waiting for one of yours.

> For these kind of applications 256 will be plenty.

And for those kind of applications, they are more than welcome to make a
convention in their implementation that they never send more than 256.
Saying the minimum max frame size is 4k is not the same as saying you must
send 4k in every frame.

> the deserve.  This is not a error-42 protection measure.

 But we shouldn't make it quite so easy for them to shoot themselves in the
face.

And with that, I'm done talking about the "internet of things", which is
such a vague statement that encompasses such varying capabilities in
endpoints that it's honestly meaningless to talk about in any real sense.
--
Peace,
  -Nick

---

From: Rob Trace <[email-masked]>
Date: Fri, 11 Jul 2014 21:30:42 +0000

We are OK with the first 4, but we agree that 5 is a bad idea.

-Rob

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Friday, July 11, 2014 10:48 AM
To: Greg Wilkins
Cc: Jeff Pinner; HTTP Working Group
Subject: Re: Striving for Compromise (Consensus?)

On 11 July 2014 00:48, Greg Wilkins <[email-masked]> wrote:
> 5b) The size of the HEADERS and CONTINUATION frames are removed from 
> the flow control window, but the they are never flow controlled.

4 doesn't seem to be well understood, but the interaction between TCP congestion window and something like the proposed 5b could mean some serious stalling/HOL issues.  More serious than the issues it purports to address.

I think that 5 is a non-starter.  Roberto's analysis on this has convinced me that this is an undesirable feature.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Jul 2014 21:53:58 +0000

In message <CANV5PPVDJchOc+FCfTtjA2Yg_n2+wAs=rQTwNB1mTSS4ZNyV=[email-masked]>, Nicholas Hurley 

>discussions, so I made one. I'm still waiting for one of yours.

You said your 4K size was based on firefox "telemetry".

I'm not in any way questioning that you know what firefox is doing
or seing, I'm pointing out that there is a whole universe of HTTP
requests that never get anywhere near Firefox,

If some random http-application doesn't need 4K, there is no rational
reason why we should force them to announce their willingness to
receive 4k frames.

But we should demand that basic HTTP interop will keep working
and in my perception that means that you can "GET /robots.txt"
and get a "I deny everything" entry back, and that you can
"GET /" and get a 3xx response back.

My back of the envelope says 256 bytes can do that, but HPACK
may even allow us to reduce it to 128.

> But we shouldn't make it quite so easy for them to shoot themselves in the face.

We are chartered to write a protocol, not a policy.

If 256 bytes is enough for basic interop, we shouldn't require more.

>endpoints that it's honestly meaningless to talk about in any real sense.

Too bad for you that they're specifically called out in our charter:

	non-browsers ("HTTP APIs")

Fortunately all we need to know about them is that they use HTTP
and generally use small transactions because they have really
small processors and bandwidths.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 15:37:26 -0700

> If 256 bytes is enough for basic interop, we shouldn't require more.

I find this reasonably convincing.  Interoperability is what we are
looking for here.

Though I'll note that the arguments regarding cost of implementation
are very much less convincing. Those devices, when presented with a
peer that doesn't know their capabilities, will have to deal with 16k
frames until their setting is seen.  And I'm not sure that 16k is so
hard for a constrained device to handle anyway.  I keep hearing about
what amazing things that even the most constrained hardware can handle
(TLS stacks, node.js, etc...).  So I'm disinclined to lend too much
weight to arguments about limited capabilities.

That said, if 256 is enough to get something working, then that's a
good argument.

An alternative approach would be to set the default and minimum to the
same value.  That has other benefits, like being able to completely
ignore a setting from the other side.  That seems pretty attractive.

---

From: <[email-masked]>
Date: Fri, 11 Jul 2014 23:06:02 +0000

On 12 Jul 2014, at 00:41, "[email-masked]" <[email-masked]> wrote:
> [snip]

Having the default and minimum the same value is a great idea - especially for resource constrained implementations that want to "ignore a setting from the other side". What value do you suggest though? I would be really, really, really surprised if anybody went for 256. For browser implementations, you'd have to wait up to one RTT before you know what the peer actually supports and can send a request safely - or speculatively send larger frames assuming the peer is going to increase from the default setting.

BTW, we still work with micro-controllers with ~100K, where 16K is a significant resource  commitment.

-Keith

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Jul 2014 16:13:28 -0700

> BTW, we still work with micro-controllers with ~100K, where 16K is a significant resource  commitment.

In those environments, do you have to deal with arbitrary peers?
Peers that might not be aware that you are so constrained?

Are those micro-controllers going to be using HTTP/2 with default
configuration such that they might get 100 requests immediately after
accepting the connection?

My point being that in environments where there is some hope of
control, you can avoid the worst by convention, standard or
gentlemen's agreement, not having to rely solely on what is in some
RFC.

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 12 Jul 2014 09:40:00 +1000

It appears that the reasons for the "Greg et al" proposal have been lost in
the debate and strawman arguments keep getting put up.   So here is a
reminder of what Large frames were proposed for - together without this
alternate proposal performs [as +/- 1]

Background

The HTTP2 protocol has a requirement to be able to transport large headers,
that exceed the payload size of a single frame at the current 16KB maximum
size.

To address this requirement, the current draft (13) includes the
CONTINUATION frames, 0 or more of which may be sent after a HEADERS or
PUSH_PROMISE frame to contain the large headers. There has been significant
criticism of the CONTINUATION design, including:

   -

   The total length of a HEADERS+CONTINUATION* sequence is not known until
   the last frame in the sequence is processed. A receiver that wishes to
   reject streams headers larger than a specific limit may have to process
   many frames and hold the results in memory before it discovers the header
   is too large.

[-1]

   -

   The size of header that an endpoint is prepared to receive is not known
   in advance. The only way a sender can know if a header too large is by
   attempting to send it and receiving an error in response. Error handling of
   headers may be difficult for an endpoint to handle efficiently and can
   result in the closure of the entire connection.

[-1]

   -

   The END_STREAM flag is not present on the CONTINUATION frame, thus it is
   possible for a stream to send CONTINUATION frames after a HEADERS frame
   that has the END_STREAM flag set. This is confusing and increases the
   complexity of the state machine required to process streams. It is highly
   desirable that a set END_STREAM flag truly indicates the last non control
   frame of a stream.

[-1]

   -

   There is a significant discontinuity in the code path required to
   process headers. Headers up to an indeterminate size (roughly 20-something
   KB) can be handled in a single frame. Headers that exceed this size must be
   handled in multiple frames of different types with different frame flags
   and stream control logic. Because the vast majority of headers sent
   (>99.99%) are below this indeterminate size, implementations will have a
   code path that is seldom executed and probably insufficiently tested. This
   invites poor and/or partial and/or incorrect implementation.

[-1]

   -

   Because of the HPACK compression algorithm, a sequence of
   HEADERS+CONTINUATIONS frames may not be interleaved with any other frame.
   This effectively makes the sequence a single large frame. Because of the
   simplicity of description and implementation it is proposed that it would
   be far simpler to meet the requirement of large headers by supporting large
   frames.

[+1]

This proposal has been prepared as it is possible to meet the requirements
of CONTINUATIONS without the complications and criticisms above.

This proposal addresses the issue of sending/receiving large HTTP headers
without giving endpoints and intermediaries unlimited resource commitments
nor unknown limits

[-1]

 Additional Frame Size Issues Addressed

The current draft (13) has maximum frame size of 16KB, which is an
arbitrary value that has been selected on the basis of experience to
provide a reasonable compromise between the efficiency of transmitting data
vs the quality of service for multiplexed channels.

Whilst this educated guess may be near optimal for today's networks and
traffic, it is entirely possible that some current and/or future networks
may require a different value to achieve an optimal balance. There have
already been proposals [1] put to the WG to reduce the frame size to
optimise multiplexing , as well as discussion that high capacity, low
latency networks can achieve satisfactory multiplexing quality of service
with large frame sizes.

This proposal addresses the issue that a fixed frame size does not allow
tuning multiplexing performance based on current/future experience.

[+0]

 It has also been noted that 16KB is near the middle of the peak of the
current HTTP Object size histogram [2], so that a small change in the frame
size may have a significant impact on the number of HTTP messages that can
be sent in a single frame, without significant impacts on QoS. The HTTP
Object size histogram has changed significantly over time and is expected
to continue to do so.

This proposal addresses the issue of tuning the frame size based on
experience of actual payload sizes.

[-0]

There have also been issues raised that a 16KB frame size does not allow
efficient data transfer [3] even when the end points are aware that only a
single stream is likely to be required for the imminent future, or
that a particular stream is of high priority.

This proposal addresses the issue of tuning the frame size for transport
efficiency for specific streams in specific situations.

[-1]

So anything that keep continuations as part of a normal implementations
just fails to address these concerns that were well received at the time.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 11 Jul 2014 18:27:15 -0700

On Fri, Jul 11, 2014 at 1:12 PM, <[email-masked]> wrote:

> That is solved by removing the reference set. Each
> HEADERS/PUSH_PROMISE/CONTINUATION frame can be interleaved with other
> frames from other streams so long as the aforementioned frame types are
> processed in order.
>
>
Unfortunately, these are orthogonal.
The reference set has zilch to do with the DoS vector here, as the DoS
vector would still be present even if the reference set didn't exist.

-=R

> (I don't buy that preventing state-changing hpack opcodes from crossing
> frame boundaries is so hard. If the *uncompressed* size of the next header
> is too big to fit in the remaining space, then finish up the current frame
> and start a new one.)
>
> -Keith
>
>
> This email message is intended only for the use of the named recipient.
> Information contained in this email message and its attachments may be
> privileged, confidential and protected from disclosure. If you are not the
> intended recipient, please do not read, copy, use or disclose this
> communication to others. Also please notify the sender by replying to this
> message and then delete it from your system.
>
>

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 15:33:56 +1200

> 2 don't stream, no need for max_size frames (!4), no interleaving (!5)

Strange that Roberto's argument for keeping status-quo was that we need
it to stream headers. But you summarize status-quo under "don't stream".
I find that case (0) above is closer to status-quo.

> changes, seems to actually be the worst in this analysis.

Matrix case (2) has more in common with "Greg et al" large frames
proposal than status-quo h2-13.
 * dont stream (large frames requests buffering on sender)
 * no need for max_size frames (large frames sends frame size matching
payload)
 * no interleaving (Greg et al large frames does not permit even
mistaken implementers to accidentally interleave frames inside each
others payload)

Greg et al proposal causes the '-' bullets to change thusly:

 - "No state commitment DoS concern" - Greg et al frame limit removes
implementations protocol mandated commitment to handle infinite-length
CONTINUATION payload. Greatlyimproving this over the status-quo.

 - "May spend extra cycles on framing" - Greg et al frames remove this
issue completely.

 - "May have HOL" - no change here, but Greg et al frames permit senders
to mediate frame size and prevent HOL blocking on outgoing connections.
status-quo actually mandates buffering "state commitment DoS concern" as
the only possible way to resolve HOL blocking.

 - "No stalling" - stalling is *always* possible with frames overhead
about 3 bytes. It is a TCP behaviour. h2 has frame overhead 8-12 bytes.

As for your priority order:

> Given that my priority order here is approximately:

> state commitment >

Greg et al. large frames leave the state commitmet equal to status quo
unless implementer decides otherwise. Implemeter may advertise *less*
state commitment than h2 specification default.

> stalling

Stalling is possible in all circumstances where more than two bytes need
to be strung together. status quo is *very bad* in encouraging this, as
are the interleaving and fragmentation proposals.

Greg et al large frames require the sender to accumulate the frame
before delivery to ensure length is accurate. This actively closes many
paths and potential HTTP/2 implementation mistakes leading to stalling.
The remaining causes are all TCP induced and need to be resolved at the
TCP layer.

> HOL

Only occurs with status quo and proposals involving CONTINUATION or
fragments.
Greg et al large frames constrains senders not to send at all unless the
frame is fully available and can be delivered.

> CPU, I think that status quo is looking pretty good.

I think that Greg et al frames proposal still looks better.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 15:42:37 +1200

> ignore a setting from the other side.  That seems pretty attractive.

When I sueggested that same point some time back you convinced me that
is was perfectly reasonable for a constrained device to reject a number
too-big frames and streams on connection initiation. Nothing has changed
since then. So why is it now such a burden to reject some frames up
front on a constrained processor?
 If the constrained processor is going to pick 256 in the first place
its something like an 8-bit processor counting memory by the byte. Even
16-bit ATOM these days have a few hundreds of KB of RAM attached.

Amos

---

From: Roberto Peon <[email-masked]>
Date: Fri, 11 Jul 2014 20:57:42 -0700

In cases where HOL blocking is acceptable, you stream.
This can be in cases where you know the entity that is sending the headers
you'd be streaming is trustworthy.

If you don't want HOL blocking you don't stream.
These don't conflict, they're simply different tradeoffs.
-=R

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 16:03:56 +1200

> impacting other coalesced clients.

This is an incorrect conclusion IMO. By requiring that the proxy buffer
without providing a total input size in advance the proxy is forced to
commit unlimited resources to *each* client. And the one which consumes
too many resources on the proxy causes stalling down each of the other
clients connections.
 Like Martin said in his priority matrix earlier; stalling is more
important than HOL.

The effect is identical to piping 9 straws and a fire hose into a sealed
container with 10 straws leaving it. The 9 input straws are going to get
backed up while the firehose content is dealt with regardess of whether
any of the firehose input gets sent on the outbound.

This is what CONTINUATION means in h2. It's bad, really bad.

> right now is the stream concurrency limit, which is large.

Agreed. Interleaving on a stateful connection (HPACK state) makes the
effects worse. When the effect without interleaving is enough to cause a
DoS there is no sane reason to believe amplifying it will help.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 16:16:54 +1200

On 12/07/2014 8:27 a.m., Martin Thomson wrote:
> On 11 July 2014 13:23, Poul-Henning Kamp wrote:

In Gerg et al framing there is nothing forcing any recipient to buffer.
It is entirely possible to stream a large frame straight into the HPACK
decoder then PROTOCOL_ERROR the connection on HPACK errors. Such
recipients will be required to buffer the decoded headers, but this is
unrelated to the framing.

The frame size up front is to:
 a) identify clearly whether to RST/431 the stream then stream the frame
into decompressor (unbuffered!) to keep consistent HPACK state. Or if
buffering

 b) shift buffering to senders as intentional incentive for senders to
avoid huge frames. They are able to do so if the recipient permits, but
incentivised not to by all the recipients which advertie small frame limits.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 16:22:15 +1200

On 12/07/2014 8:38 a.m., Poul-Henning Kamp wrote:
> In message Johnny Graetting er writes:
> 

If you advertise a large size using Greg et al frames then don't send it
the recipient will 431 respond (if its over their limit), PROTOCOL_ERROR
on a timeout. No other party or connection was harmed in any way by your
action.

If you send a HEADERS without END_HEADERS flag in h2-13 then pause, you
stall the recipient and corrupt their HPACK state.
 Also, any recipient which performs fragment streaming (eg Roberto's
implementations apparently) you will stall and corrupt the HPACk state
of the recipients outbound connections. Thus stalling all other client
connections it has interleaved.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 16:31:53 +1200

> are currently permitted to be sent but have no "semantic" mapping to

Er. So up-front knowledge of information X utilizing 8 bytes of protocol
header is important for reducing 1-bit of boolean state.

Why is information X more important than the size of the compressed
headers which is to follow? so important that its worth burning overhead
bandwidth at a ratio of 8:1 but frame size is not worth expanding at a
ratio of 2:65535 or better?

I do agree that types are important info to have up front. But size
being known up front is equally important, and has a far better
cost/benefit ratio.

> divisible which makes flow controlling them difficult

Thank you. You have given solid reason here for dropping CONTINUATION.
Which is h2-13 splitting of HEADERS content over multiple frames.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 16:40:11 +1200

> These don't conflict, they're simply different tradeoffs.

So let me ask you this:
 Why are you arguing so feircely against a proposal which sends up-front
the information to identify whether you need to make such a trust
decision in the first place?

Say you have an implementation that wants to stream. You arbitrarily
decide to send to your clients a fixed 32KB frame limit, and stream
anything under 20KB that comes back. When your clients send you new
HEADERS using Greg et al frames you get told whether it is 16KB, 256
bytes or 30KB in the frame intro. Make your decision before the first
byte of headers payload has been processed and take the
stream/non-stream code path to handle it.
 How does one do that with HEADERS+CONTNUATION? stream up to 16KB then
stall? *buffer* a "streamed" frame?

The simplest solution to these and many other issues is what Greg et al
are proposing.

Amos

> 
> On Fri, Jul 11, 2014 at 8:33 PM, Amos Jeffries wrote:
> 
>>> On 11 July 2014 11:08, Willy Tarreau wrote:

---

From: Roberto Peon <[email-masked]>
Date: Fri, 11 Jul 2014 22:17:46 -0700

> decision in the first place?

That information, about the size of the header, is not trustworthy in a DoS
scenario.

If the information about the supposed size of the header is used to
allocate buffers, it can create an asymmetric DoS scenario, where the
attacker is spending much less resource than the server-side is spending.
One accomplishes this by sending a large header size, and then either never
fulfilling the promise of those bytes. Very little bandwidth gets wasted,
and lots of memory gets consumed.

The obvious work-around is to allocate buffers as they're consumed, which
is essentially what would happen with header fragmentation.

Another issue: Lets say that the server has an internal idea of the amount
of memory it wishes to commit.
It sees that the advertised header size is greater than it is willing to
accept.
It rejects it immediately. It has now leaked information to the attackers
about the state of the server.

And then there is the fact that we do have compression, and one can make
promises till blue in the face about how much memory one is willing to
accept compressed, and it has zero to do with the state commitment
uncompressed if the server has to interpret the headers.

So, the way I look at it, it offers no practical benefit while placing
restrictions on proxies, tunnels and non-interpreting gateways w.r.t.
buffering which induce latency and which increase state commitment even
when they don't interpret. The bad outweighs the good.

> stall? *buffer* a "streamed" frame?
Why would I set such a limit?
I'd be better off setting a limit on the decompressed size, since that is
more likely to correspond to what the server has to have resident in order
to interpret it.

Best case is ~compression-state-size + compressed frame size.
Worst case is  ~compression-state-size * compressed_frame_size / 2

Even then, why advertise it? It is better to consume the headers, and give
the attackers no direct indication of the effect of the attack.
-=R

> the

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 12 Jul 2014 15:56:59 +1000

Roberto,

So it sounds like you don't like continuations any more than the proposers
of "Greg et al".
So here is a path to something that will keep everybody happy,

Support the proposal - at some frame size >=18 <=31 bits. Then
continuations are gone and the two down sides for you are:

   1. Can't stream headers
   2. We advertise a max header size (it is a frame size but the same thing
   in proposal).

Now once we have that in place, we look at some new proposals to address
those concerns.

If we want to be able to stream headers, then we can put forward a proposal
to allow header frames to be fragmented in the same way data frames
currently are. ie we use the end-segment flag that they already carry to
indicate end of a header block.    Then the same framing layer code can be
used for fragmenting data and headers.      The discussion around that
proposal would also settle if they can be interleaved or not.   Hpack would
then be updated to work with whatever results (eg by removing reference set
or whatever else is necessary).

Once we say that headers can be fragmented, then a max header frame size
setting is a bit meaningless, so we can propose to drop that and just have
a single max frame size.      We can have the debate if there should be a
new setting of max header size (expressed as a uncompressed size) or not.

cheers

On 12 July 2014 15:17, Roberto Peon <[email-masked]> wrote:

> That information, about the size of the header, is not trustworthy in a
> DoS scenario.
>
> One accomplishes this by sending a large header size, and then either
> never fulfilling the promise of those bytes. Very little bandwidth gets
> wasted, and lots of memory gets consumed.
>

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 12 Jul 2014 16:06:17 +1000

On 11 July 2014 23:18, Jeff Pinner <[email-masked]> wrote:

> > I'm with Greg, though, what is the reason for needing a distinction
> between HEADERS and DATA frames?  The same thing could easily be
> accomplished with just DATA frames logically divided into a http request of
> "headers data [trailers]" just by the END_SEGMENT flag.
>

Well firstly the framing layer shouldn't have to know how the contents of
the frame need to be processed.  That is higher layer business.    It is
only because we have built in stateful HTTP header compression into the
framing layer that this is required information.

But it would be known anyway.  It is not hard to say that the first data
segment of a HTTP stream contains headers.

But hey, I can accept calling them HEADERS instead of DATA, so long as we
have the same fragmentation mechanism rather than a different one.

> divisible which makes flow controlling them difficult

Again this is only because of the conflation of layers.   Data cannot be
arbitrarily split either when considering its semantic meaning.  It is only
because the reassembly occurs before the data is interpreted that we can
fragment data.  So its easy, lets kick HEADER decoding out of the framing
layer and then decoding headers is just the same as decoding XML data as
far as fragmentation is concerned.

Besides, I don't think there is any requirement currently to split a
HeaderBlock into Header Block Fragments on a field boundary.  In fact the
spec allows for an individual field to be of arbitrary size, so one field
may cross several frames.

regards

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 12 Jul 2014 16:46:24 +1000

> As currently specified CONTINUATION is a giant invitation to DoS attacks.

... and we're talking about changing the protocol to address that. Your way to do that is to get rid of CONTINUATION; there are other ways too.

This is in our issues list as #550 and #551; "get rid of CONTINUATION because PHK thinks it's ugly" is not.

Thanks,

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 12 Jul 2014 16:46:10 +1000

> I agree that since this has turned into a full-scale discussion of non-editorial changes there may be some "wishes" creeping in.  But I take issue with you calling this a wish list just because I d…

It's not that you didn't copy Greg's e-mail; it's that those are not issues as far as the WG is considered; they're not shared goals. We have identified the underlying issues (#549-553), and will address them.

I.e., while your purpose may be to get rid of CONTINUATION, we don't recognise that as a substantive technical issue. Likewise, adding bits and settings is a means to an end -- addressing the underlying issues -- not an end unto itself.

There are many ways to address those issues, and we're talking through a variety of them now. I'm looking for the minimal viable change to address these issues, gain consensus and ship the spec, because that's what our obligation is.

If your proposal is able to achieve consensus, that's great -- I'm not ruling it out by any means.

> Gladly! But they better have some pretty sound justification as to why the "optimal" frame length for efficient and responsive multiplexing magically changed from 16K to 64K!!!

The onus is very firmly upon those proposing a change from the WG has previously agreed to, not those who are trying to accommodate the people proposing the change. Doing it your way would only encourage people to become inflexible, and that's not what we want.

> This is NOT a tweak!  If header fragmentation can be accomplished with 1*HEADERS instead of HEADERS+CONTINUATION*, then keeping CONTINUATION is just keeping complexity for the sake of being too bot…

The fact that we have running code is good enough for now. If there's a substantive technical issue that making this change can address, we'll discuss it. If we can achieve easy consensus (i.e., no great controversy) on an aesthetic change to the protocol at this late stage, great. What I'm absolutely not willing to do is to have another 100-email thread on what colour the bikeshed is painted (nod to your collaborator there) while we still have substantive issues outstanding.

Thanks,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 12 Jul 2014 06:57:26 +0000

In message <[email-masked]>, Martin Thomson w
>Peers that might not be aware that you are so constrained?

Do they risk arbitrary peers which send requests filled with either
junk headers or cookies that don't belong ?

Not unless somebody is deliberately trying to sink them with a DoS.

Spiders are very careful to send very small requests initially,
because they get blocked if they cause trouble.

A browser pointed first time at your new $devices knows nothing
about it and simply does a "GET /" and 256 bytes is enough for
that.  It may get a SETTINGS and a page back, or it may get
a redirect to the vendors web-farm, with the serial number
in the :path or :query fields.

>accepting the connection?

I don't see that happening, and if it does, it is a DoS.

A single browser would have to encounter 100 (different!) links on
a webpage to the $device for that to happen.  Somebody put those
links there, it's reasonable to expect them to have tested the page
and they should have noticed that it didn't work too well.

And if your thermostat gets to the front page of reddit, you
don't seriously expect it work handle the traffic, do you ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 12 Jul 2014 06:50:45 +0000

In message <CABkgnnUTMTfR51hZJ__CpDWA0C8pG6HHBih9HVgKoZ=WH5AS=[email-masked]>, Martin Thomson w
>ignore a setting from the other side.  That seems pretty attractive.

There were no default in our proposal, only a minimum and a MUST
that 16k *can* be configured.

I don't see any reason, nor any gain, by forcing a server to accept
16K (compressed!) header-sets by default, if the application it
runs is never ever going to be subject to valid requests larger
than 3K ?

Who would benefit from that ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 12 Jul 2014 07:47:44 +0000

In message <CAP+FsNcuUm=hp4XF=MQX8c644vYkCRCeJsXpXYk2L=[email-masked]>

Roberto,

That doesn't make *any* sense.

If the attacker sends a bogus frame length, it doesn't matter if
it is 16 bogus bits or 31 bogus bits he sends, it's going to
desynchronize the framing layer and that's the end of the story.

>attacker is spending much less resource than the server-side is spending.

Only a fool would allocate buffers based on what an untrusted peer
tells him.

Buffers will be allocated based on the max frame size the receiver
has decided, in our proposal he announces this with SETTINGS.

If a sender sends a length longer what SETTINGS told it, I bet you
that almost everybody is going to conclude "DoS!" and just close
the connection.   This is a major advantage of the SETTINGS:  the
client cannot do the "but I didn't know.." routine.

>is essentially what would happen with header fragmentation.

No, this is not at all an obvious way to do it, because you move
the memory issue from size of allocation to number of allocations
and you force the high-pressure points to use non-optimal discontiguous
memory slices to store what in all legitimate traffic is contiguous
data.

In other words that makes the attack worse:  The attacker now forces
to you use more memory since you have the overhead of linking the
buffers together, and the CPU overhead of having to respect that
linkage.

>about the state of the server.

How is this different from it telling outright in the settings ? :-)

How is the attacker not able to similarly probe the server using
multiple CONTINUATION frames ?

>uncompressed if the server has to interpret the headers.

As I said earlier:  High-performance implementations are not going
to decompress headers until they are needed:  Why should a load-balancer
decompress User-Agent or Cookies if it doesn't care about them ?

>when they don't interpret. The bad outweighs the good.

The way you look at it seems to have very little to do with how
high-traffic-pressure HTTP devices actually work and cope with DoS
attacks...

---

From: <[email-masked]>
Date: Sat, 12 Jul 2014 21:29:39 +0000

On 11 Jul 2014, at 15:22, "[email-masked]" <[email-masked]> wrote:

> message is currently being received to decode the data properly.

Once again, thanks for taking the time to answer my questions.
In /1 you have to keep track of where you're parsing (eg headers come before "\r\n\r\n"). But, if I infer correctly from your response, the problem is that you have to decompress the headers frame before passing it to the application layer because of the shared compression state. Curse that stateful compression...

> It

I'm all for this, but I've said this before, and I'll say it again. Re-using HEADERS frames for this purpose is confusing. Are they hpack encoded? What does it mean if you get an intra-message HEADERS frame and END_HEADERS is set? Is END_STREAM allowed?

Why not just add a simple METADATA frame and be done with it?

> second frame needs an RTT to be sent, stalling all header processing.

Couldn't you solve this scenario by sending 2 bytes of padding to drain the window? Of course you wouldn't want to compress and then have to rewind - so I wouldn't ever compress unless I knew the worst case (literal w/o Huffman) could fit in the remaining window.

-Keith

---

From: <[email-masked]>
Date: Sat, 12 Jul 2014 21:35:59 +0000

https://github.com/http2/http2-spec/issues/557

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 13 Jul 2014 07:35:00 +0200

Hi Martin,

sorry, it took me a while to understand how to read your matrix,
but this morning the coffee helped me :-)

I have a comment below.

> changes, seems to actually be the worst in this analysis.

Option 5 allows agents with smaller buffers to reject CONTINUATION frames,
which removes state commitment and DoS concern. It seems to be the closest
to the situation we have in 1.1 where a message that does not fit in the
recipient's buffers is simply rejected (eg: 400 or 502).

Option 2 becomes better when you enable max_frame_size since it removes
the "May have HOL".

Best regards,
Willy

---

From: Erik Nygren <[email-masked]>
Date: Sun, 13 Jul 2014 16:04:15 -0400

I'd tend to agree that re-using HEADERS for intra-DATA is possibly
confusing, but there will likely be use-cases.  (There was a thread that
discussed this around two months or so ago --- it gets more interesting for
when the END_* flags come into play.)

As for "semantic" mapping to HTTP/1.1, chunked encoding extensions are the
best match.  ICAP has one defined, and I'm aware of at least one
proprietary chunked encoding extension used for hop-by-hop data integrity.

On Sat, Jul 12, 2014 at 5:35 PM, <[email-masked]> wrote:

> > I'm all for this, but I've said this before, and I'll say it again.
> Re-using HEADERS frames for this purpose is confusing. Are they hpack
> encoded? What does it mean if you get an intra-message HEADERS frame and
> END_HEADERS is set? Is END_STREAM allowed?
> >

---

From: Martin Thomson <[email-masked]>
Date: Sun, 13 Jul 2014 13:17:39 -0700

I try to avoid caffeine for that exact reason :)

> recipient's buffers is simply rejected (eg: 400 or 502).

It makes it easier to detect a bad actor, sure.  I don't think that it
changes the equation any though.  If you have a settings mismatch
between hops, then you trade HOL safety for buffering and stalling.

I think that this is just one of those cases where there are bad
options no matter what combination you choose.

> Option 2 becomes better when you enable max_frame_size since it removes

I think that we've converged on saying that max_frame_size (and
max_header_block_size) have some useful properties in that regard.

---

From: Martin Thomson <[email-masked]>
Date: Sun, 13 Jul 2014 13:24:47 -0700

On 13 July 2014 13:04, Erik Nygren <[email-masked]> wrote:
> best match.  ICAP has one defined, and I'm aware of at least one proprietary
> chunked encoding extension used for hop-by-hop data integrity.

Most of those extensions are not able to traverse any public networks,
nor can they be created or consumed in regular APIs.

We've tried really hard to avoid adding features for which we don't
have a use case.  I think that we really need to take a hard look at
this and consider how much more than HTTP we need to support.

---

From: <[email-masked]>
Date: Mon, 14 Jul 2014 09:55:51 +0000

On Sunday,13 July 2014 22:25, [email-masked] wrote:
> Most of those extensions [e.g. chunked encoding extensions] are not able to
> traverse any public networks, nor can they be created or consumed in regular APIs.
>

Agreed, however they were somehow possible in /1.1 (not a good enough reason to keep the feature IMO, but has been an argument used by others, including yourself, for keeping other features e.g. unlimited headers).

> We've tried really hard to avoid adding features for which we don't have a use
> case.  I think that we really need to take a hard look at this and consider how
> much more than HTTP we need to support.

Agreed.  I'm +0 on intra-message HEADERS frames, but if we keep the feature I suggest one of two paths forward:

1) At least add a section in the spec making it obvious this feature is available and give guidance for senders and recipients on how to use/deal with them (specifically, I'm thinking of the flags).

2) Add a METADATA frame type specifically for this purpose (the better option IMO)

-keith

---

From: Johnny Graettinger <[email-masked]>
Date: Mon, 14 Jul 2014 12:29:20 -0400

On Sat, Jul 12, 2014 at 1:56 AM, Greg Wilkins <[email-masked]> wrote:

>    2. We advertise a max header size (it is a frame size but the same
>    thing in proposal).
>
> If we want to be able to stream headers, then we can put forward a
> proposal to allow header frames to be fragmented in the same way data
> frames currently are. ie we use the end-segment flag that they already
> carry to indicate end of a header block.    Then the same framing layer
> code can be used for fragmenting data and headers.      The discussion
> around that proposal would also settle if they can be interleaved or not.
> Hpack would then be updated to work with whatever results (eg by removing
> reference set or whatever else is necessary).
>

This sounds an awful lot like removing continuation & header fragmenting,
just to re-add it back in later.
It presents exactly the same debate we're having now: Do we preserve the
ability to fragment headers? Do we allow the interleaving of those
fragments?

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 15 Jul 2014 09:17:26 +1000

> ability to fragment headers? Do we allow the interleaving of those

I would adjust this to "removing continuations done badly to add then
discuss adding them again done properly!"

The "Greg et al" proposal's concern is not against fragmentation - it is
against having 2 fragmentation mechanisms in the one protocol and with one
of them having non-obvious and complex impacts on the state machine.

I think Mark has now decomposed the issues rather well, and we can now
address the issues point by point and debate "should headers be
fragmented/interleaved".   Those are good debates to have and we can only
really have them properly once we have got the current bad fragmentation
mechanism out of the way.

cheers