Subject: Fwd: Stuck in a train -- reading HTTP/2 draft.

From: Mark Nottingham <[email-masked]>
Date: Tue, 17 Jun 2014 07:30:07 +1000

See input from PHK below, forwarded with permission.

I’m going to open an editorial issue so that Martin can take this as input; if he (or anyone else) feels that the points that PHK raises need more discussion, please say so.

Regards,

Begin forwarded message:

> From: "Poul-Henning Kamp" <[email-masked]>
> Subject: Stuck in a train -- reading HTTP/2 draft.
> Date: 16 June 2014 10:50:31 pm AEST
> To: Mark Nottingham <[email-masked]>

[…]

> Page 12: Reserved bits
> ----------------------
> 
> Historically RFCs have defined reserved bits opposite what would
> have been most useful later on approximately 50% of the time.
> 
> I would split the difference:  A R1 single-bit field, which must
> be sent as zero and ignored on reception, and a R2 single-bit field,
> which must be sent a zero, and where receiving a one is an error.
> 
> Page 22ff: Priority
> -------------------
> 
> I think the PRIORITY stuff is going to be trouble and will need
> revisions later on to truly work.
> 
> It will also be the first thing implementors skip in order to get
> something working fast, and they may never come back and implement
> it, unless serious numbers show it's worth the complexity.
> 
> I would restrict treatment of PRIORITY in this main RFC to a outline
> format of the frame starting with a one byte version number, and
> where the frames can go etc.
> 
> Interpretation of the frames for version #0 I would eject to a
> separate optional RFC, to make the main RFC smaller, simpler and
> thus more appetizing.
> 
> In HEADERS the PRIORITY flag should just enable a 32bit field,
> the intepretation of which goes in the side-RFC.
> 
> 
> Page 28ff: Padding
> ------------------
> 
> If padding is intended for obscuration of traffic patterns, it should
> be available in all frames to avoid "housekeeping" frames from leaking
> information about the relationships of other frames.
> 
> For this reason I would make padding part of the framing format,
> rather than the individual message types.
> 
> I'm also sceptical of demanding that padding be zeros, historically
> that has been a popular recipe for staring through compressions and
> encryptions.  I would probably specify that it is up to the sender
> to decide what to pad with, with guidance to prefer PRN bits rather
> than predictable (ie: zeros) or meaningful data.
> 
> Page 28: Typographical
> ----------------------
> 
> It's a bad idea to end a sentence with "... %d." because of exactly
> what happens in draft 12:  It might bet pushed to the next page
> and confuse everybody.   s/1/one/ ?
> 
> Page 30  HEADERS/SEGMENTS
> -------
> 
> What does END_SEGMENT do on a HEADERS frame ?
> 
> It looks like Copy&Paste error.
> 
> If it's needed, no text explains what it does and it's missing on
> CONTINUATION.
> 
> It's bogus that a lot of text separates HEADERS from CONTINUATION.
> 
> It's also pretty non-obvious why we need CONTINUATION in the
> first place:  Why not simply send HEADERS until one of them has
> the END_HEADERS bit set ?
> 
> 
> Page 34: SETTINGS
> ------------------------
> 
> Are we absolutely certain that we will never need per-stream SETTINGS ?
> 
> At least I would specify SETTINGS with non-zero stream to be ignored
> so that we can open that later.
> 
> Page 34: SETTINGS_MAX_CONCURRENT_STREAMS
> ----------------------------------------
> 
> I would make the this field mandatory in the initial SETTINGS frames
> in order to make sure people think about the DoS potential.
> 
> I also seriously doubt that anybody with a relevant DoS risk is
> going to open with a default value of "infinite" until they have
> seen some kind of "proof of work" on part of the client.
> 
> I suspect such DoS mitigation will cause a lot of SETTINGS frames,
> but that is less prone to failure than trying to define a "slow-start"
> without any benefit of real-life DoS mitigation experience.
> 
> (You should *seriously* consider trying to arrange an INTEROP with
> the specific goal of investigating DoS resistance.)
> 
> Text should be added to explain what happens if N streams are
> open but (x < N) SETTINGS_MAX_CONCURRENT_STREAMS is sent (the
> already open streams survive, new streams must wait for them to
> finish ?)
> 
> What happens of SETTINGS_MAX_CONCURRENT_STREAMS gets set to zero
> and stays there ?  How long time does the client wait for before
> opening a new connection ?
> 
> Are clients allowed to open multiple connections to bypass the
> SETTINGS_MAX_CONCURRENT_STREAMS limit ?
> 
> Are proxies ?
> 
> 
> Page 36: SETTINGS synchronization
> ---------------------------------
> 
> This is utterly bogus.
> 
> There is no way a SETTINGS frame can overtake any other frame while
> in transit, and there is no way the receiver can refuse the SETTINGS.
> 
> There is also none of the settings which require synchronization between
> the ends or which the sender cannot prepare for before sending the
> SETTINGS frame.
> 
> Simply demand that SETTINGS be applied atomically when received, before
> processing any subsequent frames and we're done.
> 
> If the sender *really* wants to know that the SETTINGS has been
> registered in the far end, it can send a PING after the SETTINGS
> and wait for the PING+ACK.
> 
> As it is now, SETTINGS causes the loss of an RTT if implemented as
> specified.
> 
> Page 39: GOAWAY
> ---------------
> 
> I would give the sender the ability to add a small-ish integer
> to the Last-Stream-ID, as a way to avoid wasted work.
> 
> Doing that, I think an opening gambit of
> 
> 	SETTINGS(MAX_CONNCURRENT_STREAMS = n)
> 	GOAWAY(Last-Stream-ID = n)
> 
> could be a very efficient DoS mitigation strategy, which would give
> legitimate users a chance.  In particular if we allow GOWAY to
> update the Last-Stream-ID once the client has proven non-hostile.
> 
> In fact, it's not obvious to me why GOAWAY isn't simply a field
> in SETTINGS to begin with ?
> 
> Page 42:  Flow Control Window
> -----------------------------
> 
> Again:  Why is WINDOW_UPDATE a separate frame type ?  a field
> in SETTINGS would do just fine.
> 
> Doing that would also allow the connection FCW to be updated
> subsequently.
> 
> For obvious DoS reasons, it's a mistake to not apply flowcontrol
> to HEADERS.
> 
> I would allocate frame Type values so that the top bit indicates
> "subject to flow-control" to simplify implementation.  Similarly
> expending a bit to mark hop-to-hop frames would make life easier
> for high performance implementations.
> 
> I also think it is a mistake to exempt the 8 byte frame header from 
> the FCW:  You can fill a lot of pipe with 1 byte DATA frames when
> the overhead isn't counted.
> 
> The explanation about SETTINGS_INITIAL_WINDOW_SIZE causing negative
> window-sizes is somewhat clumsy because some unrealted stuff breaks
> the flow.  Maybe move "A SETTINGS frame cannot alter the connection
> flow control window." and "An endpoint MUST ... FLOW_CONTROL_ERROR"
> out of the way.
> 
> Page 49: BLOCKED
> ----------------
> 
> In general sending data to tell you cannot send data is a non-bright
> idea, that has been discared in almost all protocols for the last
> three decades.
> 
> If the goal is real-time tuning, information about how many bytes
> are stuck in queue should be communicated.
> 
> If the goal is offline statistics, it would be sufficient to report
> at the end of each stream how many times the window closed and
> the max number of bytes we saw queued on a closed window.
> 
> Page 50: Error messages
> -----------------------
> 
> The RFC should propose text-messages for presentation purposes
> bearing in mind that they will be seen and reported by grandmothers
> and brothers in-law.  For a good example how not to do this,
> see the gai_strerror(3) default implemenations.
> 
> Page 51: out of order flags
> ---------------------------
> 
> This is just asking for trouble...
> 
>   The last frame in the sequence bears an END_STREAM flag, though a
>   HEADERS frame bearing the END_STREAM flag can be followed by
>   CONTINUATION frames that carry any remaining portions of the header
>   block.
> 
> The last frame should have the END_STREAM flag, always, there's
> no excuse for "whoops" details like this.
> 
> Page 51: More details in example
> --------------------------------
> 
> The example in 8.1 should show where the END_HEADERS and END_STREAM bits
> are set.
> 
> Page 51:  To strict ordering ?
> ------------------------------
> 
>   Other frames (from any stream) MUST NOT occur between either HEADERS
>   frame and the following CONTINUATION frames (if present), nor between
>   CONTINUATION frames.
> 
> Isn't this needlessly strict ?  No harm would come from DATA frames
> or SETTING frames being stuffed in there.
> 
> All this trouble could be avoided by only submitting headers for
> decompression, as a unit, when the END_HEADERS have been received.
> 
> Page 51: Gibberish
> ------------------
> 
> This needs to be translated to human readable form:
> 
>   Otherwise, frames MAY be interspersed on the stream between these
>   frames, but those frames do not carry HTTP semantics.  In particular,
>   HEADERS frames (and any CONTINUATION frames that follow) other than
>   the first and optional last frames in this sequence do not carry HTTP
>   semantics.
> 
> 
> Page 51: Trailing headers
> -------------------------
> 
> Support for trailing headers should have a bit in SETTINGS.
> 
> How dow we know the DATA section is done if there are trailing headers ?
> 
> By Content-Length ?  (what if it doesn't match ?)
> 
> By appearance of the first (trailer-)HEADER ?
> 
> Trailing headers should be marked in a way that does not force a
> high-perf load-balancer to decode all headers to find out.  Setting
> a flag bit in the first block of HEADERS/CONTINUATION is free for
> the sender (they have to find out at that time anyway) and saves a
> lot of time for the receiver.
> 
> Page 55: reverse order
> ----------------------
> 
> It would make more sense if the ":" header fields were described
> before they appear in the examples on the preceeding pages.
> 
> page 57: query part
> -------------------
> 
> The query part of the URI should get its own ":query" psuedo-field
> for reasons of othogonality and to reduce the amount of data typical
> load-balancers have to examine.
> 
> Page 59: cookie compression needs example
> -----------------------------------------
> 
> Basic confusion-prevention.
> 
> Page 59: Clarification
> ----------------------
> 
> 	[...]the sum of the uncompressed DATA frame payload lengths [...]
> 
> Should probably (better) clarify that 'uncompressed' obny pertains to the
> per frame GZIP compression.
> 
> 
> I havn't looked at the PUSH stuff, I still consider it a fundamentally
> bad idea.
> 
> 
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> [email-masked]         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe    
> Never attribute to malice what can adequately be explained by incompetence.

--
Mark Nottingham   http://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Mon, 16 Jun 2014 16:28:43 -0700

This is overall quite reasonable feedback.

>> From: "Poul-Henning Kamp" <[email-masked]>

Randomly?  I think that we could probably make a few sensible
decisions along these lines.  Probably best to handle on a
case-by-case basis though.

For instance, the bits leading up to the length field in the frame
header could be moved from R1 to R2 under this taxonomy and I'm happy
to do that if others think that it's a good idea.
https://github.com/http2/http2-spec/issues/531

>> Page 22ff: Priority

That seems to be an increasingly popular opinion.  As of this moment,
it's marked at risk and unless we get the serious numbers, I suspect
that it will be removed in one way or another.

>> Page 28ff: Padding

The guidance I've gotten is split regarding this point.  A known (zero
or otherwise) padding allows for a weak cipher to be broken more
easily.  Random padding on the other hand leaves you further exposed
to bad integrity checks.

>> Page 28: Typographical

That text has been removed.

>> It looks like Copy&Paste error.

https://github.com/http2/http2-spec/issues/397

>> It's bogus that a lot of text separates HEADERS from CONTINUATION.

It's not bogus, but it is perhaps poor document structure.

>> the END_HEADERS bit set ?

http://http2.github.io/faq/#why-the-rules-around-continuation-on-headers-frames

>> Page 34: SETTINGS

I think that the easiest way around this would be to define a new
setting that allows for the use of SETTINGS on streams.  If your peer
sets that (on stream 0, of course), then you can start sending
SETTINGS on streams.  That is, if you ever discover a need.

>> Page 34: SETTINGS_MAX_CONCURRENT_STREAMS

We already have something like that for flow control:
http://http2.github.io/http2-spec/index.html#rfc.section.6.9.3

I think that allowing the endpoint choose is best.  I'll add text for that.

>> opening a new connection ?

Fact is, zero isn't really all that special.  So the logic is the
same.  Clients that want to use new streams, but can't are going to
have to come to some conclusion on their own.  Basically, it all boils
down to how long a client is willing to wait before giving up on that
connection and making a new one.

>> SETTINGS_MAX_CONCURRENT_STREAMS limit ?

That is a good point.  https://github.com/http2/http2-spec/issues/529

I think that both come down to providing advice around concurrency
limits.  We need to work out what the messaging here is.  I don't
think that we want to encourage the use of bypass mechanisms.

>> Page 36: SETTINGS synchronization

That's not true.  You send SETTINGS, you carry on.  You only lose RTTs
if you want to apply tight limits and then open them again.

As for the rest, yes, this isn't 100% necessary, but it does give an
endpoint the ability to say at what point a setting should have been
respected.  Thus, if I set SETTINGS_MAX_CONCURRENT_STREAMS to 0, then
I know that I can kill the connection when a new stream arrives.

And the explicit acknowledgement allows for different implementation
strategies.  We've already seen that people are enqueuing packets well
ahead of what the underlying transport permits.  (That might be
suboptimal in some respects, but I'm sure that these people know what
they are doing.)

>> Page 39: GOAWAY

Except that GOAWAY explicitly says that stream n+2 is safe to retry
elsewhere.  How would your proposal here avoid issues with that?

>> Page 42:  Flow Control Window

Because you need to be able to advertise increased space.  And because
we decided not to open up the synchronization issues that come with a
rate-based or absolute window size advertisement scheme.

>> For obvious DoS reasons, it's a mistake to not apply flowcontrol

Yes, you can't use flow control as a tool to mitigate DoS that uses
HEADERS.  I think that we've reached the conclusion that we don't want
to be using that hammer to mitigate all similar problems.  I know
that's not a particularly satisfactory answer, but there are a bunch
of secondary concerns.  In other words, it was not as obvious as you
might like to think.

>> for high performance implementations.

SPDY effectively did that.  Given the status quo, I'm not sure that
(!type) and (type&mask) are measurably different performance-wise.

>> the overhead isn't counted.

This is another place where the flow control window isn't usable.  It
doesn't bother me a great deal.  It seems like there are bigger holes
in this fence.

>> flow control window." and "An endpoint MUST ... FLOW_CONTROL_ERROR"

Done.

>> Page 49: BLOCKED

Removed already.

>> Page 50: Error messages

I'm sure that people would welcome a contribution in this regard.

>> Page 51: out of order flags

Yes, we know.  It's probably not worth fixing though.

>> Page 51: More details in example

It does.

>> Page 51:  To strict ordering ?

That creates a nice state exhaustion/denial of service opportunity
that we decided not to permit.  Especially because header compression
causes the loss of a connection.

>> Page 51: Gibberish

Improved text gratefully accepted.

>> Page 51: Trailing headers

I guess that we could do it.  https://github.com/http2/http2-spec/issues/530

Not sure if it will fly, but we can see.

>> lot of time for the receiver.

It's easy, HEADERS w/ END_STREAM is worth looking at.

>> Page 55: reverse order

Already done.  Thanks.

>> page 57: query part

I think that we discussed this and decided against it.  Happy to
follow up, but I'd have thought that the gains would be marginal.

>> Page 59: cookie compression needs example

I'll look into it.

>> Page 59: Clarification

Removed.

>> I havn't looked at the PUSH stuff, I still consider it a fundamentally

Noted.

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 17 Jun 2014 11:49:46 +1000

Cherry-picking topics to respond:

> easily.  Random padding on the other hand leaves you further exposed
I am not a security person, and this is purely spit-balling, but how about
a hybrid? First n bytes must be <sentinel value, maybe zeroes>, remainder
is random/ignored.​ That way you get to detect bad packing, but also
hopefully get to mess with known-plaintext stuff. However it's more words
and more code, and I have no idea if it's worth it.

> https://github.com/http2/http2-spec/issues/397
I still don't know where these segments come from. If it's a reference to
something from 1.1, a pointer would be very helpful; otherwise some sort of
motivating text or an exemplar is sorely lacking.

> SETTINGS on streams.  That is, if you ever discover a need.
+1. Write it up as an extension setting and we keep the door open while
avoiding any YAGNI issues.

​
> suboptimal in some respects, but I'm sure that these people know what
I agree with Martin. If I use my earlier per-frame compression as an
example: say I send the equivalent of SETTINGS[TE:gzip]. Then I find my
resources being starved, maybe I'm being zip-bombed, so I send
SETTINGS[TE:gzip;q=0]. If I'm being polite, I continue to accept gzipped
messages until I get the ACK; however if it is actually a DoS attack, the
ACK may never come, so I start a timer and use SETTINGS_TIMEOUT to bail.
 The advantage here is that I can use the same timer if I reduce my
SETTINGS_MAX_CONCURRENT_STREAMS, or send any other setting where in-flight
messages could cause an error. Because there's one timer, the spec is
simpler, and probably the code is simpler.

> (!type) and (type&mask) are measurably different performance-wise.
Wouldn't play nicely with extension frames; you'd need more rules about how
to forward unrecognised end-to-end frame types (not currently possible),
how to detect/mitigate jerks working around flow control ... actually, I
thought extension frames were all flow controlled, but apparently not. We
rely on extension authors to be honest and accurate about what extension
frames are flow-controlled.

> doesn't bother me a great deal.  It seems like there are bigger holes
Indeed: ping -f

> I'm sure that people would welcome a contribution in this regard.
The equivalent of reason phrases attached to status codes, or something
richer? If the former, we kind of already have those (FLOW_CONTROL_ERROR,
STREAM_CLOSED, COMPRESSION_ERROR, etc.) if the latter, who wants to deal
with L10n?

> I guess that we could do it.
> https://github.com/http2/http2-spec/issues/530
>
To be honest, I'm surprised it isn't already a SETTING, and that the
corresponding Trailer:foo header doesn't have some way of being
communicated outside of the headers. I guess I should spelunk the archives
and minutes to read up on the discussions that resulted in almost all
hop-by-hop headers being replaced with settings or dropped, but left the
chunking stuff.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Martin Thomson <[email-masked]>
Date: Mon, 16 Jun 2014 19:52:15 -0700

On Jun 16, 2014 6:49 PM, "Matthew Kerwin" <[email-masked]> wrote:
> I am not a security person, and this is purely spit-balling, but how
about a hybrid? First n bytes must be <sentinel value, maybe zeroes>,
remainder is random/ignored.​ That way you get to detect bad packing, but
also hopefully get to mess with known-plaintext stuff. However it's more
words and more code, and I have no idea if it's worth it.

The other thing to note I'd that if you actually have IND-CCA (look it up,
you need it, it's foundational), then you are dealing with the possibility
that plaintext is all x for any value of x. And I try not to be inventive
when it comes to security mechanisms.

Yes, it is the case that padding has been attacked, but I'm inclined to
trust that TLS can handle this one.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 17 Jun 2014 13:05:44 +0200

I'll give brief (as possible) replies to some points below, as they touch
on some of my points of chronic malcontent.   But in general, many of PHKs
comments reflect the surprise that a protocol educated reader gets from
looking at some of the design decisions in the draft whose motivation is
not entirely clear.

On 16 June 2014 23:30, Mark Nottingham <[email-masked]> wrote:

> We've skipped it an only intend coming back once we have more experience
of traffic with many parallel streams in operation.

> Page 28ff: Padding

I would prefer padding be removed totally.  But failing that, I would
prefer a padding frame that could be sent between other frames using one of
the reserved bits to indicate it is padding and a 7+ integer length field
so that it could be a 1 octet frame:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |1|R|Length (6+)|  Padding (0....                               |
    +-+-+-----------+-----------------------------------------------+

Failing that then on every frame is better than on specific frame types.

But my concern remains that the transport implementation has no idea what
to do with padding. When should it use it? how long it should be? what is
the best padding to use for given data. etc. etc.    The application is the
only thing that knows and given that most applications communicate with the
transport via a simple OutputStream style abstraction, there is no easy way
to communicate frame by frame padding.       I firmly believe that it is
just not the job of the transport to either provide or explicitly transport
padding.   This should be the job of either the encryption layer or the
application layer and neither needs support from the transport to do this.

> > the END_HEADERS bit set ?

It is indeed non obvious.  But I have tried many times to come up with a
way of removing them and I have concluded that they are a necessary
consequence of other design decisions - specifically the single shared
state table of header encoding.

> > For obvious DoS reasons, it's a mistake to not apply flowcontrol

Indeed.  However I do not think that strict DoS is the main concern
here.    Instead I believe there will eventually be collusions between
applications, clients and servers to use the lack of flow control to
attempt to gain benefits vs other traffic on shared connections.   Just as
browsers increased the HTTP/1.1 connection limit, non flow control of
headers represents an incentive to use headers for large traffic.  To think
that this will not be exploited is to under estimate human nature!

However, it is nigh impossible to flow control headers when we have decided
to have a single state table for non size limited header compression.
Lots of "ugly" design decisions that I commented on and now PHK is
commenting on, flow from that decision.

Currently the WG appears resolute to stick with the single state table
decision... but if there is any doubt, then I have some proposals that I
would be keen to expand on (but am currently not spending time on as I
don't think there is currently interest in revisiting this).

> > no excuse for "whoops" details like this.

Ah for this one I think I used the phrase "MASSIVE MISTAKE verging on
hubris" when I first read the spec to describe this particular uglyness:)

Again, I have come to understand that it is a logical consequence of other
design decisions - again rooted in the single header compression state
table. There is no easy fix within the current draft.

Essentially with the current design, HEADERS+CONTINUATION* has to be
treated semantically like a single frame and once you think like that, the
END_STREAM flag is on the last frame!   Logical, but ugly!

> > or SETTING frames being stuffed in there.

See above.

> Page 51: Trailing headers

Or just punted to an extension.   Ie you can send streams with trailers (or
embedded headers), but they are just not HTTP streams.   ie in the same way
that websocket will eventually be carried in extended streams, any HTTP
like protocol that requires trailers can also be an extension.   We can
make them unsupported in the base protocol.

>From other thread:

> Page 8, appending static table
> Lacking documentation, I find it somewhat dubious that
> assigning higher indicies to the static tables very common
> entries makes sense performance wise.
>
> Forcing "GET" to have an index >30 causing it to encode into two
> bytes (or likely even three bytes) rather than one seems very
> counter inuitive.
>
> The dynmic and shifting indicies for the static table also robs
> high-performance load balancers an important optimization opportunity
> (ie: screening out non-GETs and similar).

Definitely agree with this one, also the need to copy static entries to the
dynamic table is also an issue here.   If there was interest in changing
hpack (say to support multiple state tables), then these should definitely
be reviewed - but probably not worth changing hpack just for these.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jun 2014 20:28:08 +0000

In message <[email-masked]>

>https://github.com/http2/http2-spec/issues/531

Those were the bits I was referring to.

>http://http2.github.io/faq/#why-the-rules-around-continuation-on-headers-frames

I'm not asking why we need multiple frames, that's painfully obvious
when the cookie mistake is retained.

I'm asking why the frames have different names ?

Rather than

	HEADERS	
	CONTINUATION * N
	CONTINUATION + END_HEADERS

I would do

	HEADERS * N
	HEADERS + END_HEADERS

Since I don't see how CONTINUATION differs from HEADERS in any way
other way than name and type number.

Eliminating one frame type would simplify the RFC text and 
implementations, (Ref: Antoine de Saint Exupéry's wisdom that
"Perfection is finally attained not when there is no longer anything
to add, but when there is no longer anything to take away.")

>Fact is, zero isn't really all that special.

The operative bit was "and stays there ?"

Shouldn't we at least offer some recommended minimum timeouts ?

>think that we want to encourage the use of bypass mechanisms.

I don't think its just a matter of "messaging" I think it is a
very nasty conflict between QoS and DoS mitigation.

If we state up front that clients SHALL NOT have more than one
TCP connection opened to the same destination IP#, then we lay
the ground work for the first layer of DoS mitigation strategy
without seriously inconveniencing any normal clients.

But unless servers somehow detect client side proxies and offer
them a much more generous SETTINGS_MAX_CONCURRENT_STREAMS, service
to clients behind those proxies will suck, in direct proportion
to the number of clients behind the proxy.

The necessary logic to tell a proxy from a client from a hostile
DoS-zombie is neither going to be pretty nor particularly efficient,
since access to the algorithm/source code by definition means you
can circumvent it with artificial traffic.

>if you want to apply tight limits and then open them again.

99% of those ACKs will be ignored.

I propose that the remaining 1% could get the exact same functionality
by sending SETTINGS + PING and wait for the pong.

That means less wasted bandwidth, less RFC text and less implementation
source code.

>strategies.

Using PING to get the acknowledgement and only when the strategy
actually calls for an ack, while saving RFC text and source code
gives you the same flexibility.

>elsewhere.  How would your proposal here avoid issues with that?

(I'm not sure where you get n+2 from there ?)

Then please add a SETTINGS_LAST_STREAM parameter instead:

	SETTINGS_LAST_STREAM (6):  This setting acts as a quota
	   for the amount of work that will be served.  The peer
	   SHALL NOT attempt to open streams numbered higher than
	   the current setting.

>Because you need to be able to advertise increased space.

Nobody has said anywhere that SETTINGS parameters cannot
be relative or incremental values ?

Which reminds me: Has anybody thought about consolidating 
WINDOW_UPDATE ?

As messages they suffer from a pretty big overhead, and I suspect
we will often see concurrent updates to a stream window and
the connection window at the same time.

If we defined a flag on WINDOW_UPDATE to mean that the
per stream increment should also be applied to the
per connection window, we could save 12 bytes and some
processing for that case.

>of secondary concerns.  In other words, it was not as obvious as you

I think it's very obvious:  Having retained the cookie mistake
and excempted them from flow-control, you've pretty much defined
how DoS attacks on HTTP/2 will look.

>doesn't bother me a great deal.  It seems like there are bigger holes

What fence are you referring to now ?

I'm asking because I don't see anything in the protocol that even
remotely looks like a comprehensive built in DoS resistance.

While I admire the candid listing of weaknesses in 10.5, I really
don't think "read the disclaimer" is a very usable attitude to
DoS mitigation.

>Yes, we know.  It's probably not worth fixing though.

I beg to differ.

Let me just quote the para I'm objecting to:

How do I know I have received the last CONTINUATION frame if that frame
is not marked by the END_STREAM flag ?

I think it is very much worth fixing this so that the last frame
of the stream and ONLY the last frame of the stream has the END_STREAM
flag set.

>>> The example in 8.1 should show where the END_HEADERS and END_STREAM bits

In -12 it mentions neither:

   1.  one HEADERS frame, followed by zero or more CONTINUATION frames
       (containing the message headers; see [HTTP-p1], Section 3.2), and

   2.  zero or more DATA frames (containing the message payload; see
       [HTTP-p1], Section 3.3), and

   3.  optionally, one HEADERS frame, followed by zero or more
       CONTINUATION frames (containing the trailer-part, if present; see
       [HTTP-p1], Section 4.1.2).

>that we decided not to permit.

I really don't understand that answer:  Buffering the compressed
header will take up less space than buffering the uncompressed
header ?

>Improved text gratefully accepted.

I'd happy attempt if I had any idea what it atttempts to say, but I
literally looked a that paragraph for 10 minutes without finding out.

If nobody else knows, you should strike it.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jun 2014 20:42:59 +0000

In message <CACweHNDp=[email-masked]>

>ACK may never come, so I start a timer and use SETTINGS_TIMEOUT to bail.

So how is this different from sending

SETTINGS[TE:gzip;q=0]PING[0xdeadbeef] and waiting for the PING response ?

My point is that 99% of the time the ACK is just wasted effort, so
we shouldn't make it the default.

At the very least give SETTINGS a flag "SEND_ACK" that people can
avoid setting when they don't care about ACKs.

>frames are flow-controlled.

Encoding it in the allocated type would make that honesty automatic,
and life easier for high-speed implementations.

>richer?

Something your grand-aunt can read to you in the phone, but preferably
have a chance of understanding so she doesn't have to.

My fear is that she'll call and ask what she should do about:

	"My computers says says: REFUSED_STREAM: The endpoint refuses
	the stream prior to performing any application processing"

That's a prefectly good explanation for the programmer, but not for
an end user.  Experiences has shown that most programmers don't realize
there is a difference unless we make them aware of it.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jun 2014 21:04:49 +0000

In message <[email-masked]>
, Greg Wilkins writes:

>I would prefer padding be removed totally.

I'm certainly not convinced that padding as proposed is compliant
with Gettys 1st and 3rd laws:

	1.Do not add new functionality unless an implementor cannot
	  complete a real application without it.

	3.The only thing worse than generalizing from one example is
	  generalizing from no examples at all.

I can see that padding can be very valuable in some specific security
contexts, although I find the required infrastructure support to
be a rather formidable challenge.

Ie:  How do we get W3C to support "Create this form, when you submit
replies the password field must go into a dedicated HTTP/2.0 frame
padded to a length of NN bytes" ?

It's not like anything prevents the application layer from padding
the submitted XML with whitespace, if an entry form contains a field
called "password".

>Failing that then on every frame is better than on specific frame types.

Yes, absolutely:  The more orthogonal the protocol, the faster
implementations will be possible.

>state table of header encoding.

But what difference does it make that you send:

	HEADERS
	CONTINUATION

vs.

	HEADERS
	HEADERS
	HEADERS + END_HEADERS

I don't see any value added from having two different frame types ?

>attempt to gain benefits vs other traffic on shared connections.

That's sort of comparing grand bank robbery to petty candy theft...

>to have a single state table for [...]

I really don't see why you can't just count the frames containing
the headers towards the windows, compressed or not compressed ?

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 18 Jun 2014 09:39:21 +1000

On 18 June 2014 06:28, Poul-Henning Kamp <[email-masked]> wrote:

> Martin Thomson writes:
Actually, when you account for things like stream dependency (which
shouldn't be included in continued headers), and continuations on
PUSH_PROMISE, you end up with:

    HEADERS
    [HEADERS - PRIORITY] * N
    HEADERS - PRIORITY + END_HEADERS

    PUSH_PROMISE
    [HEADERS - PRIORITY] * N
    HEADERS - PRIORITY + END_HEADERS

That's pretty complicated. I've also ignored padding, which is currently
not included in CONTINUATION frames. (Note: I've made PP followed by H
because subsequent PP frames would waste four bytes on the promised stream
id.)

> to add, but when there is no longer anything to take away.")
That quote also applies to the DRY principle, and not all of us are
pretending PUSH_PROMISE doesn't exist.

> ​​
Your preference seems to be for reusing and overloading existing frames,
while personally I find the current approach (single-purpose, single-use
frame types) simpler. That's a taste issue, so we're allowed to disagree,
and neither is "better."

If you want to prioritise bandwidth, we could add a new "SEND_ACK" flag to
SETTINGS. It doesn't add much complexity, and it saves traffic on all those
unneeded 8-byte ACKs.

Incidentally, PING/PONG costs an extra 16 bytes each way, while
SETTINGS/ACK costs 0+8, so for the cases where you do care about the
acknowledgement, ping is a relative waste of bandwidth.

> header will take up less space than buffering the uncompressed
Who's buffering headers? The whole point is that we're streaming them
through the HPACK context, *not* buffering them.

Well, you might be buffering them, if your API emits headers to the
application in a single blob, rather than streaming them up. That's nothing
to do with the spec, though.

> If nobody else knows, you should strike it.
I wouldn't do much to change it, just some minor disambiguation:

```
   ​Otherwise, frames MAY be interspersed on the stream between these
header blocks and DATA frames, but those interstitial frames do not carry
HTTP semantics. In particular, interstitial HEADERS frames (and any
CONTINUATION frames that follow) do not carry HTTP semantics.
```

Personally I'd really like someone to add something there to explain what
non-semantic metadata _means_, maybe with an example. This could also clear
up some of my residual issues with segments.

[Second message:]

>​>> >> I would allocate frame Type values so that the top bit indicates
​>​
>> >> "subject to flow-control" to simplify implementation.  Similarly
​>​
>> >> expending a bit to mark hop-to-hop frames would make life easier
​>​
>> >> for high performance implementations.
​>​
​>​
>> SPDY effectively did that.  Given the status quo, I'm not sure that
​>​
>> (!type) and (type&mask) are measurably different performance-wise.
​>​
​>​
>Wouldn't play nicely with extension frames; you'd need more rules about how
​>​
>to forward unrecognised end-to-end frame types (not currently possible),
​>​
>how to detect/mitigate jerks working around flow control ... actually, I
​>​
>thought extension frames were all flow controlled, but apparently not. We
​>​
>rely on extension authors to be honest and accurate about what extension
​>​
>frames are flow-controlled.
​>
​> ​
​> ​

No, I think it's easier now: unrecognised types are never forwarded, and
unrecognised types are not flow-controlled. That's pretty simple. Not as
useful as it could be, but simple.

​>​
>> >> Page 50: Error messages
​>​
​>​
​>​
>> >> The RFC should propose text-messages for presentation purposes
​>​
​>​
>The equivalent of reason phrases attached to status codes, or something
​>​
>richer?
​>​
​> Something your grand-aunt can read to you in the phone, but preferably
​> ​
​>
​> ​
​>
​>​
​>​
​>
​> ​
​> ​
> there is a difference unless we make them aware of it.​

That's true, but I'm pretty sure we're not in the business of designing UI
(and error messages are UI). And I, for one, do not want to be in any way
associated with any L10n of error messages, or the politics of "why are
they only in English?" or "why are they not in Swahili?"

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 18 Jun 2014 19:01:53 +1200

On 18/06/2014 11:39 a.m., Matthew Kerwin wrote:
> On 18 June 2014 06:28, Poul-Henning Kamp wrote:
> 
>> Martin Thomson writes:
>>

Most middleware proxies are buffering them un-compressed for the entire
duration of the end-to-end transaction. Having to buffer in compressed
form for a short period of frames is small by comparison.

Particularly given that a) RST_STREAM can be issued as necessary to free
up impatient clients, b) well behaved clients will treat HEADERS and
continuation as a block anyway and not consume resources for long.

> to do with the spec, though.

API - irrelevant.
application - irrelevant.
middleware - very, very relevant.

This is exactly why the protocol design made by browser people for
communication directly to server is seing so may complaints from the
middleware people.

The network substrate over which the HTTP traffic flows has to retain
the header details in parsed form for logging and transaction handling,
caching, failure recovery, etc, etc.

> header blocks and DATA frames, but those interstitial frames do not carry

"these" and "those" is unnecessary methinks.

> up some of my residual issues with segments.

Amos

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 18 Jun 2014 09:39:35 +0200

On 17 June 2014 23:04, Poul-Henning Kamp <[email-masked]> wrote:

> In message <
> [email-masked]>
> , Greg Wilkins writes:
>
>

The way I have understood it, CONTINUATION makes sense if you are going to
make END_STREAM a function of frame type.   Making END_STREAM a function of
frame type is a consequence of supporting arbitrarily large headers that
are compressed with a shared state table.   A compression state table gives
very good compression and is needed if you are going to fit 80+ requests
into a single slow start TCP window.....   why the compression state table
has to be shared???  Well I think that was a simplification that has
resulted in a very complex emergent design.

> That's sort of comparing grand bank robbery to petty candy theft...

I don't think so.   Sure full on DOS attacks get the all the media
attention just as grand bank robberies do,  but it is the taking of a
little bit more candy from every server connection that will really cost
deployers serious real money.   For example, the amount of real value taken
my users (jetty server) by the decision to break the two connection limit
far exceeds any losses made by the occasional real DOS.

Creating an incentive to send large headers is just going to put more and
more pressure of servers to support large headers.  We get away with 8KB
limits mostly now, but even a modest increase to 16KB represents a massive
investment in memory when you multiple by all the outstanding HTTP
requests.  How many billions of requests are open this instant, multiply
that by 8KB or more and that is a significant extra memory that we are
asking the world to purchase!

Sure servers can reject big headers,but the incentive will now exists to
push that limit and we have seen with the 2 connection limit that such
incentives can create pressure that eventually breaks limits that were
applied for rational reasons.

Currently a single uncompressed frame of headers already far exceed the
resource commitment most servers are prepared to make to a HTTP request, as
you have to hold those headers for the full duration of request handling
(even when waiting for a slow DB, we can put IO buffers back in the pool,
but not headers).     The numbers were published here recently an 16KB
headers would cover around 99.9% of traffic.  However there is apparently
0.1% of traffic that does use larger headers so apparently according to our
charter we must support them.       The moment you allow headers to be in
more than 1 frame, then the single shared state table becomes the limiting
factor for applying flow control etc.

>the headers towards the windows, compressed or not compressed ?

Because of the single shared hpack state table.  You don't know how large
your header frame(s) are until you start compressing them.  Once you have
started compressing them, you have started mutating the shared hpack state,
so no other headers can be compressed.  So you have to be able to finish
compressing them and transmit them so other streams can access the shared
hpack state.   If you flow control headers, then you might not be able to
send them, so you will have effectively locked the entire connection and
not just one stream.

It all comes back to hpack.    If we want simple framing that is not type
dependent and universal flow control, then we need to go to come up with a
way to avoid every stream mutating a shared state table.

> I don't see any value added from having two different frame types ?

I basically agree with you, and made pretty much the same comments when I
tuned in here a month ago.  But let try to explain some of the reasoning
for this design decision as I have understood it.

CONTINUATION is used for both HEADERS and PUSH_PROMISES.  Both HEADERS and
PUSH_PROMISE have extra fields that do not make sense in repeated instances
of themselves. EG PUSH_PROMISE has a promised stream ID and HEADERS can
have priority fields.   If we went for HEADERS+ and PUSH_PROMISE+, then
we'd have to make using these fields in continuations illegal, which is
just as fiddly as having a dedicated continuation frame.

Besides, the thing I don't like about CONTINUATION is not that it exists,
but that it does not carry the END_STREAM bit, and I think that all frames
should.   I believe a stream should exist from the first time it's ID is
mentioned until it sees a frame with the end stream bit set - and that
logic should not have to consider frame types.   The reason it does not
carry the END_STREAM bit is that a PUSH_PROMISE and it's continuation
cannot legally end a stream, so they do not have the END_STREAM bit.  The
solution to that is of course to have END_STREAM bit on all frames and if
somebody sends an illegal sequence of frame types, then that is a protocol
error (and removing the END_STREAM bit from some frames does not avoid
illegal sequences of frame types).

But because we allow HEADERS in the middle of a data stream (and the frame
be called META_DATA as it is crazy to have trailers carried in a header),
and this mid stream meta data does not half close the stream, we need to
know if a headers is mid stream or is a trailer that is half closing the
stream.  Thus we would end up with a I_AM_GOING_TO_SEND_END_STREAM_SOON
bit.    The reason for this is because we cannot interleave other frames
between HEADERS and CONTINUATIONS because of the single shared state table
of hpack.  Currently the state machine is vaguely like the simple one in
the draft because of the simplification that we can transition to half
closed once a HEADERS with END_STREAM bit set is received/sent.     Without
that simplifying assumption the real state machine is closer to the one I
proposed here  https://github.com/http2/http2-spec/issues/484
(as it is, I think the 5 paragraphs of text explaining the Closed state
indicate the draft is an over simplication and as if the TCP state machine
left TIME_WAIT and CLOSE_WAIT as exercises for the reader )

I tried several times to come up with a proposal to remove CONTINUATIONs
and/or to have EOS on all frames.  I could not find a reasonable
simplification that worked without needing to throw out the concept of a
shared compression state table.

cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 09:21:25 +0000

>follow up, but I'd have thought that the gains would be marginal.

Was that discussion before or after HPACK ?

I would expect HPACKs efficiency to increase if the constant-ish
:path part were divorced from the dynamic-ish :query part ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 10:29:37 +0000

>> through the HPACK context, *not* buffering them.

I'm starting to really hate the entire HEADER+CONTINUATION kludge
upon kludge upon kludge hackery.

My preference would be to impose sanity by simply removing CONTINUATION
and telling cookie monsters that if their HPACK compressed HTTP
headers do not fit in 16k, they should consider a diet.

If the WG insist on supporting ridiculously large HTTP headers, we
should eat one of the reserved bits in the frame header to indicate
the precense of a length-extension-field is present in front of the
payload, so that all headers can be fit into one single frame that way.

The difference between receiving cookie-monster frames and
uninterposable chains of HEADERS+CONTINUATION is that the former
is far simpler to implement, faster to process and takes up less
bandwidth.

The storage requirements will be the exact same.

A SETTINGS_MAX_FRAME_SIZE defaulting to 16k-1 would apply negative
social pressure on cookie-monsters, and avoid their WG-sanctioned
escape-hatch becoming a gateway for DoS.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 18 Jun 2014 14:58:38 +0200

> :path part were divorced from the dynamic-ish :query part ?

I think this split is a good one.

.... and while we are splitting headers, could we also split the
Date header into :date and :time.   Currently reencoding the date header is
the largest part of all the header blocks I'm sending (see example below):

Worse still, because requests come in batches that can be sent in <1s,
there is still value in putting the Date header into the HeaderTable and
reference sent, so it is at least only sent once per batch.   But this just
introduces more header table churn as you evict an entry every second.

2014-06-18 14:55:04.252:DBUG:oejhh.HpackEncoder:qtp124232201-30: encoded
':status: 200' by InRefSet to ''
2014-06-18 14:55:04.252:DBUG:oejhh.HpackEncoder:qtp124232201-30: encoded
'Server: Jetty(9.3.z-SNAPSHOT)' by InRefSet to ''
2014-06-18 14:55:04.252:DBUG:oejhh.HpackContext:qtp124232201-30:
HdrTbl[1592a441] added {D,4,Date: Wed, 18 Jun 2014 12:55:04 GMT,5e9ec769}
2014-06-18 14:55:04.252:DBUG:oejhh.HpackContext:qtp124232201-30:
RefSet[1592a441] added
2014-06-18 14:55:04.252:DBUG:oejhh.HpackEncoder:qtp124232201-30: encoded
'Date: Wed, 18 Jun 2014 12:55:04 GMT' by LitIdxNameHuffValIdxd to
'6595FcAe9cA6190dF3E371880603094d0c330406D5Ae8f'
2014-06-18 14:55:04.252:DBUG:oejhh.HpackEncoder:qtp124232201-30: encoded
'Custom: Value' by InRefSet to ''
2014-06-18 14:55:04.252:DBUG:oejhh.HpackEncoder:qtp124232201-30: encoded
'Content-Type: text/plain' by LitIdxNameHuffVal to '0f158772Fa38F7D8965d'
2014-06-18 14:55:04.252:DBUG:oejhh.HpackEncoder:qtp124232201-30: encoded
'Content-Length: 116' by LitIdxNameHuffVal to '0f1282118b'

So a header that could have been
  0f158772Fa38F7D8965d0f1282118b
plus a small time field is now
  6595FcAe9cA6190dF3E371880603094d0c330406D5Ae8f0f158772Fa38F7D8965d
0f1282118b

cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 13:11:36 +0000

In message <CAH_y2NHPqSizPrST8Ag+hr3d6FpVtL=[email-masked]>

>the largest part of all the header blocks I'm sending (see example below):

Given that the Date header is hop-by-hop and not used for anything
relevant, why don't we just point at NTP and drop it entirely ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 19:03:14 +0000

>relevant, why don't we just point at NTP and drop it entirely ?

OK, so I admit it, I'm a data-nerd...

I ran a full year of Date: headers through HPACK's Huffman table.

December compresses 2.6% better than June and July.

December is 1.5% better than August.

Saturdays are 1.6% better than Wednesdays

Saturday 20th december is 5.4% better than Wednesday 18th of june.

And yes, I still think we should just drop the Date header in HTTP/2.0

---

From: Jason Greene <[email-masked]>
Date: Wed, 18 Jun 2014 14:18:24 -0500

> that it will be removed in one way or another.

I share this opinion. The priority frames introduce way too much complexity for what is essentially a hint. Assuming it’s followed, it’s possible that the processing overhead of client directed scheduling might end up causing the very problem it’s intending to address. Finally, since HTTP/2 is already over TCP, send order already offers clients a mechanism for assigning priority (although admittedly a bit basic).

It seems this would actually be a better fit as an extension.

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Jason Greene <[email-masked]>
Date: Wed, 18 Jun 2014 14:19:35 -0500

> relevant, why don't we just point at NTP and drop it entirely ?

Pretty please!

---

From: Martin Thomson <[email-masked]>
Date: Wed, 18 Jun 2014 12:42:46 -0700

>> relevant, why don't we just point at NTP and drop it entirely ?

That change is specifically not in scope for HTTP/2, but now that we
basically done there, we can certainly discuss this.  It's certainly
in scope for the working group though.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 20:06:07 +0000

In message <[email-masked]>
>in scope for the working group though.

Why wouldn't it be in scope ?

Date is a hop-by-hop header, if it serves no use on a HTTP/2.0 to
HTTP/2.0 hop, we can eliminate it, just like we eliminated the
double CRNL after the headers, because it serves no use.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 18 Jun 2014 13:12:22 -0700

> double CRNL after the headers, because it serves no use.

It's not hop-by-hop by my reading:
https://tools.ietf.org/html/rfc7231#section-7.1.1.2

I'm not disagreeing with your premise, just the choice of the wagon
you are hitching this too.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 20:17:47 +0000

In message <[email-masked]>
>https://tools.ietf.org/html/rfc7231#section-7.1.1.2

Hmm, you may be right.

In that case we should transfer the time as a POSIX time_t in the
HEADERS frame.  Wasting time huffman encoding dates and still
using 24 bytes where 8 would be plenty is just plain stupid.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 18 Jun 2014 13:25:27 -0700

> using 24 bytes where 8 would be plenty is just plain stupid.

Yeah, that was discussed and rejected, though not permanently.  I
think that James worked out that 5 bytes was enough in the short term
with a little epoch tweaking.

And yes, we are plain stupid.  Think of the cost of parsing that stuff
as opposed to doing ntohl().

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 20:33:24 +0000

In message <[email-masked]>
>as opposed to doing ntohl().

Indeed.  Ascii Timestamps amount for about 30% if Varnish CPU load :-/

---

From: James M Snell <[email-masked]>
Date: Wed, 18 Jun 2014 14:27:26 -0700

Yep. Using a variable length encoding 5 bytes is the max we would need for
all practical purposes.

I'm all for dropping Date entirely, tho. That doesn't help us with the
other date headers, but it helps. Last-Modified and If-Modified-Since are
both great candidates for five-byte encoding.

It's too bad the WG didn't pick up on such an obvious improvement but, oh
well I guess.

- James
On Jun 18, 2014 1:36 PM, "Poul-Henning Kamp" <[email-masked]> wrote:

> In message <
> [email-masked]>
> , Martin Thomson writes:

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 19 Jun 2014 08:03:17 +1000

Date is used to calculate Age. I imagine it's actually in use by caches (?)

Coding it into the headers frame makes sense, but is there a rule
about it being set in requests?

-- 
  Matthew Kerwin, B.Sc (CompSci) (Hons)
  http://matthew.kerwin.net.au/

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 19 Jun 2014 10:32:00 +1000

As Martin said, this isn't in-scope for HTTP/2. If people want to propose separate, backwards-compatible extensions to HTTP in general, they're free to do so, but I'd ask that they use a clear Subject: line.

Regarding Date -- it's end-to-end, not hop-by-hop, and removing it is not backwards-compatible; it's used extensively by the caching model <http://httpwg.github.io/specs/rfc7234.html>, and also for other purposes (e.g., the recent discussion in TLSWG about relying upon it rather than the timestamp in TLS).

Regards,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 19 Jun 2014 14:49:44 +1000

> headers do not fit in 16k, they should consider a diet.

One thing that came up in a side conversation in NYC was the possibility of only HPACKing the HEADERS frame; subsequent CONTINUATION frames would be uncompressed (so they don't affect state, and could be flow controlled). 

I kinda like it, YMMV.

At any rate, a server (origin or proxy) is perfectly within its rights to 431 Request Header Fields Too Large any request that has a CONTINUATION, and then reset the stream. They'll learn pretty soon...

Cheers,

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 19 Jun 2014 18:31:34 +1200

> It's too bad the WG didn't pick up on such an obvious improvement but, oh

How about we proxy people collude and define a proxy SETTINGS extension
to turn on lots of these types of optimizations?
 That way we can turn around and show how slow the client->server direct
traffic format is. At least until they see the light and join in.

Amos

> On Jun 18, 2014 1:36 PM, "Poul-Henning Kamp" wrote:
> 
>> In message <

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 19 Jun 2014 06:52:54 +0000

>to turn on lots of these types of optimizations?

This is not really a proxy issue, this is a performance issue that
affects every HTTP/2.0 node in the net.

Do only proxy people care about performance, CPU-load etc ?

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 19 Jun 2014 19:41:58 +1200

> Do only proxy people care about performance, CPU-load etc ?

Trying to convince people in advance has been going in circles so far. I
am thinking we need to demonstrate the benefits and now that extensions
are permitted do not actually need to convince anyone before starting to
get some incremental gains between "the believers".

Amos

---

From: Kinkie <[email-masked]>
Date: Thu, 19 Jun 2014 12:51:50 +0200

> Do only proxy people care about performance, CPU-load etc ?

We probably care more than most: on clients CPU is abundant, and on
origin servers there's lots of processing, so fine-tuning the
transmission side of things is probably not a primary concern.

-- 
    Francesco

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Thu, 19 Jun 2014 17:28:20 +0000

> I would expect HPACKs efficiency to increase if the constant-ish :path part were divorced from the dynamic-ish :query part ?

I also think that queries (particularly on HTTPS connections are more likely to contain sensitive data (e.g. session IDs, usernames, etc.) so there might be a security argument for having them separate. Not a strong one yet IMHO but I can see one develop over time,

Richard

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Thu, 19 Jun 2014 18:20:31 +0000

> >That is a good point.  https://github.com/http2/http2-spec/issues/529

I've been working on the assumption that SETTINGS_MAX_CONCURRENT_STREAMS applies only to a single connection up to now.
I'm not sure how else it makes sense.

> If we state up front that clients SHALL NOT have more than one TCP connection opened to the same
> destination IP#, then we lay the ground work for the first layer of DoS mitigation strategy without seriously inconveniencing any normal clients.

> But unless servers somehow detect client side proxies and offer them a much more generous SETTINGS_MAX_CONCURRENT_STREAMS,
> service to clients behind those proxies will suck, in direct proportion to the number of clients behind the proxy.

Even this wouldn't be sufficient. You need multiple TCP connections. Without it, you'd have to assume connection coalescing on the
proxy- which I don't think is a given. Moreover, I don't believe it's desirable in many places. In particular, if the upstream maybe doing
some form of identification (of user, of organization, of geography, ...) based on the incoming source IP or if the proxy itself is incapable
of efficiently restricting itself to a single TCP connection. Let's take CWS for example. What is a CWS proxy? It's actually a set of multiple
servers load-balanced behind a single IP - there's no way in which we're sharing a single TCP connection for a dozen servers and
10,000+ users.

On the other hand, if you don't enforce constraints for proxies then there's a simple workaround for a client that wants to misbehave
pretend to be a proxy.

> The necessary logic to tell a proxy from a client from a hostile DoS-zombie is neither going to be pretty nor particularly efficient

Yup. No argument on this bit.

Richard

---

From: Martin Thomson <[email-masked]>
Date: Thu, 19 Jun 2014 12:18:19 -0700

> I'm not sure how else it makes sense.

That's definitely the intent here.  A distributed system like your own
can really only apply any constraints on each individual node
separately.

The question is: what recourse does a client have if they find
themselves exceeding this limit frequently or for long periods of
time.  And proxies are likely to find themselves in this situation
more easily than others, simply because they are making more requests.
It might just come down to detecting and whitelisting proxies so that
they get more connections or more concurrent streams.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 19 Jun 2014 19:28:43 +0000

In message <CABkgnnUonH0=ws=ofbdQK_R6Kj6CAWS1Rz2MrsiVDK=[email-masked]>

>they get more connections or more concurrent streams.

You're kidding about whitelisting, right ?

---

From: Jason Greene <[email-masked]>
Date: Thu, 19 Jun 2014 20:23:04 -0500

> :path part were divorced from the dynamic-ish :query part ?

I think you are on to something with separating headers which represent sensitive user data from the more transport oriented data. If it’s separate it can potentially be encrypted without the need (or desire) for the proxy to decrypt it (e.g. the bank example in the strawman thread).

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 20 Jun 2014 21:59:22 +0200

On Thu, 19 Jun 2014 21:28:43 +0200, Poul-Henning Kamp <[email-masked]>  
wrote:

> In message  
> <CABkgnnUonH0=ws=ofbdQK_R6Kj6CAWS1Rz2MrsiVDK=[email-masked]>
> , Martin Thomson writes:

You need to whitelist (or buy IP numbers) already today, so it's not an  
HTTP/2 problem.

/Martin Nilsson

-- 
Using Opera's mail client: http://www.opera.com/mail/

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 21 Jun 2014 15:50:22 +1200

On 21/06/2014 7:59 a.m., Martin Nilsson wrote:
> On Thu, 19 Jun 2014 21:28:43 +0200, Poul-Henning Kamp wrote:
> 
>> In message Martin Thomson writes:
>>> On 19 June 2014 11:20, Richard Wheeldon (rwheeldo) wrote:
>>

Thats going to be news to a vast number of small businesses and end
users runnign proxies (from all vendors).

What they are doing right now is simply disabling the Via and
X-Forwarded-For headers. Most also MITM on the traffic to capture broken
clients and HTTPS. "works perfectly" etc.

Amos

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 21 Jun 2014 05:43:09 +0000

>You need to whitelist (or buy IP numbers) already today, so it's not an  

Could you point me to the place in the HTTP/1.1 RFC that says that
proxies must be whitelisted ?

I must have missed it on my first N*100 readings...

---

From: <[email-masked]>
Date: Sat, 21 Jun 2014 15:52:19 +0000

> upon kludge upon kludge hackery.

Me too.

> ...

This is a great idea. Get rid of CONTINUATION please.
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: <[email-masked]>
Date: Sat, 21 Jun 2014 15:58:24 +0000

On 19 June 2014 06:49, [email-masked] wrote:
> One thing that came up in a side conversation in NYC was the possibility of only
> HPACKing the HEADERS frame; subsequent CONTINUATION frames would
> be uncompressed (so they don't affect state, and could be flow controlled).

How could this work? Aren't headers allowed to cross frame boundaries? You can't switch from compressed to uncompressed in the middle of a header, no?

> At any rate, a server (origin or proxy) is perfectly within its rights to
> 431 Request Header Fields Too Large any request that has a
> CONTINUATION, and then reset the stream. They'll learn pretty soon...

Good point, but I still like PHK's idea to use a length extension bit in the HEADER frames better.  Makes everything else a lot less messy.  You already require HEADER+CONTINUATION frames to be sent in one atomic chunk so having a single huge HEADER frame isn't any more of a HOL blocking issue than a sequence of HEADER+CONTINUATION frames of the same size.
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Sat, 21 Jun 2014 19:06:56 +0000

> Could you point me to the place in the HTTP/1.1 RFC that says that proxies must be whitelisted ?

Can you point me to the place in the HTTP/1.1 RFC that says that servers must do DoS prevention? It doesn't - although
the new versions (723*) do acknowledge that it might happen. In practice it's necessary. If you're running a small scale
proxy with a  couple of hundred users behind it then you'll probably never run into the problems it causes. If you're
running something with a few thousand users or more then you probably will. We've run into issues over the past few
years with certain services either blocking us or redirecting to Captcha pages to establish if there's a real user in place.

The workarounds are all pretty ugly:
- Assume that anything with an XFF header (or Forwarded) is a proxy
- Whitelist a bunch of IP addresses on the server-side
- Stop using the proxy for website <X>  (essentially a different form of whitelist)
- Switch everything over to TLS and assume that DoS over TLS is do hard for anyone to do it properly

I'd be happy to collaborate on activity that improved the status-quo in this regard,

Richard

---

From: "Martin Nilsson" <[email-masked]>
Date: Sat, 21 Jun 2014 23:01:57 +0200

> I must have missed it on my first N*100 readings...

No, I can't. And yet it is our experience from servicing a few hundred  
million users through our servers that this is the case.

/Martin Nilsson

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 21 Jun 2014 21:22:27 +0000

>million users through our servers that this is the case.

Do you have any numbers for how many sites you have had to contact
in order to get white-listed ?

---

From: "Martin Nilsson" <[email-masked]>
Date: Sat, 21 Jun 2014 23:33:39 +0200

> in order to get white-listed ?

There are 50 or so organizations that we share our active IP-ranges with.  
In addition some do detection using other mechanisms than IP number. There  
have also been quite a few instances of temporary blocks that gets  
resolved without any involvement from us as the users complains to the web  
site in question.

/Martin Nilsson

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 22 Jun 2014 18:27:29 +1000

> How could this work? Aren't headers allowed to cross frame boundaries? You can't switch from compressed to uncompressed in the middle of a header, no?

A sender would have to compose their header frames appropriately; e.g., putting all of the small / repeating headers in the compressed frame, with remaining frames in the subsequent uncompressed frames. In a pathological case, it means that most or all of the headers would be uncompressed.

Personally, I’m OK with that; it supports the weird cases, but doesn’t optimise for them. YMMV.

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 22 Jun 2014 19:39:42 +1000

On 22 June 2014 18:27, Mark Nottingham <[email-masked]> wrote:

> > How could this work? Aren't headers allowed to cross frame boundaries?
> You can't switch from compressed to uncompressed in the middle of a header,
> no?
>
> A sender would have to compose their header frames appropriately; e.g.,
> putting all of the small / repeating headers in the compressed frame, with
> remaining frames in the subsequent uncompressed frames. In a pathological
> case, it means that most or all of the headers would be uncompressed.
>
I like this idea; you can fit a fair whack of "normal" headers in 16K of
Huffman-coded (let alone delta compressed) payload, and if you insist on
sending more than that, then you have to prioritise appropriately. It also
frees us up from blocking on possibly arbitrarily many CONTINUATION frames,
and allows us to gauge the size of incoming header data sooner.

Thinking in terms of my
implementation
, I
​'d likely still want to sit on the entire header block before emitting it
to the API, but I could imagine that someone more streamey (someone with a
more advanced API, or a high-throughput intermediary) would probably
benefit a lot from being able to treat *all* frames as individual
<s>datagrams</s> packets.

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 22 Jun 2014 21:49:36 +1000

(Apologies for replying to myself)

> <s>datagrams</s> packets.
I realise I should probably clarify my thoughts on what to do if a single
header doesn't fit in a 16K frame.  The option I like best comes from one
of PHK's earlier posts, where one of the reserved bits in the frame header
is used as a "jumbo frame" marker such that if it's set the first, say,
four octets of payload space is actually an extra 32 bits of payload length
(64 petabytes is pretty stupid, but whatever). To limit the
stupid-potential there'd be an associate SETTING, I think PHK called it
"SETTINGS_MAX_FRAME_SIZE", advertising how much you're willing to receive
in a single frame. I guess it would encode the number of bits by which
you're willing to extend the length, or something like that (default=zero,
of course).

I realise this is a bit of a change from what we currently have, but as
with many of my suggestions, the minimal effort to support it is quite
simple: ignore the setting, and maybe do a little bit more
reading/arithmetic to decide whether to throw a FRAME_SIZE_ERROR. Some of
that could even be gotten around if we make setting the "jumbo frame" bit
illegal without having ACKed a SETINGS_MAX_FRAME_SIZE.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 22 Jun 2014 12:36:11 +0000

In message <CACweHNBNUpPWLw+NOE0s17LWL_1OKjWDitTGMxtaUUw6+=[email-masked]>

>you're willing to extend the length, or something like that (default=zero,

I would have it be the max length of *any* frame we're willing to accept,
and the default would then obviously be the 16kbyte currently implicit
in the standard.

And I *far* prefer to have a frame length-extension bit, to having
CONTINUATION and all that kludgery.

---

From: <[email-masked]>
Date: Mon, 23 Jun 2014 09:49:57 +0000

On Sunday,22 June 2014 14:36, [email-masked] wrote:
>>I realise I should probably clarify my thoughts on what to do if a
>>single header doesn't fit in a 16K frame.  The option I like best comes
>>from one of PHK's earlier posts, where one of the reserved bits in the
>>frame header is used as a "jumbo frame" marker such that if it's set
>>the first, say, four octets of payload space is actually an extra 32
>>bits of payload length
>
> and the default would then obviously be the 16kbyte currently implicit in the standard.
>

So are you proposing the "jumbo frame" marker for all frames, not just the HEADERS frames?  I think it's a great idea, but I know it makes a bunch of people nervous about HOL blocking if you allow more than 16K in a DATA frame.

But, to quote Roy from an earlier thread ...

On Tuesday,22 April 2014 09:00, [email-masked] wrote:
> Likewise, restricting packet sizes to a small length in order to prevent fools from HOL blocking
> their own multiplexed channels makes some sense, for browser developers.
> However, it actively harms applications of HTTP that are not interested in multiplexing
> because they only want to transmit a single large data stream.
> ... I don't think it makes sense to limit an application-level protocol to the worst case.
> ....Roy

I completely agree.

---

From: <[email-masked]>
Date: Mon, 23 Jun 2014 09:55:29 +0000

On Sunday,22 June 2014 14:36, [email-masked] wrote:
> And I *far* prefer to have a frame length-extension bit, to having CONTINUATION and all that kludgery.

I think the burden of proof is on the backers of the CONTINUATION "kludgery" to defend how that complexity could possibly be better than a single HEADERS frame with a "jumbo frame" marker - since a single HEADERS frame reduces complexity and the result is the same either way (i.e. in both cases the headers are treated as a single atomic block with the same consequence of potential for HOL blocking).

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 23 Jun 2014 10:05:17 +0000

>So are you proposing the "jumbo frame" marker for all frames, not just the 
>HEADERS frames?  I think it's a great idea, but I know it makes a bunch of 
>people nervous about HOL blocking if you allow more than 16K in a DATA frame.

Yes, the length-extension would be available on all frames, which is why
we need a SETTING to limit what we'll accept in that respect.

For huge file transfers the 16k frames are horribly suboptimal and
having the receiver bang the frame size up once "Content-Length: A_LOT"
has been received will do wonders for performance on both ends.

Obviously, you can also reduce the frame size you'll accept.  16K
is quite large for a number of high traffic sites prone to DoS.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 24 Jun 2014 16:09:46 +1000

Hi PHK,

> is quite large for a number of high traffic sites prone to DoS.

This has been discussed a lot over the life of the WG. The place where we left it was that the overhead of framing was quite small, considering that it's 8 bytes over 16K; TCP overheads are usually going to be bigger.

It's true that you can't use sendfile() here, but that's true with multiplexing regardless. It was felt that over time, kernel facilities specific to the use case of HTTP/2 will emerge if necessary, just as they did for HTTP/1.

Is there something else behind "horribly suboptimal" here? Can you give some numbers?

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 24 Jun 2014 08:28:42 +0200

Hi Mark,

> Is there something else behind "horribly suboptimal" here? Can you give some numbers?

I know some high traffic sites running with haproxy, above 100 Gbps. Such
sites don't ever make use of concurrent streams, because as PHK calls them,
they're delivering pink pixels over the net. At these rates, the problem is
not TCP overhead or any such thing, but the processing cost. A single haproxy
node can forward data at 40 Gbps with 256kB buffers, 35 Gbps with 64kB buffers
and something around 20 Gbps with 16kB buffers. At 16 kB buffers and 20 Gbps,
that's 150000 recv/send per second. That's extremely inefficient CPU cache-
wise and requires a lot of context switching. I'd say in fact that the task
processing overhead becomes huge compared to the small cost of copying or
even just splicing data between two ends. Memory bandwidth is huge with
todays processors, and were seeing the 100 Gbps NICs coming, so large
data blocks are processed with a low cost. NICs are capable of doing large
receive offloading and TCP segmentation offloading, so it's possible for
the TCP stack to process 64kB packets which are much cheaper for the stack
than 16kB packets.

So that's really a problem of data processing overhead on top of cheap
forwarding.

I would find it sad that the sites responsible for something like 75% of the
internet's traffic refrain from upgrading because of the extra infrastructure
costs :-/

Regards,
Willy

---

From: David Krauss <[email-masked]>
Date: Tue, 24 Jun 2014 15:08:14 +0800

On 2014–06–19, at 3:18 AM, Jason Greene <[email-masked]> wrote:

> I share this opinion. The priority frames introduce way too much complexity for what is essentially a hint. Assuming it’s followed, it’s possible that the processing overhead of client directed scheduling might end up causing the very problem it’s intending to address.

Anyone attempting to implement it should know enough to use an algorithm of known O(1) computational complexity, although O(log N) choices are also viable.

> Finally, since HTTP/2 is already over TCP, send order already offers clients a mechanism for assigning priority (although admittedly a bit basic).

This only works as long as the server processes requests serially, which isn’t even possible with a reverse proxy. And it moves the scheduling complexity to the client side, which now must defer low-priority requests. Terrible architecture.

Also, stream initiation order is not preserved by intermediaries, so TCP’s guarantee is not strong enough.

---

From: David Krauss <[email-masked]>
Date: Tue, 24 Jun 2014 15:14:01 +0800

> Also, stream initiation order is not preserved by intermediaries, so TCP’s guarantee is not strong enough.

Oops, that’s too pathological to be possible. Still, PRIORITY has a purpose, although it might need a trial period as an extension.

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 24 Jun 2014 20:01:05 +1200

> Oops, that’s too pathological to be possible. Still, PRIORITY has a purpose, although it might need a trial period as an extension.

More accurately middleware can potentially be load balancing a given set
of streams over N backend servers. Any one of which at low-priority may
respond faster than a "high" priority server. Leading to sub-optimal
response frame ordering on the client connection regardless of request
sent order.

Amos

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 24 Jun 2014 12:07:59 +0200

Willy,

Do you have a feel for how much processing costs would affect such proxies
if they still ran with 256kB buffers, but which contained many 16kB frames?

Is the problem here not so much the frame size, but the proxies requirement
to decode/inspect frames?

ie what would be the impost of the proxy read a 256KB buffer, quickly
scanned it for any new streams, and if none found just forwarded to entire
256KB buffer?

cheers

On 24 June 2014 08:28, Willy Tarreau <[email-masked]> wrote:

> [email-masked]>, K
> > > [email-masked] writes:
> > >>> and the default would then obviously be the 16kbyte currently
> implicit in
> > >> the standard.
> > >>
> > This has been discussed a lot over the life of the WG. The place where
> we left it was that the overhead of framing was quite small, considering
> that it's 8 bytes over 16K; TCP overheads are usually going to be bigger.
> >
> > It's true that you can't use sendfile() here, but that's true with
> multiplexing regardless. It was felt that over time, kernel facilities
> specific to the use case of HTTP/2 will emerge if necessary, just as they
> did for HTTP/1.
> >

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 24 Jun 2014 12:20:30 +0200

Hi Greg,

> scanned it for any new streams, and if none found just forwarded to entire

That cannot realistically work, because the front LB is a multiplexer in
practice. It receives hundreds of thousands of client connections that it
dispatches to tens of servers, and a single stream from a client is usually
dispatched to multiple servers depending on routing rules (path prefix,
presence of cookies or not, etc).

Instead, (I remember that it was already discussed here in the past), I
really think we'd need to support large frames for large data sets, that
are typically usable for such sites which have few streams per client but
very large ones.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 24 Jun 2014 13:24:01 +0000

>are typically usable for such sites which have few streams per client but

I agree.

Moving objects which are trivially megabytes and gigabytes in size
using 16kB frames just doesn't make any sense, in particular not
given that MTUs above 9K are actively being discussed again.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 25 Jun 2014 14:10:46 +1000

<https://github.com/http2/http2-spec/issues/540>

> given that MTUs above 9K are actively being discussed again.

The simplest way to address this would be to un-reserve the first two bits of the length <http://http2.github.io/http2-spec/#rfc.section.4.1>; that would get us back up to 64k, letting Willy get to 35 Gbps. Given that 256k only got him to 40 Gbps, that seems like a reasonable stopping point for HTTP/2.

Personally, I think that's a reasonable change -- the argument for 14 bits was to make sure people didn't abuse multiplexing and that they properly implemented continuation, etc.; I think those goals could be met by some prose in the spec and proper testing.

Doing more than 16 bits would take a lot more back-and-forth in the WG, and is likely to encounter a lot of resistance from implementers, from what I've seen.

WRT the "jumbo" frame (i.e., flagging that some prefix of the payload is actually an extension length) -- this sort of hack is necessary to back-port bigger frames onto an existing protocol. Let's not do it out of the gate.

Those are just my personal impressions. I'd very much like to hear from other folks in the WG what they think.

Regards,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 06:55:31 +0200

Hi Mark,

On Wed, Jun 25, 2014 at 02:10:46PM +1000, Mark Nottingham wrote:
> The simplest way to address this would be to un-reserve the first two bits of
> the length <http://http2.github.io/http2-spec/#rfc.section.4.1>; that would
> get us back up to 64k, letting Willy get to 35 Gbps. Given that 256k only got
> him to 40 Gbps, that seems like a reasonable stopping point for HTTP/2.

Please note something I probably didn't mention, but I was limited to
40 Gbps by the NICs, and above 256kB I started to get idle CPU again.
Other people I know doing tests with a few more NICs have already
reached slightly more than 60 Gbps. In fact building a lab at these
rates is not something you do with commodity hardware every day, so
people tend to use what they already have in their racks and try to
get the most out of it.

> Personally, I think that's a reasonable change -- the argument for 14 bits
> was to make sure people didn't abuse multiplexing and that they properly
> implemented continuation, etc.; I think those goals could be met by some
> prose in the spec and proper testing.
> 
> Doing more than 16 bits would take a lot more back-and-forth in the WG, and
> is likely to encounter a lot of resistance from implementers, from what I've
> seen.

I think we could proceed differently and that would avoid people abuse the
frame size. Instead we'd use a single bit to indicate if the size is in
bytes or in 4k pages. Yes that allows to reach 64MB per frame. But that also
means that when you want to use this, you don't have the byte granularity
anymore so the temptation to use it will only be for those sending huge
files and for whom having to send the last chunk separately is a detail
compared to the gains of sending large chunks.

> WRT the "jumbo" frame (i.e., flagging that some prefix of the payload is
> actually an extension length) -- this sort of hack is necessary to back-port
> bigger frames onto an existing protocol. Let's not do it out of the gate.

The proposal above could be much simpler in fact.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 06:39:00 +0000

In message <[email-masked]>, Mark Nottingham wri
tes:

>The simplest way to address this would be to un-reserve the first two 
>bits of the length <http://http2.github.io/http2-spec/#rfc.section.4.1>; 
>that would get us back up to 64k, [...]

Really ?

A 16bit protocol in a 64bit CPU age ?

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 25 Jun 2014 10:01:41 +0200

On 25 June 2014 06:10, Mark Nottingham <[email-masked]> wrote:

> actually an extension length) -- this sort of hack is necessary to
> back-port bigger frames onto an existing protocol. Let's not do it out of
> the gate.

But we have to backport bigger "frames" into the protocol: namely the
unlimited sized headers that we are currently stuffing into CONTINUATION
frames.

An increase to 64KB frames would not allow us to drop CONTINUATION frames,
as apparently we still need to support huge headers.

Note that with continuations, that cannot be interleaved with other frames,
we effectively have a jumbo frames anyway.  The END_HEADERS bit is the
extension bit and the extension length is at the end of the first frame (in
the header of the next "frame").

Semantically there is little difference between HEADERS+CONTINUATION* and
jumbo frames.  It is just that jumbo frames are simpler to describe,
explain, motivate, implement, understand, debug etc.

Another plus of jumbo frames is we know the total encoded length up front
and can reject large frames immediately.  CONTINUATIONS have the problem
that you don't know how big they will eventually get and they just keep
coming.  CONTINUATIONS also have a DOS vector of a client sending many 1
bytes CONTINUATIONS

The minus of jumbo frames is it makes sending huge headers a little harder
as you have to completely encode to know the length before you start
sending them.  Decoding can still be streamed if you really want.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 25 Jun 2014 17:36:03 +0900

> sending them.  Decoding can still be streamed if you really want.

In the sense that this discourages large headers, that would actually be 
a benefit.

Regards,   Martin.

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 09:35:59 +0000

On Wednesday,25 June 2014 06:11, [email-masked] wrote:
>
> The simplest way to address this would be to un-reserve the first two bits of the length
> <http://http2.github.io/http2-spec/#rfc.section.4.1>; that would get us back up to 64k,
> letting Willy get to 35 Gbps. Given that 256k only got him to 40 Gbps, that seems like a
> reasonable stopping point for HTTP/2.

Bad idea IMO.  That would really paint HTTP/2 into a corner.  With no reserved bits left, there would never be a chance to go above 64K frames.  i.e. you could never " back-port bigger frames onto an existing protocol"

> Doing more than 16 bits would take a lot more back-and-forth in the WG,
> and is likely to encounter a lot of resistance from implementers, from what I've seen.

We've been talking about jumbo frames for a week already and there hasn't been any resistance from "implementers".

I'll point out again that jumbo frames are optional.  As Matthew points out, the proposed setting can simply be ignored.

> WRT the "jumbo" frame (i.e., flagging that some prefix of the payload is actually an extension length)
> -- this sort of hack is necessary to back-port bigger frames onto an existing protocol. Let's not do it out of the gate.

Then make the frame length 64 bits and be done with it.  That's a lot of wasted bits though for those who just want to use 16K frames.

So barring a change to 64 bits, you need the CONTINUATION hack or the "jumbo frame" hack.  Which hack is better?

So far I've heard the following arguments in favour of "jumbo frames":
- Willy: higher performance for large data sets
- PHK: higher efficiency due to fewer system calls
- Greg: easier to describe and understand (keith: less error prone)
- "unlimited" HEADERS (albeit capped at some crazy large number)

So far I've heard the following arguments in favour of CONTINUATION:
- Greg: truly unlimited HEADERS

---

From: David Krauss <[email-masked]>
Date: Wed, 25 Jun 2014 18:20:18 +0800

> Bad idea IMO.  That would really paint HTTP/2 into a corner.  With no reserved bits left, there would never be a chance to go above 64K frames.  i.e. you could never " back-port bigger frames onto…

IPv6 packets only go up to 64K, so no network processor is going to get away with coarser granularity for the foreseeable future.

I’ve not been following this (my implementation is on hold as it seems HTTP/2 is going back to the drawing board) but any “jumbo frames” scheme should be amenable to hardware implementation. Needing a branchy CPU program for decoding would be painting into a corner.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 25 Jun 2014 12:23:40 +0200

Jumbo frames could easily be achieve without and extension bit if we change
the frame header to:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |R|                 Stream Identifier (31)                      |
    +-+-+-----------+---------------+---+---------------------------+
    |   Type (8)    |   Flags (8)   | R |     Length (14+)          |
    +-+-+-----------+---------------+---+---------------------------+
    |                   Frame Payload (0...)                      ...
    +---------------------------------------------------------------+

Where the length (14+) is the same variable length integer encoding used by
hpack.

This would support truly infinite frame sizes if desired, does not burn any
reserved bits and is not a hacked on extension, but just an specific length
encoding.

cheers

On 25 June 2014 11:35, <[email-masked]> wrote:

> bits of the length
> Bad idea IMO.  That would really paint HTTP/2 into a corner.  With no
> reserved bits left, there would never be a chance to go above 64K frames.
>  i.e. you could never " back-port bigger frames onto an existing protocol"
>
>
> I'll point out again that jumbo frames are optional.  As Matthew points
> out, the proposed setting can simply be ignored.
>
>
> actually an extension length)
> > -- this sort of hack is necessary to back-port bigger frames onto an
> existing protocol. Let's not do it out of the gate.
>
> Then make the frame length 64 bits and be done with it.  That's a lot of
> wasted bits though for those who just want to use 16K frames.
>
> So barring a change to 64 bits, you need the CONTINUATION hack or the
> "jumbo frame" hack.  Which hack is better?
>
> This email message is intended only for the use of the named recipient.
> Information contained in this email message and its attachments may be
> privileged, confidential and protected from disclosure. If you are not the
> intended recipient, please do not read, copy, use or disclose this
> communication to others. Also please notify the sender by replying to this
> message and then delete it from your system.
>

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 12:30:18 +0200

On Wed, Jun 25, 2014 at 06:20:18PM +0800, David Krauss wrote:
> 
> On 2014?06?25, at 5:35 PM, <[email-masked]> <[email-masked]> wrote:
> 
> IPv6 packets only go up to 64K, so no network processor is going to get away
> with coarser granularity for the foreseeable future.

Except that intermediaries are not packet processors, they are proxies and
they work with TCP streams. So they perform a read() call, retrieve as much
as they can from the system buffers, apply some processing, perform a send()
call and sleep until they're woken up again to signal that the buffer is
available again and/or that more data is available to read. That's exactly
this high frequency sleep/wake cycle that is performance critical. A lot of
syscall overhead and scheduling overhead for very little data processing.

When you have the ability to wake up to read 1 MB, and send it immediately,
at least you don't feel like you were woken up for nothing. Don't forget that
at 100 Gbps, this operation on 1 MB buffers is performed 12000 times a second!
At 64kB, that's 200000 times a second that you stop/start.

Willy

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 10:32:46 +0000

Great idea, though I would vote for the ASN.1 variable length integer style because (if I understand correctly - which is probably not true), the HPACK variable length integer is designed to work on non-byte boundaries because of the huffman encoding.

-keith

On Wednesday,25 June 2014 12:24, [email-masked] wrote:

> Jumbo frames could easily be achieve without and extension bit if we change the frame header to:

> Where the length (14+) is the same variable length integer encoding used by hpack.

> This would support truly infinite frame sizes if desired, does not burn any reserved bits and is not a hacked on extension, but just an specific length encoding.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 25 Jun 2014 20:56:50 +1000

> We've been talking about jumbo frames for a week already and there hasn't been any resistance from "implementers".

So far we’ve had a voluminous discussion among a small set of people who agree on generalities but not a single proposal, and only one of them has an actual implementation on offer.

I’d suggest that the reason why the rest of the implementers list hasn’t jumped in is because they’re busy getting draft-13 done, which we said we intended to go to WGLC any day now.

Making such a radical change to the protocol at this stage is going to need broad, strong agreement. I’m expressing doubt about the likelihood of that because so far, the interested parties haven’t converged on a single proposal, and (again) because none of the proposals are informed by running code.

Regards,

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 25 Jun 2014 13:15:37 +0200

On 25 June 2014 12:56, Mark Nottingham <[email-masked]> wrote:

> Making such a radical change to the protocol at this stage is going to
> need broad, strong agreement.

Mark,

while not disputing the fundamental point you are making, I would like to
point out that jumbo frames are really not such a radical change.
Continuations are essentially ugly jumbo frames that proxies cannot use
efficiently (and that do not apply to data frames).

I think the open issue here is do we wish to support large transfers with
large data frames?      If the WG thinks that issue is adequately handled
with 16KB frames, then the switch from continuations to jumbo frames is
essentially design aesthetics.

If there is some doubt in the WG that this issue is addressed, then I'd be
keen to put the effort in to come up with a concrete proposal and a working
implementation.

ie shouldn't we be more concerned about open issues rather than timetables.

cheers

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 25 Jun 2014 21:23:10 +1000

[email-masked] <[email-masked]> wrote:
> So barring a change to 64 bits, you need
> the CONTINUATION hack or the "jumbo
> frame" hack.  Which hack is better?

My original motivation for jumbo frames was that I wanted to break
hpack's blocking behaviour, by only compressing the HEADERS (or PP)
frame and making each CONTINUATION stand alone. Because some
individual headers are known to be >16K I needed a way to extend
CONTINUATION frames (and only them). Thus there'd be only one hack,
only useful for and only used by folks who insist on sending huge
headers -- and incidentally easy to detect and apply social
back-pressure against.

Where this has gone, it feels to me too much like a hacky workaround
with too many shortfalls and ragged corners. My opinion is that we
either bite the bullet and extend the common length field out to N*
bits (with settings for receiving variable-length frames
[max_data_length, max_headers_length, max_settings_length?] to keep
things sane), thus simultaneously fixing your data throughput concerns
and rendering CONTINUATION unnecessary, or suck it up and keep
everything the way it is.

I don't know whether it's better here to bite or suck. I suspect
people will have opinions.

*32, 64, whatever -- something everyone considers "big enough" for the
forseeable future.

---

From: Adrian Cole <[email-masked]>
Date: Wed, 25 Jun 2014 07:45:47 -0600

I'll offer another implementer pov.

TL;DR;

I'd be personally in favor of a jumbo frame, and am happier with using
a variable length integer encoding to deal with the unbounded nature
of it. Even if variable length integer is a rats nest, I'd prefer we
exhaust more energy on this one before giving up.

Continuations "feel" similar in code to concatenating data frames (to
equal a content length). Except that the latter seems more immediately
sensible as other frames can be interleaved. When you look at how to
statefully (carry end of stream flag, etc) page through continuations
under hpack, it indeed feels complicated and offers little value to
the *code*, basically makes it a spot that bugs are likely to creep
into. Jumbo is simpler, provided we address the length concern, plus
we can always bolt on constraints like a max on that. IOTW, the
framing itself is the more important bit.

Knowing that pinner's in hawaii, I'm sure we'll get at least another
implementer perspective a little later.

-A

---

From: "Ludin, Stephen" <[email-masked]>
Date: Wed, 25 Jun 2014 08:57:02 -0500

As implementors of a couple of version of h2, I would much prefer to keep
the core frame header processing as simple as possible.  The idea of
introducing a variable length integer there moves a very simple read of 8
bytes to a read, process, then possible ( and probably rarely ) read some
more to get the size.  It is not that it is necessarily hard or costly,
but it is work that seems unnecessary.  I understand the throughput
concerns and think Willy¹s suggestion of the 15th or 16th bit indicating
that the remaining size is in larger blocks ( size TBD ) is a great one.
I could get behind that.  I still have general complexity concerns around
h2 in general, but this seems a minor compromise for a larger good.

-stephen

---

From: Patrick McManus <[email-masked]>
Date: Wed, 25 Jun 2014 10:38:10 -0400

On Wed, Jun 25, 2014 at 6:56 AM, Mark Nottingham <[email-masked]> wrote:

> > We've been talking about jumbo frames for a week already and there
> hasn't been any resistance from "implementers".
>
> So far we’ve had a voluminous discussion among a small set of people who
> agree on generalities but not a single proposal, and only one of them has
> an actual implementation on offer.
>
> I’d suggest that the reason why the rest of the implementers list hasn’t
> jumped in is because they’re busy getting draft-13 done, which we said we
> intended to go to WGLC any day now.
>

indeed. If the IETF process can't converge to something we can ship after
all this time, based on experience with running code both before and during
the process, there isn't a lot of point of discussing it in this forum.
Redoing the base framing at this point is fairly close to declaring IETF
fail at this stage. I would hope we wouldn't do that.

different pov's will continue to crop up and the wg has decided to have an
extension mechanism available to try some of them out among their own
advocates. (as we know, I'm not a huge fan of that - but there it is.) This
will get runtime experience with them. Changing framing is certainly
something that can be done this way. It takes a RTT to take effect but the
swtichover can certainly take place mid stream among consensual peers. (so
it need not add a rtt of overall latency).

it would be interesting to get experience with an extension that did this
and bundled a MAX_CONCURRENT of 1 semantic (or requirement) with it
because you can't really expect to mux successfully in such an environment.
At that point the value of running h2 is kind of questionable - h2
fundamentally needs mux and priority. And that's why I think this whole
path is the wrong thing to do.

I'm looking at a spdy trace right now that sends huge frames because it was
convenient for the developer to do so - and priority doesn't work at all as
a consequence. That's real feedback that improved h2 over spdy.

-P

---

From: Johnny Graettinger <[email-masked]>
Date: Wed, 25 Jun 2014 11:48:08 -0400

FWIW, I don't buy the premise that the current framing mechanism requires
more frequent system calls, or must imply "lower performance" for large
sends. One frame != one system call to write or read that frame.

Vectored IO API's are available to write multiple frames with a single
call. One could even repeat a common DATA frame header buffer in your iovec
array: No additional allocation required, over that of a single frame.
When running over SSL, one already has to perform a mapping of frames to
TLS records. It doesn't need to be 1:1.

On Wed, Jun 25, 2014 at 10:38 AM, Patrick McManus <[email-masked]>
wrote:

> I'm looking at a spdy trace right now that sends huge frames because it
> was convenient for the developer to do so - and priority doesn't work at
> all as a consequence. That's real feedback that improved h2 over spdy.
>
> -P
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 18:12:18 +0200

On Wed, Jun 25, 2014 at 11:48:08AM -0400, Johnny Graettinger wrote:
> call.

... and they result in a data copy which is often even more expensive than
the syscall it tried to save.

When you're forwarding data between two TCP sockets and can make use of
TCP splicing, you have to play with pages (4kB). Anything not a full page
will result in a copy. And recv()+send() will result in two copies. That's
why splice() on small sizes (typically 16kB) offers no benefit : the first
and/or the last page are often incomplete or unaligned, resulting in only
the two middle ones being spliced from the CPU's L3 cache without ever
hitting memory. Also, with a 14-bit encoding, we cannot transfer 16 kB,
we can at most transfer 16kB-1, so you're always guaranteed to *copy*
4095 bytes at the end of each transfer and to misalign every data block.

16kB are *really* suboptimal for large transfers. I just got a report of
a company reaching 58 Gbps of forwarded traffic with haproxy, using a
splice size of 512kB. There *are* definitely a lot of losses to expect
from running at 16kB-1. I'd give 10-15 Gbps for the setup above, not
more.

I'm not advocating for making things complex nor for breaking the protocol
either, I'm just reporting real world scenarios which work very well with
1.1 and which are 2.0-unfriendly. It's not too late to try to fix these
corner cases. I think that the ability to use just one bit to change the
size unit is reasonable. It's exactly what TCP uses with Window Scaling
and it works pretty well. We can easily accept that the unit only changes
after the first round trip if needed, what matters is that not all the
stream is retrieved in tiny 16kB chunks.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 17:47:28 +0000

In message <[email-masked]>

>Jumbo frames could easily be achieve without and extension bit if we change

I agree that it would be cleaner to totally redesign the frame-header,
but we have to pay some respect to the fact that some people think
we're ready for Last Call.

My proposal was intended to slot as seamlessly in as possible for
that reason:

* Define one of the reserved bits as adding an 8 byte network
  byte order length field in front of the payload.  (No need to
  make it terribly complicated, 8 bytes out of 16KB+ is < epsilon.)

* Add SETTINGS defaulting to 16KB, and mention that using the
  extention length is only legal if SETTINGS > 16KB has been
  received.

* Remove every mention of CONTINUATION

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 10:56:03 -0700

> * Remove every mention of CONTINUATION

Just for clarification,

1. Assuming L1 is the 14 bits in the header and L2 is the 64 bits
before the payload, is the length of the frame:
  a. L2
  b. L1 << 64 | L2
  c. L2 << 14 | L1
  d. L1 + L2
  e. ?

2. If the bit is set, do the 8 additional bytes count toward this number?

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 17:57:24 +0000

>  extention length is only legal if SETTINGS > 16KB has been

I like this option because it requires the least amount of processing.

Needs clarification...
1. When the "jumbo" bit is set, is the possible length 14-bits + 8-bytes or just 8-bytes?  (I vote for just 8 bytes)

2. SETTINGS are only 32 bits. Depending on your answer to #1, there would need to be two or three SETTINGS (low, high, high-high) to capture the full range of possible MAX frame size values.  Perhaps a better option would be to have the setting defined as a multiple of 16K chunks (minus one?).

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 13:42:29 -0500

> It's true that you can't use sendfile() here, but that's true with multiplexing regardless. It was felt that over time, kernel facilities specific to the use case of HTTP/2 will emerge if necessary…

I’m not sure they would. OSs tend to balk at application specific APIs and prefer reusable primitives. Even if an OS did though, it still has the cost of inserting frame headers every 16K, and that is not cheap. It also seems brittle to share the framing layer.

---

From: Nicholas Hurley <[email-masked]>
Date: Wed, 25 Jun 2014 11:48:15 -0700

On Wed, Jun 25, 2014 at 3:56 AM, Mark Nottingham <[email-masked]> wrote:

> need broad, strong agreement. I’m expressing doubt about the likelihood of
> that because so far, the interested parties haven’t converged on a single
> proposal, and (again) because none of the proposals are informed by running
> code.
>

(co-)Implementer of 2 implementations chiming in...

Exactly. Jumbo frames seem like prime candidates for an extension,
especially since no one can seem to come together on a single plan for what
they would look like. And given that we've already removed things from the
spec that had some actual implementation experience (or at least broad
implementer interest - I'm thinking ALTSVC and BLOCKED), I see no reason to
add something to the spec just before WGLC.

If the procedural concerns aren't enough (they don't seem to have been in
the past, so I'm not sure they will be now), let's talk about actual
experience. We have running code, right now, from a good number of
implementers, that works with the regular-sized frames as currently in the
spec. The current frame layout is simple, easy to parse, and easy to make
decisions about. With all the perceived "complexity" that people have been
complaining about, I'm honestly a little surprised that people are asking
to add even more complexity (some of whom seem to be the same people
complaining earlier!)

I'm not convinced by the arguments around large file downloads - those are
the exception rather than the rule. Optimize for the common case. In HTTP,
that means viable multiplexing and priority, both of which are much more
effective with frame size limited to 16k like we have it now.
--
Peace,
  -Nick

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 11:54:39 -0700

> I’m not sure they would. OSs tend to balk at application specific APIs and prefer reusable primitives. Even if an OS did though, it still has the cost of inserting frame headers every 16K, and that…

I know of at least one major operating system that supports this sort
of function already.  And let's be clear: HTTP is important enough to
allocate custom kernel resources to improve performance.  I'd argue
that it's important enough to dedicate silicon to eke out a few
milliseconds or watts.

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 13:55:12 -0500

On Jun 25, 2014, at 1:48 PM, Nicholas Hurley <[email-masked]> wrote:

> Exactly. Jumbo frames seem like prime candidates for an extension, especially since no one can seem to come together on a single plan for what they would look like. And given that we've already removed things from the spec that had some actual implementation experience (or at least broad implementer interest - I'm thinking ALTSVC and BLOCKED), I see no reason to add something to the spec just before WGLC.

Extensions are length limited to 16KB, so to allow that you would have to define special framing rules, and if you have done that might as well apply it everywhere. 

> 
> If the procedural concerns aren't enough (they don't seem to have been in the past, so I'm not sure they will be now), let's talk about actual experience. We have running code, right now, from a good number of implementers, that works with the regular-sized frames as currently in the spec. The current frame layout is simple, easy to parse, and easy to make decisions about. With all the perceived "complexity" that people have been complaining about, I'm honestly a little surprised that people are asking to add even more complexity (some of whom seem to be the same people complaining earlier!)

The rough proposals are trivial changes thought. We are talking about a small change to a length value and a new setting to negotiate. 

> 
> I'm not convinced by the arguments around large file downloads - those are the exception rather than the rule. Optimize for the common case. In HTTP, that means viable multiplexing and priority, both of which are much more effective with frame size limited to 16k like we have it now.

It’s a very common use-case.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 19:02:41 +0000

In message <[email-masked]>

>before the payload, is the length of the frame:

If we make L2 8 byte wide, I'd prefer a)

>2. If the bit is set, do the 8 additional bytes count toward this number?

I'd prefer "no"

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 19:12:43 +0000

In message <0356EBBE092D394F9291DA01E8D28EC201186F24D7@sem002pd>, K.Morgan@iaea
.org writes:

>2. SETTINGS are only 32 bits. Depending on your answer to #1, there would
>need to be two or three SETTINGS (low, high, high-high) to capture the full
>range of possible MAX frame size values.  Perhaps a better option would be
>to have the setting defined as a multiple of 16K chunks (minus one?).

So this is a weak spot, because SETTINGS is only for the connection.

Ideally you should be able to bang up the frame size for a single
stream (as in: Client sess "Content-Length: ALOT" and tells server
to "just get it overhere ASAP" with a big frame size for that stream.

However, that's probably overthinking things, since connections where
that would help are very likely to have only one stream to begin with.
(If we later allow SETTINGS per stream, we get this for free.)

With a little tweak SETTINGS could be 56 bits, that's the cleanest
solution, especially for future settings we havn't thought about.

If we want to fit inside 32 bits, we can either limit the framesize
to 4GB, which is probably safe, but robs us the ability to send
huge objects as a single DATA frame.

Another way could be to go "scientific":

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Identifier (8)|
    +---------------+-------------------------------+---------------+
    |                        Value (24)             + Shift (8)     |
    +-----------------------------------------------+---------------+

Where the max framesize will then be

	Value * 2^(Shift)	/* (Value << Shift) in C syntax. */

I prefer making SETTINGS 56 bits.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 21:16:09 +0200

On Wed, Jun 25, 2014 at 11:48:15AM -0700, Nicholas Hurley wrote:
> they would look like.

Which is precisely the purpose of this discussion.

> And given that we've already removed things from the

You know, that's always the principle of cost of errors which multiplies
by ten at each step towards the end user. Fixing a bug before compiling
is cheaper than after, which is cheaper than after release which is cheaper
than after it reaches customers, which is cheaper than once customers hit
their own customers etc...

> spec.

Don't get me wrong, I don't want to dismiss anyone's work on the subject,
but you also need to accept the fact that for some low level components
like some intermediaries, implementing such a protocol is a tremendous
work and that even the smallest changes can become a nightmare to adapt,
so it is normal to observe a cool down period. To be clear, I *hope* to
be able to implement end-to-end HTTP/2 in haproxy in one year, but I'm
sure I'm dreaming...

> The current frame layout is simple, easy to parse, and easy to make

I'm personally not complaining about the protocol's complexity itself,
but am trying to warn people about choices which will definitely hurt
implementations. The difficulty that comes with the protocol is not
parsing it, it's dealing with multiplexing between N clients and M
servers in products which are designed to pass data blocks at the
fastest possible speed between both sides. And among the issues, we
observe that a few of the crucial capabilities these products rely on
(ability to transfer large blocks between the two sides) will not be
possible anymore (16kB-1 bytes is fine for 1Gbps, but that's about all).

> the exception rather than the rule.

Most of network operators and equipment providers disagree with you. On
this page, Cisco claims that video already represents 66% of all the
internet's IP traffic, and they even expect 79% for 2018 :

   http://www.cisco.com/c/en/us/solutions/service-provider/visual-networking-index-vni/index.html

And you'll easily find a few others from Akamai and other CDNs citing
similar numbers.

> Optimize for the common case.

That's what we're trying to do :-)

> In HTTP,

I absolutely agree, and we're precisely discussing the use case of the
vast majority of the internet traffic which is not concerned by this
case. OK we found how to finally make sites load fast, in large part
thanks to the good research work made by the SPDY team. Now we're
realizing that only *this* aspect was covered (and it was the most
difficult one so that's normal) and that some of the design decisions
made for this use case hurt the rest of the traffic and that it's not
too late to fix this with the most minimal changes.

I think it's the right approach to the problem to get the best in both
use cases.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 19:17:30 +0000

In message <CANV5PPV30o6bWWF=[email-masked]>

>effective with frame size limited to 16k like we have it now.

Large in this context is "Larger than 16 kilobytes" and those are the
norm, not the exception.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 21:18:23 +0200

> >2. If the bit is set, do the 8 additional bytes count toward this number?

I agree with Poul-Henning on both responses. They are the choices with the
least operations and the least chances of doing the wrong thing.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 19:30:02 +0000

In message <[email-masked]>

>that it's important enough to dedicate silicon to eke out a few

While that is true, even in kernel code 16kB framesize is suboptimal
from a performance point of view when the majority of all objects
are larger than that.

At 100 Gbit/s, you'll be north of half a million frames per second,
statistically probably very close to full million frames per second.

Being able to cut that number by a factor of 10 will matter a lot to
performance -- even if you allocate silicon.

100 Gbit/s NICs are close to shipping in bulk and some people are
already talking about 400 Gbit/s ethernet as the next step.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 25 Jun 2014 15:30:36 -0400

hey all -

> Extensions are length limited to 16KB, so to allow that you would have to
> define special framing rules, and if you have done that might as well apply
> it everywhere.
>

I don't think that's the case at all. un-acked extensions are limited that
way because you can't change the semantics of the protocol.. but once you
ack something you can do anything you like - including changing framing to
look like websockets via use one of the R bits (as an example). Go
bootstrap h3 that way if you feel so inclined to write an extension like
that.

but maybe I have that wrong -(our extension text is brand new afterall) and
extensions are meant to preclude such hackery which seems odd - given they
are essentially a hackery escape hatch that already acknowledges the
ability to change semantics of existing elements. If that's true there is
always the alpn method of proving this can work - because I really don't
think it can work and leave h2 responsive.

> It’s a very common use-case.

unfortunately its one that is in direct competition with the driving reason
for h2: prioritized mux on one connection. Minimally you should prove out
that can coexist with jumbo frames because spdy showed us it was a problem.
You simply can't send a multi megabyte frame and also then respond quickly
to a CANCEL and a new GET on the same connection which is something any h2
implementation should be able to handle responsively. My H1 application
closes a ton of connections because of that pattern - h2 doesn't have to.
Its one of its under appreciated features. The cost of that feature, 16KB
frames, is reasonable imo.

and of course, then there is TLS.

ship it.

---

From: Nicholas Hurley <[email-masked]>
Date: Wed, 25 Jun 2014 12:41:03 -0700

Accidentally hit regular reply instead of reply all on this one... (sorry
for the dupe, Jason)

On Wed, Jun 25, 2014 at 12:38 PM, Nicholas Hurley <[email-masked]>
wrote:

> On Wed, Jun 25, 2014 at 11:55 AM, Jason Greene <[email-masked]>
> wrote:
>
>>
> I don't see any text that says that.
>
>
>>
>> The rough proposals are trivial changes thought. We are talking about a
>> small change to a length value and a new setting to negotiate.
>>
>
> The same was true for both ALTSVC and BLOCKED - trivial, totally ignorable
> frames (they didn't even have a setting associated with them, IIRC)! But we
> removed them from the spec anyway, and agreed to make them extensions, in
> favor of shipping it. There's no reason to block h2 for something else that
> can be done as an extension.
>

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 19:42:57 +0000

On 25 June 2014 21:30, [email-masked] wrote:
> unfortunately its one that is in direct competition with the driving reason for h2:
> prioritized mux on one connection. Minimally you should prove out that can
> coexist with jumbo frames because spdy showed us it was a problem. ...

Adding the jumbo frame capability does not force an implementation to send or accept frames longer than 16K-1.

The default is 16K-1 and won't change unless you update the setting.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 12:48:06 -0700

The same is true for any other optional thing under the sun. It doesn't
seem to be a motivating argument for including more optional features,
though.
-=R

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 12:47:11 -0700

The size of a header block is orthogonal to the max frame size so long as
it is unlimited, as is true today and as is required today for
compatibility with the tail.
Getting rid of compatibility in the tail is a different discussion.

For any implementation which is using TLS, or not using sendfile/splice,
the size of a frame has zero correlation with the number of system calls
for a non-naive implementation, so I don't think the number of syscalls
here matters.
You read into a buffer, you append/pad/skip space for the header, read, etc.

The benefit to jumbo frames is solely that one can do fewer sendfile/splice
calls, at the cost of more framing complexity, and at the cost of having
naive implementations (especially server implementations) do things that
would affect performance/latency such that HTTP2 would be a downgrade.
Patrick mentions this, and it is a large motivator here-- getting the
muxing wrong, as is much more likely to happen with larger frame sizes,
means poor latency.

Experimentation with this seems reasonable as an extension.
-=R

On Wed, Jun 25, 2014 at 12:30 PM, Patrick McManus <[email-masked]>
wrote:

> but maybe I have that wrong -(our extension text is brand new afterall)
> and extensions are meant to preclude such hackery which seems odd - given
> they are essentially a hackery escape hatch that already acknowledges the
> unfortunately its one that is in direct competition with the driving
> reason for h2: prioritized mux on one connection. Minimally you should
> prove out that can coexist with jumbo frames because spdy showed us it was
> a problem. You simply can't send a multi megabyte frame and also then
> respond quickly to a CANCEL and a new GET on the same connection which is
> something any h2 implementation should be able to handle responsively. My
> H1 application closes a ton of connections because of that pattern - h2
> doesn't have to. Its one of its under appreciated features. The cost of
> that feature, 16KB frames, is reasonable imo.
>

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 14:50:01 -0500

> Anyone attempting to implement it should know enough to use an algorithm of known O(1) computational complexity, although O(log N) choices are also viable.

The problem isn’t algorithmic complexity. Scheduling work using priority queues has more cpu contention and cost than a typical non fair scheduler. Further buffering/queuing to ensure priority means adding latency and consuming more server resources.

> This only works as long as the server processes requests serially, which isn’t even possible with a reverse proxy.

A reverse proxy doesn’t map well to the draft priority mechanism either because the client isn’t typically aware of which resources are proxied. 

> And it moves the scheduling complexity to the client side, which now must defer low-priority requests. Terrible architecture.

Yes, and this is a good thing. There is most likely more combined CPU resources from clients than there are in endpoints and intermediaries.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 21:51:20 +0200

> from a performance point of view when the majority of all objects

16kB-1 please, or 3.99975589375 pages, thus 3 pages hence 12kB or 8 MSS
in practice for most usages :-(

> statistically probably very close to full million frames per second.

Yes and at 3 GHz, that's 3000 cycles per frame, which are easily
wasted doing a plain data copy of 16383 bytes, some cache misses +
a little bit of synchronization job. I'd rather have 30000 cycles
to forward 10 times this and avoiding the copy.

> performance -- even if you allocate silicon.

Especially if moved to sillicon, because the round-trip to hardware
is particularly expensive, which is why some chip makers have moved
the crypto accelerators into the CPU's instruction set for example.

> already talking about 400 Gbit/s ethernet as the next step.

I'm impatient :-)

Willy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 25 Jun 2014 21:52:19 +0200

So if we can boot efficient transfer of large payloads to an extension, can
be we also punt large headers to an extension?

I get it that we are late in the process and have already got a some
experience with the current implementations - but seriously how much real
usage have continuation frames got? or will get in the near future?  Have
servers that support CONTINUATIONS been tested for DOS attacks from 1 byte
CONTINUATION frames?

cheers

On 25 June 2014 21:41, Nicholas Hurley <[email-masked]> wrote:

>>> Extensions are length limited to 16KB, so to allow that you would have
>>> to define special framing rules, and if you have done that might as well
>>> apply it everywhere.
>>>
>>
>> The same was true for both ALTSVC and BLOCKED - trivial, totally
>> ignorable frames (they didn't even have a setting associated with them,
>> IIRC)! But we removed them from the spec anyway, and agreed to make them
>> extensions, in favor of shipping it. There's no reason to block h2 for
>> something else that can be done as an extension.
>>
>
>

---

From: Nicholas Hurley <[email-masked]>
Date: Wed, 25 Jun 2014 12:58:52 -0700

> their own customers etc...

Understood. But given there are already rumblings about working on h3, why
not ship what we have experience with, and when we determine something to
have room for improvement, roll it into h3?

> be able to implement end-to-end HTTP/2 in haproxy in one year, but I'm

Totally understood! I've spent the better part of the last year working on
this, so I know the amount of work involved :)

> And you'll easily find a few others from Akamai and other CDNs citing

In terms of absolute byte count, sure, I won't argue that point (I may be
dumb, but I'm not THAT dumb). But for a regular web page (which my totally
unscientific gut feeling suggest there are still way more of those than
videos loaded in terms of number of resources),
http://httparchive.org/interesting.php#responsesizes indicates that (at
least for now), 16k is a reasonable size, as the average resource (except
flash... ugly, ugly flash) fits within 2 frames at most. (Please note, I'm
not trying to pull a "64 (or in this case 16) k is be enough for everyone"
thing, just pointing out some other current, relevant data.)

> I think it's the right approach to the problem to get the best in both

Fine, but again, let's make it an extension. That gives us the time and
flexibility to really determine which approach is best. You could even do
an A/B test on different jumbo frame layouts! If my users complain of slow
downloads or bad streaming video experience in the future, and the data
suggests I can fix that without adversely affecting normal page loads
(social networks, email, etc) by implementing a jumbo frame extension, then
I'll gladly do it. However, as it stands, my users are experiencing
suboptimal page load speeds right now, and I have a way to fix that. Ship
it.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 21:59:41 +0200

Hi Roberto,

On Wed, Jun 25, 2014 at 12:47:11PM -0700, Roberto Peon wrote:
> calls, at the cost of more framing complexity,

If we just use a single bit to indicate whether the advertised length is
shifted or not, I don't think we can call that "more framing complexity",
and I would personally be fine with this (as with anything that allows
to pass 1MB or so of data I guess).

> and at the cost of having

Well, a naive implementation will not do any muxing at all, that's the
most complex part to deal with in my opinion. I think that anyone who's
able to implement the muxing correctly will not have any trouble with

     len <<= (8 * R);

And for senders, it does not change anything, they continue to send the
reserved bit to zero and nothing changes over the wire.

> Experimentation with this seems reasonable as an extension.

I suspect that we're going to rediscover the benefits of TCP's window
scaling :-/

Regards,
Willy

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 13:00:51 -0700

On 25 June 2014 12:30, Patrick McManus <[email-masked]> wrote:
> look like websockets via use one of the R bits (as an example). Go bootstrap
> h3 that way if you feel so inclined to write an extension like that.
>
> are essentially a hackery escape hatch that already acknowledges the ability
> to change semantics of existing elements. If that's true there is always the
> alpn method of proving this can work - because I really don't think it can
> work and leave h2 responsive.

You have it right.  Once you get mutual agreement to use an extension,
that extension only needs to worry about the synchronization point at
which you change anything and everything.  Here's a basic strawman:

New setting X, 0 = off (initial), 1 = on.  Once a setting has been set
to "on" on both sides and been acknowledged, you can enable the use of
the reserved bits, exactly as PHK has described.  If a peer uses the
extension prior to both acknowledging the settings and sending their
own setting enabling the setting, you are entitled to treat that as an
error.

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 20:00:10 +0000

On 25 June 2014 21:48, [email-masked] wrote:
> The same is true for any other optional thing under the sun.
> It doesn't seem to be a motivating argument for including more optional features, though.
> -=R

That's not the point.  Patrick implied that adding the jumbo frame option kills the ability to efficiently mux.  That's simply false.

On Tuesday,22 April 2014 09:00, [email-masked] said [1]:
"Likewise, restricting packet sizes to a small length in order to prevent fools from HOL blocking
 their own multiplexed channels makes some sense, for browser developers.
 However, it actively harms applications of HTTP that are not interested in multiplexing
 because they only want to transmit a single large data stream.
 ... I don't think it makes sense to limit an application-level protocol to the worst case.
 ....Roy"

So let's be straight.  There are legitimate use cases for jumbo frames.  You and Patrick just think the world is full of a bunch of what Roy calls "fools" (see above) who will HOL block their own multiplexed channels.  Roy (and I) don't disagree.  Roy (and I) don't think it makes sense to limit the protocol to protect against an unknown number of fools.

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0416.html
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:02:14 -0700

On Wed, Jun 25, 2014 at 12:51 PM, Willy Tarreau <[email-masked]> wrote:

> [email-masked]>
> > , Martin Thomson writes:
> >
> True IFF not using TLS, at which point we're doing more copies, etc.
Honestly, spending money/effort getting those libraries correct/optimized
would have a substantially larger impact than worrying about the framesize
in almost all web cases for HTTP2.

> the crypto accelerators into the CPU's instruction set for example.
We have examples of this kind of thing in hardware and working to reduce
cost today:
TCP Segment Offload (TSO) offloads making TCP segments to the NIC's
hardware.

If we're talking about non-TLS stuff, then doing this kind of simple thing
on the NIC doesn't seem that hard. It is roughly the same thing.

-=R

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 13:03:36 -0700

> be we also punt large headers to an extension?

That's an obvious and reasonable question.  I've heard enough interest
in having large header field values that I think that this would be a
little too aggressive.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:06:52 -0700

On Wed, Jun 25, 2014 at 1:00 PM, <[email-masked]> wrote:

> That's not the point.  Patrick implied that adding the jumbo frame option
> kills the ability to efficiently mux.  That's simply false.
>

A jumbo frame option requires the implementation to be smart to before the
protocol becomes effective/useful for web traffic.

> So let's be straight.  There are legitimate use cases for jumbo frames.
>  You and Patrick just think the world is full of a bunch of what Roy calls
> "fools" (see above) who will HOL block their own multiplexed channels.  Roy
> (and I) don't disagree.  Roy (and I) don't think it makes sense to limit
> the protocol to protect against an unknown number of fools.
>

Keep in mind that the original SPDY had much, much larger max framesizes,
and I designed it that way in concert with Mike.
Implementation experience, however, provided hard data that this actually
happens, and harms the user.

-=R

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 15:02:56 -0500

> response frame ordering on the client connection regardless of request

Thats a good point. My mental model was focusing on execution ordering. I can see a benefit when you have data ready to ship on the response side, and ordering the frames in a particular way is helpful to the client.
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 22:07:54 +0200

> own setting enabling the setting, you are entitled to treat that as an

The issue I'm having with the extension is very simple : I'm implementing
an intermediary which will have to adapt to both sides which disagree on
their extensions. It's the same as adapting the Connection header when one
side is 1.0 while the other one is 1.1 for example, or ensuring that one
does not use chunked-encoding when the other side does not support it.

Willy

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:12:56 -0700

> to pass 1MB or so of data I guess).
Look at this from a hardware engineer's perspective. This bit changes how
you must structure hardware buffers in order to parse things properly.
This requires far more complexity for a hardware implementation, and would
reduce the chance that we get acceleration in HW for HTTP2.

> able to implement the muxing correctly will not have any trouble with

That isn't what we've seen, however. We've seen that implementations get
that muxing is an important part of the protocol, however the implication
that small frames is essential to being able to respond quickly to a change
in circumstance is not.
Large enough frames might require a receiver to close the connection (as
we've seen from HTTP) when it is uninterested in a response when the
bandwidth is lower, for instance. Using smaller frames vastly decreases the
chance that this needs to be done, and simplifies the receiver's
implementation.

> I suspect that we're going to rediscover the benefits of TCP's window
Given that the frame size has nothing to do with RTTs or flow control, I
think that is orthogonal and completely different :)

-=R

---

From: Mike Belshe <[email-masked]>
Date: Wed, 25 Jun 2014 13:12:28 -0700

I know that you can make bulk-transport more efficient - this was
identified before the project even got started.

As it turns out, the number of folks that need super high efficiency bulk
transport of large data over the internet is very low.  Of those very few
that do need it, many are already moving to non-HTTP protocols or simply
have better solutions already.  For example:
   - many back offices use custom protocols designed for their security &
global transport requirements rather than any form of HTTP.  They aren't
stuck with HTTP since it is on their own networks.
   - streaming video transfers don't need this, because they self-throttle
bandwidth anyway to save costs.
   - consumer bulk transfers tend to be best served by protocols like
bittorrent, which are designed for this and solve much more than just the
bandwidth problem.

I'm not saying that nobody needs nor wants super high efficiency bulk
transfer over HTTP.  Sure, it would be great.  But I wouldn't spend a
single bit of complexity on it.  Just too low pri.

I propose we move forward without attempting to solve bulk transport
issues.   If it fits into extensions without additional work, great.  If
not, we'll survive.  It's just another low-priority feature, of which there
will always be more.

Mike

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 15:15:53 -0500

On Jun 25, 2014, at 3:06 PM, Roberto Peon <[email-masked]> wrote:

> A jumbo frame option requires the implementation to be smart to before the protocol becomes effective/useful for web traffic. 
> 
> 
> Keep in mind that the original SPDY had much, much larger max framesizes, and I designed it that way in concert with Mike.
> Implementation experience, however, provided hard data that this actually happens, and harms the user.

Happens from where, the browser? Since the browser controls the channel, it can simply set whatever frame size it wants.

If it wishes to implement fast downloading of large files, it can simply open a new connection using the larger size and limited to no multiplexing.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 25 Jun 2014 22:17:51 +0200

> single bit of complexity on it.  Just too low pri.

Mike,

these are good points that do address issue, even if they do so by punting
high efficiency bulk transfers to other protocols and/or extensions.

I just don't understand why the same arguments don't apply for super large
headers?

cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 20:19:35 +0000

In message <CABaLYCvFxo0qPG=[email-masked]>
, Mike Belshe writes:

>transport of large data over the internet is very low.

The segment of webmasters who usually introduce themselves as "being
in the multimedia business" would not agree with you about that.

Since their pink bits account for between one and two thirds of all
HTTP traffic, I think it would be a bad idea to marginalize them,
even if they don't participate directly and don't come to the
meetings.

And:

The length extension proposal is not *just* about solving bulk
transfer, it is *also* about getting rid of the CONTINUATION kludge.

---

From: Mike Belshe <[email-masked]>
Date: Wed, 25 Jun 2014 13:19:43 -0700

> think that is orthogonal and completely different :)

+1 to everything Roberto stated here.

Mike

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 20:20:43 +0000

In message <CAP+FsNcwTBLjn+CV0gnrpG_87U=[email-masked]>

>reduce the chance that we get acceleration in HW for HTTP2.

But the elimination of CONTINUATION would more than offset that minor
bit of complexity.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 22:22:19 +0200

> TCP Segment Offload (TSO) offloads making TCP segments to the NIC's

TSO works if you feed it with enough data. That's precisely one point
where 16kB makes it totally useless, because the cost of preparing the
fragment descriptors for the NICs overrides the savings of cutting them
into a few packets in the TCP stack.

> on the NIC doesn't seem that hard. It is roughly the same thing.

Not for an intermediary. I have to receive muxed streams from multiple
servers and deliver them over muxed connections to multiple clients.

Please consider this simple use case :

   - requests for /img /css /js /static go to server 1
   - requests for /video go to server 2
   - requests for other paths go to server 3

Clients send their requests over the same connection. The load balancer
has several connections to servers 1 and 3 behind and forwards clients'
requests over these connections to retrieve objects. In practice, a
client will go first to server 3 (GET /) then to server 1 (retrieve
page components) then to server 2 over a fresh connection and stay
there for a long time. There's no place for NIC-based acceleration
here because this MUX pulls data from one side and transfers it to
the other side in small chunks. When the video starts, if we had the
ability to splice large chunks from server 2 to client, there would
be a real benefit. With the small chunks, the benefits disappear and
we're back to doing the same recv+memcpy()+send job as for the other
servers (double to triple copy instead of zero).

Sure this is not something a common server or user-agent is even able
to detect at regular loads. But you (and you particularly) know like
me that intermediaries need to shave off everything possible to avoid
wasting time doing dumb things such as copying small data or visiting
the same byte twice, etc...

To be clear, this is not the end of the world, this will only probably
lead to a significant part of the internet not deploying what was designed
here, waiting for H3 to appear, when we could easily make it worth for
them to consider the option.

Regards,
Willy

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:22:00 -0700

The sender determines how to apportion data into frames. We were seeing
whole files transmitted as a single frame.
And again, in the web use-case this is not going to be a measurable
difference in utilization, since we're likely to be using TLS.
-=R

On Wed, Jun 25, 2014 at 1:15 PM, Jason Greene <[email-masked]>
wrote:

> > That's not the point.  Patrick implied that adding the jumbo frame
> option kills the ability to efficiently mux.  That's simply false.
> >
> > A jumbo frame option requires the implementation to be smart to before
> the protocol becomes effective/useful for web traffic.
> >
> >
> > Keep in mind that the original SPDY had much, much larger max
> framesizes, and I designed it that way in concert with Mike.
> Happens from where, the browser? Since the browser controls the channel,
> it can simply set whatever frame size it wants.
>
> If it wishes to implement fast downloading of large files, it can simply
> open a new connection using the larger size and limited to no multiplexing.
>
> --
> Jason T. Greene

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:24:06 -0700

I don't see how CONTINUATION makes a hardware implementation more
difficult, though?
It doesn't at all change the buffer management.
-=R

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 13:25:20 -0700

> does not use chunked-encoding when the other side does not support it.

Where that sucks most is where you have to re-segment.  But I have
faith in your ability to do the right thing.

Keep in mind that as proposed, you are unlikely to convince clients
(browsers in particular) to concede to having large frames thrust at
them.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:26:41 -0700

I'll note that I work at YouTube, and care deeply about video
distribution... and I'm not worried about it.

As a matter of fact, I'd really, really like to take advantage of the
prioritization scheme along with smaller frames so that I can have better
codecs in the future which do much better adaptive stuff w.r.t. bandwidth
changes.
This matters far more to me.

As it turns out, our users care deeply about the quality of their
experience.
-=R

On Wed, Jun 25, 2014 at 1:19 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <CABaLYCvFxo0qPG=
> [email-masked]>
> , Mike Belshe writes:
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 20:27:40 +0000

In message <CAP+FsNfKBazaKL4a8PE8mMtCdFhKj1tyXQ-6twAF=[email-masked]>

>It doesn't at all change the buffer management.

It sure does:  The "magic" blocking of CONTINUATION frames into one
unit where the END_* bits don't actually mean END_* at all will
seriously complicate a hardware transmit-side implementation.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 25 Jun 2014 22:28:17 +0200

Roberto,

CONTINUATION will not make hardware implementations more difficult, because
they (like many other implementations) will just not implement continuation
support.

cheers

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 15:30:35 -0500

> own setting enabling the setting, you are entitled to treat that as an

That wasn’t clear to me in the text, thank you for clarifying the intention.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:35:57 -0700

Why would the hardware care? I'd imagine it would simply forward the data
to the system. I'd be rather surprised if it did otherwise.
It wouldn't change its buffer management at all, not would it required a
memory layout change.
-=R

On Wed, Jun 25, 2014 at 1:28 PM, Greg Wilkins <[email-masked]> wrote:

> Roberto,
>
> CONTINUATION will not make hardware implementations more difficult,
> because they (like many other implementations) will just not implement
> continuation support.
>
> cheers
>
>
>

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 13:45:34 -0700

> That wasn’t clear to me in the text, thank you for clarifying the intention.

The full range of possibilities are not enumerated, no.  But that is
the intent of p4 in http://http2.github.io/http2-spec/#extensibility .
It's brief, partly as an economy thing, but mostly because it's not
clear that all avenues of extension are equally valid.  Before we know
what extension options are actually viable, I'd rather not
accidentally condone a particular non-viable one.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 13:56:32 -0700

> into a few packets in the TCP stack.

Frame size != write() size, unless you're attempting to use Splice() (which
often isn't cheaper, sadly).

> servers (double to triple copy instead of zero).

If the hardware was similar to TSO hardware, that case would work just as
it does for TSO.
You send data to the NIC via the kernel, potentially with a different API
(e.g. socket per stream), and the NIC (or kernel) fragments into frames.
It'd work just fine.

> the same byte twice, etc...
True (you've seen me push to reduce memory requirements), though my
motivation w.r.t. CPU consumption is to balance any change in resource
requirements against the change in the user's experience.

-=R

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 20:59:32 +0000

On 25 June 2014 22:12, [email-masked] wrote:
> Look at this from a hardware engineer's perspective.
> This bit changes how you must structure hardware buffers in order to parse things properly.

This is a good point, and valuable feedback for designing jumbo frames, but easily solvable with Willy's proposal or any other variation with a fixed length field for the payload length.

> This requires far more complexity for a hardware implementation,
> and would reduce the chance that we get acceleration in HW for HTTP2.

Exactly what is the http2 hardware acceleration doing? Reassembling frames?
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 21:04:58 +0000

On 25 June 2014 22:22, [email-masked] wrote:
> The sender determines how to apportion data into frames.
> We were seeing whole files transmitted as a single frame.
> -=R

The sender can only apportion data into frames less than SETTINGS_MAX_FRAME length; and if you never change it, he has to use 16K or less.

This is exactly the point. The jumbo frames eliminates the CONTINUATION "kludge" & allows use cases which need jumbo frames to have them if the peer gives consent.  All while not preventing efficient muxing with 16K frames.
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 14:05:53 -0700

On Wed, Jun 25, 2014 at 1:59 PM, <[email-masked]> wrote:

> This is a good point, and valuable feedback for designing jumbo frames,
> but easily solvable with Willy's proposal or any other variation with a
> fixed length field for the payload length.
>

Assembling or removing the framing, yes.
In the end the application doesn't care about the framing. It cares about
getting the data in the order it requested as a stream of bytes or
messages, and possibly it cares about padding.
-=R

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 25 Jun 2014 23:10:07 +0200

> often isn't cheaper, sadly).

That's clearly my intent (sorry if I didn't make it clear initially) and
for it to be cheaper, it needs to operate on large blocks.

> (e.g. socket per stream), and the NIC (or kernel) fragments into frames.

I'm not saying it does not work, I'm saying that there are two ways to
do the operations :
  - triple copy to make use of TSO and benefit from hardware acceleration,
    but at 100Gbps, the less copies the better ;

  - zero copy and many very small splice() calls that are worthless at
    these sizes.

> requirements against the change in the user's experience.

I know, we've discussed this a lot in paris. I'm not suggesting to harm
user experience, just to adjust the balance point so that when that makes
sense, applications (and thus the user's experience) do not suffer too
much from some limited initial choices.

I do expect a lot of savings from mux on the front side for small objects
for normal sites. But they're not the only ones using my product and users
pushing for larger numbers in all dimensions are becoming more common every
day. I want to pass the 100G barrier because there's increasing pressure
for it from people deploying now for the next few years, it's not just for
the fun of exhibiting large numbers (which is really enjoyable, I confess).
And I really don't want H2 to force a step back into the 10G era :-/

Best regards,
Willy

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 21:08:45 +0000

On 25 June 2014 22:26, [email-masked] wrote:
> I'll note that I work at YouTube, and care deeply about video distribution... and I'm not worried about it.
> As a matter of fact, I'd really, really like to take advantage of the prioritization scheme along with smaller
> frames so that I can have better codecs in the future which do much better adaptive stuff w.r.t. bandwidth
> changes.

And nothing about the jumbo frames proposal is stopping you.

> As it turns out, our users care deeply about the quality of their experience.

This is such a ridiculous snark I'm at a loss how to even respond.

---

From: <[email-masked]>
Date: Wed, 25 Jun 2014 21:13:04 +0000

> Assembling or removing the framing, yes.

Such acceleration is faster (or even not needed) for jumbo frames (=fewer frames).
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 14:21:42 -0700

On Wed, Jun 25, 2014 at 2:08 PM, <[email-masked]> wrote:

> prioritization scheme along with smaller

Assuming I stick lots of END_SEGMENTs in there, you might be right.
If I don't do that, however, a proxy might reframe into larger than 16k
frames and screw with the user experience.
Other servers are not likely to be as smart as our servers (as has been
demonstrated in the past), and so user experience with those may suffer.
If there are enough poor implementations, second order effects begin to
affect our usecases as they effect the browsers.

> This is such a ridiculous snark I'm at a loss how to even respond.

I'm sorry you interpreted it that way, but this consideration drives a
substantial portion of what I do.

Jumbo frames are unlikely to offer any additional quality of experience for
our users, and would not noticeably decrease CPU consumption, despite
supposedly being a motivating usecase.
As a result, jumbo frames are uninteresting for the YouTube usecase.

-=R

---

From: Roberto Peon <[email-masked]>
Date: Wed, 25 Jun 2014 14:26:57 -0700

But it does not remove the continuation 'kludge'. It substitutes one
'kludge' for another 'kludge' and doesn't change the orthogonal requirement
to handle arbitrarily large headers...

Continuation has the advantage of being a known annoyance.
There have been proposals for requiring that CONTINUATION (and similar
non-stream-initiating-HEADERS frames) carry only non-state-modifying data,
in which case many of the annoyances w.r.t. continuation go away.
I'd prefer investigation into avenues like that, personally.

Jumbo frames have the disadvantage of folks not realizing the harm they
cause, and then causing it anyway.
-=R

On Wed, Jun 25, 2014 at 2:04 PM, <[email-masked]> wrote:

> The sender can only apportion data into frames less than
> SETTINGS_MAX_FRAME length; and if you never change it, he has to use 16K or
> less.
>
> This is exactly the point. The jumbo frames eliminates the CONTINUATION
> "kludge" & allows use cases which need jumbo frames to have them if the
> peer gives consent.  All while not preventing efficient muxing with 16K
> frames.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 26 Jun 2014 00:01:17 +0200

I really don't understand why when it is pointed out that a significant
user base will have their user experience degraded without large data
frames, the response is: "go do that in an extension".

Yet when it is pointed out that a tiny fraction of traffic uses large
headers, the response is that we have to live with the known annoyance of
an acknowledged ugly kludge that has been only sparingly implemented/tested
and additional several DOS vectors.

Continuations are jumbo frames!  They are just really bad jumbo frames that
only apply to headers, can't be efficiently handled and don't have a
mechanism for end points to pre declare max acceptable sizes.   General
jumbo frames would handle the headers use-case, but also provide a solution
for those who need efficient large data.

h2 should either just support general jumbo frames for both headers and
data or no jumbo frames for either.  By supporting jumbo headers, but not
jumbo data, a double standard is being applied.

Anyway, enough said (by me) on this one.  If the chair really thinks that
the issues of large data have been given the same consideration as those
for large headers, then we have no issues to discuss here.

cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Jun 2014 22:09:41 +0000

In message <CAH_y2NGNCsKkhbmog-RXQ5PLefzjMRUkUOyee=[email-masked]>

>for those who need efficient large data.

My view exactly.

---

From: Johnny Graettinger <[email-masked]>
Date: Wed, 25 Jun 2014 18:49:08 -0400

> Clients send their requests over the same connection.

Just thinking about how I'd manage this: it seems that the load balancer
would need to have special knowledge as to whether a backend connection is
likely to send a large frame or not.

If they are, and you're not terminating SSL on either side (probably
uncommon), then you could issue read()s of just the frame header and
temporarily splice() the connections for payload-size bytes.
In all other cases though, you're better off issuing larger fixed-size
read()s, and aggregating frames to write() in user-space.

What this probably means is that you'll want to segregate your large-asset
servers from small-asset servers, and you'll want to inform your LB's pool
configuration as to which heuristic is appropriate.

But since you've now segregated your servers anyway, another option is to
have large-asset servers terminate SSL & HTTP/2, and configure a
large-asset pool in tcp-mode, distinguished by proxy IP or SNI.
This seems a whole lot simpler, allows the load balancer to splice(), and
works with SSL.

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 17:57:46 -0500

On Jun 25, 2014, at 3:22 PM, Roberto Peon <[email-masked]> wrote:

> The sender determines how to apportion data into frames. We were seeing whole files transmitted as a single frame.

Hmm I guess I just don’t see what’s so bad about this. If your peer negotiates a large frame size, you know not to bother muxing a lot of requests on it, and negotiate it down in another connection.

> And again, in the web use-case this is not going to be a measurable difference in utilization, since we're likely to be using TLS.
> -=R

I don’t want to resurrect the encryption discussions, but I will say that if I was running a common public download site I would be very tempted to send clients to a clear-text, simple and fast HTTP/1.1 connection.

Of course it would be nice if you didn’t have to switch protocols for common use cases.

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 26 Jun 2014 09:53:07 +1000

> for those who need efficient large data.

Data is a single unit, with simple properties for concatenation, so
it's trivial to split into multiple frames at one end and recombine at
the other.

The equivalent metadata unit is the header block, but that doesn't
have the same recombination properties; so while it can be split into
multiple frames, there are rules about streaming and interleaving.

Those rules result partly from the record-based nature of the
underlying headers, but mostly from the shared state. To eliminate the
shared state issue I see a couple of options:

1) allow multiple header blocks to be sent on one stream, and remove
continuations. Interleaving wouldn't be a problem, but it wouldn't be
possible to send large individual headers. More kluge required.

1.5) Only compress the HEADERS, make CONTINUATIONs stateless. Pros and
cons as above.

2) Extend HEADERS frame size and remove continuations. Removes the
ability for a very low level packet shuffler to shuffle packets
easily.

3) Extend frame size for all frame types. I understand where Roberto
is coming from on this now. It defeats the purpose of the protocol.

4) Revisit #1, but add a stream-local state, and only emit frame data
to the shared state when some boundary is reached. I think that's
fancy words for "buffer headers until END_HEADERS, then unhpack." This
appeals to me, but it defeats the commitment requirements that Martin
patiently explained to me a few months ago.

4.5) 1.5 with local state for continuations. This just occurred to me
now, so I'm not sure of the implications. OTOH two state sets
(essentially 'connection' and 'stream') is more moving parts requiring
more juggling, but probably not unreasonable. Likely to get resistance
because we're late in the game.

If we consider the status quo as option 5, its only downsides are:
END_STREAM is in the wrong place, and jerks who send ginormous headers
screw up their user experience. Also it's annoying to have to design
your low level packet shuffler, if you have one, to throw errors if
one packet has certain bit pattern (HEADERS-END_HEADERS) and the next
doesn't have another (CONTINUATION).

I think I'm pretty well decided, at least as far as my concerns are
being addressed.

> jumbo data, a double standard is being applied.

Yep. For jerks. And using it quite clearly marks them with a taint.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au

---

From: Gabriel Montenegro <[email-masked]>
Date: Thu, 26 Jun 2014 00:19:22 +0000

NYC interim discussions were very clear about not churning any more before WG LC unless something was completely broken.  We don't think that is the case here, especially as an extension would allow folks to experiment with jumbo frames for data if they so wish.

So I agree with Nicholas, Roberto, Mike on this one. 

> -----Original Message-----
> From: Poul-Henning Kamp [mailto:[email-masked]]
> Sent: Wednesday, June 25, 2014 15:10
> To: Greg Wilkins
> Cc: HTTP Working Group
> Subject: Re: #540: "jumbo" frames
> 
> In message <CAH_y2NGNCsKkhbmog-
> RXQ5PLefzjMRUkUOyee=[email-masked]>
> , Greg Wilkins writes:
> 
> >Continuations are jumbo frames!  They are just really bad jumbo frames
> >that only apply to headers, can't be efficiently handled and don't have a
> >jumbo frames would handle the headers use-case, but also provide a
> >solution for those who need efficient large data.
> 
> My view exactly.
> 
> --

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 17:28:47 -0700

> Doing more than 16 bits would take a lot more back-and-forth in the WG, and is likely to encounter a lot of resistance from implementers, from what I've seen.

Protocol failures of the class "bogged down in committee" arise from
these sorts of decisions too.  The overall protocol has co-evolved to
this point.  Changes without really strong justification, particularly
fundamental changes risk invalidating a lot of other decisions.

Sure, we might have arrived at what is only a local minimum, but
without stronger justification I'm really reluctant to act on this.
As far as it goes, Willy's numbers don't actually concern me that
much; parallelism goes a long way to addressing those sorts of
concerns.

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 21:14:42 -0500

> 1.5) Only compress the HEADERS, make CONTINUATIONs stateless. Pros and

This option could still support large headers in the uncompressed CONTINUATION frames.

I was just thinking perhaps HEADERS w/ CONTINUATION should require a total multi-frame length? 

One of the biggest problems with CONTINUATION is that a server has no idea how huge the headers will be, and is forced to buffer them until a limit is hit. If this information was known up front it could either RST_STREAM, or simply discard all subsequent CONTINUATION frames and reply with a too large status.

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 26 Jun 2014 12:18:16 +1000

Just to clarify...

> doesn't have another (CONTINUATION).
I mean to say: the current spec (i.e. treating the HEADERS and
CONTINUATIONs as a single contiguous block​) is option 5, and its downsides
are pretty minor in the grand scheme of things.

I'd be happy to consider my option 4.5 further, if it's not going to bog
everyone down in a circular discussion. That option is essentially:​ only
HEADERS is compressed (using the connection-wide HPACK state), and
subsequent CONTINUATIONs are designed such that individual headers can span
frame boundaries (i.e. introducing a per-stream state -- a buffer for a
partial header).  We buy back the HOL blocking issue (and protect ourselves
against 1-byte CONTINUATIONS, etc.), but we introduce a second state
commitment. OTOH we'd need at least one new setting, to advertise the size
of the buffer we're willing to hold (either as "X bytes per stream," or "Y
bytes shared across all streams.")

(Sorry for replying to myself.)

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 26 Jun 2014 04:22:44 +0200 (CEST)

On Thu, 26 Jun 2014, Gabriel Montenegro wrote:

> NYC interim discussions were very clear about not churning any more before 
> WG LC unless something was completely broken.  We don't think that is the 
> case here, especially as an extension would allow folks to experiment with 
> jumbo frames for data if they so wish.
>

As do I.

This discussion seems to mostly reiterate what we've already discussed 
numerous times. I think it is more important to get a protocol together that 
interops than to bikeshed forever on tiny details - and yes I think this is a 
rather tiny detail all things considered as I believe we all can live with 
what we have right now - even if it hurts a little bit more for some. And I 
think muxing and prio are two fundamental parts of the protocol that large 
frames don't really play nicely with.

Let me also remind everyone that at least two browsers only will speak HTTP/2 
over TLS and lots of the talk in this thread seem (to me) to assume plain-text 
implementations.

-- 

  / daniel.haxx.se

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 21:32:33 -0500

On Jun 25, 2014, at 9:22 PM, Daniel Stenberg <[email-masked]> wrote:

> This discussion seems to mostly reiterate what we've already discussed numerous times. I think it is more important to get a protocol together that interops than to bikeshed forever on tiny details - and yes I think this is a rather tiny detail all things considered as I believe we all can live with what we have right now - even if it hurts a little bit more for some. And I think muxing and prio are two fundamental parts of the protocol that large frames don't really play nicely with.
> 
> Let me also remind everyone that at least two browsers only will speak HTTP/2 over TLS and lots of the talk in this thread seem (to me) to assume plain-text implementations.

Also worth noting that overly restrictive browsers aren’t the sole user of HTTP.

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 26 Jun 2014 12:43:21 +1000

> I was just thinking perhaps HEADERS w/ CONTINUATION should require a total
How do you calculate that ahead of time? Even without HPACK it's a bit of a
chore; but if you have to mutate your compression context and buffer the
output so you can prefix it with a final length... well, in another world
that's why we have T-E:chunked. Too hard.

One of the biggest problems with CONTINUATION is that a server has no idea
> how huge the headers will be, and is forced to buffer them until a limit is
> hit. If this information was known up front it could either RST_STREAM, or
> simply discard all subsequent CONTINUATION frames and reply with a too
> large status.
>
>
I don't know if that's universally true. As an endpoint: if the
framing-layer machinery is buffering the headers in order to emit them as a
single blob, then yes; but the same machinery could stream the headers
piecemeal, no buffering required, and thus it wouldn't care how much header
data there is overall. The higher-level stuff (the application?) might
store the headers, but then it's that application's responsibility to tell
the sender to STFU.

I am not a proxy person. I imagine an aggregator would care more about
buffering; maybe that case really would benefit from the ability to fail
fast. Calculating the final size of (uncompressed) header data is easier if
less of said data is compressed, if that's worth anything.

Also, if it's not compressed it's easier to shout "STFU" on a stream
without having to either process the headers anyway (to keep your context
valid), or tear down the connection -- you don't need to know the final
length up front.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 25 Jun 2014 19:53:56 -0700

> One of the biggest problems with CONTINUATION is that a server has no idea how huge the headers will be, and is forced to buffer them until a limit is hit. If this information was known up front it…

This is a common thread here, but I haven't seen any way of limiting
headers that is consistently meaningful.

The obvious answer is to limit the size of the frames that contain
headers.  But in most cases that matter, the thing performing
processing is going to maintain an uncompressed copy of the headers.

So for all but a few implementations, the size that really matters is
the uncompressed size.  With Huffman coding, that's in the order of a
30% premium straight up, with significant variability.  Once you have
delta coding, the uncompressed size of a block of headers isn't quite
unbounded, but it is limited only by a multiplication factor relative
to the header table size.  More simply put, if you have a 4k header
table, you can have a magnification factor of up to 200000%.

Based on this, I concluded that it's foolish to try to limit header
block size at the framing level.  When a state exhaustion attack can
be mounted using only a handful of bytes, you are going to need
protection at another layer anyway, protection at the framing layer is
pretty redundant there.

That's not to say that the idea of prohibiting access to compression
state in CONTINUATION is a terrible idea, I just don't know that it
buys much.  Being able to multiplex CONTINUATIONS sounds attractive,
but that is really just optimizing for a case we really shouldn't be
encouraging at all.  Better in my mind to retain the cost and increase
the incentives for not doing stupid things with header fields.

---

From: Jason Greene <[email-masked]>
Date: Wed, 25 Jun 2014 22:39:57 -0500

> the incentives for not doing stupid things with header fields.

A very solid conclusion. Although I was definitely suggesting this as a possible extension of Matthew’s 1.5 (Mark’s earlier proposal). Although I admit the benefits are certainly minor so perhaps not worth the space. IMO the flow control possibility would be the biggest benefit. I actually think this discourages bad behavior, since bloated headers are throttled allowing good requests to still be processed.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 25 Jun 2014 23:50:48 -0400

On Wed, Jun 25, 2014 at 4:19 PM, Mike Belshe <[email-masked]> wrote:

>> This requires far more complexity for a hardware implementation, and
>> would reduce the chance that we get acceleration in HW for HTTP2.
>>
>>

+1

PS: Sorry for the lame, late, terse response. I am too busy for more for
now, and everyone else already wrote all the content I would have written,
although I may have phrased things differently. This is clearly a decision
about tradeoffs, and different people prioritize different things
(responsiveness vs throughput, etc) differently. My position remains
unchanged from what I've previously voiced.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 26 Jun 2014 07:11:50 +0200

Hi Johnny,

> temporarily splice() the connections for payload-size bytes.

That's the expected principle, except that splice() over 16kB is highly
inefficient.

> large-asset pool in tcp-mode, distinguished by proxy IP or SNI.

Not really because you still need to follow the request/response stream to
continue to route future requests appropriately based on the path for the
same host header.

> This seems a whole lot simpler, allows the load balancer to splice(), and

Sure but this is layer3-based routing, not l7-based as customers want today.
And HTTP/2 will make this even more important because by avoiding a new
connection to retrieve videos, we'll save one RTT and videos will start
faster.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 26 Jun 2014 07:53:21 +0200

> much; parallelism goes a long way to addressing those sorts of

Martin, while I can understand that such numbers are irrelevant to
your use case, and that you're not tempted by a last-minute change,
I'd like to mention that parallelism is orthogonal to this concern ;
parallelism is what currently makes it possible to reach close to
100G with HTTP/1.1 and if the same hardware goes back to 10 or 20G,
it's not parallelism that will bring the performance back, it's just
a definitely wasted performance by a design which does not scale as
well as the one it replaces.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 26 Jun 2014 06:05:15 +0000

In message <[email-masked]

>jumbo frames for data if they so wish.

I beg to differ:  CONTINUATION is horribly broken.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 26 Jun 2014 06:17:32 +0000

In message <[email-masked]>

>is coming from on this now. It defeats the purpose of the protocol.

In my proposal there would not be any difference for you until you eiter
send a SETTINGS saying you want to accept larger frames, or react to
the peers similar SETTINGS.

In other words:  If you don't like length extensions, nobody can 
force you to use them.

So what exact "purpose of the protocol" would my proposal defeat ?

Is there a secret purpose to specifically hurt websites with
very large objects and high traffic I havn't heard about ?  :-)

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 26 Jun 2014 16:52:55 +1000

On 26 June 2014 16:17, Poul-Henning Kamp <[email-masked]> wrote:

> In message <
> [email-masked]>
> , Matthew Kerwin writes:
>
I think the current term for those sorts of optional features is
"extensions." If you drafted a good one, I wouldn't oppose its adoption by
the wg.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 26 Jun 2014 07:00:54 +0000

In message <CACweHNBGP2sd069AsoZoxL9pbm=f_s7q4MqKqGCrmpEcxFhL=[email-masked]>

>"extensions." If you drafted a good one, I wouldn't oppose its adoption by

Getting rid of the CONTINUATION kludge is not an extension, that's a vast
improvement and reduction of complexity in the proposed protocol.

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 26 Jun 2014 17:54:20 +1000

> improvement and reduction of complexity in the proposed protocol.
Potayto, potahto. If it's a feature or functionality that is advertised in
settings and defaults to "off", it goes in an extension. Even if it's
absolutely critical to getting a *nicely* working protocol; even if, in the
end, everyone implements it (remember when 'Connection:keep-alive' was
non-standard?)

I think the initial release of HTTP2 doesn't have to be perfect, and in
fact​, thanks to the extensibility, I don't think it has to even be that
good. It just needs to provide a framework to allow it to evolve and
improve with time and experience.

Does it really matter which RFC contains the text, so long as it's out
there in the end? Is it worth blocking the publication of the
minimal/suboptimal/lame/whatever-you-want-to-call-it HTTP2, when there
clear are avenues for extensibility and improvement? My ample gut tells me
there won't be too much silicon wasted burning hard copies of the spec in
the near future, people will surely wait to see how it goes before
committing.

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 26 Jun 2014 20:11:39 +1200

On 26/06/2014 2:43 p.m., Matthew Kerwin wrote:
> On 26 June 2014 12:14, Jason Greene wrote:
> 
>>

T-E:chunked is DATA. That works just fine.

The unknown-length CONTINUATION is headers and is perpetuating the
existing HTTP/1.x problems with infinite header length.

IMO fixing that is important if we end up with CONTINUATION.

Since as others have put it "HEADERS+CONTINUATION should be handled and
sent as a unit" the full size should therefore be available as a group
to the generating client/server. A sender which does not know the size
before starting to emit the bytes is probably sending far too large a
frame sequence and the pain is yet another incentive on them to be more
reasonable.

> store the headers, but then it's that application's responsibility to tell

IIRC the use-cases for which CONTINUATION/jumbo is required are large
sets of Cookies and large Kerberos authentication tokens. In both these
cases the size is known enough in advance not to cause the server problems.

> less of said data is compressed, if that's worth anything.

Aggregator cares primarily about speed (RTT + latency + CPU cycles),
*some* extra buffer can be coped with provided the transaction is
completed fast enough to release it again before too much other traffic
is impacted. Secondly is the buffering since it is limited after all and
too many large transactions makes for either DoS or slower transactions
in a nasty feedback loop.

The middleware complaints against HPACK if you notice old discussions
have largely been about speed of the mandatory re-compress activity
expending time and CPU.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 26 Jun 2014 20:20:59 +1200

On 26/06/2014 8:25 a.m., Martin Thomson wrote:
> On 25 June 2014 13:07, Willy Tarreau wrote:

They already conceded to having infinite-sized HEADERS+CONTINUATION
jumbo framesets thrust at them.

AIUI the offer is to replace that "infinite" with a fixed and negotiated
large size. Possibly (but not necessarily) getting larger DATA from some
is a cost of getting rid of the "infinite" headers.

Amos

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 26 Jun 2014 09:52:42 +0000

In message <[email-masked]>
, Matthew Kerwin writes:

>fact=E2=80=8B, thanks to the extensibility, I don't think it has to even be
>that good. [...]

>there in the end? [...]

You're trolling now, right ?

No, HTTP2 does not need to be and certainly isn't "perfect", but it should
also not be so horribly misfigured from birth that people will prefer
HTTP/1 over it just because of that.

And yes, it does matter what RFC's contain what text, in particular for
the first one, since that's the one everybody is going to stare at
to decide if HTTP/2 is something they should bother about.

CONTINUATION in the present draft would become my goto-example for
bad protocol design and layering violations if I taught classes on
data communications.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 26 Jun 2014 09:33:46 -0700

On 26 June 2014 01:20, Amos Jeffries <[email-masked]> wrote:
> large size.

That's not quite correct.  See my post here:
http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1715.html

---

From: Johnny Graettinger <[email-masked]>
Date: Thu, 26 Jun 2014 13:25:55 -0400

> Sure but this is layer3-based routing, not l7-based as customers want

Well, technically SNI is L5, right?

I can't speak for others, but know that in my own experience and having
deployed HAProxy at pretty significant scale in the past (and thank you,
for a great product), we opted for L3 routing anyway.
This was just because it provided service isolation and better matched the
organizational structure of the teams maintaining different application
types. Performance wasn't a deep consideration.

I'm not dismissive of the assertion that customers want L7 routing of mixed
service types with wildly different response properties and performance
requirements, but I personally haven't seen it. And I'd probably try to
talk them out of it :)

> connection to retrieve videos, we'll save one RTT and videos will start

This can be overcome (preconnect / prefetch hints, etc).

---

From: Jason Greene <[email-masked]>
Date: Thu, 26 Jun 2014 12:40:58 -0500

On Jun 18, 2014, at 11:49 PM, Mark Nottingham <[email-masked]> wrote:

> At any rate, a server (origin or proxy) is perfectly within its rights to 431 Request Header Fields Too Large any request that has a CONTINUATION, and then reset the stream. They'll learn pretty soon…
> 

Since it seems likely that the jumbo frames are going to be sidelined to an extension, I really think this proposal needs a second look. It has a lot of really nice benefits including:

1. Discouraging CONTINUATIONS (slightly harder to create and they take more space)
2. Better multiplexing (other streams are not penalized by a “rude” stream)
3. Ability to RST_STREAM vs GOAWAY + close
4. Complexity of processing a standalone HEADERS frame (the desired behavior) is reduced
5. All of the benefits of the existing design are preserved, but with a cleaner solution

---

From: Jason Greene <[email-masked]>
Date: Thu, 26 Jun 2014 12:48:47 -0500

> 4. Complexity of processing a standalone HEADERS frame (the desired behavior) is reduced

^ Actually I misstated this, its slightly reduced when you are processing a HEADERS frame with following CONTINUATIONS, but this benefit is very minor so I probably shouldn’t have even mentioned it.

---

From: <[email-masked]>
Date: Thu, 26 Jun 2014 22:31:49 +0000

On 26 June 2014 19:40, [email-masked] wrote:
> Since it seems likely that the jumbo frames are going to be sidelined to an extension,
> I really think this proposal needs a second look. It has a lot of really nice benefits including:
>

Why would you still need CONTINUATION frames?  Can't you just use regular HEADERS frames without compression?

---

From: Jason Greene <[email-masked]>
Date: Thu, 26 Jun 2014 21:41:43 -0500

> Why would you still need CONTINUATION frames?  Can't you just use regular HEADERS frames without compression?

Mainly to allow them to be flow controlled, since thats based on frame type.