Subject: Framing and control-frame continuations

From: Roberto Peon <[email-masked]>
Date: Tue, 5 Feb 2013 14:28:28 -0800

The proposal for framing is that there are always 8 bytes to read for every
frame, both control and data frames. (Thanks to Jeff Pinner for modifying
this for me already!)

Generically, this is the format of a frame.

  0        1        2        3        4         5        6        7
  +--------+--------+--------+--------+--------+--------+--------+--------+
  | Length(16)      |Type(8) |Flags(8)| Num-of-Entries-or-Stream-ID-or-ID | ->
  +--------+--------+--------+--------+--------+--------+--------+--------+

  8..N
  +========+
  |  Data  |
  +========+

   Length: An unsigned 16-bit value representing the number of bytes of
   the data field.

   Type: The frame type.

   Flags: Flags related to this frame.  Flag definitions are dependent
   upon the frame type.

   Data: data associated with this control frame.  The format of this
   data is controlled by the frame type.

For a data frame, TYPE is set to zero, and the
Num-of-Entries-or-Stream-ID-or-ID field (gotta get a better name :) )
contains a '0' followed by 31 bits of stream ID. Valid flags are:

      0x01 = FLAG_FIN - signifies that this frame represents the last
      frame to be transmitted on this stream.  See Stream Close
      (Section 2.3.7) below.

      0x02 = MSG_DONE - signifies that this frame represents the last
      frame of a message.  This is relevant for layering of message-
      based protocols on top of SPDY.

For control frames, a non-zero value will exist in the TYPE field.
The following flags are valid for all control frames:

      0x01 = FLAG_FIN - signifies that this frame represents the last
      frame to be transmitted on this stream.

      0x02 = MSG_DONE - signifies that this frame represents the last
      frame of a sequence of a run of same-type control frames.

If MSG_DONE is not set for a control-frame, then the particular
control message was unable to fit within a single control-frame.
Implementations may process the control message up to the amount
received, but MUST not treat the message as done. Implementations
SHOULD finish sending the control-message as soon as possible and when
finished with the particular control message, they MUST set MSG_DONE.

What do people think about the FLAGS/TYPE arrangement? If swapped, we
gain more contiguous reserved bits.
-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 08:51:27 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <CAP+FsNemzHOH1YrDXbEqy_dMB8h=[email-masked]>

I don't think this is good enough.

You'd need ~20k of these frames a second to fill a 10GB ethernet,
and a very large fraction of present day web-objects would require
more than one frame already.

We should take a clue from TCP's history of window-scaling and not
fence ourselves into 64kbytes for all time, nor should we make the
"X is enough for Y" mistake with a fixed horizon extension mechanism.

I suggest: 

  0               1               2
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  | Mantissa(14)              | E |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

	Length = Mantissa * 2^(8*E)

(I don't particularly care if we put the E field in the left or right side.)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 08:58:14 +0000

>"X is enough for Y" mistake with a fixed horizon extension mechanism.

Hmm, maybe we should simply make the length field 32 bits, that would
be easier for everybody, and still not increase overhead too much.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 6 Feb 2013 21:21:06 +1100

> more than one frame already.

Could you please spell out why you think that's a problem?

--
Mark Nottingham   http://www.mnot.net/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 11:40:25 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <[email-masked]>, Mark Nottingham wri

>Could you please spell out why you think that's a problem?

As I said before:  If we cannot demonstrate HTTP/2 on a 10GE
media and argue plausibly that it can be handled on 1Tbit/sec,
we're writing a standard for the past and not the future.

Matt Mathis has been pushing similar agenda for IP MTU for years now:

	http://staff.psc.edu/mathis/MTU/

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 07 Feb 2013 00:47:08 +1300

On 6/02/2013 11:28 a.m., Roberto Peon wrote:
> The proposal for framing is that there are always 8 bytes to read for 
> every frame, both control and data frames. (Thanks to Jeff Pinner for 
> modifying this for me already!)
>
> For a data frame, TYPE is set to zero, and the Num-of-Entries-or-Stream-ID-or-ID field (gotta get a better name :) ) contains a '0' followed by 31 bits of stream ID. Valid flags are:
> For control frames, a non-zero value will exist in the TYPE field.  The following flags are valid for all control frames:
>
> If MSG_DONE is not set for a control-frame, then the particular control message was unable to fit within a single control-frame. Implementations may process the control message up to the amount received, but MUST not treat the message as done. Implementations SHOULD finish sending the control-message as soon as possible and when finished with the particular control message, they MUST set MSG_DONE.
>
> What do people think about the FLAGS/TYPE arrangement? If swapped, we gain more contiguous reserved bits.
> -=R

The SPDY framing is a bit clumsy, in particular its way of transmitting 
flow state as individual frames. Below is an alternative basic frame 
layout I have been working on to extend the original network-friendly 
draft frames using the SPDY and WebSockets signal types.

Using the below frame layout, in particular with two flags up front in 
specific positions we are able to transmit HTTP/2 and WebSockets frames 
on the same connection. With some small amount of magic only necessary 
during the transition period we are also able to identify SPDY v2, SPDY 
v3 and HTTP/1.x connection attempts. I am talking algorithm magic here 
*not* bits dedicated to a BOM. see "Relevant Magic" below.  I've been 
holding this off while I try to figure out what bit ranges the TLS 
handshakes are detectible with. It seems 32-bits is required if we merge 
TLS port 443 traffic into this magic, but I'm not yet completely certain 
of that.

NP: I use "flow" here to mean a series of frames which form 1..N 
requests and 1..N responses in a session from one client. A client may 
send multiple flows. For example; a proxy with many clients, browser 
with many tabs, spider doing parallel domain fetches, web service doing 
parallel fetches to a remote backend, etc. etc.

The Frame format:

         0                   1
         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
        +-+-+-----------+---------------+
        |F|C|  type     |               |
        +-+-+-----------+               +
        |        Frame Length (24)      |
        +-------------------------------+
        |       opaque ID (16)          |
        +-------------------------------+
        |     Frame Data (16...N)       |
        +-------------------------------+

  flags =  2-bits : frame flags shared by all frames with the following 
meaning:

   F = final frame. There are no more frames with this request ID 
following. This signals termination of the flow from this sender.
   C = continuation. There are more frames expected to follow in this flow.

* these are separate bits because F relates to the flow/session. C 
relates to the request, response, or entity being transferred.
   A flag combination of (F=0,C=0) signals the flow may continue but the 
object this frame relates to is now completed
   A flag combination of (F=0,C=1) signals the flow will continue with 
more frames for this particular object (request response, or entity)
   A flag combination of (F=1,C=0) signals the flow is terminating and 
the object being transferred is complete.
   A flag combination of (F=1,C=1) signals the flow is terminating and 
the object being transferred has been truncated.

  type =  6-bits : Frame Type code

* this frame layout intentionally overlaps with WebSockets in the first 
octet. Frame type codes 0x0, 0x1, 0x2, 0x8, 0x9, 0xA correlate directly 
to the WebSockets 'opcodes' of those values. We should probably reserve 
those frame types for WebSockets since from this octet onwards the WS 
frame differs markedly in field positioning. Doing this overlap allows 
us to multiplex HTTP/2 frames with WebSockets frames on the one 
connection without having to go through the Upgrade: handshake required 
with HTTP/1 protocol.

frame-length: 24-bit size numeric indicating how many octets the frame 
takes up. This size should include the headers 8 octets.

  opaque ID : 16-bit opaque identifier set by the sender to group a 
sequence of frames into a flow. The receiver must send it back on frames 
generated for the response or control frames related to that flow, or on 
frames of related responses when performing server-push.
    The request-ID of 0 is reserved for frames dealing with connection 
state, not with any particular request-response flow.

** Minimum size of Frame Data should be 16-bits, making frames 64-bit 
minimum with 32-bit alignment.

Relevant Magic:

* in HTTP/1 protocol the first 32-bits of a connection are ASCII characters.
   - The F flag is always 0 on the first frame of valid HTTP/1 traffic.
   - The C flag and type field will be random values.
   - the opaque ID field will never be 0x0.

* in SPDY protocol the first 32-bits of the connection is a control frame.
  - the F flag is set to 1.
  - the C flag is set to 0 (since the SPDY version field is 0x3)
  - the type field is 0x0 (since the SPDY version is 0x3)
  - the first octet of the frame length will be 0x1, 0x2 or 0x3 
corresponding to the SPDY version number.

* in WebSockets protocol the first 32-bits of a connection is either a 
HTTP/1 Upgrade request (to be handled as per HTTP/1) or a frame
  - the F flag may be set or not. with identical meaning to HTTP/2
  - the C bit is set to 0
  - the type field is set to a WebSockets opcode.

  * I propose in HTTP/2 the first frame be a connection feature 
negotiation frame...
  - have the F flag to 0. Differentiating it from HTTP/1 and signalling 
any WebSockets recipient to close the connection.
  - have the C flag set to 1. Differentiating it from WebSockets and 
SPDY. Also, signalling any WebSockets recipient to close the connection.
  - have the type field of 0x0. Differentiating it from HTTP/1 and 
signalling any HTTP/1 recipient with the '@ symbol as request method.
  - have a frame size limit of . Causing the \0 to be delivered to 
HTTP/1 recipients, such that their whole method string is '@'.
  - have the opaque ID of 0x0. Indicating a connection-level frame. And 
signalling any HTTP/1 recipient to abort the transaction.
  - the first 16 bits of the frame data be the version BOM.

This frame structure gives us several very important things:

* first 32-bits up front for any frame provide the magic needed to 
identify how it affects the flow, how big it is, and how to process it

* second 16-bits provide the opaque ID to group frames together for flow 
control and flow state management.

* clients and servers (including intermediaries) can maintain sessions 
and state using TCP details plus that opaque ID (IP:port:opaque-ID) as a 
unique identifier for any senders session.
For the vast majority of frames these fields are the only ones any 
intermediary software needs to actually care about for routing purposes. 
The flow opening frames are the exception here, where they are required 
processing to identify exactly what state to assign to the flow using 
that opaque ID. Possibly also some control frames to manage the flow 
control negotiation between machines - although most of these are likely 
to come under ID 0x0.

* intermediaries should be left free to re-write (or not) the opaque ID 
field as necessary to prevent collisions. This means the limit SPDY 
places on the ID counter in section 32K sessions before a connection 
MUST close is removed and any one connection is free to re-use identifiers

* with a 3-way separation of object termination (C flag) from flow 
termination (F flag) from TCP connection termination using no 
intermediary frames we are free to permit multiple requests and 
responses within one flow without additional overheads.
  + permitting simple server-push where one request fetch can trigger 
multiple responses using the same opaque ID.
  + permitting a whole browsing session to re-use the one opaque ID in 
loose equivalence to a sesison cookie.

Amos

---

From: Martin Thomson <[email-masked]>
Date: Wed, 6 Feb 2013 07:07:12 -0500

I might just be dense, but I'm not seeing the connection.  I can see
how path MTU affects network performance at the lowest layers, but I'm
not seeing how that affects application layer protocols.  How is one
big frame different from multiple smaller frames at this layer?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 12:17:25 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <[email-masked]>

>big frame different from multiple smaller frames at this layer?

Think "incoming load-balancer at the end of 1Tbit/s fiber" ?

---

From: Roland Zink <[email-masked]>
Date: Wed, 06 Feb 2013 13:46:25 +0100

The framing overhead can be saved when combining  several frames into 
one. This seems to be 8 Bytes per 64k (full frame). The processing 
overhead per frame can also be saved. Another question is how the lower 
level get a packet with a big payload if all what is delivered is in a 
frame. Nagle's algorithm to fill the MTU can cause delays.

Adding 2 bytes to the frame header for a 32bit length (instead of 16bit) 
does add some overhead for smaller frames but is more flexible. Using a 
maximum frame size of 64k may reduce the memory management cost over 
arbitrary values, but the maximum frame size can be negotiated, couldn't it?

The window size needs to be larger than 64k to fully utilize such links.

Roland

---

From: Roberto Peon <[email-masked]>
Date: Wed, 6 Feb 2013 05:40:19 -0800

At 64k, the framing overhead is 0.00012, which is pretty close to
negligible.
Arguably, if we're worried about payload efficiency at this size, then we
should be far more worried about compression efficiency, as it will have an
orders-of-magnitude greater effect on efficiency than changing the length
field here.

The idea of a 16 bit field is that it is small enough to require
implementors to get the continuation case correct because it is will occur
infrequently, as opposed to larger values where implementations are likely
to skimp and/or suffer from backup-generator style problems where the
codepath is exercised so rarely that bugs in it are not found and fixed.
Additionally, with a 16bit field a naive implementation is more likely not
to completely screw things up w.r.t. allowing interleaving in response to
new requests or streams.

In any case, if/when we implement and there are real performance
bottlenecks, we can rev the protocol-- indeed, we'll be doing this many
times as we develop HTTP/2, hopefully, as the version negotiation
mechanisms are (imho, at least) a significant work-product.
It is certainly possible that, after some implementation experience with 16
bit frame sizes we all decide that it needs an upward rev, with some
features designed to ameliorate the particular problem at hand

-=R

On Wed, Feb 6, 2013 at 4:46 AM, Roland Zink <[email-masked]> wrote:

> The framing overhead can be saved when combining  several frames into one.
> This seems to be 8 Bytes per 64k (full frame). The processing overhead per
> frame can also be saved. Another question is how the lower level get a
> packet with a big payload if all what is delivered is in a frame. Nagle's
> algorithm to fill the MTU can cause delays.
>
>>> In message <[email-masked]<[email-masked]>>,
>>> Mark Nottingham wri
>>> tes:
>>>
>>>          http://staff.psc.edu/mathis/**MTU/<http://staff.psc.edu/mathis/MTU/>
>>>
>>> --

---

From: Roberto Peon <[email-masked]>
Date: Wed, 6 Feb 2013 06:06:11 -0800

I know from past experience that one can fill up a 10GE pipe with very
small SPDY frames (1XXmillion 1 byte payloads)  when done correctly on
today's hardware, without using up all cores (though, to be fair it comes
close). This is pretty much a worst-case scenario.
I've found that the amount of cores one uses to output a certain amount of
bandwidth is far more dependent on the the NIC kernel drivers and NIC
hardware than anything else. Formatting things in memory *especially* data
frames is extremely cheap given that it requires no transformation.

I currently can't imagine what a 1T network architecture will look like
when going into a single machine.  I don't know if it'd go through the
kernel, or just DMA somewhere, or how programmable the DMA controllers will
be, etc.
I do know, though, that I could cram ~over 100M SPDY frames over the wire
on a 10GE interface on today's hardware, so I'm not worried about shoving
16M 64k frames over the wire in terms of overhead.
I do also know is that a terrabyte could be sent ~16M 64k frames. Even
assuming we do terribly and only stuff the frames with 16k, we'd only have
a piddly 67M frames per second to deal with. That is already lower than
what we can accomplish today.

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 14:19:13 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <[email-masked]>

>close). This is pretty much a worst-case scenario.

And where is the 10-20 years head-room in that ?

Remember:  Cores are unlikely to become faster, you'll just get more
and more of them.  Media on the other hand, seems to keep getting faster.

>hardware than anything else.

And you can take it from me, with my kernel-developer hat on, that
it is an issue which is being worked on aggressively, so HTTP/2
would be stupid to assume that the kernel guys don't improve this
aspect.

>when going into a single machine.

What you can or cannot imagine is not really a relevant yard-stick.

And whats more, some of use don't need to imagine, I know people
who _have_ (almost working) 1Tbit interfaces into a single machine
now.

There is absolutely no sane reason to limit HTTP/2 frames to a puny
64Kbyte at this time and date:  PDP-11 compatibility is not really
an issue any more.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 14:28:46 +0000

>The window size needs to be larger than 64k to fully utilize such links.

Just got off the phone, it was the 1980'ies which wanted their
protocol design parameters back, will you handle that ?  :-)

There is no need to negotiate a maximum frame size, because the
length comes up front, so the receiver can always chop it up to
more manageable bits if it cannot process it in the full size.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 14:30:57 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <[email-masked]>
, Roberto Peon writes:

>bottlenecks, we can rev the protocol [...]

Yeah, and 640k is enough for everybody, trust us, we've tested this.

Sorry for getting a bit sarcastic here, but I am frankly flabbergast
here that anybody can even propose a <64kByte framesize for a protocol
which will not emerge from standardization for another five years
and which is supposed to last at least 10 years after that.

Just make it 32 bit length, and move on, ok ?

---

From: Roland Zink <[email-masked]>
Date: Wed, 06 Feb 2013 15:39:02 +0100

> Just make it 32 bit length, and move on, ok ? 
+1

---

From: Brian Pane <[email-masked]>
Date: Wed, 6 Feb 2013 07:13:54 -0800

> more manageable bits if it cannot process it in the full size.

The "chop it up to more manageable bits" approach works for data
frames, where a load balancer receiving a large frame can chop it up,
thereby maintaining a limit on per-connection memory usage.  But it
doesn't work for control frames; there already are use cases where a
load balancer will need to see the entirety of a control frame in
order to decide where/whether to route it.

Brian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 15:24:28 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <CAAbTgTtC1sZD5fasfB_pLTZz=[email-masked]>
, Brian Pane writes:

>doesn't work for control frames;

And just how big do you forsee an controlframe being able to get ?

---

From: Roland Zink <[email-masked]>
Date: Wed, 06 Feb 2013 16:44:22 +0100

> And just how big do you forsee an controlframe being able to get ? 
If you want to attack a server this can be any size. For example I have 
seen a 90k URL.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 16:51:30 +0000

>If you want to attack a server this can be any size. For example I have 

And being given the length up front, what is the problem for the
server in detecting the DoS and handling it appropriately ?

---

From: Roberto Peon <[email-masked]>
Date: Wed, 6 Feb 2013 09:05:57 -0800

It strikes me that the 72X error codes really suit this conversation well :)
-=R

---

From: Ilari Liusvaara <[email-masked]>
Date: Wed, 6 Feb 2013 19:06:24 +0200

On Thu, Feb 07, 2013 at 12:47:08AM +1300, Amos Jeffries wrote:

> Magic" below.  I've been holding this off while I try to figure out
> what bit ranges the TLS handshakes are detectible with. It seems
> 32-bits is required if we merge TLS port 443 traffic into this
> magic, but I'm not yet completely certain of that.

AFAIK, the first bytes from client in current TLS connections are:

0x16 (Handshake packet)
0x03 (SSLv3 or TLS v1.x)
0x00-0x03 (At least until TLSv1.3 appears, that would use 0x04).

Then there's the SSLv2 compatiblity handshake. Hope nothing uses
that anymore.

-Ilari

---

From: Brian Pane <[email-masked]>
Date: Wed, 6 Feb 2013 09:07:03 -0800

> And just how big do you forsee an controlframe being able to get ?

I believe the proposal was "Just make it 32 bit length."

---

From: James M Snell <[email-masked]>
Date: Wed, 6 Feb 2013 11:04:39 -0800

A reasonable compromise here would seem to be: let's place limits on
control frame size but allow data frames to be variable with 32-bit
lengths. Done properly, control frames should never be too large. Doing so
ought to encourage developers to keep control-frame overhead at a minimum
while allowing optimization of data flow (honestly, I think allowing 32-bit
length control frames is just asking for trouble by encouraging bad
behavior). This would mean moving type and flags before length.

On Wed, Feb 6, 2013 at 6:30 AM, Poul-Henning Kamp <[email-masked]>wrote:

> In message <
> [email-masked]>
> , Roberto Peon writes:
>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 06 Feb 2013 20:11:06 +0100

>to completely screw things up w.r.t. allowing interleaving in response to

(I rather agree with the reasoning here, but 16 bit is a bit small...)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 19:23:48 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <CABP7Rbdv-zuAg3=O6oHRqJY0=[email-masked]>
, James M Snell writes:

>flow (honestly, I think allowing 32-bit
>behavior).

The important thing is that you have to announce your bad behaviour
up front, that makes it very easy to deal with it on the receiving end.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 6 Feb 2013 12:14:28 -0800

Unfortunately, by the time the reciever sees the frame the damage is likely
already done.  As a result of sending a large frame, the session may back
up and prevent other data from being sent on other streams either
successfully, or without undue delay.

Essentially, the receiver not being DoS'd is the smaller of the problems
here-- the sender causing random, unnecessary HoL blocking/contention and
not effectively being able to switch to using the TCP channel for higher
importance data is the bigger issue, coupled with the likelihood of
exercising all of the code regularly (and if you don't think that is an
issue, try using the not-regularly-sent parts of HTTP/1.0 and see if it is
successful on many implementations...)

-=R

On Wed, Feb 6, 2013 at 11:23 AM, Poul-Henning Kamp <[email-masked]>wrote:

> In message <CABP7Rbdv-zuAg3=O6oHRqJY0=
> [email-masked]>
> , James M Snell writes:
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 20:21:16 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <CAP+FsNdvAwThsTfp1FATETABT=fVP0pPOYkei2hWQh8Ys=[email-masked]>

>successfully, or without undue delay.

And who gets a problem ?

If a browser does this, its user exeperience suckage.

If a webservers does this, the webserver will appear sucky to people
visiting it.

What's not to like about that ?

---

From: Roberto Peon <[email-masked]>
Date: Wed, 6 Feb 2013 12:36:19 -0800

Why would I like it if the new and supposedly better stuff is worse with
naive implementations, given that a requirement for a smaller frame size
would likely do a good job of preventing the sucking in the first place? :)

This problem isn't theoretical, by the way-- we've seen it in the wild. It
is something which, unless one stops to think, one won't realize why it
would cause a problem, but that doesn't stop it from doing so.

-=R

---

From: James M Snell <[email-masked]>
Date: Wed, 6 Feb 2013 12:51:51 -0800

As opposed to endlessly debating the matter.. any chance we can see some
real metrics on all this? Specifically, it would be very informative to see
a breakdown of exactly what impact small, large and variable-sized frames
have on latency and throughput on both ends of the connection. It would
also be helpful to see what kind of processing overhead could be expected
if we allow for large control frames... for example, if we do end up going
with delta compression for headers in control frames, what would be the
impact of allowing that frames to scale up to 32-bit lengths?

(I must say, however, that I have not seen anything that would ever suggest
that we need or want control frames to scale up to 32-bit lengths. To
paraphrase a comment Mark made previously, I don't want to live in a word
with that much protocol and processing overhead.)

- James

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 06 Feb 2013 20:55:05 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <CAP+FsNdm86+Ti4iJmDy=cKXxc0uvX5KVN0KSUHu=[email-masked]>

>would likely do a good job of preventing the sucking in the first place? :)

So you're actively pushing a very complex protocol, and now you're
suddenly worried about "naive implementations" running into trouble ?

Doesn't sound very convincing to me...

Anyway, I've spent enough time you this non-sense.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 7 Feb 2013 09:41:40 +1100

PHK,

Calling the discussion "non-sense" and arguing by repeated assertion isn't going to change people's minds. Illustrating why and how adding an overhead of 8 bytes per 64k of data -- as opposed to 4M -- is too much overhead might. 

Numbers, please, not handwaving.

Thanks,

---

From: James M Snell <[email-masked]>
Date: Wed, 6 Feb 2013 14:46:28 -0800

On Wed, Feb 6, 2013 at 2:41 PM, Mark Nottingham <[email-masked]> wrote:
[snip]
> Numbers, please, not handwaving.

+1 ... many times over.

- James

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 06 Feb 2013 23:26:24 +0000

I think the issue in the end is how do we answer this question:  Will 
64k max frame size be enough / appropriate in 20 years time.

It's extremely difficult to fathom where we may be or what a computer or 
network looks like in that time frame.

64k seems too small.  4GB looks to have issues with interleaving and HOL 
blocking.  I can see some simple server authors thinking "cool I can 
send this 4GB file in one frame".  Which may be cool or not depending on 
what else is going on (e.g. may choke a down-stream proxy trying to 
interleave multiple responses - although I guess it could just repackage 
it).

I'm not really a fan of using 24 bits, but it would save 1B/frame.

In the end, we could do something like.

1. make it 32 bits
2. place minimum size limits on size of a control frame that must be 
accepted (e.g. something much smaller).  So implementations can only 
rely on that size being accepted for a control frame.
3. put stern words in about blocking and concurrency when it comes to 
choosing a frame size when sending data.

in the end, TCP has to do a bunch of work underneath to get this over 
1460 bytes of payload anyway.  Until the basic ethernet frame size 
grows, and is reliable across the internet (I'm thinking this will be a 
very long time away) we're going to be constrained by TCP window, and 
ack latency anyway.

Adrien

------ Original Message ------
From: "James M Snell" <[email-masked]>
To: "Mark Nottingham" <[email-masked]>
Cc: "Poul-Henning Kamp" <[email-masked]>; "HTTP Working Group" 
<[email-masked]>
Sent: 7/02/2013 11:46:28 a.m.
Subject: Re: Framing and control-frame continuations

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 07 Feb 2013 00:44:26 +0100

>interleave multiple responses - although I guess it could just repackage 

I do recall that over in hybi (Websocket) being able to send huge frames
via sendfile-style APIs was _extremely_ important to some participants.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 7 Feb 2013 10:50:44 +1100

> via sendfile-style APIs was _extremely_ important to some participants.

Yes, that was discussed at the F2F:
  https://github.com/http2/tmp_minutes/commit/81e8644a6826f41d7f17da88c3706d3ce4300c28#L0R899

Cheers,

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 07 Feb 2013 13:36:23 +1300

On 7/02/2013 8:04 a.m., James M Snell wrote:
> lengths. Done properly, control frames should never be too large. 
> Doing so ought to encourage developers to keep control-frame overhead 
> at a minimum while allowing optimization of data flow (honestly, I 
> think allowing 32-bit length control frames is just asking for trouble 
> by encouraging bad behavior). This would mean moving type and flags 
> before length.

See my alternative framing proposal mailed earlier.

We picked 24-bit length as an intermediary size between 16 and 32 bits 
which allows MB sized frames and also allows some frames to have the top 
8 bits defined as mandatory 0's for frames which need a smaller size 
limit. OR, the top 8 bits can be defined as a partial base when doing a 
compound mantissa. There is a vast size range that 24 bits can be used 
for. Same can be said for an aligned 16-bits, but the 'odd' 24-bits size 
lends itself better to use in complex length encodings if we want one (I 
don't think we do though).

Amos

>
>
> On Wed, Feb 6, 2013 at 6:30 AM, Poul-Henning Kamp <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     In message
>     <[email-masked] <mailto:[email-masked]>>
>     , Roberto Peon writes:
>

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 07 Feb 2013 14:21:21 +1300

> Then there's the SSLv2 compatiblity handshake. Hope nothing uses

Thank you. I was loosing hope of finding these in the TLS RFC's.

So the relevant magic for TLS is F=0,C=0, type=0x16, top byte of the 
length field being non-0x0. Meaning we can distinguish it easily from 
WebSockets and HTTP/2 client. Good news there.

Amos

---

From: Ilari Liusvaara <[email-masked]>
Date: Thu, 7 Feb 2013 04:23:44 +0200

> Thank you. I was loosing hope of finding these in the TLS RFC's.

(That was just from memory, but I verified it by dumping a TLS
ClientHello packet).
 
> length field being non-0x0. Meaning we can distinguish it easily
> from WebSockets and HTTP/2 client. Good news there.

Note:

Assuming the following:
- Implicit upgrade to TLS mid-connection is not needed. And
- There is overall connection magic (to break HTTP/1.x).

Then:
The TLS test occurs against that overall connection magic,
not against the beginning of the frame structure. 

And given that magic is thought to have the high bit of the
first byte set (the exact form needs more testing to determine),
telling it apart from TLS, which always starts with 0x16.

Now, to Websocket: I don't think websocket first byte values
are very constrained. Some extensions assign a meaning for
the RSV bits in the first byte or for other opcodes (or does
the latter require a new version of Websockets?).
Additionally there is a FIN bit there too...

-Ilari

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 07 Feb 2013 15:50:12 +1300

> - There is overall connection magic (to break HTTP/1.x).

Yes. I assume the same. This magic is the first octets of a connection 
and cannot be used elsewhere safely.
Once some other protocol has been started we must use that protocols 
defined Upgrade mechanism to change to HTTP/2 explicitly.

> the latter require a new version of Websockets?).

My understanding of WebSockets RFC is that the first frame mandates 
those bits as 0 and extensions are then negotiated which allow them to 
be used on following frames. By the time negotiation has indicated those 
bits a flag meaning we are already certain about the protocol.

Under that assumption WebSockets vs TLS is also constrained in that the 
RSV1-RSV3 bits MUST be 0 in valid WebSockets, which RSV3 overlaps with 
the '1' hex digit in 0x16.

> Additionally there is a FIN bit there too...

Intentional. So that HTTP/2 can be defined as an extension to WebSockets 
and clients Upgrade: to it from inside WebSockets cleanly - broken 
WebSockets implementations mishandling the WS Upgrade frame will see the 
HTTP/2 initial handshake as a 0-length frame with FIN set.

Amos

---

From: Ilari Liusvaara <[email-masked]>
Date: Thu, 7 Feb 2013 06:27:24 +0200

> Yes. I assume the same. This magic is the first octets of a
> connection and cannot be used elsewhere safely.

Er, what?

- I see no reason to avoid connection magic from occuring elsewhere
- Websockets needs confirmation that the server is of the right
  type for security reasons.

Now if HTTP2 contained built-in websocket-like functionality, then
confirming HTTP2 server would be enough (this would be enough even
if such functionality was optional).

> Intentional. So that HTTP/2 can be defined as an extension to
> WebSockets and clients Upgrade: to it from inside WebSockets cleanly
> - broken WebSockets implementations mishandling the WS Upgrade frame
> will see the HTTP/2 initial handshake as a 0-length frame with FIN
> set.

Upgrade: from inside websockets? That makes absolutely no sense
to me. AFAIK, there is no Upgrade: in websockets (it only uses
one to switch HTTP -> Websockets).

-Ilari

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 07 Feb 2013 18:47:01 +1300

> - I see no reason to avoid connection magic from occuring elsewhere

Your example of the uselessness of upgrading WebSockets -> HTTP/2 is a 
good one to answer that.
The framing magic only works if we are guaranteed to be looking at the 
first octets of the new traffic flow. Allowing the magics to be done 
inside an exsting WebSockets connection (and probably TLS as well) risks 
security problems from the old protocol tricking the server into 
thinking it has to perform the magics. All the issues around SSL 
re-negotiation, SNMP tunneling over HTTP/1 etc demonstrate that 
implementation flaws are easily created which open them to being 
tricked. This magic is a risky procedure at best and allowing it to be 
performed on arbitrary octets mid-way down a connection is asking for 
trouble.

>    type for security reasons.

Yes. That is my understanding also. Which reinforces my understanding of 
why the RSV bits are always 0 on first packet, with fixed format for the 
first two octets. As part of that security systems server-side input 
validation to trigger the correct server reponse.

> if such functionality was optional).

I predict that HTTP/2 will have similar security requirements on some 
traffic. Thus my investigation of the magics and how they can/could/will 
interact with other protocols expected to arrive at server ports.

> one to switch HTTP -> Websockets).

Indeed yes.  The point was that *if* one did for any reason want to do 
an upgrade from WebSockets->HTTP/2 we should be using a WebSockets 
mechanism to do so, not these magics, for the reasons above. Anyone 
wanting to do an upgrade from WebSockets would be required to justify 
that upgrade and define a process in WebSockets for doing it. We have 
one for HTTP/1 and work is underway for TLS NPN etc.

Amos

---

From: Patrick McManus <[email-masked]>
Date: Thu, 7 Feb 2013 16:31:38 +0900

HTTP/2 is a prioritized and muxxed protocol - therefore it needs a max
quanta small enough to still be responsive. As has been shown, 64KB max
size doesn't present an overhead burden, but it still represents 500ms of
serialization time on a 1mb/s network.. You could make a case that is too
much. 5mb/s is a pretty vanilla network these days and that's still 100ms
of transfer time. There is also the case of the server being able to
respond with alacrity to things like RST_STREAM in the face of unwanted
server pushes. These are cases where HTTP/2 has to work responsively.
Increasing bandwidth in the future makes it less important but the overhead
remains a constant fraction in those scenarios and doesn't, imo, create a
scaling burden.

I've worked with a number of spdy devs doing server side deployments that
make the frame size equal to the content length (assuming it fits in the
more generous spdy frame size) and have had the forseeable problems with
responsiveness. We can build a protocol that isn't susceptible to that, and
we should.

On Wed, Feb 6, 2013 at 11:19 PM, Poul-Henning Kamp <[email-masked]>wrote:

> [email-masked]>
> , Roberto Peon writes:
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 07 Feb 2013 07:46:55 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
In message <[email-masked]>
, Patrick McManus writes:

>[...] but it still represents 500ms of serialization time on a 1mb/s
>network..

Just because you _can_ send longer frames, doesn't mean that it is
a good idea to do so.

However, enforcing QoS by restricting frame-lengths is bad architecture,
when HTTP is being used for a lot of bulk applications as well.

>responsiveness. We can build a protocol that isn't susceptible to that, and

It is not the protocol being "susceptible", it is developers being
under-clued.

You cannot enforce policy with a protocol.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 07 Feb 2013 07:56:14 +0000

------ Original Message ------
From: "Patrick McManus" <[email-masked]>
To: "Poul-Henning Kamp" <[email-masked]>
Cc: "Roberto Peon" <[email-masked]>; "Mark Nottingham" 
<[email-masked]>; "HTTP Working Group" <[email-masked]>
Sent: 7/02/2013 8:31:38 p.m.
Subject: Re: Framing and control-frame continuations

<snip>
>unwanted server pushes.
<snip>

Sorry but I find the prospect of "unwanted" server pushes quite 
alarming.  This phrase and a previous mail today which indicated that 
server push would be unilateral made me write this.

Surely all server pushes should be solicited by the client first?  E.g. 
the client indicates it wants pushes.  Until it does that, it doesn't 
receive any.

Having to choke them off seems like a bad choice.

My gut tells me strongly the protocol should be "polite".  Neither end 
should ram anything significant (IOW "large") down the other end's 
throat without prior consent of the receiver (so this goes for large 
bodies on requests too).  Requests are deemed consent, so a server is 
entitled to respond.  Large request bodies are currently rude 
(unconsented) and it actually creates a lot of problems with auth.

Adrien

---

From: William Chan (陈智昌) <[email-masked]>
Date: Thu, 7 Feb 2013 00:06:08 -0800

Since the client begins the connection with a SETTINGS frame, and
there is a setting "max_concurrent_streams" or whatever it's called
that indicates the max concurrent streams that the peer can initiate
(the limit is directional), the client can disable push if desired.
Likewise, receivers can control the about of data the sender can send
by using per-stream and per-session flow control windows (I'm cleaning
up the per-session text proposal and will send it out this weekend or
early next week). Of course, you want to make sure that a single
stream can utilize BDP, so you don't want to have the window sizes too
small, although it's debatable when or not you want a slow start
approach to ramping up a single stream's window up to BDP. All up to
the implementation, since you can send WINDOW_UPDATE deltas on a
per-stream basis and thus enable a single stream to reach BDP, yet
keep the initial window size for most streams small.

In short, I believe there are enough knobs for the receiver to prevent
"rude" sender behavior. I am more worried people will use them badly
and hurt performance.

On Wed, Feb 6, 2013 at 11:56 PM, Adrien W. de Croy <[email-masked]> wrote:
> Cc: "Roberto Peon" <[email-masked]>; "Mark Nottingham" <[email-masked]>;
> "HTTP Working Group" <[email-masked]>
> Sorry but I find the prospect of "unwanted" server pushes quite alarming.
> This phrase and a previous mail today which indicated that server push would
> be unilateral made me write this.
>
> Surely all server pushes should be solicited by the client first?  E.g. the
> client indicates it wants pushes.  Until it does that, it doesn't receive
> any.
>
> should ram anything significant (IOW "large") down the other end's throat
> without prior consent of the receiver (so this goes for large bodies on
> requests too).  Requests are deemed consent, so a server is entitled to
> respond.  Large request bodies are currently rude (unconsented) and it
> actually creates a lot of problems with auth.
>
> Adrien
>
>

---

From: Patrick McManus <[email-masked]>
Date: Fri, 8 Feb 2013 01:05:26 +0900

On Thu, Feb 7, 2013 at 4:46 PM, Poul-Henning Kamp <[email-masked]>wrote:

> [email-masked]>
imo, I've made a quantifiable responsiveness argument but you haven't made
an overhead argument that shows a problem for bulk applications.

---

From: Patrick McManus <[email-masked]>
Date: Fri, 8 Feb 2013 01:07:27 +0900

On Thu, Feb 7, 2013 at 4:56 PM, Adrien W. de Croy <[email-masked]> wrote:

> This phrase and a previous mail today which indicated that server push
> would be unilateral made me write this.
>

sure disabling them completely is possible. but even a client who wants
them in general may not want some in specific because they turn out to be
duplicates of existing cache entries.

---

From: James M Snell <[email-masked]>
Date: Thu, 7 Feb 2013 10:50:45 -0800

On Wed, Feb 6, 2013 at 3:47 AM, Amos Jeffries <[email-masked]> wrote:
[snip]
> The Frame format:

+1 ... I'm still unconvinced that control frames ought to be any
longer than 16-bit but this seems like a reasonable compromise and
workable format.

- James

---

From: Eliot Lear <[email-masked]>
Date: Thu, 07 Feb 2013 20:18:00 +0100

> "rude" sender behavior. I am more worried people will use them badly
Yep.  In fact, a little experience might show that we need fewer knobs.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 7 Feb 2013 17:31:03 -0500

One potential concern with this is that the opaque id is too small. We
haven't discussed this, but the current design relies on the fact that
stream identifiers are not reused. A 16 bit id would lead to connections
being rendered useless very quickly. Jeff mentioned that even at 31 bits,
this happens too often anyway.

That could mean that we need to consider other options anyway, but I
thought the information useful to surface, regardless.

---

From: James M Snell <[email-masked]>
Date: Thu, 7 Feb 2013 14:39:57 -0800

Would it be enough to just make it a full 32 bits? Seems like anything
more than that would be complete overkill. If we run out of
stream-id's in a session at that point we just tear it down and create
a new one.

On Thu, Feb 7, 2013 at 2:31 PM, Martin Thomson <[email-masked]> wrote:
> That could mean that we need to consider other options anyway, but I thought
> the information useful to surface, regardless.
>

---

From: David Morris <[email-masked]>
Date: Thu, 7 Feb 2013 15:39:11 -0800 (PST)

> thought the information useful to surface, regardless.

I was concerned abut the 16bit Opaque ID as well, it seems small to me.

I also noted a comment when this frame header was suggested that frame
data must be a minimum of 16 bits. I'm perhaps not yet aware of the
bigger picture, but I don't understand that constraint. But it is
a desire to maintain 64bit alignment, then just expand the ID and remove
the 16 bit minimum constraing.

---

From: James M Snell <[email-masked]>
Date: Thu, 7 Feb 2013 15:42:38 -0800

+1 .. the minimum data size was a bit strange to me as well.

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 08 Feb 2013 16:27:04 +1300

>> I was concerned abut the 16bit Opaque ID as well, it seems small to me.

You seem to be looking at this ID with the SPDY limitation of no re-use 
in mind.

This frame design differs from SPDY in that the IDs are hop-by-hop and 
can be re-used. The F=1 flag on any frame signals that the ID is not 
going to be sent from that direction anymore and the recipient should 
separate the state associated with it from the next frame receive with 
that same ID - which would be constructing a new session.

Our view was that if you are juggling more than 64K active client 
sessions on one TCP connection the latency involved would lead to HOL 
blocking issues. It is more useful to simply open a second connection 
through the same uplink than to complicate scheduling, increase state, 
and packet size. Whats more you can open this second connection with a 
higher expectation that it be HTTP/2 from the start than a new 
connection is able to have.
  Remember this is an ID for grouping a whole pipeline of transactions, 
not just a single page load or request/response pair. So the data 
transfer involved with each ID value is significant.

Okay. This does limit clients to accessing about 2 billion servers 
concurrently. Which is within the limits we face today on high speed 
load balancers. So I understand if it fees a bit cramped.

>> the 16 bit minimum constraing.

"should" was the wording used, not "must".

It was for alignment.  24-bit should be enough length, but I still see 
that debate going on about 32-bit or not. If we need 32-bit length field 
we are going to have to shuffle the length and ID down leaving an 8-bit 
hole for future reserved use.

Amos

---

From: Roberto Peon <[email-masked]>
Date: Thu, 7 Feb 2013 19:56:00 -0800

I've certainly had proxies using up all ephemeral ports for HTTP to
multiple IPs before, causing queuing and blocking, so 16 bits of concurrent
ID space is definitely too small.
While I admit I'm not as worried as Jeff about having to do something every
2 billion transactions or so, I can say that 64k would cause more
connection open/close than I feel comfortable with.
Note that the flag you're proposing would require waiting for a FIN-ACK
from the other side, else you have a nasty race.
-=R

On Thu, Feb 7, 2013 at 7:27 PM, Amos Jeffries <[email-masked]> wrote:

> You seem to be looking at this ID with the SPDY limitation of no re-use in
> mind.
>
> This frame design differs from SPDY in that the IDs are hop-by-hop and can
> be re-used. The F=1 flag on any frame signals that the ID is not going to
> be sent from that direction anymore and the recipient should separate the
> state associated with it from the next frame receive with that same ID -
> which would be constructing a new session.
>
> Our view was that if you are juggling more than 64K active client sessions
> on one TCP connection the latency involved would lead to HOL blocking
> issues. It is more useful to simply open a second connection through the
> same uplink than to complicate scheduling, increase state, and packet size.
> Whats more you can open this second connection with a higher expectation
> that it be HTTP/2 from the start than a new connection is able to have.
>  Remember this is an ID for grouping a whole pipeline of transactions, not
> just a single page load or request/response pair. So the data transfer
> involved with each ID value is significant.
>
> concurrently. Which is within the limits we face today on high speed load
> balancers. So I understand if it fees a bit cramped.
>
>
>
>
> that debate going on about 32-bit or not. If we need 32-bit length field we
> are going to have to shuffle the length and ID down leaving an 8-bit hole
> for future reserved use.
>
>
> Amos
>
>

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 08 Feb 2013 08:26:57 +0100

> The Frame format:

Since the flow only ends once, isn't an end-of-flow control type more  
efficient use of bits than a flag?

/Martin Nilsson

---

From: Roberto Peon <[email-masked]>
Date: Thu, 7 Feb 2013 23:32:47 -0800

Not in cases where one side of a flow often closes after one control frame,
e.g. most HTTP GETs
-=R

On Thu, Feb 7, 2013 at 11:26 PM, Martin Nilsson <[email-masked]> wrote:

>>         +-+-+-----------+-------------**--+
>>         |F|C|  type     |               |
>>         +-+-+-----------+               +
>>         |        Frame Length (24)      |
>>         +-----------------------------**--+
>>         |       opaque ID (16)          |
>>         +-----------------------------**--+
>>         |     Frame Data (16...N)       |
>>         +-----------------------------**--+
>>
>>

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 08 Feb 2013 08:49:09 +0000

------ Original Message ------
From: "Roberto Peon" <[email-masked]>
To: "Martin Nilsson" <[email-masked]>
Cc: "HTTP Working Group" <[email-masked]>
Sent: 8/02/2013 8:32:47 p.m.
Subject: Re: Framing and control-frame continuations
>Not in cases where one side of a flow often closes after one control 
>frame, e.g. most HTTP GETs
>-=R
IME most GETS are followed by another GET.  HTTP/1.1 anyway.

Adrien

>>On Wed, 06 Feb 2013 12:47:08 +0100, Amos Jeffries 
>><[email-masked]> wrote:
>>
>>>
>>>The Frame format:
>>>
>>>          0                   1

---

From: Roberto Peon <[email-masked]>
Date: Fri, 8 Feb 2013 01:07:04 -0800

Agreed, and the trend is that is only increasing. Using a bit to indicate
end-of-stream (at least unidirectionally, assuming that can be done safely)
is most efficient in such cases.
Were it the other way around, though, of course, it might be more efficient
to do it another way.

-=R