Subject: h2 frame layout

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 29 Aug 2014 15:49:58 -0700

This comment is in reference to section 4.1 of

  http://tools.ietf.org/id/draft-ietf-httpbis-http2-14.txt

> 4.1.  Frame Format
> 
>    All frames begin with a fixed 9-octet header followed by a variable-
>    length payload.
> 
>      0                   1                   2                   3
>      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>     |                 Length (24)                   |
>     +---------------+---------------+---------------+
>     |   Type (8)    |   Flags (8)   |
>     +-+-+-----------+---------------+-------------------------------+
>     |R|                 Stream Identifier (31)                      |
>     +=+=============================================================+
>     |                   Frame Payload (0...)                      ...
>     +---------------------------------------------------------------+
> 
>                                Frame Layout

I think this has become a bit moth-eaten over time as the WG has
adjusted field lengths and perhaps added or removed a few.
I also find the use of a bit array with magic gaps to be editorially
bizarre, but this message is about the actual bits.

I should be shocked that folks are "not concerned" about the use
of a 9-octet header and the resulting effect on data alignment, but it
is true that it won't matter much to character-oriented Web traffic.
Nevertheless, I'd expect HTTP/2 to at least make an effort here
given the number of APIs that don't process character traffic.

Looking at the current spec, I think we can do better.

  1) There are only 10 defined frame types and 2 useful flags.
  2) The length is the most random of these fields and used last.
  3) The stream identifier is the least random and used first.
  4) Why are we reserving the high bit of an unsigned field?

My suggestion is that the frame layout be changed to either

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                   Stream Identifier (32)                      |
 +---------------+-----------------------------------------------+
 |Type(4)|Flag(4)|   Payload Length (24)                         |
 +=+=============================================================+
 |                   Frame Payload (0...)                      ...
 +---------------------------------------------------------------+

or

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                   Stream Identifier (32)                      |
 +---------------+-----------------------------------------------+
 | Frame Type(8) |   Payload Length (24)                         |
 +=+=============================================================+
 |                   Frame Payload (0...)                      ...
 +---------------------------------------------------------------+

Rationale for both:

  11% reduction in per-frame overhead
  A fixed 64bit layout is faster to parse than 72bits.
  A sender doesn't have to worry about 64bit alignment.
  A receiver is likely to partition based on stream id first.

Rationale for A:

  If we need more than 6 additional frame types, we might as well
    skip to a new protocol version.
  If we need more than 4 flags per frame, we should use another type.
  The useful flags are END_STREAM and END_HEADERS and they could be
    defined in general (independent of frame type).

Rationale for B:

  The current usage of frame flags in spec-14 is frightening.
  In all cases, they can and should be separate frame types.

  END_STREAM:  Orthogonal. Could be moved to the reserved bit or made
    another frame type (an extra 8 bytes at end of stream is better
    than the extra 8 flag bits per frame).

  END_HEADERS: Orthogonal. Could be just another frame type or moved
    to the beginning of the frame payload in Headers, Continuation,
    and Push_Promise.

  PADDED: Padding is just a form of ignored data and should
  be a PADDING type.  We can then remove those optional fields from the
  other frame types.  Optional fields at the beginning of otherwise fixed
  frames is parser hell.

  Header PRIORITY: What do headers have to do with frame priority?
  Why here and not on DATA?  We should instead have different
  frame types for MY_PRIORITY (this is how I am sending/reading) and
  YOUR_PRIORITY (this is how I want you to prioritize sending/reading).
  We can then remove those optional fields from the other frame types.

  Settings ACK: send an ACK frame type instead (and remove the additional
  complexity of processing requirements in the Settings type).

  Ping ACK: send a PONG type.

Alternative A would just make the protocol more efficient and faster to
parse at the cost of some extensibility. B would significantly simplify the
protocol and the spec.  Some combination of the two is also possible
(e.g., 6 bits for type and 2 flag bits for END_STREAM and END_HEADERS).

Alternatively, if END_STREAM is moved to the reserved bit E, then it
would look like:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |E|                 Stream Identifier (31)                      |
 +---------------+-----------------------------------------------+
 | Frame Type(8) |   Payload Length (24)                         |
 +=+=============================================================+
 |                   Frame Payload (0...)                      ...
 +---------------------------------------------------------------+

Yes, I know this suggestion comes late in the process.  I was busy.

Cheers,

Roy T. Fielding                     <http://roy.gbiv.com/>
Senior Principal Scientist, Adobe   <http://www.adobe.com/>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 29 Aug 2014 23:04:18 +0000

--------
In message <[email-masked]>, "Roy T. Fielding" w

>  In all cases, they can and should be separate frame types.

I'm 100% behind with Roy here.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 30 Aug 2014 09:45:06 +1000

> I'm 100% behind with Roy here.
​This lines up pretty closely with my suggestion in the
END_STREAM/CONTINUATION thread [1]. I'm +1 for removing flags from the
general frame format header.

I'm much less keen on reducing the size of frame_type field, because that
would limit my ability to play with my toys^H^H^H^Hextensions[2][3].

[1]: http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/1313.html​
​[2]: http://tools.ietf.org/html/draft-kerwin-http2-encoded-data-03
[3]: ​http://tools.ietf.org/html/draft-kerwin-http2-segments-01

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 30 Aug 2014 05:18:14 +0000

--------
In message <[email-masked]>

>would limit my ability to play with my toys^H^H^H^Hextensions[2][3].

There's a word for protocols which need 256 frame types:  "Wrong".

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 30 Aug 2014 15:56:54 +1000

Roy,

there have been several contributors (including myself) who have come into
the process late and looked at the framing layer and had a similar reaction
to what you have expressed.          For my own part,  I strongly argued
against the framing flags being a function of frame type and I expressed my
belief that it is a mistake to conflate the semantics of HTTP with the
framing layer.

As each of these new views has come in, it has woken up some other voices
with lingering levels of discontent towards the design and a new round of
debating some  design decisions results.   Unfortunately none of the
iterations of this pattern have been able to convince enough of the voices
here to achieve consensus to change some fundamental design decisions that
drive what has been described as the "uglyness" of the framing layer.
I think the problem is that none of the new voices have been able to
inspire an appetite for wholesale change and have instead resorted to try
to "fix" the protocol by arguing for change bit by bit or byte by byte.

So while there have been many minor improvements in the last few iterations
of the draft, I do not think that some of the fundamental problems with the
framing layer have been fixed and when new eyes look at tit they still have
a WTF moment with some elements of it.

Your own proposals represent both ends of the spectrum:

Your proposal A is just a trim of the current framing layer to send the
same semantics over 64 bytes rather than 72.  I've got no objections to
that, but note that making it look data aligned will not actually make it
data aligned unless padding is not a function of frame type and we require
all payloads to be padded to a word boundary, so I think more substantial
change is necessary to achieve any more substantial than a saving of 1 byte
per frame.   So it is a bit by bit improvement, but doesn't "fix" the
framing.

Your proposal B is a significant change of semantic.  I totally agree that
defining concepts like END_STREAM independently to frame type is a really
good idea, as this would facilitate carrying alternate semantics like
websocket and maybe even waka?    However, all previous attempts to achieve
consensus that this is something that should be fixed has failed, I think
mostly because the WG was wrongly chartered to primarily consider HTTP
semantics.   If it had been chartered to support multiple web semantics:
HTTP, Websocket and potential future web semantics, then we might have
ended up with a better framing layer that is not dependent on frame type or
HTTP semantics.

As much as I would like to see significant changes to the framing layer, I
don't think it is worthwhile reopening that can of worms unless we firstly
revisit the charter.     If essentially all we have to transport is HTTP in
it's current usage patterns, then the argument always devolves to that it
doesn't matter if the framing layer is ugly, fragile, inextensible, special
purpose etc.   We have made the mistake of trying to generalise from a
single example, so the non-HTTP parts of the protocol such as unused flags
and frame types are highly unlikely to be usable for anything that does not
pretend to be HTTP.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 30 Aug 2014 10:32:20 -0700

I don't see any point in engaging in a self-defeatist argument over what can be changed. There are no implementations of the final protocol because the spec has an interop break built into the versioning. 

A WG last call is based on what the group has produced so far, which means the suggestion should be evaluated on its relative merit when compared to the current spec and the willingness of implementors to implement. Of course, if most of the people here just want to be done and don't care about the quality of the protocol, then it will be a very interesting trip through the IESG last call process.

BTW, none of those suggestions change semantics. They might enable additional semantics later on, but it should be trivial to change an implementation right now. All I change is where to write or read the same data. Also, I only care about 64bit alignment within the messaging buffers.

....Roy

> On Aug 29, 2014, at 10:56 PM, Greg Wilkins <[email-masked]> wrote:
> 
> 
> Roy,
> 
> there have been several contributors (including myself) who have come into the process late and looked at the framing layer and had a similar reaction to what you have expressed.          For my own part,  I strongly argued against the framing flags being a function of frame type and I expressed my belief that it is a mistake to conflate the semantics of HTTP with the framing layer.
> 
> As each of these new views has come in, it has woken up some other voices with lingering levels of discontent towards the design and a new round of debating some  design decisions results.   Unfortunately none of the iterations of this pattern have been able to convince enough of the voices here to achieve consensus to change some fundamental design decisions that drive what has been described as the "uglyness" of the framing layer.       I think the problem is that none of the new voices have been able to inspire an appetite for wholesale change and have instead resorted to try to "fix" the protocol by arguing for change bit by bit or byte by byte.
> 
> So while there have been many minor improvements in the last few iterations of the draft, I do not think that some of the fundamental problems with the framing layer have been fixed and when new eyes look at tit they still have a WTF moment with some elements of it.
> 
> Your proposal A is just a trim of the current framing layer to send the same semantics over 64 bytes rather than 72.  I've got no objections to that, but note that making it look data aligned will not actually make it data aligned unless padding is not a function of frame type and we require all payloads to be padded to a word boundary, so I think more substantial change is necessary to achieve any more substantial than a saving of 1 byte per frame.   So it is a bit by bit improvement, but doesn't "fix" the framing.
> 
> Your proposal B is a significant change of semantic.  I totally agree that defining concepts like END_STREAM independently to frame type is a really good idea, as this would facilitate carrying alternate semantics like websocket and maybe even waka?    However, all previous attempts to achieve consensus that this is something that should be fixed has failed, I think mostly because the WG was wrongly chartered to primarily consider HTTP semantics.   If it had been chartered to support multiple web semantics: HTTP, Websocket and potential future web semantics, then we might have ended up with a better framing layer that is not dependent on frame type or HTTP semantics.
> 
> As much as I would like to see significant changes to the framing layer, I don't think it is worthwhile reopening that can of worms unless we firstly revisit the charter.     If essentially all we have to transport is HTTP in it's current usage patterns, then the argument always devolves to that it doesn't matter if the framing layer is ugly, fragile, inextensible, special purpose etc.   We have made the mistake of trying to generalise from a single example, so the non-HTTP parts of the protocol such as unused flags and frame types are highly unlikely to be usable for anything that does not pretend to be HTTP.
> 
> cheers
> 
> --

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 31 Aug 2014 11:49:41 +1000

On 30 August 2014 15:18, Poul-Henning Kamp <[email-masked]> wrote:

> --------
> In message <
> [email-masked]>
> , Matthew Kerwin writes:
>
It only *needs* the ten or so it has. I'll rephrase: I'm much less keen on
reducing the size of frame_type field to 4 bits; I don't believe we need
256 types, but 16 is too restrictive, and personally I don't want to go
introducing a non-nibble-aligned chunk into the header without good cause.
Thus, I round up to 8 bits, instead of down to 4.

Either way, I prefer something like option B > current spec > option A; but
they're all livable.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 31 Aug 2014 06:24:18 +0000

--------
In message <[email-masked]>
, Matthew Kerwin writes:

>256 types, but 16 is too restrictive,

100% agree on that.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 1 Sep 2014 08:59:27 +1000

On 31 August 2014 03:32, Roy T. Fielding <[email-masked]> wrote:

> I don't see any point in engaging in a self-defeatist argument
>

Sorry for predicting doom if we debate the detail of these proposals.   I
just think that regardless of technical merit, they will meet the same fate
within the WG as other similar proposals - unless we reconsider the charter
or there is some other external influence to stop us iterating on these
points.

>
> BTW, none of those suggestions change semantics.
>

Well maybe not strictly a semantic change, but B is certainly a significant
change that can't be made without forcing further significant change in the
protocol (which is not necessarily a bad thing).   I think the protocol has
a number of highly co-dependent design decision that while they are self
consistent, also make it very fragile to change.

In this case, the current design has tried to enforce HTTP semantics by
making it impossible for the protocol to carry END_STREAM in some locations
that are not legal HTTP frame sequences - eg within a partial
header/trailer or after a push promise.   If the protocol can carry
END_STREAM on any frame, then this is a significant change:

At the very least, implementations that were assuming the illegal HTTP end
streams could not be received would need to explicitly check for them (not
a bad thing).

Then, if a HTTP stream does receive an END_STREAM in an illegal position,
is this a protocol error or just a stream error?  There are valid reasons
for servers/clients to abort partial HTTP messages so it would be good to
make sure the protocol can handle such termination as a stream error and
not discard the entire connection.  However this means we have to define
hpack state handling of partially received headers.

We would also have to define what it means to receive a RST_STREAM frame
that did not have the END_STREAM flag set?   Is this a non-terminal error?
a protocol error?  or do we still need RST_STREAM frame?

Finally, removing END_STREAM from some frame types removes the only
significant difference between HEADER frames and  CONTINUATIONS frames (
other than the optional priority field that can be sent in it's own frame
anyway).    Thus there would be no reason to have the continuations frame
and we should instead just use multiple HEADER frames.

So while I definitely would like to see significant changes/simplification
of the framing layer, I think we have to be very careful to check that we
have not built a house of cards on those particular foundations.  I don't
think there are many easy/simple changes that can be made at this level
with a wide re-evaluation of the entire design.
regards

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 1 Sep 2014 09:34:19 +1000

On 1 September 2014 08:59, Greg Wilkins <[email-masked]> wrote:

> Well maybe not strictly a semantic change, but B is certainly a
> significant change that can't be made without forcing further significant
> change in the protocol (which is not necessarily a bad thing).   I think
> the protocol has a number of highly co-dependent design decision that while
> they are self consistent, also make it very fragile to change.
>
>
​(Snipping rest of post)

Repeating myself again, but: what do you think of an option C: remove Flags
from the general frame header, and add individual Flags fields to whichever
frame types require them?

* flags aren't universal (see: 0x1) so you can't act on them without having
inspected the frame_type field anyway
* many frame types don't have any flags at all
* by simply removing the field, the generic header goes back to being 64
bits

Thus we neither change semantics nor wobble the house of cards; we just
move the flags to the place where they make sense. Practically editorial.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 1 Sep 2014 10:34:40 +1000

On 1 September 2014 09:34, Matthew Kerwin <[email-masked]> wrote:

> Repeating myself again, but: what do you think of an option C: remove
> Flags from the general frame header, and add individual Flags fields to
> whichever frame types require them?

Matthew,   in general I think the framing layer should only know about
framing concerns and should be neutral towards any semantics conveyed in
the frame payloads.   So in principal, yes I agree than general flags (and
probably frame types other than those needed by framing) can easily be put
into the payload.

However, END_STREAM is a-priori a framing concern, so I don't think it
should be moved to the frame payload or be a function of frame type.   I
would have preferred the framing layer was designed without HTTP semantic
knowledge and was able to transport arbitrary bidirection flow controlled
streams, perhaps with segments defined so that
flushing/forwarding/segmentation behaviour can be guaranteed to a level
needed.        Other than what is required to achieve that, I see no reason
that the framing layer should know any more about what it is transporting,
even to the extent that it really should not know the difference between a
header and data frame.

cheers

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 1 Sep 2014 11:57:48 +1000

On 1 September 2014 10:34, Greg Wilkins <[email-masked]> wrote:

> header and data frame.​
tl;dr: see the diagram below

Allow me to try to reconstruct what you're saying via my thought processes:

If the framing machine were frame-type-agnostic, all frame types could
carry the END_STREAM flag, but it would only have meaning when associated
with a non-zero stream id.

Of those types that can have a non-zero stream id:
* DATA and HEADERS define an END_STREAM flag.
* PRIORITY, RST_STREAM, PUSH_PROMISE and WINDOW_UPDATE don't.
* CONTINUATION is snafu so I'm not touching it

Of those that don't define the flag, it could be argued that it's because
it doesn't make much sense; RST_STREAM can't *not* end the stream, and a
WINDOW_UPDATE or PP that does is silly. Maybe there's some small value in
the case of PRIORITY, but it still smells a bit silly. So in theory we
could reword the spec to say: the flag is defined and MUST be zero -- which
may add more validation requirements to the receivers, but it doesn't
change the general thrust of framing/streams.

Thus, at one level of thinking we can say that all frame types that have a
non-zero stream id define the END_STREAM flag, so we're on our way to
writing a generic ​framer that doesn't need to delve too deeply into the
types.

Except that SETTINGS and PING have an ACK flag that occupies the same bit
as the END_STREAM flag. So the framer has to inspect either the frame type
or maybe the stream id before it can interpret the flag. So it's not
generic.

To move things forward we could borrow from Roy's option B and move the
END_STREAM bit from Flags to somewhere else, and then use my option C and
move the remaining flags to the payload. Since both END_STREAM and Stream
Identifier are stream-related parameters, and we happen to have a reserved
bit sitting right at the top of the stream identifier, why not shove
END_STREAM there?

I'll note that our generic framer may still have some responsibility in
inspecting END_HEADERS and barfing on bad CONTINUATIONing, but that whole
mess is, as I said earlier, AFU, so whatever. But at least with this
proposal, continuation becomes the only exception to an otherwise fairly
sensible system.

tl;dr:

```
  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |   Type (8)    |                 Length (24)                   |
 +-+-------------+-----------------------------------------------+
 |E|                 Stream Identifier (31)                      |
 +=+=============================================================+
 |                   Frame Payload (0...)                      ...
 +---------------------------------------------------------------+
                            Frame Layout

The fields of the frame header are defined as:

Length:
   The length of the frame payload expressed as an unsigned 24-bit
   integer. Values greater than 214 (16,384) MUST NOT be sent unless
   the receiver has set a larger value for SETTINGS_MAX_FRAME_SIZE.

   The 8 octets of the frame header are not included in this value.

Type:
   The 8-bit type of the frame. The frame type determines the format
   and semantics of the frame. Implementations MUST ignore and discard
   any frame that has a type that is unknown.

E:
   This bit being set indicates that this frame is the last that the
   endpoint will send for the identified stream. Setting this flag
   causes the stream to enter one of the "half closed" states or the
   "closed" state (Section 5.1).

   This bit MUST remain unset (0) when sending unless explicitly
   allowed by the frame type definition.

Stream Identifier:
   A 31-bit stream identifier (see Section 5.1.1). The value 0 is
   reserved for frames that are associated with the connection as a
   whole as opposed to an individual stream.

The structure and content of the frame payload is dependent entirely
on the frame type.
```

...and move the frame field (minus END_STREAM) to the payload, if and when
it's required.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 1 Sep 2014 12:44:09 +1000

On 1 September 2014 11:57, Matthew Kerwin <[email-masked]> wrote:

> If the framing machine were frame-type-agnostic,

+1.  That is my main point.

> all frame types could carry the END_STREAM flag

that is one way of achieving it.    Roy's proposal B is another.

I'm a bit agnostic as to which is best.   The flag is probably simplest and
more conventional, but would mean that we need to review silly combinations
like RST_STREAM with no END_STREAM.    But maybe in a frame-type agnostic
world, we don't need RST_STREAM and GO_AWAY frames?   Isn't RST_STREAM just
a GO_AWAY with a non zero stream ID?

Maybe we can replace the END_STREAM flag and both GO_AWAY and RST_STREAM
frames with a single END_STREAM frame:

   -   END_STREAM(SID==0, CODE==0),  normal graceful go away
   -   END_STREAM(SID==0, CODE!=0),  error connection shutdown
   -   END_STREAM(SID!=0, CODE==0),  normal stream shutdown
   -   END_STREAM(SID!=0, CODE!=0),  reset stream style error

cheers