Subject: Permissible states for extension frames #591

From: Martin Thomson <[email-masked]>
Date: Mon, 11 Aug 2014 15:03:24 -0700

We don't really say this, but the implication is that an extension
frame can appear anywhere, for any stream. Worst case, you can have
frames appearing with any stream number at any time.

Are we OK with this?

Or... do we want to limit the sending of extension frames on streams
somehow. Note that the most permissibly constrained frame type is
PRIORITY, which can appear in any state other than "idle". A similar
constraint would be relatively easy to enact.

A tighter scope, like just "open" and the sending-permitted
"half-closed" variant, would be even easier to enact, but might reduce
the utility of extension frames.

-- https://github.com/http2/http2-spec/issues/591

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 12 Aug 2014 10:54:58 +1000

> Are we OK with this?

Not really.

If we are going to have CONTINUATIONs, then either they need to be true
jumbo frames as currently specified (can't be interleaved by anything) OR
they should be fully interleaveable by any other frame.   It is entirely
too complex and fragile to allow continuations to be interleaved by
extension frames but not normal frames.

Although I guess that would allow an extension to be defined that has a
bunch of frame types that are identical to normal frame types, except that
they can be interleaved between continuations when normal ones can't be.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 11 Aug 2014 20:00:24 -0700

Greg, this issue has nothing whatsoever to do with CONTINUATIONS, other
than to note that extensions would not be permitted within a sequence of
continuations.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 12 Aug 2014 14:51:19 +1000

> than to note that extensions would not be permitted within a sequence of

Martin,

I don't know how it is not related to continuations?    The question was
are we OK with allowing extension frames to appear anywhere for any stream?

I'm not OK with that because of continuation frames, which I believe have
to be sent in a contiguous block.   We have to specify that extensions
frames are OK anywhere except between HEADERS and CONTINUATION and between
CONTINUATION frames.

I would also be OK with extensions frames anywhere, but only if
continuations allowed interleaving.... the consensus was not to do that, so
we can't allows extension frames anywhere and have to have a more complex
specification for them.

I'm sorry if it looks like I'm trying to fight the already lost
continuation battle.   I'm not, I'm just making sure that we keep the
specification consistent with the complexities of the continuation
mechanism.

regards

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 12 Aug 2014 21:49:00 +1200

On 12/08/2014 4:51 p.m., Greg Wilkins wrote:
> On 12 August 2014 13:00, Martin Thomson wrote:
> 

I dont see why we need to do that at all.

The specs for extensions is already that they cannot redefine existing
frame semantics or layout without an explicit ACK from the other end.
Such as the non-interleaving of HEADERS+CONTINUATION.

To me that means it is perfectly legal to define an extension for making
CONTINUATION interleave, BUT must not be done without explicit
negotiation of that extension.

Amos

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 12 Aug 2014 19:59:13 +0300

> constraint would be relatively easy to enact.

Just as IMO:

I don't think "out-of-the-blue" extension frames have any use in idle
state (since state must be idle afterwards as well). And letting those
frames modify the sub-state in some manner would likely be a DOS attack
waiting to happen.

But negotated extensions are different matter, since those might define
new stream-opening frame types (or something even more radical).

-Ilari

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 13 Aug 2014 09:50:33 +1000

> I dont see why we need to do that at all.

The problem is that we have a very strict definition of no interleaving of
frames between continuations.  Not even control frames can be
interleaved.   This allows implementations to treat continuations as simply
an atomically arriving jumbo.

In fact when we discussed the more complete version of the state machine
https://github.com/http2/http2-spec/issues/484 that could handle non atomic
arrival of end_stream and  end_headers, the argument was made that the
state machine should not be complicated in that way.  It was said that
nobody would implement the proposed 1/4 closed and 3/4
<http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/att-0720/http2state.txt>
closed states because you can effectively treat the arrival of an
end_stream and the following end_headers as atomic.

If we allow arbitrary interleaving of extension frames, then we will have
extension frames arriving in the middle of these supposed atomic stream
state transitions that every implementation is going to have elided in some
not specified private way.    Thus it is highly unlikely that portable
extensions that interact with stream state will be able to be developed as
there is not a common definition for non-atomic leap between two defined
stream states.

Also any implementation that wishes to some day support the any extension
frames, cannot be written with the simplified state machine and must treat
headers and continuations as non atomic events.  This is not an impossible
task and if we supported interleaving of continuations frames by other
protocol frames, then it would have to be done anyway.  But unfortunately
because we don't support any interleaving in the default protocol, that
means that any impl that does support the full state machine cannot
actually be tested with the un extended protocol. So there are going to be
a lot of undiscovered bugs that might be exploitable once a few extensions
are implemented long after the more rigorous testing of the stream states
has been done.

The best solution would of been to have a framing layer that was entirely
independent of frame types, so it could have handles stream control without
even needing to know what an extension was.   But that horse bolted long
ago.

We could have taken a step towards that optimum solution by removing
continuations and supporting fragmentable interleavable headers with a
simple stream state machine where end_stream is on the last frame. Sure it
is more complex, but at least it can be tested in a normal use case.  But
we didn't get consensus for that.

So next best would be to allow interleaving of at least control frames
between continuation frames.  This would at least allow  testing the
complexities of 1/4 and 3/4 closed streams but without having many benefits
for this complexity.  Kind of the worst of all worlds, but probably
necessary if we want extensions able to be arbitrarily interleaved.

Failing that, all we can continue to treat continuations as a very special
case and that nothing can interleave, not even extension frames.   Only
then can implementations implement the simplified state machine that was
explicitly used to justify many design decisions.  Unfortunately this means
that all extensions, even ones that don't care about stream state, have to
known about this special case so they don't interleave at the wrong time.

Sorry for the essay, but it is really important to not allow WG fatigue to
let some less-precise wording through.  We've been told that "ugliness" is
not a technical reason for rejecting a design, so we have some ugly parts
in the specification.  The cost of ugliness comes now as every single thing
you add/extend to the specification has to be run past all the ugliness and
we have to make sure that we don't break any of the fragile reasoning or
fictional state machine used to support it.   Non-ugly design is more
robust in the face of change/extension.

regards

---

From: Mike Bishop <[email-masked]>
Date: Wed, 20 Aug 2014 20:44:23 +0000

We're okay with the current text.  An extension will specify in what states it's valid to send, so this only applies to unknown extensions on receive.  Those will be discarded anyway -- anything beyond that will complicate implementations unnecessarily.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Monday, August 11, 2014 3:03 PM
To: HTTP Working Group
Subject: Permissible states for extension frames #591

We don't really say this, but the implication is that an extension frame can appear anywhere, for any stream. Worst case, you can have frames appearing with any stream number at any time.

Are we OK with this?

Or... do we want to limit the sending of extension frames on streams somehow. Note that the most permissibly constrained frame type is PRIORITY, which can appear in any state other than "idle". A similar constraint would be relatively easy to enact.

A tighter scope, like just "open" and the sending-permitted "half-closed" variant, would be even easier to enact, but might reduce the utility of extension frames.

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 23 Aug 2014 09:55:51 +1000

On 21 August 2014 06:44, Mike Bishop <[email-masked]> wrote:

> We're okay with the current text.  An extension will specify in what
> states it's valid to send, so this only applies to unknown extensions on
> receive.  Those will be discarded anyway -- anything beyond that will
> complicate implementations unnecessarily.

Mike,

The problem with deferring to extensions to " specify in what states it's
valid to send", that there are some undefined states

The state machine as defined by the document assumes that the arrival of a
trailer is atomic and switches state directly to half closed or  closed.
But in reality there are extra states of quarter close, two quarter closed
and three quarter closed (see
https://github.com/http2/http2-spec/issues/484).

So if we wish to defer to extensions, then we should re-open 484 and define
all the states, otherwise it will be extremely difficult for any extension
to be correctly defined to work with closing streams.

cheers

---

From: Martin Thomson <[email-masked]>
Date: Fri, 22 Aug 2014 17:19:33 -0700

On 22 August 2014 16:55, Greg Wilkins <[email-masked]> wrote:
> two quarter closed

How about we not worry so much about that.  The text is very clear
that unless you have negotiated some other protocol, receiving frames
in the midst of a header block is bad and is expressly forbidden.

That is now doubly so:
https://github.com/http2/http2-spec/commit/ec57ba8fbc8d20d2ccae799a6a581666dce3d2f4

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 23 Aug 2014 12:59:32 +1000

> https://github.com/http2/http2-spec/commit/ec57ba8fbc8d20d2ccae799a6a581666dce3d2f4

Oh - I thought we were still favouring allowing arbitrary placement of
extension frames.   If we don't allow them to be within header blocks, then
we can stick with the simpler state machine, with the cost that extensions
must be HTTP semantic away... but I think they pretty much have to be
anyway (without radical change).

I'm ok with that clarification.

cheers

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 25 Aug 2014 10:21:54 +1000

Can we close this issue with that change?

I.e., does anyone want to go further than disallowing extension frames between header-bearing frames?

Regards,

On 23 Aug 2014, at 12:59 pm, Greg Wilkins <[email-masked]> wrote:

> Oh - I thought we were still favouring allowing arbitrary placement of extension frames.   If we don't allow them to be within header blocks, then we can stick with the simpler state machine, with the cost that extensions must be HTTP semantic away... but I think they pretty much have to be anyway (without radical change).
> 

--
Mark Nottingham   https://www.mnot.net/