Subject: #549: END_STREAM flag on CONTINUATION

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jul 2014 15:47:35 -0400

<https://github.com/http2/http2-spec/issues/549>

In the issue comments, Martin points out the reasoning behind the current design:

> To articulate the reasons for the current design:
> 
> 	• END_STREAM has no place on PUSH_PROMISE, or continuations thereof.
> 	• Placing END_STREAM on HEADERS ensures that there is no possibility for stupid errors where you have END_STREAM but not END_HEADERS.
> 	• CONTINUATIONS are in most respects a way to create a single frame from many. Logically, they are part of the preceding HEADERS/PUSH_PROMISE. Adding some flags from the preceding frame but not others is conceptually muddy.
> I don't recall any complaint from people actually implementing the protocol.

The first point is the one I’m focusing on; it seems like there are arguments on both sides as to how this could be confusing, making it a toss-up (and thus inclining me to close the issue with no action).

Any further discussion? Could we address the issue that people have by clarifying the text, perhaps?

--
Mark Nottingham   http://www.mnot.net/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 20 Jul 2014 22:00:34 +0000

In message <[email-masked]>, Mark Nottingham wri

>> 	* CONTINUATIONS are in most respects a way to create a single 
>>        frame from many. Logically, they are part of the preceding 
>>        HEADERS/PUSH_PROMISE. Adding some flags from the preceding
>>        frame but not others is conceptually muddy.

The fundamental problem is that CONTINUATIONS started for
one reason (superframes) and got hi-jacked for something
else (pipelining) which gives them the horrid exceptional
behaviour in the current draft.

If HTTP/2's framing layer really had superframes, they would work
for all frame-types, and all frames but the last in a superframe
would have the "... continued" bit set and nothing would
get between the frames of a superframe.  The upper layers would not
even see CONTINUATION frames, but just one large HEADERS superframe.

But that wouldn't work for pipelining, so the semantics got tweaked
and twisted to make pipelining possible without (too much) HOL
blocking.

Now that we have support longer frames, we should clean the superframe
aspect out of CONTINUATION, and define them purely in terms of their
pipelining role.

At the very least that means that the END_STREAM bit goes on the
last frame on the stream, be it a HEADERS, PUSH_PROMISE or a
pipelining CONTINUATION frame.

If we further add that no HTTP header can span frames, the question
of HPACK state disappears, and the multiplexing restrictions can
be removed.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Jeff Pinner <[email-masked]>
Date: Sun, 20 Jul 2014 16:59:56 -0700

> pipelining CONTINUATION frame.

Moving all of the flags to the last CONTINUATION frame requires even
more error handling.

Moving just one of the flags is yet another exceptional case that
requires more logic.

The flags used to be on the last frame and were moved to simplify
implementations. Moving them back is just adding implementation
complexity for no good reason.

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 21 Jul 2014 12:10:56 +1000

> clarifying the text, perhaps?

Words would really help. I'm not sure what words to suggest, though. Maybe
in §4.3 at the end of: "This allows a header block to be logically
equivalent to a single frame." add a nota bene reference to the END_STREAM
flag on HEADERS frames.

Another suggestion, which is a little bit drastic, might be to remove Flags
from the general Frame Format, and instead have each frame description
include its own Flags field (if it needs one).

My personal confusion came from the fact that an incoming DATA[END_STREAM]
basically means "close for reading," so when I saw that HEADERS also has
END_STREAM, occupying the same bit, with the same description, I assumed it
was the same flag. Thus I overlooked the subtlety whereby
HEADERS[END_STREAM] depends on END_HEADERS.

If the DATA Flags and HEADERS Flags had occupied clearly different
namespaces, I wouldn't have been as surprised. It would also help to
explain why there's no registry for flags, even though they appear to be
semi-universal (e.g. 0x4 always means END_HEADERS, 0x8 always means PADDED,
etc.; 0x1 being the exception, meaning either END_STREAM or ACK)

Incidentally there's an aesthetic appeal to making the generic frame header
64 bits, and we could save a byte on some of the administrative frames.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 21 Jul 2014 12:20:39 +1000

Moving the END_STREAM flag to the end is much of the work needed to switch
from HEADERS+CONTINUATION* to just HEADERS*  So why not just go to HEADERS*
and be done with it?

But failing that, while I see moving the flag as step in the right
direction, it is probably not a good step.  It will make unhappy those that
want to avoid significant change, but is probably not enough to make happy
those who want to get rid of continuations.

So I'm probably +0 on this one.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Glen Knowles <[email-masked]>
Date: Sun, 20 Jul 2014 21:18:20 -0700

On Sun, Jul 20, 2014 at 7:20 PM, Greg Wilkins <[email-masked]> wrote:
>
> So why not just go to HEADERS* and be done with it?
>

If there was a way to guarantee that everything required for routing was
included in the first header block I would have more sympathy for HEADERS*.
Requiring all :headers first helps, but since I know of deployments that
need other headers (Forwarded-For to name one), it's not enough.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Jul 2014 08:04:15 +0000

In message <CA+pLO_iLa7ZUq0qwCwA57siYLY1xzqw_=[email-masked]>
>Moving all of the flags to the last CONTINUATION frame requires even

Are you suggesting that ?  I havn't seen anybody else do so ?

>Moving just one of the flags is yet another exceptional case that

I think it makes everything much easier if END_STREAM always sits
on the packet which closes the stream.

---

From: David Krauss <[email-masked]>
Date: Mon, 21 Jul 2014 16:45:44 +0800

> on the packet which closes the stream.

HTTP frames aren’t packets. The implementation needs a persistent internal flag to signal cleanup at the end of the block, whether the END_STREAM is received at the start of a frame, jumbo frame, super-multi-frame, or whatever.

Since END_STREAM is only defined for HEADERS and DATA, I don’t think it’s fair to say that headers make a special case.

CONTINUATION is a special case in other ways, of course, and I hope we can see through superficial differences and give jumbo HEADERS frames a size limit pass, rather than attempt the same with a convoluted definition of what a “frame” is.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 Jul 2014 16:40:33 -0700

I’m not hearing any consensus to make a change here, and the issue itself seems more about how people perceive the flags than any concrete problem.

If someone can suggest text to clarify here, please suggest it to Martin; I’m closing the underlying issue as WONTFIX.

Regards,