Subject: END_SEGMENT and headers

From: David Krauss <[email-masked]>
Date: Wed, 16 Apr 2014 12:54:06 +0800

The END_SEGMENT flag is defined for HEADERS frames, but it seems to be meaningless. A literal reading of the draft seems to say that such a frame cannot be coalesced with subsequent CONTINUATION frames, but those frames do not also have the same (useless) segmentation capability.

Why not let END_SEGMENT and END_HEADERS be the same bit? Then headers be treated as the first segment of any stream, by some simplistic intermediaries. It also reclaims a flag bit, as they are running into short supply.

Also, is END_SEGMENT required when END_STREAM is set? It seems it should be, for the sake of protocol regularity.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 15 Apr 2014 22:00:57 -0700

Seems like a reasonable reduction/improvement to me.
-=R

On Tue, Apr 15, 2014 at 9:54 PM, David Krauss <[email-masked]> wrote:

> The END_SEGMENT flag is defined for HEADERS frames, but it seems to be
> meaningless. A literal reading of the draft seems to say that such a frame
> cannot be coalesced with subsequent CONTINUATION frames, but those frames
> do not also have the same (useless) segmentation capability.
>
> Why not let END_SEGMENT and END_HEADERS be the same bit? Then headers be
> treated as the first segment of any stream, by some simplistic
> intermediaries. It also reclaims a flag bit, as they are running into short
> supply.
>

---

From: Roberto Peon <[email-masked]>
Date: Wed, 16 Apr 2014 14:45:29 -0700

I misread that!

I withdraw that this is a good improvement, since there are protocols that
may want to have metadata in the middle of a message.

-=R

---

From: David Krauss <[email-masked]>
Date: Thu, 17 Apr 2014 07:18:06 +0800

What? This proposal does not disturb segmentation. Metadata may appear in the middle of a message, but not in the middle of the headers!

On 2014–04–17, at 5:45 AM, Roberto Peon <[email-masked]> wrote:

> I misread that!
> 
> I withdraw that this is a good improvement, since there are protocols that may want to have metadata in the middle of a message.
> 
> -=R

---

From: Roberto Peon <[email-masked]>
Date: Wed, 16 Apr 2014 16:30:26 -0700

A message might be comprised of HEADERS   DATA(200bytes)
HEADERS+CONTINUATION   DATA(20bytes)   HEADERS.

Your suggestion of making END_HEADERS mean END_SEGMENT would disallow
expression of such a message.

On Wed, Apr 16, 2014 at 4:18 PM, David Krauss <[email-masked]> wrote:

> What? This proposal does not disturb segmentation. Metadata may appear in
> the middle of a message, but not in the middle of the headers!
>
> > I withdraw that this is a good improvement, since there are protocols
> that may want to have metadata in the middle of a message.
> >
> > -=R
>
>
>

---

From: David Krauss <[email-masked]>
Date: Thu, 17 Apr 2014 07:30:45 +0800

You mean all on the same stream? No, that is not allowed, because the headers (and CONTINUATIONs) must be an uninterrupted sequence of frames. There must be exactly one HEADERS frame per stream because it changes the stream’s state.

Before the headers are complete, the server cannot dispatch the stream to an application, so there is no application there to receive the metadata…

On 2014–04–17, at 7:21 AM, Roberto Peon <[email-masked]> wrote:

> A message might be comprised of HEADERS   DATA(200bytes)   HEADERS+CONTINUATION   DATA(20bytes)   HEADERS.
> 
> Your suggestion of making END_HEADERS mean END_SEGMENT would disallow expression of such a message.
> -=R
> 
> 
> On Wed, Apr 16, 2014 at 4:17 PM, David Krauss <[email-masked]> wrote:

---

From: Roberto Peon <[email-masked]>
Date: Wed, 16 Apr 2014 16:32:12 -0700

Are you mixing up the HTTP on HTTP2 usecase with the framing part of HTTP2?
-=R

On Wed, Apr 16, 2014 at 4:30 PM, David Krauss <[email-masked]> wrote:

> You mean all on the same stream? No, that is not allowed, because the
> headers (and CONTINUATIONs) must be an uninterrupted sequence of frames.
> There must be exactly one HEADERS frame per stream because it changes the
> stream’s state.
>
> Before the headers are complete, the server cannot dispatch the stream to
> an application, so there is no application there to receive the metadata…
>
>

---

From: David Krauss <[email-masked]>
Date: Thu, 17 Apr 2014 07:33:59 +0800

No, I’m going purely by the HTTP2 spec. A HEADERS frame must initiate a new stream, and a CONTINUATION frame must be preceded on the connection by a HEADERS or CONTINUATION frame from the same stream.

---

From: David Krauss <[email-masked]>
Date: Thu, 17 Apr 2014 07:36:48 +0800

On 2014–04–17, at 7:33 AM, David Krauss <[email-masked]> wrote:

> a CONTINUATION frame must be preceded on the connection by a HEADERS or CONTINUATION frame from the same stream.

To be clear: *immediately* preceded with no intervening frames. Also, a DATA frame is not valid until after END_HEADERS which moves to the “open” state.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 16 Apr 2014 16:45:33 -0700

I remember pretty clearly how the HEADERS frames work w.r.t. continuations
:)

When an entire headers block has been received (i.e. at the point where a
HEADERS frame with END_HEADERS is set), you've received a complete set of
metadata.
This is required to start a stream, but the protocol allows HEADERS later
in a stream, potentially as part of the same message.

Assuming that HEADERS_BLOCK is defined thusly:
HEADERS_BLOCK := (HEADERS)* HEADERS_WITH_END_HEADERS ;

Then a message would be defined as:
MESSAGE :=  HEADERS_BLOCK
            (HEADERS_BLOCK | DATA)*
            (HEADERS_WITH_END_SEGMENT | DATA_WITH_END_SEGMENT)
         ;
Your proposal would not allow this, or am I misunderstanding?
-=R

On Wed, Apr 16, 2014 at 4:36 PM, David Krauss <[email-masked]> wrote:

> To be clear: *immediately* preceded with no intervening frames. Also, a
> DATA frame is not valid until after END_HEADERS which moves to the “open”
> state.
>
>

---

From: Jeff Pinner <[email-masked]>
Date: Wed, 16 Apr 2014 16:46:24 -0700

The draft states both that frames can not be coalesced across a segment
boundary, but also that segment boundaries must be preserved.

You are correct that for a HEADERS frame w/ the END_SEGMENT flag set, there
is no way a server could coalesce frames because the next frames must be
CONTINUATION frames until the END_HEADERS flag is received, so in this
case, END_SEGMENT and END_HEADERS are equivalent.

BUT

For a frame with the END_SEGMENT flag CLEARED, the HEADERS/CONTINUATION
frame sequence would still need an END_HEADERS flag set, so in this case
they are not equivalent. In this case, combining the flags would be
inserting a segment boundary where none previously existed.

On Wed, Apr 16, 2014 at 4:36 PM, David Krauss <[email-masked]> wrote:

>
> On 2014-04-17, at 7:33 AM, David Krauss <[email-masked]> wrote:
>
> DATA frame is not valid until after END_HEADERS which moves to the "open"
> state.
>
>
>

---

From: "Kulkarni, Saurabh" <[email-masked]>
Date: Wed, 16 Apr 2014 19:46:50 -0400

David,

Actually Robert is right. This is the text at the start of HEADERS frame:

"The HEADERS frame (type=0x1) carries name-value pairs. It is used to
 open a stream (Section 5.1). HEADERS frames can be sent on a stream
 in the "open" or "half closed (remote)" states.²

- Saurabh

On 4/16/14, 4:36 PM, "David Krauss" <[email-masked]> wrote:

>
>On 20140417, at 7:33 AM, David Krauss <[email-masked]> wrote:
>
>DATA frame is not valid until after END_HEADERS which moves to the ³open²
>state.
>
>

---

From: David Krauss <[email-masked]>
Date: Thu, 17 Apr 2014 08:04:23 +0800

>             (HEADERS_BLOCK | DATA)*

Ah, I did not realize this.

> Your proposal would not allow this, or am I misunderstanding?

When you said “metadata”, I thought you meant segmentation as the means of separating it, not headers as a general metadata transport.

However:

1. How is the server supposed to make late headers available to the application? They get coalesced with and replace earlier headers? What does the client-side interface look like?

2. Without segmentation, how does an application know that the headers are forwarded by an intermediary before some subsequent data that expects them? It seems that END_SEGMENT must be used with END_HEADERS anyway because otherwise a race condition is allowed to delay the headers. Either that, or END_HEADERS must imply END_SEGMENT behavior. Either way is already equivalent to my proposal. So although I am (unintentionally) proposing to disallow some valid sequences according to the protocol, they only represent unsafe practices.

---

From: David Krauss <[email-masked]>
Date: Thu, 17 Apr 2014 08:58:45 +0800

On 2014–04–17, at 7:46 AM, Jeff Pinner <[email-masked]> wrote:

> For a frame with the END_SEGMENT flag CLEARED, the HEADERS/CONTINUATION frame sequence would still need an END_HEADERS flag set, so in this case they are not equivalent. In this case, combining the flags would be inserting a segment boundary where none previously existed.

My intent was only to represent them by the same bit, not to merge the semantics. (I didn’t think it would make an actual difference, but notionally having two different flags is just easier to understand.) However, as I noted in reply to Roberto, an intermediary should not coalesce data across a header anyway although this doesn’t currently appear to be required.

Considering that:

1. An empty data frame with END_SEGMENT set, adjacent to a headers sequence, is semantically the same as setting END_SEGMENT in the HEADERS frame, and

2. A headers sequence must flush the data stream if it is to appear to be anywhere in particular, 
 
No expressive power is lost in the protocol by setting END_HEADERS = END_SEGMENT = 2, removing END_SEGMENT from the HEADERS frame, and having intermediaries treat them with equivalent semantics wrt fencing/flushing.

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 21 Apr 2014 22:56:32 +1000

On 17 April 2014 09:46, Jeff Pinner <[email-masked]> wrote:

> You are correct that for a HEADERS frame w/ the END_SEGMENT flag set,
> there is no way a server could coalesce frames because the next frames must
> be CONTINUATION frames until the END_HEADERS flag is received, so in this
​I've just come up against this issue in my implementation. Is it a
protocol ​error to receive a HEADERS with END_SEGMENT set, but END_HEADERS
cleared?

And since the CONTINUATION frame doesn't even have an END_SEGMENT flag, if
we need to send big headers at the end of a segment do we have to do
something like: HEADERS+CONTINUATION(END_HEADERS),
HEADERS(END_HEADERS+END_SEGMENT)?

Blech.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Jeff Pinner <[email-masked]>
Date: Mon, 21 Apr 2014 07:12:53 -0700

On Mon, Apr 21, 2014 at 5:56 AM, Matthew Kerwin <[email-masked]>wrote:

> I've just come up against this issue in my implementation. Is it a
> protocol error to receive a HEADERS with END_SEGMENT set, but END_HEADERS
> cleared?
>
>
Treat it similarly to END_STREAM. That is the END_SEGMENT flag would be set
on the HEADERS frame and not on any CONTINUATION frames that follow it.