Subject: Closing streams

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 5 Dec 2016 18:03:31 +0100

And finally (one more hi!),

while reading the draft, saying that the stream handling is really light-way, I was asking myself if we really need to close streams at all (given that we also just implicitly open them). When you look at section 8.1 (Life of a stream) the overhead of closing streams is actually quite large (couple of additional states only needed for this). (Btw. I also don’t really understand what the difference between a reserved and idle stream is… i guess that’s rather an interface question than state management, anyway…). To my understanding, you close connection for state management at the receiver side. However, as you announce the maximum number of streams to the other endpoint and you can use the WINDOW_UPDATE to send a zero window to make sure no additional data are send (and use BLOCK to reset-up state on the receiver if needed), I think it is a local decision at the receiver which streams to remove state for if the max number of streams is reached. I don’t really see the additional benefit the FIN bits are giving. Or has this proven to be really helpful in some real-life situation?

Mirja

---

From: Mike Bishop <[email-masked]>
Date: Mon, 5 Dec 2016 18:52:52 +0000

"Reserved" is a concept drawn from HTTP/2 that has trickled into QUIC.  Whether it needs to exist at the transport layer is a reasonable question.  In general terms, "reserved" means that you know what the peer is going to send you on that stream (from the application layer), but they haven't actually done so yet.  Now that you raise the question, I don't see a reason that couldn't be an application-layer concept and let the transport continue to see those streams as "idle" -- does anyone else?

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Monday, December 5, 2016 9:04 AM
To: [email-masked]
Subject: Closing streams

Mirja

---

From: Ian Swett <[email-masked]>
Date: Mon, 5 Dec 2016 14:01:10 -0500

A key to making streams lightweight is to ensure they're used in order.
Otherwise tracking which streams have been previously used and closed can
consume a large amount of memory.  So I believe the streams have to be
reserved or have another way to ensure they're used in order(at least
approximately).  Of course, it'd be easy enough to immediately reset the
stream to indicate you're just skipping it.

On Mon, Dec 5, 2016 at 1:52 PM, Mike Bishop <[email-masked]>
wrote:

> "Reserved" is a concept drawn from HTTP/2 that has trickled into QUIC.
> Whether it needs to exist at the transport layer is a reasonable question.
> In general terms, "reserved" means that you know what the peer is going to
> send you on that stream (from the application layer), but they haven't
> actually done so yet.  Now that you raise the question, I don't see a
> reason that couldn't be an application-layer concept and let the transport
> continue to see those streams as "idle" -- does anyone else?
>
> while reading the draft, saying that the stream handling is really
> light-way, I was asking myself if we really need to close streams at all
> (given that we also just implicitly open them). When you look at section
> 8.1 (Life of a stream) the overhead of closing streams is actually quite
> large (couple of additional states only needed for this). (Btw. I also
> don’t really understand what the difference between a reserved and idle
> stream is… i guess that’s rather an interface question than state
> management, anyway…). To my understanding, you close connection for state
> management at the receiver side. However, as you announce the maximum
> number of streams to the other endpoint and you can use the WINDOW_UPDATE
> to send a zero window to make sure no additional data are send (and use
> BLOCK to reset-up state on the receiver if needed), I think it is a local
> decision at the receiver which streams to remove state for if the max
> number of streams is reached. I don’t really see the additional benefit the
> FIN bits are giving. Or has this proven to be really helpful in some
> real-life situation?
>
> Mirja
>

---

From: Ian Swett <[email-masked]>
Date: Mon, 5 Dec 2016 17:21:04 -0500

As a high level comment, I think the current QUIC stream state machine is
quite complex, and it'd be great to simplify it.  I expect it's possible,
but don't have any concrete suggestions at the moment.

---

From: Mike Bishop <[email-masked]>
Date: Mon, 5 Dec 2016 22:42:43 +0000

Whether they’re used in order or not, there are some very long-lived streams and some very short-lived ones, as I’m sure you’ve seen.  One of the drawbacks to the two-stream-per-request mapping in HTTP, in fact, is how that maps to prioritization schemes that use “zombie” streams as placeholders for certain categories of resource.  In the current pull request, if you close the QUIC stream, you lose the ability to reprioritize it later.  (One solution in the HTTP mapping is to move PRIORITY to be a session-level frame back on stream 3, and includes both the prioritized and dependent stream IDs.)  Even so, you’ll have streams 1 and 3 living forever.  It seems like for state management, you need to be able to track contiguous regions in the same state, and you want those regions to be large so you have as few distinct records as possible.

In terms of “reserved,” that seems like it could easily be an implementation detail.  As I envision the API, at least, (haven’t looked at yours), the app probably asks the QUIC stack for a handle to a new stream (then queries to discover the stream ID), or asks the QUIC stack for a handle to a particular stream ID.  “Reserved” seems like it’s basically the stack’s counter for the first instance – even if caller #1 hasn’t actually sent data yet, you shouldn’t hand the same stream to caller #2 when they ask for a new stream.  If that’s really what it winds up being, then it doesn’t belong in the draft.  Another argument in that direction is that nothing is sent on the wire, so the other side doesn’t know it’s reserved (except if the app layer chooses to tell them).

From: Ian Swett [mailto:[email-masked]]
Sent: Monday, December 5, 2016 2:21 PM
To: Mike Bishop <[email-masked]>
Cc: Mirja Kühlewind <[email-masked]>; [email-masked]
Subject: Re: Closing streams

As a high level comment, I think the current QUIC stream state machine is quite complex, and it'd be great to simplify it.  I expect it's possible, but don't have any concrete suggestions at the moment.

On Mon, Dec 5, 2016 at 2:01 PM, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:
A key to making streams lightweight is to ensure they're used in order.  Otherwise tracking which streams have been previously used and closed can consume a large amount of memory.  So I believe the streams have to be reserved or have another way to ensure they're used in order(at least approximately).  Of course, it'd be easy enough to immediately reset the stream to indicate you're just skipping it.

On Mon, Dec 5, 2016 at 1:52 PM, Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
"Reserved" is a concept drawn from HTTP/2 that has trickled into QUIC.  Whether it needs to exist at the transport layer is a reasonable question.  In general terms, "reserved" means that you know what the peer is going to send you on that stream (from the application layer), but they haven't actually done so yet.  Now that you raise the question, I don't see a reason that couldn't be an application-layer concept and let the transport continue to see those streams as "idle" -- does anyone else?

-----Original Message-----
From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mirja Kühlewind
Sent: Monday, December 5, 2016 9:04 AM
To: [email-masked]<mailto:[email-masked]>
Subject: Closing streams

Mirja

---

From: Ian Swett <[email-masked]>
Date: Mon, 5 Dec 2016 18:27:20 -0500

A few long lived streams are fine, even if they're completely
discontiguous.  It's very simple to track open and closed stream with an
"everything under this max stream value is closed, unless it's contained in
this map" approach.  If streams aren't used or reserved in order, then one
would have to store every stream ever used, open or closed, which could
consume a large amount of memory and is not friendly to long-running
connections.

The current QUIC draft does not allow streams to be re-opened once their
closed.  I suppose an alternative is to allow streams to be closed and
re-opened later, but that brings in new challenges like the creator of the
stream having to ensure all data and the fin, or the reset, are acked
before re-using a stream.  A buggy implementation could easily cause data
corruption.

It

On Mon, Dec 5, 2016 at 5:42 PM, Mike Bishop <[email-masked]>
wrote:

> Whether they’re used in order or not, there are some very long-lived
> streams and some very short-lived ones, as I’m sure you’ve seen.  One of
> the drawbacks to the two-stream-per-request mapping in HTTP, in fact, is
> how that maps to prioritization schemes that use “zombie” streams as
> placeholders for certain categories of resource.  In the current pull
> request, if you close the QUIC stream, you lose the ability to reprioritize
> it later.  (One solution in the HTTP mapping is to move PRIORITY to be a
> session-level frame back on stream 3, and includes both the prioritized and
> dependent stream IDs.)  Even so, you’ll have streams 1 and 3 living
> forever.  It seems like for state management, you need to be able to track
> contiguous regions in the same state, and you want those regions to be
> large so you have as few distinct records as possible.
>
>
>
> In terms of “reserved,” that seems like it could easily be an
> implementation detail.  As I envision the API, at least, (haven’t looked at
> yours), the app probably asks the QUIC stack for a handle to a new stream
> (then queries to discover the stream ID), or asks the QUIC stack for a
> handle to a particular stream ID.  “Reserved” seems like it’s basically the
> stack’s counter for the first instance – even if caller #1 hasn’t actually
> sent data yet, you shouldn’t hand the same stream to caller #2 when they
> ask for a new stream.  If that’s really what it winds up being, then it
> doesn’t belong in the draft.  Another argument in that direction is that
> nothing is sent on the wire, so the other side doesn’t know it’s reserved
> (except if the app layer chooses to tell them).
>
>
>
> *From:* Ian Swett [mailto:[email-masked]]
> *Sent:* Monday, December 5, 2016 2:21 PM
> *To:* Mike Bishop <[email-masked]>
> *Cc:* Mirja Kühlewind <[email-masked]>; [email-masked]
> *Subject:* Re: Closing streams
>
>
>

---

From: Martin Thomson <[email-masked]>
Date: Wed, 7 Dec 2016 06:43:21 +0900

> but don't have any concrete suggestions at the moment.

One way to do this is to completely detach streams in both directions.
Then have response streams explicitly identify the request that they
are responding to.  Possibly at the usage layer.  I think that avoids
much of the complexity associated with reservations, but it likely
adds new and interesting twists to consider.

One twist that I'm particularly concerned about is congruence with h2.
But then Mike's proposals have already taken us away from that; maybe
that can be repaired in time, but we should be aware of the cost of
divergence.  To be explicit, we've opened several new fronts just by
taking this work on; reworking the request-response stream mapping
opens another.

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 6 Dec 2016 15:04:20 -0800

Catching up -- just a couple of thoughts.

Reserving streams did come from RFC 7540 and is meant for PUSH streams
primarily. As Ian notes, one of the ways of keeping stream-handling
light-weight is by using in-order stream creation. We didn't need to
distinguish between local and remote "reserved" states, as is done in RFC
7540, because there's no notion of PUSH in QUIC. This means PUSH messages
are sent at the HTTP level, which then turns into the HTTP layer reserving
streams at the QUIC layer through the reserve_stream primitive. The
Reserved state is only useful at the side that creates a stream, for
enforcement of "Streams MUST be created or reserved in sequential order"
(Sec 8.2). Incoming streams are allowed to be created in arbitrary order,
to allow for network reordering of incoming stream frames, and so, a PUSHed
stream does not need to be reserved at the receiver.

The state machine is basically about as complex as the HTTP/2 stream state
machine... I'm not too concerned about it. Mike's proposal does not take us
away from the congruence by much... the 2-stream mapping approach basically
means that every HTTP/2 stream becomes 2 QUIC streams. If you were to PUSH
an HTTP/2 stream, this would move the HTTP/2 stream into the "reserve
(local)" state, and the HTTP layer would then cause the 2 corresponding
QUIC streams to go into the "reserved" state (I would expect this mapping
to be static -- something like HTTP/2 stream ID x 2 and HTTP/2 stream ID  x
2 + 1.)

---

From: "Christian Huitema" <[email-masked]>
Date: Tue, 6 Dec 2016 15:06:53 -0800

> 
> As a high level comment, I think the current QUIC stream state machine is quite 
> complex, and it'd be great to simplify it.  I expect it's possible, but don't have any 
> concrete suggestions at the moment.

I did not find it particularly complex when implementing it. Basically, the spec says open, send, receive, abort and close. The only complex part is that frames can be received out of order, and that the error correction mechanism supposes an interaction between the frame level and the packet level. But even that can be managed rather easily.

There is however another issue, related to multihoming. Can we split a given stream over several packet level connections? The reasons for doing that are largely the same as splitting an application stream over several connections in MP-TCP. Each independent packet level connection can be parametrized based on its underlying path. Different paths might get for example different congestion controls. An application that sends one big stream of data could send part of the stream over each connection. If the stream level itself handles that, we get nice properties like being able to send the error corrections on a different connection than the original packet.

-- Christian Huitema

---

From: Mike Bishop <[email-masked]>
Date: Tue, 6 Dec 2016 23:24:14 +0000

Actually, in my proposal, there are no HTTP-layer stream IDs.  It directly references the QUIC stream ID in question (well, references the first stream of the pair anywhere something else needs to point to it).  I’d really like to keep away from having distinct stream IDs at two layers, even if there is a static mapping from one to the other.  (Of course, an implementation could still do so internally, if desired.)

From: Jana Iyengar [mailto:[email-masked]]
Sent: Tuesday, December 6, 2016 3:04 PM
To: Martin Thomson <[email-masked]>
Cc: Ian Swett <[email-masked]>; Mike Bishop <[email-masked]>; [email-masked]; Mirja Kühlewind <[email-masked]>
Subject: Re: Closing streams

Reserving streams did come from RFC 7540 and is meant for PUSH streams primarily. As Ian notes, one of the ways of keeping stream-handling light-weight is by using in-order stream creation. We didn't need to distinguish between local and remote "reserved" states, as is done in RFC 7540, because there's no notion of PUSH in QUIC. This means PUSH messages are sent at the HTTP level, which then turns into the HTTP layer reserving streams at the QUIC layer through the reserve_stream primitive. The Reserved state is only useful at the side that creates a stream, for enforcement of "Streams MUST be created or reserved in sequential order" (Sec 8.2). Incoming streams are allowed to be created in arbitrary order, to allow for network reordering of incoming stream frames, and so, a PUSHed stream does not need to be reserved at the receiver.

The state machine is basically about as complex as the HTTP/2 stream state machine... I'm not too concerned about it. Mike's proposal does not take us away from the congruence by much... the 2-stream mapping approach basically means that every HTTP/2 stream becomes 2 QUIC streams. If you were to PUSH an HTTP/2 stream, this would move the HTTP/2 stream into the "reserve (local)" state, and the HTTP layer would then cause the 2 corresponding QUIC streams to go into the "reserved" state (I would expect this mapping to be static -- something like HTTP/2 stream ID x 2 and HTTP/2 stream ID  x 2 + 1.)

On Tue, Dec 6, 2016 at 1:43 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 6 Dec 2016 15:43:53 -0800

On Tue, Dec 6, 2016 at 3:24 PM, Mike Bishop <[email-masked]>
wrote:

> Actually, in my proposal, there are no HTTP-layer stream IDs.  It directly
> references the QUIC stream ID in question (well, references the first
> stream of the pair anywhere something else needs to point to it).  I’d
> really like to keep away from having distinct stream IDs at two layers,
> even if there is a static mapping from one to the other.  (Of course, an
> implementation could still do so internally, if desired.)
>

Right, good point. In the abstract though, the idea remains the same --
where you would have seen HTTP/2 stream ID Y in the HTTP/2 world, you now
see QUIC stream ID Yx2 and Yx2+1.

>
>
> *From:* Jana Iyengar [mailto:[email-masked]]
> *Sent:* Tuesday, December 6, 2016 3:04 PM
> *To:* Martin Thomson <[email-masked]>
> *Cc:* Ian Swett <[email-masked]>; Mike Bishop <
> [email-masked]>; [email-masked]; Mirja Kühlewind <
> [email-masked]>