Subject: Re: Unidirectional streams PR

From: Kazuho Oku <[email-masked]>
Date: Fri, 7 Jul 2017 09:57:26 +0900

2017-06-21 16:41 GMT+09:00 Martin Thomson <[email-masked]>:
> I've created a pull request unidirectional streams.
>
> https://github.com/quicwg/base-drafts/pull/643

Thank you for working on the PR.

Going through the PR (and the discussion on this mailing list thread),
I have one question.

Could somebody clarify what a server is expected to do when there are
no more streams than can be used to send a response (i.e. when a
server has used all the Stream IDs up to 2^32-2 and then receives a
request)?

In a protocol that does 1:N mapping between request and response (HTTP
with push can be considered as such a protocol), a server will finish
using all of its streams before the client does. So there is a
possibility that by the time the server observes a new
client-initiated stream carrying a request, it might not have any
unused stream IDs that can be used for sending a response.

I can think of two ways to handle such an issue.

One way is to define a server-sent signal in the application binding
(e.g., HTTP over QUIC) that indicates the client that the request
should be replayed on a different QUIC connection.

The other way is on the client side _guess_ the possibility of server
reaching stream ID starvation, and if there's such a possibility, open
a new connection for sending requests (though making such a guess
might not be easy due to packet reordering).

Going through the PR, I could not find a change that tries to handle
the issue. Hence asking the question.

> I won't go into detail about this here, other than to point out that
> there is extensive rationale for the choices I made in the PR summary,
> a copy of which I will include for your convenience.
>
> ## Transport Changes
>
> Streams are unidirectional.  Each has three states: idle, open, and
> closed.  I separated the transitions for sending and receiving because
> that turned out to be easier to explain.  The reordering thing makes
> them different in subtle ways.
>
> That's all.  The changes in transport are relatively small and they
> simplify streams a lot.  That it also makes the transport more generic
> is a nice bonus.
>
> ## HTTP Changes
>
> This is where the bulk of the changes are.
>
> ### Stream Correlation
>
> Previously request and response were implicitly correlated, as was the
> data correlated with the request or response headers.  With this
> change, that correlation each stream has a header that explicitly
> correlates these.
>
> There are 5 types of stream: connection control, request, response,
> data, and push.  The first two have a simple header that has a type.
> The next two reference another stream in their header, which ties
> request to response and data to request or response.  Push streams
> each reference a PUSH_PROMISE using a newly minted push ID (I decided
> that was cleaner than what I proposed at the interim).
>
> I chose backward references rather than forward references for two
> reasons.  First, request and response correlation can't use forward
> references because that would mean the client would be exerting
> (uncoordinated) control over server streams.  Second, that gives the
> server the most flexibility in terms of how it answers requests (see
> #281).  The cost of using backward references is that endpoints have
> to check that the backward references aren't bad, either referencing
> the wrong type of stream, or with multiple references to the same
> stream.
>
> The presence or absence of a message body is signaled after the
> initial header block using a HAS_BODY frame.  This empty frame
> indicates that another stream will include the body of the message -
> or a promise for a body.  I would like to eliminate this stream split.
> See #245 and #557 for more details on that, though we need to fix #176
> first, which brings us back to QPACK/QCRAM again.
>
> ### Prioritization
>
> This changes prioritization so that it identifies requests.  I only
> made the minimal changes here, which means that this doesn't fix #441
> at the same time, I've left that for later (see below).
>
> ### Cancelling Pushes
>
> Because server push doesn't create streams with PUSH_PROMISE, I had to
> create a way to cancel them between the time that the PUSH_PROMISE is
> sent and when the push stream is created.  That's called RST_PUSH.
>
> ## Things That Need Improvement
>
> I haven't based this on #171.  I think that functionality is good, but
> the last time I looked at the PR I didn't like some of the changes
> Mike made there.  (That's a taste thing.)  This really assumes that we
> accept something very much like #171.
>
> This really needs QPACK/QCRAM.  Right now, it is impossible to cancel
> some types of request using RST_STREAM because not all messages will
> have a body (#176 again).  On balance, given that bodies are what you
> really want to kill, it's not disastrous, but it's a problem
> nonetheless.  I think that we all agree that this is something that we
> need to fix, but we haven't reached the point where we agree on the
> details of the fix.
>
> ## Things That Want Improvement
>
> This also really wants headers and data on the same stream .  It
> doesn't exactly need that, but merging the streams would make things a
> lot easy, both conceptually and structurally.
>
> I chose the simplest possible encoding for all of the fields that I
> touched.  That means that stream IDs are all 32 bits in size and the
> HAS_BODY frame takes an entire 9 octets.  For things that are that
> common, there are many ways in which byte efficiency could be
> improved.
>
> The push ID changes might allow us to trivially fix #441.  Use of an
> as-yet-not-created push ID as a node in the priority tree would allow
> for prioritization to use "empty" nodes.  We'd need to explicitly
> allow that though.
>
> # Fixes
>
> Closes #515, #240, #281, #175.
>

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Fri, 7 Jul 2017 11:05:41 +1000

> server has used all the Stream IDs up to 2^32-2 and then receives a

I think that the only sensible approach is to give up on the
connection before this point is reached.

A client can't know exactly how many streams a server has open, so I
think that the only sensible course of action is to initiate graceful
shutdown whenever the endpoint might be inclined to MAX_STREAM_ID that
is 2^32-1 or greater.

(In the PR, the limit is 2^32-1.)

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 7 Jul 2017 10:43:31 +0900

> (In the PR, the limit is 2^32-1.)

Thank you for the clarification.

So it would be sensible to stop initiating new requests on an existing
connection when the number of unused stream IDs of the peer becomes as
low as max-concurrent-streams multiplied by some safety factor
(something like 10 would be enough, considering the probability of
loss or reordering of packets carrying DATA frames and consecutive
MAX_STREAM_ID frames).

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Fri, 7 Jul 2017 12:04:59 +1000

> loss or reordering of packets carrying DATA frames and consecutive

Yeah, you have to allow for the possibility of the server wanting to
push some number of times in response to each of your requests.  I was
originally going to say that you hit the wall only once you
MAX_STREAM_ID hits the limit, but the server can always promise in
excess of this limit.  If you want those promises to be resolved, then
allowing more space is reasonable.  10 sounds reasonable, but we can't
know the distribution of pushes to requests (today it is close to 0 in
the aggregate, but this could change).

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 7 Jul 2017 11:21:11 +0900

> the aggregate, but this could change).

That's a good point.

Considering the complexity of guessing when the server runs out of
stream IDs, it might be easier if we could use a server-sent
CANCEL_REQUEST frame as an indication that the client needs to resend
the request on a new connection.

-- 
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Fri, 7 Jul 2017 13:33:50 +1000

> the request on a new connection.

If we ever get to the point where we have a good graceful shutdown,
then that should mostly cover this.  CANCEL_REQUEST is likely still
useful here to cover reordering.

Here's my thinking on this: the server sends a GOAWAY identifying the
highest numbered request that it has accepted (requests, not pushes).
Anything higher than this will not be processed.  The connection goes
away when responses to all those requests have been delivered,
including all the promises that attach to those.

The only trick here is that the server might be able to answer a
request on stream X, but find that it can't answer X-1, which could
arrive out of order.  So, yeah, CANCEL_REQUEST is a good fit for that.
RST_STREAM isn't useful here because the stream can be closed when the
server makes this discovery.

Either way, having the server being able to cancel requests is good
practice, because it might not be in a position to use RST_STREAM.

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 7 Jul 2017 14:05:01 +0900

> practice, because it might not be in a position to use RST_STREAM.

Thank you for the answer. I agree with your observations.

In https://mailarchive.ietf.org/arch/msg/quic/9VqjoX4acCDrxMKV2AmOQjspe3Q
Igor has suggested using DISINTEREST frame for cancelling requests
from the client-side. In the end, it might make sense for the
uni-directional-stream-only approach to have a frame that can be sent
in any direction for cancelling a request.

-- 
Kazuho Oku