Subject: Splitting transport and application error code spaces

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Aug 2017 15:40:08 +1000

A while back we decided to use a single error code space.  But in
https://github.com/quicwg/base-drafts/issues/485, I noticed that we
have an implicit requirement in the protocol not to have the transport
close streams.  If the transport resets streams, it could destroy
critical application state.

The neatest way to enforce the separation of application and transport
is to create an application error space:

  https://github.com/quicwg/base-drafts/pull/722

This splits the error codes for application protocols from the
transport codes.  To do this, it splits CONNECTION_CLOSE into
TRANSPORT_CLOSE and APPLICATION_CLOSE.  These two frames have
identical format and semantics, but use different error code spaces.
RST_STREAM and STOP_SENDING now only carry application error codes,
making it clear that resetting streams is the domain of the
application protocol.

In doing this, I noticed that the error code space is ludicrously
large.  So I have a companion PR that shrinks it to a much more
manageable 16 bits:

  https://github.com/quicwg/base-drafts/pull/723

---

From: Mike Bishop <[email-masked]>
Date: Fri, 11 Aug 2017 14:42:21 +0000

Okay with the general direction, but not wild about the frame split.  In general, nearly-identical frames make me question.

Given that the assigned 0x2 and 0x3 differ by one bit, might it be clearer to have an embedded flag in CONNECTION_CLOSE indicating the source?  That avoids having two frames with identical semantics except the registry in which you look up the value of a field.

Sent from my Windows 10 phone

From: Martin Thomson<mailto:[email-masked]>
Sent: Thursday, August 10, 2017 10:40 PM
To: QUIC WG<mailto:[email-masked]>
Subject: Splitting transport and application error code spaces

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fissues%2F485&data=02%7C01%7Cmichael.bishop%40microsoft.com%7Ce667a5b2f79c451b47f208d4e07b7337%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636380268206522856&sdata=Lo0K3hCLYeiVHx9WIZu3zDU1qp43n4Xc31RirijGbnU%3D&reserved=0, I noticed that we

  https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fpull%2F722&data=02%7C01%7Cmichael.bishop%40microsoft.com%7Ce667a5b2f79c451b47f208d4e07b7337%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636380268206522856&sdata=Fko06fKB9DbI0S58mUz%2BFyVLZ4kcxNt9e5dKRQ1XH5A%3D&reserved=0

  https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fpull%2F723&data=02%7C01%7Cmichael.bishop%40microsoft.com%7Ce667a5b2f79c451b47f208d4e07b7337%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636380268206522856&sdata=qkeITvAPa%2BeAPEB%2FCz7wtmujGcqQnzExH7R01fxCKJc%3D&reserved=0

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Aug 2017 12:39:21 -0700

Personally -- I'm +1 on this. Having a clear distinction is best.

> On 10 Aug 2017, at 10:40 pm, Martin Thomson <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Sat, 12 Aug 2017 20:04:13 +1000

On 12 August 2017 at 00:42, Mike Bishop <[email-masked]> wrote:
> Given that the assigned 0x2 and 0x3 differ by one bit, might it be clearer
> to have an embedded flag in CONNECTION_CLOSE indicating the source?  That
> avoids having two frames with identical semantics except the registry in
> which you look up the value of a field.

It all amounts to the same thing, and I think that separate types is a
better trend generally.  I'm not a huge fan of the mix of discrete
types and bit patterns, preferring the former for the sake of clarity.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 14 Aug 2017 17:00:18 -0700

There's awkwardness in this enforcement. QUIC generates RST_STREAM when it
receives a STOP_SENDING frame, and this PR changes that function quite a
bit. Specifically, it requires that QUIC send RST_STREAM with an app error
code, where this RST_STREAM is in fact triggered by receipt of a
STOP_SENDING frame.

I'm not sure exactly what this split buys us... and it changes behavior on
receiving STOP_SENDING.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 15 Aug 2017 10:48:50 +1000

> I'm not sure exactly what this split buys us... and it changes behavior on

So, what the split buys us is certainty: an application is responsible
for the lifetime of streams.  In return the application can be certain
that the transport won't destroy any state that it puts on streams at
a whim.  The split enforces this separation.

This issue with STOP_SENDING is a good one.  I have a solution to that
as well: move STOP_SENDING to HTTP.  If the required reaction
(RST_STREAM), is an application-layer action, then we should make the
trigger application-layer as well.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 15 Aug 2017 06:52:51 +0000

I'm a little unhappy about that one.  I see the logic, but that would leave each direction of a stream solely in the hands of the sender and the receiver has no way to communicate (at the transport layer) if it wishes that stream to cease.  (Other, perhaps, than starving it via flow control.)  Though I agree, it is consistent with the principle that stream lifetime is entirely in the hands of the application.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Monday, August 14, 2017 5:49 PM
To: Jana Iyengar <[email-masked]>
Cc: Mike Bishop <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Splitting transport and application error code spaces

On 15 August 2017 at 10:00, Jana Iyengar <[email-masked]> wrote:
> I'm not sure exactly what this split buys us... and it changes 
> behavior on receiving STOP_SENDING.

So, what the split buys us is certainty: an application is responsible for the lifetime of streams.  In return the application can be certain that the transport won't destroy any state that it puts on streams at a whim.  The split enforces this separation.

This issue with STOP_SENDING is a good one.  I have a solution to that as well: move STOP_SENDING to HTTP.  If the required reaction (RST_STREAM), is an application-layer action, then we should make the trigger application-layer as well.

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 15 Aug 2017 16:57:04 -0700

I like the idea of moving STOP_SENDING to the application entirely. It's
basically an HTTP/2 thing anyway, so that seems like an excellent clean-up.

Mike: QUIC currently does not use RST_STREAM without the application asking
for it outside of responding to STOP_SENDING; do you think there are
conditions under which it may be useful for the transport to issue one?
STOP_SENDING was the only thing I could think of, but moving it up to HTTP
makes that moot.

On Mon, Aug 14, 2017 at 11:52 PM, Mike Bishop <[email-masked]>
wrote:

> I'm a little unhappy about that one.  I see the logic, but that would
> leave each direction of a stream solely in the hands of the sender and the
> receiver has no way to communicate (at the transport layer) if it wishes
> that stream to cease.  (Other, perhaps, than starving it via flow
> control.)  Though I agree, it is consistent with the principle that stream
> lifetime is entirely in the hands of the application.
>
> So, what the split buys us is certainty: an application is responsible for
> the lifetime of streams.  In return the application can be certain that the
> transport won't destroy any state that it puts on streams at a whim.  The
> split enforces this separation.
>
> This issue with STOP_SENDING is a good one.  I have a solution to that as
> well: move STOP_SENDING to HTTP.  If the required reaction (RST_STREAM), is
> an application-layer action, then we should make the trigger
> application-layer as well.
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 16 Aug 2017 00:32:15 -0700

If STOP_SENDING is moved a basic application will no longer have a language
to communicate intent.
This is perhaps a broader issue - but the question is: what are the minimum
primitives a basic application with no advanced higher level protocol needs
in order to move data on multiple streams? It relates to adding bidi
streams on top of uni streams.

I’m not sure if STOP_SENDING is important in this context, or not, but I
believe it is a worthwhile question.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 16 August 2017 at 01.57.11, Jana Iyengar ([email-masked]) wrote:

---

From: Martin Thomson <[email-masked]>
Date: Thu, 17 Aug 2017 11:46:00 +1000

On 16 August 2017 at 17:32, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> in order to move data on multiple streams? It relates to adding bidi streams
> on top of uni streams.

The assertion here is that STOP_SENDING isn't a necessary primitive.
And it isn't generated or consumed by the transport, even if the
effects (RST_STREAM) are.

It is useful in HTTP, so HTTP can define it.  We might see something
like that if someone ported (for example) websockets to QUIC.  I can't
see it being used for DNS.  I might speculate and say the same about
RTP, but that's so nebulous I'd likely be wrong.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 17 Aug 2017 05:54:15 +0000

The response to STOP_SENDING may be faster if it were a part of the transport (especially if QUIC transport is implemented on a kernel level). An application on a busy system that has written 100K to the transport might not receive and process the STOP_SENDING request and then ask for a stream reset in time to save a large chunk of unnecessary data from being transmitted.

Other than this, I can see how moving this frame to the application is a minor transport simplification.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 17 Aug 2017 02:14:38 -0400

>  An application on a busy system that has written 100K to the transport
might not receive and process the STOP_SENDING request and then ask for a
stream reset in time to save a large chunk of unnecessary data from being
transmitted.

That is a good point. It is rather relevant in high volume async
transmission scenarios. It can easily be much more than 100K via mmap
transmit references. In some cases that application might not even want to
know about the stream after forwarding a file handle to the transport.
STOP_SENDING might be initiated by sub-process termination.

In other words, there can a be simple process/resource control layer
between transport and the application that does not know about data frame
formats but still manage resources. Without STOP_SENDING at transport level
this might be difficult to implement in a generic fashion.

On 17 August 2017 at 07.54.19, Lubashev, Igor ([email-masked]) wrote:

The response to STOP_SENDING may be faster if it were a part of the
transport (especially if QUIC transport is implemented on a kernel level).
An application on a busy system that has written 100K to the transport
might not receive and process the STOP_SENDING request and then ask for a
stream reset in time to save a large chunk of unnecessary data from being
transmitted.

Other than this, I can see how moving this frame to the application is a
minor transport simplification.

-----Original Message-----
*From:* Martin Thomson [[email-masked]]
*Received:* Wednesday, 16 Aug 2017, 9:46PM
*To:* Mikkel Fahnøe Jørgensen [[email-masked]]
*CC:* Jana Iyengar [[email-masked]]; QUIC WG [[email-masked]]; Mike Bishop [
[email-masked]]
*Subject:* Re: Splitting transport and application error code spaces

> in order to move data on multiple streams? It relates to adding bidi
streams

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 17 Aug 2017 12:07:07 +0200

That’s what the applicability draft should capture. If you have concrete things that need to be mentioned somewhere, you can file an issue there. However, as the base specs are still changing quite a bit, it’s a bit hard for me to track all things now…

> Am 16.08.2017 um 09:32 schrieb Mikkel Fahnøe Jørgensen <[email-masked]>:
> 
> If STOP_SENDING is moved a basic application will no longer have a language to communicate intent.
> This is perhaps a broader issue - but the question is: what are the minimum primitives a basic application with no advanced higher level protocol needs in order to move data on multiple streams? It relates to adding bidi streams on top of uni streams.
> 
> I’m not sure if STOP_SENDING is important in this context, or not, but I believe it is a worthwhile question.
> 
> Kind Regards,
> Mikkel Fahnøe Jørgensen
> 
> 
>> I like the idea of moving STOP_SENDING to the application entirely. It's basically an HTTP/2 thing anyway, so that seems like an excellent clean-up.
>> 
>> Mike: QUIC currently does not use RST_STREAM without the application asking for it outside of responding to STOP_SENDING; do you think there are conditions under which it may be useful for the transport to issue one? STOP_SENDING was the only thing I could think of, but moving it up to HTTP makes that moot. 
>> 
>> On Mon, Aug 14, 2017 at 11:52 PM, Mike Bishop <[email-masked]> wrote: