Subject: RE: QUIC for robotics and other time-sensitive applications

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 1 May 2019 14:33:53 +0000

John,

I am happy to resurrect that draft-lubashev-quic-partial-reliability for QUIC v2 (some people preferred version 02, which actually gets you a bit more features at the cost of a bit more complexity).

I would like to first understand what partially reliable streams give you that is not available with regular streams.  I.e. what are you missing if you use STREAM (with a possible subsequent RESET) for messages?  (You cannot use DATAGRAM for messages, since you would like the transport to facilitate retransmission until expiration.)

My draft-lubashev-quic-partial-reliability lists a few things that stream-per-message does not get you:

  1.  If you have multiple application message streams, you would need a header for each QUIC stream to identify the application stream.  This seems like a small burden, except for very short messages.
     *   If the order of messages in an application stream matters, you would need to reorder them yourself.  This cannot be very complicated for the application.

  1.  QUIC per-application-stream flow control is rendered ineffective.  I am actually not a huge fan of QUIC per-stream flow control, as I strongly prefer stream priorities as a signal.  But I now know of a case when per-stream flow control is, indeed useful (when there is only one slow draining stream that uses up the entire connection buffer before a higher-priority stream shows up).

Are there any other benefits you see in partially reliable streams?

  *   Igor

From: John Wason <[email-masked]>
Sent: Tuesday, April 30, 2019 5:52 PM
To: Ted Hardie <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: QUIC for robotics and other time-sensitive applications

Thanks Ted, this looks very interesting. Based on these documents the DETNET sits below the level of QUIC that we are currently discussing and would mainly affect the IP header format of the resulting UDP packets. The QUIC protocol will still require some form of partial reliability to function on a DETNET network.

    -John Wason
On 4/30/2019 5:37 PM, Ted Hardie wrote:
Hi John,

Thanks for the message.  If you have not already reviewed the work of DETNET (https://datatracker.ietf.org/wg/detnet/about/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_wg_detnet_about_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=xLXCzRykwXzX7bQ_EfAh4Rea2sEzel9wgnprQAuwPwc&e=>) here in the IETF, it might be useful to do so, as there are a number of elements of your use case which appeared to be shared with their work.  In particular, it might be useful to consider whether the integration of their proposed IP data plane (https://datatracker.ietf.org/doc/draft-ietf-detnet-dp-sol-ip/?include_text=1<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dietf-2Ddetnet-2Ddp-2Dsol-2Dip_-3Finclude-5Ftext-3D1&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=TIzKr-IO44yjBfPhhPp5mldz8QENRQGE-4WD7bbdn5k&e=>) with QUIC would be valuable in achieving your timing goals.

best regard,

Ted Hardie

On Tue, Apr 30, 2019 at 12:15 PM John Wason <[email-masked]<mailto:[email-masked]>> wrote:
Greetings QUIC WG,

The discussion on Issue #92 of webrtc-quic suggested that I post my
requirements to this list.

I am a robotics engineer and the developer of Robot Raconteur, a
communication framework for robotics and automation systems
(https://github.com/robotraconteur/robotraconteur<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_robotraconteur_robotraconteur&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=VIzDVXjr63UwHbT0rvYCzS91VAZarySeWftFNCsC2hE&e=>). Robotics
applications require different data types and QoS when compared to
business or web applications. Robotics applications need real-time
communication of isochronous data, for instance constant updates on the
position of the joints of the robot, alongside generic
non-time-sensitive data. Currently most robotics applications are using
TCP (Robot Raconteur, ROS*, OPC/UA), UDP (DDS, ROS2, ROS*), or some form
of custom Ethernet protocol (Ethernet/IP, EtherCAT, PowerLink). To add
more confusion to the situation, there is the recent introduction of
IEEE 802.1Q Time Sensitive Networking (TSN) which claims to allow TCP
and UDP to achieve hard real-time performance.

While there are many robotics protocols out there, only Robot Raconteur
and OPC/UA are compatible with existing web and cloud infrastructure.
(Some limited support for ROS/ROS2 has become available on AWS, but the
design of ROS does not play very well outside of local networks.*) Robot
Raconteur implements web browser and web server support by allowing the
websocket protocol to be used. Clients can use a URL that specifies that
the connection is a websocket, and incoming TCP connections can detect
if the connection is the Robot Raconteur protocol or an incoming
websocket connection. Robot Raconteur running inside an ASP.NET<https://urldefense.proofpoint.com/v2/url?u=http-3A__ASP.NET&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=9uchBdRzcuqtIB6lZuT0wlGoelFWgIzMVT4hTBGEVxc&e=> web
server has been demonstrated, and there is no technical reason that
other server technologies cannot accept Robot Raconteur connections
through websockets. (OPC/UA uses SOAP for its web based transport, but
support is very poor and only available in a few implementations.)

The main issue limiting Robot Raconteur for more demanding applications
is the use of TCP for data transport. TCP suffers from "head-of-line"
blocking, which means that if stale data encounters an error it can stop
fresh data from reaching the other peer. There is also the problem of
large messages blocking the transmission of smaller, more time sensitive
messages on the connection. QUIC appears to solve most of these
problems, and the webrtc-quic support under development provides a lot
of new and interesting opportunities for integrating robotics
applications with web technology. By adopting QUIC, the resulting
software will remain completely compatible with web infrastructure.

Transmitting time-sensitive data requires the ability to multiplex (and
prioritize) messages and drop messages if the data contained becomes
stale. The multiplexing issue is solved using the "streams as messages"
concept in QUIC, with built in priority handling. This design looks very
promising, and considerably simpler to use compared to SCTP streams. The
current issue that is somewhat questionable is the implementation of
partial reliability. The discussion right now is focused on DATAGRAM
extensions (draft-pauly-quic-datagram-02), which allow for "raw" access
to the socket so protocols such as RTP can tunnel through QUIC. This
requirement is quite a bit different than what is required for an
application like Robot Raconteur. Robot Raconteur is based on message
passing, where either the entire message is received or the entire
message is cancelled because the data contained is stale. The message
may be considerably larger than MTU, and the QoS may be "partial,"
meaning that a set number of attempts may be made to resend, and/or
there is an expiration time on the message. This capability is discussed
in draft-lubashev-quic-partial-reliability-03, and is similar to how
SCTP handles partial reliability. If only a raw DATAGRAM capability is
provided, then partial reliability will need to be built on top of QUIC,
defeating the purpose of using QUIC for this application. Other
applications such as online gaming have very similar performance
requirements, so I think that the partial reliability capability will be
important to many future users.

The complexity and required effort to add the partial reliability
capability to QUIC is low, so I hope that this capability is not abandoned.

Robot Raconteur has received a grant from the United States ARM
Institute and New York State to begin commercializing the technology.
This project will be done in partnership with Wason Technology, LLC,
Rensselaer Polytechnic Institute, Southwest Research Institute, and
United Technologies Research Center. The project starts soon, and
considerable development effort will be invested in the Robot Raconteur
technology starting early summer. Integrating QUIC is one of the tasks
that I hope to undertake during this project.

Please let me know if you have any questions about Robot Raconteur or
the requirements for QUIC in robotics applications.

     -John Wason

(* I am aware that there are some caveats to how ROS/ROS2 interact with
the wider world, but the technologies are not built on or compatible
with web standards so an in depth discussion is not relevant for the
current topic. ROS does have a UDP transport, but it is not commonly used.)

--
John Wason, Ph.D.
Wason Technology, LLC
16 Sterling Lake Rd
Tuxedo, NY 10987
(518) 279-6234
[email-masked]<mailto:[email-masked]>

--

John Wason, Ph.D.

Wason Technology, LLC

16 Sterling Lake Rd

Tuxedo, NY 10987

(518) 279-6234

[email-masked]<mailto:[email-masked]>

---

From: John Wason <[email-masked]>
Date: Wed, 1 May 2019 12:22:02 -0400

Igor,

It appears I misunderstood the distinction between per-stream partial 
reliability and intra-stream partial reliability. If I send one 
partial-reliability message per stream, using RESET frames should be 
sufficient. How this is exposed as an API is important, since I am not 
sure the webrtc-quic team has considered this yet. In response to your 
questions 1) Each message contains a header that should be sufficient 
for the receiver to understand the incoming stream 2) Does QUIC provide 
the type of priority that you suggest? My understanding is that QUIC 
multiplexes streams using a priority based ordering.

If I can emulate partial-reliability using RESET frames to cancel entire 
streams, that should be sufficient for my needs.

     -John

On 5/1/2019 10:33 AM, Lubashev, Igor wrote:
>
> John,
>
> I am happy to resurrect that draft-lubashev-quic-partial-reliability 
> for QUIC v2 (some people preferred version 02, which actually gets you 
> a bit more features at the cost of a bit more complexity).
>
> I would like to first understand what partially reliable streams give 
> you that is not available with regular streams.  I.e. what are you 
> missing if you use STREAM (with a possible subsequent RESET) for 
> messages?  (You cannot use DATAGRAM for messages, since you would like 
> the transport to facilitate retransmission until expiration.)
>
>  1. If you have multiple application message streams, you would need a
>     header for each QUIC stream to identify the application stream. 
>     This seems like a small burden, except for very short messages.
>      1. If the order of messages in an application stream matters, you
>         would need to reorder them yourself.  This cannot be very
>         complicated for the application.
>
>  2. QUIC per-application-stream flow control is rendered ineffective. 
>     I am actually not a huge fan of QUIC per-stream flow control, as I
>     strongly prefer stream priorities as a signal.  But I now know of
>     a case when per-stream flow control is, indeed useful (when there
>     is only one slow draining stream that uses up the entire
>     connection buffer before a higher-priority stream shows up).
>
> *From:*John Wason <[email-masked]>
> *Sent:* Tuesday, April 30, 2019 5:52 PM
> *To:* Ted Hardie <[email-masked]>
> *Cc:* IETF QUIC WG <[email-masked]>
> *Subject:* Re: QUIC for robotics and other time-sensitive applications
>
> Thanks Ted, this looks very interesting. Based on these documents the 
> DETNET sits below the level of QUIC that we are currently discussing 
> and would mainly affect the IP header format of the resulting UDP 
> packets. The QUIC protocol will still require some form of partial 
> reliability to function on a DETNET network.
>
>     -John Wason
>
>     Thanks for the message.  If you have not already reviewed the work
>     of DETNET (https://datatracker.ietf.org/wg/detnet/about/
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_wg_detnet_about_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=xLXCzRykwXzX7bQ_EfAh4Rea2sEzel9wgnprQAuwPwc&e=>)
>     here in the IETF, it might be useful to do so, as there are a
>     number of elements of your use case which appeared to be shared
>     with their work.  In particular, it might be useful to consider
>     whether the integration of their proposed IP data plane
>     (https://datatracker.ietf.org/doc/draft-ietf-detnet-dp-sol-ip/?include_text=1
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dietf-2Ddetnet-2Ddp-2Dsol-2Dip_-3Finclude-5Ftext-3D1&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=TIzKr-IO44yjBfPhhPp5mldz8QENRQGE-4WD7bbdn5k&e=>)
>     with QUIC would be valuable in achieving your timing goals.
>
>     best regard,
>
>     Ted Hardie
>
>         (https://github.com/robotraconteur/robotraconteur
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_robotraconteur_robotraconteur&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=VIzDVXjr63UwHbT0rvYCzS91VAZarySeWftFNCsC2hE&e=>).
>         Robotics
>         websocket connection. Robot Raconteur running inside an
>         ASP.NET
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__ASP.NET&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=9uchBdRzcuqtIB6lZuT0wlGoelFWgIzMVT4hTBGEVxc&e=>
>         web
>         [email-masked] <mailto:[email-masked]>
>
> -- 
> John Wason, Ph.D.
> Wason Technology, LLC
> 16 Sterling Lake Rd
> Tuxedo, NY 10987
> (518) 279-6234
> [email-masked]  <mailto:[email-masked]>

-- 
John Wason, Ph.D.
Wason Technology, LLC
16 Sterling Lake Rd
Tuxedo, NY 10987
(518) 279-6234
[email-masked]

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 1 May 2019 09:38:17 -0700

QUIC does not prioritise streams at the transport layer, that is an
implementation detail although pacing is recommended. If you bring you own
application layer protocol you can decide how to schedule data across
streams within the flow control constraints. You could add priorities to
your application level custom API. If you layer a protocol on top of H3 you
are more constrained since you must assume other implementations that won’t
necessarily do what you want them to do.

On 1 May 2019 at 18.22.22, John Wason ([email-masked]) wrote:

Igor,

It appears I misunderstood the distinction between per-stream partial
reliability and intra-stream partial reliability. If I send one
partial-reliability message per stream, using RESET frames should be
sufficient. How this is exposed as an API is important, since I am not sure
the webrtc-quic team has considered this yet. In response to your questions
1) Each message contains a header that should be sufficient for the
receiver to understand the incoming stream 2) Does QUIC provide the type of
priority that you suggest? My understanding is that QUIC multiplexes
streams using a priority based ordering.

    -John

John,

I am happy to resurrect that draft-lubashev-quic-partial-reliability for
QUIC v2 (some people preferred version 02, which actually gets you a bit
more features at the cost of a bit more complexity).

I would like to first understand what partially reliable streams give you
that is not available with regular streams.  I.e. what are you missing if
you use STREAM (with a possible subsequent RESET) for messages?  (You
cannot use DATAGRAM for messages, since you would like the transport to
facilitate retransmission until expiration.)

My draft-lubashev-quic-partial-reliability lists a few things that
stream-per-message does not get you:

   1. If you have multiple application message streams, you would need a
   header for each QUIC stream to identify the application stream.  This seems
   like a small burden, except for very short messages.
      1. If the order of messages in an application stream matters, you
      would need to reorder them yourself.  This cannot be very complicated for
      the application.

   1. QUIC per-application-stream flow control is rendered ineffective.  I
   am actually not a huge fan of QUIC per-stream flow control, as I strongly
   prefer stream priorities as a signal.  But I now know of a case when
   per-stream flow control is, indeed useful (when there is only one slow
   draining stream that uses up the entire connection buffer before a
   higher-priority stream shows up).

   - Igor

*From:* John Wason <[email-masked]> <[email-masked]>
*Sent:* Tuesday, April 30, 2019 5:52 PM
*To:* Ted Hardie <[email-masked]> <[email-masked]>
*Cc:* IETF QUIC WG <[email-masked]> <[email-masked]>
*Subject:* Re: QUIC for robotics and other time-sensitive applications

Thanks Ted, this looks very interesting. Based on these documents the
DETNET sits below the level of QUIC that we are currently discussing and
would mainly affect the IP header format of the resulting UDP packets. The
QUIC protocol will still require some form of partial reliability to
function on a DETNET network.

    -John Wason

Hi John,

Thanks for the message.  If you have not already reviewed the work of
DETNET (https://datatracker.ietf.org/wg/detnet/about/
<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_wg_detnet_about_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=xLXCzRykwXzX7bQ_EfAh4Rea2sEzel9wgnprQAuwPwc&e=>)
here in the IETF, it might be useful to do so, as there are a number of
elements of your use case which appeared to be shared with their work.  In
particular, it might be useful to consider whether the integration of their
proposed IP data plane (
https://datatracker.ietf.org/doc/draft-ietf-detnet-dp-sol-ip/?include_text=1
<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dietf-2Ddetnet-2Ddp-2Dsol-2Dip_-3Finclude-5Ftext-3D1&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=ueq3YmF7UajwgCbtLSplbhn7Ip05wFBM20Kbujzjih0&s=TIzKr-IO44yjBfPhhPp5mldz8QENRQGE-4WD7bbdn5k&e=>)
with QUIC would be valuable in achieving your timing goals.

best regard,

Ted Hardie

On Tue, Apr 30, 2019 at 12:15 PM John Wason <[email-masked]> wrote:

Greetings QUIC WG,

websocket connection. Robot Raconteur running inside an ASP.NET

     -John Wason

-- 
John Wason, Ph.D.
Wason Technology, LLC
16 Sterling Lake Rd
Tuxedo, NY 10987
(518) 279-6234
[email-masked]

-- 

John Wason, Ph.D.

Wason Technology, LLC

16 Sterling Lake Rd

Tuxedo, NY 10987

(518) 279-6234

[email-masked]

--
John Wason, Ph.D.
Wason Technology, LLC
16 Sterling Lake Rd
Tuxedo, NY 10987
(518) [email-masked]

---

From: John Wason <[email-masked]>
Date: Wed, 1 May 2019 13:33:25 -0400

Mikkel,

That makes sense, the order of streams sent shouldn't affect the wire 
protocol. For the flow control, does the protocol provide information 
about how much headroom is left? For instance, it may be very important 
that small, time sensitive streams reach their destination quickly, 
while there may be larger, slower streams that can wait. If the flow 
control is not handled correctly, the slower stream could block the fast 
streams by filling the available bandwidth and buffers.

I am not sure what you mean by "layer on top of H3".

     -John

On 5/1/2019 12:38 PM, Mikkel Fahnøe Jørgensen wrote:
> implementation detail although pacing is recommended. If you bring you 
> own application layer protocol you can decide how to schedule data 
> across streams within the flow control constraints. You could add 
> priorities to your application level custom API. If you layer a 
> protocol on top of H3 you are more constrained since you must assume 
> other implementations that won’t necessarily do what you want them to do.
>
>
> On 1 May 2019 at 18.22.22, John Wason ([email-masked] 
> <mailto:[email-masked]>) wrote:
>
>> Igor,
>>
>> sufficient. How this is exposed as an API is important, since I am 
>> not sure the webrtc-quic team has considered this yet. In response to 
>> your questions 1) Each message contains a header that should be 
>> sufficient for the receiver to understand the incoming stream 2) Does 
>> QUIC provide the type of priority that you suggest? My understanding 
>> is that QUIC multiplexes streams using a priority based ordering.
>>
>> If I can emulate partial-reliability using RESET frames to cancel 
>> entire streams, that should be sufficient for my needs.
>>
>>     -John
>>
>>> for QUIC v2 (some people preferred version 02, which actually gets 
>>> you a bit more features at the cost of a bit more complexity).
>>>
>>> I would like to first understand what partially reliable streams 
>>> give you that is not available with regular streams.  I.e. what are 
>>> you missing if you use STREAM (with a possible subsequent RESET) for 
>>> messages?  (You cannot use DATAGRAM for messages, since you would 
>>> like the transport to facilitate retransmission until expiration.)
>>>
>>>  1. If you have multiple application message streams, you would need
>>>     a header for each QUIC stream to identify the application
>>>     stream.  This seems like a small burden, except for very short
>>>     messages.
>>>      1. If the order of messages in an application stream matters,
>>>         you would need to reorder them yourself.  This cannot be
>>>         very complicated for the application.
>>>
>>>  2. QUIC per-application-stream flow control is rendered
>>>     ineffective.  I am actually not a huge fan of QUIC per-stream
>>>     flow control, as I strongly prefer stream priorities as a
>>>     signal.  But I now know of a case when per-stream flow control
>>>     is, indeed useful (when there is only one slow draining stream
>>>     that uses up the entire connection buffer before a
>>> Thanks Ted, this looks very interesting. Based on these documents 
>>> the DETNET sits below the level of QUIC that we are currently 
>>> discussing and would mainly affect the IP header format of the 
>>> resulting UDP packets. The QUIC protocol will still require some 
>>> form of partial reliability to function on a DETNET network.
>>>
>>>     -John Wason
>>>
>>>     Thanks for the message.  If you have not already reviewed the
>>>     work of DETNET (https://datatracker.ietf.org/wg/detnet/about/
>>>     On Tue, Apr 30, 2019 at 12:15 PM John Wason
>>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>>
>>>         Greetings QUIC WG,
>>>

-- 
John Wason, Ph.D.
Wason Technology, LLC
16 Sterling Lake Rd
Tuxedo, NY 10987
(518) 279-6234
[email-masked]

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 1 May 2019 17:33:08 +0000

The QUIC library API may include a facility to set/update stream priorities for the sender of data.  Webrtc-quic may include the same w/o any change to the underlying QUIC protocol.

However, if a request for a priority change of a stream is originating from the receiver of the data, QUIC transport has no facility to relay this information to the sender.  An application protocol would need to handle this.  I think it is a useful extension for QUIC v2 to push stream priority to the transport, since the transport can manage this control signal more efficiently.

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Sent: Wednesday, May 01, 2019 12:38 PM
To: Lubashev, Igor <[email-masked]>; Ted Hardie <[email-masked]>; John Wason <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: QUIC for robotics and other time-sensitive applications

QUIC does not prioritise streams at the transport layer, that is an implementation detail although pacing is recommended. If you bring you own application layer protocol you can decide how to schedule data across streams within the flow control constraints. You could add priorities to your application level custom API. If you layer a protocol on top of H3 you are more constrained since you must assume other implementations that won’t necessarily do what you want them to do.

On 1 May 2019 at 18.22.22, John Wason ([email-masked]<mailto:[email-masked]>) wrote:

Igor,

It appears I misunderstood the distinction between per-stream partial reliability and intra-stream partial reliability. If I send one partial-reliability message per stream, using RESET frames should be sufficient. How this is exposed as an API is important, since I am not sure the webrtc-quic team has considered this yet. In response to your questions 1) Each message contains a header that should be sufficient for the receiver to understand the incoming stream 2) Does QUIC provide the type of priority that you suggest? My understanding is that QUIC multiplexes streams using a priority based ordering.

If I can emulate partial-reliability using RESET frames to cancel entire streams, that should be sufficient for my needs.

    -John

  *   Igor

From: John Wason <[email-masked]><mailto:[email-masked]>
Sent: Tuesday, April 30, 2019 5:52 PM
To: Ted Hardie <[email-masked]><mailto:[email-masked]>
Cc: IETF QUIC WG <[email-masked]><mailto:[email-masked]>
Subject: Re: QUIC for robotics and other time-sensitive applications

    -John Wason

best regard,

Ted Hardie

     -John Wason

--

John Wason, Ph.D.

Wason Technology, LLC

16 Sterling Lake Rd

Tuxedo, NY 10987

(518) 279-6234

--

John Wason, Ph.D.

Wason Technology, LLC

16 Sterling Lake Rd

Tuxedo, NY 10987

(518) 279-6234

---

From: John Wason <[email-masked]>
Date: Wed, 1 May 2019 14:28:11 -0400

Would it make sense to add support for isochronous data directly to the 
QUIC protocol? DDS has the ability to set the requested update rate and 
QoS of different topics. My plan has been to emulate this functionality 
using streams at the application layer, but I am thinking now that there 
may be some benefit to having specialized flow control and QoS 
capabilities for this type of data. Besides robotics, this type of data 
is used extensively for applications like online gaming and high 
frequency trading so I think there is a large user base that would 
benefit from the extra capabilities.

On 5/1/2019 1:33 PM, Lubashev, Igor wrote:
>
> The QUIC library API may include a facility to set/update stream 
> priorities for the sender of data. Webrtc-quic may include the same 
> w/o any change to the underlying QUIC protocol.
>
> However, if a request for a priority change of a stream is originating 
> from the receiver of the data, QUIC transport has no facility to relay 
> this information to the sender.  An application protocol would need to 
> handle this.  I think it is a useful extension for QUIC v2 to push 
> stream priority to the transport, since the transport can manage this 
> control signal more efficiently.
>
> *From:* Mikkel Fahnøe Jørgensen <[email-masked]>
> *Sent:* Wednesday, May 01, 2019 12:38 PM
> *To:* Lubashev, Igor <[email-masked]>; Ted Hardie 
> <[email-masked]>; John Wason <[email-masked]>
>     It appears I misunderstood the distinction between per-stream
>     partial reliability and intra-stream partial reliability. If I
>     send one partial-reliability message per stream, using RESET
>     frames should be sufficient. How this is exposed as an API is
>     important, since I am not sure the webrtc-quic team has considered
>     this yet. In response to your questions 1) Each message contains a
>     header that should be sufficient for the receiver to understand
>     the incoming stream 2) Does QUIC provide the type of priority that
>     you suggest? My understanding is that QUIC multiplexes streams
>     using a priority based ordering.
>
>         I am happy to resurrect that
>         draft-lubashev-quic-partial-reliability for QUIC v2 (some
>         people preferred version 02, which actually gets you a bit
>         I would like to first understand what partially reliable
>         streams give you that is not available with regular streams. 
>         I.e. what are you missing if you use STREAM (with a possible
>         subsequent RESET) for messages?  (You cannot use DATAGRAM for
>         messages, since you would like the transport to facilitate
>         retransmission until expiration.)
>
>         My draft-lubashev-quic-partial-reliability lists a few things
>         that stream-per-message does not get you:
>
>          1. If you have multiple application message streams, you
>             would need a header for each QUIC stream to identify the
>             application stream.  This seems like a small burden,
>             except for very short messages.
>
>              1. If the order of messages in an application stream
>                 matters, you would need to reorder them yourself. 
>                 This cannot be very complicated for the application.
>
>             ineffective.  I am actually not a huge fan of QUIC
>             per-stream flow control, as I strongly prefer stream
>             priorities as a signal.  But I now know of a case when
>             per-stream flow control is, indeed useful (when there is
>             only one slow draining stream that uses up the entire
>         <mailto:[email-masked]>
>         *To:* Ted Hardie <[email-masked]> <mailto:[email-masked]>
>         *Cc:* IETF QUIC WG <[email-masked]> <mailto:[email-masked]>
>         Thanks Ted, this looks very interesting. Based on these
>         documents the DETNET sits below the level of QUIC that we are
>         currently discussing and would mainly affect the IP header
>         format of the resulting UDP packets. The QUIC protocol will
>         still require some form of partial reliability to function on
>         a DETNET network.
>
>         -John Wason
>
>             Thanks for the message.  If you have not already reviewed
>             the work of DETNET
>             (https://datatracker.ietf.org/wg/detnet/about/
>             here in the IETF, it might be useful to do so, as there
>             are a number of elements of your use case which appeared
>             to be shared with their work.  In particular, it might be
>             useful to consider whether the integration of their
>             proposed IP data plane
-- 
John Wason, Ph.D.
Wason Technology, LLC
16 Sterling Lake Rd
Tuxedo, NY 10987
(518) 279-6234
[email-masked]

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 1 May 2019 14:45:02 -0400

To your latest question first
> Would it make sense to add support for isochronous data directly to the
QUIC protocol?

No I don’t think so. Things go into transport if transport can do magic
difficult to do elsewhere. Sending repeated events is not that. It would
just complicate the core logic.

Please note that QUIC v2 has focus on partial reliability but it is out of
scope for v1. You can construct an overlay stream by sending an identifier
in stream payload and send each event in a new stream. This avoids
head-of-line-blocking (HOL), but it is clumsy and not always a good
solution.

On controlling transmission rate: There are algorithms that are sensitive
to prioritizing small payload over large payload such that all overall
fairness is achieved (FQ scheduler) such as weighed round robin (WRR)
https://en.wikipedia.org/wiki/Network_scheduler
An applicaition interface can receive hints that can guide such a
scheduler. The scheduler would be implement in the belly of the QUIC
transport, but it isn’t part of the transport layer protocol per se.

On visiblity of flow control. See the various packets the stream
controllers send to allow more data. Stream based limits are generally for
applications that cannot buffr more for the time being while connection
level is for overall bandwidth. A receiver can try to manage priorities via
flow credits, but it probably gets messy.

> I am not sure what you mean by "layer on top of H3”.

This is the equivalent of a HTTP 1.1. REST protocol vs a custom/dedicated
TCP protocol like FTP.
If you choose H3 your protocol design would be expected to be useful for
the majority of QUIC H3 implementations.
If you choose a lower level protocol directly on the QUIC transport layer,
you are still limited by whatever API’s make available such as scheduling,
but if you can the better choose implementations that are suitable.

Mikkel

On 1 May 2019 at 19.33.28, John Wason ([email-masked]) wrote:

Mikkel,

That makes sense, the order of streams sent shouldn't affect the wire
protocol. For the flow control, does the protocol provide information about
how much headroom is left? For instance, it may be very important that
small, time sensitive streams reach their destination quickly, while there
may be larger, slower streams that can wait. If the flow control is not
handled correctly, the slower stream could block the fast streams by
filling the available bandwidth and buffers.

    -John

Igor,

    -John

John,

   - Igor

    -John Wason

Hi John,

best regard,

Ted Hardie

Greetings QUIC WG,

     -John Wason

--
John Wason, Ph.D.
Wason Technology, LLC
16 Sterling Lake Rd
Tuxedo, NY 10987
(518) 279-6234
[email-masked]

-- 

John Wason, Ph.D.

Wason Technology, LLC

16 Sterling Lake Rd

Tuxedo, NY 10987

(518) 279-6234

[email-masked]

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 1 May 2019 20:40:27 +0100

Hi John,

On Wed, 1 May 2019, 19:28 John Wason, <[email-masked]> wrote:

> QUIC protocol? DDS has the ability to set the requested update rate and QoS
> of different topics. My plan has been to emulate this functionality using
> streams at the application layer, but I am thinking now that there may be
> some benefit to having specialized flow control and QoS capabilities for
> this type of data.
>
I used to use DDS (assuming you mean Data Distribution Service). The
abilities you mention came up recently is a side discussion abouit the
DATAGRAM frame and transport-service API (aks IETF TAPS). I shared a link
to a mammothe slide pack with some people at the time which explains some
DDS things, for convenience, here it is:

https://www.omg.org/news/meetings/workshops/RT-2007/00-T5_Hunt-revised.pdf

The DATAGRAM frame alone won't satisfy all requirements (or at least not
without some duplication in higher levels). I too am interested in partial
reliability of streams, in some form, in future versions of QUIC.

A DDS over QUIC application mapping document seems like the correct place
to express how transport capabilities map to DDS features. I wonder if such
a thing will ever emerge..

Tangent:
The particular flavour of DDS I used, based on top of UDP, had some clever
approaches for choosing when to deliver samples over via unicast or
multicast IP. I have a multicast HTTP/3 over QUIC independent draft, I
would assume some of the concepts could transfer to DDS.

Lucas

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 1 May 2019 15:43:59 -0400

I am a big fan of QUIC. I think it meets an important set of requirements.

But are those requirements so general that it amounts to a replacement for
TCP/IP in every context? I don't think so and I think it would be a very
bad idea to attempt to address every problem with the same toolkit.

Looking at the specifics of your problem, I have two major questions:

1) How big do the control messages need to be?
2) How big do the data stream messages need to be?

I am looking at a robotics control application but it is hardly a critical
one. I have this dalek in my office that I would like to be able to control
from various devices and autonomously to a limited degree. It has a camera
in its eyestalk and various telemetry data is developed.

So QUIC does look kinda sorta relevant. But is the effort required to try
and fit my application into the QUIC paradigm really worth making? Could I
do just as well by simply splitting the control and data streams? I know
that every control message is less than 64KB (much less infact).

Looking at the Web Services space, I find QUIC doesn't fit too well there
either. In particular, I note that for all my latency critical Mesh
applications, the request message is less than 8K and usually less than 1K.
So I can use UDP without any need to support request reassembly. Which
means I can support most of my information retrieval applications currently
implemented over HTTPS over a lighter weight UDP protocol that is stateless
on the server side. Message reassembly is expensive on the server side but
not an issue on the client side. So one UDP request can result in multiple
UDP responses.

So I think we should be skeptical before deciding QUIC is the new hammer to
be used to beat every nail. HTTPS is a poor match for Web Services because:

* The HTTP layer basically provides no value except for allowing multiple
services to be multiplexed on a single port.

* TCP/IP startup time introduces latency

* Vulnerable to DDoS (TCP/IP SYN flooding, TLS key exchange)

* QUIC is very complex.

The concerns that motivate QUIC are similar but really not the same as for
Web Services. And that is OK. We don't have to use the same hammer for
every problem. There are some Web Services that will layer really nicely on
QUIC (but maybe not this iteration or the next). There are other Web
Services that don't and won't.