Subject: QUIC Datagram Discussion

From: Tommy Pauly <[email-masked]>
Date: Mon, 12 Nov 2018 13:07:27 -0800

Hello QUIC,

Last week, several of us who were interested in fleshing out the use cases and details for unreliable frames in QUIC met up in an informal side meeting.

I took notes there, and wanted to share them with the whole list for documentation’s sake. Please feel free to respond with any other thoughts or opinions you may have!

Best,
Tommy

=========================

What are the use cases that would be interested in unreliable QUIC?
General interest in real time media, bundled with reliable transport. Sensor reading
VPN use case of making things look like web traffic, but allow tunneled packet to be sent unreliably
Realtime audio/video (bidirectional like RTC Web, unidirectional streaming)
WebRTC over QUIC
Video conference, augmented reality, single transport for media and data. Current spec for AR has synchronization problem.
HTTP/WebSocket/Datachannel servers currently have multiple connections that could be just one
Centralized conferencing in WebRTC
Signaling traffic (mobile core network, high signaling load)
Measurement traffic being multiplexed with data
5G wifi/cellular link multi-path control for datagrams
Use QUIC as a TURN relay into the CDN
Consolidating reverse proxy

Should work as an extension QUIC (v1)
Minimal or no changes to how applications handle datagram payload and sending

Don’t need to support fragmentation of datagrams in QUIC, just make sure that the frame boundaries are supported. Application on top generally fragments if necessary.
Application must be able to introspect maximum datagram size for the datagram channel

Try to accommodate different use cases with sensible support if possible

Should DATAGRAM frames be ACKed?
Yes. Would applications use it? Helps know what to encode next, for example.
ACKs may need different scheduling algorithms for DATAGRAMs
No flow control; but rate control. Could delay ACKs to help determine rate of consumption to influence rate of sending, packet pacing

Can there be multiple ID spaces for DATAGRAM frames?
Draft currently says no, but suggestion at meeting was to introduce a DATAGRAM_ID
DATAGRAM_ID could be used to set different congestion control per datagram “flow”
Different queues to deliver to application
Different priorities
Should be a VLI
Handles bundling of RTP vs RTCP, or multiple RTP streams

DATAGRAM_IDs are a different namespace from STREAM_IDs. They do not have equivalents such as RESET_DATAGRAM_ID, MAX_DATAGRAM_DATA, DATAGRAM_DATA_BLOCKED, MAX_DATAGRAM_IDS, or DATAGRAMS_BLOCKED.
Separate even and odd for which side initiated? But can always send in any direction without the other side opening it.

Upon migration or any connection ID length change, signal that maximum datagram size changed

Can we pass up ICMP errors for a given DATAGRAM?

Can send DATAGRAMs in 0-RTT

Add congestion window consideration—sending “immediately” still respects congestion control

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Nov 2018 13:18:06 -0800

Hi Tommy,

I think game location data is also important. This might be sent at a
relatively low frequency. In some cases with a fixed frequency like say
30Hz and not very sensitive to latency, in other cases 30Hz on average, but
depending on user input and highly sensitive to latency.

Also perhaps a grouping into high volume / low volume and latency sensitive
and latency tolerant.

There might also be cases where retransmission is of interest, but with low
priority that doesn’t block head of line. I can’t think of any use cases
right now.

Other use case could be human interface for automation control which is
similar to the game use case except people are more likely to get hurt. For
example drone control input and drone position feedback.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 12 November 2018 at 22.07.45, Tommy Pauly ([email-masked]) wrote:

Hello QUIC,

Last week, several of us who were interested in fleshing out the use cases
and details for unreliable frames in QUIC met up in an informal side
meeting.

I took notes there, and wanted to share them with the whole list for
documentation’s sake. Please feel free to respond with any other thoughts
or opinions you may have!

Best,
Tommy

General interest in real time media, bundled with reliable transport.
Sensor reading
VPN use case of making things look like web traffic, but allow tunneled
packet to be sent unreliably
Realtime audio/video (bidirectional like RTC Web, unidirectional streaming)
WebRTC over QUIC
Video conference, augmented reality, single transport for media and data.
Current spec for AR has synchronization problem.
HTTP/WebSocket/Datachannel servers currently have multiple connections that
could be just one
Centralized conferencing in WebRTC
Signaling traffic (mobile core network, high signaling load)
Measurement traffic being multiplexed with data
5G wifi/cellular link multi-path control for datagrams
Use QUIC as a TURN relay into the CDN
Consolidating reverse proxy

Should work as an extension QUIC (v1)
Minimal or no changes to how applications handle datagram payload and
sending

Don’t need to support fragmentation of datagrams in QUIC, just make sure
that the frame boundaries are supported. Application on top
generally fragments if necessary.

Application must be able to introspect maximum datagram size for the
datagram channel

No flow control; but rate control. Could delay ACKs to help determine rate
of consumption to influence rate of sending, packet pacing

Draft currently says no, but suggestion at meeting was to introduce a
DATAGRAM_ID

DATAGRAM_ID could be used to set different congestion control per datagram
“flow”
Different queues to deliver to application
Different priorities
Should be a VLI
Handles bundling of RTP vs RTCP, or multiple RTP streams

DATAGRAM_IDs are a different namespace from STREAM_IDs. They do not have
equivalents such as RESET_DATAGRAM_ID, MAX_DATAGRAM_DATA,
DATAGRAM_DATA_BLOCKED, MAX_DATAGRAM_IDS, or DATAGRAMS_BLOCKED.
Separate even and odd for which side initiated? But can always send in any
direction without the other side opening it.

Upon migration or any connection ID length change, signal that maximum
datagram size changed

Add congestion window consideration—sending “immediately” still respects
congestion control

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Nov 2018 13:21:25 -0800

Another use case is voting consensus algorithms were M/N must approve
within a deadline. I suspect the current datagram proposal is sufficient
for that.

On 12 November 2018 at 22.18.06, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Hi Tommy,

Kind Regards,
Mikkel Fahnøe Jørgensen

Hello QUIC,

Best,
Tommy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Nov 2018 13:26:01 -0800

And eventual consistency algorithms where multiple servers chatter to reach
consensus but where data loss is not essential. This could be an example
where retransmission with low priority is of interest because another
server probably already sent the old data. New data is more interesting.

On 12 November 2018 at 22.21.25, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Hi Tommy,

Kind Regards,
Mikkel Fahnøe Jørgensen

Hello QUIC,

Best,
Tommy

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 13 Nov 2018 13:37:54 +0700

Thanks for sending the notes along.

FWIW, I would be very wary of going into separate ID spaces with DATAGRAMs.
Separating congestion control within a connection is problematic -- you
want scheduling priorities, but you do not want to do two different
controllers within the same connection. I also think dragons be there, like
with flow control; keeping this simple is going to be key in getting this
done.

- jana

On Tue, Nov 13, 2018 at 4:26 AM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> And eventual consistency algorithms where multiple servers chatter to
> reach consensus but where data loss is not essential. This could be an
> example where retransmission with low priority is of interest because
> another server probably already sent the old data. New data is more
> interesting.
>
> On 12 November 2018 at 22.21.25, Mikkel Fahnøe Jørgensen (
> [email-masked]) wrote:
>
> On 12 November 2018 at 22.18.06, Mikkel Fahnøe Jørgensen (
> [email-masked]) wrote:
>
> Hi Tommy,
>
> Also perhaps a grouping into high volume / low volume and latency
> sensitive and latency tolerant.
>
> There might also be cases where retransmission is of interest, but with
> low priority that doesn’t block head of line. I can’t think of any use
> cases right now.
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 13 Nov 2018 17:43:45 +1100

This.  Keep it stupidly simple.  Also, I don't understand what the
*recipient* of a DATAGRAM_ID would do with that.  Maybe I should wait
for an actual write-up.
On Tue, Nov 13, 2018 at 5:38 PM Jana Iyengar <[email-masked]> wrote:
> FWIW, I would be very wary of going into separate ID spaces with DATAGRAMs. Separating congestion control within a connection is problematic -- you want scheduling priorities, but you do not want to do two different controllers within the same connection. I also think dragons be there, like with flow control; keeping this simple is going to be key in getting this done.
>
> - jana
>
> On Tue, Nov 13, 2018 at 4:26 AM Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
>>
>> And eventual consistency algorithms where multiple servers chatter to reach consensus but where data loss is not essential. This could be an example where retransmission with low priority is of interest because another server probably already sent the old data. New data is more interesting.
>>
>> On 12 November 2018 at 22.21.25, Mikkel Fahnøe Jørgensen ([email-masked]) wrote:
>>
>> Another use case is voting consensus algorithms were M/N must approve within a deadline. I suspect the current datagram proposal is sufficient for that.
>>
>>
>>
>> On 12 November 2018 at 22.18.06, Mikkel Fahnøe Jørgensen ([email-masked]) wrote:
>>
>> Hi Tommy,
>>
>> I think game location data is also important. This might be sent at a relatively low frequency. In some cases with a fixed frequency like say 30Hz and not very sensitive to latency, in other cases 30Hz on average, but depending on user input and highly sensitive to latency.
>>
>> Also perhaps a grouping into high volume / low volume and latency sensitive and latency tolerant.
>>
>> There might also be cases where retransmission is of interest, but with low priority that doesn’t block head of line. I can’t think of any use cases right now.
>>
>> Other use case could be human interface for automation control which is similar to the game use case except people are more likely to get hurt. For example drone control input and drone position feedback.
>>
>>
>> Kind Regards,
>> Mikkel Fahnøe Jørgensen
>>
>>

---

From: Ian Swett <[email-masked]>
Date: Tue, 13 Nov 2018 10:44:25 -0500

I agree with Jana and Martin's comments.  Adding an on-the-wire ID is
adding complexity and wire overhead with no clear value.  As Martin said,
it's also not clear what a recipient is supposed to do with it?

If you want to add an application level ID, you should do it in the
application mapping.

And congestion control within congestion control is even scarier.

Using sender side ids to allow an application to be notified of receipt or
loss(even if it's not guaranteed to deliver it to the application) is a
completely reasonable idea, but that's part of the API, not part of the
DATAGRAM frame.

On Tue, Nov 13, 2018 at 1:44 AM Martin Thomson <[email-masked]>
wrote:

> > FWIW, I would be very wary of going into separate ID spaces with
> DATAGRAMs. Separating congestion control within a connection is problematic
> -- you want scheduling priorities, but you do not want to do two different
> >> Last week, several of us who were interested in fleshing out the use
> cases and details for unreliable frames in QUIC met up in an informal side
> meeting.
> >>
> >> Video conference, augmented reality, single transport for media and
> data.. Current spec for AR has synchronization problem.
> >> Don’t need to support fragmentation of datagrams in QUIC, just make
> sure that the frame boundaries are supported. Application on top generally
> fragments if necessary.
> >>
> >> No flow control; but rate control. Could delay ACKs to help determine
> rate of consumption to influence rate of sending, packet pacing
> >>
> >> DATAGRAM_IDs are a different namespace from STREAM_IDs. They do not
> have equivalents such as RESET_DATAGRAM_ID, MAX_DATAGRAM_DATA,
> >> Separate even and odd for which side initiated? But can always send in
> any direction without the other side opening it.
> >>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 13 Nov 2018 17:57:34 +0000

> On 13 Nov 2018, at 15:44, Ian Swett <ianswett=[email-masked]> wrote:
> 
> I agree with Jana and Martin's comments.  Adding an on-the-wire ID is adding complexity and wire overhead with no clear value.  As Martin said, it’s also not clear what a recipient is supposed to do with it?

The recipient uses it to demux datagrams, much like it uses the stream ID to identify and demultiplex streams. The complexity at the QUIC layer is small, and it greatly simplifies many applications, and the demultiplexing of applications, to have this functionality as a common feature. 

> If you want to add an application level ID, you should do it in the application mapping.

It needs to be in the QUIC datagram layer, since it is common to want to demux different applications sent on a single connection. That is, we’re trying to avoid having to use mechanisms like that in RFC 7983 within QUIC, by providing the demultiplex at the datagram layer in a single, application-independent, way.

> And congestion control within congestion control is even scarier.

The suggestion was not congestion control within congestion control, but rather providing mechanisms to allow differential congestion control for sub-flows as a possible future extension. Such behaviour is widely implemented in the types of applications that use datagram service, and we have examples of how to do it in the IETF (e.g., draft-ietf-rmcat-coupled-cc-07, which has been stuck in the RFC Editor’s queue for some months now).

> Using sender side ids to allow an application to be notified of receipt or loss(even if it’s not guaranteed to deliver it to the application) is a completely reasonable idea, but that's part of the API, not part of the DATAGRAM frame.

It’s not clear what you mean by ‘sender side ids’. If the receiver is to be able to detect loss of datagrams, then we need sequence numbers in the DATAGRAM frames, and to expose those sequence numbers in the datagram API. Many applications need that functionality. 

Colin

> On Tue, Nov 13, 2018 at 1:44 AM Martin Thomson <[email-masked] <mailto:[email-masked]>> wrote:
> On Tue, Nov 13, 2018 at 5:38 PM Jana Iyengar <[email-masked] <mailto:[email-masked]>> wrote:
> >
> > FWIW, I would be very wary of going into separate ID spaces with DATAGRAMs. Separating congestion control within a connection is problematic -- you want scheduling priorities, but you do not want to do two different controllers within the same connection. I also think dragons be there, like with flow control; keeping this simple is going to be key in getting this done..
> >
> > - jana
> >
> > On Tue, Nov 13, 2018 at 4:26 AM Mikkel Fahnøe Jørgensen <[email-masked] <mailto:[email-masked]>> wrote:
> >>
> >> On 12 November 2018 at 22.21.25, Mikkel Fahnøe Jørgensen ([email-masked] <mailto:[email-masked]>) wrote:
> >>
> >> On 12 November 2018 at 22.18.06, Mikkel Fahnøe Jørgensen ([email-masked] <mailto:[email-masked]>) wrote:
> >>
> >> Hi Tommy,
> >>
> >> On 12 November 2018 at 22.07.45, Tommy Pauly ([email-masked] <mailto:[email-masked]>) wrote:
> >>
> >> Hello QUIC,
> >>
> >> Video conference, augmented reality, single transport for media and data.. Current spec for AR has synchronization problem.

-- 
Colin Perkins
https://csperkins.org/

---

From: Christian Huitema <[email-masked]>
Date: Tue, 13 Nov 2018 10:27:54 -0800

> On Nov 13, 2018, at 9:57 AM, Colin Perkins <[email-masked]> wrote:
> 

That's not the way QUIC works. Frames are carried in numbered packets. Packet loss is detected at the packet level. Loss of a datagram frame is detected when the packet that carried it is lost. There is no need for frame level detection.

-- Christian Huitema

---

From: Ian Swett <[email-masked]>
Date: Tue, 13 Nov 2018 13:44:20 -0500

On Tue, Nov 13, 2018 at 12:57 PM Colin Perkins <[email-masked]> wrote:

> it’s also not clear what a recipient is supposed to do with it?
>
>
> The recipient uses it to demux datagrams, much like it uses the stream ID
> to identify and demultiplex streams. The complexity at the QUIC layer is
> small, and it greatly simplifies many applications, and the demultiplexing
> of applications, to have this functionality as a common feature.
>
> It needs to be in the QUIC datagram layer, since it is common to want to
> demux different applications sent on a single connection. That is, we’re
> trying to avoid having to use mechanisms like that in RFC 7983 *within*
> QUIC, by providing the demultiplex at the datagram layer in a single,
> application-independent, way.
>

Ok, let's pop the stack here.  I've heard interest in the generic ability
to put two different applications on a given QUIC connection.(Roberto?),
but I think we all agree that if we do that, it's QUIC v2.  And if we do
that, making it only work for DATAGRAM seems short-sighted.  If I want to
run two entirely separate applications inside a QUIC connection, then I
definitely want the ability to use streams as well.

> The suggestion was not congestion control within congestion control, but
> rather providing mechanisms to allow differential congestion control for
> sub-flows as a possible future extension. Such behaviour is widely
> implemented in the types of applications that use datagram service, and we
> have examples of how to do it in the IETF (e.g.,
> draft-ietf-rmcat-coupled-cc-07, which has been stuck in the RFC Editor’s
> queue for some months now).
>
> loss(even if it’s not guaranteed to deliver it to the application) is a
> It’s not clear what you mean by ‘sender side ids’. If the receiver is to
> be able to detect loss of datagrams, then we need sequence numbers in the
> DATAGRAM frames, and to expose those sequence numbers in the datagram API.
> Many applications need that functionality.
>
> Colin
>
>
>
>> done..
>> >> There might also be cases where retransmission is of interest, but
>> with low priority that doesn’t block head of line. I can’t think of any use
>> cases right now.
>> >>
>> >> Other use case could be human interface for automation control which
>> is similar to the game use case except people are more likely to get hurt.
>> For example drone control input and drone position feedback.
>> >>
>> >>
>> >> Kind Regards,
>> >> Mikkel Fahnøe Jørgensen
>> >>
>> >>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 13 Nov 2018 18:55:54 +0000

> On 13 Nov 2018, at 18:27, Christian Huitema <[email-masked]> wrote:

There is if you allow multiple frames to be coalesced within a packet, otherwise you can’t tell how many frames were lost. Streams have this with the offset and length fields.

-- 
Colin Perkins
https://csperkins.org/

---

From: Roberto Peon <[email-masked]>
Date: Tue, 13 Nov 2018 19:05:49 +0000

From: QUIC <[email-masked]> on behalf of Ian Swett <ianswett=[email-masked]>
Date: Tuesday, November 13, 2018 at 10:45 AM
To: Colin Perkins <[email-masked]>
Cc: Jana Iyengar <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>, IETF QUIC WG <[email-masked]>, Martin Thomson <[email-masked]>, Tommy Pauly <[email-masked]>
Subject: Re: QUIC Datagram Discussion

On Tue, Nov 13, 2018 at 12:57 PM Colin Perkins <[email-masked]<mailto:[email-masked]>> wrote:
On 13 Nov 2018, at 15:44, Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>> wrote:

Ok, let's pop the stack here.  I've heard interest in the generic ability to put two different applications on a given QUIC connection.(Roberto?), but I think we all agree that if we do that, it's QUIC v2.  And if we do that, making it only work for DATAGRAM seems short-sighted.  If I want to run two entirely separate applications inside a QUIC connection, then I definitely want the ability to use streams as well.

Ya, I bring that (multi-protocol connection sharing) up often.
And yes on it being a Quic v2 kinda thing.
-=R

Colin

On Tue, Nov 13, 2018 at 1:44 AM Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
This.  Keep it stupidly simple.  Also, I don't understand what the
*recipient* of a DATAGRAM_ID would do with that.  Maybe I should wait
for an actual write-up.
On Tue, Nov 13, 2018 at 5:38 PM Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:
> FWIW, I would be very wary of going into separate ID spaces with DATAGRAMs. Separating congestion control within a connection is problematic -- you want scheduling priorities, but you do not want to do two different controllers within the same connection. I also think dragons be there, like with flow control; keeping this simple is going to be key in getting this done...
>
> - jana
>
> On Tue, Nov 13, 2018 at 4:26 AM Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> On 12 November 2018 at 22.21.25, Mikkel Fahnøe Jørgensen ([email-masked]<mailto:[email-masked]>) wrote:
>>
>> On 12 November 2018 at 22.18.06, Mikkel Fahnøe Jørgensen ([email-masked]<mailto:[email-masked]>) wrote:
>>
>> Hi Tommy,
>>
>> On 12 November 2018 at 22.07.45, Tommy Pauly ([email-masked]<mailto:[email-masked]>) wrote:
>>
>> Hello QUIC,
>>

--
Colin Perkins
https://csperkins.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__csperkins.org_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=QOXc5ruwvf5FD5r944G8nhNbXmMlsJJJj-6xRG9qNV8&s=BQu-VXfdP14oTFI6bjW0oM9vLJJkIarL0koPA5Q2EPg&e=>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 13 Nov 2018 22:57:08 +0000

> On 13 Nov 2018, at 18:44, Ian Swett <[email-masked]> wrote:
> On Tue, Nov 13, 2018 at 12:57 PM Colin Perkins <[email-masked] <mailto:[email-masked]>> wrote:
>> On 13 Nov 2018, at 15:44, Ian Swett <ianswett=[email-masked] <mailto:ianswett=[email-masked]>> wrote:
>> 

Sorry, I meant multiple application protocols. I don’t see this as a QUIC v2 feature, more aligning datagram features more closely with those of regular QUIC.

Colin

> https://csperkins.org/ <https://csperkins.org/>
> 
> 
> 
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: Martin Thomson <[email-masked]>
Date: Wed, 14 Nov 2018 10:34:41 +1100

> Sorry, I meant multiple application protocols. I don’t see this as a QUIC v2 feature, more aligning datagram features more closely with those of regular QUIC.

"alignment" means nothing here.  The reason we have stream IDs is
because we have a bunch of frames that all need to coordinate around
the concept of a stream.  And the transport needs that to manage flow
control, reassembly, and reset.  The transport doesn't need this
identifier.

And applications can add their own identifier if they need it.  The
reason you might add an identifier is as a common service for
application protocols, but why do that if the application protocol can
help itself?  The suggestion that you might have multiple
uncoordinated protocols operating within the same envelope is only
semi-plausible.  Any such arrangement needs more than just an
identifier, it minimally needs a way of managing the allocation of
those identifiers as well.  A composite of two protocols is in
practice a third protocol.

My conclusion here is that we don't need any such complication.  That
sort of complication is best left to other extensions, because it will
likely smother this one.

---

From: Colin Perkins <[email-masked]>
Date: Tue, 13 Nov 2018 23:53:14 +0000

> On 13 Nov 2018, at 23:34, Martin Thomson <[email-masked]> wrote:
> 

The transport does need this identifier, for demultiplexing, loss detection, congestion control, etc. Tommy’s original message identified a number of reasons why it’s useful, as did my previous messages.

> help itself?  

Because you want all protocols running over QUIC to do it the same way, to support datagram demultiplexing – a big source of complexity for UDP-based applications.

> The suggestion that you might have multiple
> semi-plausible.  

Multiple protocols and sub-flows on datagram is the typical case for WebRTC, which is one of the likely main users of this extension. We’ve spent years developing SDP BUNDLE, and RFC 7983-style heuristics for protocol demuxing. Multiple sub-flows with a datagram flow is as useful as multiple streams.

> Any such arrangement needs more than just an
> those identifiers as well.  

Managing the identifier allocation is something that can be left to the applications.

> A composite of two protocols is in practice a third protocol.
> 

I disagree. This is a single identifier added to datagram frames, that is then exposed to the applications in the API. The added complexity is trivial (and, yes, it allows us to build more complex things as optional extensions later, but there’s significant benefit even if none of those are ever implemented). 

-- 
Colin Perkins
https://csperkins.org/

---

From: Martin Thomson <[email-masked]>
Date: Wed, 14 Nov 2018 11:25:00 +1100

On Wed, Nov 14, 2018 at 10:53 AM Colin Perkins <[email-masked]> wrote:
> The transport does need this identifier, for demultiplexing, loss detection, congestion control, etc.

It certainly doesn't need this identifier for loss detection and
congestion control.  It has packet numbers for that.

And the demultiplexing case is not a necessity.  I realize that this
is what you want to do, but when other protocols have existing
techniques, why do you believe they will abandon those techniques?

> Managing the identifier allocation is something that can be left to the applications.

That suggests to me that managing the identifier encoding can also be
left to the applications.  For one, they then get to decide how big
the demux field is.  We might pick a varint encoding and find that 4
or 8 bits is what is needed, both of which would be inefficient.

---

From: Christian Huitema <[email-masked]>
Date: Tue, 13 Nov 2018 16:34:38 -0800

> On Nov 13, 2018, at 3:53 PM, Colin Perkins <[email-masked]> wrote:
> 

Why would we want that? QUIC extensions are cheap. If an application needs something more complicated than the basic datagram frame, it can define its own "complicated datagram" extension. There is no point making the basic datagram more complex with hypothetical requirements.

-- Christian Huitema

---

From: Colin Perkins <[email-masked]>
Date: Wed, 14 Nov 2018 00:58:17 +0000

> On 14 Nov 2018, at 00:34, Christian Huitema <[email-masked]> wrote:

So far as I can tell, we’ve spent several years in IETF defining protocols specifically to address this “hypothetical requirement”.  RFC 7983, SDP BUNDLE, several multiplexing drafts in AVTCORE, etc., all exist because the only demuxing point in UDP is the port, and that’s not usable due to middlebox ossification. We have the opportunity to fix this with QUIC datagrams, to introduce the multiplexing layer we need, at very minimal cost, by introducing one additional identifier analogous to that already present in QUIC streams. Why would we not want that?

Colin

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 13 Nov 2018 20:07:05 -0500

I kind of find it unlikely that you would mux on the same connection rather
than mux’ing multiple connections but perhaps there are strong use cases
for that?

Different applications would also start competing for streams. I can’t
imagine running a custom app along with HTTP/3 for instance - in the
general case - although I could imagine some add-on applications that have
learned to co-exist with HTTP/3.

Within a single applications I can imagine many different datagram types
but then that is best sorted by the application as Martin Thomson just
mentioned. In this case forcing a transport layer ID could easily spend way
more bandwidth than desirable, especially when there is only one use of
datagrams. For example, game object direction vectors might need a game
object id, but the frames are all of the same kind. Or there might be
vectors or positions indicated by a single flag. There might be hundreds of
these in a single packet, but with a forced transport ID on top of that, it
would now require two packets with larger chance of scene inconsistencies
due to loss.

On 14 November 2018 at 01.58.23, Colin Perkins ([email-masked]) wrote:

>> Because you want all protocols running over QUIC to do it the same way,
to support datagram demultiplexing – a big source of complexity for
UDP-based applications.
>
> Why would we want that? QUIC extensions are cheap. If an application
needs something more complicated than the basic datagram frame, it can
define its own "complicated datagram" extension. There is no point making
the basic datagram more complex with hypothetical requirements.

So far as I can tell, we’ve spent several years in IETF defining protocols
specifically to address this “hypothetical requirement”. RFC 7983, SDP
BUNDLE, several multiplexing drafts in AVTCORE, etc., all exist because the
only demuxing point in UDP is the port, and that’s not usable due to
middlebox ossification. We have the opportunity to fix this with QUIC
datagrams, to introduce the multiplexing layer we need, at very minimal
cost, by introducing one additional identifier analogous to that already
present in QUIC streams. Why would we not want that?

Colin

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 13 Nov 2018 20:12:50 -0500

UDP isn’t connection oriented so if you only have one port, you have a
problem.

QUIC is connection oriented and has a connection ID. So you can have
multiple connections, also on the same port. The connection ID effectively
acts as the requested DATAGRAM application ID if each application uses a
different connection. This is a more efficient encoding than repeating the
app ID for each datagram. Also, it solves lifetime issues where different
applications terminate at different points in time, avoiding resource
cleanup issues.

---

From: Colin Perkins <[email-masked]>
Date: Wed, 14 Nov 2018 01:13:41 +0000

> congestion control.  It has packet numbers for that.

Datagram sequence numbers, separate from packet numbers, are needed to detect datagram loss if packets coalescing is allowed. Datagram stream IDs are needed for congestion control to support differential stream priority, which is widely used by the types of applications that use datagrams. 

> techniques, why do you believe they will abandon those techniques?

Because existing techniques are complex and don’t work well for those applications. 

And because I believe in improving protocols, to address limitations that have caused us problems for many years, rather than just reinvent UDP, which experience shows is lacking important features. 

> or 8 bits is what is needed, both of which would be inefficient.

I tend to think that a hypothetical small inefficiency on the wire is preferable to the complexity of defining something like BUNDLE (version -53, and counting...) or the inextensible kludge that is RFC 7983.

Colin

---

From: Colin Perkins <[email-masked]>
Date: Wed, 14 Nov 2018 01:22:10 +0000

> On 14 Nov 2018, at 01:07, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> I kind of find it unlikely that you would mux on the same connection rather than mux’ing multiple connections but perhaps there are strong use cases for that?

Why would you want to open a new connection when you already have one?

> Different applications would also start competing for streams. I can’t imagine running a custom app along with HTTP/3 for instance - in the general case - although I could imagine some add-on applications that have learned to co-exist with HTTP/3.

WebRTC has been suggested as wanting exactly this use case. And sub-streams within an application are very common. 

> Within a single applications I can imagine many different datagram types but then that is best sorted by the application as Martin Thomson just mentioned. In this case forcing a transport layer ID could easily spend way more bandwidth than desirable, especially when there is only one use of datagrams. For example, game object direction vectors might need a game object id, but the frames are all of the same kind. Or there might be vectors or positions indicated by a single flag. There might be hundreds of these in a single packet, but with a forced transport ID on top of that, it would now require two packets with larger chance of scene inconsistencies due to loss.

We’re talking one extra byte per packet overhead, or if we overload the frame type, zero overhead... I really don’t think this is a concern. 

Colin

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 13 Nov 2018 20:25:42 -0500

Datagram sequence numbers, separate from packet numbers, are needed to
detect datagram loss if packets coalescing is allowed.

That doesn’t change anything: if 3 packets are in the same UDP datagram,
they either all arrive, or none does, but it is the same as if each packet
was sent by itself.

 Datagram stream IDs are needed for congestion control to support
differential stream priority, which is widely used by the types of
applications that use datagrams.

Datagram priorities can be handled in the QUIC application interface if
this is a matter of sending priorities. It does not need to happen on the
wire. For QUIC to application delivery that is different, but I’m not sure
why QUIC would buffer and delay datagrams when there is no flow control.

I tend to think that a hypothetical small inefficiency on the wire is
preferable to the complexity of defining something like BUNDLE (version
-53, and counting...) or the inextensible kludge that is RFC 7983.

It is not hypthetical. If you think voice and video, then yes. If you think
sensor data, then it can matter a lot, and even affect latency critical
applications because some data will be deferred to the next packet which
might be rate limited.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 13 Nov 2018 20:33:26 -0500

Because it removes inter-application dependencies. There can be infinitely
many applications types that would have to potentially co-exist, but likely
only a few active applications at a time. Having two or three connections
are hardly a burden in that case, especially if they don’t transmit the
same time anyway. They do, of course add some extra buffering overhead, but
likely not much compared to the transferred content.

It’s a matter of thinking about the options provided by QUIC rather than
thinking either TCP or UDP.

I could imagine a future QUIC version that allowed multi-path for
application multiplexing such that each application has it’s own set of
connection ID’s even if the physical paths are the same. Still, it is
probably simpler to have separate connections, but it would be an option.
It would totally solve the multiplexing problem except that coalescing  on
same UDP packet would not be allowed for linkability reasons.

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 13 Nov 2018 19:36:07 -0600

Over in the DOTS WG I'm currently doing AD review of a pair of documents for a
TCP+TLS "data channel" (reliable except during attacks) and a DTLS+UDP "signal
channel" (to convey control messages during attacks), that seems like it would
be a great fit for a consolidated stream+datagram QUIC functionality on a single
connection.

-Ben

On Mon, Nov 12, 2018 at 01:26:01PM -0800, Mikkel Fahnøe Jørgensen wrote:
> Yes. Would applications use it? Helps know what to encode next, for example..
>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 13 Nov 2018 18:29:12 -0800

> On Nov 13, 2018, at 4:58 PM, Colin Perkins <[email-masked]> wrote:
> 

We certainly want to define standard ways of mapping applications to QUIC. But there are many ways to do that, and at this stage in the standard process many of us would rather place the complexity in the application mapping rather than in the transport layer. We have existence proof that real time applications can be developed over a simple datagram/UDP layer. We also have very limited cycles to define a basic extension in QUIC v1. That's why a bare-bone datagram extension looks much more attractive than attempting to define something complex.

The AVT working groups may want to work on the real time applications mapping over QUIC, starting with the basic DATAGRAM frame. That would be a start.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Tue, 13 Nov 2018 18:40:06 -0800

On Nov 13, 2018, at 5:25 PM, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:

>> Datagram sequence numbers, separate from packet numbers, are needed to detect datagram loss if packets coalescing is allowed.
> 
> That doesn’t change anything: if 3 packets are in the same UDP datagram, they either all arrive, or none does, but it is the same as if each packet was sent by itself.
> 
What Mikkel says. By the way, in QUIC v1, you can only coalesce packets during the handshake phase. Once the connection is synchronized, there is only one protected packet per UDP packet. You could indeed have several datagram frames in a single QUIC packet, if we define the frame format to carry an explicit length. But even so the QUIC transport can absolutely tell the application layer which datagram frames were received and which were lost.

Colin, you may be looking at something else, the possibility for the receiving application to detect that some packets are missing from the received sequence. And yes, that can only be achieved if the datagram frames start with something like the RTP header. I assume that the application mapping will define that.

-- Christian Huitema

---

From: Colin Perkins <[email-masked]>
Date: Wed, 14 Nov 2018 08:35:29 +0000

> On 14 Nov 2018, at 02:40, Christian Huitema <[email-masked]> wrote:
> 
> On Nov 13, 2018, at 5:25 PM, Mikkel Fahnøe Jørgensen <[email-masked] <mailto:[email-masked]>> wrote:
> 

Yes, of course it can all be done over UDP-like datagram frames in an application-specific manner. With all the problems we’ve observed in doing that. I’m trying to move forward, rather than duplicate the previous known problematic design.

-- 
Colin Perkins
https://csperkins.org/

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 14 Nov 2018 04:18:07 -0500

On 14 November 2018 at 03.40.16, Christian Huitema ([email-masked])
wrote:

You could indeed have several datagram frames in a single QUIC packet, if
we define the frame format to carry an explicit length. But even so the
QUIC transport can absolutely tell the application layer which datagram
frames were received and which were lost.

@Huitema if the draft I’m reading is still current, there are two datagram
frame types: one with length and one without that extends to the end. So
coalescing within a QUIC packet is already a thing. If length was not
present, my point about additional overhead in an application ID would be
irrelevant.

https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf

The more I think about it the more sympathetic I am towards Colins concerns
when you strictly think about it in a web browser context, but QUIC has
many more use cases than that.

I can see why WebRTC would want a standard hook to sit on top of HTTP/3 but
either this can be done on by

1) having H3 standardize on a sub datagram type that mandates an ID field
early in the datagram frame. (This forces H3 application squatters to
coordinate with H3)
2) having a new datagram extension can be defined that does this. (This
amounts to the same things as 1) but now it is official and thereby more
load on official registration of application types.
3) multiplexing on connection ID by having each application use a different
path on the samme connection. (This  approach is not viable in QUIC v1
because multi-path is not supported, but it might be the best future
option.)

If 3) is attractive, it might be better to use multiple connections for now
and see where things go in the future, and influence the process. 1) can be
done trivially, but itsn’t very “standards” friendly.

2) might be all that is needed, but for both 1) and 2) HTTP/3 would need to
agree because it is its connection that is being squatted.

Apart from 3) I don’t think it is viable to have arbitrary applications
coexist on the same connection as they might make all sorts of incompatible
assumptions and simplifications that are not viable in a heterogenous
environment.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 14 Nov 2018 19:15:19 +0000

> From: Christian Huitema <[email-masked]> 
> Sent: Tuesday, November 13, 2018 9:40 PM
>
> By the way, in QUIC v1, you can only coalesce packets during the handshake phase. Once the connection is synchronized, there is only one protected packet per UDP packet.

Strictly speaking this does not seem accurate.  https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#rfc.section.12.2 implies that coalescing can do done at any time.  The leading coalesced QUIC long-header packet(s) will likely get discarded after the handshake, though.

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 15 Nov 2018 10:20:58 +0530

Colin,

If you are using something parallel to packet numbers, that's a whole other
complicated ball of wax. My understanding of the proposed DATAGRAM
extension was based on Tommy's draft, which is far simpler.

You don't need anything special for loss detection and congestion control.
What you're really after is demultiplexing. Demuxing can be arranged by the
application. I understand that it's not perfect, since you can't really get
priorities without an ID, and I don't think *this* DATAGRAM extension needs
to solve everything.

On Thu, Nov 15, 2018 at 12:45 AM Lubashev, Igor <[email-masked]> wrote:

> > By the way, in QUIC v1, you can only coalesce packets during the
> handshake phase. Once the connection is synchronized, there is only one
> protected packet per UDP packet.
>
> Strictly speaking this does not seem accurate.
> https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#rfc.section.12.2
> implies that coalescing can do done at any time.  The leading coalesced
> QUIC long-header packet(s) will likely get discarded after the handshake,
> though.
>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 15 Nov 2018 07:43:36 -0800

> On Nov 14, 2018, at 8:50 PM, Jana Iyengar <[email-masked]> wrote:
> 
> You don't need anything special for loss detection and congestion control. What you're really after is demultiplexing.

Actually, it depends on the application. Take voice over IP. When you process a voice packet, you want to know whether it is the most recent one. You can infer that from the QUIC packet number. You also want to know whether the previous segment was already received, so you know whether some form of loss compensation is required. That can only be partially inferred from the packet number, especially if you don't want to bring back head of queue blocking. Plus, there may be holes in the sequence numbers, so you would infer a segment loss when there was none. Then, if you do silence suppression, you want to know the time at which the segment was sent. 

The RTP header solves that. If I was to do voice over QUIC, my first design would be to just carry RTP packets in Datagram frames. Of course, one may be more ambitious and design a specific RTP frame extension instead, which I understand is more or less what Colin is advocating. My main reservation for the RTP frame approach is that it would be a pretty intense standardization effort, which can take a significant time. Also, I am not sure that the resulting wire image would be all that different from "RTP in datagram frames".

-- Christian Huitema

---

From: Lars Eggert <[email-masked]>
Date: Thu, 15 Nov 2018 17:51:12 +0200

Hi,

On 2018-11-15, at 17:43, Christian Huitema <[email-masked]> wrote:
> 
> Actually, it depends on the application. Take voice over IP. When you process a voice packet, you want to know whether it is the most recent one. You can infer that from the QUIC packet number.

since we don't have a transport API for QUIC, it's not clear whether packet numbers are exposed to the application at all.

> Then, if you do silence suppression, you want to know the time at which the segment was sent.

Ditto.

> The RTP header solves that. If I was to do voice over QUIC, my first design would be to just carry RTP packets in Datagram frames. Of course, one may be more ambitious and design a specific RTP frame extension instead, which I understand is more or less what Colin is advocating.

I think "something better than vanilla datagrams" is what Colin is going after.

Lars