Subject: flow control and DATAGRAM

From: Martin Thomson <[email-masked]>
Date: Mon, 29 Oct 2018 16:16:07 +1100

Hi Tommy,

Your slides - <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf>
- say that DATAGRAM frames respect connection-level flow control.  I
missed that in the draft, and I don't know how they can do that in the
face of packet loss, especially when you don't necessarily retransmit
lost DATAGRAM frames.

For that to work, you would need a bunch more machinery to make the
connection-level flow control sync between endpoints in the case that
packets are lost.  A disagreement about how much flow control is used
causes things to break down badly.  Ian and I discussed this point at
the last meeting and quickly agreed that while it might be nice to
have flow control for this stuff, the increase in complexity is
considerable and (at the time) we thought it wouldn't be worth it.

The problem that introduces is that you could end up having too many
DATAGRAM frames arrive.  The receiver has to drop something at the
point that it can't handle them.  And we say that when you acknowledge
something, you processed it.  That's tricky.

It might be easier to say that a QUIC acknowledgment for a DATAGRAM
frame doesn't mean that it was received and processed by an
application.  An endpoint might discard these frames before passing
them on to applications if it doesn't have space.  In other words,
acknowledgment of DATAGRAM means that QUIC got it, not that the
application got it.  Sadly, that means that the QUIC acknowledgment
machinery doesn't help the application that uses DATAGRAM all that
much.  Also, the lower bound on reliability is 0, which isn't the best
thing ever.

Hard choices, I know.  I don't have a good design for maintaining
connection-level flow control (or any back pressure mechanism with
equivalent properties) that doesn't add both complexity and overhead.

Cheers,
Martin

---

From: Ian Swett <[email-masked]>
Date: Mon, 29 Oct 2018 08:32:24 -0400

Good catch Martin, I missed that in the draft as well, and I also think
it's impossibly with the proposed design.

And yes, I think Martin's proposed solution is likely the only practical
one.  In my implementation, the frame is passed up to the application
immediately, so technically QUIC processed it, and it's the application's
job to decide what to do with it.

On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <[email-masked]>
wrote:

> Hi Tommy,
>
> Your slides - <
> https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf
> >

---

From: Tommy Pauly <[email-masked]>
Date: Mon, 29 Oct 2018 08:21:33 -0700

Hi Martin, Ian,

Yes, very good points!

My tendency would be to prefer what Ian's implementation does of passing these DATAGRAM frames up immediately to the application. I don't think that the acknowledgment needs to indicate that the frame was processed by the application, but merely that it has been delivered to the application (that is, the application doesn't get to do anything with the frame that can influence the acknowledgment).

The current draft indicates that the content of the DATAGRAM frames contributes to the limit used for MAX_DATA, and that if that amount is reached, the frames are blocked along with STREAM data. I think this works fine for the sender, while the receiver gets into the discussion you present. On the sender side, reaching MAX_DATA could mean dropping the DATAGRAM frames when unable to send more (and sending BLOCKED instead). Since the frames are unreliable, they can be dropped in this situation without violating the API contract. 

On the receiver side, I agree that queuing the DATAGRAM frames to let the application drive flow control in the way it does for STREAM frames adds complexity and diminishes the utility of the frame and ACKs. However, I can imagine taking a fairly simplistic approach in which the data limit is automatically increased upon reception of the frame (and the frame is immediately passed to the application). This allows the initial_max_data to put a cap on the amount of data in a given flight of DATAGRAMS, and allow the size of a flight of DATAGRAM frames to be limited by the amount of room left over from STREAM data that may be consuming the connection-wide flow control. 

Perhaps this approach needs a clearer name other than "flow control", since it has a somewhat different meaning in effect.

As for ACKs, if we never discard on the receiver side, the ACK is pretty useful for detecting if there was network-based packet loss.

Thanks,
Tommy 

> On Oct 29, 2018, at 5:32 AM, Ian Swett <[email-masked]> wrote:
> 
> Good catch Martin, I missed that in the draft as well, and I also think it's impossibly with the proposed design.
> 
> And yes, I think Martin's proposed solution is likely the only practical one.  In my implementation, the frame is passed up to the application immediately, so technically QUIC processed it, and it's the application's job to decide what to do with it.
> 
> On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Tommy,
> 
> Your slides - <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf>>

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 29 Oct 2018 12:37:36 -0700

Tommy,

Changing the semantics of an acknowledgment to include delivery up to the
application is a fundamental change to the QUIC machinery, and it doesn't
work. First, an ACK frame acknowledges packets, and you can't have
different semantics of an acknowledgment for different frames that are
carried in the same packet. Second, it interferes with RTT measurement, and
it conflates flow control with congestion control, which gets messy. (This
conflation is an interesting problem to consider theoretically, but not one
for us at this time IMO.)

I am wondering if applying a stream-level flow control for DATAGRAMs makes
sense instead. Meaning that you treat DATAGRAMs as a separate stream for
flow control purposes. You might benefit from having an offset in the
DATAGRAM frame for this purpose.

- jana

On Mon, Oct 29, 2018 at 8:21 AM Tommy Pauly <[email-masked]> wrote:

> Hi Martin, Ian,
>
> Yes, very good points!
>
> My tendency would be to prefer what Ian's implementation does of passing
> these DATAGRAM frames up immediately to the application. I don't think that
> the acknowledgment needs to indicate that the frame was processed by the
> application, but merely that it has been delivered to the application (that
> is, the application doesn't get to do anything with the frame that can
> influence the acknowledgment).
>
> The current draft indicates that the content of the DATAGRAM frames
> contributes to the limit used for MAX_DATA, and that if that amount is
> reached, the frames are blocked along with STREAM data. I think this works
> fine for the sender, while the receiver gets into the discussion you
> present. On the sender side, reaching MAX_DATA could mean dropping the
> DATAGRAM frames when unable to send more (and sending BLOCKED instead).
> Since the frames are unreliable, they can be dropped in this situation
> without violating the API contract.
>
> On the receiver side, I agree that queuing the DATAGRAM frames to let the
> application drive flow control in the way it does for STREAM frames adds
> complexity and diminishes the utility of the frame and ACKs. However, I can
> imagine taking a fairly simplistic approach in which the data limit is
> automatically increased upon reception of the frame (and the frame is
> immediately passed to the application). This allows the initial_max_data to
> put a cap on the amount of data in a given flight of DATAGRAMS, and allow
> the size of a flight of DATAGRAM frames to be limited by the amount of room
> left over from STREAM data that may be consuming the connection-wide flow
> control.
>
> Perhaps this approach needs a clearer name other than "flow control",
> since it has a somewhat different meaning in effect.
>
> As for ACKs, if we never discard on the receiver side, the ACK is pretty
> useful for detecting if there was network-based packet loss.
>
> Thanks,
> Tommy
>

---

From: Tommy Pauly <[email-masked]>
Date: Mon, 29 Oct 2018 13:01:11 -0700

Hi Jana,

I'm not suggesting any changes to when the packet gets ack'ed—specifically, I was responding to Martin's hypothetical of having an ACK to a DATAGRAM frame meaning that the application had processed the frame. My impression is that the ACK to a packet containing a DATAGRAM frame means:
- The packet made it across the network to the QUIC endpoint on the other side
- The QUIC implementation will deliver the DATAGRAM frame to application (it won't drop it locally)

Does that make sense?

Having a separate outstanding data limit for the DATAGRAM "stream" is an interesting solution to the space. It would then have the nice property of not looking like traditional flow control. It could even be measured in number of frames, rather than bytes (depending on what the limiting factors are).

Thanks,
Tommy

> On Oct 29, 2018, at 12:37 PM, Jana Iyengar <[email-masked]> wrote:
> 
> Tommy,
> 
> Changing the semantics of an acknowledgment to include delivery up to the application is a fundamental change to the QUIC machinery, and it doesn't work. First, an ACK frame acknowledges packets, and you can't have different semantics of an acknowledgment for different frames that are carried in the same packet. Second, it interferes with RTT measurement, and it conflates flow control with congestion control, which gets messy. (This conflation is an interesting problem to consider theoretically, but not one for us at this time IMO.)
> 
> I am wondering if applying a stream-level flow control for DATAGRAMs makes sense instead. Meaning that you treat DATAGRAMs as a separate stream for flow control purposes. You might benefit from having an offset in the DATAGRAM frame for this purpose.
> 
> - jana
> 
> On Mon, Oct 29, 2018 at 8:21 AM Tommy Pauly <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Martin, Ian,
> 
> Yes, very good points!
> 
> On the receiver side, I agree that queuing the DATAGRAM frames to let the application drive flow control in the way it does for STREAM frames adds complexity and diminishes the utility of the frame and ACKs.. However, I can imagine taking a fairly simplistic approach in which the data limit is automatically increased upon reception of the frame (and the frame is immediately passed to the application). This allows the initial_max_data to put a cap on the amount of data in a given flight of DATAGRAMS, and allow the size of a flight of DATAGRAM frames to be limited by the amount of room left over from STREAM data that may be consuming the connection-wide flow control. 
> 
>> On Oct 29, 2018, at 5:32 AM, Ian Swett <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Mon, 29 Oct 2018 21:28:45 +0000

> ACK to a packet containing a DATAGRAM frame means:

This only makes sense, if there is an effective flow control for DATAGRAMs.  The spec should be implementable by poll/read QUIC implementations as well as by callback-on-network-receive-thread implementations.  Unless there is an effective flow control, a buffering QUIC implementation must be able to discard DATAGRAMs, if the receiver app is not catching up.  The receiver would then have to ignore all non-DATAGRAM frames in that packet, since it would be unable to ACK the packet.  If the receiver is processing DATAGRAMs and STREAMs at different speeds, deliberate discards of packets containing DATAGRAMs would interfere with the congestion control, effectively HOL blocking STREAMs.

An effective flow control requires sender and receiver to synchronize their view of the flow control counter (MAX_DATA for connection flow control).  Relying on ACKs is very hard, since the receiver cannot know how much data the sender has sent (unless the sender is explicit about it).

If flow control for DATAGRAMs is desired, you can use mechanics such as in https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-02 or https://tools.ietf.org/html/draft-lubashev-quic-partial-reliability-03 (https://github.com/igorlord/draft-lubashev-quic-partial-reliability).  (This would be treating DATAGRAMs as belonging to a partially reliable stream, or streams.  Except you would not need to ensure ordering of DATAGRAMs in that stream.)

  *   Igor

From: Tommy Pauly <[email-masked]>
Sent: Monday, October 29, 2018 4:01 PM
To: Jana Iyengar <[email-masked]>
Cc: Ian Swett <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: flow control and DATAGRAM

Hi Jana,

Does that make sense?

Thanks,
Tommy

On Oct 29, 2018, at 12:37 PM, Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:

Tommy,

- jana

On Mon, Oct 29, 2018 at 8:21 AM Tommy Pauly <[email-masked]<mailto:[email-masked]>> wrote:
Hi Martin, Ian,

Yes, very good points!

Thanks,
Tommy

On Oct 29, 2018, at 5:32 AM, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:

On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
Hi Tommy,

Your slides - <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_wg-2Dmaterials_blob_master_ietf103_IETF103-2DQUIC-2DDatagram.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=-JAVyKK4LrDFUn-TzI96ZFlrFZYozg4GFVttnjIjGdg&s=tqUQZW-hP0GwZ7x1otPkSYpA3axGVizYu0a4JlLou6Q&e=>>
- say that DATAGRAM frames respect connection-level flow control.  I
missed that in the draft, and I don't know how they can do that in the
face of packet loss, especially when you don't necessarily retransmit
lost DATAGRAM frames.

Cheers,
Martin

---

From: Christian Huitema <[email-masked]>
Date: Mon, 29 Oct 2018 15:28:49 -0700

> On Oct 29, 2018, at 2:28 PM, Lubashev, Igor <[email-masked]> wrote:
> 

On the other hand, Datagram will be used to carry the media flows currently carried over RTP, and RTP does not include any kind of flow control. 

The known weakness of RTP is the lack of integrated congestion control between data and media, do that a parallel data download can jeopardize  the quality of the real time media. But that's an argument for counting the datagrams in the congestion control window, not an argument for flow control.

There is an argument for controlling the rate at which media is sent. But that's typically done at the app layer, by negotiating codecs and compression parameters.

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Mon, 29 Oct 2018 22:50:06 +0000

I think we are in an agreement here.  One design alternative is no flow control (only congestion control) for datagrams.  But in that case, the assertion below does not hold, since the second bullet is false for “poll/read”-based implementations (or any other implementation that is not based on “callback-on-network-receive-thread” model).

  *   Igor

From: Christian Huitema <[email-masked]>
Sent: Monday, October 29, 2018 6:29 PM
To: Lubashev, Igor <[email-masked]>
Cc: Tommy Pauly <[email-masked]>; Jana Iyengar <[email-masked]>; Ian Swett <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: flow control and DATAGRAM

-- Christian Huitema

---

From: Martin Thomson <[email-masked]>
Date: Tue, 30 Oct 2018 09:52:04 +1100

> - The QUIC implementation will deliver the DATAGRAM frame to application (it won't drop it locally)

That works.  If you allow for the possibility that an application that
is unable to process the DATAGRAM will be indistinguishable from QUIC
dropping it.  That is, if the application can't take the upcall
because it is too busy, then QUIC might not bother waiting.

> Having a separate outstanding data limit for the DATAGRAM "stream" is an interesting solution to the space. It would then have the nice property of not looking like traditional flow control. It cou…

This is something I alluded to in my email.  I don't think that it
improves things much, other than perhaps insulating streams from the
excesses of DATAGRAM frames and vice versa.

If you have a limit, then you need to ensure that both endpoints agree
on its value eventually.  With DATAGRAM, you might think that it's
possible to add a new frame that expresses how much data was sent
(that is the sum of sizes of DATAGRAM frames).  That would run afoul
of reordering problems, where the limit is exceeded when delayed
DATAGRAM frames that are already counted, arrive after the accounting
frame.

You could limit the count to just lost DATAGRAM frames.  That would
allow a receiver to repair its view of how much was used.  However
that runs afoul of two problems: acknowledgments aren't reliable, and
packets can be declared lost and still delivered.  In both cases, the
sender can believe that frames didn't arrive and so increase their
report of "lost bytes" too much.  The receiver ends up with an
inflated view of what was consumed (as opposed to the value being
short).

If you wanted to solve this problem, you would probably need to have
per-frame tracking, which makes this a lot less lightweight than I
think we want.

---

From: Roberto Peon <[email-masked]>
Date: Mon, 29 Oct 2018 23:06:42 +0000

Such per-frame tracking could be a single bit if the protocol using DATAGRAM is willing to do windowed updates (i.e. 1 in flight at once) of such things.
-=R

On 10/29/18, 3:52 PM, "QUIC on behalf of Martin Thomson" <[email-masked] on behalf of [email-masked]> wrote:

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 29 Oct 2018 19:23:59 -0700

Hey Tommy,

> I'm not suggesting any changes to when the packet gets
> ack'ed—specifically, I was responding to Martin's hypothetical of having an
> ACK to a DATAGRAM frame meaning that the application had processed the
> frame. My impression is that the ACK to a packet containing a DATAGRAM
> frame means:

... and was processed by the QUIC receiver (not necessarily by the
application). This is the same semantic as the ack of a regular frame.

> (it won't drop it locally)

In which case you will need flow control. If you agree, then we're on the
same page so far.

> Having a separate outstanding data limit for the DATAGRAM "stream" is an
> interesting solution to the space. It would then have the nice property of
> not looking like traditional flow control. It could even be measured in
> number of frames, rather than bytes (depending on what the limiting factors
> are).
>

In terms of flow control, I don't think DATAGRAM flow control is any
different than sending stream data on any stream and then canceling it --
there are no retransmissions, but the sender still accounts for it in flow
control. My thought here was basically to have exactly the same flow
control as stream-level flow control, and allow for DATAGRAM bytes to fall
within connection-level flow control as normal.

The simplest scheme would be to reserve a stream ID for DATAGRAM data --
this could be 0 or 2^62-1 -- and then MAX_STREAM_DATA uses this stream ID
for flow control of DATAGRAM octets. Alternatively, define a new frame
called MAX_DATAGRAM_DATA that is carries the largest allowed offset that
can be sent as a DATAGRAM, and introduce an offset field in the DATAGRAM
frame.

- jana

>> complexity and diminishes the utility of the frame and ACKs.. However, I
>> can imagine taking a fairly simplistic approach in which the data limit is

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 30 Oct 2018 03:54:51 +0000

Jana,

> The simplest scheme would be to reserve a stream ID for DATAGRAM data -- [...]
> -- and then MAX_STREAM_DATA uses this
> stream ID for flow control of DATAGRAM octets.

There are many dragons that way. Flow control indicates commitment to buffer data. When sender does not retransmit, when does receiver give up its commitment to wait for (and buffer) earlier data so it can advance MAX_STREAM_DATA to allow the sender to send more data? Can the signal from sender tolerate loss/reordering vs STREAM frames (assuming the datagram can be split for sending in multiple QUIC packets)? These dragons can be conquered (see my partially reliable streams draft, v2 or v3), but this is not simple. :(

- Igor

-----Original Message-----
From: Jana Iyengar [[email-masked]]
Received: Monday, 29 Oct 2018, 10:24PM
To: [email-masked] [[email-masked]]
CC: Ian Swett [[email-masked]]; QUIC WG [[email-masked]]; Martin Thomson [[email-masked]]
Subject: Re: flow control and DATAGRAM

Hey Tommy,

... and was processed by the QUIC receiver (not necessarily by the application). This is the same semantic as the ack of a regular frame.

In which case you will need flow control. If you agree, then we're on the same page so far.

In terms of flow control, I don't think DATAGRAM flow control is any different than sending stream data on any stream and then canceling it -- there are no retransmissions, but the sender still accounts for it in flow control. My thought here was basically to have exactly the same flow control as stream-level flow control, and allow for DATAGRAM bytes to fall within connection-level flow control as normal.

The simplest scheme would be to reserve a stream ID for DATAGRAM data -- this could be 0 or 2^62-1 -- and then MAX_STREAM_DATA uses this stream ID for flow control of DATAGRAM octets. Alternatively, define a new frame called MAX_DATAGRAM_DATA that is carries the largest allowed offset that can be sent as a DATAGRAM, and introduce an offset field in the DATAGRAM frame.

- jana

Thanks,
Tommy

Tommy,

- jana

Yes, very good points!

Thanks,
Tommy

Your slides - <https://github.com/quicwg/wg-materials/blob/master/ietf103/IETF103-QUIC-Datagram.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_wg-2Dmaterials_blob_master_ietf103_IETF103-2DQUIC-2DDatagram.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=BOGSfZ4hKaubJ6yww-9_8EDX3OccYvdip0g3k4t66hQ&s=qiyr1YY1MHWjhIrPfwHXN6FCdsG8wUuhAuqczkosLlI&e=>>
- say that DATAGRAM frames respect connection-level flow control.  I
missed that in the draft, and I don't know how they can do that in the
face of packet loss, especially when you don't necessarily retransmit
lost DATAGRAM frames.

Cheers,
Martin

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 29 Oct 2018 21:23:47 -0700

Igor,

I had thought that this was similar to canceling streams, but you're right.
Canceling a stream gets rid of stream flow control state entirely, where in
this case, the receiver cannot discard flow control state.

I don't have any great ideas right now, but I fear that not having flow
control simply punts the problem down the road. FWIW, this may be quite
fine for this extension, and we can surely evolve it to have flow control
when we have a way to do it.

- jana

On Mon, Oct 29, 2018 at 8:54 PM Lubashev, Igor <[email-masked]> wrote:

> [...]
> There are many dragons that way. Flow control indicates commitment to
> buffer data. When sender does not retransmit, when does receiver give up
> its commitment to wait for (and buffer) earlier data so it can advance
> MAX_STREAM_DATA to allow the sender to send more data? Can the signal from
> sender tolerate loss/reordering vs STREAM frames (assuming the datagram can
> be split for sending in multiple QUIC packets)? These dragons can be
> conquered (see my partially reliable streams draft, v2 or v3), but this is
> not simple. :(
>
> - Igor
>
> *From:* Jana Iyengar [[email-masked]]
> *Received:* Monday, 29 Oct 2018, 10:24PM
> *To:* [email-masked] [[email-masked]]
> *CC:* Ian Swett [[email-masked]]; QUIC WG [[email-masked]]; Martin
> Thomson [[email-masked]]
> *Subject:* Re: flow control and DATAGRAM
>
> Hey Tommy,
>
>
>> I am wondering if applying a stream-level flow control for DATAGRAMs
>> makes sense instead. Meaning that you treat DATAGRAMs as a separate stream
>> for flow control purposes. You might benefit from having an offset in the
>>> On the receiver side, I agree that queuing the DATAGRAM frames to let
>>> the application drive flow control in the way it does for STREAM frames
>>> adds complexity and diminishes the utility of the frame and ACKs.. However,
>>> I can imagine taking a fairly simplistic approach in which the data limit
>>> is automatically increased upon reception of the frame (and the frame is
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_wg-2Dmaterials_blob_master_ietf103_IETF103-2DQUIC-2DDatagram.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=BOGSfZ4hKaubJ6yww-9_8EDX3OccYvdip0g3k4t66hQ&s=qiyr1YY1MHWjhIrPfwHXN6FCdsG8wUuhAuqczkosLlI&e=>
>>>> >

---

From: Ian Swett <[email-masked]>
Date: Wed, 31 Oct 2018 12:00:53 -0400

If you add flow control, there's little benefit over a unidirectional
stream, so adding flow control fundamentally breaks this design.

My goal is to use this for cases when I want to use a single QUIC
connection to replace DTLS or RTP + something reliable.  WebRTC is the
obvious use case, but not the only one.  DTLS and RTP don't have flow
control, so adding it just make the application mapping a lot more complex.

On Tue, Oct 30, 2018 at 12:23 AM Jana Iyengar <[email-masked]> wrote:

> Igor,
>
> I had thought that this was similar to canceling streams, but you're
> right. Canceling a stream gets rid of stream flow control state entirely,
> where in this case, the receiver cannot discard flow control state.
>
>>> Changing the semantics of an acknowledgment to include delivery up to
>>> the application is a fundamental change to the QUIC machinery, and it
>>> doesn't work. First, an ACK frame acknowledges packets, and you can't have
>>>> My tendency would be to prefer what Ian's implementation does of
>>>> passing these DATAGRAM frames up immediately to the application. I don't
>>>> think that the acknowledgment needs to indicate that the frame was
>>>> processed by the application, but merely that it has been delivered to the
>>>> application (that is, the application doesn't get to do anything with the
>>>> frame that can influence the acknowledgment).
>>>>
>>>> As for ACKs, if we never discard on the receiver side, the ACK is
>>>> pretty useful for detecting if there was network-based packet loss.
>>>>
>>>> Thanks,
>>>> Tommy
>>>>
>>>> And yes, I think Martin's proposed solution is likely the only
>>>> practical one.  In my implementation, the frame is passed up to the
>>>> application immediately, so technically QUIC processed it, and it's the
>>>> application's job to decide what to do with it.
>>>>
>>>> On Mon, Oct 29, 2018 at 1:16 AM Martin Thomson <
>>>> [email-masked]> wrote:
>>>>
>>>>> Hi Tommy,
>>>>>
>>>>> Your slides - <

---

From: David Schinazi <[email-masked]>
Date: Wed, 31 Oct 2018 13:36:17 -0700

I agree.

Another use case is VPN, where one could want to have the control traffic
(e.g. what IKEv2 does today) be sent reliably with flow control and have
the data path (e.g. what ESP does today) be sent unreliably without flow
control. As with ESP today, flow control is handled by upper layer
protocols (e.g. TCP) inside the encryption/encapsulation.

I think it might be best to remove flow control from this proposal
entirely, and if later someone has a use-case for flow-controlled datagrams
we can define a new different extension then.

David

On Wed, Oct 31, 2018 at 9:01 AM Ian Swett <ianswett=
[email-masked]> wrote:

>>> Thomson [[email-masked] <[email-masked]>]
>>>> Having a separate outstanding data limit for the DATAGRAM "stream" is
>>>> an interesting solution to the space. It would then have the nice property
>>>> of not looking like traditional flow control. It could even be measured in
>>>>> Good catch Martin, I missed that in the draft as well, and I also
>>>>> think it's impossibly with the proposed design.
>>>>>
>>>>> martin.thomson@gmail..com <[email-masked]>> wrote:
>>>>>
>>>>>> Hi Tommy,
>>>>>>
>>>>>> Your slides - <

---

From: Mike Bishop <[email-masked]>
Date: Thu, 1 Nov 2018 04:08:24 +0000

Technically, an implementation could choose not to ACK without ignoring all frames in the packet.  Our design for frames says that they can be sent/received multiple times without error, so a spuriously-retransmitted reliable frame from that packet is wasted data, but not an error.  However, it might not be totally clear to application designers that, while ACKs can tell you which datagrams definitely did arrive, they can never tell you which ones definitely didn’t.

But it sounds like where this is going is the realization that you have to have flow control if you’re going to prohibit the remote implementation from dropping datagrams after ACKing them; but if you do allow the implementation to do that, then ACKs tell you nothing concrete about DATAGRAMs that might have been in those packets.

From: QUIC <[email-masked]> On Behalf Of Lubashev, Igor
Sent: Monday, October 29, 2018 4:29 PM
To: Tommy Pauly <[email-masked]>; Jana Iyengar <[email-masked]>
Cc: Ian Swett <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: RE: flow control and DATAGRAM

  *   Igor

From: Tommy Pauly <[email-masked]<mailto:[email-masked]>>
Sent: Monday, October 29, 2018 4:01 PM
To: Jana Iyengar <[email-masked]<mailto:[email-masked]>>
Cc: Ian Swett <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>
Subject: Re: flow control and DATAGRAM

Hi Jana,

Does that make sense?

Thanks,
Tommy

Tommy,

- jana

Yes, very good points!

Thanks,
Tommy

Cheers,
Martin