Subject: [tcpm] Thank you for the QUIC session in tcpm

From: Bob Briscoe <[email-masked]>
Date: Thu, 8 Nov 2018 11:14:44 +0000

I just wanted to thank Jana for explaining QUIC loss recovery to us (and 
QUIC CC as far as it goes).
And thank you Jana, Ian, the chairs of both WGs (and anyone else 
involved) for setting it up.

If one is not full-time on QUIC, it's very difficult to keep up with all 
the changes. But now we have a checkpoint to start from, I feel I will 
not be wasting people's time if I try to get involved - at least I only 
might say something un-QUIC occasionally, rather than nearly always. 
This has allowed people who understand how TCP cold be improved to help 
with QUIC, when working on QUIC isn't their day job.

Again, Thank you.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 8 Nov 2018 16:37:55 -0800

On Thu, Nov 8, 2018 at 3:14 AM, Bob Briscoe <[email-masked]> wrote:
>
> I just wanted to thank Jana for explaining QUIC loss recovery to us (and QUIC CC as far as it goes).
> And thank you Jana, Ian, the chairs of both WGs (and anyone else involved) for setting it up.
>
> If one is not full-time on QUIC, it's very difficult to keep up with all the changes. But now we have a checkpoint to start from, I feel I will not be wasting people's time if I try to get involved - at least I only might say something un-QUIC occasionally, rather than nearly always. This has allowed people who understand how TCP cold be improved to help with QUIC, when working on QUIC isn't their day job.
>
> Again, Thank you.
I like particularly that QUIC only reduces cwnd to 1 after the loss is
confirmed not upon RTO fires. It should be very feasible for TCP (at
least Linux) w/ TCP timestamps. It'll save a lot of spurious cwnd
reductions!

Also IMHO TCP w/ quality timestamps are almost as good as QUIC
pkt-ids. Google internally uses usec. We wish we could upstream it but
RFC needs to be updated.

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Nov 2018 01:34:34 +0000

Yuchung I brought that difference up in an email to the quic wg earlier this week. 

In app send limited case, inflight could be very small compared to cwnd. So in QUIC there is potential to send a burst out after a long idle period (with outstanding data) where TCP wouldn't. The draft claims this is okay to do because RTO may have been a result of RTT increase instead of loss. Is there data to suggest on which side we should err? i.e. data on what are the chances that an RTO is due to an RTT increase versus loss.

Do you see any safety concerns with delayed reduction of cwnd in case where the RTO is not spurious?

-----Original Message-----
From: tcpm <[email-masked]> On Behalf Of Yuchung Cheng
Sent: Thursday, November 8, 2018 4:38 PM
To: Bob Briscoe <[email-masked]>
Cc: Ian Swett <[email-masked]>; tcpm IETF list <[email-masked]>
Subject: Re: [tcpm] Thank you for the QUIC session in tcpm

> If one is not full-time on QUIC, it's very difficult to keep up with all the changes. But now we have a checkpoint to start from, I feel I will not be wasting people's time if I try to get involved…
I like particularly that QUIC only reduces cwnd to 1 after the loss is confirmed not upon RTO fires. It should be very feasible for TCP (at least Linux) w/ TCP timestamps. It'll save a lot of spurious cwnd reductions!

Also IMHO TCP w/ quality timestamps are almost as good as QUIC pkt-ids. Google internally uses usec. We wish we could upstream it but RFC needs to be updated.

> Bob Briscoe                               https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&amp;data=02%7C01%7Cpravb%40microsoft.com%7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773207316711149&amp;sdata=0pt6YfpYOaF%2B5IWEKY3LMj1idEnYbApqUvWzhtLa5qE%3D&amp;reserved=0
>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.i
> etf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microso
> ft.com%7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7cd0
> 11db47%7C1%7C0%7C636773207316711149&amp;sdata=K667a3IQG4rarQ%2FOfAlyhK
> QQ05Cea421rgb64DlEMvs%3D&amp;reserved=0

_______________________________________________
tcpm mailing list
[email-masked]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microsoft.com%7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773207316711149&amp;sdata=K667a3IQG4rarQ%2FOfAlyhKQQ05Cea421rgb64DlEMvs%3D&amp;reserved=0

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 9 Nov 2018 11:01:53 +0700

Thank you for the feedback, Bob; we really appreciate it. And of course thanks to Jana for going the extra distance here.

Cheers,

> On 8 Nov 2018, at 6:14 pm, Bob Briscoe <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 9 Nov 2018 04:44:50 -0800

On Thu, Nov 8, 2018 at 4:37 PM Yuchung Cheng <ycheng=
[email-masked]> wrote:

> the changes. But now we have a checkpoint to start from, I feel I will not
> be wasting people's time if I try to get involved - at least I only might
> say something un-QUIC occasionally, rather than nearly always. This has
> allowed people who understand how TCP cold be improved to help with QUIC,
> when working on QUIC isn't their day job.
> >
> > Again, Thank you.

+1

> pkt-ids. Google internally uses usec. We wish we could upstream it but

I agree using timestamp for the purpose.
But, if we try to utilize the TS like pkt-id, I personally think it'll be
similar to use it as extend seqno like draft-bagnulo-tcpm-esn-00 rather
than fine grade TS. :)

In any cases, it seems to me that many people think 7232 TS is not being
very efficient for today's internet.
it might be good time to think about extending TS option just like what
we're doing on ECN..  TS++?
--
Yoshi

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 12 Nov 2018 19:10:49 +0700

Thank you for that feedback, Bob. Our goal was to drive active
participation from more TCPM experts and regulars, and I hope we see more
of it. As for saying something un-QUIC, I wouldn't worry, but I understand
your point. I'm glad the presentation helped.

If it's valuable, we'd be happy to do another presentation in Prague with
changes between now and then.

On Fri, Nov 9, 2018 at 7:45 PM Yoshifumi Nishida <[email-masked]>
wrote:

>> > If one is not full-time on QUIC, it's very difficult to keep up with
>> all the changes. But now we have a checkpoint to start from, I feel I will
>> might say something un-QUIC occasionally, rather than nearly always. This
>> has allowed people who understand how TCP cold be improved to help with
>> QUIC, when working on QUIC isn't their day job.
>> >
>> > Again, Thank you.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 12 Nov 2018 19:13:32 +0700

Praveen,

The point you're raising -- that we've lost the ack clock after an RTO --
is a reasonable point. My argument is that with pacing, cwnd reduction is
unwarranted because in the extreme case, this collapses to restart after
idle, where sending at cwnd with pacing is reasonable.

The draft does not say that a sender should reduce the cwnd if it is not
pacing, which we should add. Does that make sense?

- jana

On Fri, Nov 9, 2018 at 8:34 AM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> In app send limited case, inflight could be very small compared to cwnd.
> So in QUIC there is potential to send a burst out after a long idle period
> (with outstanding data) where TCP wouldn't. The draft claims this is okay
> to do because RTO may have been a result of RTT increase instead of loss.
> Is there data to suggest on which side we should err? i.e. data on what are
> the chances that an RTO is due to an RTT increase versus loss.
>
> confirmed not upon RTO fires. It should be very feasible for TCP (at least
> Linux) w/ TCP timestamps. It'll save a lot of spurious cwnd reductions!
>
> Also IMHO TCP w/ quality timestamps are almost as good as QUIC pkt-ids.
> Google internally uses usec. We wish we could upstream it but RFC needs to
> be updated.
>
> >
> >
> > Bob
> >
> >
> > --
> > Bob Briscoe
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&amp;data=02%7C01%7Cpravb%40microsoft.com%7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773207316711149&amp;sdata=0pt6YfpYOaF%2B5IWEKY3LMj1idEnYbApqUvWzhtLa5qE%3D&amp;reserved=0
> >

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 12 Nov 2018 08:37:03 -0800

On Mon, Nov 12, 2018 at 4:13 AM, Jana Iyengar <[email-masked]> wrote:
> Praveen,
>
> The point you're raising -- that we've lost the ack clock after an RTO -- is
> a reasonable point. My argument is that with pacing, cwnd reduction is
> idle, where sending at cwnd with pacing is reasonable
I agree w/ Jana as well - the most generic way to treat burst due to
prior_inflight << cwnd is pacing. With QUIC's approach, when RTO
fires, cwnd remains unchanged until the ACK of the first
retransmission (i.e. a probe packet) comes back. Therefore the delay
is one RTT and the potential damage is an additional cwnd-worth of
burst.

Yes the worst case is more aggressive, and it can be too much for DC
incast case if pacing isn't supported - one idea is to selective
enable that if RTT variance is
very large vs RTT.

> On Fri, Nov 9, 2018 at 8:34 AM Praveen Balasubramanian
> <pravb=[email-masked]> wrote:
>>
>> (with outstanding data) where TCP wouldn't. The draft claims this is okay to
>> do because RTO may have been a result of RTT increase instead of loss. Is
>> there data to suggest on which side we should err? i.e. data on what are the
>> chances that an RTO is due to an RTT increase versus loss.
>>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Mon, 12 Nov 2018 17:35:34 +0000

Yuchung this might be worth experimenting with and getting more data on. Maybe there should be some reaction to an unconfirmed RTO even if it is not reduction all the way to LW? I like Jana's suggestion that the implementation SHOULD drop the cwnd even for unconfirmed RTO if not pacing. 

I checked the latest draft and pacing is a RECOMMENDED. I am noticing that transport algorithms in Linux and in QUIC are evolving assuming pacing which is a function usually implemented and configured outside of the transport. So the transport is potentially making an assumption which may result in safety issues. For example BBR congestion control may not work very well if pacing was disabled or misconfigured? I don’t know of any robust solution to this other than build pacing into the transport itself. Pacing is also very challenging for low latency flows because of lack of support for fine grained timers.

-----Original Message-----
From: Yuchung Cheng <[email-masked]> 
Sent: Monday, November 12, 2018 8:37 AM
To: Jana Iyengar <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; Bob Briscoe <[email-masked]>; Ian Swett <[email-masked]>; [email-masked] Extensions <[email-masked]>
Subject: Re: [tcpm] Thank you for the QUIC session in tcpm

On Mon, Nov 12, 2018 at 4:13 AM, Jana Iyengar <[email-masked]> wrote:
> Praveen,
>
> The point you're raising -- that we've lost the ack clock after an RTO 
> -- is a reasonable point. My argument is that with pacing, cwnd 
> reduction is unwarranted because in the extreme case, this collapses 
> to restart after idle, where sending at cwnd with pacing is reasonable
I agree w/ Jana as well - the most generic way to treat burst due to prior_inflight << cwnd is pacing. With QUIC's approach, when RTO fires, cwnd remains unchanged until the ACK of the first retransmission (i.e. a probe packet) comes back. Therefore the delay is one RTT and the potential damage is an additional cwnd-worth of burst.

Yes the worst case is more aggressive, and it can be too much for DC incast case if pacing isn't supported - one idea is to selective enable that if RTT variance is very large vs RTT.

>
> The draft does not say that a sender should reduce the cwnd if it is 
> not pacing, which we should add. Does that make sense?
>
> - jana
>
>> So in QUIC there is potential to send a burst out after a long idle 
>> period (with outstanding data) where TCP wouldn't. The draft claims 
>> this is okay to do because RTO may have been a result of RTT increase 
>> instead of loss. Is there data to suggest on which side we should 
>> err? i.e. data on what are the chances that an RTO is due to an RTT increase versus loss.
>>
>> > If one is not full-time on QUIC, it's very difficult to keep up 
>> > with all the changes. But now we have a checkpoint to start from, I 
>> > feel I will not be wasting people's time if I try to get involved - 
>> > at least I only might say something un-QUIC occasionally, rather 
>> > than nearly always. This has allowed people who understand how TCP 
>> > cold be improved to help with QUIC, when working on QUIC isn't their day job.
>> >
>> > Again, Thank you.
>> I like particularly that QUIC only reduces cwnd to 1 after the loss 
>> is confirmed not upon RTO fires. It should be very feasible for TCP 
>> (at least
>> Google internally uses usec. We wish we could upstream it but RFC 
>> needs to be updated.
>>
>> >
>> >
>> > Bob
>> >
>> >
>> > --
>> > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbob
>> > briscoe.net%2F&amp;data=02%7C01%7Cpravb%40microsoft.com%7C3c4d22866
>> > 8444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%
>> > 7C636776374678321324&amp;sdata=T%2BWTEc42VC%2Bz%2BsmMFyjlm37hmAwfee
>> > buPcuMYlVhgDY%3D&amp;reserved=0
>> >
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
>> ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> soft.com%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7
>> cd011db47%7C1%7C0%7C636776374678321324&amp;sdata=33WxKh5c6qL4ln5jerpx
>> Rhytfrj1iTK284pEqv5fWKY%3D&amp;reserved=0
>>
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
>> ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> soft.com%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7
>> cd011db47%7C1%7C0%7C636776374678321324&amp;sdata=33WxKh5c6qL4ln5jerpx
>> Rhytfrj1iTK284pEqv5fWKY%3D&amp;reserved=0

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 13 Nov 2018 14:00:34 +0700

Praveen,

I think that's a deployment configuration question... you're right to note
that BBR requires pacing, but I don't think the solution here is
necessarily to build pacing into the transport. It's certainly useful to
build it in as a safeguard in case the deployment environment does not
implement pacing, but I think it still is useful to describe and implement
controllers for environments that implement pacing.

- jana

On Tue, Nov 13, 2018 at 12:35 AM Praveen Balasubramanian <
[email-masked]> wrote:

> Yuchung this might be worth experimenting with and getting more data on.
> Maybe there should be some reaction to an unconfirmed RTO even if it is not
> reduction all the way to LW? I like Jana's suggestion that the
> implementation SHOULD drop the cwnd even for unconfirmed RTO if not pacing.
>
> I checked the latest draft and pacing is a RECOMMENDED. I am noticing that
> transport algorithms in Linux and in QUIC are evolving assuming pacing
> which is a function usually implemented and configured outside of the
> transport. So the transport is potentially making an assumption which may
> result in safety issues. For example BBR congestion control may not work
> very well if pacing was disabled or misconfigured? I don’t know of any
> robust solution to this other than build pacing into the transport itself.
> Pacing is also very challenging for low latency flows because of lack of
> support for fine grained timers.
>
> Cc: Praveen Balasubramanian <[email-masked]>; Bob Briscoe <
> [email-masked]>; Ian Swett <[email-masked]>; [email-masked]
> Extensions <[email-masked]>
> prior_inflight << cwnd is pacing. With QUIC's approach, when RTO fires,
> cwnd remains unchanged until the ACK of the first retransmission (i.e. a
> probe packet) comes back. Therefore the delay is one RTT and the potential
> damage is an additional cwnd-worth of burst.
>
> incast case if pacing isn't supported - one idea is to selective enable
> that if RTT variance is very large vs RTT.
>
> >

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 13 Nov 2018 14:13:12 +0700

On what to do with an unconfirmed RTO: So, my thought is that if you've hit
an RTO, it's been a REALLY LONG time, because we've gone through TLP(s) by
now and failed. (This might change if we decide to combine TLP and RTO, but
that's for a later discussion.)  It's not fully clear to me what the right
responses here are, but some thoughts follow.

1/ If this was an actual increase in the RTT, it would naturally roll into
the SRTT and RTTVAR. We might even consider resetting the RTT estimator,
but we already know that the EWMA estimator we love takes just a while to
catch on, and I don't want to litigate changing the estimator fundamentally
here.

2/ Linux fq uses min_rtt IIRC, which shouldn't change as a result of an
increase in RTT. This *might* be a good time to kick the min_rtt estimator
to pace more conservatively.

3/ I'm not convinced that it makes sense to change the cwnd. It seems clear
that an RTT increase with the same BW should increase the cwnd if anything,
so it seems reasonable to continue with the same cwnd at least. If the BW
changed, hopefully that'll be reflected in a congestion signal, but in the
absence of a congestion signal, I think we should keep the cwnd as is.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 13 Nov 2018 17:03:01 +0000

We welcome your involvement!

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Bob Briscoe
Sent: Thursday, November 8, 2018 3:15 AM
To: [email-masked]; [email-masked]; Jana Iyengar <[email-masked]>; Ian Swett <[email-masked]>
Cc: tcpm IETF list <[email-masked]>; QUIC IETF mailing list <[email-masked]>
Subject: Thank you for the QUIC session in tcpm

If one is not full-time on QUIC, it's very difficult to keep up with all the changes. But now we have a checkpoint to start from, I feel I will not be wasting people's time if I try to get involved - at least I only might say something un-QUIC occasionally, rather than nearly always. 
This has allowed people who understand how TCP cold be improved to help with QUIC, when working on QUIC isn't their day job.

Again, Thank you.

Bob

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 13 Nov 2018 19:57:26 +0000

That’s all great assuming that RTT increase was the reason for the prolonged silence. It should be possible to measure both incidences of spurious RTOs and measured RTT increases post spurious RTOs, and that should tell us which side we should err on.

For safety reasons, should we put in the text for reducing the cwnd in absence of pacing to keep consistency with TCP behavior, at least until data shows otherwise? I worry that most app developers without transport background will not have the expertise to realize the issues caused by burst after idle. The other option is to make pacing mandatory in the draft.

From: Jana Iyengar <[email-masked]>
Sent: Monday, November 12, 2018 11:13 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Yuchung Cheng <[email-masked]>; Bob Briscoe <[email-masked]>; Ian Swett <[email-masked]>; [email-masked]
Subject: Re: [tcpm] Thank you for the QUIC session in tcpm

On what to do with an unconfirmed RTO: So, my thought is that if you've hit an RTO, it's been a REALLY LONG time, because we've gone through TLP(s) by now and failed. (This might change if we decide to combine TLP and RTO, but that's for a later discussion.)  It's not fully clear to me what the right responses here are, but some thoughts follow.

1/ If this was an actual increase in the RTT, it would naturally roll into the SRTT and RTTVAR. We might even consider resetting the RTT estimator, but we already know that the EWMA estimator we love takes just a while to catch on, and I don't want to litigate changing the estimator fundamentally here.

2/ Linux fq uses min_rtt IIRC, which shouldn't change as a result of an increase in RTT. This *might* be a good time to kick the min_rtt estimator to pace more conservatively.

3/ I'm not convinced that it makes sense to change the cwnd. It seems clear that an RTT increase with the same BW should increase the cwnd if anything, so it seems reasonable to continue with the same cwnd at least. If the BW changed, hopefully that'll be reflected in a congestion signal, but in the absence of a congestion signal, I think we should keep the cwnd as is.

On Tue, Nov 13, 2018 at 12:35 AM Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Yuchung this might be worth experimenting with and getting more data on. Maybe there should be some reaction to an unconfirmed RTO even if it is not reduction all the way to LW? I like Jana's suggestion that the implementation SHOULD drop the cwnd even for unconfirmed RTO if not pacing.

-----Original Message-----
From: Yuchung Cheng <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 12, 2018 8:37 AM
To: Jana Iyengar <[email-masked]<mailto:[email-masked]>>
Cc: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; Bob Briscoe <[email-masked]<mailto:[email-masked]>>; Ian Swett <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]> Extensions <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tcpm] Thank you for the QUIC session in tcpm

> <pravb=[email-masked]<mailto:[email-masked]>> wrote:
>>
>> From: tcpm <[email-masked]<mailto:[email-masked]>> On Behalf Of Yuchung Cheng
>> To: Bob Briscoe <[email-masked]<mailto:[email-masked]>>
>> Cc: Ian Swett <[email-masked]<mailto:[email-masked]>>; tcpm IETF list <[email-masked]<mailto:[email-masked]>>
>> On Thu, Nov 8, 2018 at 3:14 AM, Bob Briscoe <[email-masked]<mailto:[email-masked]>> wrote:
>> >
>> > briscoe.net<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbriscoe.net&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092531035&sdata=rtr1fEZGcEqbSqZZIhG6HPdmfb5LzGGAFCMtoLNyTWo%3D&reserved=0>%2F&amp;data=02%7C01%7Cpravb%40microsoft.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40microsoft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092531035&sdata=SQQWIZy8UCRqj5w5I4dg3sgjvTkrygTXBci%2BOJETVE0%3D&reserved=0>%7C3c4d22866
>> > [email-masked]<mailto:[email-masked]>
>> > etf.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fetf.org&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092541035&sdata=Qrq0JPmfc48NwQ6SU9IMYEvA8x3N6OphFTxX2TpcKuk%3D&reserved=0>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micr
>> > oso
>> > ft.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092541035&sdata=Bd6AndbLA9pocmFsCvKAGMJw5nsMV4oiydwlT67wNHo%3D&reserved=0>%7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7
>> > cd0
>> [email-masked]<mailto:[email-masked]>
>>
>> ietf.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092551049&sdata=vPzeaG8Dy352omyuWVE%2BC6bQIV48o%2FgmiYzkoKP7T1s%3D&reserved=0>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> soft.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsoft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092551049&sdata=KqVybgAgN%2Bl1Ktquxl5hybWgQg05Oz9NQuUD2%2BPqQD4%3D&reserved=0>%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7
>> [email-masked]<mailto:[email-masked]>
>> ietf.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092561053&sdata=EQblZZj1sns102aGdnHO%2FvXPcRVg1cjEiUMYK%2BaJPSk%3D&reserved=0>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> soft.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsoft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092561053&sdata=XIL2TBtfeQfYNRnzCzxKQhh%2F9GdyWD9lDttd1w1waHM%3D&reserved=0>%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7

---

From: Eric Dumazet <[email-masked]>
Date: Wed, 14 Nov 2018 08:13:57 -0800

> 2/ Linux fq uses min_rtt IIRC, which shouldn't change as a result of an increase in RTT. This *might* be a good time to kick the min_rtt estimator to pace more conservatively.

linux fq does not use min_rtt, but the pacing rate set by TCP.

pacing rate for Cubic is using SRTT ( https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_input.c#L806 )

For BBR, pacing rate depends on various factors and phases, but SRTT is used, not min_rtt

https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_bbr.c#L231

> On Tue, Nov 13, 2018 at 12:35 AM Praveen Balasubramanian <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     I checked the latest draft and pacing is a RECOMMENDED. I am noticing that transport algorithms in Linux and in QUIC are evolving assuming pacing which is a function usually implemented and configured outside of the transport.. So the transport is potentially making an assumption which may result in safety issues. For example BBR congestion control may not work very well if pacing was disabled or misconfigured? I don’t know of any robust solution to this other than build pacing into the transport itself. Pacing is also very challenging for low latency flows because of lack of support for fine grained timers.
> 
>     From: Yuchung Cheng <[email-masked] <mailto:[email-masked]>>
>     To: Jana Iyengar <[email-masked] <mailto:[email-masked]>>
>     Cc: Praveen Balasubramanian <[email-masked] <mailto:[email-masked]>>; Bob Briscoe <[email-masked] <mailto:[email-masked]>>; Ian Swett <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]> Extensions <[email-masked] <mailto:[email-masked]>>
>     On Mon, Nov 12, 2018 at 4:13 AM, Jana Iyengar <[email-masked] <mailto:[email-masked]>> wrote:
>     > Praveen,
>     >
>     > <pravb=[email-masked] <mailto:[email-masked]>> wrote:
>     >>
>     >> From: tcpm <[email-masked] <mailto:[email-masked]>> On Behalf Of Yuchung Cheng
>     >> To: Bob Briscoe <[email-masked] <mailto:[email-masked]>>
>     >> Cc: Ian Swett <[email-masked] <mailto:[email-masked]>>; tcpm IETF list <[email-masked] <mailto:[email-masked]>>
>     >> On Thu, Nov 8, 2018 at 3:14 AM, Bob Briscoe <[email-masked] <mailto:[email-masked]>> wrote:
>     >> >
>     >> > briscoe.net <http://briscoe.net>%2F&amp;data=02%7C01%7Cpravb%40microsoft.com <http://40microsoft.com>%7C3c4d22866
>     >> > [email-masked] <mailto:[email-masked]>
>     >> > etf.org <http://etf.org>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micr
>     >> > oso
>     >> > ft.com <http://ft.com>%7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7
>     >> > cd0
>     >> [email-masked] <mailto:[email-masked]>
>     >>
>     >> ietf.org <http://ietf.org>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>     >> soft.com <http://soft.com>%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7
>     >> [email-masked] <mailto:[email-masked]>
>     >> ietf.org <http://ietf.org>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>     >> soft.com <http://soft.com>%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 15 Nov 2018 09:46:47 +0530

Praveen,

I think it's a good idea to require that cwnd be reduced in the absence of
pacing. I've opened #2007
<https://github.com/quicwg/base-drafts/issues/2007> to track this.

- jana

On Wed, Nov 14, 2018 at 1:27 AM Praveen Balasubramanian <[email-masked]>
wrote:

> That’s all great assuming that RTT increase was the reason for the
> prolonged silence. It should be possible to measure both incidences of
> spurious RTOs and measured RTT increases post spurious RTOs, and that
> should tell us which side we should err on.
>
>
>
> For safety reasons, should we put in the text for reducing the cwnd in
> absence of pacing to keep consistency with TCP behavior, at least until
> data shows otherwise? I worry that most app developers without transport
> background will not have the expertise to realize the issues caused by
> burst after idle. The other option is to make pacing mandatory in the
> draft.
>
>
>
> *From:* Jana Iyengar <[email-masked]>
> *Sent:* Monday, November 12, 2018 11:13 PM
> *To:* Praveen Balasubramanian <[email-masked]>
> *Cc:* Yuchung Cheng <[email-masked]>; Bob Briscoe <[email-masked]>;
> Ian Swett <[email-masked]>; [email-masked]
> *Subject:* Re: [tcpm] Thank you for the QUIC session in tcpm
>
>
>
> On what to do with an unconfirmed RTO: So, my thought is that if you've
> hit an RTO, it's been a REALLY LONG time, because we've gone through TLP(s)
> by now and failed. (This might change if we decide to combine TLP and RTO,
> but that's for a later discussion.)  It's not fully clear to me what the
> right responses here are, but some thoughts follow.
>
>
>
> 3/ I'm not convinced that it makes sense to change the cwnd. It seems
> clear that an RTT increase with the same BW should increase the cwnd if
> anything, so it seems reasonable to continue with the same cwnd at least.
> If the BW changed, hopefully that'll be reflected in a congestion signal,
> but in the absence of a congestion signal, I think we should keep the cwnd
> as is.
>
>
>
>
>
> >> > briscoe.net
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbriscoe.net&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092531035&sdata=rtr1fEZGcEqbSqZZIhG6HPdmfb5LzGGAFCMtoLNyTWo%3D&reserved=0>
> %2F&amp;data=02%7C01%7Cpravb%40microsoft.com
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40microsoft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092531035&sdata=SQQWIZy8UCRqj5w5I4dg3sgjvTkrygTXBci%2BOJETVE0%3D&reserved=0>
> %7C3c4d22866
> >> > etf.org
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fetf.org&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092541035&sdata=Qrq0JPmfc48NwQ6SU9IMYEvA8x3N6OphFTxX2TpcKuk%3D&reserved=0>
> %2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micr
> >> > oso
> >> > ft.com
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092541035&sdata=Bd6AndbLA9pocmFsCvKAGMJw5nsMV4oiydwlT67wNHo%3D&reserved=0>
> %7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7
> >> > cd0
> >> ietf.org
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092551049&sdata=vPzeaG8Dy352omyuWVE%2BC6bQIV48o%2FgmiYzkoKP7T1s%3D&reserved=0>
> %2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
> >> soft.com
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsoft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092551049&sdata=KqVybgAgN%2Bl1Ktquxl5hybWgQg05Oz9NQuUD2%2BPqQD4%3D&reserved=0>
> %7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7
> >> ietf.org
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092561053&sdata=EQblZZj1sns102aGdnHO%2FvXPcRVg1cjEiUMYK%2BaJPSk%3D&reserved=0>
> %2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
> >> soft.com
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsoft.com&data=02%7C01%7Cpravb%40microsoft.com%7C7c540bc234b8405b454008d649378123%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636776900092561053&sdata=XIL2TBtfeQfYNRnzCzxKQhh%2F9GdyWD9lDttd1w1waHM%3D&reserved=0>
> %7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 15 Nov 2018 09:47:47 +0530

Thanks for the clarification, Eric. I'm a bit surprised to see srtt be used
by both Cubic and BBR, but that discussion's not for this thread.

On Wed, Nov 14, 2018 at 9:44 PM Eric Dumazet <[email-masked]> wrote:

> > 1/ If this was an actual increase in the RTT, it would naturally roll
> into the SRTT and RTTVAR. We might even consider resetting the RTT
> estimator, but we already know that the EWMA estimator we love takes just a
> while to catch on, and I don't want to litigate changing the estimator
> fundamentally here.
> >
> pacing rate for Cubic is using SRTT (
> https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_input.c#L806 )
>
> [email-masked] <mailto:[email-masked]>> wrote:
> >
> >     Yuchung this might be worth experimenting with and getting more data
> on. Maybe there should be some reaction to an unconfirmed RTO even if it is
> not reduction all the way to LW? I like Jana's suggestion that the
> >     I checked the latest draft and pacing is a RECOMMENDED. I am
> noticing that transport algorithms in Linux and in QUIC are evolving
> assuming pacing which is a function usually implemented and configured
> outside of the transport.. So the transport is potentially making an
> assumption which may result in safety issues. For example BBR congestion
> control may not work very well if pacing was disabled or misconfigured? I
> don’t know of any robust solution to this other than build pacing into the
> transport itself. Pacing is also very challenging for low latency flows
> because of lack of support for fine grained timers.
> >
> >     Cc: Praveen Balasubramanian <[email-masked] <mailto:
> [email-masked]>>; Bob Briscoe <[email-masked] <mailto:
> [email-masked]>>; Ian Swett <[email-masked] <mailto:
> [email-masked]>>; [email-masked] <mailto:[email-masked]> Extensions <
> [email-masked] <mailto:[email-masked]>>
> >     >> Do you see any safety concerns with delayed reduction of cwnd in
> case
> >     >> where the RTO is not spurious?
> >     >>
> >     >> To: Bob Briscoe <[email-masked] <mailto:[email-masked]
> >>
> >     >> Cc: Ian Swett <[email-masked] <mailto:[email-masked]>>;
> tcpm IETF list <[email-masked] <mailto:[email-masked]>>
> >     >> > I just wanted to thank Jana for explaining QUIC loss recovery
> to us
> >     >> > (and QUIC CC as far as it goes).
> pkt-ids.
> >     >> > briscoe.net <http://briscoe.net>%2F&amp;data=02%7C01%7Cpravb%
> 40microsoft.com <http://40microsoft.com>%7C3c4d22866
> >     >> >
> >     >> > etf.org <http://etf.org
> >     >> > ft.com <http://ft.com
> >     >> ietf.org <http://ietf.org
> >     >> soft.com <http://soft.com
> >     >> ietf.org <http://ietf.org
> >     >> soft.com <http://soft.com

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 15 Nov 2018 12:15:38 -0500

On Wed, Nov 14, 2018 at 11:18 PM Jana Iyengar <[email-masked]> wrote:

> Thanks for the clarification, Eric. I'm a bit surprised to see srtt be
> used by both Cubic and BBR, but that discussion's not for this thread.
>

Just to add context (sorry to dilute the thread!), the spot in Linux TCP
BBR that Eric pointed to that is using srtt is the initialization code
(bbr_init_pacing_rate_from_rtt()). The code is using srtt here to compute a
pacing rate (via pacing_rate = gain * cwd / srtt) largely because Linux TCP
can hot-swap congestion control modules. Because of that possibility, the
cwnd and srtt that BBR inherits at initialization may have been from a
buffer-bloated CUBIC/Reno connection, where srtt may have bloated in
proportion to cwnd, so that cwd / srtt is generally a better approximation
to the available bandwidth than cwnd / min_rtt.  As you would imagine,
after that initialization, Linux TCP BBR does not use srtt (or min_rtt) to
update the pacing rate; it uses a function of the estimated bandwidth.

best,
neal

>> >     Yuchung this might be worth experimenting with and getting more
>> data on. Maybe there should be some reaction to an unconfirmed RTO even if
>> it is not reduction all the way to LW? I like Jana's suggestion that the
>> >     I agree w/ Jana as well - the most generic way to treat burst due
>> to prior_inflight << cwnd is pacing. With QUIC's approach, when RTO fires,
>> >     Yes the worst case is more aggressive, and it can be too much for
>> DC incast case if pacing isn't supported - one idea is to selective enable

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 21 Nov 2018 16:31:01 -0800

Thanks for the clarification, Neal!

On Thu, Nov 15, 2018 at 9:15 AM Neal Cardwell <[email-masked]> wrote:

>>> > On what to do with an unconfirmed RTO: So, my thought is that if
>>> you've hit an RTO, it's been a REALLY LONG time, because we've gone through
>>> TLP(s) by now and failed. (This might change if we decide to combine TLP
>>> and RTO, but that's for a later discussion.)  It's not fully clear to me
>>> what the right responses here are, but some thoughts follow.
>>> >
>>> > 2/ Linux fq uses min_rtt IIRC, which shouldn't change as a result of
>>> an increase in RTT. This *might* be a good time to kick the min_rtt
>>> estimator to pace more conservatively.
>>>
>>> >     >> Do you see any safety concerns with delayed reduction of cwnd
>>> in case
>>> >     >> On Thu, Nov 8, 2018 at 3:14 AM, Bob Briscoe <
>>> [email-masked] <mailto:[email-masked]>> wrote:
>>> >     >> >