Subject: Re: [tcpm] Linux doesn’t implement RFC3465

From: "Mark Allman" <[email-masked]>
Date: Mon, 02 Aug 2021 14:03:03 -0400

>  “This document RECOMMENDS using mechanisms like Pacing to control
>  how many bytes are sent to the network at a point of time. But if
>  it is not possible to implement pacing, an implementation MAY
>  implicitly pace their traffic by applying a limit L to the
>  increase in congestion window per ACK during slow start. In
>  modern stacks, acknowledgments are aggregated for various reason,
>  CPU optimization, reducing network load etc. Hence it is common
>  for a sender to receive an aggregated ACK that acknowledges more
>  than 2 segments. For example, a stack that implements GRO could
>  aggregate packets up to 64Kbytes or ~44 segments before passing
>  on to the TCP layer and this would result in a single ACK to be
>  generated by the TCP stack. Given that an initial window of 10
>  packets in current deployments has been working fine, the draft
>  makes a recommendation to set L=10 during slow start. This would
>  mean that with every ACK, we are probing for a new capacity by
>  sending 10 packets in addition to the previously discovered
>  capacity. Implementations MAY choose to set a lower limit if they
>  believe an increase of 10 is too aggressive."
>
> Does this sound like what we would like to say?

Not really, IMO.  I think a few things here ...

  - I agree that if pacing is in play that we don't need to worry
    about an L.

  - I think the above L=10 reasoning is at best pretty weak.  Just
    because IW=10 works OK once / connection does not mean
    continually sending "10 more" will work out OK.  It may.  It may
    not.  But, the above sort of coupling between IW-10 and L=10
    seems highly tenuous without any sort of data.

  - The real issue with picking a number is that it is so hard to
    reason about because the behavior All Depends.  E.g., consider
    something like IW=10.  We know that will allow 10 or fewer
    segments to be pumped into the network when the connection
    starts.  That's pretty easy to reason about / understand.  But,
    with L=10 we might have bursts anywhere from 2 packets to
    cwnd+10 packets on every ACK---depending on how the ACKs are
    stretched.  And, cwnd isn't a constant.  So, if the idea is to
    somehow limit bursting then sometimes we're limiting to X and
    others to Y and still others to Z.  It's an inconsistent mess.
    Making L something arbitrary without evidence seems like a bad
    path to me.

  - Of course, by making L=10 a MAY we're effectively saying "no L,
    anywhere" anyways.  If we're going to define an L it should be a
    SHOULD unless there is pacing.

  - Somewhat related to the above, it isn't clear what "10 more" is
    more than in qualitative terms, as well.  Say an ACK rolls in
    that covers 10 packets.  How were those packets sent?  Were they
    sent a back-to-back burst so "10 more" is in fact a back-to-back
    burst that is 10 more than previously (or, 2x)?  Or, did we send
    those 10 packets in 5 little bursts of 2 packets each so that
    "10 more" is actually 18 more than the previous burst size---a
    10x increase?  Of course, it could be BOTH for the same ACK!
    I.e., the segments were sent 2 at a time and aggregated
    somewhere in the middle.  This lack of clarity again makes the
    choice of L feel pretty arbitrary.

I agree we can elide L if pacing is in place.  But, twiddling with L
by feel is crude and will produce an inconsistent approach to bursts
that doesn't seem to me to be particularly helpful because we don't
really grok the implications.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 2 Aug 2021 15:01:18 -0700

The choice of L=10 is fairly arbitrary -- just like IW10. But what's the
show-stopper we should be concerned about, in the big picture of having an
interoperable and performant Internet.

The fact is that Linux CC has long moved to infinite L since 2031, with or
without pacing, for slow-start and congestion avoidance. Windows is using 8
for hystart++. We can go through 100 rounds of what's a more scientific L
with more <big-CDN> experiments. In the end a bigger L causes higher burst
and risks more losses and queues. That number and stats would continue to
change

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f9843a751d0a2057f9f3d313886e7e5e6ebaac9
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9cd981dcf174d26805a032aefa791436da709bee
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c22bdca94782f05b9337d8548bde51b2f38ef17f
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=814d488c61260521b1b3cc97063700a5a6667c8f
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e73ebb0881ea5534ce606c1d71b4ac44db5c6930

Many people have 100Mbps broadband, and stretched ACKs reaching a degree of
hundreds MTUs due to various bunching to improve performance. ABC L=2 is
really an inappropriate outdated limit.

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 2 Aug 2021 15:02:52 -0700

The fact is that Linux CC has long moved to infinite L since 2013*

---

From: "Mark Allman" <[email-masked]>
Date: Mon, 02 Aug 2021 18:36:59 -0400

> The fact is that Linux CC has long moved to infinite L since 2031,

So, if our experience is with L=\infinity and it is demonstrably OK
why don't we say *THAT* instead of "make L=5 or L=10"?  I would
submit that it makes more sense to leverage experience than it does
to make things up.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 2 Aug 2021 15:45:37 -0700

to make things up.
+1

---

From: David Lang <[email-masked]>
Date: Mon, 2 Aug 2021 15:49:21 -0700 (PDT)

I believe that Linux also defaults to fq_codel to manage congestion at it's 
interface (doesn't help if the bottleneck is a router, unless it's also running 
Linux)

I don't rememberhen that went it, but it's been several years, may actually go 
back to 2013 as well.

David  Lang

>> submit that it makes more sense to leverage experience than it does

_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 2 Aug 2021 18:56:39 -0400

>> submit that it makes more sense to leverage experience than it does

Yes, I agree that would be a great approach to take.

neal

---

From: Vidhi Goel <[email-masked]>
Date: Mon, 02 Aug 2021 16:02:01 -0700

> On Mon, Aug 2, 2021 at 3:37 PM Mark Allman <[email-masked] <mailto:[email-masked]>> wrote:
> 

So, we are saying it is fine to ignore L completely and simply increase cwnd by bytes_acked during slow start? And if this causes large bursts to be sent out (when an implementation doesn’t do pacing), that is fine? 

-
Vidhi

> On Aug 2, 2021, at 3:56 PM, Neal Cardwell <[email-masked]> wrote:
> 
> 
> 
> On Mon, Aug 2, 2021 at 6:46 PM Yuchung Cheng <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> On Mon, Aug 2, 2021 at 3:37 PM Mark Allman <[email-masked] <mailto:[email-masked]>> wrote:
>

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 2 Aug 2021 19:18:21 -0400

On Mon, Aug 2, 2021 at 7:02 PM Vidhi Goel <[email-masked]> wrote:

> So, we are saying it is fine to ignore L completely and simply increase
> cwnd by bytes_acked during slow start? And if this causes large bursts to
> be sent out (when an implementation doesn’t do pacing), that is fine?
>

Yes, I think that is the proposal on the table, and it sounds good to me.

A rationale would be:

(1) Implementations SHOULD pace (RFC 7661).

(2) Implementations that don't pace will generally be causing large bursts
for many different reasons anyway (data and/or ACK aggregation in the
network or end hosts), restart from idle,...) so having a constant L does
not provide enough protection from bursts to justify the cost in reduced
performance (in the form of slower slow-start). In support of this,
experience with this as the default behavior in Linux TCP over the
2013-2021 period suggests this works well enough in practice.

neal

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 2 Aug 2021 16:25:14 -0700

> 2013-2021 period suggests this works well enough in practice.
One simple example is:

Let's consider a sender receiving
1. one stretch ACK acking 100 packets due to ACK decimation
vs
2. 50 ACKs each acking 2 MSS within 0.1ms due to ACK compression

(2) still forms a sizable burst even with ACK clocking due to timing
compression.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 3 Aug 2021 00:46:15 +0000

In experiments a few years ago on DC networks, values over L=8 resulted in a noticeable increase in packet drops and retransmissions (without pacing). Windows TCP has been using L=8 for many years now. If we do want to specify a fallback L value for implementations that cannot pace, my suggestion would be to use the value 8.

Neal, are there cases where Linux is or can be deployed with infinite L and no pacing?

From: tcpm <[email-masked]> On Behalf Of Neal Cardwell
Sent: Monday, August 2, 2021 4:18 PM
To: Vidhi Goel <[email-masked]>
Cc: Extensions <[email-masked]>; Mark Allman <[email-masked]>
Subject: [EXTERNAL] Re: [tcpm] Linux doesn’t implement RFC3465

On Mon, Aug 2, 2021 at 7:02 PM Vidhi Goel <[email-masked]<mailto:[email-masked]>> wrote:

A rationale would be:

(2) Implementations that don't pace will generally be causing large bursts for many different reasons anyway (data and/or ACK aggregation in the network or end hosts), restart from idle,...) so having a constant L does not provide enough protection from bursts to justify the cost in reduced performance (in the form of slower slow-start). In support of this, experience with this as the default behavior in Linux TCP over the  2013-2021 period suggests this works well enough in practice.

neal

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 2 Aug 2021 20:52:42 -0400

On Mon, Aug 2, 2021 at 8:46 PM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> In experiments a few years ago on DC networks, values over L=8 resulted in
> a noticeable increase in packet drops and retransmissions (without pacing).
> Windows TCP has been using L=8 for many years now. If we do want to specify
> a fallback L value for implementations that cannot pace, my suggestion
> would be to use the value 8.
>
>
>

Yes, "infinite L and no pacing" is the default behavior for Linux TCP,
starting in 2013 for slow-start and then starting in 2015 for congestion
avoidance.

Yuchung pasted the URLs for the exact Linux commits above, which are:

But I understand that not everyone is in a position to read GPL-licensed
code. :-)

best regards,
neal

>
>
> *From:* tcpm <[email-masked]> *On Behalf Of * Neal Cardwell
> *Sent:* Monday, August 2, 2021 4:18 PM
> *To:* Vidhi Goel <[email-masked]>
> *Cc:* Extensions <[email-masked]>; Mark Allman <[email-masked]>
> *Subject:* [EXTERNAL] Re: [tcpm] Linux doesn’t implement RFC3465
>
>
>
>
>
>
>

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 2 Aug 2021 18:12:16 -0700

On Mon, Aug 2, 2021 at 5:53 PM Neal Cardwell <ncardwell=
[email-masked]> wrote:

>> In experiments a few years ago on DC networks, values over L=8 resulted
>> in a noticeable increase in packet drops and retransmissions (without
>> pacing). Windows TCP has been using L=8 for many years now. If we do want
>> to specify a fallback L value for implementations that cannot pace, my
>> suggestion would be to use the value 8.
>>
>>
>>
To be more clear: both fq_pacing and TCP pacing have been disabled by
default in Linux upstream. We do not know how much Linux senders enable
them today besides the Google servers.

Regarding L = 8, to avoid another round of why or why not. We could say
inf-L causes line-rate burst up to the stretched ACK degree so put a
comfortable L if you prefer, then mention implementation practice like
yours. At the end of the day it's ad-hoc (or "art") and subject to change.
It might be sensible to cap at cwnd to disincentivize receivers /
middle-boxes bunching up 10 rounds of ACKs.

>> (2) Implementations that don't pace will generally be causing large
>> bursts for many different reasons anyway (data and/or ACK aggregation in
>> the network or end hosts), restart from idle,...) so having a constant L
>> does not provide enough protection from bursts to justify the cost in
>> reduced performance (in the form of slower slow-start). In support of this,

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 3 Aug 2021 09:42:00 -0700

> middle-boxes bunching up 10 rounds of ACKs.
Sorry please ignore my previous message about the cwnd cap. It is
completely unnecessary -- since with ack-clocking and appropriate counting,
a correct sender would never release more than a cwnd-worth of data. I was
imagining the multiple application-limited burst could let the receiver
keep holding up ACKs, but that can never exceed a cwnd worth of data.

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 6 Aug 2021 12:53:51 -0700

Hi WG

I have been wondering if we (= IETF) should just update RFC5681 directly,
instead of another RFC3465-bis with experimental status.

Appropriate byte counting is essential but the RFC5681 of L=1 is
detrimental. There are far more people who read RFC5681 to implement the
new stack instead of RFC3465. So we should fold the experimental RFC3465
updates into RFC5681 directly, and obsolete RFC3465.

This is orthogonal to the final value of L :-)

---

From: Vidhi Goel <[email-masked]>
Date: Fri, 06 Aug 2021 12:59:38 -0700

I agree with Yuchung’s suggestion for all the reasons he provided. And its better to have it at one place.

Vidhi

> On Aug 6, 2021, at 12:53 PM, Yuchung Cheng <ycheng=[email-masked]> wrote:
> 
> Hi WG
> 
> I have been wondering if we (= IETF) should just update RFC5681 directly, instead of another RFC3465-bis with experimental status.
> 
> Appropriate byte counting is essential but the RFC5681 of L=1 is detrimental. There are far more people who read RFC5681 to implement the new stack instead of RFC3465. So we should fold the experimental RFC3465 updates into RFC5681 directly, and obsolete RFC3465.
> 
> On Tue, Aug 3, 2021 at 9:42 AM Yuchung Cheng <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> On Mon, Aug 2, 2021 at 6:12 PM Yuchung Cheng <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> On Mon, Aug 2, 2021 at 5:53 PM Neal Cardwell <ncardwell=[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> On Mon, Aug 2, 2021 at 8:46 PM Praveen Balasubramanian <pravb=[email-masked] <mailto:[email-masked]>> wrote:
> Yes, "infinite L and no pacing" is the default behavior for Linux TCP, starting in 2013 for slow-start and then starting in 2015 for congestion avoidance.
> To be more clear: both fq_pacing and TCP pacing have been disabled by default in Linux upstream. We do not know how much Linux senders enable them today besides the Google servers.
> 
> Regarding L = 8, to avoid another round of why or why not. We could say inf-L causes line-rate burst up to the stretched ACK degree so put a comfortable L if you prefer, then mention implementation practice like yours. At the end of the day it's ad-hoc (or "art") and subject to change. It might be sensible to cap at cwnd to disincentivize receivers / middle-boxes bunching up 10 rounds of ACKs.
> Sorry please ignore my previous message about the cwnd cap. It is completely unnecessary -- since with ack-clocking and appropriate counting, a correct sender would never release more than a cwnd-worth of data. I was imagining the multiple application-limited burst could let the receiver keep holding up ACKs, but that can never exceed a cwnd worth of data.
>  
> 
>  
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f9843a751d0a2057f9f3d313886e7e5e6ebaac9 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f9843a751d0a2057f9f3d313886e7e5e6ebaac9>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9cd981dcf174d26805a032aefa791436da709bee <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9cd981dcf174d26805a032aefa791436da709bee>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c22bdca94782f05b9337d8548bde51b2f38ef17f <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c22bdca94782f05b9337d8548bde51b2f38ef17f>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=814d488c61260521b1b3cc97063700a5a6667c8f <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=814d488c61260521b1b3cc97063700a5a6667c8f>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e73ebb0881ea5534ce606c1d71b4ac44db5c6930 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e73ebb0881ea5534ce606c1d71b4ac44db5c6930>
> 
> But I understand that not everyone is in a position to read GPL-licensed code. :-)
> 
> best regards,
> neal
> 
>  
>  
> 
> From: tcpm <[email-masked] <mailto:[email-masked]>> On Behalf Of Neal Cardwell
> To: Vidhi Goel <[email-masked] <mailto:[email-masked]>>
> Cc: Extensions <[email-masked] <mailto:[email-masked]>>; Mark Allman <[email-masked] <mailto:[email-masked]>>
> On Mon, Aug 2, 2021 at 7:02 PM Vidhi Goel <[email-masked] <mailto:[email-masked]>> wrote:
> 
>  
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>

---

From: Neal Cardwell <[email-masked]>
Date: Sat, 7 Aug 2021 10:57:10 -0400

I also agree with Yuchung’s suggestion, for all of the reasons he provided.

best,
neal

On Fri, Aug 6, 2021 at 3:59 PM Vidhi Goel <vidhi_goel=
[email-masked]> wrote:

> On Aug 6, 2021, at 12:53 PM, Yuchung Cheng <
> ycheng=[email-masked]> wrote:
>
> Hi WG
>
>>>>> In experiments a few years ago on DC networks, values over L=8
>>>>> resulted in a noticeable increase in packet drops and retransmissions
>>>>> (without pacing). Windows TCP has been using L=8 for many years now. If we
>>>>> do want to specify a fallback L value for implementations that cannot pace,
>>>>> my suggestion would be to use the value 8.
>>>>>
>>>>>
>>>>>
>>>>> So, we are saying it is fine to ignore L completely and simply
>>>>> increase cwnd by bytes_acked during slow start? And if this causes large
>>>>> bursts to be sent out (when an implementation doesn’t do pacing), that is
>>>>> fine?
>>>>>
>>>>>
>>>>>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 9 Aug 2021 01:40:45 -0700

I don't have a strong opinion on this yet, But, if we *could* move in this
direction, it might be good to think about the IW explanation in RFC5681 as
well?
if we do this, we might not need to discuss promoting RFC6928.
--
Yoshi

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 9 Aug 2021 09:38:10 -0700

Sorry I don't understand your suggestion. Is that related to ABC? could you
explain more

On Mon, Aug 9, 2021 at 1:41 AM Yoshifumi Nishida <[email-masked]> wrote:

>>> I have been wondering if we (= IETF) should just update RFC5681
>>> directly, instead of another RFC3465-bis with experimental status.
>>>
>>>>>> Yes, "infinite L and no pacing" is the default behavior for Linux
>>>>>> TCP, starting in 2013 for slow-start and then starting in 2015 for
>>>>>> congestion avoidance.
>>>>>>
>>>>> Regarding L = 8, to avoid another round of why or why not. We could
>>>>> say inf-L causes line-rate burst up to the stretched ACK degree so put a

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 9 Aug 2021 12:53:20 -0400

I think Yoshifumi is suggesting that if the WG re-spins  RFC5681 then in
addition to folding in discussion of ABC/RFC3465 the RFC5681bis could also
include the IW10 content in RFC6928. That could help save time in avoiding
promoting RFC6928 from experimental to proposed standard.

That sounds reasonable to me as well. Although the initial window is a
somewhat independent issue that might evolve independently from the
congestion algorithm itself, so I can imagine advantages to keeping it
separate.

neal

On Mon, Aug 9, 2021 at 12:39 PM Yuchung Cheng <ycheng=
[email-masked]> wrote:

> Sorry I don't understand your suggestion. Is that related to ABC? could
> you explain more
>
>> I don't have a strong opinion on this yet, But, if we *could* move in
>> this direction, it might be good to think about the IW explanation in
>> RFC5681 as well?

---

From: "Mark Allman" <[email-masked]>
Date: Mon, 09 Aug 2021 13:05:35 -0400

> (2) Implementations that don't pace will generally be causing
> large bursts for many different reasons anyway (data and/or ACK
> aggregation in the network or end hosts), restart from idle,...)
> so having a constant L does not provide enough protection from
> bursts to justify the cost in reduced performance (in the form of
> slower slow-start). In support of this, experience with this as
> the default behavior in Linux TCP over the 2013-2021 period
> suggests this works well enough in practice.

I think at some point someone should put some meat on the bones of
"suggests this works well enough in practice".

It isn't enough for linux to have implemented this.  Or, even had it
turned on.  E.g., if L=\infinity yet receivers ACK every other
packet then there is an effective L of 2.  We see lots of people
saying this scenario isn't the prevalent scenario these days.  If
that's true then it should be easy to provide a summary of
experience with L=\infinity.

This would all just seem better with some concrete experience rather
than the quite hand wavy statements we've seen so far.

allman

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 9 Aug 2021 10:07:13 -0700

Yes, that was my intention. Sorry for being unclear.
I thought updating RFC5681 could be impactful. So, if it would happen, I
would like to make sure we won't have another update for a long time.
--
Yoshi

On Mon, Aug 9, 2021 at 9:53 AM Neal Cardwell <[email-masked]> wrote:

>>>>>>> To be more clear: both fq_pacing and TCP pacing have been disabled
>>>>>>> by default in Linux upstream. We do not know how much Linux senders enable
>>>>>>>>> large bursts for many different reasons anyway (data and/or ACK aggregation
>>>>>>>>> in the network or end hosts), restart from idle,...) so having a constant L

---

From: Vidhi Goel <[email-masked]>
Date: Mon, 09 Aug 2021 13:32:52 -0700

I also like Yoshi’s suggestion to fold the initial window into RFC5681-bis.

> That sounds reasonable to me as well. Although the initial window is a somewhat independent issue that might evolve independently from the congestion algorithm itself, so I can imagine advantages to keeping it separate.

I think Initial Window is not independent from congestion control module, because lets say in future with higher BDPs we decide to set IW=1000, then the congestion control algorithm should be able to immediately detect congestion, if any, even at the start of the connection and for such a high initial value, probably requires other ways to detect congestion besides packet loss.
Yes, it is separate from how the algorithm works itself, but still deciding the initial value depends a lot on the congestion control algorithm.

Thanks,
Vidhi

> On Aug 9, 2021, at 10:07 AM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> I thought updating RFC5681 could be impactful. So, if it would happen, I would like to make sure we won't have another update for a long time.
> --
> Yoshi
> 
> On Mon, Aug 9, 2021 at 9:53 AM Neal Cardwell <[email-masked] <mailto:[email-masked]>> wrote:
> I think Yoshifumi is suggesting that if the WG re-spins  RFC5681 then in addition to folding in discussion of ABC/RFC3465 the RFC5681bis could also include the IW10 content in RFC6928. That could help save time in avoiding promoting RFC6928 from experimental to proposed standard.
> 
> That sounds reasonable to me as well. Although the initial window is a somewhat independent issue that might evolve independently from the congestion algorithm itself, so I can imagine advantages to keeping it separate.
> 
> neal
> 
> 
> On Mon, Aug 9, 2021 at 12:39 PM Yuchung Cheng <ycheng=[email-masked] <mailto:[email-masked]>> wrote:
> Sorry I don't understand your suggestion. Is that related to ABC? could you explain more
> 
> On Mon, Aug 9, 2021 at 1:41 AM Yoshifumi Nishida <[email-masked] <mailto:[email-masked]>> wrote:
> I don't have a strong opinion on this yet, But, if we *could* move in this direction, it might be good to think about the IW explanation in RFC5681 as well?
> On Sat, Aug 7, 2021 at 7:57 AM Neal Cardwell <ncardwell=[email-masked] <mailto:[email-masked]>> wrote:
> On Fri, Aug 6, 2021 at 3:59 PM Vidhi Goel <vidhi_goel=[email-masked] <mailto:[email-masked]>> wrote:
>> On Aug 6, 2021, at 12:53 PM, Yuchung Cheng <ycheng=[email-masked] <mailto:ycheng=[email-masked]>> wrote:
>> 
>> Hi WG
>>

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 9 Aug 2021 13:47:29 -0700

+1 to Yoshifumi's proposal as well.

On Mon, Aug 9, 2021 at 1:33 PM Vidhi Goel <vidhi_goel=
[email-masked]> wrote:

> I think Initial Window is not independent from congestion control module,
> because lets say in future with higher BDPs we decide to set IW=1000, then
> the congestion control algorithm should be able to immediately detect
> congestion, if any, even at the start of the connection and for such a high
> initial value, probably requires other ways to detect congestion besides
> packet loss.
> Yes, it is separate from how the algorithm works itself, but still
> deciding the initial value depends a lot on the congestion control
> algorithm.
>
> Thanks,
> Vidhi
>

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 9 Aug 2021 13:52:40 -0700

> "suggests this works well enough in practice".
Good point. We could add anecdotal words on the degree of stretched ACK.

Maybe we can also cite priori discussions on ACK compression / decimation a
couple years ago in tcpm / tsvwg. Are there research papers on stretch ACK
degrees too?

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 10 Aug 2021 15:55:29 +0000

Sigcomm paper from 2015:

http://www.sigcomm.org/sites/default/files/ccr/papers/2015/July/0000000-0000002.pdf

They’ve observed 11-22% of connections with stretched ACKs, ack’ing mostly up to 6 packets….

From: tcpm <[email-masked]> on behalf of Yuchung Cheng <ycheng=[email-masked]>
Date: Monday, 9. August 2021 at 22:53
To: Mark Allman <[email-masked]>
Cc: Extensions <[email-masked]>
Subject: Re: [tcpm] Linux doesn’t implement RFC3465

Maybe we can also cite priori discussions on ACK compression / decimation a couple years ago in tcpm / tsvwg. Are there research papers on stretch ACK degrees too?

allman

---

From: Martin Duke <[email-masked]>
Date: Thu, 12 Aug 2021 11:10:42 -0700

With AD hat on,

If I understand the current document state correctly, the current ABC
guidance we have is

RFC3465 (Experimental): L=2
RFC5681 (PS): L=1

There are a few PS drafts floating around that use byte counting in some
form and would benefit from a standards-track reference in this case.

1. The most comprehensive solution would be a 3465bis PS doc that obsoletes
3465 and updates 5681 with L= Infinity, 8, or whatever number the group
decides on. If we agree this is the right thing to do and someone has the
energy to drop 3465bis-00, then let's do so to have a basis of discussion.

2. If there is no such energy, or there are contentious issues that will
draw out the process to get this to RFC, than I can initiate a review of
upgrading 3465 to PS directly. I think we all agree this would be an
improvement on the status quo, but isn't worth doing if 3465bis is going to
happen quickly.

3. There might be interest in a more generalized 5681bis, but my instinct
is that this is a bigger job that will delay the most urgently needed
update -- which is not to say that we shouldn't do that document too.

4. However, I don't think it would be too much of a stretch to rope in an
IW update (6928bis) in 3465bis, if it doesn't add too much time. It all
fits in with the general theme of the maximum allowable burst on the
internet in the absence of any sort of pacing (ack-driven or
stack-enforced).

The most useful thing right now would be for people to either (1) disagree
with any of my 4 points above or (2) or volunteer to write 3465bis-00.

Martin

On Tue, Aug 10, 2021 at 8:55 AM Mirja Kuehlewind <mirja.kuehlewind=
[email-masked]> wrote:

> *From: *tcpm <[email-masked]> on behalf of Yuchung Cheng <ycheng=
> [email-masked]>
> *Date: *Monday, 9. August 2021 at 22:53
> *To: *Mark Allman <[email-masked]>
> *Cc: *Extensions <[email-masked]>
> *Subject: *Re: [tcpm] Linux doesn’t implement RFC3465
>
>
>
>
>
>
>
> Maybe we can also cite priori discussions on ACK compression / decimation
> a couple years ago in tcpm / tsvwg. Are there research papers on stretch
> ACK degrees too?
>
>
>
>

---

From: Vidhi Goel <[email-masked]>
Date: Thu, 12 Aug 2021 11:34:28 -0700

Thanks Martin for mentioning the different available options.

> If I understand the current document state correctly, the current ABC guidance we have is
> 

I want to note that the guidance in 3465 is a MUST while in 5681 is a RECOMMEND.

> 1. The most comprehensive solution would be a 3465bis PS doc that obsoletes 3465 and updates 5681 with L= Infinity, 8, or whatever number the group decides on. If we agree this is the right thing to do and someone has the energy to drop 3465bis-00, then let's do so to have a basis of discussion.

I created a GitHub repo https://github.com/goelvidhi/rfc3465-bis <https://github.com/goelvidhi/rfc3465-bis> with the current text of 3465. Neal and Yuchung have mentioned that they would be willing to contribute. (Neal, Yuchung, please correct me if I misunderstood).

> 2. If there is no such energy, or there are contentious issues that will draw out the process to get this to RFC, than I can initiate a review of upgrading 3465 to PS directly. I think we all agree this would be an improvement on the status quo, but isn't worth doing if 3465bis is going to happen quickly.

I don’t think 3465 can be upgraded to PS as-is.

> 3. There might be interest in a more generalized 5681bis, but my instinct is that this is a bigger job that will delay the most urgently needed update -- which is not to say that we shouldn't do that document too.

Yeah, it could delay the simpler and important update to 3465. I am ok with going either way - 5681-bis or 3465-bis whichever could likely make faster progress.

> 4. However, I don't think it would be too much of a stretch to rope in an IW update (6928bis) in 3465bis, if it doesn't add too much time. It all fits in with the general theme of the maximum allowable burst on the internet in the absence of any sort of pacing (ack-driven or stack-enforced).

Personally, I like the structure of 6928 more than 3465. Is it possible to rope in ABC (3465) in a 6928bis and change the title - Would like to hear more from the authors of both the RFCs.

Thanks,
Vidhi

> On Aug 12, 2021, at 11:10 AM, Martin Duke <[email-masked]> wrote:
> 
> With AD hat on,
> 
> If I understand the current document state correctly, the current ABC guidance we have is
> 
> There are a few PS drafts floating around that use byte counting in some form and would benefit from a standards-track reference in this case.
> 
> 1. The most comprehensive solution would be a 3465bis PS doc that obsoletes 3465 and updates 5681 with L= Infinity, 8, or whatever number the group decides on. If we agree this is the right thing to do and someone has the energy to drop 3465bis-00, then let's do so to have a basis of discussion.
> 
> 2. If there is no such energy, or there are contentious issues that will draw out the process to get this to RFC, than I can initiate a review of upgrading 3465 to PS directly. I think we all agree this would be an improvement on the status quo, but isn't worth doing if 3465bis is going to happen quickly.
> 
> 3. There might be interest in a more generalized 5681bis, but my instinct is that this is a bigger job that will delay the most urgently needed update -- which is not to say that we shouldn't do that document too.
> 
> 4. However, I don't think it would be too much of a stretch to rope in an IW update (6928bis) in 3465bis, if it doesn't add too much time. It all fits in with the general theme of the maximum allowable burst on the internet in the absence of any sort of pacing (ack-driven or stack-enforced).
> 
> The most useful thing right now would be for people to either (1) disagree with any of my 4 points above or (2) or volunteer to write 3465bis-00.
> 
> Martin
> 
> On Tue, Aug 10, 2021 at 8:55 AM Mirja Kuehlewind <mirja.kuehlewind=[email-masked] <mailto:[email-masked]>> wrote:
> Sigcomm paper from 2015:
> 
>  
> 
> http://www.sigcomm.org/sites/default/files/ccr/papers/2015/July/0000000-0000002.pdf <http://www.sigcomm.org/sites/default/files/ccr/papers/2015/July/0000000-0000002.pdf>
>  
> 
> From: tcpm <[email-masked] <mailto:[email-masked]>> on behalf of Yuchung Cheng <ycheng=[email-masked] <mailto:[email-masked]>>
> To: Mark Allman <[email-masked] <mailto:[email-masked]>>
> Cc: Extensions <[email-masked] <mailto:[email-masked]>>
> On Mon, Aug 9, 2021 at 10:05 AM Mark Allman <[email-masked] <mailto:[email-masked]>> wrote:
> 
>

---

From: Martin Duke <[email-masked]>
Date: Thu, 12 Aug 2021 11:39:00 -0700

> I don’t think 3465 can be upgraded to PS as-is.

Why not?

> Personally, I like the structure of 6928 more than 3465. Is it possible to
> rope in ABC (3465) in a 6928bis and
>
change the title - Would like to hear more from the authors of both the
> RFCs.

It will take a bit more editorial work, but there's no reason a single
document can't obsolete both RFCs.

> Vidhi

Thanks for writing a draft. That will move things along.

---

From: Vidhi Goel <[email-masked]>
Date: Thu, 12 Aug 2021 11:44:34 -0700

> On Thu, Aug 12, 2021 at 11:34 AM Vidhi Goel <[email-masked] <mailto:[email-masked]>> wrote:

Because L=2 as MUST is too constrained for high BDPs and most stacks don’t even follow that.

> On Aug 12, 2021, at 11:39 AM, Martin Duke <[email-masked]> wrote:
> 
> 
> 
> On Thu, Aug 12, 2021 at 11:34 AM Vidhi Goel <[email-masked] <mailto:[email-masked]>> wrote:
> Personally, I like the structure of 6928 more than 3465. Is it possible to rope in ABC (3465) in a 6928bis and 
> change the title - Would like to hear more from the authors of both the RFCs.
> 
> It will take a bit more editorial work, but there's no reason a single document can't obsolete both RFCs.
>  
> 
> Thanks,
> Vidhi
>

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 12 Aug 2021 20:55:40 +0200

> On 12. Aug 2021, at 20:44, Vidhi Goel <vidhi_goel=[email-masked]> wrote:
> 
My reading is:

* However, ABC with L=1*SMSS bytes is more conservative in a
  number of key ways (as discussed in the next section) and therefore,
  this document suggests that even though with L=1*SMSS bytes TCP
  stacks will see little performance change, ABC SHOULD be used.

* This document specifies that TCP implementations MAY use L=2*SMSS
  bytes and MUST NOT use L > 2*SMSS bytes.

* The exception to the above suggestion is during a slow start phase
  that follows a retransmission timeout (RTO).  In this situation, a
  TCP MUST use L=1*SMSS

So I don't think saying L = 2 is a MUST is a summary. For me it is more
important that it says MUST NOT be larger than 2.

Best regards
Michael

---

From: "Mark Allman" <[email-masked]>
Date: Thu, 12 Aug 2021 15:06:20 -0400

> RFC3465 (Experimental): L=2

Yep.

> 1. The most comprehensive solution would be a 3465bis PS doc that
>    obsoletes 3465 and updates 5681 with L= Infinity, 8, or
>    whatever number the group decides on. If we agree this is the
>    right thing to do and someone has the energy to drop
>    3465bis-00, then let's do so to have a basis of discussion.

Approach-wise this is OK.  But, there are issues here ...

  - First, we could probably move to L=2 pretty easily (as
    standard).  We have been using byte counting with standard
    delayed ACKs for a long time in stacks that do not use an L.
    So, we have effectively been living with this.

  - Moving to L > 2 (5, 10, \infinity) would seem to me to require
    someone to make the case that we have experience that suggests
    this is OK.  Or, that we have some suggestions that we could run
    at experimental L>2 for a while.  There have been a bunch of
    folks saying that we understand L > 2 works fine, but in pretty
    hand-wavy, mailing list sorts of ways.  I.e., there are stretch
    ACKs.  Or, there is an implementation with an L.  Or, there are
    offload systems that aggregate packets/ACKs.  When we haven't
    seen---that I am aware of---is a discussion that shows L
    > 2 is a good idea.  Or, that it isn't a bad idea.  It seems to
    me that the usual way these things get advanced is to offer some
    tangible / concrete assessment that says "hey, this is perhaps a
    reasonable thing to change".

  - I am not saying we shouldn't change L.  Perhaps we should.
    E.g., with pacing it seems like a no-brainer.  I am just saying
    that the case hasn't even been attempted.

> 3. There might be interest in a more generalized 5681bis, but my
>    instinct is that this is a bigger job that will delay the most
>    urgently needed update -- which is not to say that we shouldn't
>    do that document too.

I think a very scoped 5681bis could be done quickly.  If it is a
change to L and/or a change to IW we can use the 6298 model and make
the very tiny text change and add an appendix that justifies the
change.  So, the normative text of the RFC changes very little.

> internet in the absence of any sort of pacing (ack-driven or

Hm.  Well.  Hm.  I'd have to think about that one.  I do think that
just defining some general burst limit could prove to be thorny.
E.g., if you said the max burst size was 10 so IW=10 would be fine
then would a stretch ACK that covered 20 packets only be able to
liberate 10 packets?  Or, if we said a stretch ACK could liberate 20
packets would that imply an IW of 20 packets?  And, do we really
draw no distinction between what we can send into an unknown network
and what we can send into a network we have probed enough to have
some idea about?  This seems difficult.

allman