Subject: [tcpm] Question on CUBIC and its TCP CC RFC basics

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Tue, 8 Mar 2016 11:06:45 +0100

Hi,

I am reading the TCP CUBIC draft specification with the intend of
implementing – or rather specifically  - with the intend of implementing

in appropriately adjusted manner for SCTP. I hope that you may help with
some clarifying questions to the basic parts ?

The reference to RFC6582 and RFC2018 BUT NOT to RFC 6675 makes me wonder if
there is some

particular point which I am missing – I.e.:

It is said – section 3.1:

The

   protocol does not make any change to the fast recovery and retransmit

   of TCP-NewReno [RFC6582] and TCP-SACK [RFC2018].  During congestion

^^^^^^^^^^^^^^^

   avoidance after fast recovery, CUBIC changes the window update

  ^^^^^^^^^^^^^^^^^^

   algorithm of Standard TCP.

Apart from the missing reference to RFC5681/RFC6675 this sounds clear.
CUBIC CC does not impact CC operation during Fast Recovery but only impacts
the CC operation once Fast Recovery has completed

Is that the correct understanding ?

Hmm .. except that the ramp down factor is not 0.5 but 0.2.

It is then further said – section 3.1:

The window growth function of CUBIC uses the following function:

      W_cubic(t) = C*(t-K)^3 + W_max (Eq. 1)

   where C is a constant fixed to determine the aggressiveness of window

   growth in high BDP networks, t is the elapsed time from the last

^^^^^^^^^^

   window reduction,and K is the time period that the above function

  ^^^^^^^^^^^^^^^

   takes to increase the current window size to W_max when there is no

^^^^^^^^^^^^^^

   further loss event and is calculated by using the following equation:

   ^^^^^^^^^^^

My problem with understanding this formulation is that these two events
(last window reduction and

no further loss event) need not coincide. On the other hand then I assume
that the whole idea is that W_MAX is reached when t=K

and if this is right then we are speaking about the same event – OR am I
missing something ?

If I am right (but perhaps not) in assuming that we are speaking about the
same point in time – then

what point in time are we speaking about really ?

In my nativity I have assumed that the actual point in time would be at
exit of FR – alternatively  –  time

of last observed loss event even if this may make the slope steeper at exit
of FR.

Which is it – one of the two or something different ?

Many Thanks !

BR, Karen

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 8 Mar 2016 09:06:03 -0500

On Tue, Mar 8, 2016 at 5:06 AM, Karen Elisabeth Egede Nielsen
<[email-masked]> wrote:
> CUBIC CC does not impact CC operation during Fast Recovery but
> only impacts the CC operation once Fast Recovery has completed
>

Yes, that is correct.

> Hmm .. except that the ramp down factor is not 0.5 but 0.2.

Earlier versions used 0.2, but the current (long-time) ramp down
factor is 0.3 (eg see section 1). CUBIC cuts cwnd to 0.7x the prior
value cwnd had before entering recovery.

But at least in Linux that is not a change in the fast recovery code;
that is a different approach the congestion control module uses to
calculate the target cwnd that the fast recovery code (independent of
the congestion control module) reaches at the end of recovery.

> It is then further said – section 3.1:
...
> (last window reduction and no further loss event) need not coincide.

I think the text here in the draft is unclear. The phrase "when there
is no further loss event" is not referring to a moment in time, but
rather a condition. It might be more clearly expressed by substituting
"if" instead of "when", so that it reads:

    "K is the time period that the above function takes to increase the
    current window size to W_max if there is no further loss event"

I presume this is all in reference to:
  https://tools.ietf.org/html/draft-ietf-tcpm-cubic-01

cheers,
neal

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Tue, 8 Mar 2016 15:18:55 +0100

Hi Neal,

Thanks a lot for your valuable feedback.
Yes - Sorry this was in relation to
https://tools.ietf.org/html/draft-ietf-tcpm-cubic-01.

A little follow up inside below.

Something more - do you happen to know why the "Clamp" on the CWND increase
is not part of https://tools.ietf.org/html/draft-ietf-tcpm-cubic-01.
Is it because it is believed not to be needed given appropriate pacing or
has it simply not gotten in yet ?

Many Thanks !

> -----Original Message-----
> From: Neal Cardwell [mailto:[email-masked]]
> Sent: 8. marts 2016 15:06
> To: Karen Elisabeth Egede Nielsen <[email-masked]>
> Cc: [email-masked]; [email-masked]
> Subject: Re: [tcpm] Question on CUBIC and its TCP CC RFC basics
>
> > CUBIC CC does not impact CC operation during Fast Recovery but only
> > impacts the CC operation once Fast Recovery has completed
> >
> Earlier versions used 0.2, but the current (long-time) ramp down factor is
> 0.3
> (eg see section 1). CUBIC cuts cwnd to 0.7x the prior value cwnd had
> before
> entering recovery.
>
[Karen Elisabeth Egede Nielsen] yes that is understood :-)

> But at least in Linux that is not a change in the fast recovery code; that
> is a
> different approach the congestion control module uses to calculate the
> target cwnd that the fast recovery code (independent of the congestion
> control module) reaches at the end of recovery.
>
[Karen Elisabeth Egede Nielsen] hmm - not sure I understand this. I asume
that the CWND used _during_ FR is the 0.7*prior value CWND
And that the factor 0.7 (opposed to RFC5681/RFC6675 factor 0.5 ) impacts the
fast recovery operation (assuming PRR not in) - while it does of course not
impact the fast retransmission code lines which presumably just related to
the CWND as updated by the CC logic - is that what you mean ?
Or do you mean to say that - in Linux implementation - the 0.7*prior CWND
only impacts the CC evaluations at the _end of_ recovery ?

> > My problem with understanding this formulation is that these two
> > events (last window reduction and no further loss event) need not
> coincide.
>
> I think the text here in the draft is unclear. The phrase "when there is
> no
> further loss event" is not referring to a moment in time, but rather a
> condition. It might be more clearly expressed by substituting "if" instead
> of
> "when", so that it reads:
>
[Karen Elisabeth Egede Nielsen] ok thanks.

I think that you have confirmed that the time t starts at _end_ of fast
recovery -
Correctly understood ?

BR, Karen

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Tue, 8 Mar 2016 15:28:53 +0100

Found part of the answer myself - Clamp was removed in Linux 2.6.25

BR, Karen

> From: Karen Elisabeth Egede Nielsen [mailto:[email-masked]]
> Sent: 8. marts 2016 15:19
> To: 'Neal Cardwell' <[email-masked]>
> Cc: [email-masked]' <[email-masked]>; [email-masked]'
> <draft-
> [email-masked]>
> Subject: RE: [tcpm] Question on CUBIC and its TCP CC RFC basics
>
> Hi Neal,
>
> > that is a
> > condition. It might be more clearly expressed by substituting "if"
> > instead of

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 8 Mar 2016 10:24:56 -0500

> that the CWND used _during_ FR is the 0.7*prior value CWND

In Linux CUBIC the cwnd used during FR is determined by the Linux PRR
code (RFC 6937), operating with an ssthresh value calculated by the
congestion control module.

> the CWND as updated by the CC logic - is that what you mean ?

Yes.

> only impacts the CC evaluations at the _end of_ recovery ?

The 0.7x impacts behavior during recovery as well (though it is
indirect, in the sense of setting a target number of packets to have
in flight).

> I think that you have confirmed that the time t starts at _end_ of fast

Yes, the CUBIC t is measured relative to the end of fast recovery.

neal

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Tue, 8 Mar 2016 16:26:27 +0100

Hi Neal,

Great ! Many thanks for the information.

BR, Karen

> Sent: 8. marts 2016 16:25
> > [Karen Elisabeth Egede Nielsen] hmm - not sure I understand this. I
> > asume that the CWND used _during_ FR is the 0.7*prior value CWND
>
> In Linux CUBIC the cwnd used during FR is determined by the Linux PRR code
> (RFC 6937), operating with an ssthresh value calculated by the congestion
> control module.
>
> > And that the factor 0.7 (opposed to RFC5681/RFC6675 factor 0.5 )
> > impacts the fast recovery operation (assuming PRR not in) - while it
> > does of course not impact the fast retransmission code lines which
> > presumably just related to the CWND as updated by the CC logic - is that
> what you mean ?
>
> Yes.
>
> > Or do you mean to say that - in Linux implementation - the 0.7*prior
> > CWND only impacts the CC evaluations at the _end of_ recovery ?
>
> The 0.7x impacts behavior during recovery as well (though it is indirect,
> in the
> sense of setting a target number of packets to have in flight).
>
> > I think that you have confirmed that the time t starts at _end_ of
> > fast recovery - Correctly understood ?
>

---

From: Lisong Xu <[email-masked]>
Date: Tue, 8 Mar 2016 20:53:39 -0600

Thanks, Neal, for answering these questions. I was on my trip the whole 
day, and just came back to home.
Lisong

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Wed, 9 Mar 2016 11:26:00 +0100

HI Neal, Lisong, others

Thank you both.

Sorry for asking "stupid" questions, but could you be so kind as to comment
on the following also, :-):

My simple understanding of CUBIC CC (with ABC glasses on) is that instead of
incrementing  CWND by 1 MTU/MSS per CWND bytes acknowledged,
CUBIC CC dictates to increment CWND by 1 MTU per MTU*CWND/(CUBIC_target -
CWND) bytes acknowledged.
(here CUBIC_Target=W(t+RTT) calculated in bytes))

Can you help me understand why the "TCP friendliness" relates to an average
Reno window at a given time and not
just simply to that at any given time ( CWND_cnt_bytes =)
MTU*CWND/(CUBIC_target - CWND) should be clamped by CWND.
 I.e., never be allowed to go higher than CWND - ?

Perhaps this is just a (choice of) implementation issue (?) -  If there are
logical reason then I am probably just missing something....

Many Thanks in advance.

BR, Karen

> From: Lisong Xu [mailto:[email-masked]]
> Sent: 9. marts 2016 03:54
> To: Neal Cardwell <[email-masked]>; Karen Elisabeth Egede Nielsen
> <[email-masked]>
> >>> that
> asume
> >> impacts
> the
> CWND

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Wed, 9 Mar 2016 15:55:44 +0100

HI,

Sorry for thinking load here. ..
I understand that the slope by purpose should be lower around wmax. Sorry.

BR, Karen

> Sent: 9. marts 2016 11:26
> To: 'Lisong Xu' <[email-masked]>; 'Neal Cardwell' <[email-masked]>
> incrementing  CWND by 1 MTU/MSS per CWND bytes acknowledged, CUBIC
> CC dictates to increment CWND by 1 MTU per MTU*CWND/(CUBIC_target -
> Can you help me understand why the "TCP friendliness" relates to an
> average Reno window at a given time and not just simply to that at any
> given
> time ( CWND_cnt_bytes =) MTU*CWND/(CUBIC_target - CWND) should be
> clamped by CWND.
> Perhaps this is just a (choice of) implementation issue (?) -  If there
> are logical
> reason then I am probably just missing something....
>
> Many Thanks in advance.
>
> BR, Karen
>
> > To: Neal Cardwell <[email-masked]>; Karen Elisabeth Egede
> > Nielsen <[email-masked]>
> > Thanks, Neal, for answering these questions. I was on my trip the
> > whole day, and just came back to home.
> > Lisong
> >
> > >>> But at least in Linux that is not a change in the fast recovery
> > >>> code; that is a different approach the congestion control module
> > >>> uses to calculate the target cwnd that the fast recovery code
> > >>> (independent of the congestion control module) reaches at the end
> > >>> of recovery.
> > >>>
> > > In Linux CUBIC the cwnd used during FR is determined by the Linux
> > > PRR code (RFC 6937), operating with an ssthresh value calculated by
> > > the congestion control module.
> > >
> > >> fast recovery operation (assuming PRR not in) - while it does of
> > >> course not impact the fast retransmission code lines which

---

From: Lisong Xu <[email-masked]>
Date: Wed, 9 Mar 2016 09:23:12 -0600

>   I.e., never be allowed to go higher than CWND - ?

Hi Karen,

"tcp friendliness" is about the relation between the long-term average 
window size of cubic and that of reno. Because the long-term average 
window size depends on both the increase function (what you mentioned 
above) and the decrease parameter (i.e., the beta), we check "tcp 
friendliness" only periodically at some points.

Thanks
Lisong

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Thu, 10 Mar 2016 14:19:55 +0100

HI Lisong, Neal,

Thank you so much for the information.
I think that I will have to understand the TCP_friendly reactions a little
better.
The motivation and the analysis done. Also understanding application
limitation aspects.

In terms of the CUBIB CC application limitation then for SCTP we may (at
least initially) choose the approach of restarting the CUBIC CC slope.
Which was not the choice taken for the fix in Linux TCP, I understand from:
https://github.com/torvalds/linux/commit/30927520dbae297182990bb21d08762bcc35ce1d
But as the application limitation of SCTP is more conservative and
restricted than for TCP then this more aggressive ramp up
after application limitations may be ok. (We will see about that).

Returning to the TCP_friendly function - - -

would you say that the behavior _implemented_, analysed and tested (and
deployed) is represented by

the description in section 3.2 of the draft
https://datatracker.ietf.org/doc/draft-ietf-tcpm-cubic/?include_text=1 -
i.e.:

W_aimd(t) = W_max*beta_aimd +
                   [3*(1-beta_aimd)/(1+beta_aimd)] * (t/RTT) (Eq. 4)

I cannot say that I have understood that this is achieved by the linux
implementation which seems to relate to a cwnd calculated by
CC CUBIC to estimate the TCP CC window. But perhaps it is ok....

Have you modified this to relate to TCP CC implementing cwnd validation
during application limitations proper ?

Many Thanks.
BR, Karen

> Sent: 9. marts 2016 16:23
> Cc: Neal Cardwell <[email-masked]>; [email-masked]; draft-ietf-tcpm-
> [email-masked]
> > Sorry for asking "stupid" questions, but could you be so kind as to
> > comment on the following also, :-):
> >
> > My simple understanding of CUBIC CC (with ABC glasses on) is that
> > instead of incrementing  CWND by 1 MTU/MSS per CWND bytes
> > acknowledged, CUBIC CC dictates to increment CWND by 1 MTU per
> > MTU*CWND/(CUBIC_target -
> > given time ( CWND_cnt_bytes =) MTU*CWND/(CUBIC_target - CWND)
> should
> > be clamped by CWND.
> > Perhaps this is just a (choice of) implementation issue (?) -  If
> > there are logical reason then I am probably just missing something....
> >
> > Many Thanks in advance.
> >
> > BR, Karen
> >
> >>>> presumably just related to the CWND as updated by the CC logic - is
> >>>> that what you mean ?
> >>> Yes.
> >>>

---

From: Lisong Xu <[email-masked]>
Date: Thu, 10 Mar 2016 08:48:41 -0600

Hi Karen,

Eq 4 is implemented at lines 482-483 and 305-322.

https://github.com/torvalds/linux/blob/30927520dbae297182990bb21d08762bcc35ce1d/net/ipv4/tcp_cubic.c

Thanks
Lisong

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Fri, 11 Mar 2016 08:37:58 +0100

Hi Lisong,

Thanks.

So it seems application limitations are not taken into account in the TCP
friendliness function.
I.e. ACK's arriving while window is underutilized will make the window be
increased.
Correct ?

BR, Karen

> Sent: 10. marts 2016 15:49
> To: Karen Elisabeth Egede Nielsen <[email-masked]>; Neal
> Cardwell <[email-masked]>
> > I think that I will have to understand the TCP_friendly reactions a
> > little better.
> > In terms of the CUBIB CC application limitation then for SCTP we may
> > (at least initially) choose the approach of restarting the CUBIC CC
> > slope.
> https://github.com/torvalds/linux/commit/30927520dbae297182990bb21d08
> 7
> > 62bcc35ce1d But as the application limitation of SCTP is more
> > conservative and restricted than for TCP then this more aggressive
> > ramp up after application limitations may be ok. (We will see about
> > that).
> >
> > implementation which seems to relate to a cwnd calculated by CC CUBIC
> > to estimate the TCP CC window. But perhaps it is ok....
> >
> > Have you modified this to relate to TCP CC implementing cwnd
> > validation during application limitations proper ?
> >
> > Many Thanks.
> > BR, Karen
> >
> >> Cc: Neal Cardwell <[email-masked]>; [email-masked];
> >> draft-ietf-tcpm- [email-masked]
> >>> average Reno window at a given time and not just simply to that at
> >>> any given time ( CWND_cnt_bytes =) MTU*CWND/(CUBIC_target -
> CWND)
> >> should
> >>> be clamped by CWND.
> >> "tcp friendliness" is about the relation between the long-term
> >> average window size of cubic and that of reno. Because the long-term
> >> average window size depends on both the increase function (what you
> >> mentioned
> >>>>>>> (independent of the congestion control module) reaches at the
> >>>>>>> end of recovery.
> >>>>>>>
> >>>>> PRR code (RFC 6937), operating with an ssthresh value calculated
> >>>>> by the congestion control module.
> >>>>>
> >>>>>> presumably just related to the CWND as updated by the CC logic -
> >>>>>> is that what you mean ?
> >>>>> Yes.
> >>>>>
> >>>>> indirect, in the sense of setting a target number of packets to
> >>>>> have in flight).
> >>>>>
> >>>>>> I think that you have confirmed that the time t starts at _end_
> >>>>>> of fast recovery - Correctly understood ?

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 11 Mar 2016 08:05:08 -0500

> I.e. ACK's arriving while window is underutilized will make the window be

I suggest reading the Linux CUBIC code. The code (in the first check in the
main CUBIC function, bictcp_cong_avoid()) returns without doing anything if
the flow is not cwnd-limited, as determined by calling
tcp_is_cwnd_limited():

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/tcp_cubic.c?id=afd2ff9b7e1b367172f18ba7f693dfb62bdcb2dc#n341

So the TCP friendliness function will not use ACKs that arrive while the
connection is not cwnd-limited.

neal

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Fri, 11 Mar 2016 14:33:39 +0100

Hi Neal,

Thanks for the information. (I have not followed up by checking consistency
of the code ;-)) as my particular concern

is to understand what CUBIC CC is (or is supposed to be).

I think that it would be very useful to add this description of the TCP
friendliness function in

https://datatracker.ietf.org/doc/draft-ietf-tcpm-cubic/

as it is supposed to reflect the Linux CUBIC implementation.

BR, Karen

*From:* Neal Cardwell [mailto:[email-masked]]
*Sent:* 11. marts 2016 14:05
*To:* Karen Elisabeth Egede Nielsen <[email-masked]>
*Cc:* Lisong Xu <[email-masked]>; [email-masked]; [email-masked]
*Subject:* Re: [tcpm] Question on CUBIC and its TCP CC RFC basics

Hi Lisong,

Thanks.

neal

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 11 Mar 2016 13:43:53 +0000

On 2016-03-11, at 14:33, Karen Elisabeth Egede Nielsen <[email-masked]> wrote:
> 
> I think that it would be very useful to add this description of the TCP friendliness function in
> https://datatracker.ietf.org/doc/draft-ietf-tcpm-cubic/ <https://datatracker.ietf.org/doc/draft-ietf-tcpm-cubic/>

Agreed. The main reason for the ID is to document CUBIC so that people do not have to read GPL code to implement it.

Lars

---

From: Lisong Xu <[email-masked]>
Date: Fri, 11 Mar 2016 08:54:21 -0600

Thank you all.

I will make the following changes in the next version of cubic draft.

1. a new subsection 4.10: add the description about the behaviors of 
cubic when it is not cwnd-limited. Specifically, cubic does not do 
anything, when the flow is not limited by the cubic congestion window size.

2. section 3.1 "when there is no further loss event" --> "if there is no 
further loss event"

3. section 3.1, "t is the elapsed time from the last window reduction" 
--> "t is the elapsed time from the last window reduction (measured 
right after the fast recovery)"

Is there anything else that I missed?

Thanks
Lisong

On 3/11/2016 7:43 AM, Eggert, Lars wrote:
> On 2016-03-11, at 14:33, Karen Elisabeth Egede Nielsen 
> <[email-masked] <mailto:[email-masked]>> wrote:
>>
> Agreed. The main reason for the ID is to document CUBIC so that people 
> do not have to read GPL code to implement it.
>
> Lars

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 11 Mar 2016 10:18:07 -0500

Related to  the question of being cwnd-limited, and this sentence in
section 3.1:

    t is the elapsed time from the last window reduction

I think it is important to mention that recent versions of Linux CUBIC
take care to not include idle time in this "t". The fixes are here:

  tcp_cubic: better follow cubic curve after idle period
  http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=30927520dbae297182990bb21d08762bcc35ce1d

  tcp_cubic: do not set epoch_start in the future
  http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c2e7204d180f8efc80f27959ca9cf16fa17f67db

The issue was that after long idle periods "t" was including a lot of
time spent idle, which was implicitly treated by the algorithm as time
spent probing the network without packet loss, leading to very high
cwnd values after restarting from idle. If the code takes care to
exclude idle time, as in the above changes, then this problem does not
show up.

neal

On Fri, Mar 11, 2016 at 9:54 AM, Lisong Xu <[email-masked]> wrote:
> 1. a new subsection 4.10: add the description about the behaviors of cubic
> when it is not cwnd-limited. Specifically, cubic does not do anything, when
> the flow is not limited by the cubic congestion window size.
>
> 3. section 3.1, "t is the elapsed time from the last window reduction" -->
> "t is the elapsed time from the last window reduction (measured right after
> the fast recovery)"
>
> Agreed. The main reason for the ID is to document CUBIC so that people do
> not have to read GPL code to implement it.
>
> Lars
>
>

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Fri, 11 Mar 2016 16:20:53 +0100

Hi,

Thanks,

Well the description of TCP friendliness (which as CUBIC relates to a real
time parameter (!))

 as well as the description of CUBIC CC need to relate to what

should happen in respect to application limitation. As fully recognized in
the Linux TCP implementation

it seems, real time does not stop just because CUBIC does nothing.

BR, Karen

*From:* Lisong Xu [mailto:[email-masked]]
*Sent:* 11. marts 2016 15:54
*To:* Eggert, Lars <[email-masked]>; Karen Elisabeth Egede Nielsen <
[email-masked]>
*Cc:* Neal Cardwell <[email-masked]>; [email-masked];
[email-masked]
*Subject:* Re: [tcpm] Question on CUBIC and its TCP CC RFC basics

Thank you all.

Thanks
Lisong

On 2016-03-11, at 14:33, Karen Elisabeth Egede Nielsen <
[email-masked]> wrote:

Lars

---

From: Lisong Xu <[email-masked]>
Date: Fri, 11 Mar 2016 22:48:00 -0600

Thanks, Neal!
Lisong