Subject: Re: [tcpm] Proceeding 8312bis draft (Re: 2nd WGLC for draft-ietf-tcpm-rfc8312bis)

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 1 Mar 2022 10:54:41 +0000

In case that helps, +1 to publish as PS. We already discussed this in length when we made the decision to go for PS. Cubic is widely deployed (and didn’t break the Internet) and publishing as PS reflects this reality.

I think it also is important to document known limitation or short-comings of Cubic such that future congestion control research can take them into account accordingly. However, I don’t believe people are planning to explicitly do further experimentation with Cubic (rather than developing new schemes instead).

Mirja

From: tcpm <[email-masked]> on behalf of Vidhi Goel <vidhi_goel=[email-masked]>
Date: Monday, 28. February 2022 at 20:24
To: Neal Cardwell <ncardwell=[email-masked]>
Cc: "[email-masked] Extensions" <[email-masked]>, tcpm-chairs <[email-masked]>
Subject: Re: [tcpm] Proceeding 8312bis draft (Re: 2nd WGLC for draft-ietf-tcpm-rfc8312bis)

I would argue for:

  (2) publish the 8312bis draft as a PS doc, and

  (2b) publish an additional doc that describes any aspects of CUBIC where the community would like to see further experiments or improvements.

IMHO Michael Scharf made an excellent argument for PS status for CUBIC on Feb 22 ( https://mailarchive.ietf.org/arch/msg/tcpm/k47hlSsx1lRupy5LRTJ5pQ6LrDE/ ) by simply quoting a definition of PS from RFC 7127:

I agree with Neal’s suggestion based on PS status definition from the above RFC.

Thanks,
Vidhi

On Feb 28, 2022, at 6:29 AM, Neal Cardwell <ncardwell=[email-masked]<mailto:ncardwell=[email-masked]>> wrote:

On Mon, Feb 28, 2022 at 5:07 AM Yoshifumi Nishida <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

We've been discussing how to proceed 8312bis draft, but I think we still haven't settled yet.
In my understanding, there seems to be the following options so far.
if you have some thoughts on them or you have other options for this, please share.
BTW, please note that this discussion might affect our future drafts because we want to apply the same bar for all standard docs as much as possible.

1: Publish the draft as a non-standard doc
    a: publish as an informational doc

    b: publish as an experimental doc
        * we might need to describe what experiments are expected.

2: Publish the draft as a PS doc
    a: do additional experiments to make sure there's no concern as a PS doc.
        * we will need to decide what kinds of experiments are required, what would be the expected results before the experiments.
        * we need to delay the publication of the doc until the validation of the experiments has finished

    b: publish an additional doc that describes the checking points on the doc (and other related docs) for long term analysis.

I would argue for:

  (2) publish the 8312bis draft as a PS doc, and

  (2b) publish an additional doc that describes any aspects of CUBIC where the community would like to see further experiments or improvements.

IMHO Michael Scharf made an excellent argument for PS status for CUBIC on Feb 22 ( https://mailarchive.ietf.org/arch/msg/tcpm/k47hlSsx1lRupy5LRTJ5pQ6LrDE/ ) by simply quoting a definition of PS from RFC 7127:

   A Proposed Standard specification is stable, has resolved known

   design choices, has received significant community review, and

   appears to enjoy enough community interest to be considered valuable.

   Usually, neither implementation nor operational experience is

   required for the designation of a specification as a Proposed

   Standard.  However, such experience is highly desirable and will

   usually represent a strong argument in favor of a Proposed Standard

   designation.

   The IESG may require implementation and/or operational experience

   prior to granting Proposed Standard status to a specification that

   materially affects the core Internet protocols or that specifies

   behavior that may have significant operational impact on the

   Internet.

   A Proposed Standard will have no known technical omissions with

   respect to the requirements placed upon it.  Proposed Standards are

   of such quality that implementations can be deployed in the Internet.

   However, as with all technical specifications, Proposed Standards may

   be revised if problems are found or better solutions are identified,

   when experiences with deploying implementations of such technologies

   at scale is gathered.

It seems clear, IMHO, that CUBIC meets those criteria for being named a PS.

best regards,
neal

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Martin Duke <[email-masked]>
Date: Wed, 2 Mar 2022 12:12:34 -0800

Wearing no hats, I also support publication as PS. Quite aside from
legalistic arguments about what's in certain RFCs, it's self-evident that
Cubic has much wider deployment experience than nearly anything else that
we've granted Proposed Standard status.

On Tue, Mar 1, 2022 at 2:55 AM Mirja Kuehlewind <mirja.kuehlewind=
[email-masked]> wrote:

> In case that helps, +1 to publish as PS. We already discussed this in
> length when we made the decision to go for PS. Cubic is widely deployed
> (and didn’t break the Internet) and publishing as PS reflects this reality.
>
>
>
> I think it also is important to document known limitation or short-comings
> of Cubic such that future congestion control research can take them into
> account accordingly. However, I don’t believe people are planning to
> explicitly do further experimentation with Cubic (rather than developing
> new schemes instead).
>
>
>
> Mirja
>
>
>
>
>
>
>
> *From: *tcpm <[email-masked]> on behalf of Vidhi Goel <vidhi_goel=
> [email-masked]>
> *Date: *Monday, 28. February 2022 at 20:24
> *To: *Neal Cardwell <ncardwell=[email-masked]>
> *Cc: *"[email-masked] Extensions" <[email-masked]>, tcpm-chairs <
> [email-masked]>
> *Subject: *Re: [tcpm] Proceeding 8312bis draft (Re: 2nd WGLC for
> draft-ietf-tcpm-rfc8312bis)
>
>
>
> I would argue for:
>
>
>
>   (2b) publish an additional doc that describes any aspects of CUBIC where
> the community would like to see further experiments or improvements.
>
>
>
> IMHO Michael Scharf made an excellent argument for PS status for CUBIC on
> Feb 22 (
> https://mailarchive.ietf.org/arch/msg/tcpm/k47hlSsx1lRupy5LRTJ5pQ6LrDE/ )
> by simply quoting a definition of PS from RFC 7127:
>
>
>
>
>
> On Feb 28, 2022, at 6:29 AM, Neal Cardwell <
> ncardwell=[email-masked]> wrote:
>
>
>
>
>
> On Mon, Feb 28, 2022 at 5:07 AM Yoshifumi Nishida <[email-masked]>
> wrote:
>
> Hi,
>
>
>
> BTW, please note that this discussion might affect our future drafts
> because we want to apply the same bar for all standard docs as much as
> possible.
>
>
>
>         * we will need to decide what kinds of experiments are required,
> what would be the expected results before the experiments.
>
>         * we need to delay the publication of the doc until the
> validation of the experiments has finished
>
>
>
>     b: publish an additional doc that describes the checking points on
> the doc (and other related docs) for long term analysis.
>
>
>
> I would argue for:
>
>
>
>   (2b) publish an additional doc that describes any aspects of CUBIC where
> the community would like to see further experiments or improvements.
>
>
>
> IMHO Michael Scharf made an excellent argument for PS status for CUBIC on
> Feb 22 (
> https://mailarchive.ietf.org/arch/msg/tcpm/k47hlSsx1lRupy5LRTJ5pQ6LrDE/ )
> by simply quoting a definition of PS from RFC 7127:
>
>
>
> [email-masked]
> [email-masked]

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 3 Mar 2022 08:04:50 +0000

I support publication as PS, and I'd like to see areas of study to be 
identified where possible.

With a separate RFC to understand the topics to studied and how to 
improve the spec.

A while ago, I stepped back from further commenting on how 8312bis 
relates to other IETF drafts not through technical indifference, but 
because I saw it as important to have an RFC that relates to the 
currently deployed CUBIC. I'd be keen to rejoin this if we can figure 
out how to publish and progress.

Best wishes,
Gorry.

On 02/03/2022 20:12, Martin Duke wrote:
> legalistic arguments about what's in certain RFCs, it's self-evident 
> that Cubic has much wider deployment experience than nearly anything 
> else that we've granted Proposed Standard status.
>
> On Tue, Mar 1, 2022 at 2:55 AM Mirja Kuehlewind 
> <mirja.kuehlewind=[email-masked]> wrote:
>
>     In case that helps, +1 to publish as PS. We already discussed this
>     in length when we made the decision to go for PS. Cubic is widely
>     deployed (and didn’t break the Internet) and publishing as PS
>     reflects this reality.
>
>     I think it also is important to document known limitation or
>     short-comings of Cubic such that future congestion control
>     research can take them into account accordingly. However, I don’t
>     believe people are planning to explicitly do further
>     experimentation with Cubic (rather than developing new schemes
>     instead).
>
>     Mirja
>
>     *From: *tcpm <[email-masked]> on behalf of Vidhi Goel
>     <vidhi_goel=[email-masked]>
>     *Cc: *"[email-masked] Extensions" <[email-masked]>, tcpm-chairs
>     <[email-masked]>
>         (2b) publish an additional doc that describes any aspects of
>         CUBIC where the community would like to see further
>         experiments or improvements.
>
>         IMHO Michael Scharf made an excellent argument for PS status
>         for CUBIC on Feb 22 (
>         On Feb 28, 2022, at 6:29 AM, Neal Cardwell
>         <ncardwell=[email-masked]> wrote:
>
>         On Mon, Feb 28, 2022 at 5:07 AM Yoshifumi Nishida
>         <[email-masked]> wrote:
>
>             Hi,
>
>             BTW, please note that this discussion might affect our
>             future drafts because we want to apply the same bar for
>             all standard docs as much as possible.
>
>             * we will need to decide what kinds of experiments are
>             required, what would be the expected results before the
>             experiments.
>
>             b: publish an additional doc that describes the checking
>             points on the doc (and other related docs) for long term
>             analysis.
>
>         I would argue for:
>
>         (2b) publish an additional doc that describes any aspects of
>         CUBIC where the community would like to see further
>         experiments or improvements.
>
>         IMHO Michael Scharf made an excellent argument for PS status
>         for CUBIC on Feb 22
>         (https://mailarchive.ietf.org/arch/msg/tcpm/k47hlSsx1lRupy5LRTJ5pQ6LrDE/)
>         [email-masked] <mailto:[email-masked]>
>         <https://www.ietf.org/mailman/listinfo/tcpm>
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 9 Mar 2022 01:20:43 -0800

Hi folks,

Thanks for the feedback. I also exchanged opinions with several folks via
private e-mails.
Based on them, I am thinking we can probably separate the discussions about
how to address
the concerns that require further evaluations, such as publishing a doc,
additional experiments, etc.
This may take time, but I believe it won't block the publication of the bis
draft.

Also, it seems to me that we're converging to publish the bis draft as a PS.
I am currently hoping we can make a decision at the next WG meeting.
If you have some concerns on it, please send your thoughts to the ML or
speak up at the meeting.

Thanks,
--
Yoshi

On Mon, Feb 28, 2022 at 2:06 AM Yoshifumi Nishida <[email-masked]>
wrote:

>     b: publish an additional doc that describes the checking points on the
> doc (and other related docs) for long term analysis.
>
> Thanks,
> --
> Yoshi
>
>
> On Mon, Jan 31, 2022 at 11:43 PM Yoshifumi Nishida <[email-masked]>
> wrote:
>
>> Hello,
>>
>> After some discussions among chairs, we decided to run the 2nd WGLC on
>> draft-ietf-tcpm-rfc8312bis in consideration of the importance of the draft.
>> We'll be grateful if you could send your feedback to the ML. The WGLC
>> runs until *Feb 11*.
>>
>> If interested, you can check in-depth past discussions in the following
>> URL.
>> https://github.com/NTAP/rfc8312bis/
>>
>> Thank you so much!
>> --
>> tcpm co-chairs
>>
>>
>> On Wed, Jan 26, 2022 at 2:50 AM Lars Eggert <[email-masked]> wrote:
>>
>>> Hi,
>>>
>>> this -06 version rolls in all the changes requested during (and after)
>>> WGLC ended.
>>>
>>> I'll leave it up to the chairs to decide if another WGLC is warranted or
>>> the document can progress as-is.
>>>
>>> Thanks,
>>> Lars
>>>
>>>
>>> > On 2022-1-26, at 11:12, [email-masked] wrote:
>>> >
>>> >
>>> > A New Internet-Draft is available from the on-line Internet-Drafts
>>> directories.
>>> > This draft is a work item of the TCP Maintenance and Minor Extensions
>>> WG of the IETF.
>>> >
>>> >        Title           : CUBIC for Fast and Long-Distance Networks
>>> >        Authors         : Lisong Xu
>>> >                          Sangtae Ha
>>> >                          Injong Rhee
>>> >                          Vidhi Goel
>>> >                          Lars Eggert
>>> >       Filename        : draft-ietf-tcpm-rfc8312bis-06.txt
>>> >       Pages           : 35
>>> >       Date            : 2022-01-26
>>> >
>>> > Abstract:
>>> >   CUBIC is a standard TCP congestion control algorithm that uses a
>>> >   cubic function instead of a linear congestion window increase
>>> >   function to improve scalability and stability over fast and long-
>>> >   distance networks.  CUBIC has been adopted as the default TCP
>>> >   congestion control algorithm by the Linux, Windows, and Apple stacks.
>>> >
>>> >   This document updates the specification of CUBIC to include
>>> >   algorithmic improvements based on these implementations and recent
>>> >   academic work.  Based on the extensive deployment experience with
>>> >   CUBIC, it also moves the specification to the Standards Track,
>>> >   obsoleting RFC 8312.  This also requires updating RFC 5681, to allow
>>> >   for CUBIC's occasionally more aggressive sending behavior.
>>> >
>>> >
>>> > The IETF datatracker status page for this draft is:
>>> > https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc8312bis/
>>> >
>>> > There is also an HTML version available at:
>>> > https://www.ietf.org/archive/id/draft-ietf-tcpm-rfc8312bis-06.html
>>> >
>>> > A diff from the previous version is available at:
>>> > https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rfc8312bis-06
>>> >
>>> >
>>> > Internet-Drafts are also available by rsync at rsync.ietf.org:
>>> :internet-drafts
>>> >
>>> >

---

From: Markku Kojo <[email-masked]>
Date: Tue, 22 Mar 2022 18:57:42 +0200 (EET)

Hi Vidhi, all,

catching up ...

On Thu, 17 Feb 2022, Vidhi Goel wrote:

> I have a general question for the WG, are the "current practices" defined
> in https://datatracker.ietf.org/doc/html/rfc2914 still current? We are 22 years ahead of
> the published date of this draft and there have been magnitude of advancements in
> congestion control algorithms at the hosts and what is the right way to measure
> conformance for a congestion control. A very interesting paper about how to quantify a new
> congestion control, which most might have read, "Beyond Jain's Fairness Index: Setting the
> Bar For The Deployment of Congestion Control Algorithms"

When I am referring to fairness, I am not referring to equal fairness 
which is quite appropriately criticised in the paper you cite. Neither 
does RFC 2914 nor RFC 5033 require equal fairness. See also my other 
comments below that relate to this issue.

> I skimmed through RFC 2914 and it references obsoleted drafts for what it calls conformant
> TCP. Like below:
> 
> For TCP, this
>    means congestion control algorithms conformant with the current TCP
>    specification [RFC793, RFC1122, RFC2581].

None of these are HISTORIC, see more below.

> I don’t think we can continue to use old RFCs as a gold standard although we can use some
> the text as is if it still holds true.
>
>               Some of these may fail to implement
>               the TCP congestion avoidance mechanisms correctly because of
>             poor
>               implementation [RFC2525].  Others may deliberately be
>             implemented
>               with congestion avoidance algorithms that are more aggressive in
>               their use of bandwidth than other TCP implementations; this
>             would
>               allow a vendor to claim to have a "faster TCP".  The logical
>               consequence of such implementations would be a spiral of
>             increasingly
>               aggressive TCP implementations, or increasingly aggressive
>             transport
>               protocols, leading back to the point where there is effectively
>             no
>               congestion avoidance and the Internet is chronically congested.
> 
> For example, in this text quoted by Markku, I agree that we shouldn’t end up with
> algorithms that eventually lead to a chronically congested networks but at the same time,
> if a more aggressive approach provides better efficiency by ramping up a bit faster
> without leading to more congestion, then that shouldn’t be discouraged.

I fully agree and I have not (intentionally) criticised the feature of 
CUBIC when in the "genuine CUBIC" mode that ramps up faster. It is all OK 
once CUBIC is out of the region where std CC (Reno CC) is not able 
to utilize the available capacity (i.e., when not within the 
Reno-Friendly region). CUBIC may well ramp up faster when it does not 
steal the capacity from other (TCP-compatible) flows. This is one point 
the paper you pointed addresses and also allowed by RFC 2914 and RFC 
5033, and quite explicitly mentioned in RFC 5033 when it points to RFC 
3649.

The uncertainty we have is whether CUBIC is fair to Reno CC when it is in 
the Reno-Friendly region (i.e., whether the model and formula is correct) 
and whether CUBIC leaves the Reno-friendly region at a correct point (the 
value for C (=0.4) has not been validated in a *wide range* of 
environments.

And, very importantly, when CUBIC finds it needs to slow down (e.g., in 
the event of sudden heavy congestion), it does it very slowly as the 
paper pointed by Bob shows. This makes it to steal capacity from 
competing flows as it continues for a very long time at much higher rate 
than TCP-compatible flows do. There is the "fast convergence" feature 
with CUBIC but it really is not effective in releasing capacity to others 
when sudden congestion occurs as the measurements show and I have tried 
to explain, and I am happy to be convinced otherwise with experimental 
data.

Also, the use of MD=0.7 when in slow start is way too aggressive and 
dangerous, against the theory and design by Van Jacobson as I have 
pointed, and not justified at all in the draft. Applying MD=0.7 when a 
pkt loss is detected during slow-start results in a CUBIC sender to 
inject ~40% more packets than what is the available capacity during 
the Fast Recovery. This means that all these 40% excess packets (either 
those of the CUBIC sender or those of competing flows) are guaranteed to 
be dropped at a tail-drop bottleneck and, very importantly, note that 
these packets are so called "undelivered packets" (see RFC 2914) that 
create a danger of congestion collapse (of some degree) "by delivering 
packets through the network that are dropped before reaching their 
ultimate destination." [RFC 2914]

RFC 2914 emphasises:

  "This is probably the largest unresolved danger with respect to
   congestion collapse in the Internet today."

I urge the tcpm wg to very carefully consider the above issue.

In addition, I just noticed a little while ago when passing through the 
I-D but didn't comment yet that while the rule for changing alpha to 1 
when Wmax is reached in the Reno-friendly region is the correct thing to 
do during the normal steady state, it actually is incorrect action to 
take when in the fast convergence mode and within the Reno-friendly 
region, because it would act just opposite to what CUBIC does when in 
"genuine" CUBIC mode where it lowers the "plateau". That is, in the 
Reno-friendly region, instead of slowing down the increase rate during 
CA it actually accelerates as alpha is increased to 1 earlier than when 
not in the fast convergence mode. This has been modified in the bis 
version of the draft. I wonder if there is any experimental data 
available for evaluating the behavior?

> Faster TCP doesn’t
> necessarily mean that will cause congestion, it could just mean, instead of waiting 100s
> of RTTs to get to full link utilization, it does so in much lesser number of RTTs.
> In using Reno as a gold standard, we tend to forget that Reno is probably the cause of
> buffer bloat in the network. It basically requires network queues to be configured double
> the capacity in order to continue to have full link utilization after 50% reduction.
> And again, this might not have been evident at the time Reno was developed. We have
> realized these problems with time and we should move away from continuing to use Reno as
> the only acceptable Proposed Stds track congestion control.

Yes, having a shallower sawthooth is benefical as it allows shallower 
queues and it is not criticised.

>               It is convenient to divide flows into three classes: (1) TCP-
>               compatible flows, (2) unresponsive flows, i.e., flows that do
>             not
>               slow down when congestion occurs, and (3) flows that are
>             responsive
>               but are not TCP-compatible.  The last two classes contain more
>               aggressive flows that pose significant threats to Internet
>               performance,
> 
> 
> This second paragraph is also outdated. IIUC, TCP-compatible flows is equivalent to the
> RFC 2581 which is obsolete.

RFC 2581 is obsoleted by RFC 5681 when the doc was advanced to Draft 
Standard status, that is, the document and content is not obsoleted. How 
you are supposed to read such references is that you replace the cited 
RFC with the RFC that obsoleted the cited RFC (with some exceptions).

> I think we need to replace TCP-compatible with something else
> for two reasons:
> 
> 1. TCP is not the only transport protocol out there. What about QUIC, SCTP?

The word TCP-compatible has been selected to indicate std TCP congestion 
control behavior that the congestion controls implemented in other 
protocols than TCP are also required to follow. See the list below for 
many such protocols.

> 2. Congestion control algorithms should be evaluated in the same way for different
> transport protocols. So, a better word could be, Conformant congestion control, one which
> once accepted as Stds Track should just work for any transport protocol.
> 
> Perhaps, we need to show more experimental / deployment data for Cubic to become Stds
> Track but we shouldn’t rely on RFC2914 from the year 2000 to decide the acceptance
> criteria. If we continue to do so, then even 100 years from now, RFC 5681 will be the only
> TCP-conformant / TCP-compatible congestion control as defined by RFC 2914 and the one and
> only Stds Track congestion control to ever exist. (Please let me know if I missed any
> other Stds Track congestion control)

We have been constantly introducing TCP-conformant / TCP-compatible 
congestion controls and/or extending/modifying the existing Stds:

  RFC 3042, RFC 3448, RFC 4015, RFC 4340, RFC 4341, RFC 4342, RFC 4960,
  RFC 5348, RFC 5682, RFC 6582, RFC 6675, RFC 6298, RFC 8961, and many
  more.

Thanks,

/Markku

> Thanks,
> Vidhi
> 
>
>       On Feb 17, 2022, at 8:13 AM, Neal Cardwell
> On Thu, Feb 17, 2022 at 9:42 AM Markku Kojo <kojo=[email-masked]>
> wrote:
>       Hi Yoshi,
>
>       On Tue, 15 Feb 2022, Yoshifumi Nishida wrote:
>
>       > Hi Markku,
>       > 
>       > Thanks for the comments. I think these are very valid points. 
>       > However, I would like to check several things as a co-chair and a doc
>       shepherd before we
>       > discuss the points you've raised.
>       > 
>       > In my understanding (please correct me if I'm wrong), when this draft
>       was adopted as an WG
>       > item, I think the goal of the doc was some minor updates from RFC8312
>       which include more
>       > clarifications, minor changes and bug fixes. 
>       > However, if we try to address your concerns, I think we'll need to
>       invent a new version of
>       > CUBIC something like CUBIC++ or NewCUBIC in the end. 
>       > I won't deny the value of such doc, but, this seems not to be what we
>       agreed on
>       > beforehand.  
>       > if we proceed in this direction, I think we will need to check the WG
>       consensus whether
>       > this should be a new goal for the doc.
>       > 
>       > So, I would like to check if this is what you intend for the doc or
>       you think we can
>       > address your points while aligning with the original goal.
>       > Also, if someone has opinions on this, please share.
>
>       I think it is important that we remember the status of RFC 8312 and the 
>       decades long process that has been followed in tsv area for new 
>       TCP congestion control algorithms that have been proposed and submitted 
>       to IETF. In order to ensure that new cc algos are safe and fair, the 
>       process that has been followed for all current stds track TCP cc algos 
>       has required that the cc algo is first accepted and published as 
>       experimental RFC and only once enough supportive experimental evidence 
>       has been gathered the doc has become a candidate to be forwaded to stds 
>       track. We have even agreed on a relatively strict evaluation process to 
>       follow when cc algos are brought to the IETF to be published as 
>       experimental:
>
>       https://www.ietf.org/about/groups/iesg/statements/experimental-congestion-control/
>
>       RFC 8312 was published as "Informational" and if I recall correctly the 
>       idea was "just to publish what's out there" for the benefit of the 
>       community. RFC 8312 was never really evaluated, particularly not in the 
>       way new cc algos are supposed to be as per the agreed process.
>
>       I do not recall what/how exactly was agreed when rfc8312bis was
>       launched 
>       but I would be very interested to hear the justification why this doc 
>       does not need to follow the process mentioned above but we would like
>       to 
>       propose IETF to publish a non-evaluated Informational doc to be
>       published 
>       "with minor updates", i.e., without actual evaluation, as a stds track 
>       RFC? If the target really remains as PS then the bar should be even 
>       higher than what is described for experimental docs in the above
>       process 
>       document, i.e, what we have followod for experimental to be moved to
>       stds 
>       track.
>
>       The only justification that I have heard has beed "because CUBIC has
>       long 
>       and wide deployment experience" and "the Internet has not smelted or
>       that 
>       "we should have noticed if there were problems". We must, however, 
>       understand that in order to have noticeable bad impact CUBIC should
>       cause 
>       some sort of congestion collapse. Congestion collapse, however, is not
>       an 
>       issue with CUBIC nor with any other CC algo that applies an RTO
>       mechanisms 
>       together with correctly implemented Karn's algo that retains the 
>       backed-off RTO until an Ack is received for a new (not rexmitted) data 
>       packet. The issue is fairness to competing traffic. This cannot be 
>       observed by deploying and measuring the performance and behaviour of
>       CUBIC 
>       alone. CUBIC being more aggressive than current stds track TCP CC would 
>       just gives good performance results that one running CUBIC would be
>       happy 
>       with. One must evaluate CUBIC's impact on the competing (Reno CC)
>       traffic 
>       in range of environments which requires carefully designed active 
>       measurements with thoroughly-analyzed results (as required by the above 
>       process document, RFC 5033 and RFC 2914). What we seem to be missing is 
>       this evidence on CUBIC's impact and that is something the IETF must
>       focus 
>       on, not just that whether CUBIC can achieve better performance than
>       other 
>       existing CCs. The latter has been shown in many publications and is the 
>       majos focus in  many scientific papers proposing new algos.
>       I appreciate a lot that CUBIC has been implemented/developped and 
>       deployed for long and I wonder whether those deploying CUBIC have 
>       unpublished results the wg could review before taking the decicion?
>
>       I suggest everyone to read carefully RFC 2914 Sec 3.2 and particularly 
>       what it says about more aggressive (than RFC 5681) congestion control 
>       algorithms:
>
>         Some of these may fail to implement
>         the TCP congestion avoidance mechanisms correctly because of poor
>         implementation [RFC2525].  Others may deliberately be implemented
>         with congestion avoidance algorithms that are more aggressive in
>         their use of bandwidth than other TCP implementations; this would
>         allow a vendor to claim to have a "faster TCP".  The logical
>         consequence of such implementations would be a spiral of increasingly
>         aggressive TCP implementations, or increasingly aggressive transport
>         protocols, leading back to the point where there is effectively no
>         congestion avoidance and the Internet is chronically congested.
>
>       And:
>
>         It is convenient to divide flows into three classes: (1) TCP-
>         compatible flows, (2) unresponsive flows, i.e., flows that do not
>         slow down when congestion occurs, and (3) flows that are responsive
>         but are not TCP-compatible.  The last two classes contain more
>         aggressive flows that pose significant threats to Internet
>         performance,
>
>       As I have tried to point out there are several features with CUBIC
>       where 
>       it is likely to be (or to me it seems it obviously is) more aggressive 
>       than what is reguired to be TCP-compatible. I'm not aware of evidince 
>       presented to tcpm (or IETF/IRTF) which shows opposite (and I happy to
>       be 
>       educated what I have missed).
>
>       You may take my comments to be a part of the expert review phase 
>       performed by the IRTF/ICCRG for CUBIC. I'm not requesting to modify
>       this 
>       doc to CUBIC++ (or something) but it seems to be that this would be 
>       necessary if this doc intends to become published as PS. For
>       experimental, 
>       I think it would need some addtioinal updates and record the areas 
>       uncertainty and where more experimentation (clearly) is required.
> 
> 
> Hi Markku,
> 
> Thanks for your careful argument above. Given your argument, in order to continue to
> make constructive progress on rfc8312bis without blocking for new research and
> experimentation, I would propose that we change rfc8312bis so that it remains
> "Informational" (like RFC8312) rather than the "Standards Track" status currently
> listed at:
> 
>   https://ntap.github.io/rfc8312bis/draft-ietf-tcpm-rfc8312bis.html
> 
> As far as I can tell, that would allow all of the valuable contributions in
> rfc8312bis to carry forward and be published, while still respecting all of the
> considerations you enumerated above, which center around concerns with calling CUBIC
> "standards track".
> 
> How does that sound?
> 
> best,
> neal

---

From: Markku Kojo <[email-masked]>
Date: Tue, 22 Mar 2022 19:13:09 +0200 (EET)

Vidhi,

On Fri, 18 Feb 2022, Vidhi Goel wrote:

>
>             The other thing is that if we set a high bar for a PS doc related to CC, we should
>             apply
>             the same bar to all related docs.For example, I am personally wondering how much we
>             have
>             analyzed and performed experiments when we published RFC9002. (BTW, my intention is
>             NOT to
>             accuse a specific doc, there should be some more examples.) 
> 
>
>       Yes, IMO the bar should be the same. RFC 9002 was first targetted as experimental but was later
>       changed to PS. I did not have time to follow the progress much at all but in my opinion it would
>       probably have been better to publish it also as experimental first. On the other hand, it intends
>       to implement NewReno as close as possible, and therefore not to change the congestion control
>       principles that we have. I cannot tell how much experimental effort was put to verify the
>       behaviour of those features that are new in QUIC and/or differ from stds track TCP but hopefully
>       good enough. (Though there seems to be one issue related to resetting the exponential backoff of
>       PTO, which seems to require a more accurate specification to ensure unambiguous interpretation
>       and correct outcome, but that's an off topic here).
> 
> 
> RFC 9002 deviates from Reno as it uses `cwnd` during multiplicative decrease on a congestion event
> - https://www.rfc-editor.org/rfc/rfc9002.html#name-on-new-congestion-event. This is definitely a strong
> deviation from RFC 5681 which clearly mentions not to use cwnd.

Let me try to explain why using cwnd *without any constrains* like 
rfc8312bis text earlier did is the wrong thing to do and some perspective 
why FlightSize has been selected to be used in the stds track TCP 
congestion control RFCs, and why QUIC is not in conflict with RFC 5681:

1) Using cwnd when calculating the MD value allows arbitrarily large 
result values for cwnd because cwnd may increase way above rwnd when the 
sender is rwnd limited. Using FlightSize instead of cwnd prevents this 
from happening. That's why RFC 5681 explicitly warns about this.

2) When the sender is application data limited, the same problem may 
occur. Using FlightSize instead of cwnd in calculating MD value prevents 
also this problem from happening (but creates another problem by 
resulting in too low cwnd after MD with certain application send 
patterns as discussed, see more about this below).

To my understanding, FlightSize was selected at the time when RFC 2581 was 
published (and later TCP specs have continued to use it for consistency) 
because it resolved both problems above in a very simple way.
Moreover, the other problem of too low cwnd caused by the use of 
FlightSize was well known already at the time RFC 2581 got published and 
a bit later RFC 2861 was published with thinking that it resolves the 
other problem of too low cwnd. Therefore, it was considered to be the 
right thing to publish RFC 2581 with using FlighSize as a stds track spec 
because we followed the general guideline "be conservative in what you 
send" that all CC decisions should follow also today, that is, it is more 
adviseable to select conservative CC behaviour rather than allow 
potentially problematic too aggressive behaviour in a stds track RFC (in 
particular, if we even a bit unsure). So, this decision was not seen as a 
problem for any application limited use because RFC 2861 was already 
envisioned to solve the problem of too low cwnd.

Unfortunately, RFC 2861 turned out not to be a viable solution and it 
took a while an alternative way was introduced and later published as RFC 
7661. Meanwhile, it is not a surprise that many stacks abandoned the use 
of FlightSize because a viable solution for the application limited 
senders was not available and such application-limited traffic patterns 
were so common, creating suboptimal performance. However, AFAIK all 
stacks used other means to successfully avoid the problem 1) above and 
many did try various ways to avoid the problem 2) above. And they still 
do.

What becomes to QUIC, it does not introduce FlightSize at all, but it 
carefully tries to avoid both problems above (see Sec 7.8 of RFC 9002):

  When bytes in flight is smaller than the congestion window and
  sending is not pacing limited, the congestion window is
  underutilized.  This can happen due to insufficient application data
  or flow control limits.  When this occurs, the congestion window
  SHOULD NOT be increased in either slow start or congestion avoidance.

In addition, QUIC appropriately points to RFC 7661 as another potential 
way.
Hence, I don't see how RFC 9002 would be in conflict with RFC 5681.

So, the strict use of FlightSize in calculating the MD value is not the 
key here but making sure that the above problems become properly 
addressed in any alterative CC. Use of FlightSize is much preferred 
for consistency in the RFC series as was discussed in the FlightSize issue. 
After all, CC algorithms in RFCs describe the behavior to comply with not 
the way how to code it in the stacks. Stack implementers are, of course, 
free to adapt their code as they wish as long as the behavior remains the 
same as described in the RFC at hand. Linux, for example, has implemented 
NewReno for decades in a way that differs from the algo described in RFC 
2582/3782/6582 but the behaviour is essentially the same.

The most recent text in Sec 4.6 of the draft after discussions is now much 
better and correct compared to the version that was earlier last called. 
The only remaining issue there is that, AFAIK, the last sentence of 2nd 
para does not quite correctly capture the discussions we had nor what the 
stacks actually do:

  "Some implementations of CUBIC currently use _cwnd_ instead
   of _flight_size_ when calculating a new _ssthresh_ using Figure 5."

To my understanding those stacks prevent in one way or another the cwnd 
from increasing beyond rwnd. And, most (all?) of those stacks also 
somhow limit/disallow the cwnd from increasing when cwnd is underutilized.
Different stacks do the latter very differently, so it would have been 
very difficult to have concensus and appropriate text for a PS document 
based what the stacks do.

I would appreciate if the draft text in the above sentence could be 
further worked on to correctly capture that also the stacks that use cwnd 
in calculating the MD value actually (try to) prevent the problems 1) and 
2) above (to make sure nobody interprets the text such that it 
would be all ok to just use cwnd instead of FligtSize).

And, I'd like to conclude that I agree with Gorry in that the best way 
forward with solving the problems of application-limited senders in the 
RFC series is to collect data/experience on using RFC 7661 and advance it 
to PS. If I recall correctly, all who participated in the discussions 
considered RFC 7661 to provide a working solution.

Thanks,

/Markku

> Thanks,
> Vidhi
>
>       On Feb 18, 2022, at 2:23 AM, Markku Kojo <kojo=[email-masked]> wrote:
> 
> Hi Yoshi,
> 
> On Fri, 18 Feb 2022, Yoshifumi Nishida wrote:
>
>       Hi Markku,
>       Thanks for the comments. Yes, these are valid and important concerns. However, I have
>       mixed feelings about them.
>       I don't want to open a can of worms, but here's are some of my personally thoughts...
>       One thing is that "long and wide deployment experience" might not be a very strong claim,
>       but then, I am wondering what would be the strong one.
>       One may bring some solid experiment results, but the others might argue it uses too
>       convenient configurations, or the scale is too small or too short.
> 
> 
> Well, that has always been and IMO still is for the wg to decide when the evidence through experiments
> is good enough to proceed with publishing the doc. E.g., when RFC 5562 was considered for publication
> by the wg, the wg requested for more experiments before making the decision and Sally provided what was
> requested. And it was just about adding ECN capability to SYNACK but here we are discussing much more
> significant changes.
> 
> And more importantly, here we are mainly discussing whether there is any experimental evidence at all;
> for example, changing MD (from 0.5 to 0.7 or to whatever) must be considered separately when in slow
> start and when in congestion avoidance as I have pointed out. This requires tests that specifically
> target at studying the slow start overshoot phase and the following recovery phase. AFAIK, there are no
> such experimental results, no analysis nor evaluation presented to support the change when the sender
> is in slow start. So, we are not discussing the level of evidence but whether it exists at all, which
> is very different.
>
>       The other thing is that if we set a high bar for a PS doc related to CC, we should apply
>       the same bar to all related docs.For example, I am personally wondering how much we have
>       analyzed and performed experiments when we published RFC9002. (BTW, my intention is NOT to
>       accuse a specific doc, there should be some more examples.) 
> 
> 
> Yes, IMO the bar should be the same. RFC 9002 was first targetted as experimental but was later changed
> to PS. I did not have time to follow the progress much at all but in my opinion it would probably have
> been better to publish it also as experimental first. On the other hand, it intends to implement
> NewReno as close as possible, and therefore not to change the congestion control principles that we
> have. I cannot tell how much experimental effort was put to verify the behaviour of those features that
> are new in QUIC and/or differ from stds track TCP but hopefully good enough. (Though there seems to be
> one issue related to resetting the exponential backoff of PTO, which seems to require a more accurate
> specification to ensure unambiguous interpretation and correct outcome, but that's an off topic here).
>
>       I am not sure how we can maintain this kind of fairness between the docs at the moment.
> 
> 
> I think it is the responsibility for the entire tsv area to do it, and eventually the TSV ADs and IESG.
> The process to follow is clear and robust, and when followed it has shown to be effectual in producing
> good quality documents. AFAIK, we have not decided to change it, maybe just many of us are unaware of
> it or have forgotten its existence?
> Traditionally tsv area has been very careful in considering new CC proposals because congestion control
> is so essential for the stability and fairness of the Internet and I hope we are not sliping too much
> from there. AFAIK, that's one reason why the process was documented.
> 
> Thanks,
> 
> /Markku
>
>       Thanks,
>       --
>       Yoshi 
>       On Thu, Feb 17, 2022 at 6:41 AM Markku Kojo <[email-masked]> wrote:
>            Hi Yoshi,
>
>            > Hi Markku,
>            >
>            > Thanks for the comments. I think these are very valid points. 
>            > However, I would like to check several things as a co-chair and a doc
>            shepherd before we
>            > discuss the points you've raised.
>            >
>            > In my understanding (please correct me if I'm wrong), when this draft was
>            adopted as an WG
>            > item, I think the goal of the doc was some minor updates from RFC8312 which
>            include more
>            > clarifications, minor changes and bug fixes. 
>            > However, if we try to address your concerns, I think we'll need to invent a
>            new version of
>            > CUBIC something like CUBIC++ or NewCUBIC in the end. 
>            > I won't deny the value of such doc, but, this seems not to be what we agreed
>            on
>            > beforehand.  
>            > if we proceed in this direction, I think we will need to check the WG
>            consensus whether
>            > this should be a new goal for the doc.
>            >
>            > So, I would like to check if this is what you intend for the doc or you
>            think we can
>            > address your points while aligning with the original goal.
>            > Also, if someone has opinions on this, please share.
>
>            I do not recall what/how exactly was agreed when rfc8312bis was launched
>            does not need to follow the process mentioned above but we would like to
>            propose IETF to publish a non-evaluated Informational doc to be published
>            higher than what is described for experimental docs in the above process
>            document, i.e, what we have followod for experimental to be moved to stds
>            track.
>
>            The only justification that I have heard has beed "because CUBIC has long
>            and wide deployment experience" and "the Internet has not smelted or that
>            understand that in order to have noticeable bad impact CUBIC should cause
>            some sort of congestion collapse. Congestion collapse, however, is not an
>            issue with CUBIC nor with any other CC algo that applies an RTO mechanisms
>            observed by deploying and measuring the performance and behaviour of CUBIC
>            just gives good performance results that one running CUBIC would be happy
>            with. One must evaluate CUBIC's impact on the competing (Reno CC) traffic
>            this evidence on CUBIC's impact and that is something the IETF must focus
>            on, not just that whether CUBIC can achieve better performance than other
>            As I have tried to point out there are several features with CUBIC where
>            presented to tcpm (or IETF/IRTF) which shows opposite (and I happy to be
>            performed by the IRTF/ICCRG for CUBIC. I'm not requesting to modify this
>            necessary if this doc intends to become published as PS. For experimental,
>            > Thanks,
>            > --
>            > Yoshi
>            >
>            >
>            > On Fri, Feb 11, 2022 at 9:34 PM Markku Kojo <[email-masked]> wrote:
>            >       Hi Yoshi, all,
>            >
>            >       It seems to me that many issues that I raised have been solved,
>            thanks.
>            >       However, there are still a number of important issues that have not
>            been
>            >       addressed adequately. At least the following:
>            >
>            >       #135 on W_max. Yoshi's observation was correct that this is not
>            resolved:
>            >       the co-authors and original developpers of CUBIC (@lisongxu and
>            >       @sangtaeha) agreed in their last message that Wmax needs different
>            >       treatment for slow start and congestion avoidance and plan
>            comprehensive
>            >       (new) evaluation of it. This is obviously an open issue but the issue
>            >       was closed?
>            >
>            >       #85 (& #86 with basically same issue and these two were combined) This
>            >       (#85) is about ECN but the major issue is on using the same MD
>            >       factor in slow start and in congestion avoidance when using
>            >       loss-based CC. This (#85) remained closed even though I provided a
>            >       thorough explanation why it is wrong and against the original theory
>            and
>            >       design by Van Jacobson, against the congestion control principles (RFC
>            >       2914) and two co-authors agreed on this in their same last message to
>            >       #135 when they agreed on Wmax needing rework. This is an important
>            issue
>            >       that the wg should consider very carefully because it is not only
>            >       updating RFC 5681 but also in conflict with RFC 2914. How can
>            >       tcpm (and IETF) suggest and allow one CC algo to not follow congestion
>            >       control principles as set in RFC 2914 while requiring all other CCs to
>            >       follow RFC 2914 guidelines?
>            >       The current draft does not provide any justification for using the
>            same
>            >       MD factor in slow start as in congestion avoidance. Nor am I
>            >       awere of any experimental data that would support this change.
>            >       The fact thet CUBIC has been long deployed does not alone provide any
>            >       supporting evidence because CUBIC is likely to give good performance
>            as
>            >       it is overagressive and thereby unfair to competing traffic and users
>            >       tend to be happy when measuring the performance of the sending CUBIC
>            >       only, not the competing traffic that is badly impacted. HyStart++ is
>            >       suggested as mitigation to the problem but it cannot; HyStart++ is
>            only
>            >       applicable during initial slow start, not during slow start after RTO!
>            >       That is, the "SHOULD use HYStart++" text in Sec 4.10 is impossible
>            >       to implement as I have pointed out in my comments earlier. Using a
>            proper
>            >       MD factor in slow start is even more important if loss is detectected
>            >       during a RTO recovery because the sender is likely to face heavy
>            >       congestion in such a case and it is very bad if the sender continues
>            >       sending with overaggressive rate, stealing the capacity from and
>            causing
>            >       harm to coexisting flows. In addition, as I have explained, HyStart++
>            >       does not remove the problem even for the initial slow start as it is
>            not
>            >       shown to work always. Instead, the results with the HyStart++ draft
>            show
>            >       that it reduces 50% of rexmits and only 36% RTOs, meaning that there
>            is
>            >       likely to be a notable percentage of cases when a sender is still
>            >       in slow start when first loss is detected (i.e., HyStart++ had no
>            effect)
>            >       and a significant number of cases where a CUBIC sender is
>            >       overaggressive continuing with a 1-40% larger cwnd than what is the
>            >       available capacity. Note also that any delay-based heuristics like
>            >       HyStart++ are known to work poorly in various wireless environmens
>            where
>            >       link delay tends to vary a lot. We may come up with some other MD
>            factor
>            >       that 0.5 when in slow start and HyStart++ is in use, but that is
>            >       experimental, if not research, and definitely not ready for stds
>            track.
>            >
>            >       #114, #132, and #143 w.r.t flightsize vs. cwnd. The current text
>            >             does not quite correcly reflect what stacks that use cwnd do.
>            >             I'll comment in #143 separately.
>            >
>            >       #96 & #98: The text added does not address the problems raised which
>            are
>            >       also evidenced in the paper pointed by Bob in #96. Even though CUBIC
>            has
>            >       been modified a bit after the paper was published, it does not
>            >       automatically mean that the problem has been shown resolved:
>            experimental
>            >       evidence is required but not provided. The fact that CUBIC does not
>            >       change MD factor for fast convergence is the root of the problem
>            >       evidenced in the paper and remains so in the algo specified in this
>            >       draft. This is also a significant problem when competing with Reno CC
>            >       because CUBIC behaves much more aggressive than Reno CC when there is
>            >       sudden congestion and all competing flows must converge fast down to a
>            >       small fraction of the current cwnd to be fair to each other. This
>            again
>            >       cannot be evidenced not to be a problem by long deployment experience
>            >       unless experimental data that measures the impact on competing traffic
>            is
>            >       presented to back the claims. Adjusting just Wmax for fast convergence
>            is
>            >       not enough and is even likely to be ineffective because there tend to
>            be
>            >       several losses when sudden congestion is hit, and particularly if
>            NewReno
>            >       is in use the sender stays several RTTs in Fast Recovery being
>            >       overaggressive and then possibly continues at the same rate in CA
>            which
>            >       is unlikely to reach evan close to Wmax before a new loss hits
>            >       the sender again. That is, lower Wmax and lower additive increase
>            factor
>            >       do not compensate the use of larger MD factor when sudden congestion
>            is
>            >       encountered.
>            >
>            >       #93 & #94 & (#89) Sec 5.3 still does not address any difficult
>            >       environments, in particular buffer-bloated paths (nor does Sec 5.4).
>            >       We need evidence (results) that show CUBIC is fair towards other CCs
>            >       (Reno) also in such environments. Note that CUBIC's decision to leave
>            >       Reno-friendly region is based on the size of cwnd which tends to be
>            >       incorrect with buffer-bloated bottlenecks because with huge buffers
>            the
>            >       cwnd can be many times larger than what is actually needed to fully
>            >       utilize the available network bit-rate. Therefore, Reno CC has no
>            problem
>            >       in fully utilizing such bottleneck links and CUBIC must stay in
>            >       Reno-friendly region longer but it leaves it too early because the
>            same C
>            >       is used as with non-bloated environments. We lack experiments showing
>            >       CUBIC follows the congestion control principles and is fair to current
>            >       standard TCP CC; to my understanding no experiments with
>            buffer-bloated
>            >       bottlenecks are cited to back up the claims even though buffer bloat
>            is
>            >       very well known to be a common (difficult) environment in today's
>            >       Internet.
>            >
>            >       #90 The current text on applying undo (a response to detected false
>            >       fastrexmit) does not provide correct result if someone implements it.
>            >       I have explained the problems there in github but seem to have not
>            >       replied to latest comments by Neal. I'll reply and try to explain
>            more.
>            >       Again one major problem here is that the draft suggest a new response
>            >       algo for false fast rexmits but does not provide any experimental data
>            to
>            >       support it. Long deployment experience has been suggested as
>            >       justification but again without any carefully evaluated experimental
>            >       data and evidence there is no meat. The issue is important to solve
>            but is
>            >       not specific to CUBIC. Instead, it is general problem for all TCP CC
>            >       variants. IMHO, this is not ready for standards track but deserves a
>            draft
>            >       of its own so that it can be carefully evaluated and discussed ion the
>            >       list. AFAIK there has been no discussion on this on the tcpm list, so
>            >       those probably interested and having experience are likely to be
>            unaware
>            >       that this is part of CUBIC draft.
>            >
>            >       #88 The problem with correctness of the AIMD model and setting alpha
>            for
>            >       CUBIC requires further consideration. Bob provided an analysis that
>            >       leaves things still open. It seems that I never had time to review and
>            >       comment the analysis and clarify why the model does not work. I'll do
>            that
>            >       separately as it is important to ensure CUBIC behaves fairly as
>            intended
>            >       for the Reno-friendly region.
>            >
>            >       Best regards,
>            >
>            >       /Markku
>            >
>            >       On Mon, 31 Jan 2022, Yoshifumi Nishida wrote:
>            >
>            >       > Hello,
>            >       >
>            >       > After some discussions among chairs, we decided to run the 2nd WGLC
>            on
>            >       draft-ietf-tcpm-rfc8312bis in
>            >       > consideration of the importance of the draft. 
>            >       > We'll be grateful if you could send your feedback to the ML. The
>            WGLC runs
>            >       until *Feb 11*.
>            >       >
>            >       > If interested, you can check in-depth past discussions in the
>            following URL.
>            >       > https://github.com/NTAP/rfc8312bis/
>            >       >
>            >       > Thank you so much!
>            >       > --
>            >       > tcpm co-chairs
>            >       >
>            >       >
>            >       > On Wed, Jan 26, 2022 at 2:50 AM Lars Eggert <[email-masked]> wrote:
>            >       >       Hi,
>            >       >
>            >       >       this -06 version rolls in all the changes requested during
>            (and after)
>            >       WGLC ended.
>            >       >
>            >       >       I'll leave it up to the chairs to decide if another WGLC is
>            warranted
>            >       or the document can
>            >       >       progress as-is.
>            >       >
>            >       >       Thanks,
>            >       >       Lars
>            >       >
>            >       >
>            >       >       > On 2022-1-26, at 11:12, [email-masked] wrote:
>            >       >       >
>            >       >       >
>            >       >       > A New Internet-Draft is available from the on-line
>            Internet-Drafts
>            >       directories.
>            >       >       > This draft is a work item of the TCP Maintenance and Minor
>            >       Extensions WG of the IETF.
>            >       >       >
>            >       >       >        Title           : CUBIC for Fast and Long-Distance
>            Networks
>            >       >       >        Authors         : Lisong Xu
>            >       >       >                          Sangtae Ha
>            >       >       >                          Injong Rhee
>            >       >       >                          Vidhi Goel
>            >       >       >                          Lars Eggert
>            >       >       >       Filename        : draft-ietf-tcpm-rfc8312bis-06.txt
>            >       >       >       Pages           : 35
>            >       >       >       Date            : 2022-01-26
>            >       >       >
>            >       >       > Abstract:
>            >       >       >   CUBIC is a standard TCP congestion control algorithm that
>            uses a
>            >       >       >   cubic function instead of a linear congestion window
>            increase
>            >       >       >   function to improve scalability and stability over fast
>            and long-
>            >       >       >   distance networks.  CUBIC has been adopted as the default
>            TCP
>            >       >       >   congestion control algorithm by the Linux, Windows, and
>            Apple
>            >       stacks.
>            >       >       >
>            >       >       >   This document updates the specification of CUBIC to
>            include
>            >       >       >   algorithmic improvements based on these implementations
>            and recent
>            >       >       >   academic work.  Based on the extensive deployment
>            experience with
>            >       >       >   CUBIC, it also moves the specification to the Standards
>            Track,
>            >       >       >   obsoleting RFC 8312.  This also requires updating RFC
>            5681, to
>            >       allow
>            >       >       >   for CUBIC's occasionally more aggressive sending behavior.
>            >       >       >
>            >       >       >
>            >       >       > The IETF datatracker status page for this draft is:
>            >       >       > https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc8312bis/
>            >       >       >
>            >       >       > There is also an HTML version available at:
>            >       >       >
>            >       >       >
>            >       >       > A diff from the previous version is available at:
>            >       >       >
>            >       >       >
>            >       >       >
>            >       >       > Internet-Drafts are also available by rsync at
>            >       rsync.ietf.org::internet-drafts
>            >       >       >
>            >       >       >
>            >       >       > _______________________________________________
>            >       >       > tcpm mailing list
>            >       >       > [email-masked]
>            >       >       > https://www.ietf.org/mailman/listinfo/tcpm
>            >       >
>            >       >       _______________________________________________
>            >       >       tcpm mailing list
>            >       >       [email-masked]
>            >       >       https://www.ietf.org/mailman/listinfo/tcpm
>            >       >
>            >       >
>            >       >
>            >
>            >
>            >
>

---

From: Markku Kojo <[email-masked]>
Date: Tue, 22 Mar 2022 21:25:16 +0200 (EET)

Hi Bob, Yoshi, all,

I'm afraid Bobs statements about the process do not match what RFC 2119 
and RFC 8174 state about using the keywords and what is normative text.
Bob also seem to have missed the crucial role of RFC 5681 that has been 
intentionally and clearly set in RFC 2914 and RFC 5033.

In addition, I'd like to clarify my points about fairness that Bob seems 
to have interpreted incorrectly.

See inline below.

On Tue, 22 Feb 2022, Bob Briscoe wrote:

> Yoshi, Markku,
> 
> Many of the technical points Markku raised were important omissions from the rfc8312bis draft.
> Whatever status we give the eventual RFC, they should be highlighted as areas that need
> further work, either known weaknesses or just areas where we don't have enough data to know
> whether they are weaknesses. I tried to help find useful references for experiments already
> done around those points, and I tried to help write such text.
> 
> However, I cannot agree with some of Markku's statements about process.
>
>   #1: The rules can change
> 
> It's OK to quote RFCs, but it's also OK for us to reach consensus that certain aspects of old
> RFCs are outdated, as long as we give our reasons, and there is rough consensus around that.
> We are all consenting adults - we don't have to be limited by RFCs from the past if we
> collectively don't want to be.

I fully agree. I'd also like to note that if we want to divert from what 
has currently been documented we need to do that in a controlled manner, 
not by publishing an RFC that creates confusion but have a thorough 
discussion and modify these BCPs in those part it is seen appropriate.

> I have not noticed a need for the above "rule #1" in this discussion yet, but I'm just putting
> it out there, so if anyone needs to invoke it later, they can't be accused of inventing a rule
> for their own convenience.

Agreed.

>   #2: There is no TCP-compatibility requirement
> 
> Yes, CUBIC is not TCP-compatible. But that's a feature, not a bug. Words like 'TCP-compatible'
> or 'fair' sound nice, but do not be fooled into thinking that an RFC recommends or requires
> you to behave in a certain way just because it has used a nice-sounding word for it.

RFC 2914, RFC 5033, and RFC 5681 are all in line and compatible with 
each other and that is intentianal, and that is the current consensus 
until we decide to change it. See below that this is also currently normative.

> For an RFC to recommend or require something, it has to say SHOULD or MUST. So when BCP 41
> [RFC2914] defines 'TCP-compatible', and talks about what is necessary to be 'fair', it implies
> nothing about whether you should or must be TCP-compatible, or fair; unless it actually says
> you SHOULD or MUST be TCP-compatible, or fair. No RFC says that.

My long term understanding as well as reading of RFC 2119 and RFC 8174 
gives very different view.

RFC 2119 (the one that was valid at the time RFC 2914 was published) says:

  "These words are often capitalized."

And continues:

                                  Authors who follow these guidelines
  should incorporate this phrase near the beginning of their document:

       The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
       NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
       "OPTIONAL" in this document are to be interpreted as described in
       RFC 2119.

RFC 2914 and RFC 5033 do not include the above phrase, so the text in 
these RFCs is all *normative* without using the keywords "MUST" or 
"SHOULD". This is further clarified in RFC 8174:

  These words can be used as defined here, but using them is not
  required. Specifically, normative text does not require the use
  of these key words. They are used for clarity and consistency
  when that is what's wanted, but a lot of normative text does not
  use them and is still normative.

There are plenty of RFCs that do not use RFC 2119 language and the text is 
fully normative, including RFC 2914 and RFC 5033.

> BCP 133 [RFC5033] and others have written careful words about straying from equal flow rates
> (quoted below). That gives a checklist for people like us who come along later. But it also
> respects our right to make a decision on whether/how to publish a CC RFC based on our own
> present-day knowledge and experience. To quote the relevant guideline in BCP 133 [RFC5033]:
>
>    (1) Impact on Standard TCP, SCTP [RFC2960], and DCCP [RFC4340].
>
>        Proposed congestion control mechanisms should be evaluated when
>        competing with standard IETF congestion control [RFC2581,
>        RFC2960, RFC4340].  Alternate congestion controllers that have a
>        significantly negative impact on traffic using standard
>        congestion control may be suspect and this aspect should be part
>        of the community's decision making with regards to the
>        suitability of the alternate congestion control mechanism.
>
>        We note that this bullet is not a requirement for strict TCP-
>        friendliness as a prerequisite for an alternate congestion
>        control mechanism to advance to Experimental.  As an example,
>        HighSpeed TCP is a congestion control mechanism that is
>        Experimental, but that is not TCP-friendly in all environments.
>        We also note that this guideline does not constrain the fairness
>        offered for non-best-effort traffic.
>
>        As an example from an Experimental RFC, fairness with standard
>        TCP is discussed in Sections 4 and 6 of [RFC3649] (HighSpeed TCP)
>        and using spare capacity is discussed in Sections 6, 11.1, and 12
>        of [RFC3649].
> The rfc8312bis draft has a section on 'using spare capacity' precisely because that's what was
> considered acceptable about CUBIC and the new generation of hi-BDP CCs developed at that time,
> as I've emphasized in the last para above.

Yes, 'using spare capacity' is all fine, if it is shown to be the case. 
However, providing data and showing that capacity is not stolen from Reno 
CC when Reno CC is capable of fully using the available capacity is 
exactly the concern I have raised. This section 5.2 in rfc8312bis draft is 
not convincing at all. It cites just one paper [HKLRX06] and at least I 
have very hard time to find it providing any evidence that "indicate that 
CUBIC uses the spare bandwidth left unused by existing Reno TCP flows". 
[HKLRX06] focuses on comparing various high-speed TCP alternatives, not 
specifically comparing Reno CC and CUBIC. It has Fig 2 that shows one 
Reno CC (SACK) competing with CUBIC. The results in the figure are hard 
to interpret as distinguishing some of the graphs is quite impossible. 
However, the results seem to show that the link utilization with two SACK 
flows is roughly at the same level or even higher than the link 
utilization with SACK and CUBIC, that is, CUBIC does not even improve 
link utilization when RTT (and BDP) get higher?
In Fig 7., for example at 40ms RTT, where SACK is able to fully utilize 
the available capacity, the fairness index for CUBIC is not really 
supporting the sentence above either. So, it is hard to see how the claim
above could be backed up by data in [HKLRX06]. Maybe I'm missing 
something? 
And, definitely we are lacking results in a wide range of environments, 
just one set of network parameters are used in those tests (only RTT is 
varied).

In addition, as I have pointed out, in [HRX08] Figs 10a and 10c 
clearly demonstrate that CUBIC steals capacity from SACK (Reno) CC.

>   #3: A process for publishing EXP RFCs doesn't cover publishing non-EXP RFCs (by
>   definition)
> 
> This IESG statement on "Experimental Specification of New Congestion Control Algorithms" says
> it defines the process for a CC to be published as an Experimental RFC. It does not say it is
> the process that /all/ CCs have to follow to enter the standards track.
> 
> Before invoking that process, one has to determine whether it is applicable to this case. The
> IESG statement contains a sentence about how to determine its applicability:
>       The decision of whether a specific new congestion control schemes goes beyond the
>       principles of [RFC2914] and hence will undergo the process described in this
>       document lies with the transport area directors, who will use the expertise of the
>       transport area directorate and other congestion control experts to aid their
>       decision.
> 
> Since RFC2914, we've had RFC3246 explaining why Reno is unscalable, and RFC5033 (quoted above)
> giving guidelines for designers of new CCs. So we can debate whether the process that Markku
> quoted is relevant to this case, to help the AD to decide whether to follow the process or
> not.

The IESG statement on "Experimental Specification of New Congestion 
Control Algorithms" is fully inline with RFC 5033 which, in turn, is 
consistent with RFC 2914. It seems that Bob and many others are missing 
(have forgotten?) the long-term policy and consensus that the IETF has 
followed when new/alternate CC proposals have been introduced to the 
IETF; all such proposals are first given experimental status. This is 
also very clearly documented in RFC 5033:

   "Following the lead of HighSpeed TCP [RFC3649], alternate congestion
    control algorithms are expected to be published as "Experimental"
    RFCs until such time that the community better understands the
    solution space."

Bob is all correct when pointing out the problems with Reno CC in high 
BDP environments and what is documented in RFC 3649. RFC 5033 also 
acknowledges this and points to RFC 3649 on how to evaluate such 
proposals. RFC 3649 itself is experimental and it does not claim it is 
ready but recognizes that more work in evaluating it is needed, 
particularly in areas other than the steady-state behavior that RFC 3649 
focuses on. E.g., RFC 3649 points out a crucial test to evaluate: whether 
an alternative CC uses just the spare capacity of Reno CC or does it 
steal capacity from Reno CC when Reno CC (alone) is able to utilize the 
available capacity.

Moreover, as Bob pointed RFC 5033 gives quite clear guidelines and 
criteria to evaluate the alternate CCs and points to RFC 3649 for 
evaluating "high-speed" variants like CUBIC. If we compare the level of 
evaluation in RFC 3649 and rfc8312bis draft, the latter is way behind the 
level of justification and evaluation even though RFC 3649 is just 
experimental.

>   #4 EXP or STD? My view
> 
> Reno is worse than other CCs in some respects. Nearly everyone has abandoned it. So, if
> someone asked you why Reno is the IETF's only stds track CC, what would you say ...?

Has anyone ever thought of any alternative reasons why Reno CC has been 
largely abandoned? Might it be possible that the widely deployed CUBIC 
(and maybe some other alternatives) is much more aggressive than Reno CC 
particularly in environments where Reno CC has no problems in utilizing 
the available bandwidth and thereby CUBIC throttles competing Reno CC 
performance such that nobody finds any variant of Reno CC competitive 
enough anymore?

Note that for the global average access speeds Reno CC has hardly any 
problems in fully utilizing the available capacity, particularly when the 
queue sizes still today tend to be relatively large, if not 
buffer-bloated. There are high BDP environments, for sure, where we have 
a need for CCs that better use the avaiable capacity but such CCs still 
need to compete fairly with Reno CC in lower BDP environments. Otherwise, 
we end up exactly to the "arms race" seriously warned in RFC 2914 and 
reiterated in RFC 5033.

> As I already said, we have to stop thinking that the stds track requires a CC algorithm to be
> perfect. A CC certainly mustn't have failure or collapse modes, but it's performance doesn't
> have to be perfect in all cases.

Sure, Reno CC is not perfect, nor is CUBIC.

> For instance, CUBIC has known poor utilization over radio
> links. That didn't stop Reno from being on the stds track. For me, the IETF needs a new CC on
> the stds track, and CUBIC is the most natural candidate.
> 
> However, we must not set a precedent for any old CC to move to the stds track. So we must
> document it's weaknesses and failings, and explain why they are not serious (if that is the
> case).

Fully agree.

> New CCs need to interoperate with the most prevalent CC on the Internet (CUBIC). The IETF
> cannot even say that as long as it hasn't formally recognized CUBIC on the stds track. That is
> a failing of *us* in the tcpm WG. We can move CUBIC to stds track via the exp track if we have
> to, or we can go straight to stds track. The only difference is that the first option makes us
> look like jobsworths for longer.

What we need to understand is that whichever CC we accept to the stds 
track, we need to have a very good understanding of its performance, 
particularly when competing with the current stds track CC that has a lot 
of dependences in the RFC series. We must not create confusion by having 
a CC on stds track that deviates from that of the current stds track CC 
but we don't know why, when and how much and thereby we are not able to 
document and justify it clearly at the time of publishing such an 
alternate CC. That's why we have created and agreed on the process how to 
bring alternative CCs to the IETF such that they should undergo a thorough 
evaluation.

Thanks,

/Markku

> jobsworth
> /ˈdʒɒbzwəːθ/
> noun  derogatory•informal
> an official who upholds petty rules even at the expense of humanity or common sense.
> 1970s: from "I can't do that, it's more than my job's worth."
> 
> 
> Bob
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
> 
>

---

From: Markku Kojo <[email-masked]>
Date: Wed, 23 Mar 2022 13:14:34 +0200 (EET)

Lars,

On Mon, 21 Feb 2022, Lars Eggert wrote:

> Hi,
>
> first, as an editor (but not wearing other hats) I do object
> against reclassifying the document as Experimental or Informational,
> esp. this late in the process.
>
> CUBIC is not a new CC proposal. It's been ubiquitously deployed
> on the Internet for many years. All major TCP stacks use it by default.

Maybe I should not have pointed to the IESG statement but RFC 5033.
The IESG statement describes the IETF process while RFC 5033 provides 
the guidelines for the community, and the IESG statement explicitly points 
to (the draft of) RFC 5033. RFC 5033 is crystal clear on what kind of 
evaluation is expected from an alternate congestion control proposal to 
the IETF. "New" is borrowed from RFC5033 where it means "alternate", not 
how old the algo is or some such, and also in this context it means a new 
proposal to the IETF, i.e., an I-D submitted to the IETF.

> Given that, I don't see how the IESG statement on "Experimental 
> Specification of New Congestion Control Algorithms" applies here.
> (As an aside, that statement was specifically written when CUBIC,
> Compound and Hamilton were all jostling for RFC publication. The
> community converged on CUBIC in the meantime.)
>
> CUBIC *is* the default CC mechanism used on the Internet and in
> other IP networks. It should be published on the standards track.
>

Careful reading and understanding of RFC 5033 would be useful. RFC 5033 
and the IESG statement were written exactly to avoid a situation where 
any such alternate CC algo gets widely deployed in the global Internet 
without being first thoroughly evaluated by the IETF. The process was 
seen important to ensure thorough enough evaluation of any alternate CC 
before its deployment in the Internet.

Could you possibly explain more of your rationale when saying "don't 
see how the IESG statement applies here"?
The fact seems to be that CUBIC has taken the wrong path and has been 
deployed without any IETF evaluation. This is exactly opposite to the 
consensus that the IETF/TSV community had and what has been documented in 
RFC 5033 and the IESG statement. I'm sorry to say that it seems 
incomprehensible to me that now the (long-term) deployment in the 
Internet which was unwanted by the IETF is used in the IETF as rationale 
why CUBIC does not need any IETF evaluation. In particular, when CUBIC is 
targetted as PS which is supposed to have even higher bar than when 
targetting at Experimental.

Furthermore, despite the extensive deployment experience with CUBIC, the 
rfc8312bis draft does not cite any results comparing CUBIC vs. Reno 
that were gathered and publishhed since CUBIC was deployed. Where is the 
the extensive deployment experience documented? That is, could you 
possbly explain on what information the tcpm wg can base its decision 
whether rfc8312bis fullfils the criteria set in RFC 5033?

> Second, this is the second time that an individual raised a long
> list of issues after the WGLC has ended. We did try to work through
> all the issues that were raised after the first WGLC, but the
> individual often failed to participate in the discussion in
> a timely manner. I now see at least several of those same
> issues re-raised by the same individual.

Lars, I would have hoped that the decision on this draft and the 
remaining issues were based on arguments on the actual issues.

When publicly accusing an individual on misbehaviour it would be good to 
have the facts correct. The 2nd last call was announced by a tcpm wg 
chair: "The WGLC runs until *Feb 11*."
Below please find the timestamp as seen by ietfa.amsl.com for the 2nd wglc 
message by the individual:

  Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com
  (Postfix) with ESMTP id 7C1B53A1371; Fri, 11 Feb 2022 21:34:42 -0800
  (PST)

I have always thought that IETF is an international organization and any 
deadlines set with no specific timezone are automatically "anywhere on the 
earth" regardless of the timezone of the contributor's mail system that 
she/he uses for sending messages to the IETF lists. Could you kindly 
explain if this is incorrect such that any individual may avoid such 
a mistake in the future?

The individual raised the first list of issues after the first WGLC 
had ended with only a *single* review (thanks Yoshi) and the tcpm chairs 
explicitly asked if anyone has any thoughts on the doc. The individual 
apologised for the late list of issues and explained that the draft had 
notable conflicts with existing RFCs, which was the major reason for the 
individual to react at that late point. While checking the draft for 
for the conflicts, the individual listed also other issues that seemed to 
require more work. Could you kindly explain how this was wrong thing to 
do?

I think I can also speak for the individual on not always participating 
in the discussions on a timely manner. That is true and the individual 
also informed several times that the individual had very limited cycles 
for the IETF work. To begin with the individual also indicated that many 
issues may deserve discussion on the wg list. The editor responded:

> true, and we will bring those to the list that should be discussed here.

However, many issues were closed in github even though they were 
clearly still unresolved. Sometimes also without confirming 
with the individual.

More importantly and given that this is a wg document, would't it have 
been a viable option to take any unresolved issues to the wg list 
where the wider expertice is instead of closing them in github? By this 
far no issues has been taken to the list by the editor.

> Before going down
> the same path a second time, I would actually ask the WG to
> confirm there is consensus that the raised issues are valid
> (and if they are valid, commit to contributing to a resolution
> in a timely manner.)

First, the issues listed for the 2nd WGLC were not re-raised but they 
were more or less unresolved: no consensus has been decleared in the 
discussions on those issues.

Second, definitely these issues should be discussed on the wg list. We are 
about making very significant decisions, not only about this draft, but in 
its current form it has implications to several stds track RFCs via RFC 
5681. One important issue that has not been discussed much at all is 
whether this draft needs to update RFC 5681, and if it does, we need to 
explain and justify it carefully. The current text leaves many questions 
open on what "Updates" here actually means.

Thanks,

/Markku