Subject: [tcpm] Adoption of draft-zimmermann-tcpm-cubic?

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 18 Mar 2015 10:03:14 +0000

Michael, Yoshi, Pasi,

after the recharter of TCPM, which now specifically states " TCPM may document alternative TCP congestion control algorithms
that are known to be widely deployed", we would like to raise the question if we can start a call for adoption of https://tools.ietf.org/id/draft-zimmermann-tcpm-cubic-00.txt ? During the IETF91, there was strong consensus in the room to adopt this document and put it on standards track: https://tools.ietf.org/wg/tcpm/minutes?item=minutes-91-tcpm.html

>From our understanding, Cubic fully qualifies to be considered as a legitimate WG document after the recharter. Again, no technical work on the cubic algorithm would be done, as explained by Lars in Honolulu, just tracking what changes went into the existing implementations since the initial writing of the document. 

Best regards,
  Alex and Richard

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of IETF Secretariat
> Sent: Dienstag, 17. Februar 2015 20:10
> To: [email-masked]
> Subject: [tcpm] Milestones changed for tcpm WG
> 
> Changed milestone "Submit document obsoleting undeployed TCP extensions to
> the IESG for publication as an Informational RFC", set state to active
> from review, accepting new milestone.
> 
> URL: http://datatracker.ietf.org/wg/tcpm/charter/
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 18 Mar 2015 10:18:37 +0000

My understanding is that TCPM recharting is still pending and IESG waits for potential feedback until March 23 [1].

The chairs are well aware of the strong support for draft-zimmermann-tcpm-cubic during IETF 91. We plan to run a formal adoption call once our charter explicitly allows adoption in TCPM.

Michael

[1] http://www.ietf.org/mail-archive/web/tcpm/current/msg09489.html

> From: tcpm [mailto:[email-masked]] On Behalf Of Scheffenegger,
> Richard
> Sent: Wednesday, March 18, 2015 11:03 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [tcpm] Adoption of draft-zimmermann-tcpm-cubic?
> 
> Michael, Yoshi, Pasi,
> 
> 
> after the recharter of TCPM, which now specifically states " TCPM may
> document alternative TCP congestion control algorithms
> that are known to be widely deployed", we would like to raise the
> question if we can start a call for adoption of
> https://tools.ietf.org/id/draft-zimmermann-tcpm-cubic-00.txt ? During
> the IETF91, there was strong consensus in the room to adopt this
> document and put it on standards track:
> https://tools.ietf.org/wg/tcpm/minutes?item=minutes-91-tcpm.html
> 
> >From our understanding, Cubic fully qualifies to be considered as a
> legitimate WG document after the recharter. Again, no technical work on
> the cubic algorithm would be done, as explained by Lars in Honolulu,
> just tracking what changes went into the existing implementations since
> the initial writing of the document.
> 
> Best regards,
>   Alex and Richard
> 
>

---

From: "Zimmermann, Alexander" <[email-masked]>
Date: Wed, 18 Mar 2015 10:26:20 +0000

Hi Michael,

> Am 18.03.2015 um 11:18 schrieb Scharf, Michael (Michael) <[email-masked]>:
> 

Sounds good!

Thanks for update

Alex

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 18 Mar 2015 21:20:54 -0400

On Wed, Mar 18, 2015 at 6:26 AM, Zimmermann, Alexander
<[email-masked]> wrote:
>> The chairs are well aware of the strong support for draft-zimmermann-tcpm-cubic during IETF 91.
>> We plan to run a formal adoption call once our charter explicitly allows adoption in TCPM.

Regarding draft-zimmermann-tcpm-cubic-00:

  https://tools.ietf.org/id/draft-zimmermann-tcpm-cubic-00.txt

I support this draft.

But I did want to note that our experience with CUBIC at Google shows
that (1) it's important to consider stretch ACKs, and (2) it's critical to
carefully consider the maximum rate of cwnd increase for CUBIC
in congestion avoidance.

I would suggest the following small diffs to the draft:

Where the draft says (in sections "3.3. Concave region" and "3.4.
Convex region"):

  In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
  for each received ACK.

I'd suggest the following new text in these two spots:

  In this region, cwnd MUST be incremented by
     min((W(t+RTT) - cwnd)/cwnd, 1/2)
  for each newly-acknowledged packet.

Detailed rationale:

(1) First is the issue of the treatment of "stretch ACKs" covering
more than 2 packets.  The CUBIC paper and the Linux code through v3.18
cap the de facto maximum rate of cwnd increase (in congestion
avoidance) at 1 packet for every alternate ACK. In Google's experience
with high-BDP paths with receiver hosts using receive
offload/aggregation mechanisms (LRO and GRO in Linux terminology), the
ACKs for up to 40 or more packets at a time caused this cap of "1
packet for every alternate ACK" to result in cwnd increases that were
too slow, leading to underutilization. So our team at Google
contributed some changes in v3.19 that changed the rate of increase to
be expressed in terms of packets ACKed, rather than number of ACKs.
This approach allows full utilization even with receiver offload
mechanisms.

(2) Second is the issue of the maximum rate of increase of cwnd. As it
stands, this document says in section "3.4. Convex region" that:

   In this region, cwnd MUST be incremented by (W(t+RTT) - cwnd)/cwnd
   for each received ACK.

There is similar language in section "3.3. Concave region".

Since the rate of increase of a cubic function can be quite large, if
we take this language literally then in steep sections of the curve
this can require that the sender "MUST" increase cwnd by quite
a large amount.

I am not sure what other CUBIC implementations do, but this does not
match the CUBIC paper or the Linux implementation. As mentioned above,
the CUBIC paper and the Linux code through v3.18 actually cap the de
facto maximum rate of cwnd increase (in congestion avoidance) at 1
packet for every alternate ACK.

In our recent experiments with using CUBIC for YouTube video traffic
using the revised, stretch-ACK-savvy CUBIC logic in Linux v3.19 we saw
retransmit rates double due to v3.19 allowing cwnd to increase by 1
packet for every packet ACKed.

So in Linux v4.0 we set a limit of increasing cwnd by at most 1 packet
for every 2 packets ACKed, which restored the retransmit rates to
their previous low levels.

For reference:

(1) Commits in Linux v3.19 to make CUBIC stretch-ACK-savvy:

  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=e73ebb0881ea5534ce606c1d71b4ac44db5c6930
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=814d488c61260521b1b3cc97063700a5a6667c8f
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=9cd981dcf174d26805a032aefa791436da709bee
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d6b1a8a92a1417f8859a6937d2e6ffe2dfab4e6d

(2) Commits slated for Linux v4.0 to fix excessive cwnd growth in CUBIC:

  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=9949afa42be0b76f5832db112ce51bb6b35b2abb
  http://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d578e18ce93f5d33a7120fd57c453e22a4c0fc37

neal

---

From: "Zimmermann, Alexander" <[email-masked]>
Date: Thu, 19 Mar 2015 16:01:29 +0000

Hi Neal,

thank you so some much for this email. Especially for the summary of the
changes you folks did at Google!
See inline.

> Am 19.03.2015 um 02:20 schrieb Neal Cardwell <[email-masked]>:
> 

Ack. If you take a look at my CUBIC presentation at the last IETF
(https://tools.ietf.org/agenda/91/slides/slides-91-tcpm-3.pdf),
one goal of this exercise will be to incorporate all changes with made
to the Linux code so far. You save me a lot of time to scan the code
for changes and compare it again w/ the draft. Thanks!

> This approach allows full utilization even with receiver offload

We may also then recommend ABC for CUBIC when byte based stacks are
used…