Subject: [iccrg] Added DCTCP to the pantheon of congestion controls

From: Bob Briscoe <[email-masked]>
Date: Sat, 23 Jul 2016 12:46:36 +0100

Keith,

In the iccrg this week, Koen asked about adding DCTCP to the pantheon, 
and you said it's straightforward, but...

*1/ Not just sender algo**
*...Does the structure of the pantheon make it easier to only change the 
sender algorithm? Or is it just as easy if you want to make changes to 
the receiver, the network and to the (feedback aspect of the) TCP wire 
protocol? I know Remy included assessments of XCP and sfq_CoDel, so I 
assume all changes are possible, but is the ability to plug-in different 
contenders intended more just for the sender algorithms?

Aside: I guess that once one starts changing more than the sender, the 
throughput/delay tradeoff becomes secondary to other tradeoffs like 
incremental deployability.

*2/ Wider tradeoffs*
Deployability reminds me of a second point: We've realised TCP does not 
solely cause a dilemma between delay and throughput / utilization. TCP 
creates a hexlemma (at least). Researchers can make their TCP look good 
by reducing one impairment at the expense of another - TCP is like a 
balloon. For instance, you can reduce queuing delay, but increase loss. 
More loss causes more delay when measuring from the (short) flow 
completion time perspective, but this delay is invisible if one measures 
from an RTT-delay perspective.

See slide 6 of 
https://www.ietf.org/proceedings/96/slides/slides-96-l4s-0.pdf

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Keith Winstein <[email-masked]>
Date: Sat, 23 Jul 2016 11:46:16 -0700

Hello Bob,

Thanks for your email. The question was about the absence of ECN in the
emulated network and DCTCP among the evaluated schemes. What I said is that
it's an open-source project and that we would accept a pull request to add
a scheme, or add an emulated network scenario. (We would also accept a pull
request to add a figure of merit for evaluation.)

I don't remember opining on whether adding DCTCP would be straightforward,
and here's why. There are roughly three parts to the project:

a) A library of wrappers around existing congestion-control schemes. Each
wrapper includes a submodule reference to where the source code for the
scheme lives, and a Python file that exposes a way to build the scheme or
run a full-throttle "sender" or corresponding "receiver."
https://github.com/StanfordLPNG/pantheon/tree/master/src .

For DCTCP: Adding DCTCP would mean adding a submodule reference to a
DCTCP-like scheme and a Python wrapper that knows how to built it and run
it. If there's one in userspace, great, or if there's a Linux kernel module
you can build and then load, also great. (This is how we test cubic and
vegas.) But otherwise you may have to implement one and that's a pain. The
community would certainly benefit from somebody making the effort here.

2) A library of integration tests that run in the cloud on the Travis-CI
system, over emulated networks done with the Mahimahi containers (mm-delay
/ mm-loss / mm-link). The emulation tests produce a logfile, and scripts
turn that into a PDF report comparing all the schemes on the metrics of
throughput and signalling delay. This is the part that started producing a
PDF about 10 minutes before my talk. Right now it runs a *single* scenario
-- a particular bufferbloated cellular link, running a single full-throttle
flow from each scheme by itself -- but we do plan to add more scenarios.

For ECN: Mahimahi does support pluggable queue disciplines (the constrained
link container, mm-link, supports DropTail and DropHead with various
thresholds in bytes and packet count), but it doesn't currently ship an
implementation of RED or of a scheme that marks instead of drops. So a pull
request would have to add this. We would certainly take it.

For other figures of merit (e.g. flow completion time): I was at your L4S
talk and agree with your remarks. Evaluating the impact of packet loss on
an application-level metric, like flow completion time, will take some
effort to design the right interface so it can be measured fairly across
schemes. For example, packet loss may not affect FCT that much if the
endpoints use a rateless code to recover from loss. But it might affect it
a lot if the endpoints are relying on retransmissions.

This work should get done, but it needs somebody to put some effort into it
and have each scheme expose an interface that the evaluation code can use
to measure a quantity like FCT. (E.g. the interface might have to be
something like, transfer this file.) When we're dealing with 10+ schemes
with different codebases, it was much easier to start out with a common
denominator that is simply "a full-throttle sender of an unbounded stream."

To be clear, when you say "you can reduce queuing delay, but increase
loss," this is true if "you" is somebody who can influence behavior on the
bottleneck link. I'm not too worried about this in our evaluation of
endpoint protocols (run over emulated or actual cellular links) -- an
endpoint protocol that is able to achieve low queueing delay is probably
not going to be one that also causes a lot of loss.

3) A weekly-ish "Network Observatory" of real-world tests where the
scripted scenarios and Pantheon schemes are run over actual cellular
networks around the world, and we report the performance. Here I would say,
we'd love if ECN were deployed in these networks so we could practically
run DCTCP, but it's probably outside our ability to influence.

Best regards,
Keith

On Sat, Jul 23, 2016 at 4:46 AM, Bob Briscoe <[email-masked]>
wrote:

> Keith,
>
> In the iccrg this week, Koen asked about adding DCTCP to the pantheon, and
> you said it's straightforward, but...
>
> *1/ Not just sender algo*
> ...Does the structure of the pantheon make it easier to only change the
> sender algorithm? Or is it just as easy if you want to make changes to the
> receiver, the network and to the (feedback aspect of the) TCP wire
> creates a hexlemma (at least). Researchers can make their TCP look good by
> reducing one impairment at the expense of another - TCP is like a balloon.
> For instance, you can reduce queuing delay, but increase loss. More loss
> causes more delay when measuring from the (short) flow completion time
> perspective, but this delay is invisible if one measures from an RTT-delay
> perspective.
>
> See slide 6 of

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 24 Jul 2016 18:48:43 +0100

Keith,

Thx for the detailed response. Understood.
A couple of responses inline...

On 23/07/16 19:46, Keith Winstein wrote:
> Hello Bob,
>
> Thanks for your email. The question was about the absence of ECN in 
> the emulated network and DCTCP among the evaluated schemes. What I 
> said is that it's an open-source project and that we would accept a 
> pull request to add a scheme, or add an emulated network scenario. (We 
> would also accept a pull request to add a figure of merit for evaluation.)
>
> I don't remember opining on whether adding DCTCP would be 
> straightforward, and here's why. There are roughly three parts to the 
> project:
>
> a) A library of wrappers around existing congestion-control schemes. 
> Each wrapper includes a submodule reference to where the source code 
> for the scheme lives, and a Python file that exposes a way to build 
> the scheme or run a full-throttle "sender" or corresponding 
> "receiver." https://github.com/StanfordLPNG/pantheon/tree/master/src .
>
> DCTCP-like scheme and a Python wrapper that knows how to built it and 
> run it. If there's one in userspace, great, or if there's a Linux 
> kernel module you can build and then load, also great. (This is how we 
> test cubic and vegas.) But otherwise you may have to implement one and 
> that's a pain. The community would certainly benefit from somebody 
> making the effort here.
>
> 2) A library of integration tests that run in the cloud on the 
> Travis-CI system, over emulated networks done with the Mahimahi 
> containers (mm-delay / mm-loss / mm-link). The emulation tests produce 
> a logfile, and scripts turn that into a PDF report comparing all the 
> schemes on the metrics of throughput and signalling delay. This is the 
> part that started producing a PDF about 10 minutes before my talk. 
> Right now it runs a *single* scenario -- a particular bufferbloated 
> cellular link, running a single full-throttle flow from each scheme by 
> itself -- but we do plan to add more scenarios.
>
> For ECN: Mahimahi does support pluggable queue disciplines (the 
> constrained link container, mm-link, supports DropTail and DropHead 
> with various thresholds in bytes and packet count), but it doesn't 
> currently ship an implementation of RED or of a scheme that marks 
> instead of drops. So a pull request would have to add this. We would 
> certainly take it.
>
> For other figures of merit (e.g. flow completion time): I was at your 
> L4S talk and agree with your remarks. Evaluating the impact of packet 
> loss on an application-level metric, like flow completion time, will 
> take some effort to design the right interface so it can be measured 
> fairly across schemes. For example, packet loss may not affect FCT 
> that much if the endpoints use a rateless code to recover from loss. 
> But it might affect it a lot if the endpoints are relying on 
> retransmissions.
>
> This work should get done, but it needs somebody to put some effort 
> into it and have each scheme expose an interface that the evaluation 
> code can use to measure a quantity like FCT. (E.g. the interface might 
> have to be something like, transfer this file.) When we're dealing 
> with 10+ schemes with different codebases, it was much easier to start 
> out with a common denominator that is simply "a full-throttle sender 
> of an unbounded stream."
> loss," this is true if "you" is somebody who can influence behavior on 
> the bottleneck link. I'm not too worried about this in our evaluation 
> of endpoint protocols (run over emulated or actual cellular links)
I'm afraid it's not possible to evaluate endpoint protocols independent 
of queuing behaviour.
This is obvious when one includes schemes like XCP in the pantheon.
But it is more generally true for any TCP (see later for loss as one 
example).
> -- an endpoint protocol that is able to achieve low queueing delay is 
> probably not going to be one that also causes a lot of loss.
True.
However, any TCP will invariably increase loss when interacting with an 
/AQM/ that reduces queuing delay.

See "Insights from Curvy RED 
<http://www.bobbriscoe.net/pubs.html#CRED-insights>", which was written 
to explain why we were getting higher loss than we expected from 
fq_Codel and PIE. Fig 3 gives the essence to save you time reading the 
whole thing.

[Also, at the risk of making this completely intractable, in a cellular 
network, the scheduler interacts with TCP in ways that are difficult to 
understand by only observing the black box. See the excellent talk by 
Joachim Fabini "Access Networks –  Dormant Middleboxes 
<https://www.ietf.org/proceedings/94/slides/slides-94-hopsrg-3.pdf>" 
including his papers referenced therein.]

> networks around the world, and we report the performance. Here I would 
> say, we'd love if ECN were deployed in these networks so we could 
> practically run DCTCP, but it's probably outside our ability to influence.
The dashboard that Koen showed in his demo during the L4S BoF has a 
similar goal to your Pantheon (acknowledgements due here to Olga 
Bondarenko and Inton Tsang as well). It runs within a network of real 
equipment (currently only tried data-centre / core / DSL access / fixed 
home network). You might have seen the pull down menus of all the 
different congestion controls it can load on the hosts, and the AQMs it 
can load in the bottleneck. It is controlled by scripts that test the 
range of base RTTs, link rates, mixes of numbers of flows with different 
cc's, configuration of AQM parameters, etc. It automatically produces 
plots of mean and 99-%ile performance three dimensions at a time. It 
wasn't originally written with the goal of people contributing code 
directly - instead we use code for various schemes that is already open 
source. Perhaps it could evolve to work the other way round - we'd have 
to consider the implications in terms of support time.

Cheers

Bob

>
> Best regards,
> Keith
>
> On Sat, Jul 23, 2016 at 4:46 AM, Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Keith,
>
>     In the iccrg this week, Koen asked about adding DCTCP to the
>     pantheon, and you said it's straightforward, but...
>
>     *...Does the structure of the pantheon make it easier to only
>     change the sender algorithm? Or is it just as easy if you want to
>     make changes to the receiver, the network and to the (feedback
>     aspect of the) TCP wire protocol? I know Remy included assessments
>     of XCP and sfq_CoDel, so I assume all changes are possible, but is
>     the ability to plug-in different contenders intended more just for
>     the sender algorithms?
>
>     Aside: I guess that once one starts changing more than the sender,
>     the throughput/delay tradeoff becomes secondary to other tradeoffs
>     like incremental deployability.
>
>     *2/ Wider tradeoffs*
>     Deployability reminds me of a second point: We've realised TCP
>     does not solely cause a dilemma between delay and throughput /
>     utilization. TCP creates a hexlemma (at least). Researchers can
>     make their TCP look good by reducing one impairment at the expense
>     of another - TCP is like a balloon. For instance, you can reduce
>     queuing delay, but increase loss. More loss causes more delay when
>     measuring from the (short) flow completion time perspective, but
>     this delay is invisible if one measures from an RTT-delay perspective.
>
>     See slide 6 of
>     Bob Briscoehttp://bobbriscoe.net/
>
>
>
>
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg