Subject: [aqm] codel with low shape rates

From: Jeff Weeks <[email-masked]>
Date: Mon, 18 Jan 2016 22:11:07 +0000

Hello all,

I'm wondering if there's some data on Codel with low shape rates?

In particular, I'm talking about in the kbps ranges.

In my investigation, it seems as though the algorithm can't control latency as effectively.

For one, the algorithm requires 2 packets in the queue to operate, but at a low shape rate, it's highly likely that all packets in the queue will have high latency, so 'count' will begin to ramp up... but conversely, it's also likely that we drain the queue, and then leave the drop state (and wont re-enter it for at least an interval's worth of time (100ms)).

Effectively, we get an oscillation of "in drop state, out of drop state", and we're not in the drop state for long enough to ramp up count, because the queue itself (proportional to the shape rate) isn't very big.

Are there way to combat this, or am I misinterpreting a portion of the algorithm?

Thanks,
Jeff

________________________________
/dev/jeff_weeks.x2936
Sandvine Incorporated

---

From: Dave Täht <[email-masked]>
Date: Mon, 18 Jan 2016 14:57:46 -0800

> In particular, I'm talking about in the kbps ranges.

We recently did some testing of several codel variants at very low rates
(2mbit/384kbit asymmetric). One flent dataset is here:

http://snapon.cs.kau.se/~d/384k/

There are a couple others. I didn't take the time to create graphs and
collate results before leaving for christmas vacation, the closest
I came to that w/graphs was this post:

https://lists.bufferbloat.net/pipermail/cake/2016-January/001808.html

pie was miserable, also. (it has a 10k estimator needed)

> For one, the algorithm requires 2 packets in the queue to operate, but at a low shape rate, it's highly likely that all packets in the queue will have high latency, so 'count' will begin to ramp up…

This is not quite true - it's bytes, not packets. An MTU's worth of
bytes is the std codel limit before switching off.

Additionally, when htb is used, there is at least an htb quantum's worth
of packets that queue also. (the "cake" variant does not have this
problem. After I published the bcake vs cake results above, the cake
code got improved)

> Effectively, we get an oscillation of "in drop state, out of drop state", and we're not in the drop state for long enough to ramp up count, because the queue itself (proportional to the shape rate)…

I don't see this, we generally end up with a persistently >1 packet
queue with two or more flows going. I see is a worse problem where the
basic attributes of keeping a connection up and things like slow start,
and with multiple flows, result in count climbing excessively high,
especially with ecn in use.

More research at sub 2mbit speeds is needed. Am pretty happy with things
in the 4mbit to 40gbit range.

> Are there way to combat this, or am I misinterpreting a portion of the algorithm?

The simplest way to get decent results is to set target slightly more
than the MTU at the speed you are running at. e.g. 1Mbit = target 13ms.
This is essentially what the sqm-scripts and cake do - and the results
are still less than fully desirable.

(I would like very much to have an aqm that was knobless at these
speeds. Most of the work on trying to improve matters at these rates is
in the multitude of "cake" variants)

> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: "Agarwal, Anil" <[email-masked]>
Date: Tue, 19 Jan 2016 13:43:58 +0000

Dave, Jeff,

A quick simulation of Codel with a single queue, with -

               link rate  = 100 kbps,

               packet size = 500 bytes,

               traffic rate = 110 kbps, CBR, unresponsive

shows queue size stabilizes at 1-3 packets.

Count remains at 1.

Queueing delays are 50-86 ms, but that is understandable, given that the packet transmission time is 40 ms.

The algorithm changes state frequently (every ~400 ms), but that is how it works.

Jeff - what scenario are you studying?

Anil

-----Original Message-----
From: Codel [mailto:[email-masked]] On Behalf Of Dave Täht
Sent: Monday, January 18, 2016 5:58 PM
To: Jeff Weeks; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [Codel] [aqm] codel with low shape rates

> In particular, I'm talking about in the kbps ranges.

We recently did some testing of several codel variants at very low rates (2mbit/384kbit asymmetric). One flent dataset is here:

https://urldefense.proofpoint.com/v2/url?u=http-3A__snapon.cs.kau.se_-7Ed_384k_&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3aefVdgD3p1luGVD6HQAWZt7A&s=dlQhigd0CU9PuDq2KSsG-73u2A01K4KEagZVPJzuYc8&e=

There are a couple others. I didn't take the time to create graphs and collate results before leaving for christmas vacation, the closest I came to that w/graphs was this post:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_pipermail_cake_2016-2DJanuary_001808.html&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3aefVdgD3p1luGVD6HQAWZt7A&s=sF1-9PiJpAY41luL-EjJxf2SFPSNNEkILZ0AizH9PL0&e=

> For one, the algorithm requires 2 packets in the queue to operate, but at a low shape rate, it's highly likely that all packets in the queue will have high latency, so 'count' will begin to ramp up…

This is not quite true - it's bytes, not packets. An MTU's worth of bytes is the std codel limit before switching off.

Additionally, when htb is used, there is at least an htb quantum's worth of packets that queue also. (the "cake" variant does not have this problem. After I published the bcake vs cake results above, the cake code got improved)

> Effectively, we get an oscillation of "in drop state, out of drop state", and we're not in the drop state for long enough to ramp up count, because the queue itself (proportional to the shape rate)…

I don't see this, we generally end up with a persistently >1 packet queue with two or more flows going. I see is a worse problem where the basic attributes of keeping a connection up and things like slow start, and with multiple flows, result in count climbing excessively high, especially with ecn in use.

More research at sub 2mbit speeds is needed. Am pretty happy with things in the 4mbit to 40gbit range.

> Are there way to combat this, or am I misinterpreting a portion of the algorithm?

The simplest way to get decent results is to set target slightly more than the MTU at the speed you are running at. e.g. 1Mbit = target 13ms.

This is essentially what the sqm-scripts and cake do - and the results are still less than fully desirable.

(I would like very much to have an aqm that was knobless at these speeds. Most of the work on trying to improve matters at these rates is in the multitude of "cake" variants)

> [email-masked]<mailto:[email-masked]>

> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail

> man_listinfo_aqm&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fP

> k&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3a

> efVdgD3p1luGVD6HQAWZt7A&s=pDBHTaIvFZbGIz6xTyl2PwKyBA7KXP-nL5w-nIfT4gc&

> e=

Codel mailing list

[email-masked]<mailto:[email-masked]>

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_codel&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3aefVdgD3p1luGVD6HQAWZt7A&s=aQhUguCaMy2FpF_zUnTrqGCflO0G-FVhW2L8LT9NyDs&e=

---

From: Jeff Weeks <[email-masked]>
Date: Tue, 19 Jan 2016 17:32:24 +0000

Hi Anil, Dave;

Anil, you're numbers make sense to me, and I definitely see reasonable results when sending a constant bitrate, a bit above the shape/enforced rate.

The scenario I'm testing may not be reasonable, perhaps, but it's concerning to me.  I'm testing aggressive and bursty flows.  In these scenarios the queue utilization is not consistent, and often drains to 0 packets, which means the algorithm isn't eligible to drop for another interval's worth of time (100ms), at which point a burst of packets may be queued, and latency then spikes.

The inconsistent traffic pattern results in wildly inconsistent average latency per packet.  I suppose you could argue that's expected (inconsistent in, inconsistent out...) but I just don't like relying on uncontrollable external factors.  Perhaps this is an area where fq_codel would perform better?

Dave, What does HTB stand for?  I'm not familiar with it.

I suppose it's possible I'm seeing what you describe; count increments high, to the point where it's more likely to drain the queue, which then causes us to enter the interval wait time before re-engaging.

I definitely see more consistent results at 4Mbps+

--Jeff

________________________________
/dev/jeff_weeks.x2936
Sandvine Incorporated
________________________________
From: Agarwal, Anil [[email-masked]]
Sent: Tuesday, January 19, 2016 8:43 AM
To: Dave Täht; Jeff Weeks; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [Codel] [aqm] codel with low shape rates

Dave, Jeff,

               link rate  = 100 kbps,

               packet size = 500 bytes,

Count remains at 1.

Anil

-----Original Message-----
From: Codel [mailto:[email-masked]] On Behalf Of Dave Täht
Sent: Monday, January 18, 2016 5:58 PM
To: Jeff Weeks; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [Codel] [aqm] codel with low shape rates

Codel mailing list

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 19 Jan 2016 20:00:02 +0200

> On 19 Jan, 2016, at 19:32, Jeff Weeks <[email-masked]> wrote:
> 

Hierarchical Token Bucket.  It’s one of the two most commonly used shapers on Linux, the other being HFSC.

> Perhaps this is an area where fq_codel would perform better?

Very likely.  My experience is that flow isolation (which is what fq_codel adds over plain codel) gives the single biggest improvement to perceived traffic quality of all the techniques I’ve seen and used.  Codel also works significantly better when it only needs to signal to one flow at a time.

If you plan to try out HTB+fq_codel or HFSC+fq_codel, then I would also suggest trying out Cake.  It’s slightly more involved to set up since it’s not yet upstream, but it’s a no-knobs-needed solution using even more sophisticated and comprehensive algorithms than fq_codel.  I’ve just finished adding “triple isolation” to it, and I plan to make it the default in the near future, unless someone turns up a major problem.

- Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 19 Jan 2016 20:09:35 +0200

> On 19 Jan, 2016, at 15:43, Agarwal, Anil <[email-masked]> wrote:
> 

Codel is pretty close to being self-tuning in terms of link capacity, as long as it’s on the bottleneck and only needs to deal with one flow per instance.  The main trouble is when the inherent serialisation delay becomes comparable to the time constants assumed for the total estimated RTT.  On very slow links, the a-priori RTT estimate needs to be increased to account for that.

Cake has an advantage here in knowing the link capacity, via its built-in shaper.  This is how it knows what fine-tuning to apply.  It’s hard for a “pure” AQM qdisc to gather that information automatically.

With that self-tuning, Cake works usably well down to 64Kbps.  At that level, typical modern TCPs (especially CUBIC) saturate the link continuously from the outset, so Codel ends up constantly signalling them to slow down.  With ECN on, this is harmless.

 - Jonathan Morton