Subject: [aqm] CoDel on high-speed links

From: Steven Blake <[email-masked]>
Date: Tue, 09 Jun 2015 12:11:52 -0400

I have a question about how CoDel (as defined in
draft-ietf-aqm-codel-01) behaves on high-speed (e.g., >= 1 Gbps) links.
If this has been discussed before, please just point me in the right
direction.

In the text below, I'm using "drop" to mean either packet discard/ECN
mark.  I'm using (instantaneous) "drop frequency" to mean the inverse of
the interval between consecutive "drops" during a congestion epoch,
measured in drops/sec.

The control law for CoDel computes the next time to drop a packet, and
is given as:

  t + interval/sqrt(count)

where t is the current time, interval is a value roughly proportional to
maximum RTT (recommended 100 msec), and count is cumulative number of
drops during a congestion epoch.  It is not hard to see that drop
frequency increases with sqrt(count).  At the first drop, the frequency
is 10 drop/sec; after 100 drops it is 100 drops/sec; after 1000 drops it
is 316 drops/sec.

On a 4 Mbps link serving say 1000 packets/sec (on average), CoDel
immediately starts dropping 1% of packets and ramps up to ~10% after 100
drops (1.86 secs).  This seems like a reasonable range.  On a 10 GE link
serving 2.5 MPPs on average, CoDel would only drop 0.013% of packets
after 1000 drops (which would occur after 6.18 secs).  This doesn't seem
to be very effective.  It's possible to reduce interval to ramp up drop
frequency more quickly, but that is counter-intuitive because interval
should be roughly proportional to maximum RTT, which is link-speed
independent.

Unless I am mistaken, it appears that the control law should be
normalized in some way to average packet rate.  On a high-speed link, it
might be common to drop multiple packets per-msec, so it also isn't
clear to me whether the drop frequency needs to be recalculated on every
drop, or whether it could be recalculated over a shorter interval (e.g.,
5 msec).

Regards,
 
// Steve

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Jun 2015 19:31:57 +0300

> On 9 Jun, 2015, at 19:11, Steven Blake <[email-masked]> wrote:
> 
> On a 10 GE link
> to be very effective.

Question: have you worked out what drop rate is required to achieve control of a TCP at that speed?  There are well-known formulae for standard TCPs, particularly Reno.  You might be surprised by the result.

Fundamentally, Codel operates on the principle that one mark/drop per RTT per flow is sufficient to control a TCP, or a flow which behaves like a TCP; *not* a particular percentage of packets.  This is because TCPs are generally required to perform multiplicative decrease upon a *single* congestion event.  The increasing count over time is meant to adapt to higher flow counts and lower RTTs.  Other types of flows tend to be sparse and unresponsive in general, and must be controlled using some harder mechanism if necessary.

One such mechanism is to combine Codel with an FQ system, which is exactly what fq_codel in Linux does.  Fq_codel has been tested successfully at 10Gbps.  Codel then operates separately for each flow, and unresponsive flows are isolated.

 - Jonathan Morton

---

From: Simon Barber <[email-masked]>
Date: Tue, 09 Jun 2015 10:14:03 -0700

My concern with fq_codel is that by putting single flows into single Codel 
instances you hit the problem with Codel where it limits bandwidth on 
higher RTT paths.

Simon

Sent with AquaMail for Android
http://www.aqua-mail.com

On June 9, 2015 9:32:15 AM Jonathan Morton <[email-masked]> wrote:

> Question: have you worked out what drop rate is required to achieve control 
> of a TCP at that speed?  There are well-known formulae for standard TCPs, 
> particularly Reno.  You might be surprised by the result.
>
> Fundamentally, Codel operates on the principle that one mark/drop per RTT 
> per flow is sufficient to control a TCP, or a flow which behaves like a 
> TCP; *not* a particular percentage of packets.  This is because TCPs are 
> generally required to perform multiplicative decrease upon a *single* 
> congestion event.  The increasing count over time is meant to adapt to 
> higher flow counts and lower RTTs.  Other types of flows tend to be sparse 
> and unresponsive in general, and must be controlled using some harder 
> mechanism if necessary.
>
> One such mechanism is to combine Codel with an FQ system, which is exactly 
> what fq_codel in Linux does.  Fq_codel has been tested successfully at 
> 10Gbps.  Codel then operates separately for each flow, and unresponsive 
> flows are isolated.
>
>  - Jonathan Morton
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Jun 2015 20:30:53 +0300

Wouldn't that be a sign of dropping too much, in contrast to your previous
post suggesting it wouldn't drop enough?

In practice, statistical multiplexing works just fine with fq_codel, and
you do in fact get more throughput with multiple flows in those cases where
a single flow fails to each adequate utilisation.

Additionally, utilisation below 100% is really characteristic of Reno on
any worthwhile AQM queue and significant RTT. Other TCPs, particularly
CUBIC and Westwood+, do rather better.

- Jonathan Morton

---

From: Dave Dolson <[email-masked]>
Date: Tue, 9 Jun 2015 17:31:25 +0000

Similarly, how many queues would be required when hundreds or thousands of users share a 1Gbps link?
It seems that fq_codel is tuned for a single host or residential gateway.
Has anyone investigate this?

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Simon Barber
Sent: Tuesday, June 09, 2015 1:14 PM
To: Jonathan Morton; Steven Blake
Cc: [email-masked]
Subject: Re: [aqm] CoDel on high-speed links

Simon

---

From: Steven Blake <[email-masked]>
Date: Tue, 09 Jun 2015 13:37:12 -0400

On Tue, 2015-06-09 at 19:31 +0300, Jonathan Morton wrote:

> Question: have you worked out what drop rate is required to achieve
> control of a TCP at that speed?  There are well-known formulae for
> standard TCPs, particularly Reno.  You might be surprised by the
> result.
> 
> Fundamentally, Codel operates on the principle that one mark/drop per
> RTT per flow is sufficient to control a TCP, or a flow which behaves
> like a TCP; *not* a particular percentage of packets.  This is because
> TCPs are generally required to perform multiplicative decrease upon a
> *single* congestion event.  The increasing count over time is meant to
> adapt to higher flow counts and lower RTTs.  Other types of flows tend
> to be sparse and unresponsive in general, and must be controlled using
> some harder mechanism if necessary.

I'm not worried about controlling one TCP.  I'm worried about
controlling a multiplex of > 10K TCPs (on average).

> One such mechanism is to combine Codel with an FQ system, which is
> exactly what fq_codel in Linux does.  Fq_codel has been tested
> successfully at 10Gbps.  Codel then operates separately for each flow,
> and unresponsive flows are isolated.

FQ is very nice, but not always an option at the necessary scale for
10/100 Gbps links.

Regards,

// Steve

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Jun 2015 20:46:40 +0300

In such cases you could reasonably configure fq_codel with a very large
number of queues - I believe it supports 65536 out of the box - and/or to
provide host fairness instead of flow fairness. A planned enhancement to
cake (which really is designed for the last mile - fq_codel isn't so
specialised) is to provide both at once.

In the default mode, fq_codel still degrades gracefully when faced with an
extreme flow count, due to the natural incidence of hash collisions. Each
Codel instance then applies to traffic from a subset of flows. Even under
these conditions, a single unresponsive flow is reasonably well isolated,
only interfering with traffic unfortunate enough to end up in the same
queue by chance.

- Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Tue, 09 Jun 2015 14:29:18 -0400

On Tue, 2015-06-09 at 20:46 +0300, Jonathan Morton wrote:
> In such cases you could reasonably configure fq_codel with a very
> large number of queues - I believe it supports 65536 out of the box -
> and/or to provide host fairness instead of flow fairness. A planned
> enhancement to cake (which really is designed for the last mile -
> fq_codel isn't so specialised) is to provide both at once.
>
> In the default mode, fq_codel still degrades gracefully when faced
> with an extreme flow count, due to the natural incidence of hash
> collisions. Each Codel instance then applies to traffic from a subset
> of flows. Even under these conditions, a single unresponsive flow is
> reasonably well isolated, only interfering with traffic unfortunate
> enough to end up in the same queue by chance.

That's great, but I'm talking about how to do AQM on Nx100 Gbps packet
processing ASICs, not Linux boxes.  And I agree that FQ is desirable,
but not always cost effective or feasible to retrofit.

So back to my previous point: is CoDel as described in
draft-ietf-aqm-codel-01 suitable *by itself* as an AQM for high-speed
links multiplexing lots (> 1K) of flows.

Ex/  RTT = 25 msec
     #flows = 1K

To drop 1 packet/flow/RTT as you say, I need to drop 40K packets/sec.
CoDel ramps up to this drop rate after approximately eternity (~16e6
drops).

Has anyone tested CoDel (*not fq_codel*) in an environment with interval
~100 msec and #flows >= 1K?  It would be easy enough to tweak the
control law ramp: just scale the calculation of drop interval by a
factor inversely proportional to link speed or average number of flows.

Regards,

// Steve

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 9 Jun 2015 22:06:20 +0300

> On 9 Jun, 2015, at 21:29, Steven Blake <[email-masked]> wrote:
> 

Ah, you are talking about core networks.  That’s a little out of my area of expertise, but…

As I understand it, core networks are generally supposed to be over-provisioned.  However, even a nominally over-provisioned link can be saturated in the short-term and/or at peak load.  A straightforward AQM system is useful for coping with that.

There is a well-known theory which states that, for a dumb FIFO, queue length can be reduced from the “BDP rule of thumb” to BDP * sqrt(flows) in the case where a large number of flows is normally expected.  This is the case for core networks, but not the edge (where the link is routinely saturated by a single flow).

I think it’s reasonable to suppose that the same theory might apply to Codel parameters.  If so, taking a nice round number of 10000 flows (since you can’t predict it very precisely, an order of magnitude or two is sufficient), the new parameters would be interval=1ms and target=50us.  If you re-run your analysis using those parameters, do you get more reasonable behaviour?  Intuitively, I think you should get similar per-packet behaviour at 100G using those parameters, as at 1G using the defaults.

 - Jonathan Morton

---

From: Tim Shepard <[email-masked]>
Date: Tue, 09 Jun 2015 15:43:23 -0400

> There is a well-known theory which states that, for a dumb
> FIFO, queue length can be reduced from the "BDP rule of thumb"
> to BDP * sqrt(flows) in the case where a large number of flows
> is normally expected.

That cannot be right, it goes the wrong direction.  Maybe you
meant something like BDP / sqrt(flows) ?   (I have no idea if
that is correct, but at least it goes the right direction.)

			-Tim Shepard
			 [email-masked]

---

From: Dave Taht <[email-masked]>
Date: Tue, 9 Jun 2015 12:44:38 -0700

The below makes several mis-characterisations of codel in the first
place, and then attempts to reason from there.

On Tue, Jun 9, 2015 at 9:11 AM, Steven Blake <[email-masked]> wrote:
> drops during a congestion epoch.

No. Count is just a variable to control the curve of the drop rate. It
is not constantly
incremented, either, it goes up and down based on how successful it is
at controlling
the flow(s), only incrementing while latency exceeds the target,
decrementing slightly
after it stays below the target. The time spent below the target is
not accounted for,
so you might have a high "bang-bang" drop rate retained, when
something goes above from below.

This subtlety is something people consistently miss and something I
tried to elucidate in the
first stanford talk.

>It is not hard to see that drop
> drops (1.86 secs).

No it will wait 100ms after stuff first exceeds the target, then
progressively shoot harder based on the progress of the
interval/sqrt(count).

secondly people have this tendency to measure full size packets, or a
1k average packet. The reality is a dynamic range of 64 bytes to 64k
(gso/tso/gro offloads). So bytes is a far better proxy than packets in
order to think about this properly.

offloads of various sorts bulking up packet sizes has been a headache.
I favor reducing mss on highly congested underbuffered links (and bob
favors sub-packet windows) to keep the "signal strength" up.

The original definition of packet (circa 1962) was 1000 bits, with up
to 8 fragments. I do wish the materials that were the foundation of
packet behavior were online somewhere...

>  This seems like a reasonable range.  On a 10 GE link
> after 1000 drops (which would occur after 6.18 secs).

I am allergic to averages as a statistic in the network measurement case.

> This doesn't seem

Except that tcp's drop their rates by (typically) half on a drop, and
a matter of debate as to when on CE.

> drop, or whether it could be recalculated over a shorter interval (e.g.,

Pie took the approach of sampling, setting a rate for shooting, over a
16ms interval. That's pretty huge, but also low cost in some hardware.

Codel's timestamp per-packet control law is continuous (but you do
need to have a cheap packet timestamping ability).

Certainly in all cases more work is needed to address the problems
100gps rates have in general, and it is not just all queue theory! A
small packet is .62 *ns* in that regime. A benefit of fq in this case
is that you can parallelize fib table lookups across multiple
processors/caches, and of fq_codel is that all codels operate
independently.

-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

---

From: Steven Blake <[email-masked]>
Date: Tue, 09 Jun 2015 16:40:22 -0400

> place, and then attempts to reason from there.

Hmmm...

> is not constantly incremented, either, it goes up and down based on
> how successful it is at controlling the flow(s), only incrementing
> while latency exceeds the target, decrementing slightly after it stays
> below the target. The time spent below the target is not accounted
> for, so you might have a high "bang-bang" drop rate retained, when

I specifically mentioned "during a congestion epoch", but let me be more
precise: count is continuously incremented during an extended period
where latency exceeds the target (perhaps because CoDel isn't yet
dropping hard enough). Correct?

The fact that the drop frequency doesn't ramp down quickly when
congestion is momentarily relieved is good, but doesn't help if it takes
forever for the algorithm to ramp up to an effective drop frequency
(i.e., something greater than 1 drop/flow/minute).

> progressively shoot harder based on the progress of the

Ok.  At the first drop it is dropping at a rate of 1 packet/100 msec ==
10 drops/sec and ramps up from there.  At the 100th drop it is dropping
at a rate of 100 msec/sqrt(100) == 1 packet/10 msec == 100 drops/sec.
This just so happens to occur after 1.8 secs.

Aside: as described, CoDel's drop frequency during a congestion epoch
increases approximately linearly with time (at a rate of about 50
drops/sec^2 when interval = 100 msec).

> packet behavior were online somewhere...

I don't see how this has anything to do with the text of the draft or my
questions.

> a matter of debate as to when on CE.

Ex/ 10 GE link, ~10K flows (average).  During a congestion epoch, CoDel
with interval = 100 msec starts dropping 257 packets/sec after 5 secs.
How many flows is that effectively managing?

> processors/caches, and of fq_codel is that all codels operate

High-speed metro/core links need AQM, too.  I believe that
draft-ietf-aqm-codel-01 doesn't work for these links in the general case
(e.g., large #flows, recommended value for interval, no FQ).  IMHO the
draft should say something about this, or should adjust the algorithm
accordingly.

Regards,

// Steve

---

From: David Lang <[email-masked]>
Date: Tue, 9 Jun 2015 13:52:22 -0700 (PDT)

> How many flows is that effectively managing?

how fast are the flows ramping back up to the prior speed?

if you have 10K flows and ~250 drops/sec, over 40 seconds each could end up with 
one drop. If that keeps the link uncongested, it's doing it's job.

Unfortunantly, when there is a drop, the affected flow slows down a LOT, so if 
you are near the edge of being uncongested, you may not need to slow that many 
flows down to be uncongested. Then as the flows ramp back up, the link becomes 
congested again and some flow needs to be slowed down. Hopefully CoDel is going 
to slow down a different flow the next time.

With the extisting feedback that can be provided, it's not possible to slow all 
flows down 5%, all you could do is to slow 10% of the flows by 50% to reduce 
overall load by 5%

David Lang

---

From: Steven Blake <[email-masked]>
Date: Tue, 09 Jun 2015 18:07:02 -0400

> one drop. If that keeps the link uncongested, it's doing it's job.

According to my calculations, with RTT = 25 msec and MTU = 1500 bytes,
you need to be going around 29 Mbps average (oscillating between 2/3 and
4/3 of this) to need to see a drop every 40 seconds.  For 1 Mbps average
you need ~1.4 secs between drops.    For 10K 1 Mbps flows you then need
to drop ~7000 packets/sec (~0.8% drop frequency for MTU-sized packets on
a 10 GE link).

Of course this all assumes uniform stationary elephants which is never
the case in real life, but you see how CoDel's drop frequency (for
interval = 100 msec) is not even in the right ballpark.

> flows down 5%, all you could do is to slow 10% of the flows by 50% to reduce 

The more flows you have, the more flows you need to nuke to slow things
down even a little bit.  The more flows you need to nuke, the faster you
need to drop packets.

Regards,

// Steve

---

From: "Agarwal, Anil" <[email-masked]>
Date: Wed, 10 Jun 2015 20:58:15 +0000

To follow up on Steve's posting - here is a simple (simplistic) analysis to estimate the time taken by CoDel 
to converge to the right value of the interval I. Perhaps, similar analysis has been done elsewhere.
 
Analysis of Single Queue CoDEL for large number of TCP connections.

Given:
  Link Capacity = C bps
  Number of active TCP connections = n
  Round Trip Time = rtt seconds
  Packet size = PS bytes
  CoDel initial interval = I0 seconds

Assume:
  All TCP connections are long-lived.

  At steady state, for every TCP conn, cwnd increases linearly from W/ 2 to W
  by one packet per RTT and then drops to W / 2 when a packet is dropped or
  ECN-marked

Derive:
  bps_per_conn = C / n
  Conn max window size = W = bps_per_conn * rtt / PS (packets)
  Ideal drop interval per conn = DI1 = W / 2 * rtt (seconds)
  Ideal drop rate per conn = DR1 = 1 / DI1 (packets/sec)
  Aggregate Ideal drop rate = DR = DR1 * n (packets/sec)
  Ideal CoDel interval = II = 1 / DR (seconds)

  CoDel Packet drop rate increase per sec = DRI =~ 0.5 / I0^2
    This can be derived from the derivative of the equation I(n) = I0 / sqrt(n),
    where I(n) is the interval size after n consecutive congested intervals.
    Under sustained congestion, the packet drop rate increases
    by DRI packets/second, every second.
    DRI does not depend on C, n or rtt !

  CoDel time when Interval converges to II = T = DR / DRI (seconds)

  or
  CoDel time when Interval converges to II = T = 4 * PS / C * (n * I0 / rtt) ^ 2 (seconds)

Comments
  The time T taken by CoDel to reach the correct I value is -
    - Inversely proportional to link Capacity C
    - Proportional to the square of the number of active TCP connections
    - Proportional to the square of the initial interval value
    - Inversely proportional to the square of rtt.

  If link capacity increases and the number of TCP connections increases
  correspondingly, then T will increase.

  During time 0 to T, the queue will likely remain full and packets will
  experience tail-drops. Of course one could argue that the number of connections n 
  is likely to increase slowly, perhaps slower than the convergence time of I (?).

  In steady state, large sudden changes in n will likely result in tail-drops.

  This analysis is probably not applicable when rtt > I0 or if W is too small.

Numerical Examples
  1. C = 1 Gbps, n = 1000, rtt = 100 ms, PS = 1500 bytes, I0 = 100 ms
     DRI = 50 packets/sec/sec
     W = 8.33 packets
     Drop Rate DR = 2400 packets/sec
     T = 48 seconds

  2. C = 10 Gbps, n = 10000, rtt = 100 ms, PS = 1500 bytes, I0 = 100 ms
     DRI = 50 packets/sec/sec
     W = 8.33 packets
     Drop Rate DR = 24000 packets/sec
     T = 480 seconds

3. C = 10 Gbps, n = 11000, rtt = 100 ms, PS = 1500 bytes, I0 = 100 ms
     DRI = 50 packets/sec/sec
     W = 7.58 packets
     Drop Rate DR = 29040 packets/sec
     T = 580 seconds
     (10% increase in number of TCP connections caused T to increase by 100 seconds).

This does seem to indicate that for core and edge routers that handle large number of connections,
if the number of connections changes by large amounts , then the convergence of the CoDel Interval
value will likely take a correspondingly large amount of time.

This is probably true for other AQM schemes as well - is there any published analysis on the subject? 

The example I have used may be a bit extreme, but I hope it illustrates the issue Steve is pointing to.

Regards,
Anil Agarwal
ViaSat Inc.

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Steven Blake
Sent: Tuesday, June 09, 2015 4:40 PM
To: Dave Taht
Cc: [email-masked]
Subject: Re: [aqm] CoDel on high-speed links

Hmmm...

> > In the text below, I'm using "drop" to mean either packet 
> > discard/ECN mark.  I'm using (instantaneous) "drop frequency" to 
> > mean the inverse of the interval between consecutive "drops" during 
> > a congestion epoch, measured in drops/sec.
> >
> > The control law for CoDel computes the next time to drop a packet, 
> > and is given as:
> >
> >   t + interval/sqrt(count)
> >
> > where t is the current time, interval is a value roughly 
> > proportional to maximum RTT (recommended 100 msec), and count is 
> > cumulative number of drops during a congestion epoch.
> 
> tried to elucidate in the first stanford talk.

I specifically mentioned "during a congestion epoch", but let me be more
precise: count is continuously incremented during an extended period where latency exceeds the target (perhaps because CoDel isn't yet dropping hard enough). Correct?

The fact that the drop frequency doesn't ramp down quickly when congestion is momentarily relieved is good, but doesn't help if it takes forever for the algorithm to ramp up to an effective drop frequency (i.e., something greater than 1 drop/flow/minute).

> > frequency increases with sqrt(count).  At the first drop, the 
> >frequency  is 10 drop/sec; after 100 drops it is 100 drops/sec; after 
> >1000 drops it  is 316 drops/sec.
> >
> > immediately starts dropping 1% of packets and ramps up to ~10% after 
> > 100 drops (1.86 secs).
> 

Ok.  At the first drop it is dropping at a rate of 1 packet/100 msec ==
10 drops/sec and ramps up from there.  At the 100th drop it is dropping at a rate of 100 msec/sqrt(100) == 1 packet/10 msec == 100 drops/sec.
This just so happens to occur after 1.8 secs.

Aside: as described, CoDel's drop frequency during a congestion epoch increases approximately linearly with time (at a rate of about 50
drops/sec^2 when interval = 100 msec).

> packet behavior were online somewhere...

I don't see how this has anything to do with the text of the draft or my questions.

> >  This seems like a reasonable range.  On a 10 GE link serving 2.5 
> > MPPs on average, CoDel would only drop 0.013% of packets after 1000 
> > drops (which would occur after 6.18 secs).
> 
> > to be very effective.  It's possible to reduce interval to ramp up 
> > drop frequency more quickly, but that is counter-intuitive because 
> > interval should be roughly proportional to maximum RTT, which is 
> > link-speed independent.
> 

Ex/ 10 GE link, ~10K flows (average).  During a congestion epoch, CoDel with interval = 100 msec starts dropping 257 packets/sec after 5 secs.
How many flows is that effectively managing?

> > normalized in some way to average packet rate.  On a high-speed 
> > link, it might be common to drop multiple packets per-msec, so it 
> > also isn't clear to me whether the drop frequency needs to be 
> > recalculated on every drop, or whether it could be recalculated over 
> > a shorter interval (e.g.,
> > 5 msec).
> 

High-speed metro/core links need AQM, too.  I believe that
draft-ietf-aqm-codel-01 doesn't work for these links in the general case (e.g., large #flows, recommended value for interval, no FQ).  IMHO the draft should say something about this, or should adjust the algorithm accordingly.

Regards,

// Steve

---

From: Dave Taht <[email-masked]>
Date: Wed, 10 Jun 2015 14:56:50 -0700

I certainly would like you to try factoring in fq_codel's effects into
your calculations. 1024 queues is the default,
with 64k queues possible. If you assume 10k flows, you can readily
assume a pretty equal distribution across those available queues, with
1k codels compensating madly. I tend to think most of the future
faster world is going to have multiple queues, with every 10gigE
device I am familiar with coming with 64 or more.

Secondly you appear to be thinking in terms of very abrupt changes in
demand not characteristic of a core link. (certainly characteristic of
a route flap, however, so I am not dissing your work).

My favorite paper of last year was this one, where changes in demand
had timescales of hours, not seconds, and I would have loved to have
been in a position to try out every new aqm and fq+aqm algorithm on a
core-like router (many, many 10gigE ports is totally feasible on x86),
against the workloads here:

http://www.caida.org/publications/papers/2014/measurement_analysis_internet_interconnection/measurement_analysis_internet_interconnection.pdf

Lastly, spreadsheets are more easy to deal with than emails, if you
want to put one up with various
algos compared, that would be a goodness!?

The "all tcp sessions are long lived" is a truly fatal error in
thinking about how this stuff works, IMHO, but I realize that that is
hard to reason about.

---

From: Dave Taht <[email-masked]>
Date: Wed, 10 Jun 2015 15:37:10 -0700

On Tue, Jun 9, 2015 at 10:14 AM, Simon Barber <[email-masked]> wrote:
> instances you hit the problem with Codel where it limits bandwidth on higher
> RTT paths.

I recently did a bit of work, testing rtt_fairness from my location
(los gatos, california) to linodes in london, dallas, tokoyo, and
newark, at RTTs of roughly 145, 45, 115, and 85ms. The servers are all
using sch_fq and a modern linux (on a vm) There is a rangley box
inbetween running the sqm scripts that let me test pie, codel,
fq_codel, cake, etc.

On the long path, with pie, the download rate was generally higher
than on the shorter paths, which was kind of interesting and would
bear a repeated look at. Codel, more even, and fq_codel was very even
across all rtts.

http://snapon.lab.bufferbloat.net/~d/qdisc-stats2/download_comparison.png

http://snapon.lab.bufferbloat.net/~d/qdisc-stats2/upload_comparison.png

(rawer data in that dir or you can get it all via:
http://snapon.lab.bufferbloat.net/~d/qdisc-stats2.tgz
 toke is also working on getting buffering, drop, and delay
measurements, some of those and preliminary plot types are in there
also. pull flent from git.
 )

the rtt_fair4be dataset is noisy (and limited to my local connection
speed of 70/10mbits) If anyone would like access to these servers for
more extensive testing, I still have quite a few more gigabits to use
up, and no time to use them. Contact me offlist for access.

>> control of a TCP at that speed?  There are well-known formulae for standard
>> TCPs, particularly Reno.  You might be surprised by the result.
>>
>> per flow is sufficient to control a TCP, or a flow which behaves like a TCP;
>> *not* a particular percentage of packets.  This is because TCPs are

---

From: Dave Taht <[email-masked]>
Date: Wed, 10 Jun 2015 16:26:56 -0700

oops, I did not mean to take this offlist.

On Wed, Jun 10, 2015 at 4:25 PM, Dave Taht <[email-masked]> wrote:
> I care about this stuff, I strongly support the need for aqm on high
> speed links also...
>
> but my own focus is mostly on shorter term stuff at much lower rates,
> like wifi, and I am relying on others on the list to take apart your
> analysis. I am sufficiently lazy to want a spreadsheet to check your
> assumptions and then to go simulate what actually happens in my
> testbeds.
>
> codel is a "safe" aqm. A big goal was to do no harm.
>
> Pie hits things harder, faster, when things get out of wack, but
> tolerates a much higher delay.
>
> In "cake" we are looking at various problems, notably the stanford
> solution for an "ideal" flows and bdp relationship, and packet peeling
> for bulk offloads, and also hitting things harder when certain targets
> are exceeded, and/or using ecn more wisely.
>
> http://www.bufferbloat.net/projects/codel/wiki/Cake
>
> Some of that work might migrate back to codel. An easy mod would be to
> up the count while tail dropping, for example.
>
> And in all cases I don't think we will be applying single queue aqm
> ideas this close to the "wire", some additional latency for other
> processing is needed.
>
> Few doing codel believes in it, in it's current form, as a single
> queue aqm. It was a "piece" of the solution to bufferbloat -
> regulating tcp better at a wide range of rtts.
>
> a 5ms delay target (codel), or 16ms (pie) certainly seem like quite a
> lot at 100gbit, and trying further to size the available buffering to
> the number of flows seems like a good avenue for further research. I
> would love to find someone that could run cake at 10GigE.
>
> IO is a bad choice of variable name. I initially interpreted it as 10
> seconds, which cause a BP increase. :)
>
>> Assume:
> Not an assumption I would like to make. Do we have good numbers for
> core routers? It certainly shifted since netflix. I note that a
> network primarily in slow start behaves much worse to an aqm trying to
> shoot at the right stuff.
>
> In any state, large sudden changes in n will more likely result in
> tail drops. Slow start with IW10 is nasty. Quic is doing 10 packet IWs
> and 22 packets more, paced.
>
> However, large sudden changes happen infrequently.
>
> These all assume all tcps are at full rate. I think. Goal in life is
> to hit the fattest flows and get them to back off. Many tcps are rate
> limited in other places than the core - well, the core is hopefully
> overprovisioned, except when it was not (as per the paper I posted
> earlier). Odds are you will hit the fatter flows with more likelyhood,
> with a corresponding dropoff in rate.
>
> If your argument is that 10 minutes is too long to converge to a more
> ideal rate from 0, I agree. 64 hardware queues coupled to 64 instances
> of fq_codel with 1024 queues each may well ALSO be too many darn
> queues - but it would hopefully converge to a more ideal rate much
> faster with every change in n flows.
>
> That said we end up with another question, how much buffering is
> enough buffering when you have that many flows? Which is an old
> question, and I do wish it were possible to have a device on the core,
> routinely saturated with real traffic, that we could be looking at as
> development of realistic aqm and fq strategies continues.
>
>>
> Scaling AQM technologies to 100GigE and beyond would be a very nice
> paper to have. :)
>

---

From: Steven Blake <[email-masked]>
Date: Wed, 10 Jun 2015 20:45:43 -0400

> device I am familiar with coming with 64 or more.

As it stands the draft doesn't say anything to this effect.

> a route flap, however, so I am not dissing your work).

Things can change surprisingly rapidly when backhoe meets fiber.

Regards,

// Steve

---

From: Simon Barber <[email-masked]>
Date: Sun, 14 Jun 2015 16:10:22 -0700

Indeed - I believe that Codel will drop too much to allow maximum 
bandwidth utilization, when there are very few flows, and RTT is 
significantly greater than target. The theory is - with a Reno based CC 
the cwind gets cut in half on a drop. If the drop in cwind is greater 
than the number of packets in the queue, then the queue will empty out, 
and the link will then be idle for a while. If you want data to keep the 
data flowing uninterrupted, then you must have a full unloaded RTT's 
worth of data in the queue at that point. A drop will happen, the cwind 
will be halved (assuming a Reno TCP), and the sender will stop sending 
until one (unloaded) RTT's worth of data has been received. At that 
point the queue will just hit empty as the sender starts sending again.

Simon

On 6/9/2015 10:30 AM, Jonathan Morton wrote:
>
> Wouldn't that be a sign of dropping too much, in contrast to your 
> previous post suggesting it wouldn't drop enough?
>
> In practice, statistical multiplexing works just fine with fq_codel, 
> and you do in fact get more throughput with multiple flows in those 
> cases where a single flow fails to each adequate utilisation.
>
> Additionally, utilisation below 100% is really characteristic of Reno 
> on any worthwhile AQM queue and significant RTT. Other TCPs, 
> particularly CUBIC and Westwood+, do rather better.
>
> - Jonathan Morton
>

---

From: Dave Taht <[email-masked]>
Date: Sun, 14 Jun 2015 22:13:11 -0700

On Sun, Jun 14, 2015 at 4:10 PM, Simon Barber <[email-masked]> wrote:
> Indeed - I believe that Codel will drop too much to allow maximum bandwidth
> utilization, when there are very few flows, and RTT is significantly greater
> than target. The theory is - with a Reno based CC the cwind gets cut in half
> on a drop. If the drop in cwind is greater than the number of packets in the
> queue, then the queue will empty out, and the link will then be idle for a
> while. If you want data to keep the data flowing uninterrupted, then you
> must have a full unloaded RTT's worth of data in the queue at that point. A
> drop will happen, the cwind will be halved (assuming a Reno TCP), and the
> sender will stop sending until one (unloaded) RTT's worth of data has been
> received. At that point the queue will just hit empty as the sender starts
> sending again.
>
> Simon
>
>
>
>> any worthwhile AQM queue and significant RTT. Other TCPs, particularly CUBIC
>> and Westwood+, do rather better.
>>
>> - Jonathan Morton
>>
>

---

From: Dave Taht <[email-masked]>
Date: Sun, 14 Jun 2015 22:26:54 -0700

On Sun, Jun 14, 2015 at 4:10 PM, Simon Barber <[email-masked]> wrote:
> than target.

Interval. Not target. Interval defaults to 100ms. Target is 5ms. Dropping
behaviors stop when the queue falls below the target.

Range of tests from near zero to 300ms RTT codel does quite well with reno,
better with cubic, on single flows. 4 flows, better. fq_codel does better than
that on more than X flows in general.

You can easily do whatever experiments you like with off the shelf hardware
and RTTs around half the planet to get the observations you need to confirm
your thinking. Remember that a drop tail queue of various sizes has problems
of it's own.

I have a long overdue rant in progress of being wikified about how to use
netem correctly to properly emulate any rtt you like.

I note that a main aqm goal is not maximum bandwidth utilization, but maximum
bandwidth while still having working congestion avoidance and minimal queue
depth so other new flows can rapidly grab their fair share of the
link. The bufferbloat
problem was the result of wanting maximum bandwidth for single flows.

> The theory is - with a Reno based CC the cwind gets cut in half

flight + queue.

> must have a full unloaded RTT's worth of data in the queue at that point. A

Do the experiment? Recently landed in flent is the ability to monitor
queue depth while running another test.

> received. At that point the queue will just hit empty as the sender starts

And reno is dead. Long live reno!

---

From: Simon Barber <[email-masked]>
Date: Mon, 15 Jun 2015 11:00:50 -0700

> behaviors stop when the queue falls below the target.

In this case I specifically mean target, not interval. Dropping stops 
when queue falls below target, but by then it's too late. In the case 
I'm talking about (cwind cut by more than queue length) then a period of 
link idle occurs, and so bandwidth is hurt. It happens repeatedly.

> that on more than X flows in general.
The effect is not huge, but the bandwidth loss is there. More flows 
significantly reduce the effect, since the other flows keep the link 
busy. This bandwidth reduction effect only happens with very few flows. 
I think TCP Reno will be worse than Cubic, due to it's 50% reduction in 
cwind on drop vs Cubic's 20% reduction - but Cubic's RTT independent 
increase in cwind after the drop may make the effect happen more often 
with larger RTTs.

What results have you seen for codel on single flow for these larger RTTs?

> problem was the result of wanting maximum bandwidth for single flows.
Indeed - with many TCP CC algorithms it's just not possible to achieve 
maximum bandwidth utilization with only 5ms induced latency when the 
RTTs are long, and a single queue (no FQ, only drop tail or single queue 
AQM). The multiplicative decrease part of TCP CC simply does not allow 
it unless the decrease is smaller than the queue (PRR might mitigate a 
little here). Now add in FQ and you can have the best of both worlds.
>> queue, then the queue will empty out, and the link will then be idle for a
When cwind gets cut by N packets, the sender stops sending data while 
ACKs for N data packets are received. If the queue has less than N data 
packets, then it will empty out, resulting in an idle link at that 
point, and eventually at the receiver (hence bandwidth loss).

---

From: "Agarwal, Anil" <[email-masked]>
Date: Mon, 15 Jun 2015 23:40:32 +0000

I guess this is pointing to the age old problem - what is the right buffer size or equivalent delay limit, when packets should be dropped or ECN-marked, so that the link is never under-utilized?

For a single TCP connection, the answer is the bandwidth-delay product BDP.
For large number of connections, it is BDP / sqrt(numConnections).

Hence, one size does not fit all.
E.g., for RTT of 100 ms or 500 ms, CoDel target delay of 5 or 10 ms is too short - when handling a small number of connections.

Perhaps, there is a need for a design that adapts the queue size (or delay) target dynamically by estimating numConnections !

Anil

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Simon Barber
Sent: Monday, June 15, 2015 2:01 PM
To: Dave Taht
Cc: Jonathan Morton; [email-masked]; Steven Blake
Subject: Re: [aqm] CoDel on high-speed links

On 6/14/2015 10:26 PM, Dave Taht wrote:
>> significantly greater than target.
> Interval. Not target. Interval defaults to 100ms. Target is 5ms. 
> Dropping behaviors stop when the queue falls below the target.

In this case I specifically mean target, not interval. Dropping stops when queue falls below target, but by then it's too late. In the case I'm talking about (cwind cut by more than queue length) then a period of link idle occurs, and so bandwidth is hurt. It happens repeatedly.

> Range of tests from near zero to 300ms RTT codel does quite well with 
> reno, better with cubic, on single flows. 4 flows, better. fq_codel 
> does better than that on more than X flows in general.
The effect is not huge, but the bandwidth loss is there. More flows significantly reduce the effect, since the other flows keep the link busy. This bandwidth reduction effect only happens with very few flows. 
I think TCP Reno will be worse than Cubic, due to it's 50% reduction in cwind on drop vs Cubic's 20% reduction - but Cubic's RTT independent increase in cwind after the drop may make the effect happen more often with larger RTTs.

> You can easily do whatever experiments you like with off the shelf 
> hardware and RTTs around half the planet to get the observations you 
> need to confirm your thinking. Remember that a drop tail queue of 
> various sizes has problems of it's own.
>
> I have a long overdue rant in progress of being wikified about how to 
> use netem correctly to properly emulate any rtt you like.
>
> I note that a main aqm goal is not maximum bandwidth utilization, but 
> maximum bandwidth while still having working congestion avoidance and 
> minimal queue depth so other new flows can rapidly grab their fair 
> share of the link. The bufferbloat problem was the result of wanting 
> maximum bandwidth for single flows.
Indeed - with many TCP CC algorithms it's just not possible to achieve maximum bandwidth utilization with only 5ms induced latency when the RTTs are long, and a single queue (no FQ, only drop tail or single queue AQM). The multiplicative decrease part of TCP CC simply does not allow it unless the decrease is smaller than the queue (PRR might mitigate a little here). Now add in FQ and you can have the best of both worlds.
>> The theory is - with a Reno based CC the cwind gets cut in half on a 
>> drop. If the drop in cwind is greater than the number of packets in 
>> the queue, then the queue will empty out, and the link will then be 
>> idle for a
> flight + queue.
When cwind gets cut by N packets, the sender stops sending data while ACKs for N data packets are received. If the queue has less than N data packets, then it will empty out, resulting in an idle link at that point, and eventually at the receiver (hence bandwidth loss).
>> while. If you want data to keep the data flowing uninterrupted, then 
>> you must have a full unloaded RTT's worth of data in the queue at 
>> that point. A
>> drop will happen, the cwind will be halved (assuming a Reno TCP), and 
>> the sender will stop sending until one (unloaded) RTT's worth of data 
>> has been received. At that point the queue will just hit empty as the 
>> sender starts sending again.
>>> Additionally, utilisation below 100% is really characteristic of 
>>> Reno on any worthwhile AQM queue and significant RTT. Other TCPs, 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mai
>> lman_listinfo_aqm&d=AwICAg&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06
>> fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=uxNYRwC4x80YZPJPY
>> o3-lMaVCC_1TNJzTxVGd0F1PSs&s=6OQXBky7MzGz1dBmf7oRhwemCi5a4yAiQRm90WHX
>> FIg&e=
>
>

_______________________________________________
aqm mailing list
[email-masked]
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_aqm&d=AwICAg&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=uxNYRwC4x80YZPJPYo3-lMaVCC_1TNJzTxVGd0F1PSs&s=6OQXBky7MzGz1dBmf7oRhwemCi5a4yAiQRm90WHXFIg&e=

---

From: Dave Taht <[email-masked]>
Date: Mon, 15 Jun 2015 16:56:44 -0700

> E.g., for RTT of 100 ms or 500 ms, CoDel target delay of 5 or 10 ms is too short - when handling a small number of connections.

The codel recommendation is that the target be set to 5-10% of the
typical interval (RTT), so in the case of a sat link, interval would
be 600(?)ms, and target 5-10 % of that, 30-60ms.

The recommendation bears testing in this scenario. If you would like
to specify what you expect your (say) 98th percentile real physical
RTT, I can exhaustively simulate that rather rapidly later this week
against reno, cdg, cubic, and westwood and various aqms, against flows
from 1 to 100.

Most of this convo has also missed other advancements in tcp (reno is
thoroughly dead), like PRR. A very good read which incorporates a
discussion of PRR is

http://folk.uio.no/kennetkl/jonassen_thesis.pdf

which I plan to finish reading on the plane.

I am not sure what the pie settings would be for a sat system.

> Perhaps, there is a need for a design that adapts the queue size (or delay) target dynamically by estimating numConnections !

Not "perhaps". That is one of the avenues cake is exploring:

https://lists.bufferbloat.net/pipermail/cake/2015-June/000241.html

---

From: "Agarwal, Anil" <[email-masked]>
Date: Tue, 16 Jun 2015 00:12:29 +0000

Dave,

I guess I need to read up on "cake".

If you have time, can you simulate an RTT of 600 ms?
With a few queue drain rates from 1 Mbps to 100 Mbps.
Would help us "satellite folks" get a better understanding of CoDel parameters.

Thanks,
Anil

-----Original Message-----
From: Dave Taht [mailto:[email-masked]] 
Sent: Monday, June 15, 2015 7:57 PM
To: Agarwal, Anil
Cc: Simon Barber; Jonathan Morton; [email-masked]; Steven Blake
Subject: Re: [aqm] CoDel on high-speed links

> E.g., for RTT of 100 ms or 500 ms, CoDel target delay of 5 or 10 ms is too short - when handling a small number of connections.

The codel recommendation is that the target be set to 5-10% of the typical interval (RTT), so in the case of a sat link, interval would be 600(?)ms, and target 5-10 % of that, 30-60ms.

The recommendation bears testing in this scenario. If you would like to specify what you expect your (say) 98th percentile real physical RTT, I can exhaustively simulate that rather rapidly later this week against reno, cdg, cubic, and westwood and various aqms, against flows from 1 to 100.

Most of this convo has also missed other advancements in tcp (reno is thoroughly dead), like PRR. A very good read which incorporates a discussion of PRR is

>>> drop will happen, the cwind will be halved (assuming a Reno TCP), 
>>> and the sender will stop sending until one (unloaded) RTT's worth of 
>>> data has been received. At that point the queue will just hit empty 
>>> as the sender starts sending again.
>>>> In practice, statistical multiplexing works just fine with 
>>>> fq_codel, and you do in fact get more throughput with multiple 
>>>> flows in those cases where a single flow fails to each adequate utilisation.
>>>>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> man_listinfo_aqm&d=AwICAg&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fP
> k&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=uxNYRwC4x80YZPJPYo3-
> lMaVCC_1TNJzTxVGd0F1PSs&s=6OQXBky7MzGz1dBmf7oRhwemCi5a4yAiQRm90WHXFIg&
> e=

---

From: Dave Taht <[email-masked]>
Date: Mon, 15 Jun 2015 17:19:52 -0700

> I guess I need to read up on "cake".

some basic doc is at:

The most important thing in cake at the moment is GRO packet peeling,
which turned out desperately needed in all the new router hardware we
have encountered. Huge wins there.

the other stuff is not fully baked or implemented yet. We are in a bit
of a debate about the most troublesome and misunderstood aspect of
codel on the list over there.

> If you have time, can you simulate an RTT of 600 ms?

ok.

> With a few queue drain rates from 1 Mbps to 100 Mbps.

10,50,100,200,300 was my planned range, fed by gigE. I can't go much
faster than than that. (and even as low as 300 requires GRO offloads)

> Would help us "satellite folks" get a better understanding of CoDel parameters.

I have not looked at the very long rtt problem in several years, and
since then tcps, in particular have changed muchly (pacing in
particular)

---

From: Simon Barber <[email-masked]>
Date: Tue, 16 Jun 2015 21:37:54 -0700

The need for a full BDP comes from the way that TCP can release a fill 
BDP in one go, and in fact this happens on a restart of an idle stream. 
CWind is open to a full BDP, no data is in the pipe, so a full cwind 
burst can go out at once. Pacing is the fix for this - hence recent 
discussions of sch_fq.

Auto tuning of target is exactly what I think is needed to mitigate the 
problem with codel limiting available bandwidth usage in certain 
situations. The penalty is a little extra latency, but if used with fq, 
then that extra latency penalty is isolated from other traffic.

Simon

---

From: Simon Barber <[email-masked]>
Date: Tue, 16 Jun 2015 21:46:17 -0700

BTW I did read a paper a while ago disputing the age old rule of thumb 
of BDP/sqrt(connection) for large number of connections. It concluded 
you could get away with much less buffer. I don't have a reference 
unfortunately!

Simon

---

From: Michael Welzl <[email-masked]>
Date: Wed, 17 Jun 2015 09:15:56 +0200

> On 17 Jun 2015, at 06:37, Simon Barber <[email-masked]> wrote:
> 
> The need for a full BDP comes from the way that TCP can release a fill BDP in one go, and in fact this happens on a restart of an idle stream. CWind is open to a full BDP, no data is in the pipe, so a full cwind burst can go out at once. Pacing is the fix for this - hence recent discussions of sch_fq.
> 
> Auto tuning of target is exactly what I think is needed to mitigate the problem with codel limiting available bandwidth usage in certain situations. The penalty is a little extra latency, but if used with fq, then that extra latency penalty is isolated from other traffic.

I agree, with fq, estimating the number of recently active flows shouldn't be hard and such auto-tuning could be feasible

Cheers,
Michael

---

From: "Agarwal, Anil" <[email-masked]>
Date: Wed, 17 Jun 2015 13:59:57 +0000

Michael,

Assuming fq-aqm performs aqm independently on each queue, the estimation of the number of recently active flows must be done for each queue independently.
Not sure how fq would help.
Perhaps, the very fact that we are detecting packets as belonging to flows and mapping them to queues can be used in such an algorithm.
Note that fq does not keep track of flows or their beginnings or ends. It could.
Non-responsive UDP flows and UDP-based TCP-like-transports (e.g., QUIC and utp) would also need to be handled properly.

Alternatively, the current packet drop rate of an adaptive AQM scheme itself could be used to estimate the number of recently active flows. Just a thought.

Regards,
Anil

-----Original Message-----
From: Michael Welzl [mailto:[email-masked]] 
Sent: Wednesday, June 17, 2015 3:16 AM
To: Simon Barber
Cc: Agarwal, Anil; Dave Taht; Jonathan Morton; [email-masked]; Steven Blake
Subject: Re: [aqm] CoDel on high-speed links

Cheers,
Michael

>>> Range of tests from near zero to 300ms RTT codel does quite well 
>>> with reno, better with cubic, on single flows. 4 flows, better. 
>>> fq_codel does better than that on more than X flows in general.
>>> I have a long overdue rant in progress of being wikified about how 
>>> to use netem correctly to properly emulate any rtt you like.
>>> 
>>> I note that a main aqm goal is not maximum bandwidth utilization, 
>>> but maximum bandwidth while still having working congestion 
>>> avoidance and minimal queue depth so other new flows can rapidly 
>>> grab their fair share of the link. The bufferbloat problem was the 
>>> result of wanting maximum bandwidth for single flows.
>>>> The theory is - with a Reno based CC the cwind gets cut in half on 
>>>> a drop. If the drop in cwind is greater than the number of packets 
>>>> in the queue, then the queue will empty out, and the link will then 
>>>> be idle for a
>>> flight + queue.
>>>> while. If you want data to keep the data flowing uninterrupted, 
>>>> then you must have a full unloaded RTT's worth of data in the queue 
>>>> at that point. A
>>> Do the experiment? Recently landed in flent is the ability to 
>>> monitor queue depth while running another test.
>>> 
>>>> and the sender will stop sending until one (unloaded) RTT's worth 
>>>> of data has been received. At that point the queue will just hit 
>>>> empty as the sender starts sending again.
> man_listinfo_aqm&d=AwIFAg&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=5kt8Xa_iBhAuZj65Ag5JmQis9D5Lo2qlJ9dbjkNRVxI&s=PRXEjxkGx9UUrOR8LoJUN0b1oZ49xUO6QuH0rrbnXIE&e=

---

From: Michael Welzl <[email-masked]>
Date: Wed, 17 Jun 2015 17:31:54 +0200

> On 17 Jun 2015, at 15:59, Agarwal, Anil <[email-masked]> wrote:
> 
> Michael,
> 

it could have some notion of recently active flows... and that would of course make the implementation of aqm and fq less independent (who cares)

> Alternatively, the current packet drop rate of an adaptive AQM scheme itself could be used to estimate the number of recently active flows. Just a thought.

true, good idea...

anyway this is getting off into research... but nice thoughts i think!

cheers
michael

---

From: Vishal Misra <[email-masked]>
Date: Wed, 17 Jun 2015 11:38:39 -0400

> On Jun 17, 2015, at 11:31 AM, Michael Welzl <[email-masked]> wrote:
> 

This is exactly what our Self-Tuning PI work does:

http://dna-pubs.cs.columbia.edu/citation/paperfile/79/GLOBECOM_2003_STAQM.pdf

-Vishal
--
http://www.cs.columbia.edu/~misra/