Subject: Re: [aqm] floating a draft charter

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 4 Jun 2013 15:11:14 +0200

Hi Wes,

thanks, that's already very helpful. I was maybe even thinking about a 
milestone on something like 'requirements for initial deployment'... don't 
know if this makes sense though.

Mirja

On Thursday 30 May 2013 18:13:41 Wesley Eddy wrote:
> On 5/30/2013 8:20 AM, Mirja Kuehlewind wrote:
> > But my point is, we should not only to standardize more and more AQM
> > mechanisms because that's a large research area which maybe even should
> > be homed in the IRTF, but we need to find actually deployment strategies.
> > Having a working group on AQM will already help to make people aware of
> > the topic and maybe think about using AQM. But only standardizing more
> > and more AQM queue, might end up in investing a lot of working that
> > no-one will ever use.
>
> I suggest appending something like this to the charter:
>
>   Many AQM algorithms have been proposed in academic literature, but
>   very few are widely implemented and deployed.  A goal of the working
>   group is to produce recommendations that will actually be used, and
>   algorithms that will actually be implemented, deployed in equipment,
>   and enabled.  Towards these ends, the group actively encourages
>   participation from operators and implementers, and will coordinate
>   with the IETF OPS area and other relevant parts of the IETF and
>   Internet community.  Wider research and evaluation of AQM mechanisms
>   shall be coordinated with the IRTF/ICCRG, and significant
>   participation in this WG from the academic and research community is
>   highly desirable, when it is directly relevant to implementation and
>   deployment.
>
>
> We will definitely need to get engagement from some of the operators
> that participate in the IETF, and should solicit participation more
> widely outside as well (e.g. advertise to NANOG list, bufferbloat
> list, etc.) in order to attract operators and implementers that aren't
> already aware of this activity.
>
> Would this help to address your concern?

---

From: Jim Gettys <[email-masked]>
Date: Tue, 4 Jun 2013 11:47:10 -0400

There are a number of things I want to see:

1) packet scheduling is as important as drop/mark algorithms: it is the two
together that is such a win with fq_codel.  So I, for one, would be *very*
unhappy to see a charter that meant it was out of scope to discuss and
document scheduling and flow queuing strategies.  And packet scheduling can
interact with drop mark algorithms.

Optimal scheduling depends on where you are in the network (e.g. a host,
versus home router, vs. ISP head end, etc.); documenting what makes the
most sense where would likely help both implementors of software and
hardware and operators of those networks.  I've given some thought to the
area, but until we have running code, it's hand waving, but probably won't
be hand waving in a year or two.

2) many mark/drop algorithms can co-exist (and generating informational
RFC's to help customers write RFP's is worth while): but so far, there is
no universal "one size fits all" mark/drop algorithm that is optimally
effective everywhere.  Kathy and Van say a primary goal of CoDel was
"first, do no harm", rather than "optimal under all circumstances".  The
goal was an algorithm that never hurt you and would always be safe to have
enabled. CoDel's constants were chosen so that CoDel will work at the edge
of the network with planetary length paths in the face of highly variable
bandwidth.

But CoDel inside a datacenter (unless you mess with its constants) is not
effective in a timely fashion.

People have feared tuneable algorithms such as (W)RED can hurt them, so
what we have had for AQM is sometimes not present at all, or not enabled
(even in nodes that it would help greatly).  Note that one of Dave Taht's
discoveries was that RED had been broken in the Linux source tree for 5
years and *no one had noticed*.

Maybe someday we'll reach nirvana with a drop/mark algorithm that "just
works" everywhere well all the time and can never hurt and always works
perfectly, but that's a lofty goal left for research; in the meanwhile,
helping people understand what to use where in the network and the
tradeoffs is important.

But if CoDel (as we believe) or some other algorithm really is "first, do
no harm" without tuning,  it has great value to be on by default just to
prevent the disaster an operator not enabling any AQM at all; the world
will never be worse than that, and usually be radically better. Anything
that *requires* tuning of knobs to work is a not a candidate....

3) Some algorithms may be hard to retrofit into existing platforms.  For
example, if you can't timestamp a packet, it's really hard to use CoDel,
which is an algorithm easy to implement from scratch, but could be really
hard retrofit in an existing hardware design. I don't want to discourage
people doing things a lot better than nothing as a stop-gap, but people
need to understand that it may be a second best solution.

Again, advice as to what/where algorithms may be applicable will help
designers and operators both.  We face a 5-10 year transition here, even if
we succeed this time.

4) There are "feagtures" or lack of features that today's hardware does
that can/does make life for software very hard. Changes could make life
much easier.

For example, today's hardware makes it hard to do true head drop (which
hardware assist could make trivial).  Some have noticed that at low
bandwidth we may adjust CoDel parameters to work around the lack of head
drop and unaccounted for time in the driver's ring buffers (which CoDel
needs), since Linux's qdisc's can't see the time in the rings. So there are
a set of recommendations to hardware and OS designers that would help the
state of the network: this simple head drop feature, packet pacing, making
GSO/TSO less evil, and so on, war stories on all the places we've been
working on debloating Linux to guide other OS developers, etc.

This is really the other side of queue management: try to encourage traffic
and systems that is less bursty and gentle to the network, so the
AQM/scheduling algorithms don't have to work as hard.

Such things should go someplace in the IETF, and if not here, where?  Doing
nothing anywhere is what I want to avoid.

Best regards,
    Jim

On Thu, May 30, 2013 at 12:13 PM, Wesley Eddy <[email-masked]> wrote:

> be
> > homed in the IRTF, but we need to find actually deployment strategies.
> Having
> > a working group on AQM will already help to make people aware of the
> topic
> > and maybe think about using AQM. But only standardizing more and more AQM
> > queue, might end up in investing a lot of working that no-one will ever
> use.
>
>
> --
> Wes Eddy
> MTI Systems
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: "Rong Pan (ropan)" <[email-masked]>
Date: Wed, 5 Jun 2013 07:09:21 +0000

I understand packet scheduling is important. However, most network devices today treat these two issues separately: classed-based scheduling/queueing and drop/mark algorithms. In class-based queueing, vendors differ in how many classes that they support: 2, 4 or 8. In drop/mark algorithm, vendors tend to implement both tail-drop and WRED.

I think we can continue to treat these two issues separately as it has been done in practice. We can prove the values of delay-based AQM and we can also prove the value of FQ. The goal of AQM is to give early congestion notification before the buffer becomes full or the delay gets big. Scheduling algorithms determine bandwidth allocation and weighted fairness among different classes/flows. We don't have to mandate a particular scheduling algorithm with a particular AQM scheme. Vendors shall be free to choose their best performance-cost tradeoff point. I think it would be challenging to force vendors to adopt a fixed set.

Best Regards,

Rong

From: Jim Gettys <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, June 4, 2013 8:47 AM
To: Wesley Eddy <[email-masked]<mailto:[email-masked]>>
Cc: Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [aqm] floating a draft charter

1) packet scheduling is as important as drop/mark algorithms: it is the two together that is such a win with fq_codel.  So I, for one, would be *very* unhappy to see a charter that meant it was out of scope to discuss and document scheduling and flow queuing strategies.  And packet scheduling can interact with drop mark algorithms.

Optimal scheduling depends on where you are in the network (e.g. a host, versus home router, vs. ISP head end, etc.); documenting what makes the most sense where would likely help both implementors of software and hardware and operators of those networks.  I've given some thought to the area, but until we have running code, it's hand waving, but probably won't be hand waving in a year or two.

2) many mark/drop algorithms can co-exist (and generating informational RFC's to help customers write RFP's is worth while): but so far, there is no universal "one size fits all" mark/drop algorithm that is optimally effective everywhere.  Kathy and Van say a primary goal of CoDel was "first, do no harm", rather than "optimal under all circumstances".  The goal was an algorithm that never hurt you and would always be safe to have enabled. CoDel's constants were chosen so that CoDel will work at the edge of the network with planetary length paths in the face of highly variable bandwidth.

But CoDel inside a datacenter (unless you mess with its constants) is not effective in a timely fashion.

People have feared tuneable algorithms such as (W)RED can hurt them, so what we have had for AQM is sometimes not present at all, or not enabled (even in nodes that it would help greatly).  Note that one of Dave Taht's discoveries was that RED had been broken in the Linux source tree for 5 years and *no one had noticed*.

Maybe someday we'll reach nirvana with a drop/mark algorithm that "just works" everywhere well all the time and can never hurt and always works perfectly, but that's a lofty goal left for research; in the meanwhile, helping people understand what to use where in the network and the tradeoffs is important.

But if CoDel (as we believe) or some other algorithm really is "first, do no harm" without tuning,  it has great value to be on by default just to prevent the disaster an operator not enabling any AQM at all; the world will never be worse than that, and usually be radically better. Anything that *requires* tuning of knobs to work is a not a candidate....

3) Some algorithms may be hard to retrofit into existing platforms.  For example, if you can't timestamp a packet, it's really hard to use CoDel, which is an algorithm easy to implement from scratch, but could be really hard retrofit in an existing hardware design. I don't want to discourage people doing things a lot better than nothing as a stop-gap, but people need to understand that it may be a second best solution.

Again, advice as to what/where algorithms may be applicable will help designers and operators both.  We face a 5-10 year transition here, even if we succeed this time.

4) There are "feagtures" or lack of features that today's hardware does that can/does make life for software very hard. Changes could make life much easier.

For example, today's hardware makes it hard to do true head drop (which hardware assist could make trivial).  Some have noticed that at low bandwidth we may adjust CoDel parameters to work around the lack of head drop and unaccounted for time in the driver's ring buffers (which CoDel needs), since Linux's qdisc's can't see the time in the rings. So there are a set of recommendations to hardware and OS designers that would help the state of the network: this simple head drop feature, packet pacing, making GSO/TSO less evil, and so on, war stories on all the places we've been working on debloating Linux to guide other OS developers, etc.

This is really the other side of queue management: try to encourage traffic and systems that is less bursty and gentle to the network, so the AQM/scheduling algorithms don't have to work as hard.

Such things should go someplace in the IETF, and if not here, where?  Doing nothing anywhere is what I want to avoid.

Best regards,
    Jim

On Thu, May 30, 2013 at 12:13 PM, Wesley Eddy <[email-masked]<mailto:[email-masked]>> wrote:
On 5/30/2013 8:20 AM, Mirja Kuehlewind wrote:
> mechanisms because that's a large research area which maybe even should be
> homed in the IRTF, but we need to find actually deployment strategies. Having
> a working group on AQM will already help to make people aware of the topic
> queue, might end up in investing a lot of working that no-one will ever use.

--
Wes Eddy
MTI Systems
_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

---

From: Dave Taht <[email-masked]>
Date: Wed, 5 Jun 2013 00:44:00 -0700

On Wed, Jun 5, 2013 at 12:09 AM, Rong Pan (ropan) <[email-masked]> wrote:

>  I understand packet scheduling is important. However, most network
> devices today treat these two issues separately: classed-based
> scheduling/queueing and drop/mark algorithms. In class-based queueing,
> vendors differ in how many classes that they support: 2, 4 or 8. In
> drop/mark algorithm, vendors tend to implement both tail-drop and WRED.
>

I really have a great deal of trouble extrapolating to "vendors" here. I
suppose you mean "big iron vendors", like Juniper, or Allied Telesis, which
haven't weighed in on this list yet. In the CMTS market there are three
players, and while I know what Arris can do, I would prefer an Arris
representative to say. In terms of chipset makers ethernet support, most
don't don't support anything in hardware, neither do the low end bridge
chips, and thus they rely on software to do either AQM or FQ, in addition
to doing useful things like firewalling or NAT. In servers, home routers,
and handhelds, linux based devices are prevailant and can easily do just
about anything. (I recently had a theorist badger me about doing a particle
model for scheduling packets, because the GPU is effectively free on many
common chipsets. I'm sooo not going there, but he is). Wifi chipsets
generally have no strategy other than 802.11e for managing queues. Most of
the France Telecom's dsl devices run shortest queue first, with longest
queue drop. The list goes on and on...

>  I think we can continue to treat these two issues separately as it has
> been done in practice.
>

 Scheduling and drop algorithms have long been used in combination in
practice. Examples include the venerable wondershaper, and openwrt's QoS
system, which for most of the last 7 years or so was HFSC + SFQ + RED.
Dozens of other examples exist, but not on the gear you are familiar with.

> We can prove the values of delay-based AQM and we can also prove the value
> of FQ.
>

Sure. Thing is, we've pretty much proven that the two in combination are
better than either, singly. Do you not agree with that assessment?

> The goal of AQM is to give early congestion notification before the buffer
> becomes full or the delay gets big.
>

If you want to define AQM as Active Queue LENGTH management, then go right
ahead and try. Actually I was thinking hard about expanding the scope of
this proposed working group to be "Smart Queue Management" - which neatly
not only encompasses FQ and queue length management, but also extends out
to nagging problems like sanely rate limiting ipv6 ICMP responses to
someone probing someone's entire /64 address space, and dealing with other
sorts of attack like that, and problems like incast.

Similarly, the problem of dealing with diffserv marking remains, as to how
(or if) it would correctly interact with a SQM system.

Things like iptables don't seem to have a home within the ietf presently (?)

Scheduling algorithms determine bandwidth allocation and weighted fairness
> among different classes/flows. We don't have to mandate a particular
> scheduling algorithm with a particular AQM scheme.
>

Concur with that very much. I continue to experiment with QFQ+, DRR, WFQ,
SFQ, and other techniques.

> Vendors shall be free to choose their best performance-cost tradeoff
> point. I think it would be challenging to force vendors to adopt a fixed
> set.
>

Concur. Let the best set of ideas win both here and in the marketplace! But
an end goal should be some effective form of SQM *on by default*, on
everything.

>  Best Regards,
>
>  Rong
>
>
>
>   From: Jim Gettys <[email-masked]>
> To: Wesley Eddy <[email-masked]>
> Cc: Mirja Kuehlewind <[email-masked]>, "
> [email-masked]" <[email-masked]>
>
>  1) packet scheduling is as important as drop/mark algorithms: it is the
> two together that is such a win with fq_codel.  So I, for one, would be
> *very* unhappy to see a charter that meant it was out of scope to discuss
> and document scheduling and flow queuing strategies.  And packet scheduling
> can interact with drop mark algorithms.
>
>  But if CoDel (as we believe) or some other algorithm really is "first,
> do no harm" without tuning,  it has great value to be on by default just to
>  This is really the other side of queue management: try to encourage
> traffic and systems that is less bursty and gentle to the network, so the
>  Such things should go someplace in the IETF, and if not here, where?
>  Doing nothing anywhere is what I want to avoid.
>
>  Best regards,
>     Jim
>
>
>

-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

---

From: Andrew Mcgregor <[email-masked]>
Date: Wed, 5 Jun 2013 18:28:04 +1000

> drop/mark algorithm, vendors tend to implement both tail-drop and WRED.

I'm aware of chips, not including software forwarding, with 1, 2, 4, 5, 6,
8, 9, 16, 64 and arbitrary numbers of classes limited by available memory,
with strict- or soft-priority, DRR, WRR, and drop/mark algorithms including
tail-drop, head-drop, WRED, priority- or class- weighted buffer pressure
management with head or tail eviction, and that's without using rate ACL
mechanisms.  So even restricting the scope to ethernet switch chips there
is far more variety than you're suggesting.

Software can of course do nearly anything, and given that these days
software forwarding is slightly cheaper than silicon forwarding even up to
around 32x10G levels, it is entirely realistic to expect to run software
forwarding in many parts of the network.

> I think we can continue to treat these two issues separately as it has

Any form of buffer pressure management that takes classes into account
couples these, and two widely deployed switch silicon families that I know
of have that.  That amounts to weak AQM.

> of FQ. The goal of AQM is to give early congestion notification before the
> buffer becomes full or the delay gets big. Scheduling algorithms determine
> bandwidth allocation and weighted fairness among different classes/flows.
> We don't have to mandate a particular scheduling algorithm with a
> particular AQM scheme. Vendors shall be free to choose their best
> performance-cost tradeoff point. I think it would be challenging to force
> vendors to adopt a fixed set.
>
>  Best Regards,
>
>  Rong
>
>
More than challenging, that's impossible.  Not to mention completely
undesirable, because we're well aware that anything we have right now is
imperfect.

Andrew

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 8143 7128

---

From: Luca MUSCARIELLO <[email-masked]>
Date: Wed, 05 Jun 2013 10:44:54 +0200

AFAIK, Chipcos for home gateways (Broadcom, Ikanos, Intel), network 
processors (Cavium),
but also Cisco, Juniper and Alcatel-Lucent make a difference between 
class based queuing and
dropping algorithms. The first is implemented everywhere in different 
ways and I do not think
there would be relevant work to do there (except for HTB in Linux which 
is really bad in terms
of performance, IMHO).

Dropping algorithms is used frequently as a term to describe AQM, e.g. 
in CLI based configuration
systems you may find (in commercial Linux based products) SFQ among the 
dropping algorithms.

It is clearly an abuse of language but common practice.  I unfortunately 
agree with Dave, and we discussed about
that in Paris, that chipcos do not take care about flow based queue 
management (typically withing a given class).
Only Linux  does that, but many products make use of Linux only for the 
slow path while the fast path is managed in hardware.

Deploying QoS (including de-bloating in QoS) based on Linux is soon 
broken by the trend to distribute packet processing
in dedicated processing units and  chipco ignore completely queue 
management there.
Anytime I ask whether the chip implements deficit round round robin the 
answer is "of course!" and in the
end it  turns out to be *modified* deficit round robin, i.e. class based 
queuing with up to say 64 queues.
Which is not what I am looking for.

One solution would be to move everything to software, but the cost of 
the chip that guarantees the same switching
throughput is much higher.

I would suggest to put more emphasis that something new is being 
proposed here adding
flow based packet scheduling as a novel item in the charter (in addition 
to class based, dropping or
congestion notification).  Using Smart Queue Management seems misleading 
to me or not enough.

Best regards,
Luca

-- 
France Telecom R&D - Orange Labs
MUSCARIELLO Luca - NMP/TRM
38 - 40, rue du General Leclerc
92794 Issy Les Moulineaux Cedex 9 - France
Tel : +33 (0)1 45 29 60 37
http://perso.rd.francetelecom.fr/muscariello

On 06/05/2013 09:44 AM, Dave Taht wrote:
>
>
> On Wed, Jun 5, 2013 at 12:09 AM, Rong Pan (ropan) <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     scheduling/queueing and drop/mark algorithms. In class-based
>     queueing, vendors differ in how many classes that they support: 2,
>     4 or 8. In drop/mark algorithm, vendors tend to implement both
>     tail-drop and WRED.
>
>
> I really have a great deal of trouble extrapolating to "vendors" here. 
> I suppose you mean "big iron vendors", like Juniper, or Allied 
> Telesis, which haven't weighed in on this list yet. In the CMTS market 
> there are three players, and while I know what Arris can do, I would 
> prefer an Arris representative to say. In terms of chipset makers 
> ethernet support, most don't don't support anything in hardware, 
> neither do the low end bridge chips, and thus they rely on software to 
> do either AQM or FQ, in addition to doing useful things like 
> firewalling or NAT. In servers, home routers, and handhelds, linux 
> based devices are prevailant and can easily do just about anything. (I 
> recently had a theorist badger me about doing a particle model for 
> scheduling packets, because the GPU is effectively free on many common 
> chipsets. I'm sooo not going there, but he is). Wifi chipsets 
> generally have no strategy other than 802.11e for managing queues. 
> Most of the France Telecom's dsl devices run shortest queue first, 
> with longest queue drop. The list goes on and on...
>
>
>
>     I think we can continue to treat these two issues separately as it
>     has been done in practice.
>
> practice. Examples include the venerable wondershaper, and openwrt's 
> QoS system, which for most of the last 7 years or so was HFSC + SFQ + 
> RED. Dozens of other examples exist, but not on the gear you are 
> familiar with.
>
>     We can prove the values of delay-based AQM and we can also prove
>     the value of FQ.
>
>
> Sure. Thing is, we've pretty much proven that the two in combination 
> are better than either, singly. Do you not agree with that assessment?
>
>     The goal of AQM is to give early congestion notification before
>     the buffer becomes full or the delay gets big.
>
>
> If you want to define AQM as Active Queue LENGTH management, then go 
> right ahead and try. Actually I was thinking hard about expanding the 
> scope of this proposed working group to be "Smart Queue Management" - 
> which neatly not only encompasses FQ and queue length management, but 
> also extends out to nagging problems like sanely rate limiting ipv6 
> ICMP responses to someone probing someone's entire /64 address space, 
> and dealing with other sorts of attack like that, and problems like 
> incast.
>
> Similarly, the problem of dealing with diffserv marking remains, as to 
> how (or if) it would correctly interact with a SQM system.
>
>     Scheduling algorithms determine bandwidth allocation and weighted
>     fairness among different classes/flows. We don't have to mandate a
>     particular scheduling algorithm with a particular AQM scheme.
>
>
> Concur with that very much. I continue to experiment with QFQ+, DRR, 
> WFQ, SFQ, and other techniques.
>
>     Vendors shall be free to choose their best performance-cost
>     tradeoff point. I think it would be challenging to force vendors
>     to adopt a fixed set.
>
>
> Concur. Let the best set of ideas win both here and in the 
> marketplace! But an end goal should be some effective form of SQM *on 
> by default*, on everything.
>
>
>     Best Regards,
>
>     Rong
>
>
>
>     From: Jim Gettys <[email-masked] <mailto:[email-masked]>>
>     To: Wesley Eddy <[email-masked] <mailto:[email-masked]>>
>     Cc: Mirja Kuehlewind <[email-masked]
>     <mailto:[email-masked]>>, "[email-masked]
>     <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
>
>     1) packet scheduling is as important as drop/mark algorithms: it
>     is the two together that is such a win with fq_codel.  So I, for
>     one, would be *very* unhappy to see a charter that meant it was
>     out of scope to discuss and document scheduling and flow queuing
>     strategies.  And packet scheduling can interact with drop mark
>     algorithms.
>
>     Optimal scheduling depends on where you are in the network (e.g. a
>     host, versus home router, vs. ISP head end, etc.); documenting
>     what makes the most sense where would likely help both
>     implementors of software and hardware and operators of those
>     networks.  I've given some thought to the area, but until we have
>     running code, it's hand waving, but probably won't be hand waving
>     in a year or two.
>
>     2) many mark/drop algorithms can co-exist (and generating
>     informational RFC's to help customers write RFP's is worth while):
>     but so far, there is no universal "one size fits all" mark/drop
>     algorithm that is optimally effective everywhere.  Kathy and Van
>     say a primary goal of CoDel was "first, do no harm", rather than
>     "optimal under all circumstances".  The goal was an algorithm that
>     never hurt you and would always be safe to have enabled. CoDel's
>     constants were chosen so that CoDel will work at the edge of the
>     network with planetary length paths in the face of highly variable
>     bandwidth.
>
>     People have feared tuneable algorithms such as (W)RED can hurt
>     them, so what we have had for AQM is sometimes not present at all,
>     or not enabled (even in nodes that it would help greatly).  Note
>     that one of Dave Taht's discoveries was that RED had been broken
>     in the Linux source tree for 5 years and *no one had noticed*.
>
>     Maybe someday we'll reach nirvana with a drop/mark algorithm that
>     "just works" everywhere well all the time and can never hurt and
>     always works perfectly, but that's a lofty goal left for research;
>     in the meanwhile, helping people understand what to use where in
>     the network and the tradeoffs is important.
>
>     But if CoDel (as we believe) or some other algorithm really is
>     "first, do no harm" without tuning,  it has great value to be on
>     by default just to prevent the disaster an operator not enabling
>     any AQM at all; the world will never be worse than that, and
>     usually be radically better. Anything that *requires* tuning of
>     knobs to work is a not a candidate....
>
>     3) Some algorithms may be hard to retrofit into existing
>     platforms.  For example, if you can't timestamp a packet, it's
>     really hard to use CoDel, which is an algorithm easy to implement
>     from scratch, but could be really hard retrofit in an existing
>     hardware design. I don't want to discourage people doing things a
>     lot better than nothing as a stop-gap, but people need to
>     understand that it may be a second best solution.
>
>     Again, advice as to what/where algorithms may be applicable will
>     help designers and operators both.  We face a 5-10 year transition
>     here, even if we succeed this time.
>
>     4) There are "feagtures" or lack of features that today's hardware
>     does that can/does make life for software very hard. Changes could
>     make life much easier.
>
>     For example, today's hardware makes it hard to do true head drop
>     (which hardware assist could make trivial).  Some have noticed
>     that at low bandwidth we may adjust CoDel parameters to work
>     around the lack of head drop and unaccounted for time in the
>     driver's ring buffers (which CoDel needs), since Linux's qdisc's
>     can't see the time in the rings. So there are a set of
>     recommendations to hardware and OS designers that would help the
>     state of the network: this simple head drop feature, packet
>     pacing, making GSO/TSO less evil, and so on, war stories on all
>     the places we've been working on debloating Linux to guide other
>     OS developers, etc.
>
>     This is really the other side of queue management: try to
>     encourage traffic and systems that is less bursty and gentle to
>     the network, so the AQM/scheduling algorithms don't have to work
>     as hard.
>
>     Such things should go someplace in the IETF, and if not here,
>     where?  Doing nothing anywhere is what I want to avoid.
>
>     Best regards,
>         Jim
>
>
>
>         [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Jim Gettys <[email-masked]>
Date: Wed, 5 Jun 2013 09:19:36 -0400

On Wed, Jun 5, 2013 at 4:44 AM, Luca MUSCARIELLO <
[email-masked]> wrote:

> but also Cisco, Juniper and Alcatel-Lucent make a difference between class
> based queuing and
> dropping algorithms. The first is implemented everywhere in different ways
> and I do not think
> there would be relevant work to do there (except for HTB in Linux which is
> really bad in terms
> of performance, IMHO).
>
> Dropping algorithms is used frequently as a term to describe AQM, e.g. in
> CLI based configuration
> Deploying QoS (including de-bloating in QoS) based on Linux is soon broken
> by the trend to distribute packet processing
> One solution would be to move everything to software, but the cost of the
> chip that guarantees the same switching

There are two points I'll react to in this statement:

    1) performance: in the edge of the network, the SOC's used are already
very fast and more than fast enough to support algorithms such as fq_codel
(much faster than one's wireless or broadband connection).

The existence proofs are running code in OpenWrt on commodity home router
SOC's. And when I asked Eric Dumazet about fq_codel performance, he said I
could quote him as it taking 2% of a modern x86 CPU core at 10G speed.
 This was the point at which I got more than moderately excited about this
class of algorithms. CPU's are not what they were in the 1980's and 1990's,
and we have to get over that and move on.

So if hardware assist is not assisting, we don't have to use it, and it may
encourage hardware people to "get it right" the next generation.

   2) most application's performance is now bound by latency, and not
bandwidth.  Your web browsing doesn't get appreciably faster adding
bandwidth, but improving the latency, even if there is not competing
traffic, does improve performance, not to mention improving everything
across the board and for most things getting us out of the impossible to
configure QOS morass we're in.

As I said, what may be feasible or advisable to do at what point in the
network may vary: I don't think this group will *mandate* much at all, but
describing what can be used where for best effect is worth a lot, and
*possibly* mandating a "first, do no harm" algorithm, even if far from
ideal in many circumstances, to prevent no AQM being present at all, which
is a headache in today's network.

We're going to have a large number of different drop and queuing algorithms
over the coming years, I predict.
                                                - Jim

> I would suggest to put more emphasis that something new is being proposed
> here adding
> Tel : +33 (0)1 45 29 60 37http://perso.rd.francetelecom.fr/muscariello
>
>
>
>>  We can prove the values of delay-based AQM and we can also prove the
>> value of FQ.
>>
>
>>  The goal of AQM is to give early congestion notification before the
>> buffer becomes full or the delay gets big.
>>
>
> Concur. Let the best set of ideas win both here and in the marketplace!
> But an end goal should be some effective form of SQM *on by default*, on
> everything.
>
>
>>  Best Regards,
>>
>>  Rong
>>
>>
>>
>> host, versus home router, vs. ISP head end, etc.); documenting what makes
>> the most sense where would likely help both implementors of software and
>>  3) Some algorithms may be hard to retrofit into existing platforms.
>>  For example, if you can't timestamp a packet, it's really hard to use
>> CoDel, which is an algorithm easy to implement from scratch, but could be
>> really hard retrofit in an existing hardware design. I don't want to
>> discourage people doing things a lot better than nothing as a stop-gap, but
>> people need to understand that it may be a second best solution.
>>
>> On Thu, May 30, 2013 at 12:13 PM, Wesley Eddy <[email-masked]>wrote:
>>
> aqm mailing [email-masked]://www.ietf.org/mailman/listinfo/aqm
>
>
>

---

From: MUSCARIELLO Luca OLNC/OLN <[email-masked]>
Date: Wed, 05 Jun 2013 16:18:29 +0200

On 06/05/2013 03:19 PM, Jim Gettys wrote:
>
> On Wed, Jun 5, 2013 at 4:44 AM, Luca MUSCARIELLO 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>
>     AFAIK, Chipcos for home gateways (Broadcom, Ikanos, Intel),
>     network processors (Cavium),
>     but also Cisco, Juniper and Alcatel-Lucent make a difference
>     between class based queuing and
>     dropping algorithms. The first is implemented everywhere in
>     different ways and I do not think
>     there would be relevant work to do there (except for HTB in Linux
>     which is really bad in terms
>     of performance, IMHO).
>
>     Dropping algorithms is used frequently as a term to describe AQM,
>     e.g. in CLI based configuration
>     systems you may find (in commercial Linux based products) SFQ
>     among the dropping algorithms.
>
>     It is clearly an abuse of language but common practice.  I
>     unfortunately agree with Dave, and we discussed about
>     that in Paris, that chipcos do not take care about flow based
>     queue management (typically withing a given class).
>     Only Linux  does that, but many products make use of Linux only
>     for the slow path while the fast path is managed in hardware.
>
>     Deploying QoS (including de-bloating in QoS) based on Linux is
>     soon broken by the trend to distribute packet processing
>     Anytime I ask whether the chip implements deficit round round
>     robin the answer is "of course!" and in the
>     end it  turns out to be *modified* deficit round robin, i.e. class
>     based queuing with up to say 64 queues.
>     One solution would be to move everything to software, but the cost
>     of the chip that guarantees the same switching
>     1) performance: in the edge of the network, the SOC's used are 
> already very fast and more than fast enough to support algorithms such 
> as fq_codel (much faster than one's wireless or broadband connection).
>
> The existence proofs are running code in OpenWrt on commodity home 
> router SOC's. And when I asked Eric Dumazet about fq_codel 
> performance, he said I could quote him as it taking 2% of a modern x86 
> CPU core at 10G speed.  This was the point at which I got more than 
> moderately excited about this class of algorithms. CPU's are not what 
> they were in the 1980's and 1990's, and we have to get over that and 
> move on.

For what concerns performance, I am thinking about the following kind of 
home gateway.
A triple play offer (voice, Internet, TV)  deployed using a MIPS based 
gateway.
It incorporates 4 Gigabit Ethernet ports, wifi @2.5GHz b/g/n, wifi @5GHz 
a/n,
of course USB, SATA for your local storage.
It must support FTTH and xDSL, IPv4/6 plus some software like samba, DLNA.

For instance:
http://www.ikanos.com/products/communications-processors/fusiv-vx185183/

Packet processing is distributed in hardware to let the central unit 
take care of other stuff.
Do not think to have a multi-core Intel processor there.

You can implement your packet scheduling in hardware (micro code, 
software in the end)
but it is *not* an open environment. You code your queuing now, the 
environment evolves
and nobody maintains your code which is broken as the micro-code evolves.

So the problem is not really software/hardware but open/closed and 
re-usability
(Linux matters not only because it is open but also because it is used 
everywhere).

>
> So if hardware assist is not assisting, we don't have to use it, and 
> it may encourage hardware people to "get it right" the next generation.
>
> across the board and for most things getting us out of the impossible 
> to configure QOS morass we're in.
>
> As I said, what may be feasible or advisable to do at what point in 
> the network may vary: I don't think this group will *mandate* much at 
> all, but describing what can be used where for best effect is worth a 
> lot, and *possibly* mandating a "first, do no harm" algorithm, even if 
> far from ideal in many circumstances, to prevent no AQM being present 
> at all, which is a headache in today's network.
>
> We're going to have a large number of different drop and queuing 
> algorithms over the coming years, I predict.

Yes, I hope. But if you want them deployed I suspect we'll wait to have 
either open hardware
or to have *true* FQ  in the chip of home gateways. Also because most of 
the delay is measured
in the user's home uplink. Having everything in software is easy and 
useful but does not cover
most of the current market.

Best regards
Luca