Subject: Re: [aqm] think once to mark, think twice to drop: draft-ietf-aqm-ecn-benefits-02

From: Bob Briscoe <[email-masked]>
Date: Mon, 13 Apr 2015 16:11:15 +0100

David,

Returning from a fortnight offlist...

I think your conception of how ECN works is incorrect. You describe 
ECN as if the AQM marks one packet when it drops another packet. You 
say that the ECN-mark speeds up the retransmission of the dropped 
packet. On the contrary, the idea of classic ECN [RFC3168] is that 
the ECN marks replace the drops. In all known testing (except 
pathological cases), classic ECN effectively eliminates drops for all 
ECN-capable packets.

Nonetheless, I do agree with your sentiment that the perfect is the 
enemy of the good. We can remove most of the really bloat-induced 
latency without ECN. So the message must be clear: Deploy AQM now. No 
need to wait for ECN. But implementations SHOULD allow ECN packets to 
be classifed into a separately configurable instance of the AQM algo. {Note 1}

Similarly, this WG made sure we did not deprecate RED in the AQM 
recommendations. Because, in existing equipment, even a poorly tuned 
RED is usually much better than a bloated buffer with no AQM.

Just as the WG mustn't confuse messages, you mustn't get confused by 
a discussion about the potential for more ambitious reductions in 
latency. Koen started the thread with reference to our presentation 
in the ICCRG in the IRTF, where 'R' in both cases stands for Research.

And I believe it is valid for the ECN benefits draft (in the IETF AQM 
WG) to point to the potential of ECN, by using insights from research 
in progress.

There are a number of different contributions to unnecessary latency, 
not just the one popularised as bufferbloat:

[In the following, I will use the term 'short message(s)' as 
shorthand for either a short interactive flow or a long flow 
consisting of short interactive messages (like in a game), or video 
frames or voice datagrams or anything where the perceived latency 
depends on the latency of each 'message', especially a string of 
messages with serial dependency as is typical in Web.]

#1 The well-known bufferbloat problem, where a long-running flow 
fills a bloated buffer, delays short messages.
   - AQM and/or flow queuing can remove this delay, without needing ECN.

#2 A loss causes head-of-line blocking for short message(s) while 
waiting for the retransmission.
   - AQM without ECN cannot remove this delay.
   - Flow queuing cannot remove this delay, if the losses are self-induced.
   - FEC can remove this delay without needing ECN, but the increased 
redundancy is equivalent to poorer utilisation (altho only the short 
flows need the redundancy).
   - ECN can remove this delay.

#3 A loss near the end of a short flow can lead to multi-RTT delay.
   - AQM without ECN cannot remove this delay.
   - Techniques like tail-loss probe and RTO-restart can mitigate 
this delay, without ECN, but not remove it.
   - ECN can remove this delay.

#4 The Reno/Cubic sawtooth causes variation in delay between 1 and 2 
base RTTs. This can affect short messages.
   - AQM without ECN cannot remove this delay unless configured to 
sacrifice utilisation.
   - Flow queuing removes this delay variation if caused by a 
separate long-running flow.{Note 2}
   - A change to the TCP algo (e.g. DCTCP) can remove this delay 
variation. Smaller sawteeth imply a much higher signalling rate, 
which in turn requires ECN, otherwise drop probability would be 
excessive. (This was the main point Koen was making.)
   - Therefore, ECN can remove this delay.

#5 Slow-starts{Note 3} cause spikes in delay.
   - AQM without ECN cannot remove this delay, and typically AQM is 
designed to allow such bursts of delay in the hope they will 
disappear of their own accord.
   - Flow queuing can remove the effect of these delay bursts on 
other flows, but only if it gives all flows a separate queue from the 
start.{Note 2}
   - Delay-based softening of slow-start, such as Hybrid Slow-Start 
in Linux, can mitigate these variations, but with increased risk of 
coming out of SS early, causing significantly longer completion time.
   - ECN with AQM based on the instantaneous queue limits this delay, 
without the risk of longer completion time.

#6 Slow-starts{Note 3} can cause runs of losses, which in turn cause delays.
   - AQM without ECN cannot remove these delays.
   - Flow queuing cannot remove these losses, if self-induced.
   - Delay-based SS like HSS can mitigate these losses, with 
increased risk of longer completion time.
   - ECN can remove these losses, and the consequent delays.

Summary:
* AQM alone solves the main problem
* Flow queuing solves or mitigates most of the remaining secondary problems.
* ECN has the potential to solve all the remaining secondary problems 
(pending further research to prove some of them).

Whether flow queuing is applicable depends on the scale. The work I'm 
doing with Koen is to reduce the cost of the queuing mechanisms on 
our BNGs (broadband network gateways). We're trying to reduce the 
cost of per-customer queuing at scale, so per-flow queuing is simply 
out of the question. Whereas ECN requires no more processing than drop.

ECN has potential cheating problems, but we have per-customer queues anyway.
Using flow as the unit of allocation also has its own problems, with 
no proposed solutions.

Bob

{Note 1}: Your general point is that the perfect can be the enemy of 
the good. Here's the presentation I gave in TSVAREA straight after VJ 
presented CoDel in 2012 entitled "DCTCP & CoDel; the Best is the 
Friend of the Good."
<http://www.bobbriscoe.net/presents/1207ietf/1207-tsvarea-dctcp.pdf>

{Note 2}:  A lone flow can cause this delay variation to itself, but 
that's irrelevant because, if the delay were not in the network it 
would be at the sender.

{Note 3}: Delay and loss spikes can equivalently be caused when 
Cubic's window rises to seek out newly available capacity after 
another flow finishes or the link rate varies.

At 05:16 30/03/2015, David Lang wrote:
>On Sat, 28 Mar 2015, Scheffenegger, Richard wrote:
>
>>David,
>>
>>Perhaps you would care to provide some text to address the 
>>misconception that you pointed out? (To wait for a 100% fix as a 
>>90% fix appears much less appealing, while the current state of art is at 0%)
>
>Ok, you put me on the spot :-) Here goes.
>
>>If you think that aqm-recommendations is not strogly enough worded. 
>>I think this particular discussion (to aqm or not) really belongs 
>>there. The other document (ecn benefits) has a different target in 
>>arguing for going those last 10%...
>
>so here is my "elevator pitch" on the problem. Feel free to take 
>anything I say here for any purpose, and I'm sure I'll get corrected 
>for anything I am wong on
>
>Problem statement: Transmit buffers are needed to keep the network 
>layer fully utilized, but excessive buffers result in poor latency 
>for all traffic. This latency is frequently bad enough to cause some 
>types of traffic to fail entirely.
>
>   <link to more background goes here, including how separate 
> benchmarks for throughput and latency have mislead people, "packet 
> loss considered evil", cheaper memory encouraging larger buffers, 
> etc. Include tests like netperf-wrapper and ping latency while 
> under load, etc. Include examples where buffers have resulted in 
> latencies so long that packets are retransmitted before the first 
> copy gets to the destination>
>
>Traditionally, transmit buffers have been sized to handle a fixed 
>number of packets. Due to teh variation in packet sizes, it is 
>impossible to tune this value to both keep the link fully utilized 
>when small packets dominate the trafific without having the queue 
>size be large enough to cause latency problems when large packets 
>dominate the traffic.
>
>Shifting to Byte Queue Lengths where queues are allowed to hold a 
>variable number of packets depending on how large they are makes it 
>possible to manually tune the transmit buffer size to get good 
>latency under all traffic conditions at a given speed. However, this 
>step forward revealed two additional problems.
>
>1. whenever the data rate changes, this value needs to be manually 
>changed (multi-link paths loose a link, noise degrades max 
>throughput on a link, etc)
>
>2. high volume flows (i.e. bulk downloads) can starve other flows 
>(DNS lookups, VoIP, Gaming, etc). this happens because space in tue 
>queue is on a first-com-first-served basis, so the high-volume 
>traffic fills the queue (at which point it starts to be dropped), 
>but all other traffic that tries to arrive is also dropped. It turns 
>out that these light flows tend to have a larger effect on the user 
>experience than heavier flows, because things tend to be serialized 
>behind the lighter flows (DNS lookup before doing a large download, 
>retrieving a small HTML page to find what additional resources need 
>to be fetched to display a page), or the user experience is directly 
>effected by light flows (gaming lag, VoIP drops, etc)
>
>Active Queue Management addresses these problems by adapting the 
>amount of data that is buffered to match the data transmission 
>capacity, and prevents high volume flows from starving low-volume 
>flows without the need to implement QoS classifications.
>
>   <insert link about how you can't trust QoS tags that are made by 
> other organizations, ways that it can be abused, etc>
>
>This is possible because AQM algoithms don't have to drop the new 
>packet that arrives, the algorithm can decide to drop the packet for 
>one of the heavy flows rather than for one of the lightweight flows.
>
><insert references to currently favored AQM options here, PIE, 
>fq_codel, cake, ???. Also links to failed approaches>
>
>Turning on aqm on every bottleneck link makes the Internet usable 
>for everyone, no matter what sort of application they are using.
>
><insert link on how to deal with equipment you can't configure by 
>throttling bandwidth before the bottleneck oand/or doing ingress 
>shaping of traffic>
>
>
>While AQM makes the network usable, there is still additional room 
>for improvement. While dropping packets does result in the TCP 
>senders slowing down,and eventually stabilizing at around the right 
>speed to keep the link fully utilized, the only way that senders 
>have been able to detect problems is to discover that they have not 
>received an ack for the traffic within the allowed time. This causes 
>a 'bubble' in the flow as teh dropped packet must be retransmitted 
>(and sometimes a significant amount of data after the dropped packet 
>that did make it to the destination, but could not be acked because 
>fo the missing packet).
>
>This "bubble" in the data flow can be greatly compressed by 
>configuring the AQM algorithm to send an ECN packet to the sender 
>when it drops a packet in a flow. The sender can then adapt faster, 
>slowing down it's new data, and re-sending the dropped packet 
>without having to wait for the timeout. This has two major effects 
>by allowing the sender to retransmit the packet sooner the dealy on 
>the dropped data is not as long, and because the replacement data 
>can arrive before the timeout of the following packets, they may not 
>need to be re-sent. by configuring the AQM algorithm to send the ECN 
>notification to the sender only when the packet is being dropped, 
>the effect of failure of the ECN packet to get through to the sender 
>(the notification packet runs into congestion and gets dropped, some 
>network device blocks it, etc) is that the ECN enabled case devolves 
>to match the non-ECN case in that the sender will still detect the 
>dropped packet via the timeout waiting for the ack as if ENCN was not enabled.
>
><insert link to possible problems that can happen here, including 
>the potential for an app to 'game' things if packets are marked at a 
>different level than when they are dropped.>
>
>
>So a very strong recommendation to enable Active Queue Management, 
>while the different algorithms have different advantages and levels 
>of testing, even the 'worst' of the set results in a night-and-day 
>improvement for usability compared to unmanaged buffers.
>
>Enabling ECN at the same point as dropping packets as part of 
>enabling any AQM algorithm results in a noticable improvement over 
>the base algorithm without ECN. When compared to the baseline, the 
>improvement added by ECN is tiny compared to the improvement from enabling AQM.
>
>   Is it fair to say that plain aqm vs aqm+ecn variation is on the 
> same order of difference as the differences between the different 
> AQM algorithms?
>
>
>
>Future research items (which others here may already have done, and 
>would not be part of my 'elevator pitch')
>
>I believe that currently ECn triggers the exact same slowdown that a 
>missed packet does, and it may be appropriate to have the sender do 
>a less drastic slowdown.
>
>It would be very interesing to provide soem way for the application 
>sending the traffic to detect dropped packets and ECN responses. For 
>example, a streaming media source (especially an interactive one 
>like video conferencing) could adjust the bitrate that it's sending.
>
>David Lang
>
>_______________________________________________
>aqm mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/aqm

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: David Lang <[email-masked]>
Date: Mon, 13 Apr 2015 14:46:42 -0700 (PDT)

On Mon, 13 Apr 2015, Bob Briscoe wrote:

> I think your conception of how ECN works is incorrect. You describe ECN as if 
> the AQM marks one packet when it drops another packet. You say that the 
> ECN-mark speeds up the retransmission of the dropped packet. On the contrary, 
> the idea of classic ECN [RFC3168] is that the ECN marks replace the drops. In 
> all known testing (except pathological cases), classic ECN effectively 
> eliminates drops for all ECN-capable packets.

That's what I thought, and if that was the case, then marking packets as 
ECN-capable would mean that they would have an advantage over non-ECN packets 
(by not getting dropped, so getting a higher share of bandwidth)

that's what the gaming ECN thread was about, and if I understood the responses, 
I was being told that marking packets as ECN-capable, but not slowing down 
(actually responding to ECN) would not let an application get any advantage 
because the packets would just end up getting dropped anyway, since marking and 
dropping happen at the same level, even on ECN-capable flows.

If the packets are just marked, but not dropped, then the ECN-capable flows will 
occupy a disproportinate share of the available buffer space, since they just 
get marked instead of dropped.

David Lang

> Nonetheless, I do agree with your sentiment that the perfect is the enemy of 
> the good. We can remove most of the really bloat-induced latency without ECN. 
> So the message must be clear: Deploy AQM now. No need to wait for ECN. But 
> implementations SHOULD allow ECN packets to be classifed into a separately 
> configurable instance of the AQM algo. {Note 1}
>
> recommendations. Because, in existing equipment, even a poorly tuned RED is 
> usually much better than a bloated buffer with no AQM.
>
> Just as the WG mustn't confuse messages, you mustn't get confused by a 
> discussion about the potential for more ambitious reductions in latency. Koen 
> started the thread with reference to our presentation in the ICCRG in the 
> IRTF, where 'R' in both cases stands for Research.
>
> And I believe it is valid for the ECN benefits draft (in the IETF AQM WG) to 
> point to the potential of ECN, by using insights from research in progress.
>
> There are a number of different contributions to unnecessary latency, not 
> just the one popularised as bufferbloat:
>
> [In the following, I will use the term 'short message(s)' as shorthand for 
> either a short interactive flow or a long flow consisting of short 
> interactive messages (like in a game), or video frames or voice datagrams or 
> anything where the perceived latency depends on the latency of each 
> 'message', especially a string of messages with serial dependency as is 
> typical in Web.]
>
> #1 The well-known bufferbloat problem, where a long-running flow fills a 
> bloated buffer, delays short messages.
> #2 A loss causes head-of-line blocking for short message(s) while waiting for 
> the retransmission.
> redundancy is equivalent to poorer utilisation (altho only the short flows 
> need the redundancy).
>  - Techniques like tail-loss probe and RTO-restart can mitigate this delay, 
> without ECN, but not remove it.
> #4 The Reno/Cubic sawtooth causes variation in delay between 1 and 2 base 
> RTTs. This can affect short messages.
>  - AQM without ECN cannot remove this delay unless configured to sacrifice 
> utilisation.
>  - Flow queuing removes this delay variation if caused by a separate 
> long-running flow.{Note 2}
>  - A change to the TCP algo (e.g. DCTCP) can remove this delay variation. 
> Smaller sawteeth imply a much higher signalling rate, which in turn requires 
> ECN, otherwise drop probability would be excessive. (This was the main point 
> Koen was making.)
>  - AQM without ECN cannot remove this delay, and typically AQM is designed 
> to allow such bursts of delay in the hope they will disappear of their own 
> accord.
>  - Flow queuing can remove the effect of these delay bursts on other flows, 
> but only if it gives all flows a separate queue from the start.{Note 2}
>  - Delay-based softening of slow-start, such as Hybrid Slow-Start in Linux, 
> can mitigate these variations, but with increased risk of coming out of SS 
> early, causing significantly longer completion time.
>  - ECN with AQM based on the instantaneous queue limits this delay, without 
> the risk of longer completion time.
>
>  - Delay-based SS like HSS can mitigate these losses, with increased risk of 
> longer completion time.
> Whether flow queuing is applicable depends on the scale. The work I'm doing 
> with Koen is to reduce the cost of the queuing mechanisms on our BNGs 
> (broadband network gateways). We're trying to reduce the cost of per-customer 
> queuing at scale, so per-flow queuing is simply out of the question. Whereas 
> ECN requires no more processing than drop.
>
> Using flow as the unit of allocation also has its own problems, with no 
> proposed solutions.
>
>
> Bob
>
> {Note 1}: Your general point is that the perfect can be the enemy of the 
> good. Here's the presentation I gave in TSVAREA straight after VJ presented 
> CoDel in 2012 entitled "DCTCP & CoDel; the Best is the Friend of the Good."
> {Note 2}:  A lone flow can cause this delay variation to itself, but that's 
> irrelevant because, if the delay were not in the network it would be at the 
> sender.
>
> {Note 3}: Delay and loss spikes can equivalently be caused when Cubic's 
> window rises to seek out newly available capacity after another flow finishes 
> or the link rate varies.
>
>>> Perhaps you would care to provide some text to address the misconception 
>>> that you pointed out? (To wait for a 100% fix as a 90% fix appears much 
>>> less appealing, while the current state of art is at 0%)
>> 
>>> If you think that aqm-recommendations is not strogly enough worded. I 
>>> think this particular discussion (to aqm or not) really belongs there. The 
>>> other document (ecn benefits) has a different target in arguing for going 
>>> those last 10%...
>> 
>> so here is my "elevator pitch" on the problem. Feel free to take anything I 
>> say here for any purpose, and I'm sure I'll get corrected for anything I am 
>> wong on
>> 
>> Problem statement: Transmit buffers are needed to keep the network layer 
>> fully utilized, but excessive buffers result in poor latency for all 
>> traffic. This latency is frequently bad enough to cause some types of 
>> traffic to fail entirely.
>>
>>   <link to more background goes here, including how separate benchmarks for 
>> throughput and latency have mislead people, "packet loss considered evil", 
>> cheaper memory encouraging larger buffers, etc. Include tests like 
>> netperf-wrapper and ping latency while under load, etc. Include examples 
>> where buffers have resulted in latencies so long that packets are 
>> retransmitted before the first copy gets to the destination>
>> 
>> Traditionally, transmit buffers have been sized to handle a fixed number of 
>> packets. Due to teh variation in packet sizes, it is impossible to tune 
>> this value to both keep the link fully utilized when small packets dominate 
>> the trafific without having the queue size be large enough to cause latency 
>> problems when large packets dominate the traffic.
>> 
>> Shifting to Byte Queue Lengths where queues are allowed to hold a variable 
>> number of packets depending on how large they are makes it possible to 
>> manually tune the transmit buffer size to get good latency under all 
>> traffic conditions at a given speed. However, this step forward revealed 
>> two additional problems.
>> 
>> 1. whenever the data rate changes, this value needs to be manually changed 
>> (multi-link paths loose a link, noise degrades max throughput on a link, 
>> etc)
>> 
>> 2. high volume flows (i.e. bulk downloads) can starve other flows (DNS 
>> lookups, VoIP, Gaming, etc). this happens because space in tue queue is on 
>> a first-com-first-served basis, so the high-volume traffic fills the queue 
>> (at which point it starts to be dropped), but all other traffic that tries 
>> to arrive is also dropped. It turns out that these light flows tend to have 
>> a larger effect on the user experience than heavier flows, because things 
>> tend to be serialized behind the lighter flows (DNS lookup before doing a 
>> large download, retrieving a small HTML page to find what additional 
>> resources need to be fetched to display a page), or the user experience is 
>> directly effected by light flows (gaming lag, VoIP drops, etc)
>> 
>> Active Queue Management addresses these problems by adapting the amount of 
>> data that is buffered to match the data transmission capacity, and prevents 
>> high volume flows from starving low-volume flows without the need to 
>> implement QoS classifications.
>>
>>   <insert link about how you can't trust QoS tags that are made by other 
>> organizations, ways that it can be abused, etc>
>> 
>> This is possible because AQM algoithms don't have to drop the new packet 
>> that arrives, the algorithm can decide to drop the packet for one of the 
>> heavy flows rather than for one of the lightweight flows.
>> 
>> <insert references to currently favored AQM options here, PIE, fq_codel, 
>> cake, ???. Also links to failed approaches>
>> 
>> Turning on aqm on every bottleneck link makes the Internet usable for 
>> everyone, no matter what sort of application they are using.
>> 
>> throttling bandwidth before the bottleneck oand/or doing ingress shaping of 
>> traffic>
>> 
>> 
>> While AQM makes the network usable, there is still additional room for 
>> improvement. While dropping packets does result in the TCP senders slowing 
>> down,and eventually stabilizing at around the right speed to keep the link 
>> fully utilized, the only way that senders have been able to detect problems 
>> is to discover that they have not received an ack for the traffic within 
>> the allowed time. This causes a 'bubble' in the flow as teh dropped packet 
>> must be retransmitted (and sometimes a significant amount of data after the 
>> dropped packet that did make it to the destination, but could not be acked 
>> because fo the missing packet).
>> 
>> This "bubble" in the data flow can be greatly compressed by configuring the 
>> AQM algorithm to send an ECN packet to the sender when it drops a packet in 
>> a flow. The sender can then adapt faster, slowing down it's new data, and 
>> re-sending the dropped packet without having to wait for the timeout. This 
>> has two major effects by allowing the sender to retransmit the packet 
>> sooner the dealy on the dropped data is not as long, and because the 
>> replacement data can arrive before the timeout of the following packets, 
>> they may not need to be re-sent. by configuring the AQM algorithm to send 
>> the ECN notification to the sender only when the packet is being dropped, 
>> the effect of failure of the ECN packet to get through to the sender (the 
>> notification packet runs into congestion and gets dropped, some network 
>> device blocks it, etc) is that the ECN enabled case devolves to match the 
>> non-ECN case in that the sender will still detect the dropped packet via 
>> the timeout waiting for the ack as if ENCN was not enabled.
>> 
>> <insert link to possible problems that can happen here, including the 
>> potential for an app to 'game' things if packets are marked at a different 
>> level than when they are dropped.>
>> 
>> 
>> So a very strong recommendation to enable Active Queue Management, while 
>> the different algorithms have different advantages and levels of testing, 
>> even the 'worst' of the set results in a night-and-day improvement for 
>> usability compared to unmanaged buffers.
>> 
>> Enabling ECN at the same point as dropping packets as part of enabling any 
>> AQM algorithm results in a noticable improvement over the base algorithm 
>> without ECN. When compared to the baseline, the improvement added by ECN is 
>> tiny compared to the improvement from enabling AQM.
>>
>>   Is it fair to say that plain aqm vs aqm+ecn variation is on the same 
>> order of difference as the differences between the different AQM 
>> algorithms?
>> 
>> 
>> 
>> Future research items (which others here may already have done, and would 
>> not be part of my 'elevator pitch')
>> 
>> I believe that currently ECn triggers the exact same slowdown that a missed 
>> packet does, and it may be appropriate to have the sender do a less drastic 
>> slowdown.
>> 
>> It would be very interesing to provide soem way for the application sending 
>> the traffic to detect dropped packets and ECN responses. For example, a 
>> streaming media source (especially an interactive one like video 
>> conferencing) could adjust the bitrate that it's sending.
>> 
>> David Lang
>>

---

From: John Leslie <[email-masked]>
Date: Mon, 13 Apr 2015 20:25:16 -0400

David Lang <[email-masked]> wrote:
>> I think your conception of how ECN works is incorrect. You describe ECN as 
>> if the AQM marks one packet when it drops another packet. You say that the 
>> ECN-mark speeds up the retransmission of the dropped packet. On the 
>> contrary, the idea of classic ECN [RFC3168] is that the ECN marks replace 
>> the drops. In all known testing (except pathological cases), classic ECN 
>> effectively eliminates drops for all ECN-capable packets.
> 
> That's what I thought, and if that was the case,

   What does "that" refer to -- I honestly can't tell from your text.

   (We have an awful lot of talking past each other when we talk ECN:
it would really help to be as clear as possible.)

> then marking packets as ECN-capable would mean that they would have an
> advantage over non-ECN packets (by not getting dropped, so getting a
> higher share of bandwidth)

   I wish I could go back in time an excise every phrase saying "same as
a packet drop" from the ECN RFCs.

> that's what the gaming ECN thread was about, and if I understood the 
> responses, I was being told that marking packets as ECN-capable, but not 
> slowing down (actually responding to ECN) would not let an application get 
> any advantage because the packets would just end up getting dropped anyway, 
> since marking and dropping happen at the same level, even on ECN-capable 
> flows.

   No, I don't think any of us were saying that.

   A few of us _were_ saying that abuse like that is a small percentage
of the abuse where the sender simply doesn't reduce the sending rate if
packet loss is detected.

   And several folks _did_ say that it was important to have the marking
and dropping threshholds the same _and_ the reaction to marking the same
as the reaction to dropping be the same "so that neither would suffer in
comparison to the other".

   (They are wrong, BTW, but most of us just let this canard die down.)

> If the packets are just marked, but not dropped, then the ECN-capable flows 
> will occupy a disproportinate share of the available buffer space, since 
> they just get marked instead of dropped.

   True.

   But so what?

   What do you want to do about it? (That's my trademark question, BTW.)

   If you are the node experiencing congestion, and your buffer is full,
you absolutely should drop the marked packet. No disagreement there.

   (This can happen, BTW, in PIE, which marks packets as the enter the
buffer; and the buffer may fill before they can be sent.)

   But if your buffer isn't even close to full, what's wrong with ECN
capable packets using a larger share of the buffer? If they go to a
responsive flow, the reaction will be quicker than if they were dropped.

   Of course, if they go to a non-responsive flow, they would get a
slight advantage, but why is that _your_ concern? And there's really
nothing you can do to counter the advantage that doesn't break the ECN
paradigm for responsive flows.

   If instead, you are a node later on the path, there's nothing wrong
with dropping ECN packets with Congestion-Experienced marks _when_ you
need to drop packets. But there's no reason to "punish" them for having
occupied a "disproportionate share" of somebody else's buffer.

   It would be much better if we could think in terms of ECN-marking
_well_before_ there is any need to drop packets because the buffer is
full; and treating any "advantage" this might give them an an incentive
for more flows to enable ECN.

   One of the things I'm quite sure Bob Briscoe is thinking -- but not
writing -- is that he believes ECN can solve a lot of the problems he
named _if_ the ECN marking occurs at a lower threshhold than packet drop.
We are _not_ chartered to "fix ECN" in this WG: so that discussion doesn't
really belong in this thread; but you and Bob are unlikely to communicate
fully if you ignore that possibility.

   Hope this helps...

--
John Leslie <[email-masked]>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 15 Apr 2015 10:23:04 +0100

David,

At 22:46 13/04/2015, David Lang wrote:
>>ECN as if the AQM marks one packet when it drops another packet. 
>>You say that the ECN-mark speeds up the retransmission of the 
>>dropped packet. On the contrary, the idea of classic ECN [RFC3168] 
>>is that the ECN marks replace the drops. In all known testing 
>>(except pathological cases), classic ECN effectively eliminates 
>>drops for all ECN-capable packets.
>
>That's what I thought, and if that was the case, then marking 
>packets as ECN-capable would mean that they would have an advantage 
>over non-ECN packets (by not getting dropped, so getting a higher 
>share of bandwidth)

This is a common fallacy. An ECN-capable TCP achieves the same 
throughput as an otherwise identical non-ECN TCP. The fallacy comes 
from people who think that the network caps the rate by removing 
packets. But the source determines the rate by how many drop or mark 
signals it sees. For today's ('classic') ECN, the source's rate 
reduction in response to either is the same; both voluntary as well.

There will be a tiny difference in goodput, because of the 
retransmissions. However, the loss (or marking) probability that TCP 
uses to determine its rate is a fraction of a percent, so this 
difference is in the noise.

>responses, I was being told that marking packets as ECN-capable, but 
>not slowing down (actually responding to ECN) would not let an 
>application get any advantage because the packets would just end up 
>getting dropped anyway, since marking and dropping happen at the 
>same level, even on ECN-capable flows.

Nope.

I recall that thread. People came up with a number of complicated 
arguments for why it is hard to game ECN, but none were solid. Below 
I have given a simple argument that I think is solid on its own. I 
thought about intevening at the time, but this stuff needs care and 
time that I didn't have then.

1) For the most simple and complete argument, all you need to know 
about the ECN behaviour of AQMs is: under normal load conditions, an 
AQM decides it's time to send a congestion signal irrespective of 
whether the next packet is ECN-capable or not. Then, if the next 
packet is ECN-capable, it marks it, else it drops it. This is from 
RFC3168, which also requires the source to respond equally to either 
a loss or a mark. I call this 'classic' ECN.{Note 1}

2) I will try to correct your misunderstanding about "marking and 
dropping at the same level even on on ECN-capable flows". However, to 
determine whether ECN can be gamed, there's no need to go there. So 
I'll come back to that as a post-script{Note 2}.

3) I will prove that it is as easy to game loss as it is to game ECN, 
first considering sender cheating, then receiver cheating:

3a) Sender Cheating
 From the sender's point of view, the only difference between a loss 
and an ECN mark is that it has to retransmit a loss. But that has 
nothing to do with the rate it can go at. If it has been programmed 
to ignore congestion feedback (and instead to go at a constant 
unresponsive rate{Note 2}), it is as easy for it to ignore loss 
feedback as ECN feedback. See {Note 3} for an example.

3b) Receiver Cheating
* An ECN receiver can best fool an ECN-capable TCP sender into going 
faster by only feeding back a small fraction of ECN marks.{Note 4}
* A non-ECN receiver could fool a non-ECN TCP sender into going 
faster by only revealing a small fraction of the losses. However, it 
would have to ACK undelivered bytes, and most TCP-based apps won't 
work unless all bytes are delivered.{Note 5}

So it seems that it's easier for a receiver to game ECN than loss. However:
* returning to the ECN case, the sender can validate the receiver by 
randomly setting an ECN mark itself on a very small proportion of 
packets (probably only on unusually high rate connections). Then if 
it doesn't see ECN feedback on the ACK of any one of its 
self-inserted marks, it can close the connection.

In summary,
* a sender can't game ECN any more easily than it can game loss.
* a receiver can only game ECN if the sender doesn't take measures to 
prevent it.{Note 6}

>If the packets are just marked, but not dropped, then the 
>ECN-capable flows will occupy a disproportinate share of the 
>available buffer space, since they just get marked instead of dropped.

Nope.

The arrival rates will be the same, whether or not ECN is used (see 
earlier). And recall  that TCP drives the marking or loss probability 
at very small fractions in all normal conditions.

Example: if there are 10 flows in a 100Mb/s link, 5 ECN and 5 
non-ECN, they will all arrive at the buffer at 10Mb/s (all other 
factors being equal). Then, if the loss or marking probability is 
0.5%, the AQM will be marking but not dropping 1 in 200 packets in 
the ECN flows whereas it would drop 1 in 200 from the non-ECN flows.

So, assuming tail drop, if there were 399 packets in this buffer, on 
average 200 would be ECN-capable (20 in each flow) with one marked; 
and 199 would be non-ECN-capable (20 in each flow except one with 
19). And one of those 199 would be a retransimssion from an earlier loss.

[Of course, we would hope that there would be 4 packets in the 
buffer, not 400. The proportions would still be the same on average. 
I merely used 399 to avoid fractions of packets for the averages.]

===Footnotes===

{Note 1} To be concrete, I've assumed classic ECN [RFC3168]. The 
argument is similar for research approaches like "think once to mark, 
twice to drop", but let's not make it more complicated than it needs to be.

{Note 2} RFC3168 (and draft-ietf-aqm-recommendation) require that, 
whenever the AQM decides it is time to signal congestion on the next 
packet, if the queue has been persistently long, the AQM must only 
use drop as a congestion signal, irrespective of whether the next 
packet is ECN-capable or not.

So, if a source naively just continues to increase its window until 
it drives the queue into overload, then it would cause the AQM to 
turn off ECN and consequently not be able to game ECN. But the simple 
strategy of sending at a high but constant rate avoids driving the 
queue into overload. So that's the strategy I described for gaming 
ECN. Because strategies that don't work are irrelevant if there's a 
strategy that does work.

{Note 3} Examples to show source cheating is as easy with loss as ECN:
* An ECN source sends at a constant unresponsive 90Mb/s through a 
100Mb/s bottleneck. In parallel some other responsive flows (say 10 
non-ECN TCP flows) squeeze themselves into the remaining 10Mb/s. They 
will cause themselves (say) 0.5% loss probability, while the 
unresponsive flow will experience 0.5% marking and zero loss.
* A non-ECN source can just as easily send unresponsively at 90.5Mb/s 
as 90Mb/s. The other flows will still drive loss to about 0.5%, which 
the unresponsive flow will now experience as well. Nonetheless, after 
it retransmits the 0.5% loss it still achieves goodput of about 90Mb/s.

{Note 4} Again, feeding back no marks at all would be naive, because 
it would drive the bottleneck into overload, causing it to turn off 
ECN (and driving the loss-rate over a cliff). A better strategy is to 
feedback only a small proportion. Because TCP's rate depends on the 
square root of the congestion probability, to download N times 
faster, the receiver should feed back only about 1 in N^2 of the 
marks or losses. E.g. to go 90 times faster, feed back 1 in 8100 
marks (or losses).

{Note 5} There are two classes of apps that use TCP but can get away 
without reliable delivery:
i) Some streaming media apps are designed with a loss-tolerant 
encoding, so they can use TCP but play out the media even if some 
retransmissions haven't arrived yet (e.g. using a raw socket at the receiver).
ii) In the specific case of HTTP, a hacked receiver can open another 
connection to the same server and download the byte-ranges it needs 
to repair the holes in the other connection.

{Note 6} ConEx (congestion exposure [RFC6789]) provides a 
comprehensive framework for the network to prevent senders and 
receivers from cheating. However, for this argument, we don't need to 
go there either.

Cheers

Bob

>David Lang

[snip]

>>Using flow as the unit of allocation also has its own problems, 
>>with no proposed solutions.

[snip]

>>At 05:16 30/03/2015, David Lang wrote:

[snip]

>>>senders slowing down,and eventually stabilizing at around the 
>>>right speed to keep the link fully utilized, the only way that 
>>>senders have been able to detect problems is to discover that they 
>>>have not received an ack for the traffic within the allowed time. 
>>>This causes a 'bubble' in the flow as teh dropped packet must be 
>>>retransmitted (and sometimes a significant amount of data after 
>>>the dropped packet that did make it to the destination, but could 
>>>not be acked because fo the missing packet).
>>>when it drops a packet in a flow. The sender can then adapt 
>>>faster, slowing down it's new data, and re-sending the dropped 
>>>packet without having to wait for the timeout. This has two major 
>>>effects by allowing the sender to retransmit the packet sooner the 
>>>dealy on the dropped data is not as long, and because the 
>>>replacement data can arrive before the timeout of the following 
>>>packets, they may not need to be re-sent. by configuring the AQM 
>>>algorithm to send the ECN notification to the sender only when the 
>>>packet is being dropped, the effect of failure of the ECN packet 
>>>to get through to the sender (the notification packet runs into 
>>>congestion and gets dropped, some network device blocks it, etc) 
>>>is that the ECN enabled case devolves to match the non-ECN case in 
>>>that the sender will still detect the dropped packet via the 
>>>timeout waiting for the ack as if ENCN was not enabled.
>>>the potential for an app to 'game' things if packets are marked at 
>>>a different level than when they are dropped.>
>>>
>>>while the different algorithms have different advantages and 
>>>levels of testing, even the 'worst' of the set results in a 
>>>night-and-day improvement for usability compared to unmanaged buffers.
>>>Future research items (which others here may already have done, 
>>>and would not be part of my 'elevator pitch')
>>>I believe that currently ECn triggers the exact same slowdown that 
>>>a missed packet does, and it may be appropriate to have the sender 
>>>do a less drastic slowdown.
>>>It would be very interesing to provide soem way for the 
>>>application sending the traffic to detect dropped packets and ECN 
>>>responses. For example, a streaming media source (especially an 
>>>interactive one like video conferencing) could adjust the bitrate 
>>>that it's sending.
>>>David Lang

---

From: David Lang <[email-masked]>
Date: Wed, 15 Apr 2015 13:15:21 -0700 (PDT)

On Wed, 15 Apr 2015, Bob Briscoe wrote:

> 3) I will prove that it is as easy to game loss as it is to game ECN, first 
> considering sender cheating, then receiver cheating:

I think there is a key difference

cheating non-ECN requires changes to the TCP stack code

cheating ECN requires an iptables rule to zero out the ECN bit on received 
packets before they hit the TCP stack.

I've seen enough people doing cargo-cult network configs to 'improve 
gaming' (MTU 576 as one example) that I expect that if there is any noticable 
advantage for doing so, there will be a significant population doing so.

> 3a) Sender Cheating
> From the sender's point of view, the only difference between a loss and an 
> ECN mark is that it has to retransmit a loss. But that has nothing to do with 
> the rate it can go at. If it has been programmed to ignore congestion 
> feedback (and instead to go at a constant unresponsive rate{Note 2}), it is 
> as easy for it to ignore loss feedback as ECN feedback. See {Note 3} for an 
> example.
>
> 3b) Receiver Cheating
> * An ECN receiver can best fool an ECN-capable TCP sender into going faster 
> by only feeding back a small fraction of ECN marks.{Note 4}
> * A non-ECN receiver could fool a non-ECN TCP sender into going faster by 
> only revealing a small fraction of the losses. However, it would have to ACK 
> undelivered bytes, and most TCP-based apps won't work unless all bytes are 
> delivered.{Note 5}
>
> * returning to the ECN case, the sender can validate the receiver by randomly 
> setting an ECN mark itself on a very small proportion of packets (probably 
> only on unusually high rate connections). Then if it doesn't see ECN feedback 
> on the ACK of any one of its self-inserted marks, it can close the 
> connection.

do any TCP stacks do this?

won't this cause the datarate to slow down, something that will seem 'bad' for 
the people who would need to code this (it will show up on benchmarks as random 
slowdowns, right?)

> * a sender can't game ECN any more easily than it can game loss.

I think it's more "there is a way to detect gaming ECN"

> * a receiver can only game ECN if the sender doesn't take measures to prevent 
> it.{Note 6}
>
> The arrival rates will be the same, whether or not ECN is used (see earlier). 
> And recall  that TCP drives the marking or loss probability at very small 
> fractions in all normal conditions.
>
> Example: if there are 10 flows in a 100Mb/s link, 5 ECN and 5 non-ECN, they 
> will all arrive at the buffer at 10Mb/s (all other factors being equal). 
> Then, if the loss or marking probability is 0.5%, the AQM will be marking but 
> not dropping 1 in 200 packets in the ECN flows whereas it would drop 1 in 200 
> from the non-ECN flows.
>
> So, assuming tail drop, if there were 399 packets in this buffer, on average 
> 200 would be ECN-capable (20 in each flow) with one marked; and 199 would be 
> non-ECN-capable (20 in each flow except one with 19). And one of those 199 
> would be a retransimssion from an earlier loss.
>
> [Of course, we would hope that there would be 4 packets in the buffer, not 
> 400. The proportions would still be the same on average. I merely used 399 to 
> avoid fractions of packets for the averages.]
>
>
> ===Footnotes===
>
> * An ECN source sends at a constant unresponsive 90Mb/s through a 100Mb/s 
> bottleneck. In parallel some other responsive flows (say 10 non-ECN TCP 
> flows) squeeze themselves into the remaining 10Mb/s. They will cause 
> themselves (say) 0.5% loss probability, while the unresponsive flow will 
> experience 0.5% marking and zero loss.
> * A non-ECN source can just as easily send unresponsively at 90.5Mb/s as 
> 90Mb/s. The other flows will still drive loss to about 0.5%, which the 
> unresponsive flow will now experience as well. Nonetheless, after it 
> retransmits the 0.5% loss it still achieves goodput of about 90Mb/s.

well, sort of.

if the retransmissions are due to dropped packets, it may require retransmitting 
some packets that actually got through (depending on the link latency and rate 
of transmission, those added packets may not be acked before they also get 
retransmitted), this will waste some bandwidth

if the sender slows down, it may drop significantly below the 90Mb/s rate and 
have to ramp up to that rate again.

> {Note 4} Again, feeding back no marks at all would be naive, because it would 
> drive the bottleneck into overload, causing it to turn off ECN (and driving 
> the loss-rate over a cliff). A better strategy is to feedback only a small 
> proportion. Because TCP's rate depends on the square root of the congestion 
> probability, to download N times faster, the receiver should feed back only 
> about 1 in N^2 of the marks or losses. E.g. to go 90 times faster, feed back 
> 1 in 8100 marks (or losses).

that assumes that the raw sending rate is enough higher to drive the link into 
overload itself rather than just being enough higher to demand an unfair share 
of the link

you use the example of a 100Mb link with one app getting 90Mb and trying to get 
more.

Instead think of that same 100Mb link, but with 100 1Mb flows going over it. 
Would ignoring ECN feedback allow one app to get a 50Mb flow, squeezing the 
other apps to .5Mb each?

> {Note 5} There are two classes of apps that use TCP but can get away without 
> reliable delivery:
> i) Some streaming media apps are designed with a loss-tolerant encoding, so 
> they can use TCP but play out the media even if some retransmissions haven't 
> arrived yet (e.g. using a raw socket at the receiver).
> connection to the same server and download the byte-ranges it needs to repair 
> the holes in the other connection.

the TCP stack will demand reliable delivery, even if the app doesn't. There 
isn't any provision for an app to use TCP but allow 'holes' in the data.

David Lang