Subject: [aqm] review of draft-ietf-aqm-ecn-benefits-02

From: "Fred Baker (fred)" <[email-masked]>
Date: Tue, 24 Mar 2015 23:26:06 +0000

> Abstract
> 
>    This document describes the potential benefits when applications
>    enable Explicit Congestion Notification (ECN).  It outlines the
>    principal gains in terms of increased throughput, reduced delay and
>    other benefits when ECN is used over network paths that include
>    equipment that supports ECN-marking.  It also identifies some
>    potential problems that might occur when ECN is used.  The document
>    does not propose new algorithms that may be able to use ECN or
>    describe the details of implementation of ECN in endpoint devices,
>    routers and other network devices.

Dumb question: Is there any intention to comment on DCTCP, which has its own interpretation of ECN?

> 1.  Introduction

>    A network device (router, middlebox, or other device that forwards
>    packets through the network) that does not support AQM, typically
>    uses a drop-tail policy to discard excess IP packets when its queue
>    becomes full.

For the record, the device sending the packet should implement something regarding its own queues. One alternative is to freeze a process that presents a packet to a queue that is full until the queue opens up, but another way is to mark the packet.

> 2.2.  Bleaching and middlebox requirements to deploy ECN
> [snip]
>    Some network devices have been observed to implement a policy that
>    erases or "bleaches" the ECN marks at a network edge (resetting these
>    to zero).  This may be implemented for various reasons (including
>    normalising packets to hide which equipment supports ECN).  This
>    policy prevents use of ECN by applications.  A network device should
>    therefore not remark an ECT(0) or ECT(1) mark to zero.

Is this RFC 2119 “SHOULD”?

>    A network device must not change a packet with a CE mark to a zero

Is this RFC 2119 "MUST NOT”?

Section 3 and 4; my one point of puzzlement was that there was no reference to a test or paper supporting the relevant values.

> 4.1.  Avoiding Capacity Overshoot
> 
>    Internet transports do not know apriori how much capacity exists
>    along a network path.  Transports therefore try to measure the
>    capacity available to an application by probing the network path with
>    increasing traffic to the point where they detect the onset of
>    congestion (such as TCP or SCTP Slow Start).
> 
>    ECN can help capacity probing algorithms (such as Slow Start) from
>    significantly exceeding the bottleneck capacity of a network path.

This statement is a case in point of “can you point to a paper?”. Imagine, if you will, that the best value of cwnd for some session is N. A TCP session starts with IW packets in a burst. In the second RTT, as acks come back, 2*IW packets are sent, followed by 4*IW, 8*IW, and 16*IW packets until either cwnd crosses ssthresh or there is a loss. Suppose, just for fun, that for k a power of 2, N = k*IW+1. At least in theory, I shouldn’t see a lot of congestion on that RTT, and in the subsequent RTT I could see cwnd/2-1 packets marked and perhaps quite a few lost. That is no different than slow-start terminated by loss. I’m not sure I see how ECN avoids the case.

> 6.2.  Detecting ECN receiver feedback cheating

>    The transport at endpoint receivers must not try to conceal reception

Is this RFC 2119 “MUST NOT”?

>    of CE-marked packets in the ECN feedback information that they

> 7.  Conclusion

This fairly screams “RFC 2119”...

I think the document is well-written and clear. Primarily, I have two questions: to what extent is it an academic paper making assertions, and needing to justify them, and to what extent is it a “considerations” document that gives guidance on the use of ECN in a network (RFC 2119).

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Wed, 25 Mar 2015 18:12:47 +0000

Hi all,

Related to DCTCP and different (more) marking ECN than dropping (let's call it ECN++ in this mail), the talk I gave in iccrg (Data Center to the Home) shows that it is possible to have fairness between ECN++ flows (DCTCP, Relentless TCP, Scalable TCP, ...) and drop based Reno/Cubic flows.

The ECN++ flows typically respond to marking proportional to 1/p (with p the marking or dropping probability), while the Reno flavors respond proportional to 1/p^0.5 (one over square root of p).

This means that the only difference between marking and dropping is that an AQM has to think twice before it drops, and that is what we want, right? We mark fast by comparing our congestion indicator (derived from the queue size or packet sojourn time, or PI controller) with a random generated value. For a drop decision we just can compare the congestion indicator with the maximum of 2 random values (= thinking twice and is resulting in a drop probability which is the square of the marking probability). This will compensate the square root in Reno-like TCPs. If it is a problem to generate 2 random values per packet, you can keep your previous random value, as it is (pseudo) independent of the newly generated.

As this is a very simple relation between marking and dropping, AND it gives extra advantages, it is worth considering. The EXTRA advantages are:
- low latency AND high throughput (compared to low latency OR high throughput)
- less variability in flow fairness between competing flows (because of the high marking probability, the flows get more signals and will stray of less). If you get one drop every 10 seconds, and you had bad luck that your flow got 2 drops in a row, you have reduced your throughput by 4, compared to the other flow who should have had the second mark, running still at full throughput.
- The marking will scale to higher throughputs, every flow will get the same signal rate independent from the throughput (preferably every millisecond). It is a solution which scales to the future.

If we want to promote the use of ECN, let's make sure we get all the benefits, and have a solution that doesn't need to be revised after x years anyway. This is an opportunity to do it better this time, with lots of benefits which might convince people to finally use ECN.

I propose to recommend the "think twice" concept, or at least describe its extra benefits in the draft.

Regards,
Koen.

---

From: Michael Welzl <[email-masked]>
Date: Wed, 25 Mar 2015 14:03:36 -0500

Hi,

Below:

> On 25. mar. 2015, at 13.12, De Schepper, Koen (Koen) <[email-masked]> wrote:
> 
> Hi all,
> 

This draft is not specifying a new behavior for ECN, which is what you propose. Thus I think this is out of scope of this document.

Cheers,
Michael

---

From: David Collier-Brown <[email-masked]>
Date: Thu, 26 Mar 2015 16:36:09 -0400

> This draft is not specifying a new behavior for ECN, which is what you propose. Thus I think this is out of scope of this document.

I might suggest this is an implementation of a relationship between ECN 
and drop, based on an observation of how fairness might be achieved.

The implementation doesn't belong, unless you have an appendix of 
interesting implementation concerns.  The observation that

  * there is a relationship between ECN and drop
  * there are fairness problems to consider, and
  * there are values of the relationship that minimize or exclude the
    fairness problem

is new, a genuinely good thing, and arguably deserves mention as a benefit.

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
[email-masked]           |                      -- Mark Twain

---

From: Michael Welzl <[email-masked]>
Date: Thu, 26 Mar 2015 23:03:37 -0500

> On 26. mar. 2015, at 15.36, David Collier-Brown <[email-masked]> wrote:
> 
> I might suggest this is an implementation of a relationship between ECN and drop, based on an observation of how fairness might be achieved.  
> 
> The implementation doesn't belong, unless you have an appendix of interesting implementation concerns.

Yes -

>  The observation that 
> 	• there is a relationship between ECN and drop
> 	• there are fairness problems to consider, and
> 	• there are values of the relationship that minimize or exclude the fairness problem

Every item in your list is so vague that I can't see the value of adding it. I think these things have to be coupled with a specific algorithm (such as the one described above) to make sense - but I think we agree that this document isn't the place for a new algorithm.

Cheers,
Michael

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Fri, 27 Mar 2015 15:40:20 +0000

Hi all,

Agreed that there implementations should not be described. Also agree that we should describe all the benefits of using ECN, so I propose to reformulate something as follows:

ECN allows to introduce and deploy state of the art ECN based congestion controllers, 
and to guarantee throughput fairness by applying in the network an appropriate 
relation between marking and dropping.

The extra benefits of supporting these improved ECN based congestion controllers are:

- low latency AND high throughput (compared to low latency OR high throughput for drop 
based congestion controllers)
- reduced variability in flow fairness between competing flows because of the high 
marking rate
- scalability to higher throughputs because the marking rate is independent from the 
throughput

Disadvantage is that congestion controllers that are optimized for drop signals cannot 
use ECN without experiencing high throughput unfairness.

Regards,
Koen.

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Michael Welzl
Sent: vrijdag 27 maart 2015 5:04
To: [email-masked]
Cc: [email-masked]
Subject: Re: [aqm] think once to mark, think twice to drop: draft-ietf-aqm-ecn-benefits-02

>>> On 25. mar. 2015, at 13.12, De Schepper, Koen (Koen) 
>>> <[email-masked]>
>>>  wrote:
>>> 
>>> Hi all,
>>> 

Yes -

> fairness problem is new, a genuinely good thing, and arguably deserves mention as a benefit.

Cheers,
Michael

_______________________________________________
aqm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/aqm

---

From: David Lang <[email-masked]>
Date: Fri, 27 Mar 2015 10:32:16 -0700 (PDT)

> based congestion controllers)

I think that you are overstating things when you say that without ECN you are 
forced to choose between low latency OR high throughput. that doesn't match what 
people are reporting when they use simple fq_codel without ECN

> - reduced variability in flow fairness between competing flows because of the 
> high marking rate

This is plausable, but how unfair do things get in practice? Especially if you 
have a large number of flows so that you are statistically unlikely to keep 
picking on the same flow.

> - scalability to higher throughputs because the marking rate is independent 
> from the throughput

The drop rate is also independent of the throughput, so I don't see how this 
works unless you are doing statistical dropping.

If you take the approach that there is a probability P that you should drop/mark 
a packet, these statements make a lot of sense. But if you are doing something 
like fq_codel that is based on incoming packets, not probability checks, they 
make less sense.

Yes, you can analyse the results by saying that there was a probability P of a 
packet being dropped, but that's not what's being done under the covers.

> Disadvantage is that congestion controllers that are optimized for drop 
> signals cannot use ECN without experiencing high throughput unfairness.

This is being disputed heavily by others here who are saying that adding ECN 
cannot possibly cause unfairness (in either direction)

David Lang

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 27 Mar 2015 18:11:33 +0000

Hi David,

> > - low latency AND high throughput (compared to low latency OR high
> > throughput for drop based congestion controllers)
> 
> I think that you are overstating things when you say that without ECN you
> are forced to choose between low latency OR high throughput. that doesn't
> match what people are reporting when they use simple fq_codel without ECN

I disagree. At the least when you are using TCP, a drop will cause head-of-line blocking on the receiver, for at least 1 RTT; Agreed, there are ways to mitigate this (FEC-encoded transports).

But the "OR" is exactly the correct description here: FEC has inheritent overhead thus reduced bandwidth, and loss-recovery suffers from head-of-line blocking, thus higher latency (to the application, where it actually matters).

FQ-Codel runs with high bandwidth, but the drops induce latency in the end-hosts nevertheless...
Thus FQ-Codel with ECN would still be more effective than without ECN.

Richard

---

From: John Leslie <[email-masked]>
Date: Fri, 27 Mar 2015 14:36:59 -0400

Scheffenegger, Richard <[email-masked]> wrote:
> 
>... At the least when you are using TCP, a drop will cause head-of-line
> blocking on the receiver, for at least 1 RTT;

   Yes.

   This is a trade-off: many folks believe that a good AQM has enough
benefits for typical TCP flows to overcome that. (YMMV)

> Agreed, there are ways to mitigate this (FEC-encoded transports).

   Indeed.

   And this is happening. I haven't heard of it happening in TCP itself;
but I wouldn't be surprised to hear of implementations which advertise
themselves as TCP doing this...

   FEC does _not_ have to be particularly expensive in bandwidth (though
the implementations we hear about mosty are...).

> But the "OR" is exactly the correct description here: FEC has inherent
> overhead thus reduced bandwidth, and loss-recovery suffers from
> head-of-line blocking,

   Yes. (though there are possible mitigations)...

> thus higher latency (to the application, where it actually matters).

   It depends upon the application, at least partly.

   The bulk of TCP transport today, we believe, is trying to optimize
for quick transport of medium-sized files. We already see "optimizations"
which send whole files before looking for feedback. These optimizations
seem to be working pretty well...

> end-hosts nevertheless...

   Yes.

   But this isn't causing folks to demand their outgoing routers disable
FQ-CoDel, etc...

> Thus FQ-Codel with ECN would still be more effective than without ECN.

   Absolutely!

   And that is where we want to go!

--
John Leslie <[email-masked]>

---

From: Vishal Misra <[email-masked]>
Date: Fri, 27 Mar 2015 14:45:13 -0400

If you reduce latency, the dynamics of TCP are such that it will necessarily increase loss rate. On a bottlenecked link, the relationship of throughput to the RTT and loss rate of TCP is roughly the following (happy stop supply link to papers):

throughput = K/(RTT*sqrt(p))

where K is some constant, p is the loss rate and RTT is the round trip time. If you reduce latencies, to maintain the same throughput (that of the bottlenecked link), the loss rate has to necessarily go up.
So reducing latencies has the impact of increasing loss rates which affects things in bad ways as has been pointed out.

With ECN, it is the _marking_ rate that goes up and TCP follows the same dynamics. Nothing is dropped, no harm done.
That’s why ECN widely adopted is a win-win.

-Vishal
--
http://www.cs.columbia.edu/~misra/

> On Mar 27, 2015, at 2:36 PM, John Leslie <[email-masked]> wrote:
> 
> Scheffenegger, Richard <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: David Lang <[email-masked]>
Date: Fri, 27 Mar 2015 12:08:51 -0700 (PDT)

> Thus FQ-Codel with ECN would still be more effective than without ECN.

The key thing is that we are talking about different scales here.

You are talking about a 1RTT latency that will happen if the congestion gets bad 
enough, while I'm looking at the effects of overbuffering that causes the 
effective utilization of links to be well below 50% in practice. In such a case, 
doing something like fq_codel without ECN will get you 90%+ of the theoretical 
best case and is such a massive improvement that the vast majority of the users 
are not going to notice any improvement from ECN marking instead of dropping.

But the discussions here have explicitly said that ECN marking happens at the 
same time that drops happen, so it's not a case of doing ECN instead of 
dropping. While ECN could be used that way, that's not what's being advocated 
(and marking independently of dropping does raise all the fairness issues that 
we discussed a week or so ago)

David Lang

---

From: David Lang <[email-masked]>
Date: Fri, 27 Mar 2015 12:18:19 -0700 (PDT)

If your link is the bottleneck, adding buffers (allowing latency to increase) 
doesn't decrease loss over the long run, it just hides it in the short run.

If you don't have the bandwidth to send the packets, nothing you do with buffers 
or latency will let you get more packets through.

If you push too hard to decrease latency, then you run into times when the link 
is idle, so throughput can suffer a little bit.

But right now, real-world buffers commonly get into tens of seconds worth of 
traffic. Far beyond what's needed or reasonable to keep the links busy. Using 
active queue management to keep the links busy without letting the backlong 
build up results in both increased throughput for users, and decreases of 
latency of a couple orders of magnatude.

by comparison, ECN on a network that's operating without excessive buffering and 
has a large number of flows, is going to only have a small effect on latency and 
througput overall.

Yes, when a packet is lost it causes a 'large' amount of latency as the sender 
times out and retransmits, but if this is only happening every few thousand 
packets, it's a minor effect.

David Lang

On Fri, 27 Mar 2015, Vishal Misra wrote:

> If you reduce latency, the dynamics of TCP are such that it will necessarily 
> increase loss rate. On a bottlenecked link, the relationship of throughput to 
> the RTT and loss rate of TCP is roughly the following (happy stop supply link 
> to papers):
>

---

From: KK <[email-masked]>
Date: Fri, 27 Mar 2015 12:36:08 -0700

The discussion about adding buffers and the impact of buffers should be
considered relative to the time scales when congestion occurs and when it
is relieved by the dynamics of the end-system protocols. The reason we
have buffering is to handle transients at the points where there is a
mismatch in available bandwidth. We don¹t look to just throw buffers in
front of a bottleneck for Œlong run¹ overload.

While active queue management undoubtedly seeks to keep the backlog
build-up at a manageable level so as to not allow latency to grow and
still keep the links busy to the extent possible, the complement that ECN
provides is to mitigate the impact of the drop that AQM uses to signal
end-points to react to the transient congestion. ECN has the benefit when
you have flows that have small windows, where the impact of loss is more
significant. 

As you say, "when a packet is lost it causes a 'large' amount of latency
as the sender times out and retransmits, but if this is only happening
every few thousand packets, it's a minor effect.². But this is the case
for flows that are long-lived. If the flows are short-lived (and I believe
empirical evidence suggests that they are a significant portion of the
flows), then it is not a minor effect any more.

Thanks,
-- 
K. K. Ramakrishnan
Professor
Dept. of Computer Science and Engineering
University of California, Riverside
Rm. 332, Winston Chung Hall
Tel: (951) 827-2480
Web Page: http://www.cs.ucr.edu/~kk/

On 3/27/15, 12:18 PM, "David Lang" <[email-masked]> wrote:

>> where K is some constant, p is the loss rate and RTT is the round trip
>>time. If you reduce latencies, to maintain the same throughput (that of
>>the bottlenecked link), the loss rate has to necessarily go up.
>> So reducing latencies has the impact of increasing loss rates which
>>affects things in bad ways as has been pointed out.
>>
>> With ECN, it is the _marking_ rate that goes up and TCP follows the
>>same dynamics. Nothing is dropped, no harm done.
>> That¹s why ECN widely adopted is a win-win.
>>
>> -Vishal
>> --

---

From: David Lang <[email-masked]>
Date: Fri, 27 Mar 2015 13:17:36 -0700 (PDT)

On Fri, 27 Mar 2015, KK wrote:

> front of a bottleneck for ?long run¹ overload.

In theory you are correct. However in practice, you are wrong.

throughput benchmarks don't care how long the data sits in buffers, so larger 
buffers improve the benchmark numbers (up until the point that they cause 
timeouts)

But even if the product folks aren't just trying to maximize throughput, they 
size the buffers based on the worst case bandwidth/latency. So you have products 
with buffers that can handle 1Gb links with 200ms speed-of-light induces latency 
being used for 1.5Mb/768K 20ms DSL lines without any changes.

I'm not saying that ECN doesn't provide value, but the statement that without 
ECN you have the choice of low-latency OR good througput is only true if you 
ignore what's in place today.

It also does a dissservice because it implies that if you use something other 
than ECN, it's going to hurt your performance. This discourages people from 
enabling pie or fq_codel because they have read about how bad they are and how 
they will increase latency because they drop packets. This isn't just a 
theoretical "someone may think this", I've seen this exact argument trotted out 
a couple times recently.

> flows), then it is not a minor effect any more.

Even an occasional lost packet in a short flow is a minor effect compared to the 
current status quo of high latency on all packets.

Yes, many web pages are made up of many different items, fetched from many 
different locations, so avoiding packet losses on these flows is desirable.

But it's even more important to keep latency low while the link is under load, 
otherwise your connections end up being serialized, which kills performance even 
more.

As an example (just to be sure we are all talking about the same thing)

user clicks a link
DNS lookup
small page fetch
N resources to fetch, add to queue
for each resource in the queue (up to M in parallel)
   DNS lookup (may be cached)
   page fetch (some small, some large, some massive)
   may trigger more resources to fetch that get added to queue

it's common for there to be a few massive resources to fetch in a page that get 
queued early (UI javascript libraries or background images)

If a packet gets lost from one of the large fetches, it doesn't have that big of 
an effect. If it gets lost from one of the small fetches, it has more of an 
effect.

But if the first resource to be fetch causes latency to go to 500ms (actually a 
fairly 'clean' network by today's standards), then all of the DNS lookups, TCP 
handshakes, etc that are needed for all the other resources end up taking far 
longer than the time that would be lost due to a dropped packet.

This is a better vs best argument. Nobody disputes that something like 
fq_codel/pie/cake/whatever + ECN would be better than just 
fq_codel/pie/cake/whatever, but the way this is being worded make it sound that 
static buffer sizes + tail-drop + ECN is better than fq_codel/pie/cake/whaever 
because these other queueing algorithms will cause packet loss.

David Lang

---

From: Vishal Misra <[email-masked]>
Date: Fri, 27 Mar 2015 16:21:09 -0400

I don’t think anyone is making that argument. ECN without AQM is completely pointless (what decides the marking rate?). AQM is good but it really kicks in when coupled with ECN. 

-Vishal

> On Mar 27, 2015, at 4:17 PM, David Lang <[email-masked]> wrote:
> 
> 
> This is a better vs best argument. Nobody disputes that something like fq_codel/pie/cake/whatever + ECN would be better than just fq_codel/pie/cake/whatever, but the way this is being worded make it sound that static buffer sizes + tail-drop + ECN is better than fq_codel/pie/cake/whaever because these other queueing algorithms will cause packet loss.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sat, 28 Mar 2015 14:50:42 +0000

David,

Perhaps you would care to provide some text to address the misconception that you pointed out? (To wait for a 100% fix as a 90% fix appears much less appealing, while the current state of art is at 0%)

If you think that aqm-recommendations is not strogly enough worded. I think this particular discussion (to aqm or not) really belongs there. The other document (ecn benefits) has a different target in arguing for going those last 10%...

Best regards,
   Richard Scheffenegger

> Am 27.03.2015 um 16:17 schrieb "David Lang" <[email-masked]>:
> 
> throughput benchmarks don't care how long the data sits in buffers, so larger buffers improve the benchmark numbers (up until the point that they cause timeouts)
> 
> But even if the product folks aren't just trying to maximize throughput, they size the buffers based on the worst case bandwidth/latency. So you have products with buffers that can handle 1Gb links with 200ms speed-of-light induces latency being used for 1.5Mb/768K 20ms DSL lines without any changes.
> 
> I'm not saying that ECN doesn't provide value, but the statement that without ECN you have the choice of low-latency OR good througput is only true if you ignore what's in place today.
> 
> It also does a dissservice because it implies that if you use something other than ECN, it's going to hurt your performance. This discourages people from enabling pie or fq_codel because they have read about how bad they are and how they will increase latency because they drop packets. This isn't just a theoretical "someone may think this", I've seen this exact argument trotted out a couple times recently.
> 
> Even an occasional lost packet in a short flow is a minor effect compared to the current status quo of high latency on all packets.
> 
> Yes, many web pages are made up of many different items, fetched from many different locations, so avoiding packet losses on these flows is desirable.
> 
> But it's even more important to keep latency low while the link is under load, otherwise your connections end up being serialized, which kills performance even more.
> 
> it's common for there to be a few massive resources to fetch in a page that get queued early (UI javascript libraries or background images)
> 
> If a packet gets lost from one of the large fetches, it doesn't have that big of an effect. If it gets lost from one of the small fetches, it has more of an effect.
> 
> But if the first resource to be fetch causes latency to go to 500ms (actually a fairly 'clean' network by today's standards), then all of the DNS lookups, TCP handshakes, etc that are needed for all the other resources end up taking far longer than the time that would be lost due to a dropped packet.
>

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Sun, 29 Mar 2015 15:19:59 +0000

Hi David,

There seem to be some misunderstandings:

> (and marking independently of dropping does raise all the fairness
> issues that we discussed a week or so ago)
What I proposed is to add the benefits of marking differently than dropping in the AQM, more specific to apply for dropping p^2 and for marking just p. This will run fair, as Reno (and others fair to Reno) have a steady state rate that is proportional to 1/p^0.5 and many improved congestion controllers all advocate to have a rate proportional to 1/p. So marking ECN different (with squared p) than dropping allows to deploy the better 1/p schemes for marking. They can still be drop compatible (behave as reno/cubic on experiencing drop).

So the advantages I described were in the context of doing the better marking and better congestion control for marking: 

> I think that you are overstating things when you say that without
> ECN you are forced to choose between low latency OR high throughput
I was not comparing ECN with non-ECN, but ECN+Reno and for example ECN+DCTCP.
You can have a much more frequent marking probability than dropping probability, so we should use that.
DCTCP which has a rate = 2/(p.rtt) when controlled with an AQM that calculates a p. It can have full utility with a very small buffer, as it reduces its window only with a very small amount when receiving a mark (it needs around 2 marks per RTT at steady state). If a window is halved as with reno, you need a buffer of 1 BDP, if a window is set to 70% as with cubic you still need a buffer of 43% of BDP.

>> - reduced variability in flow fairness between competing flows
>> because of the high marking rate
> This is plausable, but how unfair do things get in practice?
> Especially if you have a large number of flows so that you are
> statistically unlikely to keep picking on the same flow.
If marking is more frequent, it is indeed statistically unlikely to keep picking on the same flow even with 2 flows (no need for many flows).

>> - scalability to higher throughputs because the marking rate is 
>> independent from the throughput
> The drop rate is also independent of the throughput, so I don't
> see how this works unless you are doing statistical dropping.
In case of reno, for a specific RTT, the drop rate (rate*p) will be 1.22*p^0.5/RTT (drops become very seldom), while for DCTCP it is 2/RTT (marks always twice per RTT, independent from p).

>> Disadvantage is that congestion controllers that are optimized for 
>> drop signals cannot use ECN without experiencing high throughput
>> unfairness.
> This is being disputed heavily by others here who are saying that
> adding ECN cannot possibly cause unfairness (in either direction)
I meant that if we mark more often than dropping, flows that use ECN and are not adapted to behave different to marking will reduce their window too much which results in reduced throughput.

Hope this clarifies.

Regards,
Koen.

-----Original Message-----
From: David Lang [mailto:[email-masked]] 
Sent: vrijdag 27 maart 2015 20:09
To: Scheffenegger, Richard
Cc: De Schepper, Koen (Koen); [email-masked]; Michael Welzl; [email-masked]
Subject: RE: [aqm] think once to mark, think twice to drop: draft-ietf-aqm-ecn-benefits-02

On Fri, 27 Mar 2015, Scheffenegger, Richard wrote:

>> I think that you are overstating things when you say that without ECN 
>> you are forced to choose between low latency OR high throughput. that 
>> doesn't match what people are reporting when they use simple fq_codel 
>> without ECN
>
>

You are talking about a 1RTT latency that will happen if the congestion gets bad enough, while I'm looking at the effects of overbuffering that causes the effective utilization of links to be well below 50% in practice. In such a case, doing something like fq_codel without ECN will get you 90%+ of the theoretical best case and is such a massive improvement that the vast majority of the users are not going to notice any improvement from ECN marking instead of dropping.

But the discussions here have explicitly said that ECN marking happens at the same time that drops happen, so it's not a case of doing ECN instead of dropping. While ECN could be used that way, that's not what's being advocated (and marking independently of dropping does raise all the fairness issues that we discussed a week or so ago)

David Lang

---

From: David Lang <[email-masked]>
Date: Sun, 29 Mar 2015 21:16:03 -0700 (PDT)

On Sat, 28 Mar 2015, Scheffenegger, Richard wrote:

> David,
>
> Perhaps you would care to provide some text to address the misconception that 
> you pointed out? (To wait for a 100% fix as a 90% fix appears much less 
> appealing, while the current state of art is at 0%)

Ok, you put me on the spot :-) Here goes.

> If you think that aqm-recommendations is not strogly enough worded. I think 
> this particular discussion (to aqm or not) really belongs there. The other 
> document (ecn benefits) has a different target in arguing for going those last 
> 10%...

so here is my "elevator pitch" on the problem. Feel free to take anything I say 
here for any purpose, and I'm sure I'll get corrected for anything I am wong on

Problem statement: Transmit buffers are needed to keep the network layer fully 
utilized, but excessive buffers result in poor latency for all traffic. This 
latency is frequently bad enough to cause some types of traffic to fail 
entirely.

   <link to more background goes here, including how separate benchmarks for 
throughput and latency have mislead people, "packet loss considered evil", 
cheaper memory encouraging larger buffers, etc. Include tests like 
netperf-wrapper and ping latency while under load, etc. Include examples where 
buffers have resulted in latencies so long that packets are retransmitted before 
the first copy gets to the destination>

Traditionally, transmit buffers have been sized to handle a fixed number of 
packets. Due to teh variation in packet sizes, it is impossible to tune this 
value to both keep the link fully utilized when small packets dominate the 
trafific without having the queue size be large enough to cause latency problems 
when large packets dominate the traffic.

Shifting to Byte Queue Lengths where queues are allowed to hold a variable 
number of packets depending on how large they are makes it possible to manually 
tune the transmit buffer size to get good latency under all traffic conditions 
at a given speed. However, this step forward revealed two additional problems.

1. whenever the data rate changes, this value needs to be manually changed 
(multi-link paths loose a link, noise degrades max throughput on a link, etc)

2. high volume flows (i.e. bulk downloads) can starve other flows (DNS lookups, 
VoIP, Gaming, etc). this happens because space in tue queue is on a 
first-com-first-served basis, so the high-volume traffic fills the queue (at 
which point it starts to be dropped), but all other traffic that tries to arrive 
is also dropped. It turns out that these light flows tend to have a larger 
effect on the user experience than heavier flows, because things tend to be 
serialized behind the lighter flows (DNS lookup before doing a large download, 
retrieving a small HTML page to find what additional resources need to be 
fetched to display a page), or the user experience is directly effected by 
light flows (gaming lag, VoIP drops, etc)

Active Queue Management addresses these problems by adapting the amount of data 
that is buffered to match the data transmission capacity, and prevents high 
volume flows from starving low-volume flows without the need to implement QoS 
classifications.

   <insert link about how you can't trust QoS tags that are made by other 
organizations, ways that it can be abused, etc>

This is possible because AQM algoithms don't have to drop the new packet that 
arrives, the algorithm can decide to drop the packet for one of the heavy flows 
rather than for one of the lightweight flows.

<insert references to currently favored AQM options here, PIE, fq_codel, cake, 
???. Also links to failed approaches>

Turning on aqm on every bottleneck link makes the Internet usable for everyone, 
no matter what sort of application they are using.

<insert link on how to deal with equipment you can't configure by throttling 
bandwidth before the bottleneck oand/or doing ingress shaping of traffic>

While AQM makes the network usable, there is still additional room for 
improvement. While dropping packets does result in the TCP senders slowing 
down,and eventually stabilizing at around the right speed to keep the link fully 
utilized, the only way that senders have been able to detect problems is to 
discover that they have not received an ack for the traffic within the allowed 
time. This causes a 'bubble' in the flow as teh dropped packet must be 
retransmitted (and sometimes a significant amount of data after the dropped 
packet that did make it to the destination, but could not be acked because fo 
the missing packet).

This "bubble" in the data flow can be greatly compressed by configuring the AQM 
algorithm to send an ECN packet to the sender when it drops a packet in a flow. 
The sender can then adapt faster, slowing down it's new data, and re-sending the 
dropped packet without having to wait for the timeout. This has two major 
effects by allowing the sender to retransmit the packet sooner the dealy on the 
dropped data is not as long, and because the replacement data can arrive before 
the timeout of the following packets, they may not need to be re-sent. by 
configuring the AQM algorithm to send the ECN notification to the sender only 
when the packet is being dropped, the effect of failure of the ECN packet to get 
through to the sender (the notification packet runs into congestion and gets 
dropped, some network device blocks it, etc) is that the ECN enabled case 
devolves to match the non-ECN case in that the sender will still detect the 
dropped packet via the timeout waiting for the ack as if ENCN was not enabled.

<insert link to possible problems that can happen here, including the potential 
for an app to 'game' things if packets are marked at a different level than 
when they are dropped.>

So a very strong recommendation to enable Active Queue Management, while the 
different algorithms have different advantages and levels of testing, even the 
'worst' of the set results in a night-and-day improvement for usability compared 
to unmanaged buffers.

Enabling ECN at the same point as dropping packets as part of enabling any AQM 
algorithm results in a noticable improvement over the base algorithm without 
ECN. When compared to the baseline, the improvement added by ECN is tiny 
compared to the improvement from enabling AQM.

   Is it fair to say that plain aqm vs aqm+ecn variation is on the same order of 
difference as the differences between the different AQM algorithms?

Future research items (which others here may already have done, and would not be 
part of my 'elevator pitch')

I believe that currently ECn triggers the exact same slowdown that a missed 
packet does, and it may be appropriate to have the sender do a less drastic 
slowdown.

It would be very interesing to provide soem way for the application sending the 
traffic to detect dropped packets and ECN responses. For example, a streaming 
media source (especially an interactive one like video conferencing) could 
adjust the bitrate that it's sending.

David Lang