Subject: [aqm] Fwd: [tsvwg] Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP

From: Dave Taht <[email-masked]>
Date: Mon, 14 Oct 2013 21:35:58 -0700

It is my general assumption that membership in the aqm wg is a subset
of tsvwg, but in case it isn't...

---------- Forwarded message ----------
From: Bob Briscoe <[email-masked]>
Date: Tue, Sep 17, 2013 at 5:33 AM
Subject: [tsvwg] Guidelines for Adding Congestion Notification to
Protocols that Encapsulate IP
To: tsvwg IETF list <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>

TSV folks,

We've just posted an update to the guidelines for adding congestion
notification to L2 and tunnelling protocols, e.g. Ethernet, L2TP, GRE,
PPTP, GTP etc. Especially of current interest in mobile and separately
in data centres.

Pls read/review and send any comments to the list.
<http://tools.ietf.org/html/draft-briscoe-tsvwg-ecn-encap-guidelines>

It's currently still an individual draft; intended status: BCP.
We're looking for support for adoption as a item of work in tsvwg.

We've dealt with most of the outstanding issues from previous reviews,
esp. those from Gorry (see summary of diffs below).

--------------------------------
Summary of Diffs from -02 to -03

      *  Scope section:

         +  Added dependence on correct propagation of traffic class
            information

         +  For the feed-backward mode, deemed multicast and anycast out
            of scope

      *  Ensured all guidelines referring to subnet technologies also
         refer to tunnels and vice versa by adding applicability
         sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and
         5.

      *  Added Security Considerations on ensuring congestion signal
         fields are classed as immutable and on using end-to-end
         congestion signal integrity technologies rather than hop-by-
         hop.

Bob

> From: <[email-masked]>
> To: John Kaippallimalil <[email-masked]>,
>         Bob Briscoe
>         <[email-masked]>,
>         "Bob J.Briscoe" <[email-masked]>,
>         Pat Thaler
>         <[email-masked]>,
>         Patricia Thaler <[email-masked]>
> Subject: New Version Notification for
>         draft-briscoe-tsvwg-ecn-encap-guidelines-03.txt
>
>
> A new version of I-D, draft-briscoe-tsvwg-ecn-encap-guidelines-03.txt
> has been successfully submitted by Bob Briscoe and posted to the
> IETF repository.
>
> Filename:       draft-briscoe-tsvwg-ecn-encap-guidelines
> Revision:       03
> Title:          Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP
> Creation date:  2013-09-17
> Group:          Individual Submission
> Number of pages: 29
> URL: http://www.ietf.org/internet-drafts/draft-briscoe-tsvwg-ecn-encap-guidelines-03.txt
> Status: http://datatracker.ietf.org/doc/draft-briscoe-tsvwg-ecn-encap-guidelines
> Htmlized: http://tools.ietf.org/html/draft-briscoe-tsvwg-ecn-encap-guidelines-03
> Diff: http://www.ietf.org/rfcdiff?url2=draft-briscoe-tsvwg-ecn-encap-guidelines-03
>
> Abstract:
>    The purpose of this document is to guide the design of congestion
>    notification in any lower layer or tunnelling protocol that
>    encapsulates IP.  The aim is for explicit congestion signals to
>    propagate consistently from lower layer protocols into IP.  Then the
>    IP internetwork layer can act as a portability layer to carry
>    congestion notification from non-IP-aware congested nodes up to the
>    transport layer (L4).  Following these guidelines should assure
>    interworking between new lower layer congestion notification
>    mechanisms, whether specified by the IETF or other standards bodies.
>
>
>
>
>
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat

________________________________________________________________
Bob Briscoe,                                                  BT

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 22 Oct 2013 00:40:39 +0100

Dave,

Thx for cross-posting. It's quite possible there are aqm people not on tsvwg.

1) Did you have views on the ecn-encap-guidelines I-D itself? Is it helpful?
I think it's fairly non-controversial.

2) In case I'm misunderstood, I don't see much 
latency gain from ECN as currently defined 
(RFC3168). I'm still banging on about ECN in 
lower layers 'cos early tests show promise that 
we may have found a way to make the ultra-low 
queuing delay of data centre TCP incrementally 
deployable on the public Internet

         We = Mirja Kuehlewind, David Wagner, Juan Manuel Reyes Espinosa and I.

It can be deployed with most existing kit, 
without any new AQM implementation needed. We use 
WRED in an unusual configuration. The idea in a nutshell:
* For ECN-capable packets, shift the job of hiding bursts from network to host:
    - ECN-capable packets map to a WRED policy with queue smoothing turned off
    - so the network signals ECN with zero smoothing delay
    - then the transport can hide bursts of ECN 
signals from itself (like DCTCP does)
    - the transport knows
      o that it's ECN-capable
      o whether it's TCP or RTP etc,
      o whether its in congestion avoidance or slow-start,
      o and it knows its RTT,
      o so it can know whether to respond immediately or to smooth the signals,
      o and if so, over what time
    - then short RTT flows can smooth the signals 
with only the delay of their /own/ RTT
      o so they can fill troughs and absorb peaks that longer RTT flows cannot
    - a TCP only needs to smooth the signals if in congestion avoidance
      o in slow start, it can respond immediately, thus reducing overshoot

There's more to it - I've asked to present the whole story in Vancouver.

Bob

At 05:35 15/10/2013, Dave Taht wrote:
> > Title:          Guidelines for Adding 
> Congestion Notification to Protocols that Encapsulate IP
> > URL: 
> http://www.ietf.org/internet-drafts/draft-briscoe-tsvwg-ecn-encap-guidelines-03.txt
> > Status: 
> http://datatracker.ietf.org/doc/draft-briscoe-tsvwg-ecn-encap-guidelines
> > Htmlized: 
> http://tools.ietf.org/html/draft-briscoe-tsvwg-ecn-encap-guidelines-03
> > Diff: 
> http://www.ietf.org/rfcdiff?url2=draft-briscoe-tsvwg-ecn-encap-guidelines-03
> >
> > Abstract:
>Fixing bufferbloat with cerowrt: 
>http://www.teklibre.com/cerowrt/subscribe.html
>_______________________________________________
>aqm mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/aqm

---

From: Matthew Ford <[email-masked]>
Date: Tue, 22 Oct 2013 11:56:04 +0100

> There's more to it - I've asked to present the whole story in Vancouver.

Cool, looking forward to it. I'll prime the question pump now if I may:

Does the transport need to know that it is connected to a network that will signal ECN in this fashion?

Mat

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 25 Oct 2013 22:16:09 +0100

Mat,

>that will signal ECN in this fashion?

Here's the outline of the talk I'm planning, that I sent to Martin 
originally. Most of the second half addresses your question.

Exec summary
* Early tests show promise that we may have found a way to make the 
ultra-low queuing delay of data centre TCP incrementally deployable 
on the public Internet
* For rtcweb, we need to address
   a) cc for r-t media [rmcat w-g in progress]
   b) Making TCP nicer
   c) minimise ability of TCP to bloat queues [AQM w-g now in progress]
   This addresses b) & c)

The problem
* All AQMs delay dropping for about one (hard-coded) worst-case RTT, 
in case a burst dissipates (allegedly a 'good queue' according to Van Jacobson)
* For a flow with 1/10 or 1/100 of this RTT (e.g. from a CDN or your 
home media server), any congestion signal is delayed tens or hundreds 
of its own RTTs by these AQMs.
* A TCP flow in slow-start doesn't need the burst smoothed anyway
   - delaying the signal just makes slow-start overshoot more
   - a TCP in slow-start knows that it won't allow the burst to 
dissipate anyway

The solution: make ECN also mean "Immediate Congestion Notification"?
* For ECN-capable packets, shift the job of hiding bursts from network to host:
   - the network signals ECN with no smoothing delay
   - then the transport can hide bursts of ECN signals from itself
   - the transport knows
     o whether it's TCP or RTP etc,
     o whether its in congestion avoidance or slow-start,
     o and it knows its RTT,
     o so it can know whether to respond immediately or to smooth the signals,
     o and if so, over what time
   - then short RTT flows can smooth the signals with only the delay 
of their /own/ RTT
     o so they can fill troughs and absorb peaks that longer RTT flows cannot
   - a TCP only needs to smooth the signals if in congestion avoidance
     o in slow start, it can respond immediately, thus reducing overshoot

Incremental Deployment:
* Immediate congestion notification doesn't need new AQM implementation
   - it can use the widely implemented WRED algorithm with an 
unexpected configuration
* The network classifies packets for this AQM treatment based on 
their ECN-capability
   - Without ECN, it smoothes the queue before signalling drops
   - With ECN, it signals immediately, without any smoothing delay
   - (as today, the operator can still use WRED with the Diffserv field too)
* For TCP apps, the stack will use 'DCTCP' (we've tweaked it), if the 
ends negotiate ECN with the accurate feedback capability.
* It should 'just work' if an RTP app or a Reno TCP uses ECN.

The request:
* Much more evaluation to do, but first we want to know:
   - if the idea works, would the IETF have an appetite for tweaking 
the definition of ECN so it is merely equivalent to drop in the long 
term, but the dynamics need not be equivalent.

Much better than the ECN that didn't get deployed
* This is Explicit and Immediate Congestion Notification (EICN?)
   - same wire protocol, much greater benefits
* The advantage of the original ECN (avoiding congestive loss) was 
too small to be worth the deployment hassle
* Predictable ultra-low latency without loss too (similar to 
DCTCP-ECN) would be worth deploying
* But we all thought DCTCP could only be deployed in isolation (e.g. 
data centres)
   - we all thought DCTCP traffic would starve alongside today's TCP traffic
   - because in a DCTCP queue, the ECN threshold is lower than you 
would trigger drop
   - and we thought ECN & drop had to be equivalent.
* We believe we've found a way to ensure DCTCP-ECN traffic doesn't starve
   - we still make DCTCP-ECN equivalent to drop in the long-run, but 
not in its dynamics

More info
A paper with early results is under submission  (available on 
request). It shows neither this approach nor legacy traffic can 
starve the other under a wide range of conditions, but we have a lot 
more work to test the limits.

Bob

---

From: John Leslie <[email-masked]>
Date: Sat, 26 Oct 2013 08:16:32 -0400

(If this continues, we've got to fix the title and cross-posting...)

> in case a burst dissipates (allegedly a 'good queue' according to Van 

   This assertion is going to need a lot of support.

   Bob is a man after my own heart suggesting that an ECN notification
may be sent earlier than a packet drop would be indicated. I don't know
if we can get there; but IMHO that is essential to getting ECN deployed
and used.

   I don't think I agree with Bob that what's hard-coded is necessarily
a "worst-case" RTT -- and I'm quite sure I'm not willing to make any
pronouncement about "all AQMs".

   I suggest the talk might be more useful if Bob outlined the AQMs
currently in widespread use and detailed _how_ they delay dropping
for an estimated RTT.

> of its own RTTs by these AQMs.

   Clearly, RTTs differing by a factor of ten are quite common at most
nodes traversed in a typical path; and it seems _very_ suboptimal to
have the responsibility for guessing the RTT at the node which must
drop packets.

>   - a TCP in slow-start knows that it won't allow the burst to 

   A critical point! (It seems obvious to me, but is it obvious to
everyone?)

>   - then the transport can hide bursts of ECN signals from itself

   But can we get there from here?

   The node doing the ECN notification _can't_ know how the transport
will react; and the transport receiving and ECN notification can't know
whether the forwarding node has "smoothed" the signal. (It is truly a
shame we haven't left any bits for signals like this!)

>     o and if so, over what time

   Yes, but it can't know what smoothing may already have been applied.

>     o in slow start, it can respond immediately, thus reducing overshoot

   This would, IMHO, improve "slow start".

>   - it can use the widely implemented WRED algorithm with an 

   Bob is beginning to lose me here. Does he mean that a forwarding node
would apply WRED for both drop and ECN, but with different parameters?

>   - Without ECN, it smoothes the queue before signalling drops

   Bob has lost me now -- apparently he doesn't mean different
parameters... and I don't recognize this "smoothing" step in WRED.

>   - (as today, the operator can still use WRED with the Diffserv field too)

   (Do we need to confuse this discussion by adding diffserv?)

> ends negotiate ECN with the accurate feedback capability.

   Have we settled on "accurate feedback" already? I thought that was
still under discussion. (I don't follow exactly what it adds...)

> * It should 'just work' if an RTP app or a Reno TCP uses ECN.

   I don't see any way for a Reno transport using ECN to avoid being
starved if ECN arrives earlier (without notice).

> term, but the dynamics need not be equivalent.

   There's a good question there; but I don't think we're ready for it.

   I'd really like to discuss the dynamics of responding more quickly
but perhaps less drastically for almost any real-time flow.

   But proving "equivalence in the long term" seems too hard.

> too small to be worth the deployment hassle

   Actually, I don't agree that was the problem -- instead I believe
the code has been deployed but administratively suppressed because
the operators don't trust the transports. There _is_ a significant
improvement from one-RTT reaction instead of several (to detect a
drop), but the whole process is just too complicated, while the
opportunity for abuse remains obvious.

> DCTCP-ECN) would be worth deploying

   I'm optimistic that latency will become an easier argument.

>   - and we thought ECN & drop had to be equivalent.

   (I'm not sure we'll succeed at breaking that "equivalence"...)

>   - we still make DCTCP-ECN equivalent to drop in the long-run, but 

   (I'm still not sure it's worth arguing the "long-run".)

--
John Leslie <[email-masked]>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sat, 26 Oct 2013 22:42:23 +0000

HI John,

(aqm chair hat off, and accecn author hat on)

> still under discussion. (I don't follow exactly what it adds...)

The current requirements draft give a concise list of what benefits it should bring.

There is also the (individual) draft on the TCP ECN modifications, that try to deliver these benefits within the confines of the TCP header without requiring additional bits.

The main issue with traditional ECN/TCP is, that you only get one bit of information per RTT (delivered over N segments sent in an RTT), thus the extent of congestion (number of CE marks per this flows RTT), and the fine-grained sequencing of the marks is lost.

AccECN aims to deliver these two signals to the sender - but what the sender is then doing exactly with this much more detailed information is out of scope for the AccECN drafts so far.

http://tools.ietf.org/html/draft-ietf-tcpm-accecn-reqs

http://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn

( http://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-option )

Any feedback on these drafts during the TCPM session is appreciated.

Richard Scheffenegger

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Oct 2013 01:36:39 +0000

John, inline...

>currently in widespread use and detailed _how_ they delay dropping

You're right. The 'about' in my sentence was meant to indicate some 
leeway. The specifics depend on each AQM...

The only AQM I know of that doesn't smooth over some nominal RTT is 
DCTCP itself.

* CoDel was designed for 'interval' to be a worst-case (largest) RTT, 
which it recommends to be set to 100ms. One the queue has exceeded 
threshold, CoDel delays for time 'interval' before starting to signal 
congestion.

  - I already said how sluggish CoDel would be for a flow with a much 
shorter RTT than CoDel (others have made this point, e.g. for data 
centres : 
https://lists.bufferbloat.net/pipermail/codel/2012-August/000448.html).
  - And in the other direction, we already know that utilisation 
suffers fairly badly for flows with RTT significantly larger than 100ms.

* PIE suppresses all drops for time max_burst (set to 100ms by 
default) from when the drop probability it calculates (but doesn't 
necessarily use) first rises above zero. This is very similar to 
CoDel, and similar comments are applicable.

* RED requires the constant for its exponentially weighted moving 
average (w_q) to be set taking into account how many packets are 
likely to arrive at the link in a 'typical' RTT. Reverse engineering 
the values recommended by Sally Floyd in the RED paper and in her 
famous RED parameters Web page 
<http://www.icir.org/floyd/REDparameters.txt>, she recommended a 
'typical' RTT of about 130ms.

[BTW, I know of people who don't calculate w_q, but just use the 
value of "0.002" that Sally recommended for her 45Mb/s link in the 
original RED paper simulations (and repeated at 
<http://www.icir.org/floyd/REDparameters.txt>). This was calculated 
assuming about 500 packets arrive at a link (from all flows) in a 
typical RTT. Links have got a lot faster since 1993. Nonetheless, she 
was considering 45Mb/s for an aggregated link in those days, and it 
happens to be about right for a single user today.]

>have the responsibility for guessing the RTT at the node which must

For packets that do not support ECN, the dropping node has to make a 
guess at the RTT, so as not to drop packets unnecessarily, because 
drop is an impairment as well as a congestion signal. So a transport 
cannot 'undrop' packets.

Our point though is that a network node doesn't have to mimic this 
behaviour for ECN packets, because ECN is not an impairment. So a 
transport can un-ECN-mark packets (by smoothing out bursts itself).

>shame we haven't left any bits for signals like this!)

Well, we do have ECT(1) still only assigned experimentally and never 
used, which we could decide to use for this immediate ECN. However, 
first I want to see whether people think it might be feasible to just 
redefine the meaning of CE.

Rationale: So few buffers have ECN support turned on anyway that we 
should be able to redefine ECN so that many more will want to turn it on.

For those AQMs that already support ECN, we believe this 
retrospective change will make them only a little worse than they are 
already (and the operator can update them by simple reconfiguration 
anyway, and is more likely to do so, given these are clearly 
early-adopter networks).

>    Yes, but it can't know what smoothing may already have been applied.

Yes. If this is a problem, we will have to consider using ECT(1) not CE.
But it's pretty academic when so few buffers support ECN.

The tiny proportion that do support ECN will already smooth by a 
'typical RTT' of about 100ms.

If a 20ms RTT flow adds smoothing over its own RTT to this, it will 
be smooth over 120ms.
The main problem there is not the extra 20ms, it's the original 
100ms, which we won't lose unless we make this change somehow.

>parameters... and I don't recognize this "smoothing" step in WRED.

I do mean that a forwarding node would apply WRED for both drop and 
ECN, but with different parameters.

Each WRED policy-map includes a setting for this smoothing parameter, 
which Cisco calls the exponential-weighting-constant. Many people 
don't notice it's there and they just leave it at the default. For 
instance, Cisco set it to
2^(-9) ~ 0.002 by default for each of the WRED policy-maps (see 
http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/fswfq26.html#wp1039982).

>    (Do we need to confuse this discussion by adding diffserv?)

A non-Diffserv network still doesn't need to worry about Diffserv.

I put this in parentheses because, if WRED is used today, it is 
usually used with Diffserv, and I didn't want anyone to worry that 
they wouldn't be able to continue to do this (e.g. BT use WRED with 
Diffserv in enterprise networks, as do many other carriers).

>still under discussion. (I don't follow exactly what it adds...)

See response from Richard Scheffenegger. Essentially the TCPM WG has 
accepted the requirements doc, but not decided between the mechanisms on offer.

>starved if ECN arrives earlier (without notice).

We haven't tested legacy Reno with ECN yet (we figured legacy Reno 
without ECN is a lot more prevalent, so focused on this first). 
Nonetheless, Reno-ECN is unlikely to starve, because starvation is 
about long-running behaviour, and once a flow has run for more than a 
couple of 100ms RTTs, the immediate ECN signals should be no 
different from a smoothed ECN. I suspect Reno-ECN might be worse in 
its short-term dynamics. But remember Reno-ECN is likely to be a tiny 
corner-case.

>    There's a good question there; but I don't think we're ready for it.

At this stage, even we haven't got many answers. So I'm not asking 
the IETF to answer the question right now. I'm merely saying, /if/ 
our idea works, is there at least an /appetite/ in the IETF for 
reconsidering the definition of ECN?

We wanted to make the IETF aware of this research early, because it 
might want to at least hold off on any actions that would otherwise 
close off this option.

And if we find that any change is completely out of the question, we 
have to try a different tack (e.g. ECT(1)).

>    But proving "equivalence in the long term" seems too hard.

This should be the easy part, because the longer that conditions are 
stable, a smoothed signal should tend towards an unsmoothed signal, 
all other factors being equal.

Equivalence during dynamics is the hard part, and I'm suggesting we 
don't sweat too much about that, as long as the performance 
evaluations are not too far apart.

>opportunity for abuse remains obvious.

I agree. That's the 'deployment hassle' side of my sentence - the 
extra trust-enhancing mechanisms that seemed necessary were too much 
pain for the small gain.

>    (I'm still not sure it's worth arguing the "long-run".)

I mean competing long-running ECN & non-ECN flows stabilise at 
predictable rates, rather than one ratchetting itself down to nothing 
over time (starvation).

That's the primary concern of congestion control 'fairness', before 
anyone starts worrying about what the relative rates are. Given apps 
get different relative rates with different RTTs, with different size 
objects or by opening multiple flows, we don't need to sweat so much 
about precisely equal flow rates; but we must sweat about stable convergence.

Results so far show that the proposed idea is at least very robust 
against starvation.

Bob

---

From: Dave Taht <[email-masked]>
Date: Sun, 27 Oct 2013 21:22:28 -0700

As is probably well known at this point, I make a clear distinction
between networking problems in the data center and ones in the wild
and wooly world.

The data center portion of the universe is a couple hundred meters in
diameter, the other, I dunno, let's say 8x from here to the moon
(3,075,200,000m).

There are all sorts of things that work and are needed in the data
center that probably won't work outside it. People run straighter
cables, do microwave, use pause frames at layer 2, etc, etc, in order
to wring out the last nanosecond of performance, and certainly running
without loss is important in that world.

>> > * Early tests show promise that we may have found a way to make the

I'm failing to get excited lacking paper, testable code, and other proofs.

>> > ultra-low queuing delay of data centre TCP incrementally deployable

At the moment the edge of the internet is using things like cable and
gpon which have 2ms or more of inherent latency built into their
grant/response structures.

>> >   c) minimise ability of TCP to bloat queues [AQM w-g now in progress]

I do strongly feel that webrtc needs good aqm and packet scheduling in
order to succeed, and that the webrtc folk should be testing out what
has already been developed (red, sfqred, codel, fq_codel, pie) and the
aqm folk testing the webrtc code.

I look forward to some interesting conference calls.

I gave the webrtc codebases a whirl against what aqm and packet
scheduling techniques we have already this past summer, the initial
results were encouraging, but I was able to easily crash most of the
browsers sooner than I could take a good, repeatable set of
measurements. That said, things over there are moving along smartly.

> You're right. The 'about' in my sentence was meant to indicate some leeway.
> The specifics depend on each AQM...
>
> The only AQM I know of that doesn't smooth over some nominal RTT is DCTCP
> itself.

DCTCP fits uncomfortably into the aqm catagory.

>
> * CoDel was designed for 'interval' to be a worst-case (largest) RTT, which
> it recommends to be set to 100ms. One the queue has exceeded threshold,

Which is 5ms of delay.

> CoDel delays for time 'interval' before starting to signal congestion.

Which it then decreases until it finds something approximating the
RTT. For a system in a reasonably steady state, it will find a decent
value and stay there.

For bursty traffic it is not necessarily helpful, and slow start is
interesting...

>  - I already said how sluggish CoDel would be for a flow with a much shorter

But you've never published a measurement, unless there is one in your new paper?

> RTT than CoDel (others have made this point, e.g. for data centres :

You might want to read the full thread... a stumbling block on fixing
the srtt was the stochastic hashing... so now there is a full blown
non-approximated fq scheduler that implements pacing.

http://www.ietf.org/mail-archive/web/aqm/current/msg00259.html

And I look forward to a certain upcoming presentation in ICCRG with
great anticipation as to additional fallout from this....

>  - And in the other direction, we already know that utilisation suffers
> fairly badly for flows with RTT significantly larger than 100ms.

"target" and "interval" have always been variables in the fq_codel and
codel codebase and ecn is supported.

tc qdisc add dev your_device root fq_codel target 500us interval 10ms ecn

I don't think the pain of adding that one line of configuration would
affect a data center deployment any. I would certainly like to see
benchmarks in a data center environment. I personally lack 10Gig hw.

I note that once you get below 1ms on a typical box today, you start
hitting other bottlenecks in (for example) the cpu scheduler.

> * PIE suppresses all drops for time max_burst (set to 100ms by default) from
> when the drop probability it calculates (but doesn't necessarily use) first
> rises above zero. This is very similar to CoDel, and similar comments are
> applicable.

This too has a variable target value, although there are several other
magic constants that I don't fully understand. I would certainly like
to see some benchmarks in a data center environment. The pie code I
have for linux shoots for a target of 20ms by default, for some
reason.

> * RED requires the constant for its exponentially weighted moving average
> (w_q) to be set taking into account how many packets are likely to arrive at
> the link in a 'typical' RTT. Reverse engineering the values recommended by
> Sally Floyd in the RED paper and in her famous RED parameters Web page
> <http://www.icir.org/floyd/REDparameters.txt>, she recommended a 'typical'
> RTT of about 130ms.
>
> [BTW, I know of people who don't calculate w_q, but just use the value of
> "0.002" that Sally recommended for her 45Mb/s link in the original RED paper
> simulations (and repeated at <http://www.icir.org/floyd/REDparameters.txt>).
> This was calculated assuming about 500 packets arrive at a link (from all
> flows) in a typical RTT. Links have got a lot faster since 1993.
> Nonetheless, she was considering 45Mb/s for an aggregated link in those
> days, and it happens to be about right for a single user today.]
>
>

Congestion takes time to occur. Consider tcp dynamics as flows also
only increase relative to RTT, as well.

A core question is what do you want response time to congestion to be under?

100ms? 5ms? 1ms? 500us?

>> nodes traversed in a typical path; and it seems _very_ suboptimal to

2.4 billion folk connect to the internet generally at RTTs between 20 and 80ms.

In a data center environment I have no data.

>> have the responsibility for guessing the RTT at the node which must

I certainly favor the ongoing development of end to end cc capable of
monitoring the RTT and doing the right thing.

> For packets that do not support ECN, the dropping node has to make a guess
> at the RTT, so as not to drop packets unnecessarily, because drop is an
> impairment as well as a congestion signal. So a transport cannot 'undrop'
> packets.
>
> Our point though is that a network node doesn't have to mimic this behaviour
> for ECN packets, because ECN is not an impairment. So a transport can
> un-ECN-mark packets (by smoothing out bursts itself).
>
>

The end of slow start is an ECN notification or a packet drop. What am
I missing here?

>> > * For ECN-capable packets, shift the job of hiding bursts from network
>> > to
>> > host:
> Well, we do have ECT(1) still only assigned experimentally and never used,
> which we could decide to use for this immediate ECN. However, first I want
> to see whether people think it might be feasible to just redefine the
> meaning of CE.

A lot of the NONCE logic has been discussed in other rfcs.

>
> Rationale: So few buffers have ECN support turned on anyway that we should
> be able to redefine ECN so that many more will want to turn it on.
>
> For those AQMs that already support ECN, we believe this retrospective
> change will make them only a little worse than they are already (and the
> operator can update them by simple reconfiguration anyway, and is more
> likely to do so, given these are clearly early-adopter networks).
>
>
>> >   - the transport knows
> The tiny proportion that do support ECN will already smooth by a 'typical
> RTT' of about 100ms.
>
> If a 20ms RTT flow adds smoothing over its own RTT to this, it will be
> smooth over 120ms.
> The main problem there is not the extra 20ms, it's the original 100ms, which
> we won't lose unless we make this change somehow.
>
>
> I do mean that a forwarding node would apply WRED for both drop and ECN, but
> with different parameters.
>
> Each WRED policy-map includes a setting for this smoothing parameter, which
> Cisco calls the exponential-weighting-constant. Many people don't notice
> it's there and they just leave it at the default. For instance, Cisco set it
> to
> I put this in parentheses because, if WRED is used today, it is usually used
> with Diffserv, and I didn't want anyone to worry that they wouldn't be able
> to continue to do this (e.g. BT use WRED with Diffserv in enterprise
> networks, as do many other carriers).

Perhaps we need a taxonomy so we can all talk to areas of the network
we care about? RED as currently defined will not work correctly on
variable rate networks, like wireless...

I have never thought we'd end up with one aqm or tcp to rule them all..

DC: Data Center
WI: Wireless
FL: Fixed Line

> We haven't tested legacy Reno with ECN yet (we figured legacy Reno without
> ECN is a lot more prevalent, so focused on this first). Nonetheless,
> Reno-ECN is unlikely to starve, because starvation is about long-running
> behaviour, and once a flow has run for more than a couple of 100ms RTTs, the
> immediate ECN signals should be no different from a smoothed ECN. I suspect
> Reno-ECN might be worse in its short-term dynamics. But remember Reno-ECN is
> likely to be a tiny corner-case.
>
>
>> > The request:
> At this stage, even we haven't got many answers. So I'm not asking the IETF
> to answer the question right now. I'm merely saying, /if/ our idea works, is
> there at least an /appetite/ in the IETF for reconsidering the definition of
> ECN?

I wouldn't mind it, but frankly my own concern was addressing the
security issue, not the current definition.

> We wanted to make the IETF aware of this research early, because it might
> want to at least hold off on any actions that would otherwise close off this
> option.

>
> And if we find that any change is completely out of the question, we have to
> try a different tack (e.g. ECT(1)).
>
>
> This should be the easy part, because the longer that conditions are stable,
> a smoothed signal should tend towards an unsmoothed signal, all other
> factors being equal.
>
> Equivalence during dynamics is the hard part, and I'm suggesting we don't
> sweat too much about that, as long as the performance evaluations are not
> too far apart.
>
>
> I agree. That's the 'deployment hassle' side of my sentence - the extra
> trust-enhancing mechanisms that seemed necessary were too much pain for the
> small gain.

I

>> >   - and we thought ECN & drop had to be equivalent.

I'm looking forward to being convinced otherwise.

> I mean competing long-running ECN & non-ECN flows stabilise at predictable
> rates, rather than one ratchetting itself down to nothing over time
> (starvation).
>
> That's the primary concern of congestion control 'fairness', before anyone
> starts worrying about what the relative rates are. Given apps get different
> relative rates with different RTTs, with different size objects or by
> opening multiple flows, we don't need to sweat so much about precisely equal
> flow rates; but we must sweat about stable convergence.
>
> Results so far show that the proposed idea is at least very robust against
> starvation.
>
>
> Bob
>
>
>> --

-- 
Dave Täht

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Oct 2013 07:16:49 +0000

Dave,

>between networking problems in the data center and ones in the wild

The two are converging. It will gradually become 
less sensible to think of the two as separate.

>diameter, the other, I dunno, let's say 8x from here to the moon

DCTCP was fixed ages ago for global RTTs. 
Download the Linux version from Stanford.

>without loss is important in that world.

Not relevant. Don't be confused by the name 
DCTCP. It's only a few lines of code different 
from TCP, and it's not only relevant in data 
centres. It's for low queuing delay.

It's only confined to a data centre while we 
can't work out how it co-exists with current 
Internet traffic - that's what we're doing.

>I'm failing to get excited lacking paper, testable code, and other proofs.

I'll send the paper, but agreed we've only just 
got started, so you don't have to be excited yet. A lot more testing to do.

>grant/response structures.

You're saying you're not interested in low 
queuing delay solutions because there's often 
fluff in badly designed access networks anyway? 
That sounds like a poor excuse for not aspiring 
to lower delay. What excuse is there for the 
100ms of signalling delay on ECN packets in CoDel 
then? That's 50 times more than this 2ms 
(currently) inherent access network delay (that 
some of us are working to remove as well).

If you put an AQM in a host that is the 
bottleneck for a transfer to your media server in 
the next room of your house, at the same time as 
transfers to the wide area, you really don't want 
to introduce a nominal global RTT into both feedback loops.

>aqm folk testing the webrtc code.

The AQM for DCTCP was developed 2 years before 
all of those (except RED). Why decide to build on 
CoDel in 2012, when DCTCP had already been out 
there and deployed for 2 years with excellent 
results? And tcp-rcv-cheat had been out there for 5 years.

>DCTCP fits uncomfortably into the aqm catagory.

Oh dear. Is this Dave Taht saying "I don't like 
it cos it's different to tradition"?

DCTCP shows just how low you can get queuing 
delay even with a brain-dead simple step AQM as 
long as you fix the right problem (smooth in TCP, 
not in the AQM). Doesn't this tell you something 
about where the problem probably was?

> > it recommends to be set to 100ms. One the queue has exceeded threshold,

105ms of delay on top of RTT delay before it can possibly do anything.

By far the most common case on an access link is 
a flow arriving when the link's empty.

>RTT. For a system in a reasonably steady state, it will find a decent

Until the queue empties (which it should if the 
control system is working), then it has to start 
over, because another flow may have a different RTT.

>there is one in your new paper?

Lots of experiments in the paper, but not the 
dynamics yet - we deliberately focused on the 
long-running behaviour first, to check the critical starvation issues first.

>http://www.ietf.org/mail-archive/web/aqm/current/msg00259.html

As I understand it, this only works for an AQM on 
a host that has the benefit of being told the RTT from higher up the stack.

>codel codebase

It's not use having variables you don't know how 
to set. That was the lesson from RED that DCTCP 
started to solve (concerning variable RTT), and 
CoDel continued to solve (concerning line 
variable rate). However, unfortunately CoDel chose the not invented here route.

>and ecn is supported.

AFAICT, ECN is only supported in CoDel in the 
sense that it is treated as equivalent to drop. 
The factors where ECN is not equivalent to drop have not been catered for.

And ECN certainly hasn't been considered as 
something to exploit beyond what drop can do. For 
instance, to take the approach I'm suggesting, 
but using CoDel not RED, for ECN-capable packets 
you would use interval as zero. Then the 
end-system would smooth out bursts of ECN marks instead of CoDel doing it.

>hitting other bottlenecks in (for example) the cpu scheduler.

As above, not a good reason. We have to knock 
down each source of delay, one after the other.

>magic constants that I don't fully understand.

It takes a few minutes of reading to understand 
all the PIE variables (about the same complexity 
as CoDel). I think PIE has the following constants (defaults in []):
* target_del (the target queuing delay [20ms] as you say)
* max_burst (has a similar role to interval in CoDel [100ms])
* deq_threshold (there's enough packets in the 
queue to measure the line-rate [10KB])
* Tupdate, which determines how often the line 
rate is tested (which shouldn't affect 
performance unless the machine can't cope with a high enough update rate).

All the other variables are dependents of these, 
or autotuned (in the case of alpha and beta).

>I would certainly like
>to see some benchmarks in a data center environment.

Just to be clear, for this conversation I'm not 
focused on a data centre environment - I'm making 
DCTCP applicable to the public Internet.

>The pie code I
> > simulations (and repeated at 
> <http://www.icir.org/floyd/REDparameters.txt>).

Er no. As a queue builds, it causes delay. 
Regular TCP makes this worse than it has to be by 
using large saw-teeth, so they become a large 
inherent variable source of delay. VJ calls this 
a good queue. But if a variant of TCP can remove 
this variable source of delay, it can't be good.

>A core question is what do you want response time to congestion to be under?

Wrong question. Answer depends on RTT. That's the 
whole point. Consider that you may have been brain-washed.

>generally at RTTs between 20 and 80ms.

The Internet includes your home network, 
potentially a cache in your home gateway, I/O 
lanes within your own machine, etc etc. No-one 
"connects to the Internet at an RTT of" anything. 
Today, all these 2.4B folk use connections over 
the Internet with a very wide range of RTTs.

>In a data center environment I have no data.

Typically 100us or 200us base RTT. And all that 
cheap data centre kit can be and will be re-used in the public Internet too.

>monitoring the RTT and doing the right thing.

Good.

>The end of slow start is an ECN notification or a packet drop. What am

A traditional AQM (RED, CoDel, PIE) suppresses 
any signal (ECN or drop) for another 100ms or so 
after slow start has pushed the queue over the 
delay threshold. That's on top of the feedback 
delay of 1RTT, because it doesn't know this is 
slow-start, so it waits to see if the queue goes away.

The transport knows whether it's in slow-start, 
so it doesn't need to do any smoothing here - it 
can drop straight out of slow-start without smoothing.

The transport knows when it's in congestion 
avoidance too, of course, then it can smooth out 
bursts of ECN by doing its own EWMA over the 
actual RTT (that it knows properly).

Ie the AQM can't be selective about when it 
smooths bursts, because it doesn't know what 
phase each transport flow is in, but each transport does and therefore can.

>I have never thought we'd end up with one aqm or tcp to rule them all..

Again, this immediate ECN stuff is for public 
Internet (not just data centres).

>security issue, not the current definition.

It seems strange that you weren't aware of solutions to this then.

>I'm looking forward to being convinced otherwise.

Yup, I don't want to take up too much of anyone's 
time with this until we've proved it. But we 
decided a heads up was important. That's all.

Bob