Subject: Re: [aqm] ECT(1)

From: Bob Briscoe <[email-masked]>
Date: Thu, 1 Oct 2015 11:46:59 +0100

Andrew,

DualQ might be a red herring for the special case you are thinking of 
within the Googleplex. But only a company with a brain the size of 
Google could countenance building bandwidth enforcer (BwE), in the 
belief that knowledge can be collected of the relative priorities, QoS 
requirements, bandwidth requirements, timing and location of all its 
applications. Then it might make sense to collect demand from all hosts 
up a control hierarchy and drive rate-limiting policies back down the 
hierarchy to every host which then sets Diffserv markings for all its 
different apps which determine routing and scheduling against a 
hierarchically organised master-plan.

But for the rest of us surely it's BwE that is the red herring - I mean 
us outside Google's internal networks, on the Internet, which consists 
of independent actors with no master Gosplan coordinating us all.

For most of the billions on the Internet who are not on large campus 
networks, geography and economics determines that the access network is 
the bottleneck - the part that is most spread-out physically and 
therefore least able to advantage from economies of scale (aggregation).

For instance, to demonstrate Coupled DualQ we chose the downstream queue 
in a broadband network gateway (BNG) - each of these queues is shaped to 
the rate of each residential broadband line. It would also be applicable 
for the upstream queue in each residential gateway. And it would also be 
applicable for the equivalent queue entering the access network from 
each end in other access network technologies: the HFC-node and the 
cable modem in cable, or the RNC and the user-equipment in cellular.

In these logical links, traffic is only for one user or one small site 
(e.g. a home) so it is very sparse. Diffserv is for aggregated traffic - 
Diffserv is useless if all traffic to and from a home at any one time is 
latency sensitive (e.g. Web, VoIP, and interactive video).

The simple argument for why DualQ:
a) Today's 'classic' TCPs are the problem in these sparse access links, 
the drastic saw-teeth vary the queue {Note 1}
b) Even with a good replacement TCP with small sawteeth (like DCTCP), it 
won't be able to keep queuing delay low if there might be 'classic' TCP 
traffic in the same queue, which implies 2 queues minimum.
c) I don't care how we do that, but the coupling was a neat way to do 
that without the network operator making /any/ capacity management 
judgements on behalf of the users.

An Internet service that gives /all/ traffic ultra-low queueing delay 
and near-zero congestion loss would be extremely valuable. Not having to 
ask permission for QoS is easy for both customer and network operator. 
Capacity sharing is then orthogonal to that.

Just because we show that TCP Cubic and DCTCP flows will get the same 
rates within the user's own line, that doesn't stop the user's 
applications using more or less aggressive congestion controls, or none 
at all, or multiple flows - all the usual ways of varying capacity 
shares still apply when the network operator doesn't enforce anything.

Actually, even if you do capacity sharing with BwE, if some hosts are 
using a 'classic' TCP, they will still cause varying queuing delay for 
other traffic. However, in your WAN scenario you might have sufficient 
aggregation for the resulting queuing delay not to be a problem.

Bob

{Note 1} I expected the slow-start overshoot would also be a problem, 
but (so far) just reducing the congestion avoidance sawteeth seems to 
remove virutally all the queuing delay - probably because we haven't 
done tests with sparse but bursty workloads yet. Anyway, we've got ideas 
on how to reduce the slow-start overshoot once ECN marking is more frequent.

On 07/08/15 01:51, Andrew Mcgregor wrote:
> I believe getting a DSCP deployed for this is a non-starter; that 
> space is a complete mess, and if we tie this proposal to cleaning up 
> that mess we'll get nowhere.  The evil bit doesn't fly either, for a 
> lot of reasons.
>
> That leaves us with ECT(1).
>
> So, how bad is that?  Well, not very.  In places which are ECN-enabled 
> but not dual-queue, DCTCP or another 1/p wrt ECN marking transport 
> will still respond to loss; sure, we'll drive the network into a lossy 
> regime, but those parts of the network are guaranteed to be there 
> anyway due to the presence of TCP.  Provided the loss response is 
> still sane, that will equilibrate out and we will end up with a 
> reasonable outcome. No, it will not be fair, but in the real world TCP 
> never is since the equilibrium takes hours to establish while 
> essentially no flows last that long; in real world practice, the few 
> flows that do last that long need special protection because they are 
> so fragile (BGP, I'm looking at you), or else nobody really cares how 
> long they take to complete.  TCP does not share capacity in any 
> reasonable manner, it is a circuit breaker for avoiding congestion 
> collapse, and nobody is proposing running completely oblivious traffic 
> on the internet.  DCTCP still has enough congestion avoidance.
>
> The dual-queue solution is not the only way to deploy either; it's a 
> nice solution, if one actually wants to achieve capacity sharing by 
> router feedback, but there are other ways to do the capacity sharing 
> (for example 
> http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p1.pdf). In an 
> admission controlled environment, separating the queues is not 
> necessary for sane coexistence.  Further I'm sure there are other 
> single-queue solutions, involving AQM control systems, although I 
> don't have a proposal right now (nor do I think one is necessary).
>
> I do think that assuming dual queue to be necessary for deployment is 
> a red herring.
>
> On 7 August 2015 at 06:38, Mikael Abrahamsson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On Tue, 4 Aug 2015, Bob Briscoe wrote:
>
>         *Combining ECT(0) and CE with a globally assigned DSCP solely
>         during initial deployment of L4S seems the least worst choice.
>
>
>     Having the same bits in the header mean different things in
>     combination with DSCP seems like a really hard to get deployed
>     Internet-wide.
>
>     ECN is just now gaining traction and seems like it might actually
>     see real deployment. Repurposing those bits just now would most
>     likely just cause confusion.
>
>     I started using ECN when it first appeared in the Linux kernel
>     around 2001 or whenever it was. I had to immediately turn it off
>     because some firewalls dropped those packets. Now almost 15 years
>     later after this sitting in the operating systems for at least 10
>     years, we're now getting to a point where we're ready to start
>     turning it on widely because things do not break when it's turned on.
>
>     So whatever you come up with now that requires host stack changes,
>     expect 5-10 years at least until it can be deployed. This means
>     you have to be really sure this is what you actually want before
>     you start to push for deployment. Also, deployment impacts should
>     be taken a lot into account when deciding what to do.
>
>     So how sure are you that L4S as it currently stands is the way to
>     go? If you think you're going to invent something new in 2-3
>     years, then please wait until then. Experimentation is all fine
>     and dandy, but until we can actually get DSCP codepoints working
>     on Internet-wide scale, this approach isn't feasable for that
>     use-case (which for me is close to "the only" use-case).
>
>     My proposal has been before that we should try to get 7 DSCP
>     codepoints deployed by using 000xxx, and nudge providers to
>     incrementally just not bleach them and treat them as BE in their
>     core networks, so we can use them on the edge to influence AQM there.
>
>     So, if we're going to invent new meaning of ECN bits in
>     combination with DSCP, then that needs to be coupled with work of
>     getting some DSCP working Internet-wide in a fashion that someone
>     actually believes will work out, as in actually getting
>     significant Internet-wide deployment.
>
>     -- 
>     Mikael Abrahamsson    email: [email-masked]
>     <mailto:[email-masked]>
>
>
>     _______________________________________________
>     aqm mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/aqm
>
>
>
>
> -- 
> Andrew McGregor | SRE [email-masked] 
> <mailto:[email-masked]> | +61 4 1071 2221
>
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Andrew Mcgregor <[email-masked]>
Date: Fri, 2 Oct 2015 13:00:03 +1000

On 1 October 2015 at 20:46, Bob Briscoe <[email-masked]> wrote:

> within the Googleplex. But only a company with a brain the size of Google
> could countenance building bandwidth enforcer (BwE), in the belief that
> knowledge can be collected of the relative priorities, QoS requirements,
> bandwidth requirements, timing and location of all its applications. Then
> it might make sense to collect demand from all hosts up a control hierarchy
> and drive rate-limiting policies back down the hierarchy to every host
> which then sets Diffserv markings for all its different apps which
> determine routing and scheduling against a hierarchically organised
> master-plan.
>

Well, it does make sense, the paper isn't theoretical but deployed reality
(it takes a fairly large team to maintain the system, even though the
majority of the data collection is automated).

But that wasn't really my point... it's that alternate capacity sharing
mechanisms can be envisaged that don't involve solely deduction from router
signals.  BwE is a configured system for sharing capacity.  One can imagine
a distributed, signalled system that would do something similar, but with
policy communicated from the users; obviously there are enormous trust and
security issues there, but also quite a potential benefit.  That, however,
would be an entirely different project than either AQM or TCP-Prague is
heading for, and complementary to them.

> But for the rest of us surely it's BwE that is the red herring - I mean us
> outside Google's internal networks, on the Internet, which consists of
> independent actors with no master Gosplan coordinating us all.
>

No single plan, no, but anyone with a substantial CDN has to coordinate
with the providers in which the CDN edge is embedded, so there are
negotiated plans for coordination.  It isn't inconceivable to automate that
negotiation, to achieve the benefits of time-of-day optimisation etc.

> networks, geography and economics determines that the access network is the
> bottleneck - the part that is most spread-out physically and therefore
> least able to advantage from economies of scale (aggregation).
>

Clearly true.

For instance, to demonstrate Coupled DualQ we chose the downstream queue in
> a broadband network gateway (BNG) - each of these queues is shaped to the
> rate of each residential broadband line. It would also be applicable for
> the upstream queue in each residential gateway. And it would also be
> applicable for the equivalent queue entering the access network from each
> end in other access network technologies: the HFC-node and the cable modem
> in cable, or the RNC and the user-equipment in cellular.
>
> c) I don't care how we do that, but the coupling was a neat way to do that
> without the network operator making /any/ capacity management judgements on
> behalf of the users.
>
> An Internet service that gives /all/ traffic ultra-low queueing delay and
> near-zero congestion loss would be extremely valuable. Not having to ask
> permission for QoS is easy for both customer and network operator. Capacity
> sharing is then orthogonal to that.
>

Indeed, that's an outstanding goal.  Please don't think I'm against it!

Just because we show that TCP Cubic and DCTCP flows will get the same rates
> within the user's own line, that doesn't stop the user's applications using
> more or less aggressive congestion controls, or none at all, or multiple
> flows - all the usual ways of varying capacity shares still apply when the
> network operator doesn't enforce anything.
>

I should say, I really like this work, and it's a great demonstration of
one way of doing something really good.  I'm simply saying, I think we
should look wider than this particular solution for a while before trying
to converge the evolution of ECN and DCTCP, especially since there are
going to be places where for one reason or another deploying DualQ as
specified is pretty difficult, but something sufficiently similar might be
doable if the transport has just enough flexibility.

> aggregation for the resulting queuing delay not to be a problem.

Small enough buffers that we don't care about queue delay fluctuations,
actually.  But yes, large scale means that the classic-ish TCP doesn't
matter very much (although it still isn't a capacity sharing mechanism).

>
>
>
> Bob
>
> {Note 1} I expected the slow-start overshoot would also be a problem, but
> (so far) just reducing the congestion avoidance sawteeth seems to remove
> virutally all the queuing delay - probably because we haven't done tests
> with sparse but bursty workloads yet. Anyway, we've got ideas on how to
> reduce the slow-start overshoot once ECN marking is more frequent.
>
>
>
>
> I believe getting a DSCP deployed for this is a non-starter; that space is
> a complete mess, and if we tie this proposal to cleaning up that mess we'll
> get nowhere.  The evil bit doesn't fly either, for a lot of reasons.
>
> So, how bad is that?  Well, not very.  In places which are ECN-enabled but
> not dual-queue, DCTCP or another 1/p wrt ECN marking transport will still
> respond to loss; sure, we'll drive the network into a lossy regime, but
> those parts of the network are guaranteed to be there anyway due to the
> presence of TCP.  Provided the loss response is still sane, that will
> equilibrate out and we will end up with a reasonable outcome.  No, it will
> not be fair, but in the real world TCP never is since the equilibrium takes
> hours to establish while essentially no flows last that long; in real world
> practice, the few flows that do last that long need special protection
> because they are so fragile (BGP, I'm looking at you), or else nobody
> really cares how long they take to complete.  TCP does not share capacity
> in any reasonable manner, it is a circuit breaker for avoiding congestion
> collapse, and nobody is proposing running completely oblivious traffic on
> the internet.  DCTCP still has enough congestion avoidance.
>
> The dual-queue solution is not the only way to deploy either; it's a nice
> solution, if one actually wants to achieve capacity sharing by router
> feedback, but there are other ways to do the capacity sharing (for example
> <http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p1.pdf>
> admission controlled environment, separating the queues is not necessary
> for sane coexistence.  Further I'm sure there are other single-queue
> solutions, involving AQM control systems, although I don't have a proposal
> right now (nor do I think one is necessary).
>
> I do think that assuming dual queue to be necessary for deployment is a
> red herring.
>
> On 7 August 2015 at 06:38, Mikael Abrahamsson <[email-masked]> wrote:
>
>> *Combining ECT(0) and CE with a globally assigned DSCP solely during
>>> initial deployment of L4S seems the least worst choice.
>>>
>>
>> Having the same bits in the header mean different things in combination
>> with DSCP seems like a really hard to get deployed Internet-wide.
>>
>> ECN is just now gaining traction and seems like it might actually see
>> real deployment. Repurposing those bits just now would most likely just
>> cause confusion.
>>
>> I started using ECN when it first appeared in the Linux kernel around
>> 2001 or whenever it was. I had to immediately turn it off because some
>> firewalls dropped those packets. Now almost 15 years later after this
>> sitting in the operating systems for at least 10 years, we're now getting
>> to a point where we're ready to start turning it on widely because things
>> do not break when it's turned on.
>>
>> So whatever you come up with now that requires host stack changes, expect
>> 5-10 years at least until it can be deployed. This means you have to be
>> really sure this is what you actually want before you start to push for
>> deployment. Also, deployment impacts should be taken a lot into account
>> when deciding what to do.
>>
>> So how sure are you that L4S as it currently stands is the way to go? If
>> you think you're going to invent something new in 2-3 years, then please
>> wait until then. Experimentation is all fine and dandy, but until we can
>> actually get DSCP codepoints working on Internet-wide scale, this approach
>> isn't feasable for that use-case (which for me is close to "the only"
>> use-case).
>>
>> My proposal has been before that we should try to get 7 DSCP codepoints
>> deployed by using 000xxx, and nudge providers to incrementally just not
>> bleach them and treat them as BE in their core networks, so we can use them
>> on the edge to influence AQM there.
>>
>> So, if we're going to invent new meaning of ECN bits in combination with
>> DSCP, then that needs to be coupled with work of getting some DSCP working
>> Internet-wide in a fashion that someone actually believes will work out, as
>> in actually getting significant Internet-wide deployment.
>>
>> --
> Andrew McGregor | SRE |  <[email-masked]>[email-masked] | +61
> 4 1071 2221
>
>
> aqm mailing [email-masked]://www.ietf.org/mailman/listinfo/aqm
>
>
> --

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 1071 2221

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 16 Oct 2015 01:24:25 +0100

Jonathan,

Will you be writing up this ELR idea soon? You might not realise that 
your descriptions of it so far in emails have not been understandable 
(to me), so a proper write up is needed. I'm not even sure what problem 
it is trying to solve (don't spend time explaining by email - write it 
up properly pls).

We are currently finalising an Internet draft on an identifiers for L4S 
(aimed for the tsvwg, which owns ECN and Diffserv standardisation, but 
I'll cross-post to aqm too). It should appear before Monday's IETF 
deadline. It will be written as if it is a draft to standardise the 
chosen identifier), with all the proposals and their pros and cons in an 
appendix.

Inline...

On 07/08/15 08:55, Jonathan Morton wrote:
>
> A flow-isolating AQM such as fq_codel would be able to deal with 
> different flows responding differently to CE.
>
> The problem is that core and semi-core network nodes don't have the 
> resources to do flow isolation at line rate, or (equivalently) they 
> deal with too many simultaneous flows to make a meaningful flow 
> isolation scheme practically implementable.  This, I assume, is where 
> the dual queue proposal comes in.
>
Yes, and because flow-queuing overrides the packet rate choices of 
applications (breaking the end-to-end principle). In testing, fq 
completely rips apart CBR and variable rate video when running alongside 
long-running TCP flows, when they would otherwise work just fine in a FIFO.

I've been whingeing about this ever since fq_CoDel was first mooted. But 
I didn't want to make too much noise until we had fully tested a better 
alternative.

> Fundamentally, dual queue needs to be able to identify the packets 
> belonging to each queue.  DSCP is a non-starter because it doesn't, in 
> practice, survive end to end.
>
Not a complete non-starter. A valid deployment model is for ISPs to 
initially deploy L4S for their own services (CDNs, premium services etc) 
as local-network only. Then as more e2e ECN sources migrate to L4S (and 
any attempt to use the DSCP gets bleached), each ISP could choose to 
remove the DSCP rule from its L4S queues leaving only the ECN rule, so 
that ECN alone could become the identifer, and it would work e2e.

At the moment we're favouring ECT(1), but I keep wavering as I find new 
issues with each approach. Watch for the draft.
>
> I'm wary of using the ECN field for the same purpose - it could work, 
> but promotimg CE packets from classic ECN flows could result in 
> spurious retransmissions due to reordering.
>
I'd need the experts to confirm this, but I don't think this would lead 
to spurious retransmits - at least very rarely - and very rare spurious 
retransmissions are not a problem.

Reason: If a classic CE-marked packet arrives earlier than the rest of 
the flow around it, TCP will see one dupack. Then subsequent (non-CE) 
packets will continue to advance the cumulative ACK number closing the 
gap towards the one packet up ahead, so there will be no further 
dupacks. With SACK, it will report the gap on the SACK of the CE packet, 
then again, subsequent packets will close the gap, so no ill effects - I 
believe.

An odd packet delivered early might confuse the smart mechanisms (in 
e.g. Linux) for characterising the level of persistent re-ordering, 
which are intended to prevent spurious retransmits. But a good algo 
should be able to ignore outliers.

3 classic CEs in a row would cause a spurious retransmit. But the 
probability of that from a good AQM should be tiny.

Also, reordered CE packets would be very rare in the first place because:
i) it is quite unusual to experience more than one bottleneck queue on a 
path;
ii) even then, reordering would only occur if there was simultaneous 
mixing of classic and L4S traffic, which would be more unlikely in an 
access link, which is where most bottlenecks are located.

> Simply using the presence of ECN capability is also a bad idea, 
> because Apple's imminent deployment of classic ECN to millions of end 
> hosts will break that assumption.
>
Agreed. Altho the deployment seemed to have happened, I looked at 
packets emitted from my iPhone after it autoinstalled the public release 
of iOS 9.0.2, a few days ago, and SYNs did not attempt to negotiate ECN. 
I've tried to find out what's (not) happening from Apple, but...?

> A more robust deployment option would be to use a new transport 
> protocol ID.  Then it wouldn't be TCP any more, and you can throw away 
> some cruft while you're at it.  I'll leave the problems with that idea 
> to your fertile imaginations.
>
Nah. Lots of reasons why the protocol ID is not useful for this. Pasting 
from our draft to be...

* A new protocol ID would need to be paired with the old one for each 
transport (TCP, SCTP, UDP, etc.);
* In IPv6, there can be a sequence of Next Header fields, and it would 
not be obvious which one would be expected to identify a network service 
like L4S;
* A new protocol ID would rarely provide an end-to-end service, because 
It is well-known that new protocol IDs are often blocked by numerous 
types of middlebox;
* The approach is not a solution for AQMs below the IP layer;

The middlebox bullet is the killer (which is why DCCP, SCTP etc still 
don't work e2e).

> As a gentle reminder, ELR is designed to solve the same problem while 
> avoiding these deployment problems.  It does "burn" ECT(1), but for a 
> good cause and - crucially - doesn't require the network to be aware 
> of it before endpoints can safely turn it on.
>
ELR sounds similar to previously published stuff (VCP maybe?). But until 
you write it up, we can't tell for sure.

Bob

> - Jonathan Morton

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Fri, 16 Oct 2015 10:42:17 +0200

Bob Briscoe <[email-masked]> writes:

> completely rips apart CBR and variable rate video when running
> alongside long-running TCP flows, when they would otherwise work just
> fine in a FIFO.

Are these tests available somewhere? Don't think I've actually seen a
well-specified test scenario that demonstrates this, and would love to
poke into it some more...

-Toke