Subject: Re: [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S

From: <[email-masked]>
Date: Fri, 2 Aug 2019 08:29:32 +0000

Hi Jonathan,

could you provide a real world example of links which are consecutively narrower than sender access links?

I could figure out a small campus network which has a bottleneck at the Internet access and a second one connecting the terminal equipment. But in a small campus network, the individual terminal could very well have a higher LAN access bandwidth, than the campus - Internet connection (and then there's only one bottleneck again).

There may be a tradeoff between simplicity and general applicability. Awareness of that tradeoff is important. To me, simplicity is the design aim. 

Regards,

Ruediger 

-----Ursprüngliche Nachricht-----
Von: tsvwg <[email-masked]> Im Auftrag von Jonathan Morton
Gesendet: Dienstag, 9. Juli 2019 17:41
An: Bob Briscoe <[email-masked]>
Cc: tcpm IETF list <[email-masked]>; [email-masked]; tsvwg IETF list <[email-masked]>
Betreff: Re: [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S

> On 13 Jun, 2019, at 7:48 pm, Bob Briscoe <[email-masked]> wrote:
> 
>       1.  It is quite unusual to experience queuing at more than one
>           bottleneck on the same path (the available capacities have to
>           be identical).

Following up on David Black's comments, I'd just like to note that the above is not the true criterion for multiple sequential queuing.

Many existing TCP senders are unpaced (aside from ack-clocking), including FreeBSD, resulting in potentially large line-rate bursts at the origin - especially during slow-start.  Even in congestion avoidance, each ack will trigger a closely spaced packet pair (or sometimes a triplet).  It is then easy to imagine, or to build a testbed containing, an arbitrarily long sequence of consecutively narrower links; upon entering each, the burst of packets will briefly collect in a queue and then be paced out at the new rate.

TCP pacing does largely eliminate these bursts when implemented correctly.  However, Linux' pacing and IW is specifically (and apparently deliberately) set up to issue a 10-packet line-rate burst on startup.  This effect has shown up in SCE tests to the point where we had to patch this behaviour out of the sending kernel to prevent an instant exit from slow-start.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 2 Aug 2019 10:47:06 +0100

> On 2 Aug, 2019, at 9:29 am, <[email-masked]> <[email-masked]> wrote:
> 
> Hi Jonathan,
> 

A progressive narrowing of effective link capacity is very common in consumer Internet access.  Theoretically you can set up a chain of almost unlimited length of consecutively narrowing bottlenecks, such that a line-rate burst injected at the wide end will experience queuing at every intermediate node.  In practice you can expect typically three or more potentially narrowing points:

1: Core to Access network peering.  Most responsible ISPs regularly upgrade these links to minimise congestion, subject to cost effectiveness.  Some consumer ISPs however are less responsible, and permit regular congestion here, often on a daily and/or weekly cycle according to demand.  Even the responsible ones may experience short-term congestion here due to exceptional events.  Even if the originating server's NIC is slower than the peering link, queuing may occur here if the link is congested overall due to statistical multiplexing.

2: Access to Backhaul provisioning shaper.  Many ISPs have a provisioning shaper to handle "poverty tariffs" with deliberately limited capacity.  It may also be used to impose a sanity check on inbound traffic bursts, to limit backhaul network traffic to that actually deliverable to the customer (especially when the backhaul network is itself subcontracted on a gigabytes-carried basis, as is common in the UK).  In the ADSL context it's often called a BRAS.

3: Backhaul to head-end device.  Generally the backhaul network is sized to support many head-end devices, each of which serves some number of consumer last-mile links.  I'm being deliberately generic here; it could be a CMTS on a cable network, a DSLAM in a phone exchange, a cell tower, or a long-range wifi hub.  In many cases the head-end device shares several subscribers' lines over a common last-mile medium, so there is still some statistical multiplexing.  In the particular case of a cell tower, the subscriber usually gets less link capacity (due to propagation conditions) than his tariff limit.

4: CPE bufferbloat mitigation shaper.  This is *post-last-mile* ingress shaping, with AQM and FQ, to reduce the effects of the still-ubiquitous dumb FIFOs on the above bottlenecks, especially the head-end device.  The IQrouter is a ready-made CPE device which does this.

5: LAN, wifi, or powerline link.  Most people now have GigE, but 100base-TX is still in use in cheaper CPE and some low-end computers, and this will be a further bottleneck in cases where the last mile is faster.  For example, the Raspberry Pi has only just upgraded to GigE in its latest versions, and used 100base-TX before.  Wifi is also a likely bottleneck, especially if the mobile station is on the opposite side of the house from the base CPE, and is additionally a "bursty MAC" with heavy aggregation.  Some CPE now runs airtime-fairness and FQ-AQM on wifi to help manage that.

I think the above collection is not at all exotic.  Not all of these will apply in every case, or at all times in any given case, but it is certainly possible for all five to apply consecutively.

 - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Fri, 2 Aug 2019 04:10:11 -0700

> A progressive narrowing of effective link capacity is very common in consumer Internet access.  Theoretically you can set up a chain of almost unlimited length of consecutively narrowing bottleneck…

0: Container and vm users are frequently using htb + something to keep
their bandwidths under control.

0.5: Cloudy providers use "something" to also rate limit traffic.
Policers and shapers, I assume.

> _______________________________________________
> Ecn-sane mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/ecn-sane

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Dave Taht <[email-masked]>
Date: Fri, 2 Aug 2019 05:05:04 -0700

> Policers and shapers, I assume.

Stuff in the cloud thus far is looking quite jittery; sub-ms marking
thresholds do not look feasible. I have no idea what sorts of
software jitter and burstyness exist in NFV and ddpk based implementatons.

> > 3: Backhaul to head-end device.  Generally the backhaul network is sized to support many head-end devices, each of which serves some number of consumer last-mile links.  I'm being deliberately ge…

There are also bursts here from the vpn crypto engine.

> > 4: CPE bufferbloat mitigation shaper.  This is *post-last-mile* ingress shaping, with AQM and FQ, to reduce the effects of the still-ubiquitous dumb FIFOs on the above bottlenecks, especially the…

I would say "inbound shaper" to be clear. And its far, far wider than
just that commercial product, Nearly everyone using this stuff
shapes both in and outbound - be it untangle, netduma, asus "adaptive
qos", edgerouter's or eero's sqm implemention, all of openwrt, dd-wrt,
and deratives, bsd-based pfsense and opfsense, preseem does a bump in
the wire for WISPs, streamboost, I think the list of off the shelf
home router qos systems that shape inbound is well above 80-90%, and
users have been trained to turn it on in both directions
and off only when they run out of cpu. We see new product sales driven
by the cpu cost of having to shape inbound, too.

Policers have become quite useless in the past decade.

> >
> > 5: LAN, wifi, or powerline link.  Most people now have GigE, but 100base-TX is still in use in cheaper CPE and some low-end

I'd so love to see powerline gain fq and AQM. I see it used a lot in
busy apt buildings to drag stuff from room to room. It can be *awful*

>computers, and this will be a further bottleneck in cases where the last mile is faster.  For example, the Raspberry Pi has only just upgraded to GigE in its latest versions, and used 100base-TX before.  Wifi is also a likely bottleneck, especially if the mobile station is on the opposite side of the house from the base CPE, and is additionally a "bursty MAC" with heavy aggregation.  Some CPE now runs airtime-fairness and FQ-AQM on wifi to help manage that.

"some" includes most of qca's ath9k or ath10k products. (40% of the AP
market?) Prominently known fq-codel for wifi users are ~3m chromebook
users and ~3m google wifi, (
http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf
) and nearly everyone else in the 802.11s meshy market... meraki is a
known sfq + codel user... starting in 2014...

A large number of wifi APs and p2p links oft have a faster wifi speed
than their 100base-tx link, and thus use the ethernet (with either a
short fifo or fq_codel) to smooth the bursts out. I can point to a few
ubnt products that I know a bit too much about. There's also some
802.11ad and ay.... there is a quite remarkable amount of 100base-tx
gear still being deployed.

If it helps any to those here doing simulation, long ago I put a
"slot" and statistical distribution of busty mac delay feature into
netem. I can say now that that was used to help guide the development
of google "stadia", and certainly "slotting" is a MAJOR tool that I'd
plug into the SCE work to better emulate the real orld behaviors of
bursty macs. google has collected WAY more examples characterizing
real world micro(bursts) than I could ever deal with, and I hope they
publish that data someday for others to use.

> > I think the above collection is not at all exotic.  Not all of these will apply in every case, or at all times in any given case, but it is certainly possible for all five to apply consecutively.

For bursty.

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 2 Aug 2019 15:15:23 +0200

Hi Ruediger,

> On Aug 2, 2019, at 10:29, <[email-masked]> <[email-masked]> wrote:
> 
> Hi Jonathan,
> 

	Just an example from a network you might be comfortable with, in DTAGs internet access network there typically are traffic limiting elements at the BNGs (or at the BRAS for the legacy network), I am not 100% sure whether these are implemented as policers or shapers, but they tended to come with >= 300ms buffering. Since recently, the BNG/BRAS traffic shaper's use the message field of the PPPoE Auth ACK to transfer information about the TCP/IPv4 Goodput endusers can expect on their link as a consequence of the BNG/BRAS"s traffic limiter. In DOCSIS and GPON networks the traffic shaper seems mandated by the standards, in DSL networks it seems optional (but there even without a shaper the limited bandwidth of the access link would be a natural traffic choke point).
 Fritzbox home router's now use this information to automatically set egress (and I believe also) ingress traffic shaping on the CPE to reduce the bufferbloat users experience. I have no insight in what Telekom's own Speedport routers do, but I would not be surprised if they would do the same (at least for egress). 
	As Jonathan and Dave mentioned, quite a number of end-users, especially the latency sensitive ones, employ their own ingress and egress traffic shapers at their home routers as the 300ms buffers of the BNG's are just not acceptable for any real-timish uses (VoIP, on-line twitch gaming, even for interactive sessions like ssh 300ms delay are undesirable). E.g. personally, I use an OpenWrt router with an FQ AQM for both ingress and egress (based on Jonathan's excellent cake qdisc) that allows a family of 5 to happily share a 50/10 connection between video streaming and interactive use with very little interference between the users, the same link with out the FQ-AQM active makes interactive applications feel like submerged in molasses once the link gets saturated...
	As far as I can tell there is a number of different solutions that offer home-router based bi-directional traffic shaping to solve bufferbloat" from home (well, not fully solve it, but remedy its consequences), including commercial options like evenroute's iqrouter, and open-source options like OpenWrt (with sqm-scripts as shaper packet). 
	It is exactly this use case and the fact that latency-sensitive users often opt for this solution, that causes me to ask the L4S crowd to actually measure the effect of L4S on RFC3168-FQ-AQMs in the exact configuration it is actually used today, to remedy the same issue L4S wants to tackle.

Best Regards
	Sebastian

---

From: <[email-masked]>
Date: Mon, 5 Aug 2019 07:26:51 +0000

Hi Sebastian,

the access link is the bottleneck, that's what's to be expected. As far as I know, in the operator world shapers here by and large removed policers.

A consecutive chain of narrower links results, if the Home Gateway runs with an additional ingress or egress shaper operating below the access bandwidth, if I get you right. 

I understand that you aren't interested in having 300ms buffer delay and may be some jitter for a phone conversation using best effort transport. A main driver for changes in consumer IP access features in Germany are publications of journals and regulators comparing IP access performance of different providers. Should one provider have an advantage over the others by deploying a solution as you (and Bob's team) work on, it likely will be generally deployed.

As far as I can see, latency aware consumers still are a minority and gamers seem to be a big group belonging here. Interest in well performing gaming seems to be growing, I guess (for me at least it's an impression rather than a clear trend).

I'd personally prefer an easy to deploy and operate standard solution offering Best Effort based transport being TCP friendly and at the same time congestion free for other flows at a BNG for traffic in access direction (and for similar devices in other architectures of course). 

Fighting bufferbloat in the upstream direction the way you describe it doesn't construct a chain of links which are consecutively narrower than the bottleneck link, I think.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Freitag, 2. August 2019 15:15
An: Geib, Rüdiger <[email-masked]>
Cc: Jonathan Morton <[email-masked]>; [email-masked]; ECN-Sane <[email-masked]>; [email-masked]
Betreff: Re: [Ecn-sane] [tsvwg] ECN CE that was ECT(0) incorrectly classified as L4S

Hi Ruediger,

Best Regards
	Sebastian

---

From: <[email-masked]>
Date: Mon, 5 Aug 2019 09:35:48 +0000

Jonathan Morton marked [JM] below, Ruediger Geib [RG].

> There may be a tradeoff between simplicity and general applicability. Awareness of that tradeoff is important. To me, simplicity is the design aim. 

[JM] A progressive narrowing of effective link capacity is very common in consumer Internet access.  Theoretically you can set up a chain of almost unlimited length of consecutively narrowing bottlenecks, such that a line-rate burst injected at the wide end will experience queuing at every intermediate node.  In practice you can expect typically three or more potentially narrowing points:

[RG] deleted. Please read https://tools.ietf.org/html/rfc5127#page-3 , first two sentences. That's a sound starting point, and I don't think much has changed since 2005. 

[RG] About the bursts to expect, it's probably worth noting that today's most popular application generating traffic bursts is watching video clips streamed over the Internet. Viewers dislike the movies to stall. My impression is, all major CDNs are aware of that and try their best to avoid this situation. In particular, I don't expect streaming bursts to overwhelm access link shaper buffers by design. And that, I think, limits burst sizes of the majority of traffic.

[RG] Other people use their equipment to communicate and play games (that's what I see when I use commuters). Unless gaming pictures are rendered on a server or live pictures of communicating persons are streamed, there should be no bursts. Still I miss the consecutively narrowing bottlenecks with queues being built at each instance with a likelihood justifying major engineering and deployment efforts. Any solution for Best Effort service which is TCP friendly and support scommunication expecting no congestion at the same time should be easy to deploy and come with obvious benefits. 

[RG] I found Sebastian's response sound. I think, there are people interested in avoiding congestion at their access.

[RG] I'd like to repeat again what's important to me: no corner case engineering. Is there something to be added to Sebastian's scenario?

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 5 Aug 2019 13:59:47 +0300

> [RG] deleted. Please read https://tools.ietf.org/html/rfc5127#page-3 , first two sentences. That's a sound starting point, and I don't think much has changed since 2005. 

As I said, that reference is *usually* true for *responsible* ISPs.  Not all ISPs, however, are responsible vis a vis their subscribers, as opposed to their shareholders.  There have been some high-profile incidents of *deliberately* inadequate peering arrangements in the USA (often involving Netflix vs major cable networks, for example), and consumer ISPs in the UK *typically* have diurnal cycles of general congestion due to under-investment in the high-speed segments of their network.

To say nothing of what goes on in Asia Minor and Africa, where demand routinely far outstrips supply.  In those areas, solutions to make the best use of limited capacity would doubtless be welcomed.

> [RG] About the bursts to expect, it's probably worth noting that today's most popular application generating traffic bursts is watching video clips streamed over the Internet. Viewers dislike the m…

In my personal experience with YouTube, to pick a major video streaming service not-at-all at random, the bursts last several seconds and are essentially ack-clocked.  It's just a high/low watermark system in the receiving client's buffer; when it's full, it tells the server to stop sending, and after it drains a bit it tells the server to start again.  When traffic is flowing, it's no different from any other bulk flow (aside from the use of BBR instead of CUBIC or Reno) and can be managed in the same way.

The timescale I'm talking about, on the other hand, is sub-RTT.  Packet intervals may be counted in microseconds at origin, then gradually spaced out into the millisecond range as they traverse the successive bottlenecks en route.  As I mentioned, there are several circumstances when today's servers emit line-rate bursts of traffic; these can also result from aggregation in certain link types (particularly wifi), and hardware offload engines which try to treat multiple physical packets from the same flow as one.  This then results in transient queuing delays as the next bottleneck spaces them out again.

When several such bursts coincide at a single bottleneck, moreover, the queuing required to accommodate them may be as much as their sum.  This "incast effect" is particularly relevant in datacentres, which routinely produce synchronised bursts of traffic as responses to distributed queries, but can also occur in ordinary web traffic when multiple servers are involved in a single page load.  IW10 does not mean you only need 10 packets of buffer space, and many CDNs are in fact using even larger IWs as well.

These effects really do exist; we have measured them in the real world, reproduced them in lab conditions, and designed qdiscs to accommodate them as cleanly as possible.  The question is to what extent they are relevant to the design of a particular technology or deployment; some will be much more sensitive than others.  The only way to be sure of the answer is to be aware, and do the appropriate maths.

> [RG] Other people use their equipment to communicate and play games

These are examples of traffic that would be sensitive to the delay from transient queuing caused by other traffic.  The most robust answer here is to implement FQ at each such queue.  Other solutions may also exist.

> Any solution for Best Effort service which is TCP friendly and support scommunication expecting no congestion at the same time should be easy to deploy and come with obvious benefits. 

Well, obviously.  Although not everyone remembers this at design time.

> the access link is the bottleneck, that's what's to be expected.

It is typically *a* bottleneck, but there can be more than one from the viewpoint of a line-rate burst.

> [RG] I'd like to repeat again what's important to me: no corner case engineering. Is there something to be added to Sebastian's scenario?

He makes an essentially similar point to mine, from a different perspective.  Hopefully the additional context provided above is enlightening.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 5 Aug 2019 13:00:26 +0200

Hi Ruediger,

> On Aug 5, 2019, at 09:26, <[email-masked]> <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	Mostly, then again there are situations with 1Gbps Plans where it is not the actual access link, but rather the CPEs Gigabit ethernet LAN ports that are the true bottleneck, but that does not substantially change the issue, it is still the upstream shaper/policer that needs to be worked around.

> As far as I know, in the operator world shapers here by and large removed policers.

	Good to know, shapers are somewhat nicer to user traffic than hard policers, at least that is my interpretation.

> A consecutive chain of narrower links results, if the Home Gateway runs with an additional ingress or egress shaper operating below the access bandwidth, if I get you right. 

	Yes, as you state below, this only is true for the ingress direction, egress shaping works reliably and is typically not suffering from this. That said, if the egress link bandwidth is larger than a servers connection this issue can appear also for the egress direction. For example overly hot peering/transit links can cause downstream bottlenecks considerably narrower than the internet access link's upload direction, but that, while unfortunate, is not at the core of my issue.

> 
> I understand that you aren't interested in having 300ms buffer delay and may be some jitter for a phone conversation using best effort transport.

	+1

> A main driver for changes in consumer IP access features in Germany are publications of journals and regulators comparing IP access performance of different providers.

	Good to know,

> Should one provider have an advantage over the others by deploying a solution as you (and Bob's team) work on, it likely will be generally deployed.

	I do not believe that these mechanisms are actually in play in the German market, as an example for roughly a decade the DOCSIS-ISPs offer higher bandwidth for same or less money than the incumbent telco and yet only managed to get 30% of the customers of their ~75% of possible customers, so only 75*0.3 = 22.5 % market share, with the incumbent only reaching 250/40 for the masses while the DOCSIS ISPs offer 1000/50. And unlike latency, bandwidth (or rather rate) is the number that consumers understand intuitively.
	If anything will expedite the roll-out of L4S style AQMs it is the capability to use those to implement the "special services" that the EU net neutrality regulation explicitly allows, as that is a product that can be actually sold to customers, but I might be too pessimistic here.

> As far as I can see, latency aware consumers still are a minority and gamers seem to be a big group belonging here. Interest in well performing gaming seems to be growing, I guess (for me at least…

	Put that way, I see a way for ISPs to distinguish themselves from the rest by being gaming friendly, but unless this results in gamers paying more I fail to see the business case that management probably needs before green-lighting the funds required to implement this. This is where cablelabs approach to mandate this in the specs is brilliant. 

> Fighting bufferbloat in the upstream direction the way you describe it doesn't construct a chain of links which are consecutively narrower than the bottleneck link, I think.

	Yes, fully agreed, that said, and ISPs CPE should implement an AQM to really solve the latency issues for end-users. The initial L4S paper side-stepped that requirement by making sure the uplinks were not saturated during the test, and state that that needs a real solution for proper roll-out. In theory the ISP could do the uplink shaping on its end (and to constrain users to their contracted rates, ISPs do this already) but as in the downstream case, running an AQM in front of a bottleneck as opposed to behind it makes everything much easier. Also with uplinks typically << downlinks, the typically weak CPE CPUs will still be able to AQM the uplink, nicely distributing that computation load away from the BNG/BRAS big iron ....

Best Regards
	Sebastian

---

From: <[email-masked]>
Date: Mon, 5 Aug 2019 11:47:51 +0000

Hi Sebastian,

thanks. Three more remarks:

I'm happy for any AQM design which comes at low implementation cost and allows me to add value to network operation (be it saving cost, be it enabling value added services). And I think the representatives of other operators are so too.

For most consumers, streaming is the most bandwidth hungry application. I think someone now working for Google published research, that at the time when Internet access bandwidth no longer had an impact on the streaming quality, consumers started to lose interest in "access speed" as an important measure of quality of their Internet access. I think it takes a while until John Doe requires a n*100 Mbit/s Internet access, because any access below 100 Mbit/s causes congestion for the services consumed by John. 

I see many people around me conveniently use their smartphone to access the Internet. A handheld display likely requires less bandwidth for an acceptable display quality, than a large screen. That doesn't say, the latter disappear. But maybe only one or two of them need to be served per consumer access. That will work with 100 Mbit/s or less for a while (other bandwidth hungry applications will arrive some day; I prefer the copper access lines in the ground to be replaced by fiber ones).
 
Regards, Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Montag, 5. August 2019 13:00
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; ECN-Sane <[email-masked]>; [email-masked]
Betreff: Re: [Ecn-sane] [tsvwg] ECN CE that was ECT(0) incorrectly classified as L4S

Hi Ruediger,

	+1

	Good to know,

Best Regards
	Sebastian

---

From: <[email-masked]>
Date: Mon, 5 Aug 2019 12:16:37 +0000

As I said, no corner-case engineering. On my vacation site abroad, Netflix worked as well as it does at home. Germany is often criticised as a laggard in developing a competitive broadband access market. My interest in IETF work being set up to work around commercial problems between ISPs and ISPs or CDNs in a rich country like the UK or US is low. 

If your technical standardization activities help to make using the Internet more convenient in African countries without raising extra cost or requiring extra skills, I'm sure that's a fair market. I know that these countries are struggling to improve the services operated in their networks. Conditions there (and some Asian countries) differ strongly from those in many other parts of the world.
It might be a good thing to clarify under which scenarios the problem solutions you work on create the most significant benefit.

In the market that I am aware of, there's a single regular bottleneck, which is the consumer access or terminal.

-----Ursprüngliche Nachricht-----
Von: Jonathan Morton <[email-masked]> 
Gesendet: Montag, 5. August 2019 13:00
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Betreff: Re: [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 5 Aug 2019 15:47:34 +0200

Hello Ruediger,

> On Aug 5, 2019, at 13:47, <[email-masked]> <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	Yes, that is my premise too. The only cost saving opportunity I can see in both proposals would be if they allowed to run a fully saturated network without the adverse effects on latency and loss. Value added services, sure for the few users sensitive to latency under load. Maybe with the PR the 5G roll-out is getting in regards to low-latency it might be possible to convince more consumers that this is actually valuable?

> 
> For most consumers, streaming is the most bandwidth hungry application.

	I have no statistical numbers on this topic, but on the list of things that cause issues for latency sensitive home networks the following items come up repeatedly:
A) Streaming in (Youtube, Netflix, Amazon, ...) 
B) Streamin out (aka twitch and friends)
C) File sharing with bittorrent (a slight challenge for FQ-AQMs die to lots of parallel flows and a misdesigned back-off mechanism (react to 100ms induced latency under load?))
D) OS updates, (especially windows update when lveraging P2P technolgy from a close by CDN was/is notorious for being a tad too aggressive)

How these stack up proportionally to each other, I have no clue. Typically the reports are that perceived interactivity in FPS games goes down the drain if and combination of A-D are concurrently active.

> I think someone now working for Google published research, that at the time when Internet access bandwidth no longer had an impact on the streaming quality, consumers started to lose interest in "access speed" as an important measure of quality of their Internet access. I think it takes a while until John Doe requires a n*100 Mbit/s Internet access, because any access below 100 Mbit/s causes congestion for the services consumed by John. 

	That is a good question, as far as I see it, ISPs in Germany still seem to leverage access-rates as their main attraction (giga this and giga that), even though as you note higher rates have diminishing returns for most use-cases.

> 
> I see many people around me conveniently use their smartphone to access the Internet.

	So do I, but I realize how laggy this feels even for "simple" browsing duty (but I accept that for the ease of use and immediacy)... And it is not guaranteed that the smartphone uses the mobile network, it might just as well use wifi. But the latency/bottleneck issue also exists with smartphones, where the variable bandwidth nature of the radios and the opaqueness of 2-4G modems makes things even less enjoyable than on fixed networks (I see multi second stalls when browsing on a phone, versus 300ms on the fixed line without my AQM.

> A handheld display likely requires less bandwidth for an acceptable display quality, than a large screen.

	For browsing, I am not sure that is a real point, given that smartphone display resolution crossed from high to ridiculous some years ago (at least without glasses).

> That doesn't say, the latter disappear. But maybe only one or two of them need to be served per consumer access. That will work with 100 Mbit/s or less for a while

	I agree, I run a family of 5 on a 50/10 link including concurrent streaming (in SD) and thanks to employing a competent FQ-AQM on my router (ingress & egress) this works quite well even with interactive sessions. But without that AQM system the link feels noticeably worse... (and this is the reason why I want to see data, that L4S senders will not invalidate the effectiveness of my setup)

Best Regards
	Sebastian

> (other bandwidth hungry applications will arrive some day; I prefer the copper access lines in the ground to be replaced by fiber ones)
> 
> Regards, Ruediger
>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 5 Aug 2019 18:55:36 +0300

> On 5 Aug, 2019, at 3:16 pm, <[email-masked]> <[email-masked]> wrote:
> 
> As I said, no corner-case engineering.

> In the market that I am aware of, there's a single regular bottleneck, which is the consumer access or terminal.

Let me explain one more time: this is *not* a corner case.  This is normal operation on today's Internet; faster core networks feed into larger numbers of slower edge networks in several stages.  If you haven't noticed it yourself, that's fortunate for you - but that "single regular bottleneck" is an illusory simplification, which only applies at relatively long timescales.

However, high-fidelity ECN markers *will* notice the difference, and the resulting congestion signals will appear at the receiver and be fed back to the sender.  That represents an engineering problem for which either a solution must be devised, or the fact that it is not a problem must be established.  At present, we're still working through the maths to determine the boundaries of acceptable operation.

It's even possible that conventional RFC-3168 AQMs notice it as well, depending on their design.  Codel, for example, is specifically designed to ignore transient queuing (if it persists for less than one 'interval', which is taken as an estimate of the RTT) and only act when a persistent standing queue exists.

And it's actually a very important problem when a dumb FIFO bottleneck is immediately followed by a slightly narrower AQM bottleneck.  In this case the dumb FIFO dilutes the benefit of the AQM significantly, because the AQM can't see that most of the queue exists in the upstream FIFO, and even if it could, it cannot apply any intelligence to it.  This is the scenario Sebastian is most concerned about, and for which I have tried to compensate in Cake with "ingress mode" shaping - because Cake is specifically designed to be deployed into that topology.

Most of the problem would go away if that dumb FIFO were merely replaced by a simple AQM.  This is a straightforward & deployable engineering solution that has been known for many years, and yet almost nobody has actually deployed it.  I would urge you to do what you can on that front; judging by your e-mail address, you probably have more influence where it counts than I do.

I repeat: this is normal operation on today's Internet.

> If your technical standardization activities help to make using the Internet more convenient in African countries without raising extra cost or requiring extra skills, I'm sure that's a fair market. I know that these countries are struggling to improve the services operated in their networks.

May I refer you to some useful work already done and widely deployed?  This is now the default in most Linux and Apple devices, and is also available in FreeBSD.  It is used by some of the better CPE devices as part of "Advanced QoS" and "Airtime Fairness", eg. in the Netgear R7000.

	https://tools.ietf.org/html/rfc8289
	https://tools.ietf.org/html/rfc8290

The problem is that it's not deployed at most of the actual bottlenecks, which mainly exist in ISPs' equipment if the core networks are indeed overprovisioned.  If it was deployed, congestion on the Internet would be a much easier problem than it is today.  And this should be especially applicable to less-developed areas of the Internet.

All it takes is enough ISPs manning up, talking to their hardware vendors, and saying: "We expect to have congestion occasionally/frequently (delete as applicable).  What AQM does your hardware support, and how do we turn it on?"  And if the answer is negative, being prepared to take business to a competitor who does.  The technology exists; money talks.

In the less developed parts of the world, one could easily jury rig an AQM router together using a Raspberry Pi and a couple of USB Ethernet dongles.  With the latest Raspberry Pi 4, that would work for up to a gigabit link; with older models, you can still handle 100Mbps.  These things are cheap enough to practically be disposable, and consume very little power.  If there's demand for that sort of thing, I and my colleagues could probably arrange for a ready-made SD card image to be built and published.

Or you could standardise on a consumer CPE router and build a no-knobs mesh-networking image for that.  There's a bunch of groups who regularly attend Battlemesh, who could help with that.  Be prepared to change models every few years as the old ones go out of production.

 - Jonathan Morton

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 6 Aug 2019 11:49:54 +0200 (CEST)

On Mon, 5 Aug 2019, Sebastian Moeller wrote:

> 	That is a good question, as far as I see it, ISPs in Germany still 
> seem to leverage access-rates as their main attraction (giga this and 
> giga that), even though as you note higher rates have diminishing 
> returns for most use-cases.

It's great marketing. Same way as car manufacturers make prototypes and 
high end cars ("halo cars") to sell their high-volume mainstream cars. 
It's brand building. So don't judge the customer interest for actual 
product sales, by the marketing you see. They might not correlate 
directly.

There is also one more thing that people nowadays do that wasn't directly 
on your list. Software downloads. Either on game console or on a PC, these 
downloads can easily be in tens of gigabytes. I personally have a 250/100 
connection at home, and download of a large modern game can take 30-60 
minutes, because it's 50 GB. I do not know what congestion avoidance 
algorithms are used, but it seems to me that at least some of these 
software download services do not actually create congestion. They do very 
slow ramp-ups and from what I can see they typically keep the utilisation 
of my Internet connection below congestion (as in perhaps averaging at 80% 
of capacity).

Personally I frequently see several potential congestion points between 
user device and what it's communicating with.

There is the ISP-CDN or ISP-ISP interconnect point.
There might be congestion on the core-core links.
There is the uplink to the BNG or whatever.
There is the user-unique shaper
There is the L2 aggregation network (DOCSIS/*PON/ETTH)
There is the in-house wifi network.

So even if the ISP does a great job, we might have the user-unique shaper 
and the wifi both congesting and the wifi might be slower than the 
user-access link. This is the case in my home sometimes. Even with a great 
wifi setup (multiple 5GHz APs) I frequently get congestion there resulting 
in lower speeds than my 250 megabit/s Internet access speed. So this means 
traffic might encounter half of the time my 250 megabit/s ISP shaper as 
the bottleneck, then sporadically it encounters my wifi lowering the speed 
even more, and then returning to my ISP shaper being the slowest point.

So I think your suggestion of what we should test is useful.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: <[email-masked]>
Date: Tue, 6 Aug 2019 14:34:57 +0000

I'd like to sort Mikael's list a little....

-----Ursprüngliche Nachricht-----
Von: Mikael Abrahamsson <[email-masked]> 

Of course congestion occurs. But the probability of it matters (the less likely it is, the less likely are efforts to work around it) 

Congestion is not probable in a well dimensioned backbone and at paid peerings. I think the effort put into congestion avoidance by engineering is high at these locations and whether protocol design to deal with congestion there (and make a gain as compared to todays transport performance under congestion there) is worth a larger effort. Bulk transport optimisation makes sense there, that includes suitable AQM.

Public peerings and not well dimensioned networks may suffer from regular congestion. I'm not sure to which extent technical standards can significantly improve service quality in that situation. IP transport must work as good as possible also in such a situation, of course. 

The following are access issues:

   There is the uplink to the BNG or whatever.
   There is the user-unique shaper
   There is the L2 aggregation network (DOCSIS/*PON/ETTH) There is the in-house wifi network.

Maybe one can add LTE and 5G, these are layer 2 standards missing above. Shared L2 may result in annoying performance. To me, L3 protocol design improving IP performance over one or more L2 protocols standardised by other SDOs sounds good. I wonder to which extent that's feasible.   

As you know, the user-unique (BNG or the like) shaper is my favourite site where I'd appreciate improvements. 

Home Gateways are a mass market product. I'm not familiar with ways to impact the vendors of that segment (but agree that it's worth trying). Also wireless scheduling of IP traffic certainly is an interesting topic. I'm not sure whether IETF has sufficient impact to push for improved packet transport performance there (again, it's worth trying).

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 6 Aug 2019 18:27:57 +0300

> On 6 Aug, 2019, at 5:34 pm, <[email-masked]> <[email-masked]> wrote:
> 

Obviously the application of AQM will not magically improve total throughput.  What it can do, however, is reduce latency and packet loss, and thereby improve perceived reliability of the service.  It may even improve goodput for the same throughput - an increase in efficiency.

This is especially true under emergency overload conditions, which is when people most desperately want a functioning network, but it is most likely to collapse under the strain.  Often a disaster will incidentally knock out some proportion of network infrastructure in the area, turning a previously well-proportioned network into an under-proportioned one, simultaneously with a sharp increase in demand as people try to find out what is going on or communicate with friends and relatives, and emergency response teams also try to coordinate their essential work.

So IMO networks should be designed to work well when congested, even when they are also designed to never *be* congested.  Technical specifications already exist and are well tested for this purpose.  Having them built in and turned on from the factory would be a great step forward.

 - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Tue, 6 Aug 2019 08:35:56 -0700

> This is especially true under emergency overload conditions, which is when people most desperately want a functioning network, but it is most likely to collapse under the strain.  Often a disaster…

Overbuffering, the silent killer:
http://blog.cerowrt.org/post/bufferbloat_on_the_backbone/

> So IMO networks should be designed to work well when congested, even when they are also designed to never *be* congested.  Technical specifications already exist and are well tested for this purpos…

I live in california, where I kind of expect the network to collapse
in the next Big One.

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740