Subject: [aqm] the side effects of 330ms lag in the real world

From: Dave Taht <[email-masked]>
Date: Mon, 28 Apr 2014 18:24:20 -0700

pretty wonderful experiment and video http://livingwithlag.com/

-- 
Dave Täht

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 29 Apr 2014 09:08:49 +0200 (CEST)

> pretty wonderful experiment and video http://livingwithlag.com/

Just so that everybody realises that this is an advertisement.

Also, what access method has 300 ms access latency, let alone 3 seconds? 
None that I know of, the meaningful comparison would be ADSL2+ at around 
25ms and 3G at around 50-100ms.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Tue, 29 Apr 2014 07:21:31 +0000

On Apr 29, 2014, at 3:08 AM, Mikael Abrahamsson <[email-masked]> wrote:

> Also, what access method has 300 ms access latency, let alone 3 seconds? None that I know of, the meaningful comparison would be ADSL2+ at around 25ms and 3G at around 50-100ms.

Well, we could discuss international communications. I happen to be at Infocom in Toronto, VPN’d into Cisco San Jose, and did a ping to you:

ping -c 10 swm.pp.se
PING swm.pp.se (212.247.200.143): 56 data bytes
   ...
--- swm.pp.se ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 249.368/342.038/456.376/71.902 ms

3 seconds is unusually high, although naval satcom is frequently in the 1.5-2.5 ms range. But I have a sample that I measured in a Dublin hotel that observed standing latencies of around 280 ms to San Jose, frequent latencies of seven seconds, and a peak of 9286 ms. Yes, the hotel DSL was horrendously misconfigured. It makes a great graphic for presentations.

> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

----------------------------------------------------
The ignorance of how to use new knowledge stockpiles exponentially. 
   - Marshall McLuhan

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 29 Apr 2014 09:56:10 +0200 (CEST)

On Tue, 29 Apr 2014, Fred Baker (fred) wrote:

> Well, we could discuss international communications. I happen to be at 
> Infocom in Toronto, VPN’d into Cisco San Jose, and did a ping to you:

Yes, but as soon as you hit the long distance network the latency is the 
same regardless of access method. So while I agree that understanding the 
effect of latency is important, it's no longer a meaningful way of selling 
fiber access. If your last-mile is fiber instead of ADSL2+ won't improve 
your long distance latency.

---

From: Tristan Seligmann <[email-masked]>
Date: Tue, 29 Apr 2014 09:43:39 +0200

> 25ms and 3G at around 50-100ms.

Latencies on my ADSL2+ connection in Johannesburg, South Africa to my
ISP, a host in the UK, and a host in the USA:

--- www.mweb.co.za ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9011ms
rtt min/avg/max/mdev = 33.720/38.594/66.841/9.507 ms

--- anor.aduial.net ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9010ms
rtt min/avg/max/mdev = 197.966/201.981/211.580/4.619 ms

--- speedtest.atlanta.linode.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9001ms
rtt min/avg/max/mdev = 285.859/292.451/331.214/13.285 ms

And of course, I have bufferbloat mitigation in play.
-- 
mithrandi, i Ainil en-Balandor, a faer Ambar

---

From: Dave Taht <[email-masked]>
Date: Tue, 29 Apr 2014 08:46:46 -0700

> On Tue, 29 Apr 2014, Fred Baker (fred) wrote:

A couple points here.

1) The video went viral, and garnered over 600,000 new hits in the 12
hours since I posted
 it here.

there is pent up demand for less latency. While the ad conflates
bandwidth with latency,
they could have published their RTTs on their local fiber network,
which is probably a
great deal less than dsl or cable. That counts for a lot when
accessing local services.

2) There is a lot of things an ISP can do to improve apparent latency
on the long haul

A)  co-locating with a major dns server like f-root to reduce dns latency
B)  co-locating with major services like google and netflix

publishing ping times to google for example might be a good tactic.

C) Better peering

> your long distance latency.

Well, it chops a great deal from the baseline physical latency, and most
people tend to access resources closer to them than farther away. An
american in paris might want to access the NYT, but Parisians La Monde.

Similarly most major websites are replicated and use CDNs to distribute
their data closer to the user. The physical RTT matters more and more
in the last mile the more resources are co-located in the local data center.

-- 
Dave Täht

---

From: Jim Gettys <[email-masked]>
Date: Tue, 29 Apr 2014 12:44:38 -0400

> your long distance latency.

​FIOS bufferbloat is a problem too.

Measured bufferbloat, symmetric 25/25 service in New Jersey at my inlaw's
house is 200ms (on the ethernet port of the Actiontec router provided by
Verizon).  So latency under load is the usual problem.

Why would you think the GPON guys are any better in principle than cable or
DSL?  Cable and DSL may be somewhat worse, just because it is older and
downward compatibility means that new modems on low bandwidth tiers are
even more grossly over buffered.
You can look at the netalyzr scatter plots in
http://gettys.wordpress.com/2010/12/06/whose-house-is-of-glasse-must-not-throw-stones-at-another/

Now, if someone gives me real fiber to the home, with a real switch fabric
upstream, rather than gpon life might be somewhat better (if the switches
aren't themselves overbuffered....  But so far, it isn't.
                                      - Jim

                                  - Jim

                                                         - Jim

​

> Bloat mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/bloat
>
>

---

From: Jim Gettys <[email-masked]>
Date: Tue, 29 Apr 2014 12:57:31 -0400

On Tue, Apr 29, 2014 at 12:44 PM, Jim Gettys <[email-masked]> wrote:

> Why would you think the GPON guys are any better in principle than cable
> or DSL?  Cable and DSL may be somewhat worse, just because it is older and
​Oh, I forgot to note: Netalyzr topped out at about 20M​bps: most of why
GPON looks good on those plots is just that the common lowest tier of
service (at the time that data was taken) it can't easily detect the bloat.
But if you perform other tests that don't have a bandwidth limit, the bloat
is there.

This reminds me I should ask Nick Weaver if he has more recent data...
                                             - Jim

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Tue, 29 Apr 2014 19:01:36 +0200

Jim Gettys <[email-masked]> writes:

> upstream, rather than gpon life might be somewhat better (if the switches aren't
> themselves overbuffered.... But so far, it isn't.

As a data point for this, I have fibre to my apartment building and
ethernet into the apartment. I get .5 ms to my upstream gateway and
about 6 ms to Google. Still measured up to ~20 ms of bufferbloat while
running at 100 Mbps...

http://files.toke.dk/bufferbloat/data/karlstad/cdf_comparison.png

However, as that graph shows, it is quite possible to completely avoid
bufferbloat by deploying the right shaping. And in that case fibre
*does* have a significant latency advantage. The best latency I've seen
to the upstream gateway on DSL has been ~12 ms.

-Toke

---

From: Dave Taht <[email-masked]>
Date: Tue, 29 Apr 2014 10:02:08 -0700

On Tue, Apr 29, 2014 at 9:44 AM, Jim Gettys <[email-masked]> wrote:
> FIOS bufferbloat is a problem too.
>

ESR's link, before and after the cerowrt SQM treatment:

https://www.bufferbloat.net/projects/codel/wiki/RRUL_Rogues_Gallery#Verizon-FIOS-Testing-at-25Mbit-up-and-25Mbit-down

> downward compatibility means that new modems on low bandwidth tiers are even
> more grossly over buffered.

Well, buffering on the DSLAM or CMTS needs to be more actively
managed. Fixed limits are much like conventional policing, always
either too large or too small to handle sustained or bursty traffic
respectively.

I have been fiddling with Tim Shepard's "udpburst" tool as a quick
means of measuring head end buffering, even with fq_codel present on
the inbound. (It's not suitable for open internet use as yet, but code
in progress can be had or enhanced at
https://github.com/dtaht/isochronous ). I just added ecn and tos
setting support to it.

server: ./udpburst -S -E -D 32 # Server mode, enable ECN marking, set
dscp to 0x20 (CS1)

client:

This is from a 22Mbit down CMTS

d@nuc:~/git/isochronous$ ./udpburst -f 149.20.63.30 -E -C -d -n 400 -s 1400
1400 bytes -- received 382 of 400 -- 365 consecutive 0 ooo 0 dups 2 ect

........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
.... .. .   ... . ...  . ..    ..  .  .
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2   2 2   2       2 2 2   2   2 2 2     2   2 2         2 2     2

or roughly 512k of buffering.

A DSL link (6400 down)

d@puck:~/git/isochronous$ ./udpburst -f snapon.lab.bufferbloat.net -n
100 -C -d -s 1000
1000 bytes -- received 71 of 100 -- 71 consecutive 0 ooo 0 dups 0 ect

......................................................................

 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

or roughly 64k worth of buffering. Interestingly the bandwidth disparity between
the server (gigE in isc.org's co-lo), is so great that fq_codel can't
kick in before
the 64k dslam buffer is overrun.

-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

---

From: Jim Gettys <[email-masked]>
Date: Tue, 29 Apr 2014 13:07:21 -0400

On Tue, Apr 29, 2014 at 1:01 PM, Toke Høiland-Jørgensen <[email-masked]>wrote:

> bufferbloat by deploying the right shaping​

And in that case fibre
> to the upstream gateway on DSL has been ~12 ms.

​Media access is a killer on Cable too, putting the latency floor at around
8ms on my Docsis 3.0 Comcast service, though you can sometimes get lucky
and piggyback. to somewhat lower latency, IIRC conversations with Greg
White about how cable works.
                                       - Jim

---

From: Greg White <[email-masked]>
Date: Tue, 29 Apr 2014 18:09:15 +0000

FYI piggybacked requests in DOCSIS eliminate the possibility of request collisions (and resulting backoff/retry).  In DOCSIS 3.0 they will result in lower latency only as a result of eliminating these events.  In a lightly loaded DOCSIS 3.0 network (few neighbors contending for bandwidth) the collision probability will be low anyway, so it won't make much difference.  In a highly utilized network (especially in the case where a lot of neighbors are sending at rates below the piggybacking threshold) it can make a bigger difference.

Packet rates above ~200pps will be sufficient to ensure piggybacking.

I usually consider ~6ms to be the nominal upstream MAC latency, and ~0.7ms to be the equivalent on downstream, making the RTT on the DOCSIS 3.0 link close to 7ms (depending on configuration).

-Greg

From: Jim Gettys <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 29, 2014 at 11:07 AM
To: Toke Høiland-Jørgensen <[email-masked]<mailto:[email-masked]>>
Cc: bloat <[email-masked]<mailto:[email-masked]>>, "Fred Baker (fred)" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Mikael Abrahamsson <[email-masked]<mailto:[email-masked]>>
Subject: Re: [aqm] [Bloat] the side effects of 330ms lag in the real world

On Tue, Apr 29, 2014 at 1:01 PM, Toke Høiland-Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

​Media access is a killer on Cable too, putting the latency floor at around 8ms on my Docsis 3.0 Comcast service, though you can sometimes get lucky and piggyback. to somewhat lower latency, IIRC conversations with Greg White about how cable works.
                                       - Jim

-Toke

---

From: Dave Taht <[email-masked]>
Date: Tue, 29 Apr 2014 20:21:59 -0700

> http://files.toke.dk/bufferbloat/data/karlstad/cdf_comparison.png

I need to note that what this wonderfully flat CDF for the measurement
stream shows is that short flows under fq_codel leap to the head of
the queue ever better as you get more and more bandwidth available.

The background load flows not shown on this graph are experiencing
5-20ms worth of latency in each direction as per codel's algorithm.

A better test (in progress) would measure typical voip behaviors....

> bufferbloat by deploying the right shaping.

It does not "completely avoid bufferbloat", the fq_codel "fast queue"
merely eliminates queuing delay for sparse flows, things like arp, syn,
syn/ack, dns, ntp, etc, as  well as the first packet of any flow that
has not built up a queue yet.

(which is, admittedly, quite a lot of bufferbloat reduction)

The rest of the magic comes from codel.

> to the upstream gateway on DSL has been ~12 ms.

And reduced RTT = money.

this piece states observed average RTTs at peak times were 17ms for fiber,
28ms for cable, and 44ms for DSL.

http://www.igvita.com/2012/07/19/latency-the-new-web-performance-bottleneck/

I don't know if the underlying report measures baseline unloaded last mile RTT.

-- 
Dave Täht

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 30 Apr 2014 08:16:48 +0200 (CEST)

> Why would you think the GPON guys are any better in principle than cable or

Fiber != GPON. Fiber in Sweden is 99% active ethernet.

And the advertisement isn't about bufferbloat, I doubt they even know what 
that is... Upside is that the equipment used in Sweden is usually cheap 
ethernet switches which don't have buffering and might also use policing 
instead of shaping which makes bufferbloat problem basically go away, but 
on the other hand creates other problems.