Subject: [aqm] ping loss "considered harmful"

From: Dave Taht <[email-masked]>
Date: Sun, 1 Mar 2015 19:57:37 -0800

On this thread over here, an otherwise pretty clueful user chose
openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
*higher ping loss*.

http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125

(I note that both fq_codel enabled QoS systems outperformed
streamboost by a lot, which I am happy about)

wow. It never registered to me that users might make a value judgement
based on the amount of ping loss, and in looking back in time, I can
think of multiple people that have said things based on their
perception that losing pings was bad, and that sqm-scripts was "worse
than something else because of it."

sqm-scripts explicitly *deprioritizes* ping. In particular, this
reduces the impact of ping floods from ipv6 to your entire /64, or to
your whole ipv4, fairly well. And I had made the point that
prioritizing ping was a bad idea here (including some dripping sarcasm
later in the piece).

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

but wow, it never occurred to me - in all these years - that ping was
the next core metric on simple tests. I can be really dumb.

I use netperf-wrapper and tend to ignore most of the ping data, but
certainly on some benchmarks we have published ping doesn't look as
good as the other stuff, *because it is deprioritized below all the
other traffic*. Not strictly rate limited - as some systems do by
default, including openwrt, which is impossible to get right - just
deprioritized....

How can we fix this user perception, short of re-prioritizing ping in
sqm-scripts?

-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

---

From: Andrew Mcgregor <[email-masked]>
Date: Mon, 2 Mar 2015 15:00:40 +1100

Hash all pings into one fq_codel bucket reserved for the purpose, so if
we're getting DoSed we drop them, otherwise if they stay thin they get
prioritised?

On 2 March 2015 at 14:57, Dave Taht <[email-masked]> wrote:

> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 1071 2221

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 2 Mar 2015 05:05:18 +0100 (CET)

> How can we fix this user perception, short of re-prioritizing ping in

People will make all kinds of judgement calls based on things that are not 
very relevant.

Some people think a traceroute with few hops in it to be better than one 
with more hops in it. So a lot of providers use their MPLS networks and 
make the intermediate nodes not show up in traceroute.

I don't know how many times I have experienced people have used PING and 
opened trouble tickets when a core router had ICMP ping packet loss (even 
though all core router platforms have rate limiters for ICMP).

When you say that you de-prioritize ping, do you mean only ECHO REQUEST 
and ECHO REPLY, or do you mean all ICMP? Perhaps a solution would be to 
assure 5% of the link bandwidth for ICMP?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: David Lang <[email-masked]>
Date: Sun, 1 Mar 2015 20:06:46 -0800 (PST)

> than something else because of it."

People make the assumption that if ping packets are being dropped, their data is 
getting dropped as well. This is part of the "dropped packets are bad, so buffer 
so you don't drop packets" mentality

To address this, the test results should first show all the stats without 
showing dropped packets, or if you need to show them, add a comment at that 
point that says that data packets are given priority over ping packets, so 
dropped ping packets don't imply that data packets would be dropped.

David Lang

> Cerowrt-devel mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

---

From: Brian Trammell <[email-masked]>
Date: Mon, 2 Mar 2015 10:40:37 +0100

hi Dave,

> On 02 Mar 2015, at 04:57, Dave Taht <[email-masked]> wrote:
> 

I am not proud of the fact that I am not surprised by this.

> How can we fix this user perception, short of re-prioritizing ping in

The real solution is to create a utility called "ping" that uses traffic that gets prioritized the same way as the traffic you care about instead of ICMP echo request/reply. Users don't care about the packets on the wire so much as they do that you're supposed to ping things.

We have a protocol for this <ippm-chair-hat> called TWAMP </ippm-chair-hat> which shares the unfortunate property with all such approaches that it requires the far end to be running a reflector (the advantage of ICMP is the reflector is built into the kernel everywhere) or worse a control protocol for setting reflectors up.

Gaming protocols do this right - latency measurement is built into the protocol. For the web we already have tools and reasonably well accepted metrics (time to first byte). Wrap $YOUR_FAVORITE_BROWSER's perftools in a ping -w <url> utility and this problem goes away in that space. I presume that rtcweb clients in the browser will take advantage of these tools and provide measurements the same way games do.

Andrew's hack is probably easier to actually get deployed in a reasonable amount of time though. :)

Cheers,

Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 2 Mar 2015 11:17:33 +0100 (CET)

On Mon, 2 Mar 2015, Brian Trammell wrote:

> Gaming protocols do this right - latency measurement is built into the 
> protocol.

I believe this is the only way to do it properly, and the most likely 
easiest way to get this deployed would be to use the TCP stack.

We need to give users an easy-to-understand metric on how well their 
Internet traffic is working. So the problem here is that the users can't 
tell how well it's working without resorting to ICMP PING to try to figure 
out what's going on.

For instance, if their web browser had insight into what the TCP stack was 
doing then it could present information a lot better to the user. Instead 
of telling the user "time to first byte" (which is L4 information), it 
could tell the less novice user about packet loss, PDV, reordering, RTT, 
how well concurrent connections to the same IP address are doing, tell 
more about *why* some connections are slow instead of just saying "it took 
5.3 seconds to load this webpage and here are the connections and how long 
each took". For the novice user there should be some kind of expert system 
that collects data that you can send to the ISP that also has an expert 
system to say "it seems your local connection delays packets", please 
connect to a wired connection and try again". It would know if the problem 
was excessive delay, excessive delay that varied a lot, packet loss, 
reordering, or whatever.

We have a huge amount of information in our TCP stacks that either are 
locked in there and not used properly to help users figure out what's 
going on, and there is basically zero information flow between the 
applications using TCP and the TCP stack itself. Each just tries to do its 
best on its own layer.

---

From: Dave Dolson <[email-masked]>
Date: Mon, 2 Mar 2015 10:47:07 +0000

I'm rather new to the aqm community, but IMHO, it is wrong to deprioritize the ping traffic by default. I would not have expected a forwarding agent to do this. 

And I think measuring ping times and loss is a reasonable thing to do, never expecting forwarding agents along the path to place more value on some IP packets than others. (Especially in my own network/lab when I did not configure such a policy)

There aren't many tools available to an end user. Ping, traceroute, speed test... The network is a black box to most users. 

As for the flood attack aspect, of course a flood of pings should wait their turn in a queue and be dropped as the queue fills.

It would be appropriate if this was fair to different ping flows in the same way TCP SYN packets are treated fairly. Treat ping flood like TCP SYN flood.  

My 2cents. 
-Dave Dolson

---

From: Andrew Mcgregor <[email-masked]>
Date: Mon, 2 Mar 2015 21:49:22 +1100

So, are you suggesting that, for example, Chrome's rather extensive network
debugging information get more publicised?  We can probably arrange that.

On 2 March 2015 at 21:47, Dave Dolson <[email-masked]> wrote:

> I'm rather new to the aqm community, but IMHO, it is wrong to deprioritize
> the ping traffic by default. I would not have expected a forwarding agent
> to do this.
>
> And I think measuring ping times and loss is a reasonable thing to do,
> never expecting forwarding agents along the path to place more value on
> some IP packets than others. (Especially in my own network/lab when I did
> not configure such a policy)
>
> There aren't many tools available to an end user. Ping, traceroute, speed
> test... The network is a black box to most users.
>
> As for the flood attack aspect, of course a flood of pings should wait
> their turn in a queue and be dropped as the queue fills.
>
> It would be appropriate if this was fair to different ping flows in the
> same way TCP SYN packets are treated fairly. Treat ping flood like TCP SYN
> flood.
>
> My 2cents.
> -Dave Dolson
>
>
>
> To: [email-masked] <
> [email-masked]>; [email-masked] <[email-masked]>; bloat <
> [email-masked]>

---

From: [email-masked]
Date: Mon, 2 Mar 2015 07:44:18 -0500 (EST)

On my own web server (running nginx) I provide an HTTP 1.1 accessible statistics service.  It returns a single JSON structure with the underlying packet statistics for the server and the current connection. Since this packet is inserted into the HTTP 1.1 stream upon request, it provides both the statistics and a single-round-trip "ping" that can measure queue delay on demand.

Changing server code is a lot easier than changing browser code.  Anyone who would like to develop a "standards-track" option for HTTP 1.1 is quite welcome to do so...  right now it is a "RESTful" interface, so measurements of the client side must be obtained indirectly, but simple JavaScript code in a web page can access this with standard HTML5 code.

I'm not looking to support this code or enhance it.  I've thought about putting it up on GitHub.

In any case, it's easy to add a little widget to the corner of all the web pages from a site that displays latency and net statistics at various levels in a very pretty user-understandable form.

To see how this works, just go to the alt.reed.com site (preferably using an iPhone or Android phone) to see a trivial test app I put out a number of years ago for my personal zeroth-order diagnostics.

nginx (as most know) is an incredibly tight, low overhead HTTP 1.1 server that lets you add plugins for lots of things other than retrieving web pages from disk.  It's not the pile of spaghetti that Apache has become. It supports WebSockets and HTTPS quite well, too.

On Monday, March 2, 2015 5:54am, "Jonathan Morton" <[email-masked]> said:

> 
>> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <[email-masked]> wrote:
>>
>>> Gaming protocols do this right - latency measurement is built into the protocol.
>>
>> I believe this is the only way to do it properly, and the most likely easiest way
>> to get this deployed would be to use the TCP stack.
>>
>> We need to give users an easy-to-understand metric on how well their Internet
>> traffic is working. So the problem here is that the users can't tell how well
>> it's working without resorting to ICMP PING to try to figure out what's going on.
>>
>> For instance, if their web browser had insight into what the TCP stack was doing
>> then it could present information a lot better to the user. Instead of telling
>> the user "time to first byte" (which is L4 information), it could tell the less
>> novice user about packet loss, PDV, reordering, RTT, how well concurrent
>> connections to the same IP address are doing, tell more about *why* some
>> connections are slow instead of just saying "it took 5.3 seconds to load this
>> webpage and here are the connections and how long each took". For the novice user
>> there should be some kind of expert system that collects data that you can send
>> to the ISP that also has an expert system to say "it seems your local connection
>> delays packets", please connect to a wired connection and try again". It would
>> know if the problem was excessive delay, excessive delay that varied a lot,
>> packet loss, reordering, or whatever.
>>
>> We have a huge amount of information in our TCP stacks that either are locked in
>> there and not used properly to help users figure out what's going on, and there
>> is basically zero information flow between the applications using TCP and the TCP
>> stack itself. Each just tries to do its best on its own layer.
> 
> This seems like an actually good idea.  Several of those statistics, at least,
> could be exposed to userspace without incurring any additional overhead in the
> stack (except for the queries themselves), which is important for high-performance
> server users.  TCP stacks already track RTT, and sometimes MinRTT - the difference
> between these values is a reasonable lower-bound estimate of induced latency.
> 
> For stacks which don’t already track all the desirable data, a socket option
> could be used to turn that on, allocating extra space to do so.  To maximise
> portability, therefore, it might be necessary to require that option before
> statistics requests will be valid, even on stacks which do collect it all anyway.
> 
> Recent versions of Windows, even, have a semi-magic system which gives a little
> indicator of whether your connection has functioning Internet connectivity or not.
>  This could be extended, if Microsoft saw fit, to interpret these statistics and
> notify the user that their connection was behaving badly in the ways we now find
> interesting.  Whether Microsoft will do such a thing (which would undoubtedly piss
> off every major ISP on the planet) is another matter, but it’s a concept
> that can be used by Linux desktops as well, and with less political fallout.
> 
> Now, who’s going to knuckle down and implement it?
> 
>  - Jonathan Morton
>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 2 Mar 2015 12:54:45 +0200

> I believe this is the only way to do it properly, and the most likely easiest way to get this deployed would be to use the TCP stack.
> 
> We need to give users an easy-to-understand metric on how well their Internet traffic is working. So the problem here is that the users can't tell how well it's working without resorting to ICMP PING to try to figure out what's going on.
> 
> For instance, if their web browser had insight into what the TCP stack was doing then it could present information a lot better to the user. Instead of telling the user "time to first byte" (which is L4 information), it could tell the less novice user about packet loss, PDV, reordering, RTT, how well concurrent connections to the same IP address are doing, tell more about *why* some connections are slow instead of just saying "it took 5.3 seconds to load this webpage and here are the connections and how long each took". For the novice user there should be some kind of expert system that collects data that you can send to the ISP that also has an expert system to say "it seems your local connection delays packets", please connect to a wired connection and try again". It would know if the problem was excessive delay, excessive delay that varied a lot, packet loss, reordering, or whatever.
> 
> We have a huge amount of information in our TCP stacks that either are locked in there and not used properly to help users figure out what's going on, and there is basically zero information flow between the applications using TCP and the TCP stack itself. Each just tries to do its best on its own layer.

This seems like an actually good idea.  Several of those statistics, at least, could be exposed to userspace without incurring any additional overhead in the stack (except for the queries themselves), which is important for high-performance server users.  TCP stacks already track RTT, and sometimes MinRTT - the difference between these values is a reasonable lower-bound estimate of induced latency.

For stacks which don’t already track all the desirable data, a socket option could be used to turn that on, allocating extra space to do so.  To maximise portability, therefore, it might be necessary to require that option before statistics requests will be valid, even on stacks which do collect it all anyway.

Recent versions of Windows, even, have a semi-magic system which gives a little indicator of whether your connection has functioning Internet connectivity or not.  This could be extended, if Microsoft saw fit, to interpret these statistics and notify the user that their connection was behaving badly in the ways we now find interesting.  Whether Microsoft will do such a thing (which would undoubtedly piss off every major ISP on the planet) is another matter, but it’s a concept that can be used by Linux desktops as well, and with less political fallout.

 - Jonathan Morton

---

From: Brian Trammell <[email-masked]>
Date: Mon, 2 Mar 2015 15:45:10 +0100

> On 02 Mar 2015, at 11:54, Jonathan Morton <[email-masked]> wrote:
> 
> 

So there seem to me to be three separate but related problems we want to solve here:

(1) How to get users who don't care what ping is useful information about their applications' network performance.

(2) How to get users who do care what ping is information that actually reflects application performance when they type ping (or, more generally, how to make sure that the common diagnostic tools neither (a) provide misleading information or (b) require network misconfiguration to ensure "proper" operation of the diagnostic tools (cf. speedtest.net and its ilk).

(3) How to get application developers tools they can use to integrate network measurement into their apps without having to roll their own (i.e., helping them to solve (1), and enabling the creation of tools for (2) ).

This is an approach to (3)... but (as with many things) the key to getting it deployed and used on endpoints would be defining a universal interface to it. "Yet another API to learn on every platform" == "I'm just going to use ping, thank you very much."

> Recent versions of Windows, even, have a semi-magic system which gives a little indicator of whether your connection has functioning Internet connectivity or not.  This could be extended, if Micros…

This is, I'm afraid, the kicker. As long as everyone has an interest in pointing the finger at everyone else, people will choose to interpret the metrics how they like, and fail to respond to metrics they don't, no matter how good they are.

> Now, who’s going to knuckle down and implement it?

web10g.org is a start, for Linux anyway.

Cheers,

Brian

---

From: David Lang <[email-masked]>
Date: Mon, 2 Mar 2015 10:36:00 -0800 (PST)

> debugging information get more publicised?  We can probably arrange that.

That would be good. I have no idea what debugging info you are referring to.

David Lang

---

From: Wes Felter <[email-masked]>
Date: Mon, 02 Mar 2015 14:06:33 -0600

What about a token bucket policer, so more than N ICMP/second gets 
penalized but a normal ping won't be.

-- 
Wes Felter

---

From: Dave Dolson <[email-masked]>
Date: Mon, 2 Mar 2015 20:33:56 +0000

Would you do that to TCP or UDP traffic?

At IETF I often hear laments about middle-boxes breaking the internet by being "clever" with certain types of traffic.
It seems that policing ICMP falls into that category.

There may have been bugs in the past, but I'm not aware that ICMP packets are any more dangerous than UDP or TCP. And if the RFCs can be believed, ICMPv6 is critical to determining Path-MTU. Don't drop those.

One may wish to rate-limit ICMP (or DNS or TCP) flows as a matter of network policy, but in my opinion this should be kept orthogonal to solving buffer bloat.

Taken to the extreme, a network should support full utilization of a link doing only ping. If I wish to use my connection to the internet to ping hosts at full line rate, why not?

David Dolson
Senior Software Architect, Sandvine Inc.

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Wes Felter
Sent: Monday, March 02, 2015 3:07 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [aqm] ping loss "considered harmful"

-- 
Wes Felter

---

From: Dave Taht <[email-masked]>
Date: Mon, 2 Mar 2015 12:38:29 -0800

On Mon, Mar 2, 2015 at 12:06 PM, Wes Felter <[email-masked]> wrote:
> What about a token bucket policer, so more than N ICMP/second gets penalized
> but a normal ping won't be.

If I haven't expressed this too many times before, I want to thank you
all for existing.

It is the quality and caliber of all the folk on these mailing lists
coming up with new ideas, that keeps me going.

I've had a really rough could days (bronchitus, the virgin controversy
here: https://plus.google.com/u/0/107942175615993706558/posts/E1yMgbWW81C
), and I will try to write more later, but I thought I would add a bit
of light to the debate by showing you the default openwrt firewall
rules:

http://git.openwrt.org/?p=openwrt.git;a=blob;f=package/network/config/firewall/files/firewall.config;h=d149e77957de1c80358c1e8131920867a193c9ed;hb=HEAD

Problems: They do a (very low) syn limiter (25 syns/sec I think) by
default, which made sense 10 years ago, but not now - and I had run
into trouble with it while attempting to benchmark the alexa top 1
million in parallel through two routers at isc (at 300mbit) a few
years ago, so I ripped that out of cerowrt. It was a really puzzling
set of results, I had no idea why so many syn requests were dropping
on the floor til I inspected the actual firewall rules generated.

Similarly, they rate limit icmp at 1000/sec, which is too high at low
rates and too low at others. So I ripped that out of cerowrt, and in
sqm-scripts, I just held tossed it into the background bucket for for
a min of 30% of the background bandwidth.

I note that there are conflicting definitions of CS1 (background).
Comcast, re-marks about 90% I see to CS1 from whatever it was
originally, in the hope that it is treated as background. The ancient
firmware in commercial home routers *prioritizes* CS1 on etheret and
*deprioritizes it* on wifi, into the 802.11e background queue, when
enabled. CeroWrt tries to cons

I haven't seen ANY shipped open source FQ/AQM/QoS system that squashes
inbound dscp values, either, til cero, and despite the various rfcs
out there recommending that.

Openwrt's firewall system is, sadly, one of the better and saner
firewall rule sets that exists. Other rulesets I have seen in
commercial deployments are amazingly more complex, antiquated, and
frequently wrong.

as for fq/aqm/QoS I have pointed out the flaws in the comments on
wondershaper here:

and those flaws, exist, in copy-pasted code in nearly every subsequent
shaper I have examined *and had the time to fix*.

EVERY open source shaper I have seen, also (aside from sqm-scripts)
does not handle ipv6 right, nor do they handle dsl/atm framing issues
correctly. Notably streamboost got it terribly, terribly wrong, as did
Netgear's new "Dynamic QoS", and d-links DIR-5500. Openwrt's
qos-scripts has similar problems. So does gargoyle's stuff, and
dd-wrts... and whatever they were using at nznog on my last talk.

In the original design for sqm-scripts, I'd tried to use 3 tiers of
DRR + fq_codel - for classification, and failed due to in some
circumstances DRR getting "stuck" on a queue when codel did a drop on
it. (a bug that may only have existed at the time). So we went with
HTB, with a max of 30% for prio, a min of 30% for background (due to
all the mismarked traffic, 5% wasn't enough) and background and BE
being able to use up all the bandwidth.

That said, however free.fr did DRR + fq_codel in their implemention
benchmarks out quite nicely,  tho I can't find the relevant comparison
benchmarks at the moment.

particularly the prio queue helps for multicast IPTV traffic. I tried
to discuss some of these issues here, but the wg was not interested at
the time for a document like this.

http://snapon.lab.bufferbloat.net/~d/draft-taht-home-gateway-best-practices-00.html

Since then I specified the combined rate limiter + fq_codel idea
called "cake" with up to 8 progressive levels of DRR for
classification, which jonathon did a GREAT first attempt at, but he
didn't believe me that strict rate limits were not the right thing
until he got a bit of user feedback....

I should write up all the design decisions that went into sqm-scripts
as it exists today - it came out of me using varous shapers in various
environments for 12+ years and for example, I had completely forgotten
why I had deprioritized icmp the way I did! - and rip out a few test
bits (like prioritizing AF42 for mosh) that are not correct for the
real world... whenever I manage to get out of bed.

---

From: David Lang <[email-masked]>
Date: Mon, 2 Mar 2015 12:39:42 -0800 (PST)

> Taken to the extreme, a network should support full utilization of a link doing only ping. If I wish to use my connection to the internet to ping hosts at full line rate, why not?

what's going on here isn't that pings are being rate limited, but rather that 
the TCP/UDP traffic is being given priority over the ping traffic. This means 
that when you max out the pipe, pings will suffer.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Mar 2015 13:53:13 -0800

On 3/2/2015 1:40 AM, Brian Trammell wrote:
...
> The real solution is to create a utility called "ping" that uses 
> traffic that gets prioritized the same way as the traffic you care
> about instead of ICMP echo request/reply. Users don't care about
> the packets on the wire so much as they do that you're supposed to
> ping things.

There are three separate problems:

1. a ping that doesn't use ICMP
	there are dozens of these

2. needing a reflector
	ping gets around this only because the reflector is widely
	deployed (and integrated into most OSes)

3. using the same port as the traffic you care about
	transport protocol is only one problem (ICMP being a "transport
	protocol" by virtue of using the IP protocol number field)

	the other is differential prioritization based on port number

	there's no easy solution to that;
	every service would need an integrated
	ping reflector

I suspect #3 is the ultimate killer of this idea.

Joe

---

From: David Lang <[email-masked]>
Date: Mon, 2 Mar 2015 15:14:58 -0800 (PST)

> I suspect #3 is the ultimate killer of this idea.

The service you are trying to contact acts as a reflector for TCP traffic. If 
you send a syn you will get back a syn-ack from the TCP stack of the receiving 
system.

For UDP systems, it gets more interesting and service specific. But for TCP 
systems it works today.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Mar 2015 15:25:26 -0800

On 3/2/2015 3:14 PM, David Lang wrote:
> The service you are trying to contact acts as a reflector for TCP
> traffic. If you send a syn you will get back a syn-ack from the TCP
> stack of the receiving system.

Sure, but SYNs and SYN-ACKs don't get prioritized the same as
non-control TCP segments. And they could have been spoofed by a middlebox.

Joe

---

From: David Lang <[email-masked]>
Date: Mon, 2 Mar 2015 15:34:24 -0800 (PST)

> non-control TCP segments. And they could have been spoofed by a middlebox.

well, this is exactly the sort of thing that http heartbeat (the core of 
heartbleed) was designed to provide.

It's not perfect, you can see where you really get the response from (vi the 
traceroute), and if you are going through a MITM (i.e. transparent proxy), all 
you are ever going to be able to test is the network between yourself and the 
proxy. It's up to the people who own the proxy to test beyond that.

David Lang

---

From: Andrew Mcgregor <[email-masked]>
Date: Tue, 3 Mar 2015 11:07:25 +1100

Maybe I should mention https://github.com/apenwarr/blip

HTTP ping, using deliberate 204 responses.  Will run over whatever version
of HTTP/SPDY/QUIC your browser happens to be using at the time.

On 3 March 2015 at 10:34, David Lang <[email-masked]> wrote:

> It's not perfect, you can see where you really get the response from (vi
> the traceroute), and if you are going through a MITM (i.e. transparent
> proxy), all you are ever going to be able to test is the network between
> yourself and the proxy. It's up to the people who own the proxy to test
> beyond that.
>
> David Lang
>
>

---

From: Wes Felter <[email-masked]>
Date: Mon, 02 Mar 2015 23:54:06 -0600

> Would you do that to TCP or UDP traffic?

No, so I see your point. If an ICMP scan was dropped an attacker could 
simply switch to another protocol.

> One may wish to rate-limit ICMP (or DNS or TCP) flows as a matter of network policy, but in my opinion this should be kept orthogonal to solving buffer bloat.

Keeping in mind the original context which was subnet scans, I agree. 
But is there a way to do this in Linux? I guess it would use conntrack 
rather than tc.

(As an aside, I got a chuckle from seeing someone from a DPI company 
advocate treating all protocols equally.)

-- 
Wes Felter

---

From: [email-masked]
Date: Tue, 03 Mar 2015 01:45:25 -0500

> applications using TCP and the TCP stack itself. Each just tries to do its

You might want to touch base with the 'web10g' crew, who are working on
instrumenting the Linux network stack to expose more of the inner variables
for analysis and tuning.

http://web10g.org/

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Tue, 3 Mar 2015 17:20:19 +0000

> On Mar 1, 2015, at 7:57 PM, Dave Taht <[email-masked]> wrote:
> 

IMHO, ping should go at the same priority as general traffic - the default class, DSCP=0. When I send one, I am asking whether a random packet can get to a given address and get a response back. I can imagine having a command-line parameter to set the DSCP to another value of my choosing.

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 03 Mar 2015 12:29:31 -0500

On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
> 
>> On Mar 1, 2015, at 7:57 PM, Dave Taht <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>
> IMHO, ping should go at the same priority as general traffic - the
> default class, DSCP=0. When I send one, I am asking whether a random
> packet can get to a given address and get a response back. I can imagine
> having a command-line parameter to set the DSCP to another value of my
> choosing.
> 

I generally agree, however ...

The DSCP of the response isn't controllable though, and likely the DSCP
that is ultimately received will not be the one that was sent, so it
can't be as simple as echoing back the same one.  Ping doesn't tell you
latency components in the forward or return path (some other protocols
can do this though).

So, setting the DSCP on the outgoing request may not be all that useful,
depending on what the measurement is really for.

-- 
Wes Eddy
MTI Systems

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Tue, 3 Mar 2015 18:00:14 +0000

> On Mar 3, 2015, at 9:29 AM, Wesley Eddy <[email-masked]> wrote:
> 

Note that I didn’t say “I demand”… :-)

I share the perception that ping is useful when it’s useful, and that it is at best an approximation. If I can get a packet to the destination and a response back, and I know the time I sent it and the time I received the response, I know exactly that - messages went out and back and took some amount of total time. I don’t know anything about the specifics of the path, of buffers en route, or delay time in the target. Traceroute tells me a little more, at the cost of a more intense process. In places I use ping, I tend to send a number of them over a period of time and observe on the statistics that result, not a single ping result.

---

From: Dave Taht <[email-masked]>
Date: Tue, 3 Mar 2015 21:24:59 -0800

> Note that I didn’t say “I demand”… :-)

My point was A), I have seen tons of shapers out there that actually
prioritize ping over other traffic. I figure everyone here will agree
that is a terrible practice, but I can certainly say it exists, as it
is a dumb mistake replicated in tons of shapers I have seen... that
makes people in marketing happy.

Already put up extensive commentary on that bit of foolishness on
"wondershaper must die".

Please feel free to review any shapers or firewall code you might have
access to for the same sort of BS and/or post the code somewhere for
public review. A BCP for these two things would be nice.

And B) Deprioritizing ping (slightly) as I do came from what has
happened to me multiple times when hit by a bot that ping floods the
network. One time, 30+ virtual windows boxes in a lab got infected by
something that went nuts pinging the entire 10/8 network we were on.
It actually DID melt the switch - and merely getting to isolating that
network off from the rest was a PITA, as getting to the (SFQ-ing)
router involved was nearly impossible via ssh. (like, 2 minutes
between keystrokes).

Thus, ping, deprioritized. I tend to feel deprioritizing it slightly
is much more important in the post ipv6 world.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 4 Mar 2015 09:12:02 +0100 (CET)

> enabled. CeroWrt tries to cons

This is the default I have seen in quite a few 4 queue L2 devices, I 
believe it comes from IEEE recommendations:

https://community.extremenetworks.com/extreme/topics/default_802_1p_priority_to_transmit_queue_mapping

     Product           802.1p Priority/CoS    Transmit Queue

Matrix N; non-Policy Priority, 'show port priority-queue'
  -4 or 8 queues-              0      4&8 Qs-> 1
     Fast Ethernet ports       1               0
                               2               0
                               3               1
                               4               2
                               5               2
                               6               3
                               7               3

Since a lot of products will mark IP PREC part of TOS directly into .1p 
bits, this means CS0 and CS3 goes into higher priority queues compared to 
CS1 and CS2.

http://www.hp.com/rnd/device_help/help/hpwnd/webhelp/HPJ4121A/qos_priority_map.html 
seems to indicate HP does the same.

http://alliedtelesis.com/manuals/GS900M_Series_Web_Browser_User_Guide_revA/aw1001299.html 
says the same.

However, I find devices that do differently by default as you have already 
discovered.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 4 Mar 2015 09:14:03 +0100 (CET)

> instrumenting the Linux network stack to expose more of the inner variables

I know about them. I have been pitching my idea since 2008 or so. We need 
more people working on this.

---

From: [email-masked]
Date: Wed, 4 Mar 2015 12:34:41 -0500 (EST)

It's a heavy burden to place on ICMP ping to say that it should tell you about all aspects of its path through all the networks between source and destination.

On the other hand, I'll suggest that Fred's point - treat ICMP Ping like any other IP datagram with the same header options is the essence of Ping's function.

I'd suggest that a more flexible rule would be for the echo reply to set header options (including DSCP) based on the ping packet's content tells it to be set to.

DSCP should not be changed en route, so the receiver of the echo reply should be able to know what DSCP was used on the reply packet.

Clearly the value of Ping is its standardized form and its ubiquity.  Being able to control all header options from the sender is useful for that function.  If the receiver cannot satisfy the request (e.g. it doesn't support the DSCP mechanism), it can just refuse to set it. That way, Ping acquires an option, but the option is upward compatible if not supported.

(I specifically talk about all header options here, rather than DSCP in particular.  For example, one could request ECN marking in the same way, with the same rules. I'm not a big fan of DSCP because I think the code points are poorly defined and so forth, but that's irrelevant to the thinking about Ping vs. "envelope" option - fully end-to-end modular services like ECN clearly should be testable in this way, and in the case of ECN, the notion of just not doing it if you can't do it fits into the Ping conceptual framework).

On Tuesday, March 3, 2015 1:00pm, "Fred Baker (fred)" <[email-masked]> said:

> I share the perception that ping is useful when it’s useful, and that it is
> at best an approximation. If I can get a packet to the destination and a response
> back, and I know the time I sent it and the time I received the response, I know
> exactly that - messages went out and back and took some amount of total time. I
> don’t know anything about the specifics of the path, of buffers en route, or
> delay time in the target. Traceroute tells me a little more, at the cost of a more
> intense process. In places I use ping, I tend to send a number of them over a
> period of time and observe on the statistics that result, not a single ping
> result.
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 4 Mar 2015 20:45:45 +0100 (CET)

On Wed, 4 Mar 2015, [email-masked] wrote:

> DSCP should not be changed en route, so the receiver of the echo reply 
> should be able to know what DSCP was used on the reply packet.

Changing the DSCP bits by the ISP is definitely not unheard of, so don't 
count on it.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 05 Mar 2015 13:56:33 -0500

> Already put up extensive commentary on that bit of foolishness on

Its possible to detect such a shaper prioritizing ICMP echo/reply by
doing a an HTTP fetch concurrent with a ping and then and see if the
TCP data packet get significantly delayed relative to the ICMP echo
and echo reply packets.  You'd have to do a tcpdump and match the ICMP
echo to the echo reply and see if later the ICMP RTT looks very
different from the TCP RTT.  It might be that the SYN and SYN ACK are
not delayed but the plain old TCP date packets are.

If anyone has a small amount of spare time and wants to put together a
shell script its certainly doable.

Curtis

---

From: Dave Taht <[email-masked]>
Date: Thu, 5 Mar 2015 12:53:30 -0800

I had spoken to someone at nznog that promised to combine mrtg +
smokeping or cacti + smokeping so as to be able to get long term
latency and bandwidth numbers on one graph. cc added.

On Thu, Mar 5, 2015 at 12:38 PM, Matt Taggart <[email-masked]> wrote:
>> based on the amount of ping *loss*, rather than latency, and in looking back in time, I can
> This thread makes me realize that my standard method of measuring latency
> over time might have issues. I use smokeping
>
>   http://oss.oetiker.ch/smokeping/

in sqm-scripts's case, possibly, all you have been collecting is
largely worst case behavior, which I don't mind collecting as it tends
to be pretty good. :)

However, I have been unclear. In the main (modern - I don't know what
version you have) sqm code, IF you enable dscp squashing on inbound
(the default), you do end up with a single fq_codel queue, not 3, no
classification or ping prioritization. (it is the default because of
all the re-marking I have seen from comcast)

So if you are, as I am, monitoring your boxes from the outside, there
is no classification and prioritization present for ping.

do a tc -s qdisc show ifbwhatever (varies by platform) to see how many
queues you have. Example of a single queued inbound rate limiter +
fq_codel (yea! packet drop AND ecn working great!)

root@lorna-gw:~# tc -s qdisc show dev ifb4ge00
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0
direct_qlen 32
 Sent 168443514948 bytes 334370551 pkt (dropped 0, overlimits
143273498 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 110: parent 1:10 limit 1001p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
 Sent 168443514948 bytes 334370551 pkt (dropped 17480, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 125872421 ecn_mark 1044
  new_flows_len 0 old_flows_len 1

root@lorna-gw:~# uptime
 12:45:35 up 54 days, 22:33,  load average: 0.05, 0.05, 0.04

dscp classification in general, is only useful from within your own
network, going outside.

> which is a really nice way of measuring and visualizing packet loss and
> variations in latency. I am using the default probe type which uses fping
> (ICMP http://www.fping.org/ ).

I LOVE smokeping and wish very much we had a way to combine it with
mrtg data to see latency AND bandwidth at the same time.

>
> It has been working well, I set it up for a site in advance of setting up
> SQM and then afterwards I can see the changes and determine if more tuning
> is needed.  But if ICMP is having it's priority adjusted (up or down), then
> the results might not reflect the latency of other services.
>
> Fortunately the nice thing is that many other probe types exist
>
>   http://oss.oetiker.ch/smokeping/probe/index.en.html
>
> So which probe types would be good to use for bufferbloat measurement? I
> guess the answer is "whatever is important to you", but I also suspect
> there is a set of things that ISPs are known to mess with.
> HTTP? But also maybe HTTPS in case they are doing some sort of transparent
> proxy?
> DNS?
> SIP?
> I suppose you could even do explicit checks for things like Netflix (but
> then it's easy to go off on a tangent of building a net neutrality
> observatory).
>
> On a somewhat related note, I was once using smokeping to measure a fiber
> link to a bandwidth provider and had it configured to ping the router IP on
> the other side of the link. In talking to one of their engineers, I learned
> that they deprioritize ICMP when talking _with_ their routers, so my
> measurement weren't valid. (I don't know if they deprioritize ICMP traffic
> going _through_ their routers)

I do strongly recomend deprioritizing ping slightly, and as I noted, I
have seen many a borken
script that actually prioritized it, which is foolish, at best.

I keep hoping multiple (many!) someones here will go have lunch with
their company's oft lonely, oft starving sysadmin(s), to ask them what
they are doing as to firewalling, QoS and traffic shaping. Most of the
ones I have talked are quite eager to show off their work, which is
unfortunately often of wildly varying quality and complexity.

I find that an offer of saki and sushi are most conducive to getting
that conversation started.

I certainly would like to see more default corporate
firewall/QoS/shaping rules than I have personally, for various
platforms. Someone's got to have some good ideas in them... and it
would be nice to know how far the bad ones, have propagated.

> Matt Taggart
> [email-masked]

---

From: Rich Brown <[email-masked]>
Date: Thu, 5 Mar 2015 14:50:33 -0500

On Mar 5, 2015, at 1:56 PM, Curtis Villamizar <[email-masked]> wrote:

> doing a an HTTP fetch concurrent with a ping... 

For an easy (but imprecise) way test the HTTP response times, try Blip - http://gfblip.appspot.com/ (or read about it on github: https://github.com/apenwarr/blip) Blip sends short http requests to a couple hosts and measures the response time of the error pages. 

> and then and see if the
> Bloat mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/bloat

---

From: Matt Taggart <[email-masked]>
Date: Thu, 05 Mar 2015 12:38:13 -0800

-- 
Matt Taggart
[email-masked]