Subject: [aqm] Sane analysis of "typical traffic"

From: Dave Taht <[email-masked]>
Date: Tue, 15 Jul 2014 14:00:01 -0700

changing the title as this is not relevant to the aqm document...

... but to an attitude that is driving me absolutely crazy.

On Tue, Jul 15, 2014 at 10:46 AM, Akhtar, Shahid (Shahid)
<[email-masked]> wrote:
> Dave,
>
> The message of the results that we presented in November is that it is possible, with currently deployed access hardware, to configure RED so that it consistently improves the end user experience of common network services over Tail-Drop (which is most often configured), and that this improvement can be achieved with a fixed set of RED configuration guidelines.
>
> We did not run experiments with sfq_codel because it is not deployed in access networks today. We ran experiments with plain CoDel to understand the difference between a well-configured RED and a more recent single-bucket AQM in our target scenarios, and as reported, didn't observe significant differences in application QoE.

Your "application" was a bunch of video streams. Not web traffic, not
voip, not, gaming, not bittorrent, not a family of four doing a
combination of these things, nor a small business that isn't going to
use HAS at all.

Please don't over generalize your results. "RED proven suitable for
family of couch potatoes surfing the internet and watching 4 movies at
once over the internet but not 5, at 8mbit/sec" might have been a
better title for this paper.

In this fictional family, just one kid under the stair, trying to do
something useful, interactive and/or fun, can both wreck the couch
potatoes' internet experience, and have his own, wrecked also.

>
> Additional inline clarifications below.
>
> -Shahid.
>
> -----Original Message-----
> From: Dave Taht [mailto:[email-masked]]
> Sent: Monday, July 14, 2014 2:00 PM
> To: Akhtar, Shahid (Shahid)
> Cc: Fred Baker (fred); John Leslie; [email-masked]
> Subject: Re: [aqm] Obsoleting RFC 2309
>
> On Mon, Jul 14, 2014 at 11:08 AM, Akhtar, Shahid (Shahid) <[email-masked]> wrote:
>> Hi Fred, All,
>>
>> Let me an additional thought to this issue.
>>
>> Given that (W)RED has been deployed extensively in operators' networks, and most vendors are still shipping equipment with (W)RED, concern is that obsoleting 2309 would discourage research on trying to find good configurations to make (W)RED work.
>>
>> We had previously given a presentation at the ICCRG on why RED can still provide value to operators (http://www.ietf.org/proceedings/88/slides/slides-88-iccrg-0.pdf). We have a paper at Globecom 2014 that explains this study much better, but I cannot share a link to it until the proceedings are available.
>
> My problem with the above preso and no doubt the resulting study is that it doesn't appear cover the classic, most basic, bufferbloat scenario, which is
>
> "1 stream up, 1 stream down, one ping (or some form of voip-like traffic)" and usually on an edge network with asymmetric bandwidth.

Two additional analyses of use from the download perspective might be
Arris's analysis of the benefits of red and fq over cable head ends:

http://snapon.lab.bufferbloat.net/~d/trimfat/Cloonan_Paper.pdf

and the cable labs work which focused more on the effects of traffic
going upstream which has been discussed fairly extensively here.

> SA: We tried to cover the typical expected traffic over the Internet.

I don't know where you get your data, but my measured edge traffic
looks nothing like yours. Sure bandwidth wise, theres the netflix
spike 3 hours out of the day but the rest sure isn't HAS.

>Most of the traffic is now HAS traffic (as per the sandvine report), so if only a single stream is present, it is likely to be HAS.

>The closest approximation of a continuous TCP stream, as you mention, would be a progressive download which can last long enough to look continuous. These were modeled together with other types of traffic.

You keep saying, download, download, download. I am saying merely
please ALWAYS try an upload at the same time you are testing downloads
- be it videoconferencing (which can easily use up that 1.6mbit link),
a youtube upload, a rsync backup, a scp, anything...

It needent be the crux of your paper! But adding several tests of that
sort does need to inform your total modeling experience.

If you do that much, you we get a feel for how present day systems
interact with things like ack clocking which will do very interesting
things to your "downstream to the couch potato" performance metrics.

>
> It's not clear from the study that this is a 8mbit down 1mbit up DSL network (?),
>
> SA: In the study presented, It was 8M down and 1.6M up - slide 9

Thx.

>
> nor is it clear if RED is being applied in both directions or only one direction onl?
>
> SA: AQMs (including RED) were only applied in downstream direction - slide 9

Are you going to follow up with stuff that looks at the upstream direction?

> (and the results you get from an asymmetric network are quite interesting, particularly in the face of any cross traffic at all)
>
> SA: I am not sure what you mean by cross-traffic, the DSL link only goes to one residence (or business). The typical traffic to that was modeled.

I do need a consistent name for this that matches people's
expectations. Does not cross traffic make sense in a DSL context?
Competing?

I mean someone doing *anything* that involves sending data upstream
other than tiny tcp acks.

> And I do keep hoping that someone will do a study of how prevalent fq is on dsl devices, I keep accumulating anecdotal data that it's fairly common.
>
>> One of the major reasons why operators chose not to deploy (W)RED was a number of studies and research which gave operators conflicting messages on the value of (W)RED and appropriate parameters to use. Some of these are mentioned in the presentation above.
>>
>> In it we show that the previous studies which showed low value for RED used web traffic which had very small file sizes (of the order of 5-10 packets), which reduces the effectives of all AQMs which work by dropping or ECN marking of flows to indicate congestion. Today's traffic is composed of mostly multi-media traffic like HAS or video progressive download which has much larger file sizes and can be controlled much better with AQMs and in our research we show that RED can be quite effective with this traffic, with little tuning needed for typical residential access flows.

I agree, incidentally, that RED is OK for fat non-interactive video
flows over tcp. And I approve of people coming up with ways to tune it
properly for all sorts of environments and fully documenting how to do
it.

>
> I tend to disagree with this, (well, not the trendline to big video
> flows) but my own studies are focused on retaining high performance gaming, voip, and videoconferencing traffic in the face of things like HAS video traffic. I would be much happier about your study had you also run that sort of traffic while testing your video downloads and aqms... and why is it so hard to get folk to try sfq_codel, etc, while they try everything else?
>
> SA: We did not model smaller flows (in terms of throughput). Small

LATENCY, damn it.

>UDP flows, which I assume all/most of the above would fall in, would benefit significantly with RED - slide 13 shows much lower average queue length with RED.

Average queue length is valueless, according to Kathy and Van - and a
few other people, including me.

>Other studies have shown that small UDP flows with lots of TCP traffic benefit significantly with RED achieving lower packet loss (M. Mays et al, “Reasons not to deploy RED”).

Hmm... you really, really, really need to spend some time playing with
real traffic over real, asymmetric networks that goes in both
directions.

>>
>> Prefer John's proposal of updating 2309 rather than obsoleting, but if we can have some text in Fred's draft acknowledging the large deployment of (W)RED and the need to still find good configurations - that may work. I can volunteer to provide that text.
>>
>> -Shahid.
>
>>
>> -----Original Message-----
>> From: aqm [mailto:[email-masked]] On Behalf Of Fred Baker (fred)
>> Sent: Monday, July 14, 2014 2:06 AM
>> To: John Leslie
>> Cc: [email-masked]
>> Subject: Re: [aqm] Obsoleting RFC 2309
>>
>>
>> On Jul 3, 2014, at 10:22 AM, John Leslie <[email-masked]> wrote:
>>
>>> It would be possible for someone to argue that restating a
>>> recommendation from another document weakens both statements; but I
>>> disagree: We should clearly state what we mean in this document, and
>>> I believe this wording does so.
>>
>> The argument for putting it in there started from the fact that we are obsoleting 2309, as stated in the charter. I would understand a document that updates 2309 to be in a strange state if 2309 is itself made historic or obsolete. So we carried the recommendation into this document so it wouldn't get lost.
>>
>> _______________________________________________
>> aqm mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/aqm
>
>
>
> --
> Dave Täht
>
> NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

---

From: "Akhtar, Shahid (Shahid)" <[email-masked]>
Date: Thu, 17 Jul 2014 15:51:20 +0000

Dave,

Web traffic was part of the study - you should be able to see page load times on slide 12

Lower levels of traffic were also tested (slides 11,12), however the highest impact of AQM was observed at high traffic levels

We included the largest components of Internet traffic (HAS, Web Traffic, Video Progressive download) which constitute over 75% of all traffic (https://www.sandvine.com/downloads/general/global-internet-phenomena/2014/1h-2014-global-internet-phenomena-report.pdf) at peak hour

The study presented in Nov was our initial work and is not the only component in development of any configuration guidelines

-Shahid.

-----Original Message-----
From: Dave Taht [mailto:[email-masked]] 
Sent: Tuesday, July 15, 2014 4:00 PM
To: Akhtar, Shahid (Shahid)
Cc: Fred Baker (fred); John Leslie; [email-masked]
Subject: Sane analysis of "typical traffic"

> We did not run experiments with sfq_codel because it is not deployed in access networks today. We ran experiments with plain CoDel to understand the difference between a well-configured RED and a m…

Your "application" was a bunch of video streams. Not web traffic, not voip, not, gaming, not bittorrent, not a family of four doing a combination of these things, nor a small business that isn't going to use HAS at all.

Please don't over generalize your results. "RED proven suitable for family of couch potatoes surfing the internet and watching 4 movies at once over the internet but not 5, at 8mbit/sec" might have been a better title for this paper.

In this fictional family, just one kid under the stair, trying to do something useful, interactive and/or fun, can both wreck the couch potatoes' internet experience, and have his own, wrecked also.

> My problem with the above preso and no doubt the resulting study is 
> that it doesn't appear cover the classic, most basic, bufferbloat 
> scenario, which is
>

Two additional analyses of use from the download perspective might be Arris's analysis of the benefits of red and fq over cable head ends:

and the cable labs work which focused more on the effects of traffic going upstream which has been discussed fairly extensively here.

> SA: We tried to cover the typical expected traffic over the Internet.

I don't know where you get your data, but my measured edge traffic looks nothing like yours. Sure bandwidth wise, theres the netflix spike 3 hours out of the day but the rest sure isn't HAS.

>The closest approximation of a continuous TCP stream, as you mention, would be a progressive download which can last long enough to look continuous. These were modeled together with other types of t…

You keep saying, download, download, download. I am saying merely please ALWAYS try an upload at the same time you are testing downloads
- be it videoconferencing (which can easily use up that 1.6mbit link), a youtube upload, a rsync backup, a scp, anything...

It needent be the crux of your paper! But adding several tests of that sort does need to inform your total modeling experience.

If you do that much, you we get a feel for how present day systems interact with things like ack clocking which will do very interesting things to your "downstream to the couch potato" performance metrics.

Thx.

> (and the results you get from an asymmetric network are quite 
> interesting, particularly in the face of any cross traffic at all)
>

I do need a consistent name for this that matches people's expectations. Does not cross traffic make sense in a DSL context?
Competing?

I mean someone doing *anything* that involves sending data upstream other than tiny tcp acks.

>> In it we show that the previous studies which showed low value for RED used web traffic which had very small file sizes (of the order of 5-10 packets), which reduces the effectives of all AQMs whi…

I agree, incidentally, that RED is OK for fat non-interactive video flows over tcp. And I approve of people coming up with ways to tune it properly for all sorts of environments and fully documenting how to do it.

LATENCY, damn it.

>UDP flows, which I assume all/most of the above would fall in, would benefit significantly with RED - slide 13 shows much lower average queue length with RED.

Average queue length is valueless, according to Kathy and Van - and a few other people, including me.

>Other studies have shown that small UDP flows with lots of TCP traffic benefit significantly with RED achieving lower packet loss (M. Mays et al, “Reasons not to deploy RED”).

Hmm... you really, really, really need to spend some time playing with real traffic over real, asymmetric networks that goes in both directions.

> NSFW: 
> https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_i
> ndecent.article

--
Dave Täht

---

From: Dave Taht <[email-masked]>
Date: Thu, 17 Jul 2014 09:30:06 -0700

> We included the largest components of Internet traffic (HAS, Web Traffic, Video Progressive download) which constitute over 75% of all traffic (https://www.sandvine.com/downloads/general/global-int…

I continually am mind-blown by statistical legerdemain.

Just look at the size of the ps3/ps4/xbox gamer market for example. I
don't care about the amount of traffic at all. I care that in a
all-night aming session or a hour long videconference that the
relatively sparse latency sensitive packets get through - in both
directions - regardless of what else is going on on the network.

> The study presented in Nov was our initial work and is not the only component in development of any configuration guidelines

Got any gamers helping write your specs?

-- 
Dave Täht

---

From: Andrew Mcgregor <[email-masked]>
Date: Sun, 20 Jul 2014 12:21:44 -0700

Traffic volume has basically nothing to do with traffic value.  Consider
the DNS... a tiny fraction of upstream traffic must work in order for
anything else to work.

Similarly, a VoIP call is a tiny amount of traffic, but rather sensitive to
loss.

Games can be quite chatty, but even so they are sensitive to latency
variability (latency compensation works much better if the latency is
fairly consistent).

Whereas, if my HD video stream downgrades one codec notch, that will halve
the bandwidth and I will barely notice the difference.

On asymmetric DSL, the upstream value is usually much higher than
downstream... if I send anything but ACKs I probably care a lot about
whatever that traffic is.

On 17 July 2014 09:30, Dave Taht <[email-masked]> wrote:

> > Lower levels of traffic were also tested (slides 11,12), however the
> highest impact of AQM was observed at high traffic levels
> >
> > We included the largest components of Internet traffic (HAS, Web
> Traffic, Video Progressive download) which constitute over 75% of all
> traffic (
> https://www.sandvine.com/downloads/general/global-internet-phenomena/2014/1h-2014-global-internet-phenomena-report.pdf)
> at peak hour
>
> > The study presented in Nov was our initial work and is not the only
> component in development of any configuration guidelines
>
> >> The message of the results that we presented in November is that it is
> possible, with currently deployed access hardware, to configure RED so that
> it consistently improves the end user experience of common network services
> over Tail-Drop (which is most often configured), and that this improvement
> can be achieved with a fixed set of RED configuration guidelines.
> >>
> >> We did not run experiments with sfq_codel because it is not deployed in
> access networks today. We ran experiments with plain CoDel to understand
> the difference between a well-configured RED and a more recent
> single-bucket AQM in our target scenarios, and as reported, didn't observe
> significant differences in application QoE.
> >
> voip, not, gaming, not bittorrent, not a family of four doing a combination
> of these things, nor a small business that isn't going to use HAS at all.
> >
> family of couch potatoes surfing the internet and watching 4 movies at once
> over the internet but not 5, at 8mbit/sec" might have been a better title
> for this paper.
> >
> >>> Given that (W)RED has been deployed extensively in operators'
> networks, and most vendors are still shipping equipment with (W)RED,
> concern is that obsoleting 2309 would discourage research on trying to find
> good configurations to make (W)RED work.
> >>>
> >>> We had previously given a presentation at the ICCRG on why RED can
> still provide value to operators (
> http://www.ietf.org/proceedings/88/slides/slides-88-iccrg-0.pdf). We have
> a paper at Globecom 2014 that explains this study much better, but I cannot
> share a link to it until the proceedings are available.
> >>
> >> "1 stream up, 1 stream down, one ping (or some form of voip-like
> traffic)" and usually on an edge network with asymmetric bandwidth.
> >
> > I don't know where you get your data, but my measured edge traffic looks
> nothing like yours. Sure bandwidth wise, theres the netflix spike 3 hours
> out of the day but the rest sure isn't HAS.
> >
> >
> >>Most of the traffic is now HAS traffic (as per the sandvine report), so
> if only a single stream is present, it is likely to be HAS.
> >
> >>The closest approximation of a continuous TCP stream, as you mention,
> would be a progressive download which can last long enough to look
> continuous. These were modeled together with other types of traffic.
> >
> > You keep saying, download, download, download. I am saying merely please
> ALWAYS try an upload at the same time you are testing downloads
> > - be it videoconferencing (which can easily use up that 1.6mbit link), a
> youtube upload, a rsync backup, a scp, anything...
> >
> >> SA: I am not sure what you mean by cross-traffic, the DSL link only
> goes to one residence (or business). The typical traffic to that was
> modeled.
> >
> > I do need a consistent name for this that matches people's expectations.
> Does not cross traffic make sense in a DSL context?
> > Competing?
> >
> >> And I do keep hoping that someone will do a study of how prevalent fq
> is on dsl devices, I keep accumulating anecdotal data that it's fairly
> common.
> >>
> >>> One of the major reasons why operators chose not to deploy (W)RED was
> a number of studies and research which gave operators conflicting messages
> on the value of (W)RED and appropriate parameters to use. Some of these are
> mentioned in the presentation above.
> >>>
> >>> In it we show that the previous studies which showed low value for RED
> used web traffic which had very small file sizes (of the order of 5-10
> packets), which reduces the effectives of all AQMs which work by dropping
> or ECN marking of flows to indicate congestion. Today's traffic is composed
> of mostly multi-media traffic like HAS or video progressive download which
> has much larger file sizes and can be controlled much better with AQMs and
> in our research we show that RED can be quite effective with this traffic,
> with little tuning needed for typical residential access flows.
> >
> properly for all sorts of environments and fully documenting how to do it.
> >
> >>
> >> flows) but my own studies are focused on retaining high performance
> gaming, voip, and videoconferencing traffic in the face of things like HAS
> video traffic. I would be much happier about your study had you also run
> that sort of traffic while testing your video downloads and aqms... and why
> is it so hard to get folk to try sfq_codel, etc, while they try everything
> else?
> >>
> >>UDP flows, which I assume all/most of the above would fall in, would
> benefit significantly with RED - slide 13 shows much lower average queue
> length with RED.
> >
> >>Other studies have shown that small UDP flows with lots of TCP traffic
> benefit significantly with RED achieving lower packet loss (M. Mays et al,
> “Reasons not to deploy RED”).
> >
> real traffic over real, asymmetric networks that goes in both directions.
> >
> >>>
> >>> Prefer John's proposal of updating 2309 rather than obsoleting, but if
> we can have some text in Fred's draft acknowledging the large deployment of
> (W)RED and the need to still find good configurations - that may work. I
> can volunteer to provide that text.
> >>>
> >>> -Shahid.
> >>
> >>>
> >>> The argument for putting it in there started from the fact that we are
> obsoleting 2309, as stated in the charter. I would understand a document
> that updates 2309 to be in a strange state if 2309 is itself made historic
> or obsolete. So we carried the recommendation into this document so it
> wouldn't get lost.
> >>>
> https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>
>
>
> --
> Dave Täht
>
> NSFW:
> https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 1071 2221

---

From: "Akhtar, Shahid (Shahid)" <[email-masked]>
Date: Mon, 21 Jul 2014 12:33:16 +0000

Agreed that applications which constitute a small amount of traffic may carry a large value to the end user. The objective of modeling the largest components was to model the majority of TCP traffic. The amount of VoIP traffic and gaming traffic, for example is not listed individually in the referenced Sandrine report (report lists components more than 1%), so these are expected to be a small portion of the total download traffic. The impact of this traffic is expected to be little/negligible on the large TCP components.

The impact of the large TCP traffic on the small UDP flows (VoIP, gaming, DNS are all good examples) with/without AQM is another issue. This was not studied in the study presented last Nov. However, this issue has been studied to some extent earlier (see M. Mays et al, “Reasons not to deploy RED”), where the authors found a 4 times reduction in UDP packet loss with lots of TCP traffic. This is somewhat intuitive as the UDP flows are less likely to find the queue to be full with AQM.

From: Andrew Mcgregor [mailto:[email-masked]]
Sent: Sunday, July 20, 2014 2:22 PM
To: Dave Taht
Cc: Akhtar, Shahid (Shahid); [email-masked]<mailto:[email-masked]>; Fred Baker (fred); John Leslie
Subject: Re: [aqm] Sane analysis of "typical traffic"

Traffic volume has basically nothing to do with traffic value.  Consider the DNS... a tiny fraction of upstream traffic must work in order for anything else to work.

Similarly, a VoIP call is a tiny amount of traffic, but rather sensitive to loss.

Games can be quite chatty, but even so they are sensitive to latency variability (latency compensation works much better if the latency is fairly consistent).

Whereas, if my HD video stream downgrades one codec notch, that will halve the bandwidth and I will barely notice the difference.

On asymmetric DSL, the upstream value is usually much higher than downstream... if I send anything but ACKs I probably care a lot about whatever that traffic is.

On 17 July 2014 09:30, Dave Taht <[email-masked]<mailto:[email-masked]>> wrote:
On Thu, Jul 17, 2014 at 8:51 AM, Akhtar, Shahid (Shahid)

> From: Dave Taht [mailto:[email-masked]<mailto:[email-masked]>]
> Cc: Fred Baker (fred); John Leslie; [email-masked]<mailto:[email-masked]>
> On Tue, Jul 15, 2014 at 10:46 AM, Akhtar, Shahid (Shahid) <[email-masked]<mailto:[email-masked]>> wrote:
>> Dave,
>>
>> From: Dave Taht [mailto:[email-masked]<mailto:[email-masked]>]
>> Cc: Fred Baker (fred); John Leslie; [email-masked]<mailto:[email-masked]>
>> On Mon, Jul 14, 2014 at 11:08 AM, Akhtar, Shahid (Shahid) <[email-masked]<mailto:[email-masked]>> wrote:
>>> Hi Fred, All,
>>>
>>> From: aqm [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Fred Baker
>>> (fred)
>>> Cc: [email-masked]<mailto:[email-masked]>
>>> On Jul 3, 2014, at 10:22 AM, John Leslie <[email-masked]<mailto:[email-masked]>> wrote:
>>>
>>> [email-masked]<mailto:[email-masked]>

--
Dave Täht

_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

--
Andrew McGregor | SRE | [email-masked]<mailto:[email-masked]> | +61 4 1071 2221