Subject: Re: [aqm] [AQM Evaluation Guidelines]

From: Nicolas KUHN <[email-masked]>
Date: Tue, 15 Apr 2014 10:23:49 +0200

Hi Shahid Akhtar, all,

After the discussions in London IETF 89, there has been lots of modifications to the draft organisation (as we already advertised on the list). 
The authors and I just discussed your points considering the size of the files for the bulk transfer and the HTTP traffic. 
In the current version of the draft, we consider the values mentioned in the cablelabs work [0].

As a result, we consider bulk TCP transfers (repeating 5MB file transmission) and realistic HTTP web traffic (repeated download of 700kB). As a reminder, please find here the comments of Shahid Akhtar regarding these values:

> "Bulk TCP transfer: (continuous file transmission, or repeating 5MB file transmission);"
> The largest type of traffic on the Internet that uses bulk TCP - or large files via TCP is progressive download. The average speed of a Youtube video is over 3Mbps and it is on average 4.5 minutes long - so the average file size is considerably larger than 5MB. We have used an average of 38MB assuming 480p video and 4.5 minutes for some of our experiments. The inter-arrival time of these files should have a realistic random distribution as well as size of the files. 

> "Realistic HTTP web traffic (repeated download of 700kB);"
> According to statistics that Google collected from 2010 about web traffic, the average GET file size for web traffic was only 7KB. It is very important to have the size of these files correct since AQMs can help transfers of small files when large file downloads are also occurring. The effect of AQM on a 7KB file would be very different than on a 700KB file. Web traffic also has a complex inter-arrival time relationship between the GET requests as well as the pages that produce these GET requests. However to make the testing simpler, a repeated download of a 7KB file with a Pareto distributed size and Pareto distributed inter-arrival time may be an OK compromise.

We expect to stick with the cable labs values [0]. Please let us know if you have any comments regarding this choice. 

Kind regards, 

Nicolas KUHN

[0] http://www.cablelabs.com/wp-content/uploads/2013/11/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf

On Feb 20, 2014, at 6:48 AM, "Akhtar, Shahid (Shahid)" <[email-masked]> wrote:

> Hi Nicholas and other authors,
> 
> Thank you for producing this document. I had some (long) comments where I have also suggested some solutions/text, hopefully you find them helpful. When quoting from the document, I have used quotes and I have listed my comments as 1-4.
> 
> 
> 1. The number of tests recommended per AQM scheme seems to be quite large as well as many tests that may be best done together are done independently.
> 
> For example, there are different tests for fairness between flows (section 3.2.3), level of traffic (mild, medium, heavy) (section 3.2.4), different types of bursty traffic (3.2.2) and different network environments (section 3.3). In order to obtain a realistic estimate of AQM comparison, many of these effects may need to be tested together. For example to increase level of traffic (in section 3.2.4) only long term TCP flows of 50s are proposed. It would be better to test the level of congestion with realistic traffic (and with realistic proportions) as proposed in section 3.2.4. Another example would be to test fairness between flows of same type - e.g. long term TCP flows when multiple types of TCP flows are present at a AQM controlled buffer or test fairness between flows of different types - e.g. long term TCP flows and web flows. 
> 
> I noticed that in section 3.3 different network environments are mentioned. I suggest that tests with appropriate realistic traffic be done on each of these enviroments with various levels of congestion. One could measure fairness from these tests without the need for additional tests. This may have the benefit of reducing overall number of tests as well.
> 
> 
> 2. There are four network enviroments mentioned in the document (section 3.3):
> Wifi - in home network.
> Data-Center communications.
> BB wireless/satellite.
> Low and high buffers
> 
> This is a good set, however, recommed to add:
> 
> Access networks - and further split access networks into DSL/FTTH networks and Cable access networks. Cable access networks may have much larger number of flows than DSL/FTTH since they are shared by a large number of customers
> 
> There is another type of network environment - narrowband wireless (mobile) networks where the AQM could be placed on the eNodeB, but perhaps this is a complicated scenario, and best left for later or in another document.
> 
> 
> 3. I noticed that the modeling of different types of realistic traffic could be made more accurate (section 3.2.2)
> 
> "Bulk TCP transfer: (continuous file transmission, or repeating 5MB file transmission);"
> The largest type of traffic on the Internet that uses bulk TCP - or large files via TCP is progressive download. The average speed of a Youtube video is over 3Mbps and it is on average 4.5 minutes long - so the average file size is considerably larger than 5MB. We have used an average of 38MB assuming 480p video and 4.5 minutes for some of our experiments. The inter-arrival time of these files should have a realistic random distribution as well as size of the files. 
> 
> "Realistic HTTP web traffic (repeated download of 700kB);"
> According to statistics that Google collected from 2010 about web traffic, the average GET file size for web traffic was only 7KB. It is very important to have the size of these files correct since AQMs can help transfers of small files when large file downloads are also occurring. The effect of AQM on a 7KB file would be very different than on a 700KB file. Web traffic also has a complex inter-arrival time relationship between the GET requests as well as the pages that produce these GET requests. However to make the testing simpler, a repeated download of a 7KB file with a Pareto distributed size and Pareto distributed inter-arrival time may be an OK compromise.
> 
> I noticed that adaptive streaming is not modelled as a type of realistic traffic. It is probably the most important and largest flow on the Internet today - if we look at a recent Sandvine report of Internet traffic, about 75% of Internet traffic is real-time entertainment of which the largest component is adaptive streaming based video traffic. In order to simulate adaptive streaming traffic, one would need a client that changes video quality rates. To keep testing simpler, an option may be to generate realistic chunk sized files at regular intervals. Average netflix speeds are around 2Mbps. Typical chunk sizes range from 2s to 10s, but 4s may be a good average. The chunk sizes should be varied with some random distribution as is typically observed from real codecs. 
> 
> 
> 4. In section 2.2.5, QoE metrics are mentioned. It may be valuable to spell out some standard models or expressions to derive QoE from network parameters so that these QoE can be compared between AQMs. In the literature there is work that derives closed form expressions for QoE. The following are examples and suggestions:
> 
> Video - the work by Johan de Vriendt et al , "Model for estimating QoE of Video delivered using HTTP Adaptive Streaming", IFIP/IEEE Workshop on QoE Centric Management, May 2013 may be useful here. The authors used over 500 samples to derive their a closed form expressions. The authors derive the MOS = alpha*(average bit rate) + beta*(standard deviation of bit rates) + sigma*(no of bit rate switches) + gamma and then produce good values for alpha, beta, sigma and gamma.
> 
> Voice - The work in "Voice quality prediction models and their application in VoIP networks" by L. Sun et al has closed form expressions for MOS scores for most commercial codes. They derive a long closed form expression with parameters a-j for key codecs (and then provide a-j for most commercial codecs) based on end-to-end and packet loss.  
> 
> Web - The delay for one average web GET file could be used as the representative for the page load time
> 
> Key metric for AQMs on access networks may be QoE since it provides and end-user view. We have seen that often the absolute goodput does not matter as much as the stability of the flow for Video (which tends to be the dominant traffic). Larger goodput may provide higher VQ levels for Video, but if higher volatility is also produced then that higher VQ level may not produce a higher QoE.
> 
> On the other hand, best metrics inter data center traffic may be goodput and delay since most of that traffic is likely to be inter machine traffic.
> 
> Best regards,
> 
> Shahid Akhtar.
> 
> -----Original Message-----
> From: aqm [mailto:[email-masked]] On Behalf Of Nicolas KUHN
> Sent: Friday, February 07, 2014 5:58 AM
> To: Richard Scheffenegger
> Cc: [email-masked]
> Subject: [aqm] [AQM Evaluation Guidelines]
> 
> Dear all, 
> 
> On the behalf of the contributors to the AQM Evaluation Guidelines, I encourage active discussion on the draft that we have written. 
> I just submitted the draft as an individual draft to the I-D Submission Tool [0].
> 
> Please let us know what you think of the current document. 
> 
> Kind regards, 
> 
> Nicolas KUHN
> 
> [0] http://datatracker.ietf.org/doc/draft-kuhn-aqm-eval-guidelines/
> 
> 
> On Feb 5, 2014, at 10:01 PM, "Scheffenegger, Richard" <[email-masked]> wrote:
> 
>> Hi,
>> 
>> a new month, a new status report.
>> 
>> First of all, Wes and I as chairs would like to thank the editors who have stepped forward to work on the AQM Evaluation Guideline draft. We are really thankful for their burst of efforts in the last couple weeks!
>> 
>> We expect that that a document will be ready for submission into the I-D repository well before cutoff, as an individual draft, so that the WG can see what the state of thinking is currently. Also, once the document is published on datatracker we'd like to encourage active discussion on it.
>> 
>> If the submitted -00 draft is felt to have a fairly complete outline of what the current thinking in the WG is, we may be able to ask the WG for formal adoption during this IETF meeting, or shortly thereafter.
>> 
>> 
>> 
>> 
>> - WG Milestones:
>>  - Submit AQM recommendations to IESG for publication, obsoleting RFC
>> 2309 (Goal: January 2014)
>>    - draft-ietf-aqm-recommendation is accepted towards this milestone
>>    - http://datatracker.ietf.org/doc/draft-ietf-aqm-recommendation/
>>    - the draft has been updated per comments received
>>    - if the authors are comfortable, a WGLC might be made on the next 
>> revision
>>    - we would like to hear from other authors of RFC 2309 on this 
>> document, if anyone has contacts to them.
>> 
>> 
>>  - Submit AQM algorithm evaluation guidelines to IESG for publication 
>> as Informational (Goal: July 2014)
>>    - An editor team has come forth and is working on this
>>    - A draft should be available for discussion in the London IETF
>>    - We encourage discussion on the list and during the meeting, if this 
>>       draft should be adopted by the working group.
>> 
>>  - Submit first algorithm specification to IESG for publication as 
>> Proposed Standard (Goal: December 2014)
>>    - Since any Proposed Standard algorithm should be in line with the 
>> recommendations and be passable versus the evaluation guidelines, this 
>> milestone is dependend on the progress of the two work items above.
>>    - Currently the only algorithm spec with a complete and active 
>> individual-submission draft is PIE
>> 
>> - Other items:
>>  - draft-pan-aqm-pie is under active work as a proposed algorithm:
>>    http://tools.ietf.org/id/draft-pan-aqm-pie-00.txt, however the draft
>>    has expired and should be refreshed.
>>  - draft-nichols-tsvwg-codel is expired; Dave Taht or others may 
>> revive it and/or describe pairing with FQ/SFQ algorithms:
>>    http://tools.ietf.org/id/draft-nichols-tsvwg-codel-01.txt
>>  - Other algorithm specifications are welcome!
>>    - Though, we are not planning on adopting algorithms until 
>> recommendations and evaluation guidelines are mostly stable
>> 
>> 
>> Richard Scheffenegger
>> 
>> _______________________________________________
>> aqm mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/aqm
> 
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Tue, 15 Apr 2014 12:28:23 +0200

Nicolas KUHN <[email-masked]> writes:

> and realistic HTTP web traffic (repeated download of 700kB). As a reminder,
> please find here the comments of Shahid Akhtar regarding these values:

The Cablelabs work doesn't specify web traffic as simply "repeated
downloads of 700KB", though. Quoting from [0], the actual wording is:

> "Webs" indicates the number of simultaneous web users (repeated
> downloads of a 700 kB page as described in Appendix A of [White]),

Where [White] refers to [1] which states (in the Appendix):

> The file sizes are generated via a log-normal distribution, such that
> the log10 of file size is drawn from a normal distribution with mean =
> 3.34 and standard deviation = 0.84. The file sizes (yi) are calculated
> from the resulting 100 draws (xi ) using the following formula, in
> order to produce a set of 100 files whose total size =~ 600 kB (614400
> B):

And in the main text it specifies (in section 3.2.3) the actual model
for the web traffic used:

> Model single user web page download as follows:
>
> - Web page modeled as single HTML page + 100 objects spread evenly
> across 4 servers. Web object sizes are currently fixed at 25 kB each,
> whereas the initial HTML page is 100 kB. Appendix A provides an
> alternative page model that may be explored in future work.
> 
> - Server RTTs set as follows (20 ms, 30 ms, 50 ms, 100 ms).
> 
> - Initial HTTP GET to retrieve a moderately sized object (100 kB HTML
> page) from server 1.
> 
> - Once initial HTTP GET completes, initiate 24 simultaneous HTTP GETs
> (via separate TCP connections), 6 connections each to 4 different
> server nodes
> 
> - Once each individual HTTP GET completes, initiate a subsequent GET
> to the same server, until 25 objects have been retrieved from each
> server.

Which is a pretty far cry from just saying "repeated downloads of 700
KB" and, while still somewhat bigger, matches the numbers from Google
better in terms of distribution between page sizes and other objects.
And, more importantly, it features the kind of parallelism and
interactions that a real web browser does; which, as Shahid mentioned is
(can be) quite important for the treatment it receives by an AQM.

As such I would be strongly in favour of changing the draft to actually
describe realistic web client behaviour, rather than just summarising it
as "repeated downloads of 700KB".

-Toke

[1] http://www.cablelabs.com/downloads/pubs/PreliminaryStudyOfCoDelAQM_DOCSISNetwork.pdf

[2] https://developers.google.com/speed/articles/web-metrics

---

From: Nicolas KUHN <[email-masked]>
Date: Tue, 15 Apr 2014 15:57:19 +0200

Thank you for detailing the content of the Cable Labs document and where these 700kB come from. 

> as "repeated downloads of 700KB".

I understand that it may be a drastic simplification to just summarise the web client behaviour as only repeated downloads of 700kB. However, the draft may not detail realistic web client behaviour: I believe that it may be out of topic and the draft cannot contain such level of complexity for all the covered protocols/traffic. 
I propose that the following changes: 

Was: 
	- Realistic HTTP web traffic (repeated download of 700kB);
Changed by:
	- Realistic HTTP web page downloads: the tester should at least consider repeated downloads of 700kB - for a more accurate web traffic, a single user web page download [White] may exploited;

What do you think ?

Regards, 

Nicolas

---

From: Dave Taht <[email-masked]>
Date: Tue, 15 Apr 2014 09:30:54 -0700

>> as "repeated downloads of 700KB".

+100.

> What do you think ?

An AQM evaluation guide MUST include evaluations against real traffic
patterns. Period. The
white PLT model was decent; the repeated single flow 700k download
proposal is nuts. (I can certainly see
attempting to emulate DASH traffic, however)

I have further pointed out some flaws in the white PLT model in
previous emails - notably as to the effect of not emulating DNS
traffic - and have been working towards acquiring a reasonable
distribution of DNS hit, miss, and fill numbers to plug into it for
some time. That has required work - work on finding a decent web
benchmark - and work on acquiring statistics that make sense - and
some of that work is beginning to bear fruit. Dnsmasq, for example has
sprouted the ability to collect statistics, we are trying to get the
chrome web page benchmarker working again, and so on. You ignore the
overhead of DNS lookups to your peril. There are other overheads worth
looking at, too...

Similarly I regard testing a correct emulation of bittorrent's
real-world behavior in an AQM'd environment as pretty critical.
[white] was not even close in this respect. (but it was a good first
try!)

Overall I suggest that we also adopt the same tests that other WGs are
proposing for their protocols. rmcat
had a good starter set here:

http://www.ietf.org/proceedings/89/slides/slides-89-rmcat-2.pdf

-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

---

From: David Collier-Brown <[email-masked]>
Date: Tue, 15 Apr 2014 13:19:42 -0400

On 04/15/2014 12:30 PM, Dave Taht wrote:
> proposing for their protocols. rmcat had a good starter set here:

That might be a substantial time-saver in the end, although a time cost
rigfht now...

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
[email-masked]           |                      -- Mark Twain

---

From: "Eggert, Lars" <[email-masked]>
Date: Tue, 15 Apr 2014 22:08:38 +0000

> as "repeated downloads of 700KB".

Fully agree.

And if the 700KB download is left in, please call it something other than "realistic HTTP web traffic", which it's certainly not. Maybe call it "ringtone download" :-)

Lars

---

From: "Akhtar, Shahid (Shahid)" <[email-masked]>
Date: Tue, 15 Apr 2014 22:47:06 +0000

Hi Nicholas,

Please see comments inline.

Best regards,

-Shahid.

________________________________
From: Nicolas KUHN [mailto:[email-masked]]
Sent: Tuesday, April 15, 2014 3:24 AM
To: Akhtar, Shahid (Shahid)
Cc: [email-masked]; Richard Scheffenegger
Subject: Re: [aqm] [AQM Evaluation Guidelines]

Hi Shahid Akhtar, all,

SA: The Cablelabs document as mentioned below has a set of tests for Bulk TCP for uploading traffic, they use a single continuous TCP flow for certain testcases and multiple TCP flows with 20MB or 250KB files repeatedly being sent upstream for other testcases. Their application is upstream traffic which behaves differently from downstream traffic. A Sandvine document on Internet traffic (https://www.sandvine.com/downloads/general/global-internet-phenomena/2013/exposing-the-technical-and-commercial-factors-underlying-internet-quality-of-experience.pdf) shows that real-time entertainment (Netflix, YouTube and others) constitute 68% of downstream traffic, (Figure 9) of which YouTube (progessive download) is at 17% of total traffic. Using repeated transfers of TCP files as suggsted in the Cablelabs work would miss the effect of the random on-off patterns that PD flows generate on the downstream portion. So I would still recommend the use of PD (YouTube) statistics such as above for modeling the Bulk TCP traffic downstream traffic.
"Realistic HTTP web traffic (repeated download of 700kB);"
According to statistics that Google collected from 2010 about web traffic, the average GET file size for web traffic was only 7KB. It is very important to have the size of these files correct since AQMs can help transfers of small files when large file downloads are also occurring. The effect of AQM on a 7KB file would be very different than on a 700KB file. Web traffic also has a complex inter-arrival time relationship between the GET requests as well as the pages that produce these GET requests. However to make the testing simpler, a repeated download of a 7KB file with a Pareto distributed size and Pareto distributed inter-arrival time may be an OK compromise.

SA: I believe Toke has already responded that the CableLabs work mentioned below refers to another document, which uses 25KB for responses to web page GETs. This is hugely better than using repeated 700KB downloads and I am OK with using the model in the other document. I am still concerned that the average size of the reponse maybe less than 25KB, The statistics collected by Google (https://developers.google.com/speed/articles/web-metrics) show an average size of 7KB, but they are 4 years old and I cannot find a more recent number.

The Cablelabs work does not mention any HTTP adaptive streaming (HAS) traffic models - would that be included in the guidelines?
Kind regards,

Nicolas KUHN

On Feb 20, 2014, at 6:48 AM, "Akhtar, Shahid (Shahid)" <[email-masked]<mailto:[email-masked]>> wrote:

Best regards,

Shahid Akhtar.

-----Original Message-----
From: aqm [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Nicolas KUHN
Sent: Friday, February 07, 2014 5:58 AM
To: Richard Scheffenegger
Cc: [email-masked]<mailto:[email-masked]>
Subject: [aqm] [AQM Evaluation Guidelines]

Dear all,

Kind regards,

Nicolas KUHN

On Feb 5, 2014, at 10:01 PM, "Scheffenegger, Richard" <[email-masked]<mailto:[email-masked]>> wrote:

Hi,

- WG Milestones:

- Other items:

Richard Scheffenegger

_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

---

From: Greg White <[email-masked]>
Date: Wed, 16 Apr 2014 15:28:05 +0000

All,

As has been mentioned, CableLabs work on AQM evaluations has been (so far) entirely devoted to evaluating performance for upstream AQM algorithms that would be implemented in a residential cable modem.   While the traffic is, of course, bi-directional, the focus was on identifying realistic test cases that would provide insight on the performance of the algorithms that we were considering (in both an absolute and a relative sense).  For evaluating downstream AQM performance I would definitely add some additional traffic types, and video traffic would be at the front of my mind there.  In order to do that justice, I think you'd want to have a HAS model that incorporates adaptive bitrate capability (but what algorithm?) and estimates QoE using something like the de Vriendt model.  For progressive download, it would seem that the salient metrics would be initial buffering delay and then something related to the number and duration of rebuffering events.

I agree with others that reducing the web model to a single 700kB file download is unrealistic.  Also, for recent work we've increased the object sizes to correspond to a projected 2018 page size (extrapolating from http://httparchive.org/trends.php) of about 7MB.

For our 2018 model of Internet traffic (again, upstream focused) where the user's broadband speed is projected to be 1Gbps down and 200Mbps up, we're using the following 9 traffic loads:

N F1 F2 Fs    W VG C T
1 0  0  -     1 1  0 0
2 1  1  inf   1 1  0 0
3 3  3  50MB  1 1  0 0
4 3  3  50MB  1 1  6 0
5 1  1  19MB* 1 1  0 0
6 3  3  50MB  4 4  0 0
7 3  3  50MB  4 4  6 0
8 5  5  2.5MB 4 4  6 0
9 3  3  50MB  4 1  0 100

where:
N: traffic load index
F1: number of simultaneous FTP uploads with 20ms RTT
F2: number of simultaneous FTP uploads with 100ms RTT
Fs: FTP filesize
W: number of simultaneous web users
VG: number of simultaneous VoIP/gaming sessions
C: CBR data rate (Mbps)
T: number of torrent (LEDBAT) connections
*Filesize and repetition pattern chosen to exercise DOCSIS "powerboost" feature

Since evaluation comes before deployment (by a long lead time in some cases), and, once deployed, algorithms may have a long life, I would encourage the AQM Evaluation Guidelines draft to consider not just what traffic looks like historically, but what we can reasonably predict for the future.

-Greg

From: <Akhtar>, "Shahid (Shahid)" <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 15, 2014 at 4:47 PM
To: Nicolas KUHN <[email-masked]<mailto:[email-masked]>>
Cc: Richard Scheffenegger <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [aqm] [AQM Evaluation Guidelines]

Hi Nicholas,

Best regards,

-Shahid.

Cc: [email-masked]<mailto:[email-masked]>; Richard Scheffenegger

Hi Shahid Akhtar, all,

Nicolas KUHN

Best regards,

Shahid Akhtar.

-----Original Message-----
From: aqm [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Nicolas KUHN
Sent: Friday, February 07, 2014 5:58 AM
To: Richard Scheffenegger
Cc: [email-masked]<mailto:[email-masked]>
Subject: [aqm] [AQM Evaluation Guidelines]

Dear all,

Kind regards,

Nicolas KUHN

Hi,

- WG Milestones:

- Other items:

Richard Scheffenegger

---

From: Dave Taht <[email-masked]>
Date: Wed, 16 Apr 2014 09:25:18 -0700

On Wed, Apr 16, 2014 at 8:28 AM, Greg White <[email-masked]> wrote:
> All,
>
> As has been mentioned, CableLabs work on AQM evaluations has been (so far)
> entirely devoted to evaluating performance for upstream AQM algorithms that
> would be implemented in a residential cable modem.   While the traffic is,
> of course, bi-directional, the focus was on identifying realistic test cases
> that would provide insight on the performance of the algorithms that we were
> considering (in both an absolute and a relative sense).  For evaluating
> downstream AQM performance I would definitely add some additional traffic
> types, and video traffic would be at the front of my mind there.  In order
> to do that justice, I think you'd want to have a HAS model that incorporates
> adaptive bitrate capability (but what algorithm?) and estimates QoE using
> something like the de Vriendt model.

+1

> For progressive download, it would
> seem that the salient metrics would be initial buffering delay and then
> something related to the number and duration of rebuffering events.
>
> I agree with others that reducing the web model to a single 700kB file
> download is unrealistic.  Also, for recent work we've increased the object
> sizes to correspond to a projected 2018 page size (extrapolating from
> http://httparchive.org/trends.php) of about 7MB.

While the trendline is impressively "up":

http://httparchive.org/trends.php?s=All&minlabel=Nov+15+2010&maxlabel=Apr+1+2014

I'm reluctant to do a linear extrapolation of that trendline from 3
years of data. Perhaps
a longer term store, like that of archive.org, could be explored. I'm
not one to predict
a "Peak Web", but the idea of a typical web page load cracking 7MB in
5 years disturbs
me.

>
> For our 2018 model of Internet traffic (again, upstream focused) where the

I do like, very much, trying to come up with projections for what traffic will
look like in 5 years. I'll ask around...

> user's broadband speed is projected to be 1Gbps down and 200Mbps up, we're
> using the following 9 traffic loads:
>
> N F1 F2 Fs    W VG C T
> 1 0  0  -     1 1  0 0
> 2 1  1  inf   1 1  0 0
> 3 3  3  50MB  1 1  0 0
> 4 3  3  50MB  1 1  6 0
> 5 1  1  19MB* 1 1  0 0
> 6 3  3  50MB  4 4  0 0
> 7 3  3  50MB  4 4  6 0
> 8 5  5  2.5MB 4 4  6 0
> 9 3  3  50MB  4 1  0 100
>
> where:
> N: traffic load index

I note that the average RTT in the US is apparently 38ms,
and trending downward at about 1ms/year on wired connections.

Key web sites in major cities (e.g. googles of the world) are
below 5ms. I wish I knew netflix's typical RTT from ISP
co-location facilities... high bandwidth needing providers
will continue to also reduce their RTT's as much as they can.

> W: number of simultaneous web users

I have two mental models for cable internet connectivity - one is
in the home, the other is in small (and increasingly larger) business.

Given the first concept, it is hard to extrapolate as far as your
larger models, and given the second, I'd imagine web, voip,
and videoconferencing traffic to be a much larger percentage
of the overall model.

> VG: number of simultaneous VoIP/gaming sessions

We really need a good model for videconferencing.

I have no idea (except for live streaming delivery) what
the CBR stream is intended to represent. Security cameras backhauled to
Big Brother?

Another item is the amount of "noise" traffic - dns, iot devices reporting
tempurature, weather, etc, being transported.

> T: number of torrent (LEDBAT) connections

Torrent does not behave as per your model; I just had a long meeting
with brahm over that. Presently per "new" torrent you typically see dozens of
sessions "connected", but only 6 or so sessions exchanging data in
both directions,
with a new connection explored and rotated to every 15 seconds or so.

After a file is fully transferred, the same logic remains, but upload only.
Anticipating that a given site would have 16 simultaneous torrents over
a 100 active connections being redistributed seems overlarge, and I imagine
that a typical number is bound much lower by the bittorrent software.

> *Filesize and repetition pattern chosen to exercise DOCSIS "powerboost"

Yes, I'd like not only to have that but to actually have running code in
linux that could emulate it... :)

>
>
> Since evaluation comes before deployment (by a long lead time in some
> cases), and, once deployed, algorithms may have a long life, I would
> encourage the AQM Evaluation Guidelines draft to consider not just what
> traffic looks like historically, but what we can reasonably predict for the
> future.

+1

Now that we are transporting hi resolution video, voice, web, and data
in addition to the traditional email and chat, I have great difficulty assuming
that bandwidth demands will continue to grow as they have, as the new stuff
is mostly "fixed" in its bandwidth needs, rather than dynamic like tcp.

... Unless someone comes up with a way of encapsulating other senses or
some application that truly demands high bandwidth that can work at
multiple RTTs.

There is a lot downward pressure to "do more with less", for example,
as with things like webrtc that are bundled in the browser and don't
require a lot of javascript support, and a lot of caching of core code
that takes place, and a lot of work in reducing RTTs in new protocols
like QUIC.

(but I freely admit I could be wrong and would love to work on projections
 that make sense!)

> -Greg
>
>
> From: <Akhtar>, "Shahid (Shahid)" <[email-masked]>
> To: Nicolas KUHN <[email-masked]>
> Cc: Richard Scheffenegger <[email-masked]>, "[email-masked]" <[email-masked]>
>
> After the discussions in London IETF 89, there has been lots of
> modifications to the draft organisation (as we already advertised on the
> list).
> The authors and I just discussed your points considering the size of the
> files for the bulk transfer and the HTTP traffic.
> As a result, we consider bulk TCP transfers (repeating 5MB file
> transmission) and realistic HTTP web traffic (repeated download of 700kB).
> As a reminder, please find here the comments of Shahid Akhtar regarding
> these values:
>
> The largest type of traffic on the Internet that uses bulk TCP - or large
> files via TCP is progressive download. The average speed of a Youtube video
> is over 3Mbps and it is on average 4.5 minutes long - so the average file
> size is considerably larger than 5MB. We have used an average of 38MB
> assuming 480p video and 4.5 minutes for some of our experiments. The
> inter-arrival time of these files should have a realistic random
> distribution as well as size of the files.
>
> SA: The Cablelabs document as mentioned below has a set of tests for Bulk
> TCP for uploading traffic, they use a single continuous TCP flow for certain
> testcases and multiple TCP flows with 20MB or 250KB files repeatedly being
> sent upstream for other testcases. Their application is upstream traffic
> which behaves differently from downstream traffic. A Sandvine document on
> Internet traffic
> (https://www.sandvine.com/downloads/general/global-internet-phenomena/2013/exposing-the-technical-and-commercial-factors-underlying-internet-quality-of-experience.pdf)
> shows that real-time entertainment (Netflix, YouTube and others) constitute
> 68% of downstream traffic, (Figure 9) of which YouTube (progessive download)
> is at 17% of total traffic. Using repeated transfers of TCP files as
> suggsted in the Cablelabs work would miss the effect of the random on-off
> patterns that PD flows generate on the downstream portion. So I would still
> recommend the use of PD (YouTube) statistics such as above for modeling the
> Bulk TCP traffic downstream traffic.
>
> According to statistics that Google collected from 2010 about web traffic,
> the average GET file size for web traffic was only 7KB. It is very important
> to have the size of these files correct since AQMs can help transfers of
> small files when large file downloads are also occurring. The effect of AQM
> on a 7KB file would be very different than on a 700KB file. Web traffic also
> has a complex inter-arrival time relationship between the GET requests as
> well as the pages that produce these GET requests. However to make the
> testing simpler, a repeated download of a 7KB file with a Pareto distributed
> size and Pareto distributed inter-arrival time may be an OK compromise.
>
> SA: I believe Toke has already responded that the CableLabs work mentioned
> below refers to another document, which uses 25KB for responses to web page
> GETs. This is hugely better than using repeated 700KB downloads and I am OK
> with using the model in the other document. I am still concerned that the
> average size of the reponse maybe less than 25KB, The statistics collected
> by Google (https://developers.google.com/speed/articles/web-metrics) show an
> average size of 7KB, but they are 4 years old and I cannot find a more
> recent number.
>
> The Cablelabs work does not mention any HTTP adaptive streaming (HAS)
> traffic models - would that be included in the guidelines?
>
> Kind regards,
>
> Nicolas KUHN
>
> [0]
> http://www.cablelabs.com/wp-content/uploads/2013/11/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf
>
> On Feb 20, 2014, at 6:48 AM, "Akhtar, Shahid (Shahid)"
> <[email-masked]> wrote:
>
> Thank you for producing this document. I had some (long) comments where I
> have also suggested some solutions/text, hopefully you find them helpful.
> When quoting from the document, I have used quotes and I have listed my
> comments as 1-4.
>
>
> 1. The number of tests recommended per AQM scheme seems to be quite large as
> well as many tests that may be best done together are done independently.
>
> For example, there are different tests for fairness between flows (section
> 3.2.3), level of traffic (mild, medium, heavy) (section 3.2.4), different
> types of bursty traffic (3.2.2) and different network environments (section
> 3.3). In order to obtain a realistic estimate of AQM comparison, many of
> these effects may need to be tested together. For example to increase level
> of traffic (in section 3.2.4) only long term TCP flows of 50s are proposed.
> It would be better to test the level of congestion with realistic traffic
> (and with realistic proportions) as proposed in section 3.2.4. Another
> example would be to test fairness between flows of same type - e.g. long
> term TCP flows when multiple types of TCP flows are present at a AQM
> controlled buffer or test fairness between flows of different types - e.g.
> long term TCP flows and web flows.
>
> I noticed that in section 3.3 different network environments are mentioned.
> I suggest that tests with appropriate realistic traffic be done on each of
> these enviroments with various levels of congestion. One could measure
> fairness from these tests without the need for additional tests. This may
> have the benefit of reducing overall number of tests as well.
>
>
> Access networks - and further split access networks into DSL/FTTH networks
> and Cable access networks. Cable access networks may have much larger number
> of flows than DSL/FTTH since they are shared by a large number of customers
>
> There is another type of network environment - narrowband wireless (mobile)
> networks where the AQM could be placed on the eNodeB, but perhaps this is a
> complicated scenario, and best left for later or in another document.
>
>
> The largest type of traffic on the Internet that uses bulk TCP - or large
> files via TCP is progressive download. The average speed of a Youtube video
> is over 3Mbps and it is on average 4.5 minutes long - so the average file
> size is considerably larger than 5MB. We have used an average of 38MB
> assuming 480p video and 4.5 minutes for some of our experiments. The
> inter-arrival time of these files should have a realistic random
> distribution as well as size of the files.
>
> According to statistics that Google collected from 2010 about web traffic,
> the average GET file size for web traffic was only 7KB. It is very important
> to have the size of these files correct since AQMs can help transfers of
> small files when large file downloads are also occurring. The effect of AQM
> on a 7KB file would be very different than on a 700KB file. Web traffic also
> has a complex inter-arrival time relationship between the GET requests as
> well as the pages that produce these GET requests. However to make the
> testing simpler, a repeated download of a 7KB file with a Pareto distributed
> size and Pareto distributed inter-arrival time may be an OK compromise.
>
> I noticed that adaptive streaming is not modelled as a type of realistic
> traffic. It is probably the most important and largest flow on the Internet
> today - if we look at a recent Sandvine report of Internet traffic, about
> 75% of Internet traffic is real-time entertainment of which the largest
> component is adaptive streaming based video traffic. In order to simulate
> adaptive streaming traffic, one would need a client that changes video
> quality rates. To keep testing simpler, an option may be to generate
> realistic chunk sized files at regular intervals. Average netflix speeds are
> around 2Mbps. Typical chunk sizes range from 2s to 10s, but 4s may be a good
> average. The chunk sizes should be varied with some random distribution as
> is typically observed from real codecs.
>
>
> 4. In section 2.2.5, QoE metrics are mentioned. It may be valuable to spell
> out some standard models or expressions to derive QoE from network
> parameters so that these QoE can be compared between AQMs. In the literature
> there is work that derives closed form expressions for QoE. The following
> are examples and suggestions:
>
> Video - the work by Johan de Vriendt et al , "Model for estimating QoE of
> Video delivered using HTTP Adaptive Streaming", IFIP/IEEE Workshop on QoE
> Centric Management, May 2013 may be useful here. The authors used over 500
> samples to derive their a closed form expressions. The authors derive the
> MOS = alpha*(average bit rate) + beta*(standard deviation of bit rates) +
> sigma*(no of bit rate switches) + gamma and then produce good values for
> alpha, beta, sigma and gamma.
>
> Voice - The work in "Voice quality prediction models and their application
> in VoIP networks" by L. Sun et al has closed form expressions for MOS scores
> for most commercial codes. They derive a long closed form expression with
> parameters a-j for key codecs (and then provide a-j for most commercial
> codecs) based on end-to-end and packet loss.
>
> Key metric for AQMs on access networks may be QoE since it provides and
> end-user view. We have seen that often the absolute goodput does not matter
> as much as the stability of the flow for Video (which tends to be the
> dominant traffic). Larger goodput may provide higher VQ levels for Video,
> but if higher volatility is also produced then that higher VQ level may not
> produce a higher QoE.
>
> On the other hand, best metrics inter data center traffic may be goodput and
> delay since most of that traffic is likely to be inter machine traffic.
>
> Best regards,
>
> Shahid Akhtar.
>
> On the behalf of the contributors to the AQM Evaluation Guidelines, I
> encourage active discussion on the draft that we have written.
> First of all, Wes and I as chairs would like to thank the editors who have
> stepped forward to work on the AQM Evaluation Guideline draft. We are really
> thankful for their burst of efforts in the last couple weeks!
>
> We expect that that a document will be ready for submission into the I-D
> repository well before cutoff, as an individual draft, so that the WG can
> see what the state of thinking is currently. Also, once the document is
> published on datatracker we'd like to encourage active discussion on it.
>
> If the submitted -00 draft is felt to have a fairly complete outline of what
> the current thinking in the WG is, we may be able to ask the WG for formal
> adoption during this IETF meeting, or shortly thereafter.
>
>
>
>
> - WG Milestones:

-- 
Dave Täht

---

From: Greg White <[email-masked]>
Date: Fri, 18 Apr 2014 19:13:37 +0000

>look like in 5 years. I'll ask around...

Yes, it would be interesting to get some other views on this.

>that a typical number is bound much lower by the bittorrent software.

6 sessions * 16 torrents seems to end up pretty close to 100 total
sessions.  Am I missing something? I neglected to mention that the torrent
traffic is limited to 50% of the user's upstream data rate, based on
evidence (like you suggest) that torrent clients by default limit their
traffic, and it is apparently well known that running it unlimited is not
recommended (probably due partly to bufferbloat, but once established, the
mindset might still remain even after bufferbloat is eliminated).

>linux that could emulate it... :)

The code we've released for the ns2 model is available if you want to use
that as a start. Or, I can walk you through the math if you'd rather.

>some application that truly demands high bandwidth that can work at

Yes, I hear you and don't disagree.  On the other hand it is hard to argue
strongly against 30+ years of rock-solid consistent 50% annual increase...
Plus, I'm probably not the only one who can remember at one point thinking
"I'll never fill a 20 MB hard drive, why, that's 20 *million* bytes!"