Subject: Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

From: Polina Goltsman <[email-masked]>
Date: Mon, 1 Feb 2016 17:30:42 +0100

Dear Wesley,

I apologize for the long delay. Below are responses to your emails 
regarding the draft-ietf-aqm-eval-guidelines.

------------------------------------------------------------------------

On 22.01.16 15:50, Wesley Eddy wrote:
> On 12/7/2015 7:32 AM, Polina Goltsman wrote:
>>
>> In the abstract, the document says that it describes characterization 
>> guidelines for an AQM proposal, to decide whether it should be 
>> adopted by the AQM WG. The WG currently has two AQMs 
>> (dropping/marking policy) in last call. Did someone evaluate these 
>> AQMs according to the specified guidelines?
>>
>
>
> In my opinion, for "standardization" it would be good to have crisp 
> guidelines.  For simply publishing RFCs that are experimental (not 
> standardized), it is much less important.
>
I have a question regarding "standardization". As I understand, no 
evaluation was performed based on these guidelines. Since the intended 
status of draft-ietf-aqm-eval-guidelines is "informational" too, this is 
also not an issue. Am I correct?
(see also an email from Dave Täht, quoted at the end of this email)
>
>> Moreover, it seems to me that the WG is about to conclude. What 
>> exactly is the purpose of standardizing this document then ?
>>
>
>
> It's definitely true that the activity level has been low, and so 
> we're trying to wrap up the outstanding work items before it flattens 
> completely.  This document is not proposed for standards track, only 
> "Informational".  The current goal as I see it (with co-chair hat on) 
> is to see if we have rough consensus that this is a useful 
> contribution to the community going forward, and that any small issues 
> can be addressed.
>
> As I understood your review (please correct if I'm wrong), a main 
> issue you see is that there isn't specific guidance about numeric 
> values or ranges to use in evaluations?  Nicolas explained why this 
> might be a bit difficult to do in general (and specific values 
> published in 2016 may be moot by 2018), so as I understand, one of the 
> limitations of this document is that it is only going to be able to 
> provide general descriptive guidance in terms of what kinds of tests 
> to run, and what types of things to look for, but not prescriptive 
> values and conditions to be met.  Do you think that's okay, even 
> though it's probably less directly useful than if there were specific 
> values

Yes, my concern was that the document did not contain enough detailed 
description of a test setup (capacities, RTT, etc). However, as Nicolas 
explained, the intention is that the test setup is selected by a tester  
for some specific environment. The results of an evaluation will then 
provide guidance for selecting an AQM for this specific environment. 
 From my point of view this is completely reasonable approach, however 
IMO  *the document should state this explicitly*.

 From answers to our (mine and Roland's) reviews we found out that the 
goals of the document are:

>We  believe that

>"

 >  AQM schemes need to be compared against both performance and

 >  deployment categories.

>"

(this one is present in the document in section 4.3)

>The main idea was to present the  available and useful metrics early

>in the document, so that the  tester can choose between them those that

>are of interest for the considered  scenario.

and finally

>The  scope of this document is to present how to assess the performance of 
AQMs – without focusing on a specific context ...
>...
>The unspecified  parameters do not require specific knowledge on AQMs – the unspecified 
parameters are (1) the RTT or the capacities, which are related to the 
network underneath; (2) and the characteristics of the traffic 
generation, which is related to the >traffic that is run between clients 
and servers.

(which I understand as stated above, correct me if I am wrong)
(this goal will be referred as [*] below)

Section 1.2 is intended to specify the goals of the document. I am not 
sure that everybody will understand the above stated goals from the text 
of Section 1.2. and the document would benefit if these goals were 
stated more explicitly (Well, I could not understand them without 
explanations from the authors).

------------------------------------------------------------------------

On 22.01.16 19:32, Klatsky, Carl wrote:
>
> Wes and all,
>
> My comment is in regard to Polina’s comment “The WG currently has two 
> AQMs (dropping/marking policy) in last call. Did someone evaluate 
> these AQMs according to the specified guidelines?”.  As I read over 
> draft-ietf-aqm-eval-guidelines, I did not think the objective of this 
> memo was to arrive at consensus to select one specific AQM.  I thought 
> the objective was to publish guidelines for implementers & service 
> providers on how they can select an AQM that is best for their 
> environment. And further that both AQMs in last call would complete 
> the process as valid AQMs for implementers & service providers as 
> available to use in their environment, with the guidelines helping 
> them to decide which is best for them. Some may chose PIE, some may 
> chose FQ_CODEL.  And possibly other future AQMs would go through the 
> IETF process for publication, and that would be an additional option 
> for implementers & service providers to evaluate according to the 
> guidelines as best for their environment.
>
> Is my understand of draft-ietf-aqm-eval-guidelines correct?
>
> Regards,
>
> Carl Klatsky
>
> Comcast
>
Again,  this is not what the document says. The document says (several 
times) that it is intended to be used by the AQM WG to evaluate 
proposals. If this (guidelines for the WG) is not the intended goal, 
then it is probably a good idea to remove this goal from the document.

------------------------------------------------------------------------

Next topic. Assuming that my understanding of the goals of the document 
(Section 1.2 + above) are correct, I find the following a little bit 
confusing.

1) The document does contain specific numbers in some scenarios, which, 
considering the goal [*], I find quite confusing ... . The RTTs in range 
[5ms-560ms] in Section 6 are, according to some TCP evaluation 
guidelines, currently seen RTTs in the Internet, where the latter is for 
satellite links, while the capacities "between 10Mbps an 100Mbps" in 
Section 8 are probably available Ethernet rates in range of available 
ADSL speeds. None of this makes sense if I decide to evaluate an AQM for 
a gigabit LAN in an office. So I think it might be a good idea to either 
introduce these numbers as "for example" and mentioning the setups for 
which these values are applicable (e.g., "for example the RTT range for 
general Internet is ...").

2) The document says:

> The guidelines also discuss methods to understand the
>  various aspects associated with safely deploying and operating the
>  AQM scheme.

I can't understand whether the document is intended to include 
requirements for an AQM specification document or not. Sections 8.3, 11, 
and 12 include requirements ("SHOULD") for an AQM proposal.

Since RFC7567 does not provide a recommendation in form of a list of 
what should be present in an AQM specification document, it might be a 
good idea to include this list in the eval-guidelines. However I suggest 
that these requirements be present as a separate section in the 
document, since AQM authors and testers are likely to be different people.

Moreover, since Sections 8.3, 11, and 12 include requirements for AQM 
proposals, what part of the document evaluates aspects associated with 
safe deployment of AQM? In particular:

Section 8.3 (stability analysis): an AQM proposal may or may not cover 
the particular test setup (capacity, RTT, etc), so the tester may need 
to specify whether their specific parameters are covered by an AQM 
proposal, and if not, perform the required analysis themselves.

Section 11 (implementation): AQM proposal SHOULD provide pseudocode. If 
the selected test setup includes specific device, then discussing 
device-specific implementation could be a part of an evaluation result. 
An AQM proposal cannot possibly discuss implementation on every 
available hardware.

(On the other hand, Section 12 requires to specify AQM parameters if 
they are not default, so it can be considered as an evaluation of safe 
deployment.)

3) Section 9.2 includes comments about DNS requests and SYN packets. I 
understand that this is a valid concern for an evaluation, but the 
scenarios in Section 9.2 do not evaluate this. I think it might be a 
good idea to provide a separate scenario with appropriate traffic mix 
and metrics, or, as a simpler solution, to discuss DNS and SYN packets 
in Section 2.2 - flow start-up metric.

4)  The document currently suggests using TCP congestion control 
according to RFC5681 with SACK. Is it even possible to use pure RFC5681 
on a real end-system? I think it might be a good idea to instead require 
that an implementation of a network stack on end hosts is disclosed in 
the evaluation results. For example, implementation of loss recovery can 
affect results and it may be not configurable: if I understand 
correctly, both proportional rate reduction (rfc6937) and a replacement 
for TCP ABC  cannot be turned off on Linux.

(the issue of network stack implementation on end-system is raised in 
*Yee-Ting Li, Douglas Leith, and Robert ******N. Shorten. 2007. 
Experimental evaluation of TCP protocols for****high-speed networks.)*

P.S. Is RFC5681 really NewReno and not Reno?

------------------------------------------------------------------------

On 22.01.16 15:52, Wesley Eddy wrote:
> With my co-chair hat on, I agree with this sentiment that it will 
> never be perfect.    To my knowledge, there aren't other resources 
> that have better information and guidance than this document when it 
> comes to the topic of characterizing AQMs.  The goal has been to 
> produce something useful to the community, compared to the current 
> guidance available (which is basically nothing), so I'm hopeful that 
> we can get a version of it with rough consensus to go forward.
I am not very familiar with the following topic, so if the following 
concerns are rubbish, please ignore them.

The sections for evaluating AQMs' performance (as opposed to deployment 
safety) can be divided into two groups: scenarios in sections 5 (except 
for 5.3), 6, 8, and 9.2 evaluate long-lived TCP flows, while scenarios 
in section 7 and 9.1 evaluate  traffic mixes including multimedia and 
web flows. There are several guidelines for evaluation of TCPs, some of 
which suggest evaluating TCP with AQM deployed. For eval-guidelines, 
they can be used as a source for additional information on how to choose 
and calculate metrics for the first group of scenarios. For the second 
group of scenarios, eval-guidelines  includes suggestion on how to 
generate "web-like" traffic and does not include suggestions on how to 
generate multimedia traffic. For both scenarios the document suggests 
using delay-throughput graph as well as "Metrics such as end-to-end 
latency, jitter, flow completion time MAY be generated" without 
suggesting particular metric for particular type of traffic (to my 
understanding, FCT does not make sense for a video stream and jitter for 
web flow).

Is there literature on how to generate and evaluate multimedia streams, 
including what metrics to choose and how to calculate them. Although 
there are some references in Section 2 (e.g., interval between 
consecutive losses), I think that the readers could benefit from 
additional references on the topic. Do I understand correctly that the 
universally accepted metric for web-flows is flow completion time?

------------------------------------------------------------------------

On 22.01.16 19:57, Dave Täht wrote:
> The evaluation guide needs to be executable, or rather, turned into
> public code and a standardized benchmark suite. Eventually.
>
> Iteratively, flent has many tests that have proven valuable and quite a
> few that have not.
>
> The tests in the aqm guide, need to be created, iterated on, and examined.
Totally agree. Implementing the tests for flent could be a perfect 
analogue for an "interoperability" test for these guidelines. Without 
any available result it is hard to say, whether the guidelines are 
useful or not.

Regards,
Polina