Subject: Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

From: Wesley Eddy <[email-masked]>
Date: Fri, 22 Jan 2016 09:50:32 -0500

On 12/7/2015 7:32 AM, Polina Goltsman wrote:
>
> In the abstract, the document says that it describes characterization 
> guidelines for an AQM proposal, to decide whether it should be adopted 
> by the AQM WG. The WG currently has two AQMs (dropping/marking policy) 
> in last call. Did someone evaluate these AQMs according to the 
> specified guidelines?
>

In my opinion, for "standardization" it would be good to have crisp 
guidelines.  For simply publishing RFCs that are experimental (not 
standardized), it is much less important.

> Moreover, it seems to me that the WG is about to conclude. What 
> exactly is the purpose of standardizing this document then ?
>

It's definitely true that the activity level has been low, and so we're 
trying to wrap up the outstanding work items before it flattens 
completely.  This document is not proposed for standards track, only 
"Informational".  The current goal as I see it (with co-chair hat on) is 
to see if we have rough consensus that this is a useful contribution to 
the community going forward, and that any small issues can be addressed.

As I understood your review (please correct if I'm wrong), a main issue 
you see is that there isn't specific guidance about numeric values or 
ranges to use in evaluations?  Nicolas explained why this might be a bit 
difficult to do in general (and specific values published in 2016 may be 
moot by 2018), so as I understand, one of the limitations of this 
document is that it is only going to be able to provide general 
descriptive guidance in terms of what kinds of tests to run, and what 
types of things to look for, but not prescriptive values and conditions 
to be met.  Do you think that's okay, even though it's probably less 
directly useful than if there were specific values?

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 22 Jan 2016 09:52:02 -0500

On 12/14/2015 4:09 AM, Kuhn Nicolas wrote:
>
> Dear Polina, all,
>
> 1)The explanation was mostly on why parameters are not specified in 
> the document because the words unspecified was in a bold font in your 
> email ; this made me think that this was your concern about the document.
>
> 2)I think it is not an issue to have some parameters specified in the 
> document.
>
> You do not “have to” write a longer review. Do it  only if you think 
> it is necessary and helpful in having a better document J.
>
> As Gorry said at IETF93, “this doc will never be perfect”[1].
>

With my co-chair hat on, I agree with this sentiment that it will never 
be perfect.    To my knowledge, there aren't other resources that have 
better information and guidance than this document when it comes to the 
topic of characterizing AQMs.  The goal has been to produce something 
useful to the community, compared to the current guidance available 
(which is basically nothing), so I'm hopeful that we can get a version 
of it with rough consensus to go forward.

Please do correct me if I'm wrong about this though.

---

From: "Klatsky, Carl" <[email-masked]>
Date: Fri, 22 Jan 2016 18:32:15 +0000

Wes and all,

My comment is in regard to Polina's comment "The WG currently has two AQMs (dropping/marking policy) in last call. Did someone evaluate these AQMs according to the specified guidelines?".  As I read over draft-ietf-aqm-eval-guidelines, I did not think the objective of this memo was to arrive at consensus to select one specific AQM.  I thought the objective was to publish guidelines for implementers & service providers on how they can select an AQM that is best for their environment.  And further that both AQMs in last call would complete the process as valid AQMs for implementers & service providers as available to use in their environment, with the guidelines helping them to decide which is best for them.  Some may chose PIE, some may chose FQ_CODEL.  And possibly other future AQMs would go through the IETF process for publication, and that would be an additional option for implementers & service providers to evaluate according to the guidelines as best for their environment.

Is my understand of draft-ietf-aqm-eval-guidelines correct?

Regards,

Carl Klatsky
Comcast

From: aqm [mailto:[email-masked]] On Behalf Of Wesley Eddy
Sent: Friday, January 22, 2016 9:51 AM
To: [email-masked]; Polina Goltsman
Subject: Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

In the abstract, the document says that it describes characterization guidelines for an AQM proposal, to decide whether it should be adopted by the AQM WG. The WG currently has two AQMs (dropping/marking policy) in last call. Did someone evaluate these AQMs according to the specified guidelines?

In my opinion, for "standardization" it would be good to have crisp guidelines.  For simply publishing RFCs that are experimental (not standardized), it is much less important.

Moreover, it seems to me that the WG is about to conclude. What exactly is the purpose of standardizing this document then ?

It's definitely true that the activity level has been low, and so we're trying to wrap up the outstanding work items before it flattens completely.  This document is not proposed for standards track, only "Informational".  The current goal as I see it (with co-chair hat on) is to see if we have rough consensus that this is a useful contribution to the community going forward, and that any small issues can be addressed.

As I understood your review (please correct if I'm wrong), a main issue you see is that there isn't specific guidance about numeric values or ranges to use in evaluations?  Nicolas explained why this might be a bit difficult to do in general (and specific values published in 2016 may be moot by 2018), so as I understand, one of the limitations of this document is that it is only going to be able to provide general descriptive guidance in terms of what kinds of tests to run, and what types of things to look for, but not prescriptive values and conditions to be met.  Do you think that's okay, even though it's probably less directly useful than if there were specific values?

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 22 Jan 2016 13:47:06 -0500

On 1/22/2016 1:32 PM, Klatsky, Carl wrote:
>
> Wes and all,
>
> My comment is in regard to Polina’s comment “The WG currently has two 
> AQMs (dropping/marking policy) in last call. Did someone evaluate 
> these AQMs according to the specified guidelines?”.  As I read over 
> draft-ietf-aqm-eval-guidelines, I did not think the objective of this 
> memo was to arrive at consensus to select one specific AQM.  I thought 
> the objective was to publish guidelines for implementers & service 
> providers on how they can select an AQM that is best for their 
> environment. And further that both AQMs in last call would complete 
> the process as valid AQMs for implementers & service providers as 
> available to use in their environment, with the guidelines helping 
> them to decide which is best for them. Some may chose PIE, some may 
> chose FQ_CODEL.  And possibly other future AQMs would go through the 
> IETF process for publication, and that would be an additional option 
> for implementers & service providers to evaluate according to the 
> guidelines as best for their environment.
>

Yes, you're correct!  My assumption is that someone like a service 
provider would have an idea of some of the ranges of values (rates, 
delays, asymmetries, etc) appropriate for their environment, and would 
be able to use the evaluation guidelines effectively.

I do also (personally) think that if there's a desire to go 
standards-track (rather than just experimental) with AQM algorithms, 
that having a fairly explicit evaluation of the algorithms with regard 
to the guidelines would be very helpful, exactly as Polina asked about.  
But I don't think this has really happened, and don't think it's 
necessary at all for experimental RFCs.

---

From: Dave Täht <[email-masked]>
Date: Fri, 22 Jan 2016 10:57:46 -0800

The evaluation guide needs to be executable, or rather, turned into
public code and a standardized benchmark suite. Eventually.

Iteratively, flent has many tests that have proven valuable and quite a
few that have not.

The tests in the aqm guide, need to be created, iterated on, and examined.

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 22 Jan 2016 19:17:35 +0000

On Jan 22, 2016, at 10:47 AM, Wesley Eddy <[email-masked]> wrote:
> I do also (personally) think that if there's a desire to go standards-track (rather than just experimental) with AQM algorithms, that having a fairly explicit evaluation of the algorithms with regard to the guidelines would be very helpful, exactly as Polina asked about.  But I don't think this has really happened, and don't think it's necessary at all for experimental RFCs.

As I recall the discussion, we decided up front that since there is no interoperability requirement among AQM algorithms (the requirement is that they interoperate well with TCP and UDP based applications; the AQM algorithms don' actually talk to each other, and the point is to drop or mark at the right rate and with the right pattern to encourage transport layer sessions to behave well), we didn't need to recommend a single AQM algorithm for all equipment or all uses. What we did need to do was identify some AQM algorithms that actually worked, and give guidance to the vendors and operators on their use.

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 22 Jan 2016 15:25:35 -0500

> As I recall the discussion, we decided up front that since there is no interoperability requirement among AQM algorithms (the requirement is that they interoperate well with TCP and UDP based appli…

I agree with all of this.  The characterization guidelines are aimed at 
helping to identify the AQM algorithms that actually work, or cases 
where they don't work as well (i.e. where some harmful or unintended 
consequence results).