Subject: [aqm] WGLC on draft-ietf-aqm-eval-guidelines

From: Wesley Eddy <[email-masked]>
Date: Mon, 10 Aug 2015 09:43:42 -0400

As chairs, Richard and I would like to start a 2-week working
group last call on the AQM characterization guidelines:

https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/

Please make a review of this, and send comments to the list or
chairs.  Any comments that you might have will be useful to us,
even if it's just to say that you've read it and have no other
comments.

Thanks!

-- 
Wes Eddy
MTI Systems

---

From: "LAUTENSCHLAEGER, Wolfram (Wolfram)" <[email-masked]>
Date: Thu, 13 Aug 2015 11:58:34 +0000

Hi all,

I read the latest version of the draft, and I found it useful. The draft
addresses a comprehensive range of topics for AQM characterization. What I
am not so happy with, is the description of the corresponding experiments.
Some critical points of my first review
https://mailarchive.ietf.org/arch/msg/aqm/OwPTGmXLpEmCChpgE7ZFqFnnT64 still
persist. I would like to regard these experiments as initial proposals
(which is good to have) that might undergo substantial revision in practice
later on. In general I have the feeling that the combinatorial number of
mandatory experiments is close to infinity. Not only that I doubt this will
ever be done; but who is subsequently going to judge the huge amount of
results?

Here are some minor comments:

Section 2.7 defines goodput/delay scatter plots in two different ways: On
with reference to [HAYE2013], the other definition with reference to
[WINS2014]. I would prefer to have only one definition, namely [WINS2014]. 
- [HAYE2013] depends on a parameter variation across certain range (e.g.
traffic load, or buffer size) that is not defined in most of our
experiments.
- [WINS2014] depends only on randomized replication of otherwise identical
experiments. This should be applicable to any of the evaluation experiments.
(In fact, it is unavoidable anyway.)

Section 4.3: The term "long-lived non application-limited UDP" is somewhat
infinite bandwidth. What the authors probably mean is "long-lived UDP flow
from unresponsive application" to make it clear that no application layer
congestion control is present like in NFS.

Section 2.1: Formula on flow completion time: mismatch of dimensions (Byte
vs. Mbps)

Wolfram Lautenschlaeger

-----Ursprüngliche Nachricht-----
Von: aqm [mailto:[email-masked]] Im Auftrag von Wesley Eddy
Gesendet: Montag, 10. August 2015 15:44
An: [email-masked]
Betreff: [aqm] WGLC on draft-ietf-aqm-eval-guidelines

As chairs, Richard and I would like to start a 2-week working group last
call on the AQM characterization guidelines:

Please make a review of this, and send comments to the list or chairs.  Any
comments that you might have will be useful to us, even if it's just to say
that you've read it and have no other comments.

Thanks!

--
Wes Eddy
MTI Systems

_______________________________________________
aqm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/aqm

---

From: Roland Bless <[email-masked]>
Date: Wed, 19 Aug 2015 00:03:05 +0200

Hi,

> even if it's just to say that you've read it and have no other

"Unfortunately", we (Polina and I) did a thorough review,
which is attached. TL;DR: from our point-of-view
the I-D needs a major revision.

Regards,
 Roland

---

From: Dave Taht <[email-masked]>
Date: Tue, 18 Aug 2015 15:07:43 -0700

> the I-D needs a major revision.

I am so tired of this document that I can hardly bear to read it
again, but I agree with the majority of the comments.

Sometimes I do wish we could do graphics and charts as the IEEE does.

-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 18 Aug 2015 22:38:06 -0400

> Sometimes I do wish we could do graphics and charts as the IEEE does.

We can add any type of graphics that are necessary, they will
just only show up in the PDF version of the RFC, with only
references to the PDF version in the TXT copy.  See, for
instance:
https://tools.ietf.org/pdf/rfc6687.pdf

Are there particular figures that need to be added to this AQM
document to strengthen it?

-- 
Wes Eddy
MTI Systems

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 18 Aug 2015 22:52:47 -0400

> the I-D needs a major revision.

Many thanks for the detailed review.

I think a majority of the comments could be addressed in an update, if
the authors agree.

There were only a couple of the "major issues" that I thought I should
comment on as a co-chair of the WG:

> 3) the overall number of tests and parameter combinations is really
>   high

Are there particular permutations (or classes of permutations) that you
can suggest to remove?  There's a balancing act between including enough
to satisfy people that want to find edge cases and thoroughly
characterize an algorithm, and the desire for a more easily tractable
suite of tests.

> 4) from the discussed end-to-end metrics only latency/goodput metrics
>   are used in the scenarios and for some of the scenarios these metrics
>   are not suitable to show the desired behavior

It would be easier for the editors to improve this if you could suggest
specific metrics to add to specific scenarios, I think.

> 5) some sections in this document (e.g., 7.3, 10, 13) specify requirements
>   for an AQM standard(/draft) and not requirements for a performance
>   evaluation, so these sections should be moved to
[draft-ietf-aqm-recommendation]

That one is now an RFC (7567), so hopefully they're already reflected
if they were critical requirements.

In any case, I agree with you that requirements themselves should not
be conveyed in this document, but rather it should be just aimed at
characterizing algorithm behavior with regard to the requirements
(for ones that are applicable to verification by testing).

-- 
Wes Eddy
MTI Systems

---

From: Roland Bless <[email-masked]>
Date: Fri, 21 Aug 2015 22:18:50 +0200

Hi Wes,

> characterize an algorithm, and the desire for a more easily tractable

Right now, I can't, it was merely an observation...

> specific metrics to add to specific scenarios, I think.

We included some considerations in the comments to the individual
sections already. As a general notice, The document could benefit
by cross-referencing the scenarios against the documents referenced in
Major Issue 6.  Regarding particular metrics, suggesting one
would require exact understanding of particular test goals, proper
argumentation, and thus more time than LC deadline allowed.

Regards,
 Roland