Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-ca-bench-meth-02.txt

From: Al Morton <[email-masked]>
Date: Wed, 01 Aug 2012 21:36:40 -0400

At 02:21 PM 7/31/2012, Tom Alexander wrote:
>Hello, Mike,
>
>Thanks for incorporating my feedback. The -03 draft looks very good 
>and I have no further comments.
>
>Best regards,
>
>- Tom

Mike and Sarah,

I read the draft on the flight-out, and have the following comments.
(these are comments on the version 02 text, sorry I couldn't keep
up with your really fast update...)

Al (as a participant)

-=-=-=-=-

end of Section 1:
    ...
    The nature of application-driven networks is such that a single
    dropped packet inherently changes the input stimulus from a network
    perspective.  While application flows will be specified in great
    detail, it simply is not practical to require totally repeatable
    input stimulus.

It seems that the input stimulus *should* be repeatable when there
is no loss.  Whether the measured results are repeatable or not depends
on the DUT.

When there is loss (in overload) the input will change (TCP connections
will react, etc.), and the offered load will change thus the measured
result will change. Therefore several trials at the same conditions will
be needed along with averaging.

-----

Section 2 Scope, 2nd para.
...
    None of the metrics captured through this methodology are specific to
    a device and the results are DUT implementation independent.
Suggest
    a device and the format of the results are DUT implementation independent.
                 ^^^^^^^^^^^^^^
Question of this sentence:
    If this wasn't what you were thinking, then I don't understand what
we are testing - the results must be dependent on the DUT implementation,
but the design of the test must not be.

Further down, para 3:
...
          While this list may become obsolete,
    these are a subset of devices that fall under this scope of testing.
s/of devices/of all devices/

-------

In 3.7.3.  TCP Stack Considerations,

It seems to me that the scenario of DUT as TCP Middlebox is a likely
candidate for these methods, and probably should be mentioned explicitly
here (since TCP options and other parameters might change and affect
the testing - these facts need to be measured and noted).

-------

In 4.1.1.  Objective

    To determine the maximum rate through which a device is able to
s/through/at/
    establish and complete application flows as defined by
    draft-ietf-bmwg-ca-bench-term-00.
Is this under loss-less conditions? failure-less conditions?

-------

In  4.1.3.  Procedure

...
    This procedure MAY be repeated any reasonable number of times with
    the results being averaged together.
and the Standard Deviation SHOULD also be reported (I think)

------

---

From: Chuck McAuley <[email-masked]>
Date: Thu, 2 Aug 2012 16:37:00 +0000

Al,

There are other things that can impact application traffic behavior beyond
a dropped packet.

Off the top of my head:
1. Out of order packets from load balanced connections
2. Layer 7 Content Manipulation
3. Higher average latency of one stateful connection over another.  IE one
transaction can get "ahead" at the cost of another, altering all traffic
behavior after that point. This is similar to #1.

Chuck McAuley

---

From: Al Morton <[email-masked]>
Date: Thu, 02 Aug 2012 12:45:31 -0400

At 12:37 PM 8/2/2012, Chuck McAuley
wrote:

There are other things that can impact application traffic behavior
beyond

a dropped packet.

Off the top of my head:

3. Higher average latency of one stateful connection over
another.  IE one

transaction can get "ahead" at the cost of another,
altering all traffic

Good point Chuck, it seems that we only have

nominal control over #3 in a test set-up.

Worth discussing the implications on input stream

repeatability a bit, hopefully today...

Al