Subject: [ippm] initial comments on draft-constantine-tcp-throughput-tm-02

From: Al Morton <[email-masked]>
Date: Thu, 18 Mar 2010 15:37:58 -0400

Hi Barry and all,

I'm sharing a few comments on this draft as they occur to me,
and they'll be spread out over several messages as interrupts
ebb and flow.

It seems that there's a bit of intro material worth sorting-out,
but I think it will help to understand the throughput & capacity
testing landscape as IPPM considers this new work.

more to follow,
Al

>1. Introduction
>
>    Even though RFC2544 was meant to benchmark network equipment

in the lab-only (a key point to include)

>    and
>    used by network equipment manufacturers (NEMs),

some

>network providers
>    have used it to benchmark operational networks in order to
>    provide SLAs (Service Level Agreements) to their business customers.

Comment:
This is both unfortunate and true.  However, *some* test equipment
manufacturers simply put RFC2455 in test devices (hand-held), while others
enhanced the methodology in ways that are unknown to me and probably
most of the standards community.  Thus, methods deployed for raw
capacity testing are/were "like-RFC2544" more or less.

>    Network providers are coming to the realization that RFC2544 testing
>    and TCP layer testing are required to more adequately ensure end-user
>    satisfaction.

I see interest in both raw capacity testing (without TCP's restrictions)
and TCP (this drafting effort and others making measurements using a
variety of methods).

>    Therefore, the network provider community desires to measure network
>    throughput performance at the TCP layer.

Maybe so, but only if the results will pass the test of a
valid metric, and I think a rather wide spectrum of methods
are actually wanted.

>    Measuring TCP throughput provides

Suggest instead:
A standardized metric of TCP Throughput would provide ...

>    ... a meaningful measure with respect to the end user's
>    application SLA (and ultimately reach some level of TCP testing
>    interoperability which does not exist today).

---

From: Vinayak Hegde <[email-masked]>
Date: Wed, 24 Mar 2010 01:59:54 +0530

Hi,

I am adding a couple more comments to the draft:

1. TCP throughput testing has a lot of variable and intermediate hops
(which can affect the results). Barry has done a good job of narrowing
down and pinning the scope so that the draft itself is very useful.
But given these complexity of the task, it would be nice to see a
minimal list of parameters that should be added to each of the
benchmark tests so that there can be atleast some meaningful
comparison of results across networks / network conditions. Examples
could include:
     1 TCP window size
     2. MSS
     3. Number of connections
     4. Congestion algorithm used at the end-points
     5. RTT (steady state as the draft notes).
     6. Various stack tunables (such as the sysctl output for network params)

2. The range of variance in some of the above parameters
    1. Most notably latency - RTT on a live network
    2. The effect of the mix of traffic (and the ramping curve of
stateless traffic such as UDP) on the latency.

One last point was whether there might be merit in instrumenting any
of the intermediate hops for checking congestion. Would it be possible
to share code that does some of the measurement and generates streams
for testing (as a reference implementation).

-- Vinayak

On Fri, Mar 19, 2010 at 1:07 AM, Al Morton <[email-masked]> wrote:
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>