Subject: [ippm] TCP Throughput Testing Methodology draft

From: Ed Gastle <[email-masked]>
Date: Thu, 24 Jun 2010 15:00:26 -0400

Some comments on the TCP Throughput Testing Methodology draft:

   - The acronym MSS is referred to in the Introduction but not defined
   until 3.1 - suggest it be defined when the acronym is first used.
   - The Introduction seems to too narrowly focus the draft to the "network
   provider community". It seems this test methodology would also be useful in
   a LAN/Campus (private managed network) environment.
   - The Introduction (last paragraph)  as well as section 3 recommends that
   RFC2544 tests be performed as a precursor to conducting TCP throughput
   tests. The stated purpose of this step is to "verify the integrity of the
   network before conducting TCP testing." What network integrity
   characteristics that affect TCP throughput tests should be clearly spelt out
   in the introduction. It seems the main parameter is frame loss and a stable
   delay (absence of jitter). But what level of frame loss and jitter that are
   acceptable are not defined. In addition there are many ways to test loss
   without performing RFC2544 tests. With this in mind it seems recommending
   RFC2544 is too restrictive a requirement. My main point here is the network
   performance parameters that must be in place to make performing TCP
   throughput testing accurate should be more clearly defined and explained.
   The method of measuring the network for these parameters should be left
   open.
   - The BDP calculation in section 3.2 should have the units of measure
   defined (BDP in Bits, RTT in ms, Bandwidth in bps)
   - Although there are many factors at play (TCP Window in particular) it
   seems counter-intuitive that maximum achievable TCP throughput in some cases
   INCREASES with delay. Referring to Table 3.2 a link with 100Mb of bandwidth
   and 2 ms of delay has greater TCP throughput than a link with 1 ms of delay.
   Are the calculations here correct?

Ed Gastle

---

From: "Gilles Forget" <[email-masked]>
Date: Fri, 25 Jun 2010 00:28:32 -0400

Bonjour Ed,

Thank you very much for all of your very constructive comments.

About RFC2544, I would just like to add that it was never meant for network
testing. This RFC was written with lab testing in mind only, right at the
beginning in section 3, it talks about DUTs (Devices Under Test).

Plus the complete title is : Benchmarking Methodology for Network
Interconnect Dev. However, it's a fact of life that it has been used for at
least 10 years now, not only for lab testing but for complete end to end
network tests and that is one of the reasons why we have decided to mention
it in the draft. I know by field experience that it leads to finger pointing
situations between business customers and network providers. On the other
hand, it's hard for us to be very precise about how many frame loss or how
much jitter is acceptable without entering the SLA (Service Level Agreement)
area. SLAs are not only based on technical variables, they are also based on
business and political variables that we can not define in the present
document.

Concerning your last comment, you are absolutely right, there is an error in
Table 3.2. The error comes from versions 5 of Excel spreadsheets and I have
corrected it in the calculation formula of version 6.

At 100M, the theoretical TCP throughput should be 90.699 Mbps for all of the
3 calculated RTTs because we reach the maximum frame per seconds at all 3
delays.

However, we have to be aware that it is still possible to achieve a better
TCP throughput in some cases when delay increases because it permits more in
flight bytes between sites.

Thanks,

Gilles Forget, CCNP.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ed
Gastle
Sent: June-24-10 3:00 PM
To: [email-masked]
Subject: [ippm] TCP Throughput Testing Methodology draft

*	The acronym MSS is referred to in the Introduction but not defined
until 3.1 - suggest it be defined when the acronym is first used.
*	The Introduction seems to too narrowly focus the draft to the
"network provider community". It seems this test methodology would also be
useful in a LAN/Campus (private managed network) environment. 
*	The Introduction (last paragraph)  as well as section 3 recommends
that RFC2544 tests be performed as a precursor to conducting TCP throughput
tests. The stated purpose of this step is to "verify the integrity of the
network before conducting TCP testing." What network integrity
characteristics that affect TCP throughput tests should be clearly spelt out
in the introduction. It seems the main parameter is frame loss and a stable
delay (absence of jitter). But what level of frame loss and jitter that are
acceptable are not defined. In addition there are many ways to test loss
without performing RFC2544 tests. With this in mind it seems recommending
RFC2544 is too restrictive a requirement. My main point here is the network
performance parameters that must be in place to make performing TCP
throughput testing accurate should be more clearly defined and explained.
The method of measuring the network for these parameters should be left
open.
*	The BDP calculation in section 3.2 should have the units of measure
defined (BDP in Bits, RTT in ms, Bandwidth in bps)
*	Although there are many factors at play (TCP Window in particular)
it seems counter-intuitive that maximum achievable TCP throughput in some
cases INCREASES with delay. Referring to Table 3.2 a link with 100Mb of
bandwidth and 2 ms of delay has greater TCP throughput than a link with 1 ms
of delay. Are the calculations here correct?

Ed Gastle

---

From: Barry Constantine <[email-masked]>
Date: Sat, 26 Jun 2010 03:51:29 -0700

Ed,

- Regarding the applicability to enterprise network groups and network providers, this is a very good point.  We have been focused upon network providers since this methodology is envisioned as a qualification / turn-up test, but agree it is equally applicable to enterprise groups.

- Al and others made similar comments relating to other means besides RFC-2544 to pre-qualify the network; I added some text to broaden the scope of the functional network testing; obviously more clarification is required in this area.

- I received the updates from Gilles for the BDP calculations.

These are all very good comments and will be incorporated into the next draft version which will be published prior to Maastricht.

Thanks,
Barry

Principal Member of Technical Staff

JDSU Communication Test (formerly Acterna)
Emerging Markets and Technology Research
One Milestone Center Court
Germantown, MD 20876
(W) 240-404-2227
(C) 301-325-7069
________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Gilles Forget
Sent: Friday, June 25, 2010 12:29 AM
To: 'Ed Gastle'; [email-masked]
Subject: Re: [ippm] TCP Throughput Testing Methodology draft

Bonjour Ed,

About RFC2544, I would just like to add that it was never meant for network testing. This RFC was written with lab testing in mind only, right at the beginning in section 3, it talks about DUTs (Devices Under Test).
Plus the complete title is : Benchmarking Methodology for Network Interconnect Dev. However, it's a fact of life that it has been used for at least 10 years now, not only for lab testing but for complete end to end network tests and that is one of the reasons why we have decided to mention it in the draft. I know by field experience that it leads to finger pointing situations between business customers and network providers. On the other hand, it's hard for us to be very precise about how many frame loss or how much jitter is acceptable without entering the SLA (Service Level Agreement) area. SLAs are not only based on technical variables, they are also based on business and political variables that we can not define in the present document.

Concerning your last comment, you are absolutely right, there is an error in Table 3.2. The error comes from versions 5 of Excel spreadsheets and I have corrected it in the calculation formula of version 6.
At 100M, the theoretical TCP throughput should be 90.699 Mbps for all of the 3 calculated RTTs because we reach the maximum frame per seconds at all 3 delays.
However, we have to be aware that it is still possible to achieve a better TCP throughput in some cases when delay increases because it permits more in flight bytes between sites.

Thanks,
Gilles Forget, CCNP.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ed Gastle
Sent: June-24-10 3:00 PM
To: [email-masked]
Subject: [ippm] TCP Throughput Testing Methodology draft

 *   The acronym MSS is referred to in the Introduction but not defined until 3.1 - suggest it be defined when the acronym is first used.
 *   The Introduction seems to too narrowly focus the draft to the "network provider community". It seems this test methodology would also be useful in a LAN/Campus (private managed network) environment.
 *   The Introduction (last paragraph)  as well as section 3 recommends that RFC2544 tests be performed as a precursor to conducting TCP throughput tests. The stated purpose of this step is to "verify the integrity of the network before conducting TCP testing." What network integrity characteristics that affect TCP throughput tests should be clearly spelt out in the introduction. It seems the main parameter is frame loss and a stable delay (absence of jitter). But what level of frame loss and jitter that are acceptable are not defined. In addition there are many ways to test loss without performing RFC2544 tests. With this in mind it seems recommending RFC2544 is too restrictive a requirement. My main point here is the network performance parameters that must be in place to make performing TCP throughput testing accurate should be more clearly defined and explained. The method of measuring the network for these parameters should be left open.
 *   The BDP calculation in section 3.2 should have the units of measure defined (BDP in Bits, RTT in ms, Bandwidth in bps)
 *   Although there are many factors at play (TCP Window in particular) it seems counter-intuitive that maximum achievable TCP throughput in some cases INCREASES with delay. Referring to Table 3.2 a link with 100Mb of bandwidth and 2 ms of delay has greater TCP throughput than a link with 1 ms of delay. Are the calculations here correct?
Ed Gastle

---

From: <[email-masked]>
Date: Sun, 27 Jun 2010 15:12:31 -0400

There is definitely a high applicability to large/national enterprise organizations for this type of testing.  Cox is performing more TCP layer testing to qualify end-customer network service and a standard test methodology is a good idea.

Our own IT group actually had an issue as well that wound up being a TCP RWIN problem for an application served out of our Atlanta data center with clients on the West coast.

Brent Bowers
Director, Commercial Engineering
CCIE #13530
Cox Communications, Inc.
619.266.5436 (Office)
619.972.5436 (Mobile)

On Jun 26, 2010, at 3:51 AM, Barry Constantine wrote:

Ed,

Thanks,
Barry

JDSU Communication Test (formerly Acterna)
Emerging Markets and Technology Research
One Milestone Center Court
Germantown, MD 20876
(W) 240-404-2227
(C) 301-325-7069
________________________________
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Gilles Forget
Sent: Friday, June 25, 2010 12:29 AM
To: 'Ed Gastle'; [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] TCP Throughput Testing Methodology draft

Bonjour Ed,

About RFC2544, I would just like to add that it was never meant for network testing. This RFC was written with lab testing in mind only, right at the beginning in section 3, it talks about DUTs (Devices Under Test).
Plus the complete title is : Benchmarking Methodology for Network Interconnect Dev. However, it’s a fact of life that it has been used for at least 10 years now, not only for lab testing but for complete end to end network tests and that is one of the reasons why we have decided to mention it in the draft. I know by field experience that it leads to finger pointing situations between business customers and network providers. On the other hand, it’s hard for us to be very precise about how many frame loss or how much jitter is acceptable without entering the SLA (Service Level Agreement) area. SLAs are not only based on technical variables, they are also based on business and political variables that we can not define in the present document.

Thanks,
Gilles Forget, CCNP.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Ed Gastle
Sent: June-24-10 3:00 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [ippm] TCP Throughput Testing Methodology draft

<ATT00001..txt>