Subject: RE: [bmwg] Some issues in RFC-2544

From: Jim McQuaid <[email-masked]>
Date: Fri, 14 Feb 2003 13:33:06 -0500

A quick thought about item #2 below.

You wrote:

	"in most cases leads to queue buildup in the DUT, resulting in
extemely high latency measurement."

This is true of one class of architectures but is not true of some others.
L2 switches, for example and other hardware level switching tends to
function without much queueing.  So the test and the metric, as currently
described, provides a pretty clear insight into that issue.

Put another way, the high latency is not a "problem" but a characteristic.
The test method is designed to find semi-realistic (but testable) boundary
conditions in performance.  Requiring the latency to be measured at
throughput rate seems to achieve this goal.  And yes, some devices will
reveal their 'worst' behavior under these conditions.  That is, in effect, a
successful test method.

Looking forward to other BMWGers thoughts on this.

Jim McQuaid

NetIQ Raleigh

-----Original Message-----
From: Dawit Birhanu [mailto:[email-masked]]
Sent: Friday, February 14, 2003 1:17 PM
To: [email-masked]
Cc: Lane Wigley
Subject: [bmwg] Some issues in RFC-2544

A number of test equipement verndors have implemented RFC2544 benchmarking
suite to their test. Router vendors as well as customers have greately
benefited from this. However, we have seen some issues with these tests with
respect to performance and latency testing. The purpose of this e-mail is to
initiate a discussion on this issues.

1. RFC2544 Throughput Testing

RFC 2544 defines throughput as "the fastest rate at which the count of test
frames transmitted by the DUT is equal to the number of test frames sent to
it by the test equipment." The issue that we have seen is that the no-drop
rate usually overestimates the actuall performance of the DUT. The
measurement error depends on the duration of the test and size of buffering
in the DUT. However, RFC-2544 does not specify nor require the reporting of
test duration and max/min/avg latency measurements during throughput
testing. Note that the test duration T, the minimum latency Dmin, the
maximum latency Dmax and the measurement error due to buffering Eb can be
related as follows:

Eb = (Dmax - Dmin)/T

This is assuming that the latency variation is mostly attributed to
buffering.

There are two possible solutions to this problem:

a. Redefine throughput as "the fastest rate at which count of test frames
transmitted by DUT is qual to the number of test frames sent to it by the
test equipment without exceeding a specified maximum latency. The test
duration should be at least 1000 (or 100) times the specified maximum
latency".

b. Require the vendor to include Test duration and maximum latency in the
test report.

2. RFC2544 Latency Testing

RFC2544 requires that latency be measured while sending traffic at the
no-drop throughput rate. This in most cases leads to queue buildup in the
DUT, resulting in extemely high latency measurement.

One possible solution for this problem is to require that latency be
measured at 90%, 95% and/or 99% of measured throughput.

Thanks,
Dawit Birhanu
[email-masked]

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Dawit Birhanu" <[email-masked]>
Date: Fri, 14 Feb 2003 11:31:41 -0800

The high latency could sometimes be a characteristics. But it could as well
be a "problem". We can see queue builup even when a DUT is truely capable of
supporting linerate throughput. Here are some of the reasons why this could
happen:

1. Even if the clock rate used by the DUT and the Test equipment are
nominally identical for similar interface types, in reality there is slight
difference. Therefore, it is possible that the Test equipment may use a
slightly higher clock rate than the DUT. In this case queue buildup is
unavoidable.

2. Some link layer protocols (eg. PPP) use bit/byte stuffing. For such types
of link layer protocols, it is possible that there may not be bit/byte
stuffing sent by Test equipment but the DUT may have to generate bit/byte
stuffing due to changes in TTL, checksum, CRC, etc. The occurance of this is
rare. But when it occurs, queue buildup is be unavoidable.

3. In some cases DUTs may generate L2 keepalives or other control plane
frames. But the Test equipment may not generate this additional frames. This
would also result in queue buildup.

Thanks,
-dawit

> From: [email-masked] [mailto:[email-masked]]On Behalf Of Jim
> McQuaid
> Sent: Friday, February 14, 2003 10:33 AM
> To: [email-masked]'; [email-masked]
> Cc: Lane Wigley
> Subject: RE: [bmwg] Some issues in RFC-2544
>
>

---

From: "Dawit Birhanu" <[email-masked]>
Date: Fri, 14 Feb 2003 11:31:41 -0800

Thanks,
-dawit

---

From: "Bruce Thomson" <[email-masked]>
Date: Sat, 15 Feb 2003 15:26:03 +0900

The idea of measuring latency seems to only be useful in that is counterbalances
the tendancy of equipment with vast buffers to appear faster than it really is.
Latency measurements themselves are not at all a precise measure of performance,
as pointed out by several people.

It seems more useful to take a step back and ask what we would really like to
know about the unit. As a user, I think I would like to know:

1. How fast can data be forwarded sustainably?
2. How much additional burst capacity is provided by buffering?
3. How much latency exists under no-load conditions?

#1 of course needs to be approximated without having the test be of
infinite duration. The other two can be measured directly.

Bruce Thomson
FM-net

---

From: "Bruce Thomson" <[email-masked]>
Date: Sat, 15 Feb 2003 15:26:03 +0900

Bruce Thomson
FM-net

---

From: David Newman <[email-masked]>
Date: Thu, 20 Feb 2003 08:06:46 -0800 (PST)

> 3. How much latency exists under no-load conditions?

None. The latency test described in 2544 is conducted at, and only at, the 
throughput level.

The topic of latency vs. load has been discussed extensively on this list, 
with many suggesting that a new metric be introduced to allow measurements 
at OLOADs other than the throughput level. There appears to be consensus 
that it is useful to describe the slope of the curve leading up to the 
throughput level.

The "delay" metric in the current dsterm ID is one means of describing
this curve. Note that we deliberately used a term other than latency to
avoid confusion with "latency" as described in 1244 and 2544.

dn

---

From: David Newman <[email-masked]>
Date: Thu, 20 Feb 2003 08:06:46 -0800 (PST)

dn