Subject: RE: [bmwg] Some issues in RFC-2544

From: "John Dawson" <[email-masked]>
Date: Fri, 14 Feb 2003 14:11:01 -0800

There are a lot of additional situations in which queue build-up occurs.
Assume for a moment that a DUT can't sustain wire-speed indefinitely (or at
least for the duration of the test).  The throughput rate would therefore be
something less than 100%.  If the DUT employs buffering, then the amount of
data that can be forwarded without loss (throughput) will equal the
state-state forwarding capacity of the system plus the amount of data that
can be held in buffer.  The end result here is that the buffers are always
expected to be full at the end of the throughput test -- if the buffers were
not full, then the DUT would be able to handle more data which would mean
the DUT could support a higher offered load. Taking the latency measurement
half-way through the test just means that we are measuring half of the
buffering depth and not all of the buffering depth.

Now depending upon how the DUT and test equipment is architected, a latency
measurement at the throughput rate can yield vastly different results.
Buffering is not necessarily uniform for all port pairs and some ports may
have slower forwarding rates than others.  Since throughput measurements
force all traffic to slow-down if even one port pair is congested, then we
could have very good latency characteristic on one pair of ports while
another pair of ports could show drastic buffering effects.  This suggests
that reporting latency as a single number isn't always that meaningful -- we
actually need a distribution function to characterize latency properly.

Several years ago, I had presented research on latency measurements that
demonstrated effects caused by clock synchronization between transmit
sources.  Some test systems can have drift between transmitters while others
may not.  This research illustrated the fact that latency can be greatly
affected by the way test systems are internally built -- not a good approach
for providing a reproducible and un-biased measurement.

I would be very against including any latency criteria in the measurement of
Throughput.  You are essentially taking a characteristic (latency) that is
not being measured very well, and mixing it into a measurement that is well
understood (throughput).

I would say go fix latency measurements but leave throughput alone.

John
-----Original Message-----
From: Jim McQuaid [mailto:[email-masked]] 
Sent: Friday, February 14, 2003 12:33 PM
To: [email-masked]'; Jim McQuaid; [email-masked]
Cc: Lane Wigley
Subject: RE: [bmwg] Some issues in RFC-2544

I think that #1 and #3 are the normal problems of conducting any test
properly.  #2 is more interesting, but any equipment that truly maintains
"line rate" will have to have the overhead to deal with some of these
conditions, e.g. MTU changes as discussed under Frame sizes.  Again the
build-up of latency is a clue to this function.

Clearly, publishing the duration of the latency test is very important; 2544
says that tests SHOULD be 120 seconds in length with the latency measured
after 60 seconds.  But the reporting format does not specifically include
reporting the duration of the test, which ought to be a requirement.  It
seems to be that what Dawit is really saying is that the test duration is
significant and ought to be a "variable" reported at the very least.

Jim McQuaid

NetIQ Raleigh

-----Original Message-----
From: Dawit Birhanu [mailto:[email-masked]]
Sent: Friday, February 14, 2003 2:32 PM
To: Jim McQuaid; [email-masked]
Cc: Lane Wigley
Subject: RE: [bmwg] Some issues in RFC-2544

The high latency could sometimes be a characteristics. But it could as well
be a "problem". We can see queue builup even when a DUT is truely capable of
supporting linerate throughput. Here are some of the reasons why this could
happen:

1. Even if the clock rate used by the DUT and the Test equipment are
nominally identical for similar interface types, in reality there is slight
difference. Therefore, it is possible that the Test equipment may use a
slightly higher clock rate than the DUT. In this case queue buildup is
unavoidable.

2. Some link layer protocols (eg. PPP) use bit/byte stuffing. For such types
of link layer protocols, it is possible that there may not be bit/byte
stuffing sent by Test equipment but the DUT may have to generate bit/byte
stuffing due to changes in TTL, checksum, CRC, etc. The occurance of this is
rare. But when it occurs, queue buildup is be unavoidable.

3. In some cases DUTs may generate L2 keepalives or other control plane
frames. But the Test equipment may not generate this additional frames. This
would also result in queue buildup.

Thanks,
-dawit

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of Jim
> McQuaid
> Sent: Friday, February 14, 2003 10:33 AM
> To: [email-masked]'; [email-masked]
> Cc: Lane Wigley
> Subject: RE: [bmwg] Some issues in RFC-2544
>
>
> A quick thought about item #2 below.
>
> You wrote:
>
> 	"in most cases leads to queue buildup in the DUT, resulting in
> extemely high latency measurement."
>
> This is true of one class of architectures but is not true of some others.
> L2 switches, for example and other hardware level switching tends to
> function without much queueing.  So the test and the metric, as currently
> described, provides a pretty clear insight into that issue.
>
> Put another way, the high latency is not a "problem" but a characteristic.
> The test method is designed to find semi-realistic (but testable) boundary
> conditions in performance.  Requiring the latency to be measured at
> throughput rate seems to achieve this goal.  And yes, some devices will
> reveal their 'worst' behavior under these conditions.  That is,
> in effect, a
> successful test method.
>
> Looking forward to other BMWGers thoughts on this.
>
> Jim McQuaid
>
> NetIQ Raleigh
>
>
>
> -----Original Message-----
> From: Dawit Birhanu [mailto:[email-masked]]
> Sent: Friday, February 14, 2003 1:17 PM
> To: [email-masked]
> Cc: Lane Wigley
> Subject: [bmwg] Some issues in RFC-2544
>
>
>
> A number of test equipement verndors have implemented RFC2544 benchmarking
> suite to their test. Router vendors as well as customers have greately
> benefited from this. However, we have seen some issues with these
> tests with
> respect to performance and latency testing. The purpose of this
> e-mail is to
> initiate a discussion on this issues.
>
> 1. RFC2544 Throughput Testing
>
> RFC 2544 defines throughput as "the fastest rate at which the
> count of test
> frames transmitted by the DUT is equal to the number of test
> frames sent to
> it by the test equipment." The issue that we have seen is that the no-drop
> rate usually overestimates the actuall performance of the DUT. The
> measurement error depends on the duration of the test and size of
> buffering
> in the DUT. However, RFC-2544 does not specify nor require the
> reporting of
> test duration and max/min/avg latency measurements during throughput
> testing. Note that the test duration T, the minimum latency Dmin, the
> maximum latency Dmax and the measurement error due to buffering Eb can be
> related as follows:
>
> Eb = (Dmax - Dmin)/T
>
> This is assuming that the latency variation is mostly attributed to
> buffering.
>
> There are two possible solutions to this problem:
>
> a. Redefine throughput as "the fastest rate at which count of test frames
> transmitted by DUT is qual to the number of test frames sent to it by the
> test equipment without exceeding a specified maximum latency. The test
> duration should be at least 1000 (or 100) times the specified maximum
> latency".
>
> b. Require the vendor to include Test duration and maximum latency in the
> test report.
>
> 2. RFC2544 Latency Testing
>
> RFC2544 requires that latency be measured while sending traffic at the
> no-drop throughput rate. This in most cases leads to queue buildup in the
> DUT, resulting in extemely high latency measurement.
>
> One possible solution for this problem is to require that latency be
> measured at 90%, 95% and/or 99% of measured throughput.
>
> Thanks,
> Dawit Birhanu
> [email-masked]
>
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/bmwg
>
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: David Newman <[email-masked]>
Date: Sat, 15 Feb 2003 09:16:34 -0800 (PST)

> I would say go fix latency measurements but leave 

strongly concur on this point

The goal (I think) of latency measurements is to describe the basic delay
introduced by the DUT/SUT, not to try to divine buffer state or other
conditions that are not externally observable.

Reinterpreting 2544 also takes us down the path of trying to redefine
existing terms. it's taken us long enough to make clear that throughput !=
forwarding rate; let's not go changing those meanings now.

As Jerry noted, the dsterm draft introduces a new term, delay, that 
should help address John's concerns about latency.

dn

---

From: David Newman <[email-masked]>
Date: Sat, 15 Feb 2003 09:16:34 -0800 (PST)

dn

---

From: "Dawit Birhanu" <[email-masked]>
Date: Mon, 17 Feb 2003 23:35:01 -0800

> forwarding rate; let's not go changing those meanings now.
The problem with the current definition "throughput = no drop rate" is that
it
can result in dramatically different measurement values depending on the
duration
of the test. 2544 does not specify the duration of the test. Also it does
NOT specify how long the receiver must wait to receive the last packet after
the transmitter has completed transmitting.

---

From: "Dawit Birhanu" <[email-masked]>
Date: Mon, 17 Feb 2003 23:35:01 -0800