Subject: RE: [bmwg] Some issues in RFC-2544

From: Jim McQuaid <[email-masked]>
Date: Tue, 18 Feb 2003 13:05:28 -0500

Right. So the individual tester always has to use their best judgement in
balancing these factors for any particular DUT.  That's why testers get the
big bucks!  ;-)

Jim

-----Original Message-----
From: Dawit Birhanu [mailto:[email-masked]]
Sent: Tuesday, February 18, 2003 1:00 PM
To: Jim McQuaid; David Newman; John Dawson
Cc: [email-masked]
Subject: RE: [bmwg] Some issues in RFC-2544

Jim,

I rechecked RFC2544 and it has both 2 second residual time and 60 second
test duration.  (my mistake).

But the main problem that I have is that depending on the size of buffering,
two routers with identical forwarding capability could result in different
performance measurment values. Note that 2 second residual time with 60
second test duration could potentially result in overestimation of the
throughput performance by upto 3.3% (2/60). This suggests that, to decrease
the over-estimation we have to decrease the residual time or increase the
test duration.

-dawit

> -----Original Message-----
> From: Jim McQuaid [mailto:[email-masked]]
> Sent: Tuesday, February 18, 2003 5:21 AM
> To: [email-masked]; David Newman; John Dawson
> Cc: [email-masked]
> Subject: RE: [bmwg] Some issues in RFC-2544
>
>
> First of all, 2544 does deal with the issue of how long a receiver should
> wait in section 23; "Wait for two seconds for any residual frames to be
> received."
>
> Section 24, titled "Trial Duration" is pretty clear in my opinion.  Most
> importantly, it states the GOAL of the test duration, "The aim of these
> tests is to determine the rate continuously supportable by the DUT."  That
> is a lot better than requiring some specific time, because,
> obviously, for a
> given test time, there is a buffer size large enough to "fool" the test.
> This section further states that the duration SHOULD be at least
> 60 seconds,
> which is a pretty large amount of traffic to buffer at any common network
> speed today, I think.  (Also a long time to age a frame . . .)  Finally,
> there is a reference to "binary search" for throughput, which in practice
> means starting with the highest possible value of throughput we
> can test for
> / expect and observing {typically} some frame loss, than working
> down using
> what I will call a gross interval, until you have  bracketed the
> frame rate
> for throughput, followed by a finer-grained repeat of the same thing.  By
> starting at the top, so to speak, we establish that there is a frame loss
> condition.  If the device is capable of sustained wire speed,
> then our only
> focus would be the test duration.
>
> I haven't looked closely at 2544 for a while, but it seems to be
> sufficient
> already re: your concerns about duration, waiting, etc.
>
> Jim McQuaid
>
> -----Original Message-----
> From: Dawit Birhanu [mailto:[email-masked]]
> Sent: Tuesday, February 18, 2003 2:35 AM
> To: David Newman; John Dawson
> Cc: [email-masked]
> Subject: RE: [bmwg] Some issues in RFC-2544
>
>
>
> > Reinterpreting 2544 also takes us down the path of trying to redefine
> > existing terms. it's taken us long enough to make clear that
> throughput !=
> > forwarding rate; let's not go changing those meanings now.
> The problem with the current definition "throughput = no drop
> rate" is that
> it
> can result in dramatically different measurement values depending on the
> duration
> of the test. 2544 does not specify the duration of the test. Also it does
> NOT specify how long the receiver must wait to receive the last
> packet after
> the transmitter has completed transmitting.
>
> >
> > As Jerry noted, the dsterm draft introduces a new term, delay, that
> > should help address John's concerns about latency.
> >
> > dn
> >
> > _______________________________________________
> > bmwg mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/bmwg
> >
>
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/bmwg
>
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Dawit Birhanu" <[email-masked]>
Date: Tue, 18 Feb 2003 10:42:14 -0800

Allowing different values of and requiring the inclusion of residual time
and test duration in the test report would have the following advantages:

 - Better indication of the test condition (test environment) to the user
 - Controlling the effect of buffering on throughput measurement
 - Increasing testing efficient. For example if we use a residual time of 5
msec, we can decrease the test duration (for example to 5 second).

Thanks,
-dawit

> From: [email-masked] [mailto:[email-masked]]On Behalf Of Jim
> McQuaid
> Sent: Tuesday, February 18, 2003 10:05 AM
> To: [email-masked]'; Jim McQuaid; David Newman; John Dawson
> Cc: [email-masked]

---

From: "Dawit Birhanu" <[email-masked]>
Date: Tue, 18 Feb 2003 10:42:14 -0800

Thanks,
-dawit