Subject: RE: [bmwg] Some issues in RFC-2544

From: "John Dawson" <[email-masked]>
Date: Tue, 18 Feb 2003 10:15:37 -0800

There is no over-estimate.  Throughput is a measure of offered load that is
forwarded without loss.  It is not a measure of forwarding rate.

Since the offered load is independent of the residual time, the forwarding
rate may be changing but the throughput measurement is a constant.  This
characteristic has been known for quite some time.

John 

-----Original Message-----
From: Dawit Birhanu [mailto:[email-masked]] 
Sent: Tuesday, February 18, 2003 10:00 AM
To: Jim McQuaid; David Newman; Dawson, John [BLVWA:382:EXCH]
Cc: [email-masked]
Subject: RE: [bmwg] Some issues in RFC-2544

Jim,

I rechecked RFC2544 and it has both 2 second residual time and 60 second
test duration.  (my mistake).

But the main problem that I have is that depending on the size of buffering,
two routers with identical forwarding capability could result in different
performance measurment values. Note that 2 second residual time with 60
second test duration could potentially result in overestimation of the
throughput performance by upto 3.3% (2/60). This suggests that, to decrease
the over-estimation we have to decrease the residual time or increase the
test duration.

-dawit

> -----Original Message-----
> From: Jim McQuaid [mailto:[email-masked]]
> Sent: Tuesday, February 18, 2003 5:21 AM
> To: [email-masked]; David Newman; John Dawson
> Cc: [email-masked]
> Subject: RE: [bmwg] Some issues in RFC-2544
>
>
> First of all, 2544 does deal with the issue of how long a receiver should
> wait in section 23; "Wait for two seconds for any residual frames to be
> received."
>
> Section 24, titled "Trial Duration" is pretty clear in my opinion.  Most
> importantly, it states the GOAL of the test duration, "The aim of these
> tests is to determine the rate continuously supportable by the DUT."  That
> is a lot better than requiring some specific time, because,
> obviously, for a
> given test time, there is a buffer size large enough to "fool" the test.
> This section further states that the duration SHOULD be at least
> 60 seconds,
> which is a pretty large amount of traffic to buffer at any common network
> speed today, I think.  (Also a long time to age a frame . . .)  Finally,
> there is a reference to "binary search" for throughput, which in practice
> means starting with the highest possible value of throughput we
> can test for
> / expect and observing {typically} some frame loss, than working
> down using
> what I will call a gross interval, until you have  bracketed the
> frame rate
> for throughput, followed by a finer-grained repeat of the same thing.  By
> starting at the top, so to speak, we establish that there is a frame loss
> condition.  If the device is capable of sustained wire speed,
> then our only
> focus would be the test duration.
>
> I haven't looked closely at 2544 for a while, but it seems to be
> sufficient
> already re: your concerns about duration, waiting, etc.
>
> Jim McQuaid
>
> -----Original Message-----
> From: Dawit Birhanu [mailto:[email-masked]]
> Sent: Tuesday, February 18, 2003 2:35 AM
> To: David Newman; John Dawson
> Cc: [email-masked]
> Subject: RE: [bmwg] Some issues in RFC-2544
>
>
>
> > Reinterpreting 2544 also takes us down the path of trying to redefine
> > existing terms. it's taken us long enough to make clear that
> throughput !=
> > forwarding rate; let's not go changing those meanings now.
> The problem with the current definition "throughput = no drop
> rate" is that
> it
> can result in dramatically different measurement values depending on the
> duration
> of the test. 2544 does not specify the duration of the test. Also it does
> NOT specify how long the receiver must wait to receive the last
> packet after
> the transmitter has completed transmitting.
>
> >
> > As Jerry noted, the dsterm draft introduces a new term, delay, that
> > should help address John's concerns about latency.
> >
> > dn
> >
> > _______________________________________________
> > bmwg mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/bmwg
> >
>
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/bmwg
>

---

From: "Dawit Birhanu" <[email-masked]>
Date: Tue, 18 Feb 2003 10:45:38 -0800

RE: [bmwg] Some issues in RFC-2544
While the offered load and forwarding rate are independent of the residual
time, throughput is not since for the same offered load there may or may not
be loss depending on the size of the residual time and buffer size.

-dawit
  -----Original Message-----
  From: [email-masked] [mailto:[email-masked]]On Behalf Of John
Dawson
  Sent: Tuesday, February 18, 2003 10:16 AM
  To: [email-masked]
  Cc: [email-masked]
  Subject: RE: [bmwg] Some issues in RFC-2544

  There is no over-estimate.  Throughput is a measure of offered load that
is forwarded without loss.  It is not a measure of forwarding rate.

  John

  -----Original Message-----
  From: Dawit Birhanu [mailto:[email-masked]]
  Sent: Tuesday, February 18, 2003 10:00 AM
  To: Jim McQuaid; David Newman; Dawson, John [BLVWA:382:EXCH]
  Cc: [email-masked]
  Subject: RE: [bmwg] Some issues in RFC-2544

  Jim,

  But the main problem that I have is that depending on the size of
buffering,
  two routers with identical forwarding capability could result in different
  performance measurment values. Note that 2 second residual time with 60
  second test duration could potentially result in overestimation of the
  throughput performance by upto 3.3% (2/60). This suggests that, to
decrease
  the over-estimation we have to decrease the residual time or increase the
  test duration.

  -dawit

---

From: "Dawit Birhanu" <[email-masked]>
Date: Tue, 18 Feb 2003 10:45:38 -0800

-dawit
  -----Original Message-----

  John

  -----Original Message-----
  From: Dawit Birhanu [mailto:[email-masked]]
  Sent: Tuesday, February 18, 2003 10:00 AM
  To: Jim McQuaid; David Newman; Dawson, John [BLVWA:382:EXCH]
  Cc: [email-masked]
  Subject: RE: [bmwg] Some issues in RFC-2544

  Jim,

  -dawit