Subject: RE: [bmwg] Some issues in RFC-2544

From: "John Dawson" <[email-masked]>
Date: Tue, 18 Feb 2003 11:16:45 -0800

Offered load is independent of the residual time.  Offered load is the
number of frames sent by a port divided by the test duration.  
 
Forwarding rate, however, is highly dependent upon the residual time.  The
forwarding rate is the offered load (total bits or total packets) divided by
the sum of the test duration and the residual time component that actually
carries traffic.
 
If you have lots of buffers on a box then you are more likely to get a
better throughput rate -- no question about it.  Your proposals are trying
to completely eliminate the effect of buffering from the equation.  When I
am trying to compare performance of switches, I care how much buffering is
present.  Throughput helps me evaluate the amount of buffering that exists
on the device.  This is especially true in the half-duplex test cases.
 
Throughput does not measure the forwarding rate -- it never has.  
I see no reason to change the measurement after so many years.
 
John
 
-----Original Message-----
From: Dawit Birhanu [mailto:[email-masked]] 
Sent: Tuesday, February 18, 2003 10:46 AM
To: Dawson, John [BLVWA:382:EXCH]
Cc: [email-masked]
Subject: RE: [bmwg] Some issues in RFC-2544

While the offered load and forwarding rate are independent of the residual
time, throughput is not since for the same offered load there may or may not
be loss depending on the size of the residual time and buffer size.
 
-dawit

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of John
Dawson
Sent: Tuesday, February 18, 2003 10:16 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] Some issues in RFC-2544

There is no over-estimate.  Throughput is a measure of offered load that is
forwarded without loss.  It is not a measure of forwarding rate.

Since the offered load is independent of the residual time, the forwarding
rate may be changing but the throughput measurement is a constant.  This
characteristic has been known for quite some time.

John 

-----Original Message----- 
From: Dawit Birhanu [mailto:[email-masked] <mailto:[email-masked]> ] 
Sent: Tuesday, February 18, 2003 10:00 AM 
To: Jim McQuaid; David Newman; Dawson, John [BLVWA:382:EXCH] 
Cc: [email-masked] 
Subject: RE: [bmwg] Some issues in RFC-2544 

Jim, 

I rechecked RFC2544 and it has both 2 second residual time and 60 second 
test duration.  (my mistake). 

But the main problem that I have is that depending on the size of buffering,

two routers with identical forwarding capability could result in different 
performance measurment values. Note that 2 second residual time with 60 
second test duration could potentially result in overestimation of the 
throughput performance by upto 3.3% (2/60). This suggests that, to decrease 
the over-estimation we have to decrease the residual time or increase the 
test duration. 

-dawit 

> -----Original Message----- 
> From: Jim McQuaid [mailto:[email-masked]
<mailto:[email-masked]> ] 
> Sent: Tuesday, February 18, 2003 5:21 AM 
> To: [email-masked]; David Newman; John Dawson 
> Cc: [email-masked] 
> Subject: RE: [bmwg] Some issues in RFC-2544 
> 
> 
> First of all, 2544 does deal with the issue of how long a receiver should 
> wait in section 23; "Wait for two seconds for any residual frames to be 
> received." 
> 
> Section 24, titled "Trial Duration" is pretty clear in my opinion.  Most 
> importantly, it states the GOAL of the test duration, "The aim of these 
> tests is to determine the rate continuously supportable by the DUT."  That

> is a lot better than requiring some specific time, because, 
> obviously, for a 
> given test time, there is a buffer size large enough to "fool" the test. 
> This section further states that the duration SHOULD be at least 
> 60 seconds, 
> which is a pretty large amount of traffic to buffer at any common network 
> speed today, I think.  (Also a long time to age a frame . . .)  Finally, 
> there is a reference to "binary search" for throughput, which in practice 
> means starting with the highest possible value of throughput we 
> can test for 
> / expect and observing {typically} some frame loss, than working 
> down using 
> what I will call a gross interval, until you have  bracketed the 
> frame rate 
> for throughput, followed by a finer-grained repeat of the same thing.  By 
> starting at the top, so to speak, we establish that there is a frame loss 
> condition.  If the device is capable of sustained wire speed, 
> then our only 
> focus would be the test duration. 
> 
> I haven't looked closely at 2544 for a while, but it seems to be 
> sufficient 
> already re: your concerns about duration, waiting, etc. 
> 
> Jim McQuaid 
> 
> -----Original Message----- 
> From: Dawit Birhanu [mailto:[email-masked] <mailto:[email-masked]> ] 
> Sent: Tuesday, February 18, 2003 2:35 AM 
> To: David Newman; John Dawson 
> Cc: [email-masked] 
> Subject: RE: [bmwg] Some issues in RFC-2544 
> 
> 
> 
> > Reinterpreting 2544 also takes us down the path of trying to redefine 
> > existing terms. it's taken us long enough to make clear that 
> throughput != 
> > forwarding rate; let's not go changing those meanings now. 
> The problem with the current definition "throughput = no drop 
> rate" is that 
> it 
> can result in dramatically different measurement values depending on the 
> duration 
> of the test. 2544 does not specify the duration of the test. Also it does 
> NOT specify how long the receiver must wait to receive the last 
> packet after 
> the transmitter has completed transmitting. 
> 
> > 
> > As Jerry noted, the dsterm draft introduces a new term, delay, that 
> > should help address John's concerns about latency. 
> > 
> > dn 
> > 
> > _______________________________________________ 
> > bmwg mailing list 
> > [email-masked] 
> > https://www1.ietf.org/mailman/listinfo/bmwg
<https://www1.ietf.org/mailman/listinfo/bmwg>  
> > 
> 
> _______________________________________________ 
> bmwg mailing list 
> [email-masked] 
> https://www1.ietf.org/mailman/listinfo/bmwg
<https://www1.ietf.org/mailman/listinfo/bmwg>

---

From: "Dawit Birhanu" <[email-masked]>
Date: Tue, 18 Feb 2003 13:52:52 -0800

Message
John,

I would like to make a distinction between the required residual time (as
specified in RFC2544), which is a test parameter, and the time after traffic
stopped until the last packet is received (or latency experienced by the
last packet), which is a measurable parameter. I would argue that the former
has no impact on or relationship with forwarding rate. The later does not
impact forwarding rate but could be interrelated with forwarding rate. In
other words, setting the required residual time to 2 sec or 2 msec does not
affect the forwarding rate of the DUT.

I agree that changing the definition of "throughput" measurement is painful
and has a ripple effect. However, I don't see any reason why we should NOT
allow the tester to use different values for required residual time and test
duration, depending on the test cases and REQUIRE the tester to include
these values in the test report.

Thanks,
-dawit

 -----Original Message-----
From: John Dawson [mailto:[email-masked]]
Sent: Tuesday, February 18, 2003 11:17 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] Some issues in RFC-2544

  John

  -----Original Message-----
  From: Dawit Birhanu [mailto:[email-masked]]
  Sent: Tuesday, February 18, 2003 10:46 AM
  To: Dawson, John [BLVWA:382:EXCH]
  Cc: [email-masked]
  Subject: RE: [bmwg] Some issues in RFC-2544

    While the offered load and forwarding rate are independent of the
residual time, throughput is not since for the same offered load there may
or may not be loss depending on the size of the residual time and buffer
size.

    -dawit
      -----Original Message-----
      From: [email-masked] [mailto:[email-masked]]On Behalf Of
John Dawson
      Sent: Tuesday, February 18, 2003 10:16 AM
      To: [email-masked]
      Cc: [email-masked]
      Subject: RE: [bmwg] Some issues in RFC-2544

      There is no over-estimate.  Throughput is a measure of offered load
that is forwarded without loss.  It is not a measure of forwarding rate.

      Since the offered load is independent of the residual time, the
forwarding rate may be changing but the throughput measurement is a
constant.  This characteristic has been known for quite some time.

      John

      -----Original Message-----
      From: Dawit Birhanu [mailto:[email-masked]]
      Sent: Tuesday, February 18, 2003 10:00 AM
      To: Jim McQuaid; David Newman; Dawson, John [BLVWA:382:EXCH]
      Cc: [email-masked]
      Subject: RE: [bmwg] Some issues in RFC-2544

      Jim,

      I rechecked RFC2544 and it has both 2 second residual time and 60
second
      test duration.  (my mistake).

      But the main problem that I have is that depending on the size of
buffering,
      two routers with identical forwarding capability could result in
different
      performance measurment values. Note that 2 second residual time with
60
      second test duration could potentially result in overestimation of the
      throughput performance by upto 3.3% (2/60). This suggests that, to
decrease
      the over-estimation we have to decrease the residual time or increase
the
      test duration.

      -dawit

      > From: Jim McQuaid [mailto:[email-masked]]

---

From: "Dawit Birhanu" <[email-masked]>
Date: Tue, 18 Feb 2003 13:52:52 -0800

Message
John,

Thanks,
-dawit

 -----Original Message-----
From: John Dawson [mailto:[email-masked]]
Sent: Tuesday, February 18, 2003 11:17 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] Some issues in RFC-2544

  John

  -----Original Message-----
  From: Dawit Birhanu [mailto:[email-masked]]
  Sent: Tuesday, February 18, 2003 10:46 AM
  To: Dawson, John [BLVWA:382:EXCH]
  Cc: [email-masked]
  Subject: RE: [bmwg] Some issues in RFC-2544

    -dawit
      -----Original Message-----

      John

      -----Original Message-----
      From: Dawit Birhanu [mailto:[email-masked]]
      Sent: Tuesday, February 18, 2003 10:00 AM
      To: Jim McQuaid; David Newman; Dawson, John [BLVWA:382:EXCH]
      Cc: [email-masked]
      Subject: RE: [bmwg] Some issues in RFC-2544

      Jim,

      -dawit