Subject: RE: [bmwg] Some issues in RFC-2544 - measuring receive rate

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 19 Feb 2003 09:40:54 -0800

Dawit,

Measuring receive rate is NOT trivial, but solvable.  Measuring Offered Load
(transmit rate) has the same problem.  How do you measure throughput when
the offered load does not math the intended load due to backpresssure?
Anyone who has implemented RFC1242/RFC2544 or RFC2285/RFC2889 has considered
this and developed there own technique.

If you exam RFC1242, RFC2285, and RFC2432, they do not actually specify how
to implement the metrics.  These RFCs only define them.  This type of
information is found in the methodology document (for example RFC 2889
appendix B).

I'm saving you email for the companion methodology to dsmterm.  If you have
a particular method for measuring rates during the test duration, I would
like to discus them with you off the reflector.

Jerry.

> -----Original Message-----
> From: Dawit Birhanu [mailto:[email-masked]]
> Sent: Tuesday, February 18, 2003 11:49 AM
> To: [email-masked]; Perser, Jerry
> Subject: RE: [bmwg] Some issues in RFC-2544
> 
> 
> 
> Parser,
> 
> You are right channel capacity as defined in dsmterm-04 will 
> eliminate the
> effect of buffering. However, measuring receive rate (after 
> the buffer) is
> NOT trivial. If traffic is started at t=0, and stopped at 
> t=Ts, first bit of
> first packet is received at t=Tf, last bit of last packet is 
> recieved at
> t=Tl, and the number of packets received is N, what is the 
> forwarding rate.
> Is it N/Ts or N/Tl or N/(Tl-Tf)? Or do we take some time slot 
> between t=0
> and T=Ts, count the number of packets received during the 
> time slot and
> divide it by the size of the time slot? The draft does not 
> specify this.
> 
> Thanks,
> -dawit
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Perser, Jerry
> > Sent: Tuesday, February 18, 2003 10:49 AM
> > To: [email-masked]
> > Subject: RE: [bmwg] Some issues in RFC-2544
> >
> >
> > Dawit,
> >
> > I propose solving the buffer problem by creating a new metric.
> >
> > More DUT have some sort of input buffer.  By measuring 
> before the buffer,
> > the actual forwarding process is not measured.  Changing 
> the test duration
> > can minimize the affects of buffering.  But it will never remove it.
> >
> > In dsmterm-04, there is a metric called Channel Capacity.  
> It was inspired
> > by John Dawson's comments on the reflector a few years ago.  Channel
> > Capacity is defined as "The maximum forwarding rate [RFC2285] at
> > which none
> > of the offered packets are dropped by the DUT/SUT."
> >
> > Channel Capacity is equal or less than Throughput depending upon
> > buffer size
> > and test duration.  Both are a maximum rate at which none 
> of the offered
> > packets are dropped.  The Difference is where on the DUT to 
> measure the
> > rate.  By measuring after the buffer, Channel Capacity 
> reports the actual
> > forwarding performance.
> >
> > Jerry.
> >
> > > -----Original Message-----
> > > From: Dawit Birhanu [mailto:[email-masked]]
> > > Sent: Tuesday, February 18, 2003 10:00 AM
> > > To: Jim McQuaid; David Newman; John Dawson
> > > Cc: [email-masked]
> > > Subject: RE: [bmwg] Some issues in RFC-2544
> > >
> > >
> > >
> > > Jim,
> > >
> > > I rechecked RFC2544 and it has both 2 second residual time
> > > and 60 second
> > > test duration.  (my mistake).
> > >
> > > But the main problem that I have is that depending on the
> > > size of buffering,
> > > two routers with identical forwarding capability could result
> > > in different
> > > performance measurment values. Note that 2 second residual
> > > time with 60
> > > second test duration could potentially result in 
> overestimation of the
> > > throughput performance by upto 3.3% (2/60). This suggests
> > > that, to decrease
> > > the over-estimation we have to decrease the residual time or
> > > increase the
> > > test duration.
> > >
> > > -dawit
> > >
> > > > -----Original Message-----
> > > > From: Jim McQuaid [mailto:[email-masked]]
> > > > Sent: Tuesday, February 18, 2003 5:21 AM
> > > > To: [email-masked]; David Newman; John Dawson
> > > > Cc: [email-masked]
> > > > Subject: RE: [bmwg] Some issues in RFC-2544
> > > >
> > > >
> > > > First of all, 2544 does deal with the issue of how long a
> > > receiver should
> > > > wait in section 23; "Wait for two seconds for any residual
> > > frames to be
> > > > received."
> > > >
> > > > Section 24, titled "Trial Duration" is pretty clear in my
> > > opinion.  Most
> > > > importantly, it states the GOAL of the test duration, "The
> > > aim of these
> > > > tests is to determine the rate continuously supportable by
> > > the DUT."  That
> > > > is a lot better than requiring some specific time, because,
> > > > obviously, for a
> > > > given test time, there is a buffer size large enough to
> > > "fool" the test.
> > > > This section further states that the duration SHOULD be at least
> > > > 60 seconds,
> > > > which is a pretty large amount of traffic to buffer at any
> > > common network
> > > > speed today, I think.  (Also a long time to age a frame . .
> > > .)  Finally,
> > > > there is a reference to "binary search" for throughput,
> > > which in practice
> > > > means starting with the highest possible value of throughput we
> > > > can test for
> > > > / expect and observing {typically} some frame loss, than working
> > > > down using
> > > > what I will call a gross interval, until you have  bracketed the
> > > > frame rate
> > > > for throughput, followed by a finer-grained repeat of the
> > > same thing.  By
> > > > starting at the top, so to speak, we establish that there
> > > is a frame loss
> > > > condition.  If the device is capable of sustained wire speed,
> > > > then our only
> > > > focus would be the test duration.
> > > >
> > > > I haven't looked closely at 2544 for a while, but it seems to be
> > > > sufficient
> > > > already re: your concerns about duration, waiting, etc.
> > > >
> > > > Jim McQuaid
> > > >
> > > > -----Original Message-----
> > > > From: Dawit Birhanu [mailto:[email-masked]]
> > > > Sent: Tuesday, February 18, 2003 2:35 AM
> > > > To: David Newman; John Dawson
> > > > Cc: [email-masked]
> > > > Subject: RE: [bmwg] Some issues in RFC-2544
> > > >
> > > >
> > > >
> > > > > Reinterpreting 2544 also takes us down the path of trying
> > > to redefine
> > > > > existing terms. it's taken us long enough to make clear that
> > > > throughput !=
> > > > > forwarding rate; let's not go changing those meanings now.
> > > > The problem with the current definition "throughput = no drop
> > > > rate" is that
> > > > it
> > > > can result in dramatically different measurement values
> > > depending on the
> > > > duration
> > > > of the test. 2544 does not specify the duration of the
> > > test. Also it does
> > > > NOT specify how long the receiver must wait to receive the last
> > > > packet after
> > > > the transmitter has completed transmitting.
> > > >
> > > > >
> > > > > As Jerry noted, the dsterm draft introduces a new term,
> > > delay, that
> > > > > should help address John's concerns about latency.
> > > > >
> > > > > dn
> > > > >
> > > > > _______________________________________________
> > > > > bmwg mailing list
> > > > > [email-masked]
> > > > > https://www1.ietf.org/mailman/listinfo/bmwg
> > > > >
> > > >
> > > > _______________________________________________
> > > > bmwg mailing list
> > > > [email-masked]
> > > > https://www1.ietf.org/mailman/listinfo/bmwg
> > > >
> > >
> > > _______________________________________________
> > > bmwg mailing list
> > > [email-masked]
> > > https://www1.ietf.org/mailman/listinfo/bmwg
> > >
> > _______________________________________________
> > bmwg mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/bmwg
> >
> 
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 19 Feb 2003 09:40:54 -0800

Dawit,

Jerry.