Subject: RE: [bmwg] Terminology Question: Throughput

From: "Player, Timmons" <[email-masked]>
Date: Fri, 14 Nov 2003 05:44:27 -1000

So, just to clarify the point...  Following the conventions of 1242 would
mean rewording the throughput defintions to use input frames per second or
input bits per second as the measurement units, right?

This was the main point of the original question.  Supposing a device has a
(as currently defined) {en,de}cryption throughput of x input Mbps and y
output Mbps, then the vendor will report the max(x,y) as the throughput.
Specifying where the measurement is made removes any, err, opportunity for
specmanship. :)

Timmons

> -----Original Message-----
> From: Tim Van Herck [mailto:[email-masked]]
> Sent: Friday, November 14, 2003 9:05 AM
> To: Perser, Jerry
> Cc: [email-masked]'
> Subject: Re: [bmwg] Terminology Question: Throughput
> 
> 
> 	Jerry,
> > Okay,
> > 
> > Let me restate the problem assuming the loss detection 
> happens at both the
> > ingress and egress:
> 
> Loss detection happens on the egress side but throughput as 
> not a metric 
> that is specific to ingress or egress, I would say.
> 
> > Throughput is a measured RATE at the ingress of the DUT/SUT 
> [RFC1242]
> > Forwarding Rate is a measured RATE at the egress of the 
> DUT/SUT [RFC1242]
> > 
> > As I understand Timmons question on ipsec I-D:
> > 
> > 9.3.1 IPsec Tunnel Throughput
> >    Definition:
> >       The forwarding rate through an IKE/IPsec tunnel at
> which none of
> >       the offered frames are dropped by the device under test.
> > 
> > Does the measurement Units follow RFC1242 or RFC2285?
> 
> I would say we stick to the definition of throughput as stated in 
> RFC1242 but remember that this can happen in two distinct topologies 
> which will be described in detail in the methodology draft.
> 
> For IPSec Tunnel Throughput this would be the throughput of an SUT 
> consisting of two identical back-to-back connected IPSec devices.
> 
> 	Tim-
> 
> > -----Original Message-----
> > From: Jim McQuaid [mailto:[email-masked]]
> > Sent: Thursday, November 13, 2003 11:53 AM
> > To: 'John Dawson'; [email-masked]'
> > Subject: RE: [bmwg] Terminology Question: Throughput
> > 
> > 
> > Yes, correct.  And you state it better than I did.  You 
> have to *measure* on
> > both ingress and egress, but the metric is the ingress rate 
> at which the
> > throughput condition is satisfied.
> > 
> > There was a discussion years back about "goodput" (and 
> other suggested
> > nomenclature) which was kind of throughput with some amount 
> of dropped
> > frames allowed or ignored or deducted in some fashion.  But 
> that discussion
> > never yielded any consensus about what is acceptable or 
> useful in practice
> > when it comes to dropping frames.
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On 
> Behalf Of John
> > Dawson
> > Sent: Thursday, November 13, 2003 2:46 PM
> > To: [email-masked]'
> > Subject: RE: [bmwg] Terminology Question: Throughput
> > 
> > 
> > I agree with Jerry on this one.  The throughput measurement 
> as not as clear
> > as you think (Jim). 
> > Yes, you do look at the number of packets/frames sent 
> (ingress) and compare
> > it to the number received (egress), and you try to find a 
> maximum rate that
> > can be offered on the ingress without loss.
> > However, when it comes time to report the final "rate", you 
> must report the
> > offered load -- which is purely an ingress calculation.  
> The reported
> > Throughput rate has nothing to due with the received 
> (egress) rate, so long
> > as all packets are received.
> > John D. 
> > -----Original Message----- 
> > From: Jim McQuaid [mailto:[email-masked]] 
> > Sent: Thursday, November 13, 2003 11:26 AM 
> > To: 'Perser, Jerry'; [email-masked]' 
> > Subject: RE: [bmwg] Terminology Question: Throughput 
> > 
> > 
> > No.  You cannot measure throughput without measuring 
> ingress and egress. 
> > The definition of throughput perhaps allows some ambiguity. 
>  However, how 
> > would one detect "dropped" if only looking at ingress?  
> Likewise, the 
> > definition of frame loss rate clearly talks about forwarded 
> (egress) frames;
> > 
> > the 'percentage' can only be calculated if you have both 
> ingress and egress.
> > 
> > 
> > 
> > Finally, the definition of the throughput procedure is 
> extremely clear about
> > 
> > HOW to measure throughput: by counting ingress and egress 
> and comparing it. 
> > Test equipment implementing this test did exactly that.  It 
> is necessary to 
> > achieve a sufficently long test duration to overcome 
> buffering effects, 
> > hence the reference in 'frame loss rate' to "steady state." 
> > You said that your issue is that 'throughput' cannot be 
> both an ingress and 
> > egress metric.  In one sense, that's true.  It's a metric 
> about throughput, 
> > namely the relationship of inputs to outputs and it requires two 
> > simultaneous measurements to be computed. 
> > cheers, 
> > Jim McQuaid 
> > NetIQ Raleigh 
> > ------------------------------------------ 
> > 1242 definition of throughput: 
> > "The maximum rate at which none of the offered frames are 
> dropped by the 
> > device." 
> > 1242 definition of frame loss rate: 
> > "Percentage of frames that should have been forwarded by a 
> network device 
> > under steady state (constant)load that were not forwarded 
> due to lack of 
> > resources." 
> > 1944/2544 definition of throughput benchmark procedure: 
> >  'Procedure:  Send a specific number of frames at a 
> specific rate through 
> > the DUT and then count the frames that are transmitted by 
> the DUT. ' 
> > 
> > 
> > 
> > 
> > -----Original Message----- 
> > From: Perser, Jerry [mailto:[email-masked]] 
> > Sent: Thursday, November 13, 2003 12:38 PM 
> > To: Jim McQuaid; [email-masked]' 
> > Subject: RE: [bmwg] Terminology Question: Throughput 
> > 
> > 
> > Jim, 
> > The definition leads one to believe that it is an ingress 
> method.  "The 
> > maximum rate at which none of the offered frames are 
> dropped by the device" 
> > [RFC1242] mentions offered frames, not forwarded frames.  Also the 
> > measurement units are in N-octet input frames per second.  
> IMHO, ingress and
> > 
> > input are the same. 
> > My issue is that throughput can not both be an ingress 
> metric AND an egress 
> > metric.  In any DUT/SUT that buffers (store and forward 
> devices) the maximum
> > 
> > zero loss rate at the ingress will be higher than the 
> egress.  Vendors will 
> > use this ambiguity to engage in "specsmanship". 
> > Jerry. 
> > 
> >>-----Original Message----- 
> >>From: Jim McQuaid [mailto:[email-masked]] 
> >>Sent: Thursday, November 13, 2003 5:52 AM 
> >>To: 'Perser, Jerry'; [email-masked]' 
> >>Subject: RE: [bmwg] Terminology Question: Throughput 
> >>
> >>
> >>I think the concept that throughput can be measured at the 
> ingress of 
> >>anything is incorrect.  Throughput is the ability of a DUT to 
> >>forward 100% 
> >>of the offered load.  The definition more or less requires 
> >>measurement on 
> >>both the ingress and egress and a comparison.  This is just 
> >>1242 stuff. 
> >>
> >>Jim McQuaid 
> >>NetIQ Raleigh 
> >>
> >>-----Original Message----- 
> >>From: [email-masked] [mailto:[email-masked]]On Behalf Of 
> >>Perser, Jerry 
> >>Sent: Wednesday, November 12, 2003 6:45 PM 
> >>To: [email-masked]' 
> >>Subject: RE: [bmwg] IPSec Terminology Question: 
> Encryption/Decryption 
> >>Thr oughput 
> >>
> >>
> >>Tim, 
> >>
> >>Before we can agree, I like to get some basic terminology correct: 
> >>
> >>- Throughput is measured at the ingress of the DUT/SUT 
> >>- Forwarding Rate is measured at the egress of the DUT/SUT 
> >>- Layer 2 measurements are in frames 
> >>- Layer 3 measurements are in packets 
> >>
> >>If anyone disagrees with these definitions, please point to 
> >>the correct 
> >>definition document.  I bring this up because I don't think 
> >>we are talking 
> >>apples to apples here. 
> >>
> >>Jerry. 
> >>
> >>
> >>
> >>>-----Original Message----- 
> >>>From: Tim Van Herck [mailto:[email-masked]] 
> >>>Sent: Wednesday, November 12, 2003 3:17 PM 
> >>>To: Perser, Jerry 
> >>>Cc: [email-masked]' 
> >>>Subject: Re: [bmwg] IPSec Terminology Question: 
> >>
> >>Encryption/Decryption 
> >>
> >>>Throughput 
> >>>
> >>>
> >>>    Jerry, 
> >>>
> >>>Let me restate that. We will define a tput test with two 
> >>>ipsec devices 
> >>>in a back to back setup on which we only measure at ingress 
> >>
> >>ports and 
> >>
> >>>may report the projected egress rate/Mbps. 
> >>>
> >>>In other scenario's we can use the defined framesizes to 
> >>>measure rates. 
> >>>In all cases the definition of a rate doesn't change, just 
> >>>the framesize 
> >>>  used to calculate the throughput changes. Agree ? 
> >>>
> >>>See section 9.1 
> >>>
> >>>    Tim- 
> >>>
> >>>
> >>>>If you are going to measure at the ingress AND egress, then 
> >>>
> >>>you will need 
> >>>
> >>>>two different definitions for the two metrics. 
> >>>>
> >>>>Jerry. 
> >>>>
> >>>>
> >>>>
> >>>>>-----Original Message----- 
> >>>>>From: Tim Van Herck [mailto:[email-masked]] 
> >>>>>Sent: Wednesday, November 12, 2003 10:46 AM 
> >>>>>To: Player, Timmons 
> >>>>>Cc: [email-masked]' 
> >>>>>Subject: Re: [bmwg] IPSec Terminology Question: 
> >>>>
> >>>Encryption/Decryption 
> >>>
> >>>>>Throughput 
> >>>>>
> >>>>>
> >>>>> Timmons, 
> >>>>>
> >>>>>Like Michele already mentioned this will be addresses in 
> >>>>
> >>>the meth I-D 
> >>>
> >>>>>but to answer the question, we will indeed document that you can 
> >>>>>represent throughput in four ways : 
> >>>>>
> >>>>>- measured at ingress in pps (clear traffic) 
> >>>>>- measured at ingress in Mbps (clear traffic) 
> >>>>>- measured at egress in pps (crypto traffic) 
> >>>>>- measured at egress in Mbps (crypto traffic & using crypto 
> >>>>
> >>>pkt size) 
> >>>
> >>>>> Tim- 
> >>>>>
> >>>>>
> >>>>>
> >>>>>>When performing an IPSec Encryption/Decryption Throughput 
> >>>>>
> >>>>>the draft isn't 
> >>>>>
> >>>>>
> >>>>>>clear as to which side of the DUT pps or Mbps should be 
> >>>>>
> >>>>>measured.  I'm 
> >>>>>
> >>>>>
> >>>>>>pretty sure that Mbps will be different for ingress and 
> >>>>>
> >>>>>egress.  And pps 
> >>>>>
> >>>>>
> >>>>>>could be different for ingress and egress depending on the 
> >>>>>
> >>>>>size of the input 
> >>>>>
> >>>>>
> >>>>>>packet, at least for the encryption throughput case. 
> >>>>>
> >>>>>
> >>>>>
> >>>>>_______________________________________________ 
> >>>>>bmwg mailing list 
> >>>>>[email-masked] 
> >>>>>https://www1.ietf.org/mailman/listinfo/bmwg 
> >>>>>
> >>>>
> >>>>
> >>>>_______________________________________________ 
> >>>>bmwg mailing list 
> >>>>[email-masked] 
> >>>>https://www1.ietf.org/mailman/listinfo/bmwg 
> >>>
> >>_______________________________________________ 
> >>bmwg mailing list 
> >>[email-masked] 
> >>https://www1.ietf.org/mailman/listinfo/bmwg 
> >>
> > 
> > _______________________________________________ 
> > bmwg mailing list 
> > [email-masked] 
> > https://www1.ietf.org/mailman/listinfo/bmwg 
> > 
> > _______________________________________________
> > bmwg mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/bmwg
> 
> 
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/bmwg
> 

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Tim Van Herck <[email-masked]>
Date: Mon, 17 Nov 2003 13:13:30 -0800

Timmons,

> input bits per second as the measurement units, right?

Correct.

> Specifying where the measurement is made removes any, err, opportunity for

We will clarify in the methodology document how this needs to be 
reported but you are correct that we need to be very carefull with this 
metric since it is prone to different interpretations.

My take on it would be to report a throughput as the offered load at 
which there is no packetloss anymore. Since we also defined different 
packetsizes in the term doc we can now report the rate that is offered 
(in pps) and convert it with the correct packetsize to a throughput 
number in Mbps. No room anymore for specmanship in that case ! For every 
  reported PPS there will be an associated Mbps depending on which link 
it is measured and which transform is applied to it.

	Tim-

---

From: Tim Van Herck <[email-masked]>
Date: Mon, 17 Nov 2003 13:13:30 -0800

Timmons,

Correct.

	Tim-