Subject: [bmwg] Wire speed definition

From: Ramesh Menon <[email-masked]>
Date: Fri, 19 Jan 2001 09:41:40 -0800

Is the term "Wire speed" defined in any of the RFCs. I have been
looking at some recent third party tests where they run RFC 2544 like
throughput tests but  for a certain distribution of frame sizes. Based
on this frame size distribution throughput, they claim a DUT to have 
wire speed performance although it is clear that this particular DUT cannot
do
100% of wire rate at each of the frame sizes as in RFC 2544/1242.
Is this the usual industry practice or is this just the exception. I would 
very much like your thoughts on the state of things today and what
you think it should be.

While the distribution of frame sizes reported in the literature based
on actual measurements (Thompson '97 (MCI) '97 all the way up to 
McCreary '00 (caida-AIX)) are fairly consistent, the actual distribution can
be
varied within that range. This has implications on whether a DUT is wire
speed
or not. Perhaps RFC2544 should be extended to run a test for a well defined
distribution of frames? 

Thanks
Ramesh Menon
--
Ramesh G. Menon, Ph.D.                       NARUS, Inc.
[email-masked]                               3950 Fabian Way
(650) 475 9228                                       Palo Alto, CA. 94303. 

_______________________________________________
bmwg mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/bmwg

---

From: "John Dawson" <[email-masked]>
Date: Fri, 19 Jan 2001 15:40:36 -0800

RFC 2544 and its' parent 1944 don't use the term wire-speed.  Instead the
term maximum frame rate or theoretical frame rate is used.  Wire-speed and
line-rate are often used inter-changeably with maximum frame rate.  The
formula for calculating the maximum frame rate is not too difficult for LAN
based systems (it is the line speed in bits per second divided by the packet
size which includes preambles and any leading gap).  For a mixed frame size
environment, the average frame size must be deterministic in order to
calculate the maximum frame rate.

It is very possible to manipulate the benchmark to report better numbers.
This has occurred over and over again, sometimes by mistake and sometimes
overtly for purposes of selling product.

One way to manipulate the test is to run an unspecified distribution of
frame sizes. It is possible for a product to operate at wire-speed
(theoretical maximum frame rate) at one packet size but not another.  Since
a product is likely to operate at the theoretical maximum frame rate for
large packets, it is possible to hide poor operation at small packet sizes
by creating a distribution that favors one packet size or another.  Section
9 of RFC 2544 provides details as to which frame sizes should be tested.
Anytime a report shows performance numbers for only packet size, or doesn't
include the recommended sizes, the user should question what is up. Also
section 26.1 of the RFC provides additional guidance which states 

   We assume that if a single value is desired for advertising purposes
   the vendor will select the rate for the minimum frame size for the
   media. If this is done then the figure MUST be expressed in frames
   per second.

Another way to manipulate the benchmark is to report results based upon
intended load instead of offered load.  In this case, even though the test
was intended to run at a specific data rate, the use of pause frames (for
full-duplex) or back-pressure/forward-pressure (for half-duplex) can cause
the offered load to be something completely different then intended.  If a
report doesn't cite offered loads specifically, then the assumption should
be made that intended load measurements were made and the results are once
again suspect.

The RFC also cautions that in WAN environments, tests should be made above
the maximum theoretical frame rate as header compression on a WAN link can
bump-up the maximum achievable frame rate.

RFC 2544 already does provide guidance on running tests using multiple frame
sizes.  It is in section 18 of the document.

18. Multiple frame sizes

   This document does not address the issue of testing the effects of a
   mixed frame size environment other than to suggest that if such tests
   are wanted then frames SHOULD be distributed between all of the
   listed sizes for the protocol under test.  The distribution MAY
   approximate the conditions on the network in which the DUT would be
   used. The authors do not have any idea how the results of such a test
   would be interpreted other than to directly compare multiple DUTs in
   some very specific simulated network.

I think that there is enough due diligence in the RFC such that extending
the RFC even further doesn't make sense.  Even though it is a noble goal you
are likely to open up even more wiggle room for the marketeers to bend the
rules.  If anything, I would suggest that more energy needs to be focused on
explaining how to interpret results so that people can know when someone is
trying to pull the wool over their eyes or simply making an honest mistake.

John

-----Original Message-----
From: Ramesh Menon [mailto:[email-masked]]
Sent: Friday, January 19, 2001 9:42 AM
To: [email-masked]'
Subject: [bmwg] Wire speed definition

Thanks
Ramesh Menon

---

From: Scott Bradner <[email-masked]>
Date: Fri, 19 Jan 2001 20:27:58 -0500 (EST)

> RFC 2544 and its' parent 1944 don't use the term wire-speed.

and I think that was an omission 
too many people are using the term "wire speed" in their own ways

Internet average packet size, Internet packet size mix, and minimum
sized packets are all definitions I've seen - to me its only the last
one that makes sense

Scott

---

From: Bob Mandeville <[email-masked]>
Date: Mon, 22 Jan 2001 06:25:33 -0800 (PST)

I agree with Jim who agrees with Scott.  That is why
RFC 2285 on switch testing terminology refers the
question of packet size to 2544 (and antecedent). 
There was nothing to be added or improved.

--- Jim McQuaid <[email-masked]> wrote:
> I agree with Scott.  The only meaningful definition
> is handling the maximum
> possible frame rate, i.e. the minimum size packets. 
> This is the "implied"
> definition of wire speed, even if the reality is
> quite different.
> 
> Every so often there has been discussion of
> "average" traffic or "typical"
> traffic.  It is possible to imagine coming up with
> some defined 'bag of
> frames' that represents "typical traffic" but in
> reality the consensus is
> never there.  There is no such thing as ""typical"
> traffic for testing
> purposes.  The well-defined (if artificial) traffic
> loads of 2544 and others
> are the workable, implementable and consensus ways
> to do this, it seems.
> 
> Jim McQuaid
> 
> NetIQ
> 
> From: Scott Bradner [mailto:[email-masked]]
> Sent: Friday, January 19, 2001 8:28 PM
> To: [email-masked]
> Subject: [bmwg] RE: Wire speed definition
> 
> 
> 

__________________________________________________
Do You Yahoo!?
Yahoo! Auctions - Buy the things you want at great prices. 
http://auctions.yahoo.com/

---

From: Scott Poretsky <[email-masked]>
Date: Mon, 22 Jan 2001 10:04:45 -0500

Jim/Scott B,

When referring to Wire Rate we are actually referring to the physical 
interface.  Wire rate (a.k.a. Line Rate) is used in ITU documents to 
describe the forwarding capacity of the physical interface.  Since it is 
the physical interface, the ITU describes wire rate in terms of bits per 
second independent of packet sizes.  For example, the line rate of OC-192c 
is 10Gbps, independent of the packet sizes offered to the Sonet engine from 
Layer 2 technology (PPP or GigE) and Layer 2.5 or 3 (MPLS or IP).  It is 
important that IETF maintains consistency with known standards while 
technologies such a GMPLS are developed for interoperability between MPLS 
capable routers and optical switches.

Scott P

At 07:41 AM 1/22/01 -0500, Jim McQuaid wrote:
>I agree with Scott.  The only meaningful definition is handling the maximum
>possible frame rate, i.e. the minimum size packets.  This is the "implied"
>definition of wire speed, even if the reality is quite different.
>
>Every so often there has been discussion of "average" traffic or "typical"
>traffic.  It is possible to imagine coming up with some defined 'bag of
>frames' that represents "typical traffic" but in reality the consensus is
>never there.  There is no such thing as ""typical" traffic for testing
>purposes.  The well-defined (if artificial) traffic loads of 2544 and others
>are the workable, implementable and consensus ways to do this, it seems.
>
>Jim McQuaid
>
>NetIQ
>

---

From: Jim McQuaid <[email-masked]>
Date: Mon, 22 Jan 2001 10:24:30 -0500

So never using the term 'wire rate' was a very foresighted choice, eh?

jim

-----Original Message-----
From: Scott Poretsky [mailto:[email-masked]]
Sent: Monday, January 22, 2001 10:05 AM
To: Jim McQuaid
Cc: 'Scott Bradner'; [email-masked]
Subject: RE: [bmwg] RE: Wire speed definition

Jim/Scott B,

Scott P

---

From: Scott Bradner <[email-masked]>
Date: Mon, 22 Jan 2001 11:03:52 -0500 (EST)

> We already have parameters for router packet   
> forwarding performance - Forwarding Rate and Throughput.  I hope these
> parameters provide us sufficient coverage.

what started this discussion is that vendors are now useing the term
"line rate" when talking about router performance and that the use
of this term is quite misleading since different people mean different
things by it

Scott

---

From: Jim McQuaid <[email-masked]>
Date: Mon, 22 Jan 2001 11:30:47 -0500

Let's focus on the goal.

If it is to benchmark for router performance, we have what we need.

If it is to define the term "wire rate" as it applies to router performance,
we have a candidate (maximum frame rate for smallest legal frame size) but
no specific plan to define it.

And - in the context of device testing - we don't actually care about the
raw media speed or bit rate except as it determines the max frame rate for
small packets.

Jim

-----Original Message-----
From: Scott Bradner [mailto:[email-masked]]
Sent: Monday, January 22, 2001 11:04 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; Jim McQuaid
Subject: RE: [bmwg] RE: Wire speed definition

Scott

---

From: "John Dawson" <[email-masked]>
Date: Thu, 25 Jan 2001 15:17:40 -0800

I don't believe that the RFC precludes what you are saying.

The RFC recommends typical sizes that should be tested and allows 
for a vendor to do mixed size tests that they believe may represent
a realistic network.  Therefore vendors are allowed to show lots
of details with respect to possible throughput and worst case 
throughput.

The original question however was :

  When can a vendor claim his product does wire-speed forwarding?

As I stated early, I believe that the RFC already provides guidance on 
this issue.  It is found in section 26.1.

I don't believe that it is appropriate to use a mixed frame size test 
as the only data point when attempting to establish a claim that a 
product is wire-speed capable.

John

-----Original Message-----
From: Jambi Ganbar [mailto:[email-masked]]
Sent: Thursday, January 25, 2001 2:47 PM
Cc: [email-masked]
Subject: RE: [bmwg] RE: Wire speed definition

I would like to see vendors specify typical packet rate and worst case
scenario.  If a vendor has unique features that I like and has some packet
loss at worst case and no packet loss at realistic case I believe that
they will still be considered.
 MCI is not in the business of selling routers.
Jambi
vBNS Engineering
(703) 886 1714

On Thu, 25 Jan 2001, Jim McQuaid wrote:

> Would you buy routers for MCI based on their ability to handle a "typical
> traffic definition" as determined by studies done by CAIDA?
> 
> From: Jambi Ganbar [mailto:[email-masked]]
> Sent: Thursday, January 25, 2001 3:43 PM
> Cc: [email-masked]
> I'd like to point out that while there is no typical traffic definition
> there are some studies that do map out a typical packet size distribution
> which could be used for equipment performance validation.
> Caida has published the following paper:
> http://www.caida.org/outreach/papers/AIX0005/ which should provide some
> useful data on the subject.
> I'd be happy to see more pointers for information on the subject.
> Jambi
> 
> vBNS Engineering
> (703) 886 1714
> 
> On Mon, 22 Jan 2001, Jim McQuaid wrote:
>

---

From: Ramesh Menon <[email-masked]>
Date: Thu, 25 Jan 2001 15:36:18 -0800

I posed this question originally but had to drop out of 
the thread for a bit. Jambi, thanks for guiding it back
to the original question. 

Jim, you asserted in an earlier mail:
"Let's focus on the goal. If it is to benchmark for router 
performance, we have what we need."

Routers are not the only interesting devices our there that
need benchmarking. The great thing about 2544 is that it can
used for benchmarking NICs, analyzers etc. The design (and
price point) for a lot of these are different from routers.
Some of these have no concept of forwarding and as such are
are optimized for other metrics, including price/performance. 
While it may do less than 100% at 64 byte frames it can keep 
up with every situation out there bar synthetic traffic. 

The reality on the ground is that it is not easy for engineers
in companies with marketing departments to go out and say that their
card (not *router*) can keep up with only 75% at the smallest frame size.
This is even if none of their customers would care.

Quoting Jim again "... "implied" definition of wire speed, even if the 
reality is quite different". I have spent quite a bit of time on standards 
bodies before and I would argue that if we don't take *reality* into
account,
we are quickly going to be written off to oblivion. DUT vendors that can do 
100% will report at every frame size, those that cant must have the option
to
report for a mix. I would urge this group to adopt a more pragmatic and
practical approach to this issue.

Ramesh

> From: Jambi Ganbar [mailto:[email-masked]]
> Sent: Thursday, January 25, 2001 12:43 PM
> Cc: [email-masked]

---

From: Jim McQuaid <[email-masked]>
Date: Fri, 26 Jan 2001 07:06:01 -0500

The original charter of the working group was network devices as distinct
from systems components.  So I repeat, if the goal is to benchmark routers
(or things handling routing / forwarding decisions in the "middle" of the
network) then we have what we need.

1944/2544 was not intended to be either a way to test systems or their
components, e.g. NIC cards, nor end to end services.  It may, in fact, shed
a lot of useful light on those testing & benchmarking procedures, but if
there is a requirement to benchmark these other things, let's define that,
just as IPPM split off from BMWG to look at the end to end performance
issues.

Jim McQuaid
NetIQ

-----Original Message-----
From: Ramesh Menon [mailto:[email-masked]]
Sent: Thursday, January 25, 2001 6:08 PM
To: Jim McQuaid; 'Jambi Ganbar'
Cc: [email-masked]
Subject: RE: [bmwg] RE: Wire speed definition

Ramesh

> From: Jim McQuaid [mailto:[email-masked]]
> Sent: Thursday, January 25, 2001 2:20 PM
> To: 'Jambi Ganbar'
> Cc: [email-masked]
> > purposes.  The well-defined (if artificial) traffic loads 
> of 2544 and
> others

---

From: Ramesh Menon <[email-masked]>
Date: Thu, 25 Jan 2001 15:07:51 -0800

Ramesh