Subject: [bmwg] I-D Action: draft-ietf-bmwg-2544-as-00.txt

From: [email-masked]
Date: Mon, 08 Aug 2011 07:50:39 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Benchmarking Methodology Working Group of the IETF.

	Title           : RFC 2544 Applicability Statement: Use on Real-World Networks Considered Harmful
	Author(s)       : Scott Bradner
                          Kevin Dubray
                          Jim McQuaid
                          Al Morton
	Filename        : draft-ietf-bmwg-2544-as-00.txt
	Pages           : 9
	Date            : 2011-08-06

   Benchmarking Methodology Working Group (BMWG) has been developing key
   performance metrics and laboratory test methods since 1990, and
   continues this work at present.  Recent application of the methods
   beyond their intended scope is cause for concern.  This memo
   clarifies the scope of RFC 2544 and other benchmarking work for the
   IETF community.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-2544-as-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-bmwg-2544-as-00.txt

---

From: David Newman <[email-masked]>
Date: Mon, 08 Aug 2011 09:37:39 -0700

I support this clarification and would like to see it move forward.

Two nits:

1. I have a long-standing allergy to the term "real-world networks" as
it begs the question of "real for whom?" I would much prefer this term
be replaced with "production networks" rather than perpetuating the
notion of "unreal" and "real" kinds of packets (in the latter case, one
bucket seems like way too little).

Over the past 20 years, I've repeatedly had vendors' marketing
representatives, upon learning of some suboptimal result from RFC 2544
testing, wave their hands and complain "that's not real world!" My
explanation that some one-size-fits-all definition of reality was never
the goal, and that we're instead trying to describe the limits of DUT
performance, doesn't seem to help. When I ask what they'd instead like
to see, the response is usually some equally nonrandom definition of
reality carefully crafted to make the DUT perform better.

I think a division between test and production networks would be cleaner.

(I am aware that RFCs 1242 and 2544 use the term "real world." I wish
they didn't; aside from the white-box terminology split for latency in
1242, no other concept has caused more grief with these documents.)

2. Sections 4.2 and 5 refer to overload conditions in a throughput test.
Overloads are possible in running a search algorithm, but the throughput
rate itself is by definition not an overload condition. A DUT with
line-rate throughput is never overloaded in this test.

Suggested alternative wording for section 4.2:

RFC 2544 methods, specifically to determine Throughput as defined in
   [RFC1242] and other benchmarks, may overload the
   resources of the device under test, and may cause failure modes in
   the device under test.

Section 5:

   Operating test equipment on production networks according to the
   methods described in [RFC2544], where overload is a possible outcome,
   would no doubt be harmful to user traffic performance.

dn

On 8/8/11 7:50 AM, [email-masked] wrote:
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: Al Morton <[email-masked]>
Date: Sun, 14 Aug 2011 22:19:17 -0400

Hi David,

Thanks for your careful review of the AS.
Both your points below have merit and deserve a bit more
thought/discussion, especially the title/terminology (point 1 below).

We discussed "real-world" vs. "production" networks among the co-authors
prior to releasing the initial version of this memo,
and ended-up using the RFC2544 term with explanation. But since
this memo is about clarifying RFC2544, I can see the value of using
a more concise term, especially in the title.

On overload conditions, I agree with your suggested changes.

Other opinions?
Al
(as co-author)

At 12:37 PM 8/8/2011, David Newman wrote:
> > A New Internet-Draft is available from the on-line 
> Internet-Drafts directories. This draft is a work item of the 
> Benchmarking Methodology Working Group of the IETF.
> >