Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

From: "Jay Karthik \(jakarthi\)" <[email-masked]>
Date: Mon, 28 Aug 2006 21:54:51 -0400

Hello Authors,

I had the following comments on the terminology ID. I will follow up
with my comments on the methodology draft shortly.

Most of the comments are just nits.

- Introduction : "faster test duration" could be replaced by "shorter
test duration"

- Introduction :  "Multiple benchmarks are made for each Benchmark Plane
during each Phase."  I suggest replacing 'made' with 'proposed' or
'suggested'.

- Need to fix "These benchmarks White Box benchmarks ..... " in
Introduction.

- The white box benchmarks could be a very important for many and I
recommend adding the CPU Utilization and Available Memory benchmarks to
Table 1 for all 3 phases.

- 3.1.2 Configuration Sets. We should mention forwarding rate, besides
feature and scaling limit in the definition of the term.

- I am wondering if we need a new term called "Peak Load" or something
else to indicate this, which is basically to denote a sample
configuration set beyond which the device under test becomes unusable or
unresponsive to any command line instruction. If you decide to add this,
another usual term would be "duration of unresponsiveness" which should
be ideally 0.

Cheers -Jay

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Monday, July 17, 2006 9:31 AM
To: [email-masked]
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

A WG Last Call period for the Internet-Drafts on

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking

will be open from 17 July 2006 through 28 August 2006 (6 weeks).

URLs for these Internet-Drafts are:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.tx
t

These versions incorporate comments resulting from implementation.
The authors believe that all comments from the working group have been
addressed, dating back to June 2003 when version 00 of the terminology
draft was published.

These drafts are entering the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
At this time, we are seeking volunteers to complete the BMWG Last Call
Review Template in the message of 4/27/04 titled, "Refinements to the
BMWG Last Call Process".
Volunteers should contact [email-masked] A copy of the template is
available here:
http://home.comcast.net/~acmacm/IDcheck/LastCallTemplate.txt

Whether or not you decide to volunteer for the directed review, please
*read* and express your opinion on whether or not this Internet-Draft
should be given to the Area Directors for consideration as an
Informational RFC.  Send your comments to this list or [email-masked] 

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Arun Gandhi" <[email-masked]>
Date: Tue, 29 Aug 2006 11:54:44 -0400

Hello Authors,

I am little late in responding wrt the open dates but I suppose my feedback
will get evaluated. I will also follow up with the feedback on methodology
document shortly.

<-- Section 1 (Introduction): 

In general I believe this draft should not just address multiple protocols
operational simultaneously but "different services" being overlaid in a
fully operational box.

> Routers in an operational network are simultaneously configured with
> multiple protocols and security policies while forwarding traffic and
> being managed.

Arun> IMHO, replace "protocols" with "services" as in ISP world (which is
Arun> the motivation for this draft) there may be different service sets
Arun> such as l3vpn, l2vpns, VPLS, etc. along with routing and signaling
Arun> protocols configured.

> It is helpful to accelerate these network operational conditions so 
> that the router under test can be benchmarked with faster test 
> duration.

Arun> This needs revision/clarification.

> Multiple benchmarks are made for each Benchmark Plane during each 
> Phase.

Arun> Benchmarks are defined for each plane to be characterized in all 
Arun> phases.

> These benchmarks White Box benchmarks are provided in Appendix 1 for
> additional DUT behavior measurements.

Arun> Two points:
Arun> 1) This line needs revision.
Arun> 2) A new term "White" box is added without being explained. Also, 
Arun> the impression I get reading this draft is that it's a black box
Arun> test. Also, since the motivation is the benchmarking of a router
Arun> in ISP domain then for sure it is a black box testing. Regardless
Arun> we should monitor/characterize other system parameters such as 
Arun> "memory usage", "CPU utilization", "install/bring up times", and 
Arun> so on.

<-- Section 3.2.1 (Control Plane)

> The Control Plane defines the Configuration, Startup Conditions, and
> Instability Conditions of the control protocols. Control Plane 
> protocols may include routing protocols, multicast protocols, and 
> MPLS protocols.     

Arun> IMHO, this statement should be revised to be "The Control Plane 
Arun> defines the Configuration, Startup Conditions, and Instability
Arun> Conditions of the control plane protocols, which may include 
Arun> routing protocols, multicast protocols, and MPLS protocols.  

<-- Section 3.3.2.3 (Control Plane)

> Measurement units: sessions

Arun> IMHO, this should be replaced with "number" (as "sessions" in not
Arun> a measuring unit) 

Thanks,
Arun

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Monday, July 17, 2006 9:31 AM
To: [email-masked]
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking will be open from 17 July 2006 through
28 August 2006 (6 weeks).

URLs for these Internet-Drafts are:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.txt
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.txt