Subject: RE: [bmwg] Draft draft-ietf-bmwg-mcastm-10.txt Last Call

From: "Perser, Jerry" <[email-masked]>
Date: Thu, 23 Jan 2003 11:49:21 -0800

Rebuttal inline:

>The scope of this draft is not intended to be a test plan; rather, 
>the intent is to provide a methodology w/meaningful & repeatable 
>metrics.  The test vendors or test designers can utilize the
>methodologies in this draft as they need to in order to determine & 
>characterize the behavior of the SUT.  As always, the SUT 
>configuration should be included as part of the test description.

Please clarify how you define "test plan" versus "methodology".  What I am
looking for is a methodology that can help me write the test plan.  For
example, I have a switch with N ports, the ratio of multicast to unicast is
A:B.  Per figure 1, what order does the source transmit frames?  I am
failing to extract this methodology from the draft.  Please respond with
section, paragraph, sentence in the draft that answers this methodology
question.

>Interesting.  This draft does not state that any of these 
>methodologies are restricted to benchmarking two ports; please refer 
>to Figure 1 as an example of a typical setup that can be scaled to 
>accommodate any test plan.

Read the paragraph between "3. Test set up" and figure 1.  It states
"Methodologies for multiple ingress, multiple egress scenarios are beyond
the scope of this document."  Since figure 1 (and figure 2) have multiple
egress, they fall under the scenario "beyond the scope of this document".

A zero, one, or two port test is the only setup that falls under "the scope
of this document".  I exclude a zero and one port test as meaningless.
Please show me how to get more than two ports and still stay within "the
scope of this document" (per section 3).

>Hmm.  I looked into the archives because I seem to recall that you 
>actually approved of the latest version of the draft and found that 
>on Thu 10/24/2002 6:38 PM, you wrote:

Words of encouragement does not constitute approval.  It's heading in the
right direction.  I am still concerned about the lack of detail in the
methodology.  Any reason for not including my text change of "versus" to
"and"?

>In addition, on Mon 12/17/2001 4:07 PM, you suggested we remove appendix A:
>"I have a solution to the Appendix A issue.  If we remove Appendix A, there
>are no algorithms or equations lost.  The current Appendix is just
restating
>the bmwg charter.  The only reference to Appendix A is the Mixed Class
>Throughput test."

Sarcasm on the reflector biting me on the ass again.  Go back to 1999 at the
46th IETF meeting.  I asked for detail on Appendix A.  Between 1999 and
2001, no details, then in version 4 you gutted Appendix A.  I also accused
you of writing a IXIA proprietary methodology.

I apologize for the sarcasm and the accusation.  What I was looking for in
1999, 2001, and now in 2003 is a detail methodology for mixing unicast and
multicast traffic that: A.)any test vendor can implement; B.) different
implementations will produce similar results.  I know of two implementation
that will comply with this draft, but have two different methodologies that
will result in different results.

>Perhaps you don't have the latest version of the draft?  You can find it at

>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-mcastm-10.txt .
Sections 8.1 
>& 8.2 have been completely redone and I have removed all ambiguity
regarding 
>methodologies & metrics.

My version is labeled draft-ietf-bmwg-mcastm-10.txt.  Did you post multiple
version 10 (like you did on version 8)?  And yet it still does answer the
questions I asked Tue, 2 Oct 2001 08:43:43.  Please show me where these
questions are answered in the draft:

	Where does setup A join?
	Is setup A transmitting multicast packets?
	Is setup B transmitting multicast packets?
	Is the traffic homogeneous or nonhomogeneous?  
	When does setup B join?  
	Does the statement "then setup B traffic will join multicast group
addresses
	not joined by the ports in this setup", refer to setup A or only
setup B
	ports "not joined"?  
	Does setup B ever send multicast packets?

I am having a hard time implementing these sections.  What would be helpful
is the level of detail found in section 5.1 Multicast Latency.  I am getting
the sense that sections 8.1 and 8.2 can not be implemented due to the lack
of methodology.  Any comments on why, as both author of the draft and author
of TCL multicast scripts, you have yet implemented these sections?

>Section 3.2.3 Aggregated Multicast Throughput (AMT) from 2434 was intended
to be 
>utilized as a scaling exercise to determine how well a SUT/DUT will behave
under 
>varying conditions.  In the mcast draft, section 4.3 addresses this
methodology:
>"If any frame loss is detected, one receive port MUST leave the group(s)
and the 
>sender will transmit again.".
>
>In answer to your question, the number of ports are varied while the
offered load >stays the same. 

Kevin Dubray, is this correct?  If so, we need to change RFC2432 to reflect
this.

Debbie, you missed the next paragraph (before I talk about Plagiarism).
RFC2432 has "Issues" that are be resolved in the Methodology.  Can you point
to where in the draft you addressed them.

>We can go over the details over coffee one morning if you'd like; however I

>suggest that we've probably taken up enough bandwidth over this issue and
we 
>should move on to subjects of more technical interest.

I meet at Starbucks 8AM on Fridays.  Talk is usually non-business. If you
show up, Michele and I can change the subject.

Jerry.

___________________________________________________________ 
Jerry Perser                         Spirent Communications
Director of Test Methodology         SmartBits Division 
Phone: 818.676.2320                  26750 Agoura Road 
Lab:   818.676.2337                  Calabasas, CA, 91302 
Cell:  818.292.6457                  Corp:  818.676.2300

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "David Newman" <[email-masked]>
Date: Thu, 23 Jan 2003 12:13:59 -0800

> A.)any test vendor can implement; B.) different
> implementations will produce similar results.

Not similar. Identical.

Repeatability is one of the bedrock requirements of any benchmark. We've got
to work together to eliminate ambiguities in the current ID that would lead
to nonrepeatable results.

dn

---

From: "David Newman" <[email-masked]>
Date: Thu, 23 Jan 2003 12:13:59 -0800

Not similar. Identical.

dn