Subject: [bmwg] Draft draft-ietf-bmwg-mcastm-10.txt Last Call

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 22 Jan 2003 15:13:04 -0800

It seems that we have been her before.  Once again I have concerns that the
"Methodology for IP Multicast Benchmarking" draft is not ready.  Some of
these go back to the 1999 meetings.  I would like to open this up for
rebuttal, not from just the authors, but the entire BMWG community:

1. Single ingress port.  Section 3 (end of first paragraph) state
"Methodologies for multiple ingress, multiple egress scenarios are beyond
the scope of this document."

Every single multicast test I have been asked to do has required either
multiple ingress or multiple egress ports.  I have yet run across someone
who wants to benchmark only two ports on a DUT/SUT.

The question I have is two part: A.) Does anyone has a real world experience
where you would ONLY benchmark two ports?  B.) Should this draft be expanded
to include the common scenarios, or another draft be written to cover these
scenarios?

2. Ambiguous.  If you go back in the archives, you will find that there have
been issues with the drafts ambiguity in sections 4.1, 8.1, and 8.2 since
1999.  Back in 1999, we had appendix A to give a hint of what to expect the
offered load to look like.  Without appendix A (or the algorithms used to
make appendix A), how can one predict what packets will be offered to the
DUT/SUT?

Since the packets are to be offered can not be predicted, how do we know
that two scripts provided by two vendors will give similar results on the
same DUT/SUT?  Since they both comply with the ambiguous methodology, both
results are right.  Can someone argue that a DUT/SUT will always give the
same throughput no matter how unicast and multicast packets are offered?

3. RFC2432 harmony.  In reading RFC2432, I found a few things
different/missing in the methodology draft.  In RFC2432, the Aggregated
Multicast Throughput talks about rate as a function of number of destination
ports.  While the methodology's procedure has a fixed offered load and
varies the ports as a function of OLOAD.  Which variable is fixed, the
number of ports or the offered load?

Also RFC2432 has sections labeled "Issues:".  I can not find where RFC 2432
section "3.2.3 Aggregated Multicast Throughput" Issues are addressed in the
draft.  They are not addressed in draft's section "4.3 Aggregated Multicast
Throughput".  Do the authors need to follow RFC2432?

4. Plagiarism. Most of the draft's objectives is a copy the RFC2432's
definitions.  Section 8 is a copy of RFC 2432 section 3.6 (minus the grammar
error I pointed out in 1999).  Section 8's paragraph serves no propose if
the reader reads and follows section 1's instructions (second paragraph).

While there is no technical issue here, I would like to raise a moral one.
My professors frown on submitting any work that was not original.
Submitting other people work was grounds for expulsion.  As educated
authors, how much of the previous work we should copy and label our own?

Jerry.

___________________________________________________________ 
Jerry Perser                         Spirent Communications
Director of Test Methodology         SmartBits Division 
Phone: 818.676.2320                  26750 Agoura Road 
Lab:   818.676.2337                  Calabasas, CA, 91302 
Cell:  818.292.6457                  Corp:  818.676.2300
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Debby Stopp <[email-masked]>
Date: Thu, 23 Jan 2003 09:09:32 -0800

> the scope of this document."

The scope of this draft is not intended to be a test plan; rather, the
intent is to provide a methodology w/meaningful & repeatable metrics.  The
test vendors or test designers can utilize the methodologies in this draft
as they need to in order to determine & characterize the behaviour of the
SUT.  As always, the SUT configuration should be included as part of the
test description.

> to include the common scenarios, or another draft be written

Interesting.  This draft does not state that any of these methodologies are
restricted to benchmarking two ports; please refer to Figure 1 as an example
of a typical setup that can be scaled to accommodate any test plan.

> make appendix A), how can one predict what packets will be

Hmm.  I looked into the archives because I seem to recall that you actually
approved of the latest version of the draft and found that on Thu 10/24/2002
6:38 PM, you wrote:
"Debby,

The wording looks good.  I have one small change.  Can we change "multicast
class frames vs. unicast class frames" to "multicast class frames and
unicast class frames"?  The versus reminds me of battle-bots.  If they are
fighting for the same bandwidth, then this becomes a Differentiated Services
methodology.

Jerry."

In addition, on Mon 12/17/2001 4:07 PM, you suggested we remove appendix A:
"I have a solution to the Appendix A issue.  If we remove Appendix A, there
are no algorithms or equations lost.  The current Appendix is just restating
the bmwg charter.  The only reference to Appendix A is the Mixed Class
Throughput test."

You also suggested some alternate wording, which I have incorporated into
the document.

Perhaps you don't have the latest version of the draft?  You can find it at
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-mcastm-10.txt
<http://www.ietf.org/internet-drafts/draft-ietf-bmwg-mcastm-10.txt>  .
Sections 8.1 & 8.2 have been completely redone and I have removed all
ambiguity regarding methodologies & metrics.

> number of ports or the offered load?

Section 3.2.3 Aggregated Multicast Throughput (AMT) from 2434 was intended
to be utilized as a scaling exercise to determine how well a SUT/DUT will
behave under varying conditions.  In the mcast draft, section 4.3 addresses
this methodology:
"If any frame loss is detected, one receive port MUST leave the group(s) and
the sender will transmit again.".

In answer to your question, the number of ports are varied while the offered
load stays the same.

************

We can go over the details over coffee one morning if you'd like; however I
suggest that we've probably taken up enough bandwidth over this issue and we
should move on to subjects of more technical interest.

Debby Stopp 
Principal Software Engineer
(818) 444-2365<?xml:namespace prefix = o ns =
"urn:schemas-microsoft-com:office:office" />

IXIA
26601 W. Agoura Rd. 
Calabasas 
CA 91302 
Tel: 818 871 1800 (US) +1 818 871 1800 (international) 

[email-masked]
http://www.ixiacom.com <http://www.ixiacom.com/>

---

From: Debby Stopp <[email-masked]>
Date: Thu, 23 Jan 2003 09:09:32 -0800

Jerry."

************

Debby Stopp 

IXIA
26601 W. Agoura Rd. 
Calabasas 
CA 91302 

[email-masked]

---

From: "Perser, Jerry" <[email-masked]>
Date: Thu, 23 Jan 2003 11:49:21 -0800

Rebuttal inline:

>The scope of this draft is not intended to be a test plan; rather, 
>the intent is to provide a methodology w/meaningful & repeatable 
>metrics.  The test vendors or test designers can utilize the
>methodologies in this draft as they need to in order to determine & 
>characterize the behavior of the SUT.  As always, the SUT 
>configuration should be included as part of the test description.

Please clarify how you define "test plan" versus "methodology".  What I am
looking for is a methodology that can help me write the test plan.  For
example, I have a switch with N ports, the ratio of multicast to unicast is
A:B.  Per figure 1, what order does the source transmit frames?  I am
failing to extract this methodology from the draft.  Please respond with
section, paragraph, sentence in the draft that answers this methodology
question.

>Interesting.  This draft does not state that any of these 
>methodologies are restricted to benchmarking two ports; please refer 
>to Figure 1 as an example of a typical setup that can be scaled to 
>accommodate any test plan.

Read the paragraph between "3. Test set up" and figure 1.  It states
"Methodologies for multiple ingress, multiple egress scenarios are beyond
the scope of this document."  Since figure 1 (and figure 2) have multiple
egress, they fall under the scenario "beyond the scope of this document".

A zero, one, or two port test is the only setup that falls under "the scope
of this document".  I exclude a zero and one port test as meaningless.
Please show me how to get more than two ports and still stay within "the
scope of this document" (per section 3).

>Hmm.  I looked into the archives because I seem to recall that you 
>actually approved of the latest version of the draft and found that 
>on Thu 10/24/2002 6:38 PM, you wrote:

Words of encouragement does not constitute approval.  It's heading in the
right direction.  I am still concerned about the lack of detail in the
methodology.  Any reason for not including my text change of "versus" to
"and"?

>the bmwg charter.  The only reference to Appendix A is the Mixed Class

Sarcasm on the reflector biting me on the ass again.  Go back to 1999 at the
46th IETF meeting.  I asked for detail on Appendix A.  Between 1999 and
2001, no details, then in version 4 you gutted Appendix A.  I also accused
you of writing a IXIA proprietary methodology.

I apologize for the sarcasm and the accusation.  What I was looking for in
1999, 2001, and now in 2003 is a detail methodology for mixing unicast and
multicast traffic that: A.)any test vendor can implement; B.) different
implementations will produce similar results.  I know of two implementation
that will comply with this draft, but have two different methodologies that
will result in different results.

>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-mcastm-10.txt .
Sections 8.1 
>& 8.2 have been completely redone and I have removed all ambiguity
regarding 
>methodologies & metrics.

My version is labeled draft-ietf-bmwg-mcastm-10.txt.  Did you post multiple
version 10 (like you did on version 8)?  And yet it still does answer the
questions I asked Tue, 2 Oct 2001 08:43:43.  Please show me where these
questions are answered in the draft:

	Where does setup A join?
	Is setup A transmitting multicast packets?
	Is setup B transmitting multicast packets?
	Is the traffic homogeneous or nonhomogeneous?  
	When does setup B join?  
	Does the statement "then setup B traffic will join multicast group
addresses
	not joined by the ports in this setup", refer to setup A or only
setup B
	ports "not joined"?  
	Does setup B ever send multicast packets?

I am having a hard time implementing these sections.  What would be helpful
is the level of detail found in section 5.1 Multicast Latency.  I am getting
the sense that sections 8.1 and 8.2 can not be implemented due to the lack
of methodology.  Any comments on why, as both author of the draft and author
of TCL multicast scripts, you have yet implemented these sections?

>Section 3.2.3 Aggregated Multicast Throughput (AMT) from 2434 was intended
to be 
>utilized as a scaling exercise to determine how well a SUT/DUT will behave
under 
>varying conditions.  In the mcast draft, section 4.3 addresses this
methodology:
>"If any frame loss is detected, one receive port MUST leave the group(s)
and the 
>sender will transmit again.".
>
>In answer to your question, the number of ports are varied while the
offered load >stays the same. 

Kevin Dubray, is this correct?  If so, we need to change RFC2432 to reflect
this.

Debbie, you missed the next paragraph (before I talk about Plagiarism).
RFC2432 has "Issues" that are be resolved in the Methodology.  Can you point
to where in the draft you addressed them.

>should move on to subjects of more technical interest.

I meet at Starbucks 8AM on Fridays.  Talk is usually non-business. If you
show up, Michele and I can change the subject.

Jerry.