Subject: [bmwg] WG Last Call: draft-ietf-bmwg-mcastm-10.txt

From: Kevin Dubray <[email-masked]>
Date: Wed, 15 Jan 2003 09:27:22 -0500

BMWG'ers:

A WG Last Call period for the Internet-Draft on
       "Methodology for IP Multicast Benchmarking,"

       <draft-ietf-bmwg-mcastm-10.txt>

will be open from 15 January 2003 through 30 January 2003.

The editor feels this latest version of the I-D has addressed all 
outstanding feedback from the previous WG Last Calls.

If you feel that this draft should not be given to the Area Directors
for consideration in progressing the I-D to an Informational RFC,
please cite your basis to this list or [email-masked]. Weight
will be given to those issues for which alternative solutions
are offered.  This must be done prior to close of the last call
period. Of course, notes of support are always welcome.

A URL for the Internet-Draft is:

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-mcastm-10.txt

-Kevin

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: taha <[email-masked]>
Date: Tue, 28 Jan 2003 11:11:36 -0800 (PST)

This document has been updated several times in response to the
given comments, and is now in a position to be reviewed for possible
consideration in becoming an informational RFC.

      Ali Taha
      Research Assistant
      Communication Sciences Institute
      University of Southern California
      Email: [email-masked]

> -----Original Message-----
> From: Kevin Dubray [mailto:[email-masked]]
> Sent: Wednesday, January 15, 2003 6:27 AM
> To: [email-masked]
> Subject: [bmwg] WG Last Call: draft-ietf-bmwg-mcastm-10.txt
>
>
>
> BMWG'ers:
>

---

From: "Perser, Jerry" <[email-masked]>
Date: Tue, 28 Jan 2003 12:50:15 -0800

On the subject of meaningful, the Aggregated Multicast Throughput
methodology does not agree with RFC2432.  

In reading RFC2432, several of us interpret the definition (and the
discussion) as the number of ports to be fixed while the offered load is
varied to find the maximum zero loss point.  Yet the draft (and the draft's
authors) state that the offered load is fixed and the number of ports is
varied.  The opposite of RFC2432.

I have asked the author of RFC2432 to clarify which is the independent and
dependent variable.  When this is clarified, we can then discuss if the
methodology is meaningful.  Ali, do you have an opinion on which way the
methodology should follow?

Jerry.

> From: David Newman [mailto:[email-masked]]
> Sent: Tuesday, January 28, 2003 12:23 PM
> To: [email-masked]
> Subject: RE: FW: [bmwg] WG Last Call: draft-ietf-bmwg-mcastm-10.txt
> 
> 
> 
> There are multiple open concerns about ambiguities in draft 
> 10. The result
> of these ambiguities will be that different testers will 
> obtain different
> results, even though each complies with the draft's procedures.
> 
> That the document has been updated several times shows 
> admirable persistence
> on the part of the authors, but it should not be the standard 
> for deciding
> whether to move forward.
> 
> Let's instead ask the question: Is a given benchmark 
> repeatable, stressful,
> and meaningful?
> 
> Unfortunately, on the issue of repeatability I do not believe 
> we are quite
> there yet with draft 10. I will let others comment on the other two
> requirements.
> 
> dn
> 
>

---

From: "Hickman, Brooks" <[email-masked]>
Date: Wed, 29 Jan 2003 12:44:54 -0800

I have been reviewing the methodology and have the following comments. I
have not finished reviewing the remaining sections, but wanted to give some
feedback sooner(Should have been before last call) than later.

Section 4.1. Mixed Class Throughput

I believe the main point in regards to the ambiguity is the fact that the
methodology does not address whether or not a test port may be BOTH a
mulitcast source and unicast source. In such a case, defining the ratio of
the two traffic classes does not address how the two classes are mixed
within the same aggregate traffic stream. I have reviewed the thread that
Jerry Perser submitted on the reflector regarding the mixed class test:

http://www1.ietf.org/mail-archive/working-groups/bmwg/current/msg00139.html

In Jerry Perser's suggestion, the methodology would specify that:

"The MT group SHOULD not join any multicast groups or transmit unicast
frames."

This would make the mixed class aggregate stream applicable only on the
egress side and would therefore not require the methodology to define how
the "tester" would mixed the multicast and unicast traffic.

"A minimum of one multicast transmit port MUST be configured to transmit
multicast traffic to a DUT/SUT port that is configured to receive multicast
traffic." 

Section 3 specifies that the methodologies for multiple ingress, multiple
egress scenarios are beyond the scope of this document. If this is the case,
I would suggest changing the wording to "One multicast port MUST..." so as
to not contradict section 3. The sentence implies testing with multiple
multicast transmit ports. This is minor, but a little confusing. 

"Throughput measurement is defined in RFC1242 [Br91]. A search algorithm
MUST be utilized to determine the maximum offered frame rate with a zero
frame loss rate."

RFC1242 does not specifically address multiple classes of traffic. I would
suggest elaborating on this a little. For example, are both the multicast
traffic AND unicast traffic varied to find the throughput or only the
multicast traffic? While it may be inferred by defining the ratio,
specifying how the two classes of traffic are varied may make it a little
less confusing. 

Also, the results section specify what measurements to make, but I did not
see anything regarding the reporting format.

4.2.  Scaled Group Forwarding Matrix

The methodology states:

"On each iteration, the receive ports SHOULD incrementally join 10 multicast
groups until a user defined maximum number of groups is reached."

I would suggest making the number of multicast groups that are joined on
each iteration a set up parameter and that the parameter be reported. 

"In addition, all other reporting parameters of the scaled group forwarding
matrix methodology MUST be reflected in the results report, such as the
transmitted packet size(s) and offered load of the packet stream for each
source port."

The "all others" and "MUST" wording has me wondering if there are other
parameters, besides the ones that are specifically mentioned, that need to
be included. 

4.3. Aggregated Multicast Throughput

After reviewing both the terminology and the methodology document, I believe
the procedure in the methodology needs to be changed to reflect the offered
load being varied to determine the throughput while keeping the number of
destination interfaces constant. Would suggest that the procedure also
include mentioning that testing SHOULD be repeated with different numbers of
multicast destination interfaces.

---

From: "Hickman, Brooks" <[email-masked]>
Date: Wed, 29 Jan 2003 12:44:54 -0800

The methodology states:

---

From: "Debby Stopp" <[email-masked]>
Date: Thu, 30 Jan 2003 16:47:03 -0800

I'm home sick today, so I can't respond to all comments but I did notice
something from a previous thread:

----- Original Message -----
From: "Hickman, Brooks" <[email-masked]>

> specifying how the two classes of traffic are varied may make it a little

This was discussed at length back in Oct 2001 - please see thread
http://www.ietf.org/mail-archive/working-groups/bmwg/current/thrd10.html for
details.  I believe that if you combine:
     The offered load per each DUT/SUT port MUST not exceed the maximum
     bandwidth capacity of any configured receive DUT/SUT ports.

with

     Throughput measurement is defined in RFC1242 [Br91]. A search
     algorithm MUST be utilized to determine the maximum offered frame
     rate with a zero frame loss rate.

it is implied that you must vary the offered load, & again, that would
depend on whether you were chosing:

          a) As an independent rate for unicast class and multicast
             class of traffic OR
          b) As an aggregate rate comprised of a ratio of multicast
             class to unicast class of traffic.

as to how that offered load was varied.

Debby Stopp 
Principal Software Engineer
(818) 444-2365

IXIA
26601 W. Agoura Rd. 
Calabasas 
CA 91302 
Tel: 818 871 1800 (US) +1 818 871 1800 (international) 

[email-masked]