Subject: RE: [bmwg] bmwg-multicast document review

From: Debby Stopp <[email-masked]>
Date: Thu, 15 Nov 2001 16:12:35 -0800

See my comments below.

> >>  Section 3.1. Test Considerations 
>    
> >> Moreover, no provisions are 
> >> made for traffic-affecting factors, such as congestion control or 
> >> service differentiation mechanisms.  Modifications to the 
> specified 
> >> collection procedures might need to be made to accommodate the 
> >> transmission media actually tested.  These accommodations MUST be 
> >> presented with the test results. 
> I don't agree with this language.  We have suffered for too 
> many years with throughput and performance 
> tests that allow congestion control and forward pressure 
> techniques to skew results.  It is time to close 
> these holes.  This paragraph essentially grants license to 
> change any collection procedures to accommodate 
> some designer agenda.  If the procedures laid out in this RFC 
> can not be used to characterized a specific 
> media, then this RFC should not be used as a basis for 
> analyzing that technology.  Although the authors 
> specify that any accommodations must be presented with the 
> test results, this action allows anyone to 
> re-engineer the test process and then leaves it to the 
> ingenuity of the reader to catch the tricks that may 
> be employed to mis-direct the less educated. 
> Are we agreeing to a process here? or are we granting 
> everyone free license to re-write their own test? 

This paragraph specifically worded to allow someone to use the methodology
w/out regard to media.  The intention is not to overload the receive ports &
thus deal w/congestion & flow control <there's already an RFC for that>, but
rather to measure the ability to forward IPMC traffic in a particular
scenario.

Thus, I will add a sentence to this section stating that these tests MUST be
run w/out flow control, etc, like this:

********
3.1.	Test Considerations

The procedures outlined below are written without regard for specific
physical layer or link layer protocols. The methodology further assumes a
uniform medium topology. Issues regarding mixed transmission media, such as
speed mismatch, headers differences, etc., are not specifically addressed.
Flow control, QoS and other traffic-affecting mechanisms MUST be disabled.
Modifications to the specified collection procedures might need to be made
to accommodate the transmission media actually tested.  These accommodations
MUST be presented with the test results.
*********

> >> Section 4.1 Mixed Class Throughput 
> >> Result 
> >> Parameters to be measured SHOULD include the frame loss 
> and percent 
> >> loss for each class of traffic per destination port.  The ratio of 
> >> unicast traffic to multicast traffic MUST be reported. 
> Let's specify what must be measured, not what should be measured. 

I can reword this section <4.1> as follows:
********************************
Result

Parameters to be measured SHOULD include the frame loss and percent loss for
each class of traffic per destination port.  The ratio of unicast traffic to
multicast traffic MUST be reported.

The nature of the traffic stream contributing to the result MUST be
reported, specifically number of source and destination ports within the
multicast group. Parameters to be measured MUST include number of multicast
frames offered, number of unicast frames offered, number of multicast frames
received and number of unicast frames received.  In addition, all other
reporting parameters of mixed class throughput MUST be reflected in the
results report, such as the transmitted packet size(s) and the granularity
of the measurement process.

Results for mixed class throughput MUST include the following parameters:
the maximum aggregate offered frames per second from the source ports for
both multicast and unicast frames at which no loss forwarding is observed
and the ratio of unicast traffic to multicast traffic at each destination
port.

********************************

Note - as the test duration must be included in all results reports, I can
reword this section as follows:
********************************
3.1.5.	Trial Duration

The duration of the test portion of each trial SHOULD be at least 30
seconds.  This parameter MUST be included as part of the results reporting
for each methodology.
********************************

I prefer to put this in a more generalized section, as I believe all tests
should always report the test duration and I prefer not to cut and paste the
same words throughout the document.

For section 4.2, I have reworded as follows:
********************************
Results

Parameters to be measured MUST include the offered load and forwarding rate
as a function of the total number of multicast groups for each test
iteration.

The nature of the traffic stream contributing to the result MUST be
reported, specifically number of source and destination ports within the
multicast group.  In addition, all other reporting parameters of the scaled
group forwarding matrix methodology MUST be reflected in the results report,
such as the transmitted packet size(s) and offered load of the packet stream
for each source port.

Result reports MUST include the following parameters for each iteration: the
number of frames offered, number of frames received per each group, number
of multicast groups and forwarding rate, in frames per second.  Constructing
a table that contains the forwarding rate vs. number of groups is desirable.

********
You had some comments <as did others> re: the aggregated test; here's a
revised version, see draft for revised section.
********

> >> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput 

this section has changed significantly based on Kevin's comments - please
review again.

> >> Section  5. Forwarding Latency 

This section has been reworded a bit thanks to Kevin's comments, see draft.

Debby Stopp 
Principal Software Engineer

IXIA
26601 W. Agoura Rd. 
Calabasas 
CA 91302 
Tel: 818 871 1800 (US) +1 818 871 1800 (international) 

[email-masked]
http://www.ixiacom.com 

_______________________________________________
bmwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/bmwg

---

From: Debby Stopp <[email-masked]>
Date: Thu, 15 Nov 2001 16:12:35 -0800

See my comments below.

********
3.1.	Test Considerations

********

Debby Stopp 

IXIA
26601 W. Agoura Rd. 
Calabasas 
CA 91302 

[email-masked]
http://www.ixiacom.com

---

From: "John Dawson" <[email-masked]>
Date: Thu, 15 Nov 2001 17:30:28 -0800

>> Parameters to be measured MUST include number of multicast frames
offered,
>> number of unicast frames offered, number of multicast frames received and

>> number of unicast frames received.

You are assuming that test duration is a constant.  This is not true.  Test 
duration (Time in seconds) is a measured quantity.  This is the
differentiator
between OfferedLoad and IntendedLoad.  You can not report offered load
unless 
time is a measurement.  All methodology MUST include a measurement of the
TIME
it takes to transmit the offered frames.

If you measure time at the receive side, then and only then is the
forwarding 
rate accurately known.

By including Time as a required measurement parameter, you will
automatically
address the issues associated with flow control and forward/back-pressure.

I do not believe that it is appropriate to put test duration into the
general
overhead section.  Test duration MUST be measured, but SHOULD also be
reported.
If you choose not to report time directly, then time can be deduced by
calculating
backwards from the Offered Load.  Time == FramesOffered/OfferedLoad.

It is also important to note that there are two methods that have been
suggested
as a valid methodology for determining offered Load.  One is based upon
transmitting
a fixed quantity of packets and measuring the time required to transmit the
packets.
If multiple ports are transmitting, then TIME must be measured on each port.
The
other method, is to transmit packets for a fixed time.  The number of
packets
transmitted by each port then becomes the measured quantity.

I prefer the fixed TIME approach as it represents a uniform loading
(pressure) for
all ports across a given test.  If a fixed quantity of packets are used,
then the
test can be very non-uniform.  For instance, assume that half-duplex systems
are
being tested using bi-directional traffic and further assume that one port
captures
the wire.  In this scenario, you actually have two separate uni-directional
streams
where the first stream transmits for the first half of the test and the
other stream
transmits for the second half of the test (after the first stream goes
idle).  In
this scenario, the offered loads are not what you intended.

Unless you make TIME a parameter that MUST be measured, you can not
determine the
offered load -- you will never know what happened during the test and you
will be
reporting bogus numbers.

John

-----Original Message-----
From: Debby Stopp [mailto:[email-masked]]
Sent: Thursday, November 15, 2001 4:13 PM
To: Dawson, John [BLVWA:382:EXCH]; [email-masked]
Subject: RE: [bmwg] bmwg-multicast document review

See my comments below.

********
3.1.	Test Considerations

********

Debby Stopp 

IXIA
26601 W. Agoura Rd. 
Calabasas 
CA 91302 

[email-masked]
http://www.ixiacom.com

---

From: "John Dawson" <[email-masked]>
Date: Thu, 15 Nov 2001 17:30:28 -0800

offered,

John

-----Original Message-----
From: Debby Stopp [mailto:[email-masked]]
Sent: Thursday, November 15, 2001 4:13 PM
To: Dawson, John [BLVWA:382:EXCH]; [email-masked]
Subject: RE: [bmwg] bmwg-multicast document review

See my comments below.

********
3.1.	Test Considerations

********

Debby Stopp 

IXIA
26601 W. Agoura Rd. 
Calabasas 
CA 91302 

[email-masked]
http://www.ixiacom.com