Subject: re: [bmwg] I-D ACTION:draft-ietf-bmwg-mcastm-11.txt

From: Debby Stopp <[email-masked]>
Date: Thu, 20 Mar 2003 12:10:47 -0800

Let's stop using the terms LIFO and FIFO because these terms are used to
define how entities arriving at a queue get serviced.  If you disagree, then
I would suggest you review the enclosed links:  The National Institute of
Standards and Technology (http://www.nist.gov/dads/HTML/lastfirstout.html)
and the General Services Administration
http://glossary.its.bldrdoc.gov/fs-1037/dir-020/_2990.htm for the
appropriate use of these definitions.  Unless, of course, you wish to
redefine these terms for meauring delay between two separate events.  In
that case, a terminology document will be required.

Now that we cease to use this terms inappropriately, let's get to your
perceived issue.  You believe that the authors have introduced the LIFO
goalposts from RFC 1242.  I disagree.  I don't see any explicit references
to RFC 1242 in section 6.1 or 6.2.  The group join delay measures the time
elapsed between two different externally observable events.  It is
irrelevent for this test if the DUT is a "store and forward" or a "bit
forwarding" device.  

	According to Section 6.1, the Group Join delay time is defined as
the difference between two timestamps A and B where:

	Timestamp A MUST be the time the last bit of the IGMP group
membership report is sent from the destination test port
	timestamp B MUST be the time the first bit of the first valid
multicast frame is forwarded on the egress interface of the DUT/SUT. 

The packets sent to and received from the egress interface are different.
One packet is an IP packet with an IGMP message.  This packet is not being
forwarded on the data plane but rather being processed by the state machine.
The other packet is an IP or UDP packet with a destination multicast address
that has been replicated and forwarded on the data plane once the join is
complete. Please note that any inferences to the LIFO are your preceived
notions.  

I am sure that there are differing opinions on where the goalpost should be
placed.  We elected to start the clock when the last bit of the IGMP message
is placed on the wire in order to exclude the time to transmit IGMP packet;
we elected to stop the clock on receipt of first bit of the forwarded
multicast packet in order to exclude the size of the validatation packet as
part of the metric.  Regardless of the goalpost placement, the objective of
the methodology is to create a meaningful and repeatable  metric.  I believe
we meet that objective.

Debby Stopp

-----Original Message-----
From: David Newman [mailto:[email-masked]] 
Sent: Tuesday, March 18, 2003 10:59 AM
To: [email-masked]
Subject: RE: [bmwg] I-D ACTION:draft-ietf-bmwg-mcastm-11.txt

Section 1.

"This document defines tests for measuring and reporting the forwarding,
latency and IGMP group membership characteristics of  devices that support
IP multicast routing protocols."

s/routing //

This document covers forwarding, not routing. This is consistent with
"forwarding" in the abstract of 2432.

Section 3.1.4

"The TTL in IGMP control plane messages is in compliance with the version of
IGMP in use."

s/is/MUST be/

Section 6.1

Timestamp A MUST be the time the last bit of the IGMP group membership
report is sent from the destination test port; timestamp B MUST be the time
the first bit of the first valid multicast frame is forwarded on the egress
interface of the DUT/SUT.

s/first bit/last bit/

Add at end:

IGMP group membership MUST be verified through externally observable means.

The LIFO method given here will not produce meaningful numbers for
store-and-forward devices. Two reasons:

1. Store-and-forward devices need the entire packet before processing begins
2. It's impossible to validate group membership with the "FO" in LIFO.

On point 1, some of the authors have objected that LIFO vs. FIFO is
irrelevant since this is a test of delay and not latency as defined in 1242
and 2432.

This is a red herring: While WHAT we measure has changed, HOW we measure it
has not. The authors introduced the LIFO goalposts from 1242; they cannot
then claim that LIFO is irrelevant.

IOW, the fact that we measure a two-way conversation multiple packets doesn'
t remove the requirement for meaningful starting and ending points. RFC 2432
doesn't define these, and the ending point with the FO method given here isn
't meaningful. This brings us to point 2, the lack of a validation
requirement.

Simple analogy illustrates the validation issue. I have a friend who is a
firearms collector. He sends away for a subscription to Guns & Ammo
magazine. My friend then receives a magazine in a manila envelope. He rips
open the envelope only to discover he has received Gun Control Gazette
(something I'm more likely to subscribe to).

The "FO" in LIFO is the equivalent of my friend receiving the envelope. A
LILO method would allow my friend to ascertain that his join request has
failed.

Note that LILO itself doesn't guarantee a valid join; a separate step of
externally observable verification is needed for that. Still, it's not
possible to do this separate step with LIFO.

Section 6.2 on group leave gets it right: LILO.

dn

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of 
> [email-masked]
> Sent: Thursday, March 06, 2003 3:24 AM
> To: IETF-Announce:
> Cc: [email-masked]
> Subject: [bmwg] I-D ACTION:draft-ietf-bmwg-mcastm-11.txt
>
>
> A New Internet-Draft is available from the on-line Internet-Drafts 
> directories. This draft is a work item of the Benchmarking Methodology 
> Working Group of the IETF.
>
> 	Title		: Methodology for IP Multicast Benchmarking
> 	Author(s)	: D. Stopp, B. Hickman
> 	Filename	: draft-ietf-bmwg-mcastm-11.txt
> 	Pages		: 28
> 	Date		: 2003-3-5
>
> The purpose of this draft is to describe methodology specific to the 
> benchmarking of multicast IP forwarding devices. It builds upon the 
> tenets set forth in RFC 2544, RFC 2432 and other IETF Benchmarking 
> Methodology Working Group (BMWG) efforts.  This document seeks to 
> extend these efforts to the multicast paradigm. The BMWG produces two 
> major classes of documents: Benchmarking Terminology documents and 
> Benchmarking Methodology documents. The Terminology documents present 
> the benchmarks and other related terms. The Methodology documents 
> define the procedures required to collect the benchmarks cited in the 
> corresponding Terminology documents.
>
> A URL for this Internet-Draft is: 
> http://www.ietf.org/internet-drafts/draft-ietf-bmwg-mcastm-11.txt
>
> To remove yourself from the IETF Announcement list, send a message to 
> ietf-announce-request with the word unsubscribe in the body of the 
> message.
>
> Internet-Drafts are also available by anonymous FTP. Login with the 
> username "anonymous" and a password of your e-mail address. After 
> logging in, type "cd internet-drafts" and then
> 	"get draft-ietf-bmwg-mcastm-11.txt".
>
> A list of Internet-Drafts directories can be found in 
> http://www.ietf.org/shadow.html or 
> ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>
> Internet-Drafts can also be obtained by e-mail.
>
> Send a message to:
> 	[email-masked].
> In the body type:
> 	"FILE /internet-drafts/draft-ietf-bmwg-mcastm-11.txt".
>
> NOTE:	The mail server at ietf.org can return the document in
> 	MIME-encoded form by using the "mpack" utility.  To use this
> 	feature, insert the command "ENCODING mime" before the "FILE"
> 	command.  To decode the response(s), you will need "munpack" or
> 	a MIME-compliant mail reader.  Different MIME-compliant mail readers
> 	exhibit different behavior, especially when dealing with
> 	"multipart" MIME messages (i.e. documents which have been split
> 	up into multiple messages), so check your local documentation on
> 	how to manipulate these messages.
>
>
> Below is the data which will enable a MIME compliant mail reader 
> implementation to automatically retrieve the ASCII version of the 
> Internet-Draft.
>

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: David Newman <[email-masked]>
Date: Thu, 20 Mar 2003 12:51:29 -0800 (PST)

> define how entities arriving at a queue get serviced. 

um, no, they are the goalposts you added in draft 11. Here is your 
language from section 6.1:

Timestamp A MUST be the time the last bit of the IGMP group membership
report is sent from the destination test port; timestamp B MUST be the
time the first bit of the first valid multicast frame is forwarded on the
egress interface of the DUT/SUT.

You can argue the point all you like, but this is a LIFO method.

BTW let me reiterate that I'm not at all opposed to the use of such
goalposts;  I just think they need to be applied in a meaningful way.

> to RFC 1242 in section 6.1 or 6.2.  

The first reference in a bmwg context to LIFO and FIFO methods is in RFC 
1242. You have reused those methods here (albeit inconsistently, since 
section 6.2 of the ID specifies LILO).

The group join delay measures the time
> the methodology is to create a meaningful and repeatable  metric.  I believe

You have failed to meet the objective if, by your method, a bogus (ie, 
non-subscribed) group is counted as subscribed. The current ID allows 
such a situation.

You need the entire packet to verify subscription, and you can't do that 
with a strict LIFO method.

I would actually be OK with the LIFO method provided you add language 
stating that group membership MUST be verified using externally 
observable methods. 

You MAY also want to consider giving implementers some help here by
stating that test instrument's rx interface MUST cache the entire packet
(mandatory to verify correct group membership) and then subtract out the 
frame's (not packet's) rx time to get the LIFO number.

If you do stick with LIFO, you would then need to change section 6.2 to
LIFO (it's currently LILO) for consistency.

dn

---

From: David Newman <[email-masked]>
Date: Thu, 20 Mar 2003 12:51:29 -0800 (PST)

dn

---

From: Scott Poretsky <[email-masked]>
Date: Wed, 26 Mar 2003 10:34:40 -0500

I believe this is the most recent mail on this subject. My apologies if it
is not. Reading this thread I have come to a conclusion I would like to
share in hope that it will resolve the issue.

I believe it is inappropriate to apply queuing terms of FIFO, LIFO, and
LILO (and even FILO if it were ever suggested) to the tests described in
sections 6.1 and 6.2 (Group Join Delay and Group Leave Delay, respectively)
of draft-ietf-bmwg-mcastm-11. The Group Join Delay test describes
scenarios in which an offered control plane message is used to drive
multicast forwarding on the data plane of that same interface. First
multicast traffic is sent to the DUT. The delay is the time difference for
the Group Join control message to be sent to the DUT and the multicast data
to be received by the test device on that same interface. The packet
forwarding delay of the multicast traffic through the DUT is never
considered.  Since an action is made on the control plane to drive a 
response on the data plane using a single interface, the terms FIFO, LIFO, 
and LILO do not apply.

The Group Leave Delay case is the opposite test in which the
delay is measured as the time difference for the control message to be sent
to the DUT and the multicast data to stop being received by the test device
on that same interface. Again, the packet forwarding delay of the
multicast traffic through the DUT is never considered and since an action 
is made on the control plane to drive a response on the data plane using a 
single interface, the terms FIFO, LIFO, and LILO do not apply.

This is indeed an excellent router benchmark for hundreds to thousands of 
simultaneous multicast joins.  For example the longer it takes to join all 
1K groups is the longer it takes that 1K'th multicast customer to start 
receiving traffic.

Note that these test cases are not specific to IGMP as they can also be 
applied to PIM.

If we all agree not to apply the terms FIFO, LIFO, and LILO is there still
an issue?

Scott

---

From: Scott Poretsky <[email-masked]>
Date: Wed, 26 Mar 2003 10:34:40 -0500

Scott