From mailman-owner@ietf.org Thu Nov 1 05:07:12 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA08974 for ; Thu, 1 Nov 2001 05:07:11 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id FAA12267 for ; Thu, 1 Nov 2001 05:07:16 -0500 (EST) Date: Thu, 1 Nov 2001 05:07:16 -0500 (EST) Message-Id: <200111011007.FAA12267@optimus.ietf.org> From: mailman-owner@ietf.org Subject: ietf.org mailing list memberships reminder To: bmwg-archive@ietf.org X-No-Archive: yes Precedence: bulk X-Mailman-Version: 1.0 Precedence: bulk List-Id: Session Initiation Protocol This is a reminder, sent out once a month, about your ietf.org mailing list memberships. It includes your subscription info and how to use it to change it or unsubscribe from a list. You can visit the URLs to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on. In addition to the URL interfaces, you can also use email to make such changes. For more info, send a message to the '-request' address of the list (for example, bmwg-request@ietf.org) containing just the word 'help' in the message body, and an email message will be sent to you with instructions. If you have questions, problems, comments, etc, send them to mailman-owner@ietf.org. Thanks! Passwords for bmwg-archive@lists.ietf.org: List Password // URL ---- -------- bmwg@ietf.org YGWv http://www1.ietf.org/mailman/options/bmwg/bmwg-archive@lists.ietf.org From bmwg-admin@ietf.org Thu Nov 1 07:50:54 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA13513 for ; Thu, 1 Nov 2001 07:50:54 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id HAA29711; Thu, 1 Nov 2001 07:50:51 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id HAA27153 for ; Thu, 1 Nov 2001 07:03:25 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA11047; Thu, 1 Nov 2001 07:03:19 -0500 (EST) Message-Id: <200111011203.HAA11047@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: bmwg@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Thu, 01 Nov 2001 07:03:19 -0500 Subject: [bmwg] I-D ACTION:draft-ietf-bmwg-firewall-03.txt Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Benchmarking Methodology Working Group of the IETF. Title : Benchmarking Methodology for Firewalls Author(s) : B. Hickman, D. Newman, S. Tadjudin, T. Martin Filename : draft-ietf-bmwg-firewall-03.txt Pages : 23 Date : 31-Oct-01 This document provides methodologies for the performance benchmarking of firewalls. It provides methodologies in four areas: forwarding, connection, latency and filtering. In addition to defining the tests, this document also describes specific formats for reporting the results of the tests. A previous document, 'Benchmarking Terminology for Firewall Performance' [1], defines many of the terms that are used in this document. The terminology document SHOULD be consulted before attempting to make use of this document. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-bmwg-firewall-03.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-bmwg-firewall-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-bmwg-firewall-03.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011031112009.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-bmwg-firewall-03.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-bmwg-firewall-03.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011031112009.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From daemon@optimus.ietf.org Thu Nov 1 07:50:56 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA13526 for ; Thu, 1 Nov 2001 07:50:56 -0500 (EST) Received: (from daemon@localhost) by optimus.ietf.org (8.9.1a/8.9.1) id HAA29728 for bmwg-archive@odin.ietf.org; Thu, 1 Nov 2001 07:51:02 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id HAA29711; Thu, 1 Nov 2001 07:50:51 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id HAA27153 for ; Thu, 1 Nov 2001 07:03:25 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA11047; Thu, 1 Nov 2001 07:03:19 -0500 (EST) Message-Id: <200111011203.HAA11047@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: bmwg@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Thu, 01 Nov 2001 07:03:19 -0500 Subject: [bmwg] I-D ACTION:draft-ietf-bmwg-firewall-03.txt Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Benchmarking Methodology Working Group of the IETF. Title : Benchmarking Methodology for Firewalls Author(s) : B. Hickman, D. Newman, S. Tadjudin, T. Martin Filename : draft-ietf-bmwg-firewall-03.txt Pages : 23 Date : 31-Oct-01 This document provides methodologies for the performance benchmarking of firewalls. It provides methodologies in four areas: forwarding, connection, latency and filtering. In addition to defining the tests, this document also describes specific formats for reporting the results of the tests. A previous document, 'Benchmarking Terminology for Firewall Performance' [1], defines many of the terms that are used in this document. The terminology document SHOULD be consulted before attempting to make use of this document. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-bmwg-firewall-03.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-bmwg-firewall-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-bmwg-firewall-03.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011031112009.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-bmwg-firewall-03.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-bmwg-firewall-03.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011031112009.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From bmwg-admin@ietf.org Thu Nov 15 19:20:10 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29192 for ; Thu, 15 Nov 2001 19:20:10 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA11668; Thu, 15 Nov 2001 19:19:35 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA11637 for ; Thu, 15 Nov 2001 19:19:33 -0500 (EST) Received: from racerx.ixiacom.com (64-60-75-69-cust.telepacific.net [64.60.75.69]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29150 for ; Thu, 15 Nov 2001 19:19:31 -0500 (EST) Received: by racerx.ixiacom.com with Internet Mail Service (5.5.2653.19) id ; Thu, 15 Nov 2001 16:12:36 -0800 Message-ID: <9A9C83C019F35D47A570460E87D5D8ABD198CC@racerx.ixiacom.com> From: Debby Stopp To: "'John Dawson'" , bmwg@ietf.org Subject: RE: [bmwg] bmwg-multicast document review Date: Thu, 15 Nov 2001 16:12:35 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org See my comments below. > >> Section 3.1. Test Considerations > > >> Moreover, no provisions are > >> made for traffic-affecting factors, such as congestion control or > >> service differentiation mechanisms. Modifications to the > specified > >> collection procedures might need to be made to accommodate the > >> transmission media actually tested. These accommodations MUST be > >> presented with the test results. > I don't agree with this language. We have suffered for too > many years with throughput and performance > tests that allow congestion control and forward pressure > techniques to skew results. It is time to close > these holes. This paragraph essentially grants license to > change any collection procedures to accommodate > some designer agenda. If the procedures laid out in this RFC > can not be used to characterized a specific > media, then this RFC should not be used as a basis for > analyzing that technology. Although the authors > specify that any accommodations must be presented with the > test results, this action allows anyone to > re-engineer the test process and then leaves it to the > ingenuity of the reader to catch the tricks that may > be employed to mis-direct the less educated. > Are we agreeing to a process here? or are we granting > everyone free license to re-write their own test? This paragraph specifically worded to allow someone to use the methodology w/out regard to media. The intention is not to overload the receive ports & thus deal w/congestion & flow control , but rather to measure the ability to forward IPMC traffic in a particular scenario. Thus, I will add a sentence to this section stating that these tests MUST be run w/out flow control, etc, like this: ******** 3.1. Test Considerations The procedures outlined below are written without regard for specific physical layer or link layer protocols. The methodology further assumes a uniform medium topology. Issues regarding mixed transmission media, such as speed mismatch, headers differences, etc., are not specifically addressed. Flow control, QoS and other traffic-affecting mechanisms MUST be disabled. Modifications to the specified collection procedures might need to be made to accommodate the transmission media actually tested. These accommodations MUST be presented with the test results. ********* > >> Section 4.1 Mixed Class Throughput > >> Result > >> Parameters to be measured SHOULD include the frame loss > and percent > >> loss for each class of traffic per destination port. The ratio of > >> unicast traffic to multicast traffic MUST be reported. > Let's specify what must be measured, not what should be measured. I can reword this section <4.1> as follows: ******************************** Result Parameters to be measured SHOULD include the frame loss and percent loss for each class of traffic per destination port. The ratio of unicast traffic to multicast traffic MUST be reported. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. Parameters to be measured MUST include number of multicast frames offered, number of unicast frames offered, number of multicast frames received and number of unicast frames received. In addition, all other reporting parameters of mixed class throughput MUST be reflected in the results report, such as the transmitted packet size(s) and the granularity of the measurement process. Results for mixed class throughput MUST include the following parameters: the maximum aggregate offered frames per second from the source ports for both multicast and unicast frames at which no loss forwarding is observed and the ratio of unicast traffic to multicast traffic at each destination port. ******************************** Note - as the test duration must be included in all results reports, I can reword this section as follows: ******************************** 3.1.5. Trial Duration The duration of the test portion of each trial SHOULD be at least 30 seconds. This parameter MUST be included as part of the results reporting for each methodology. ******************************** I prefer to put this in a more generalized section, as I believe all tests should always report the test duration and I prefer not to cut and paste the same words throughout the document. For section 4.2, I have reworded as follows: ******************************** Results Parameters to be measured MUST include the offered load and forwarding rate as a function of the total number of multicast groups for each test iteration. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. In addition, all other reporting parameters of the scaled group forwarding matrix methodology MUST be reflected in the results report, such as the transmitted packet size(s) and offered load of the packet stream for each source port. Result reports MUST include the following parameters for each iteration: the number of frames offered, number of frames received per each group, number of multicast groups and forwarding rate, in frames per second. Constructing a table that contains the forwarding rate vs. number of groups is desirable. ******** You had some comments re: the aggregated test; here's a revised version, see draft for revised section. ******** > >> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput this section has changed significantly based on Kevin's comments - please review again. > >> Section 5. Forwarding Latency This section has been reworded a bit thanks to Kevin's comments, see draft. Debby Stopp Principal Software Engineer IXIA 26601 W. Agoura Rd. Calabasas CA 91302 Tel: 818 871 1800 (US) +1 818 871 1800 (international) debby@ixiacom.com http://www.ixiacom.com _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From daemon@ns.ietf.org Thu Nov 15 19:20:11 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29205 for ; Thu, 15 Nov 2001 19:20:11 -0500 (EST) Received: (from daemon@localhost) by optimus.ietf.org (8.9.1a/8.9.1) id TAA11741 for bmwg-archive@odin.ietf.org; Thu, 15 Nov 2001 19:20:13 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA11668; Thu, 15 Nov 2001 19:19:35 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id TAA11637 for ; Thu, 15 Nov 2001 19:19:33 -0500 (EST) Received: from racerx.ixiacom.com (64-60-75-69-cust.telepacific.net [64.60.75.69]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29150 for ; Thu, 15 Nov 2001 19:19:31 -0500 (EST) Received: by racerx.ixiacom.com with Internet Mail Service (5.5.2653.19) id ; Thu, 15 Nov 2001 16:12:36 -0800 Message-ID: <9A9C83C019F35D47A570460E87D5D8ABD198CC@racerx.ixiacom.com> From: Debby Stopp To: "'John Dawson'" , bmwg@ietf.org Subject: RE: [bmwg] bmwg-multicast document review Date: Thu, 15 Nov 2001 16:12:35 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org See my comments below. > >> Section 3.1. Test Considerations > > >> Moreover, no provisions are > >> made for traffic-affecting factors, such as congestion control or > >> service differentiation mechanisms. Modifications to the > specified > >> collection procedures might need to be made to accommodate the > >> transmission media actually tested. These accommodations MUST be > >> presented with the test results. > I don't agree with this language. We have suffered for too > many years with throughput and performance > tests that allow congestion control and forward pressure > techniques to skew results. It is time to close > these holes. This paragraph essentially grants license to > change any collection procedures to accommodate > some designer agenda. If the procedures laid out in this RFC > can not be used to characterized a specific > media, then this RFC should not be used as a basis for > analyzing that technology. Although the authors > specify that any accommodations must be presented with the > test results, this action allows anyone to > re-engineer the test process and then leaves it to the > ingenuity of the reader to catch the tricks that may > be employed to mis-direct the less educated. > Are we agreeing to a process here? or are we granting > everyone free license to re-write their own test? This paragraph specifically worded to allow someone to use the methodology w/out regard to media. The intention is not to overload the receive ports & thus deal w/congestion & flow control , but rather to measure the ability to forward IPMC traffic in a particular scenario. Thus, I will add a sentence to this section stating that these tests MUST be run w/out flow control, etc, like this: ******** 3.1. Test Considerations The procedures outlined below are written without regard for specific physical layer or link layer protocols. The methodology further assumes a uniform medium topology. Issues regarding mixed transmission media, such as speed mismatch, headers differences, etc., are not specifically addressed. Flow control, QoS and other traffic-affecting mechanisms MUST be disabled. Modifications to the specified collection procedures might need to be made to accommodate the transmission media actually tested. These accommodations MUST be presented with the test results. ********* > >> Section 4.1 Mixed Class Throughput > >> Result > >> Parameters to be measured SHOULD include the frame loss > and percent > >> loss for each class of traffic per destination port. The ratio of > >> unicast traffic to multicast traffic MUST be reported. > Let's specify what must be measured, not what should be measured. I can reword this section <4.1> as follows: ******************************** Result Parameters to be measured SHOULD include the frame loss and percent loss for each class of traffic per destination port. The ratio of unicast traffic to multicast traffic MUST be reported. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. Parameters to be measured MUST include number of multicast frames offered, number of unicast frames offered, number of multicast frames received and number of unicast frames received. In addition, all other reporting parameters of mixed class throughput MUST be reflected in the results report, such as the transmitted packet size(s) and the granularity of the measurement process. Results for mixed class throughput MUST include the following parameters: the maximum aggregate offered frames per second from the source ports for both multicast and unicast frames at which no loss forwarding is observed and the ratio of unicast traffic to multicast traffic at each destination port. ******************************** Note - as the test duration must be included in all results reports, I can reword this section as follows: ******************************** 3.1.5. Trial Duration The duration of the test portion of each trial SHOULD be at least 30 seconds. This parameter MUST be included as part of the results reporting for each methodology. ******************************** I prefer to put this in a more generalized section, as I believe all tests should always report the test duration and I prefer not to cut and paste the same words throughout the document. For section 4.2, I have reworded as follows: ******************************** Results Parameters to be measured MUST include the offered load and forwarding rate as a function of the total number of multicast groups for each test iteration. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. In addition, all other reporting parameters of the scaled group forwarding matrix methodology MUST be reflected in the results report, such as the transmitted packet size(s) and offered load of the packet stream for each source port. Result reports MUST include the following parameters for each iteration: the number of frames offered, number of frames received per each group, number of multicast groups and forwarding rate, in frames per second. Constructing a table that contains the forwarding rate vs. number of groups is desirable. ******** You had some comments re: the aggregated test; here's a revised version, see draft for revised section. ******** > >> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput this section has changed significantly based on Kevin's comments - please review again. > >> Section 5. Forwarding Latency This section has been reworded a bit thanks to Kevin's comments, see draft. Debby Stopp Principal Software Engineer IXIA 26601 W. Agoura Rd. Calabasas CA 91302 Tel: 818 871 1800 (US) +1 818 871 1800 (international) debby@ixiacom.com http://www.ixiacom.com _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From bmwg-admin@ietf.org Thu Nov 15 20:32:37 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01129 for ; Thu, 15 Nov 2001 20:32:37 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA13850; Thu, 15 Nov 2001 20:32:18 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA13821 for ; Thu, 15 Nov 2001 20:32:16 -0500 (EST) Received: from zrc2s03g.us.nortel.com (h66s122a103n47.user.nortelnetworks.com [47.103.122.66]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01108 for ; Thu, 15 Nov 2001 20:32:13 -0500 (EST) Received: from smtprch2.nortel.com (erchg0k.us.nortel.com [47.113.64.104]) by zrc2s03g.us.nortel.com (8.9.3+Sun/8.9.1) with ESMTP id TAA26436 for ; Thu, 15 Nov 2001 19:31:45 -0600 (CST) Received: from zsc4c000.us.nortel.com by smtprch2.nortel.com; Thu, 15 Nov 2001 19:24:39 -0600 Received: by zsc4c000.us.nortel.com with Internet Mail Service (5.5.2653.19) id ; Thu, 15 Nov 2001 17:30:30 -0800 Message-ID: <5D630265EF50D311ABB60008C7917DB6083E24CC@zsc4c004.us.nortel.com> From: "John Dawson" To: Debby Stopp , bmwg@ietf.org Subject: RE: [bmwg] bmwg-multicast document review Date: Thu, 15 Nov 2001 17:30:28 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C16E3E.45FBEB90" X-Orig: Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C16E3E.45FBEB90 Content-Type: text/plain; charset="iso-8859-1" >> Parameters to be measured MUST include number of multicast frames offered, >> number of unicast frames offered, number of multicast frames received and >> number of unicast frames received. You are assuming that test duration is a constant. This is not true. Test duration (Time in seconds) is a measured quantity. This is the differentiator between OfferedLoad and IntendedLoad. You can not report offered load unless time is a measurement. All methodology MUST include a measurement of the TIME it takes to transmit the offered frames. If you measure time at the receive side, then and only then is the forwarding rate accurately known. By including Time as a required measurement parameter, you will automatically address the issues associated with flow control and forward/back-pressure. I do not believe that it is appropriate to put test duration into the general overhead section. Test duration MUST be measured, but SHOULD also be reported. If you choose not to report time directly, then time can be deduced by calculating backwards from the Offered Load. Time == FramesOffered/OfferedLoad. It is also important to note that there are two methods that have been suggested as a valid methodology for determining offered Load. One is based upon transmitting a fixed quantity of packets and measuring the time required to transmit the packets. If multiple ports are transmitting, then TIME must be measured on each port. The other method, is to transmit packets for a fixed time. The number of packets transmitted by each port then becomes the measured quantity. I prefer the fixed TIME approach as it represents a uniform loading (pressure) for all ports across a given test. If a fixed quantity of packets are used, then the test can be very non-uniform. For instance, assume that half-duplex systems are being tested using bi-directional traffic and further assume that one port captures the wire. In this scenario, you actually have two separate uni-directional streams where the first stream transmits for the first half of the test and the other stream transmits for the second half of the test (after the first stream goes idle). In this scenario, the offered loads are not what you intended. Unless you make TIME a parameter that MUST be measured, you can not determine the offered load -- you will never know what happened during the test and you will be reporting bogus numbers. John -----Original Message----- From: Debby Stopp [mailto:dstopp@ixiacom.com] Sent: Thursday, November 15, 2001 4:13 PM To: Dawson, John [BLVWA:382:EXCH]; bmwg@ietf.org Subject: RE: [bmwg] bmwg-multicast document review See my comments below. > >> Section 3.1. Test Considerations > > >> Moreover, no provisions are > >> made for traffic-affecting factors, such as congestion control or > >> service differentiation mechanisms. Modifications to the > specified > >> collection procedures might need to be made to accommodate the > >> transmission media actually tested. These accommodations MUST be > >> presented with the test results. > I don't agree with this language. We have suffered for too > many years with throughput and performance > tests that allow congestion control and forward pressure > techniques to skew results. It is time to close > these holes. This paragraph essentially grants license to > change any collection procedures to accommodate > some designer agenda. If the procedures laid out in this RFC > can not be used to characterized a specific > media, then this RFC should not be used as a basis for > analyzing that technology. Although the authors > specify that any accommodations must be presented with the > test results, this action allows anyone to > re-engineer the test process and then leaves it to the > ingenuity of the reader to catch the tricks that may > be employed to mis-direct the less educated. > Are we agreeing to a process here? or are we granting > everyone free license to re-write their own test? This paragraph specifically worded to allow someone to use the methodology w/out regard to media. The intention is not to overload the receive ports & thus deal w/congestion & flow control , but rather to measure the ability to forward IPMC traffic in a particular scenario. Thus, I will add a sentence to this section stating that these tests MUST be run w/out flow control, etc, like this: ******** 3.1. Test Considerations The procedures outlined below are written without regard for specific physical layer or link layer protocols. The methodology further assumes a uniform medium topology. Issues regarding mixed transmission media, such as speed mismatch, headers differences, etc., are not specifically addressed. Flow control, QoS and other traffic-affecting mechanisms MUST be disabled. Modifications to the specified collection procedures might need to be made to accommodate the transmission media actually tested. These accommodations MUST be presented with the test results. ********* > >> Section 4.1 Mixed Class Throughput > >> Result > >> Parameters to be measured SHOULD include the frame loss > and percent > >> loss for each class of traffic per destination port. The ratio of > >> unicast traffic to multicast traffic MUST be reported. > Let's specify what must be measured, not what should be measured. I can reword this section <4.1> as follows: ******************************** Result Parameters to be measured SHOULD include the frame loss and percent loss for each class of traffic per destination port. The ratio of unicast traffic to multicast traffic MUST be reported. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. Parameters to be measured MUST include number of multicast frames offered, number of unicast frames offered, number of multicast frames received and number of unicast frames received. In addition, all other reporting parameters of mixed class throughput MUST be reflected in the results report, such as the transmitted packet size(s) and the granularity of the measurement process. Results for mixed class throughput MUST include the following parameters: the maximum aggregate offered frames per second from the source ports for both multicast and unicast frames at which no loss forwarding is observed and the ratio of unicast traffic to multicast traffic at each destination port. ******************************** Note - as the test duration must be included in all results reports, I can reword this section as follows: ******************************** 3.1.5. Trial Duration The duration of the test portion of each trial SHOULD be at least 30 seconds. This parameter MUST be included as part of the results reporting for each methodology. ******************************** I prefer to put this in a more generalized section, as I believe all tests should always report the test duration and I prefer not to cut and paste the same words throughout the document. For section 4.2, I have reworded as follows: ******************************** Results Parameters to be measured MUST include the offered load and forwarding rate as a function of the total number of multicast groups for each test iteration. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. In addition, all other reporting parameters of the scaled group forwarding matrix methodology MUST be reflected in the results report, such as the transmitted packet size(s) and offered load of the packet stream for each source port. Result reports MUST include the following parameters for each iteration: the number of frames offered, number of frames received per each group, number of multicast groups and forwarding rate, in frames per second. Constructing a table that contains the forwarding rate vs. number of groups is desirable. ******** You had some comments re: the aggregated test; here's a revised version, see draft for revised section. ******** > >> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput this section has changed significantly based on Kevin's comments - please review again. > >> Section 5. Forwarding Latency This section has been reworded a bit thanks to Kevin's comments, see draft. Debby Stopp Principal Software Engineer IXIA 26601 W. Agoura Rd. Calabasas CA 91302 Tel: 818 871 1800 (US) +1 818 871 1800 (international) debby@ixiacom.com http://www.ixiacom.com ------_=_NextPart_001_01C16E3E.45FBEB90 Content-Type: text/html; charset="iso-8859-1" RE: [bmwg] bmwg-multicast document review

>> Parameters to be measured MUST include number of multicast frames offered,
>> number of unicast frames offered, number of multicast frames received and
>> number of unicast frames received.

You are assuming that test duration is a constant.  This is not true.  Test
duration (Time in seconds) is a measured quantity.  This is the differentiator
between OfferedLoad and IntendedLoad.  You can not report offered load unless
time is a measurement.  All methodology MUST include a measurement of the TIME
it takes to transmit the offered frames.

If you measure time at the receive side, then and only then is the forwarding
rate accurately known.

By including Time as a required measurement parameter, you will automatically
address the issues associated with flow control and forward/back-pressure.

I do not believe that it is appropriate to put test duration into the general
overhead section.  Test duration MUST be measured, but SHOULD also be reported.
If you choose not to report time directly, then time can be deduced by calculating
backwards from the Offered Load.  Time == FramesOffered/OfferedLoad.

It is also important to note that there are two methods that have been suggested
as a valid methodology for determining offered Load.  One is based upon transmitting
a fixed quantity of packets and measuring the time required to transmit the packets.
If multiple ports are transmitting, then TIME must be measured on each port.  The
other method, is to transmit packets for a fixed time.  The number of packets
transmitted by each port then becomes the measured quantity.

I prefer the fixed TIME approach as it represents a uniform loading (pressure) for
all ports across a given test.  If a fixed quantity of packets are used, then the
test can be very non-uniform.  For instance, assume that half-duplex systems are
being tested using bi-directional traffic and further assume that one port captures
the wire.  In this scenario, you actually have two separate uni-directional streams
where the first stream transmits for the first half of the test and the other stream
transmits for the second half of the test (after the first stream goes idle).  In
this scenario, the offered loads are not what you intended.

Unless you make TIME a parameter that MUST be measured, you can not determine the
offered load -- you will never know what happened during the test and you will be
reporting bogus numbers.

John



-----Original Message-----
From: Debby Stopp [mailto:dstopp@ixiacom.com]
Sent: Thursday, November 15, 2001 4:13 PM
To: Dawson, John [BLVWA:382:EXCH]; bmwg@ietf.org
Subject: RE: [bmwg] bmwg-multicast document review


See my comments below.

> >>  Section 3.1. Test Considerations
>   
> >> Moreover, no provisions are
> >> made for traffic-affecting factors, such as congestion control or
> >> service differentiation mechanisms.  Modifications to the
> specified
> >> collection procedures might need to be made to accommodate the
> >> transmission media actually tested.  These accommodations MUST be
> >> presented with the test results.
> I don't agree with this language.  We have suffered for too
> many years with throughput and performance
> tests that allow congestion control and forward pressure
> techniques to skew results.  It is time to close
> these holes.  This paragraph essentially grants license to
> change any collection procedures to accommodate
> some designer agenda.  If the procedures laid out in this RFC
> can not be used to characterized a specific
> media, then this RFC should not be used as a basis for
> analyzing that technology.  Although the authors
> specify that any accommodations must be presented with the
> test results, this action allows anyone to
> re-engineer the test process and then leaves it to the
> ingenuity of the reader to catch the tricks that may
> be employed to mis-direct the less educated.
> Are we agreeing to a process here? or are we granting
> everyone free license to re-write their own test?

This paragraph specifically worded to allow someone to use the methodology
w/out regard to media.  The intention is not to overload the receive ports &
thus deal w/congestion & flow control <there's already an RFC for that>, but
rather to measure the ability to forward IPMC traffic in a particular
scenario.

Thus, I will add a sentence to this section stating that these tests MUST be
run w/out flow control, etc, like this:

********
3.1.    Test Considerations

The procedures outlined below are written without regard for specific
physical layer or link layer protocols. The methodology further assumes a
uniform medium topology. Issues regarding mixed transmission media, such as
speed mismatch, headers differences, etc., are not specifically addressed.
Flow control, QoS and other traffic-affecting mechanisms MUST be disabled.
Modifications to the specified collection procedures might need to be made
to accommodate the transmission media actually tested.  These accommodations
MUST be presented with the test results.
*********


> >> Section 4.1 Mixed Class Throughput
> >> Result
> >> Parameters to be measured SHOULD include the frame loss
> and percent
> >> loss for each class of traffic per destination port.  The ratio of
> >> unicast traffic to multicast traffic MUST be reported.
> Let's specify what must be measured, not what should be measured.

I can reword this section <4.1> as follows:
********************************
Result

Parameters to be measured SHOULD include the frame loss and percent loss for
each class of traffic per destination port.  The ratio of unicast traffic to
multicast traffic MUST be reported.

The nature of the traffic stream contributing to the result MUST be
reported, specifically number of source and destination ports within the
multicast group. Parameters to be measured MUST include number of multicast
frames offered, number of unicast frames offered, number of multicast frames
received and number of unicast frames received.  In addition, all other
reporting parameters of mixed class throughput MUST be reflected in the
results report, such as the transmitted packet size(s) and the granularity
of the measurement process.

Results for mixed class throughput MUST include the following parameters:
the maximum aggregate offered frames per second from the source ports for
both multicast and unicast frames at which no loss forwarding is observed
and the ratio of unicast traffic to multicast traffic at each destination
port.

********************************

Note - as the test duration must be included in all results reports, I can
reword this section as follows:
********************************
3.1.5.  Trial Duration

The duration of the test portion of each trial SHOULD be at least 30
seconds.  This parameter MUST be included as part of the results reporting
for each methodology.
********************************

I prefer to put this in a more generalized section, as I believe all tests
should always report the test duration and I prefer not to cut and paste the
same words throughout the document.



For section 4.2, I have reworded as follows:
********************************
Results

Parameters to be measured MUST include the offered load and forwarding rate
as a function of the total number of multicast groups for each test
iteration.

The nature of the traffic stream contributing to the result MUST be
reported, specifically number of source and destination ports within the
multicast group.  In addition, all other reporting parameters of the scaled
group forwarding matrix methodology MUST be reflected in the results report,
such as the transmitted packet size(s) and offered load of the packet stream
for each source port.

Result reports MUST include the following parameters for each iteration: the
number of frames offered, number of frames received per each group, number
of multicast groups and forwarding rate, in frames per second.  Constructing
a table that contains the forwarding rate vs. number of groups is desirable.


********
You had some comments <as did others> re: the aggregated test; here's a
revised version, see draft for revised section.
********


> >> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput

this section has changed significantly based on Kevin's comments - please
review again.


> >> Section  5. Forwarding Latency

This section has been reworded a bit thanks to Kevin's comments, see draft.
 

Debby Stopp
Principal Software Engineer


IXIA
26601 W. Agoura Rd.
Calabasas
CA 91302
Tel: 818 871 1800 (US) +1 818 871 1800 (international)

debby@ixiacom.com
http://www.ixiacom.com

------_=_NextPart_001_01C16E3E.45FBEB90-- _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From daemon@ns.ietf.org Thu Nov 15 20:32:39 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01142 for ; Thu, 15 Nov 2001 20:32:39 -0500 (EST) Received: (from daemon@localhost) by optimus.ietf.org (8.9.1a/8.9.1) id UAA13879 for bmwg-archive@odin.ietf.org; Thu, 15 Nov 2001 20:32:41 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA13850; Thu, 15 Nov 2001 20:32:18 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id UAA13821 for ; Thu, 15 Nov 2001 20:32:16 -0500 (EST) Received: from zrc2s03g.us.nortel.com (h66s122a103n47.user.nortelnetworks.com [47.103.122.66]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01108 for ; Thu, 15 Nov 2001 20:32:13 -0500 (EST) Received: from smtprch2.nortel.com (erchg0k.us.nortel.com [47.113.64.104]) by zrc2s03g.us.nortel.com (8.9.3+Sun/8.9.1) with ESMTP id TAA26436 for ; Thu, 15 Nov 2001 19:31:45 -0600 (CST) Received: from zsc4c000.us.nortel.com by smtprch2.nortel.com; Thu, 15 Nov 2001 19:24:39 -0600 Received: by zsc4c000.us.nortel.com with Internet Mail Service (5.5.2653.19) id ; Thu, 15 Nov 2001 17:30:30 -0800 Message-ID: <5D630265EF50D311ABB60008C7917DB6083E24CC@zsc4c004.us.nortel.com> From: "John Dawson" To: Debby Stopp , bmwg@ietf.org Subject: RE: [bmwg] bmwg-multicast document review Date: Thu, 15 Nov 2001 17:30:28 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C16E3E.45FBEB90" X-Orig: Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C16E3E.45FBEB90 Content-Type: text/plain; charset="iso-8859-1" >> Parameters to be measured MUST include number of multicast frames offered, >> number of unicast frames offered, number of multicast frames received and >> number of unicast frames received. You are assuming that test duration is a constant. This is not true. Test duration (Time in seconds) is a measured quantity. This is the differentiator between OfferedLoad and IntendedLoad. You can not report offered load unless time is a measurement. All methodology MUST include a measurement of the TIME it takes to transmit the offered frames. If you measure time at the receive side, then and only then is the forwarding rate accurately known. By including Time as a required measurement parameter, you will automatically address the issues associated with flow control and forward/back-pressure. I do not believe that it is appropriate to put test duration into the general overhead section. Test duration MUST be measured, but SHOULD also be reported. If you choose not to report time directly, then time can be deduced by calculating backwards from the Offered Load. Time == FramesOffered/OfferedLoad. It is also important to note that there are two methods that have been suggested as a valid methodology for determining offered Load. One is based upon transmitting a fixed quantity of packets and measuring the time required to transmit the packets. If multiple ports are transmitting, then TIME must be measured on each port. The other method, is to transmit packets for a fixed time. The number of packets transmitted by each port then becomes the measured quantity. I prefer the fixed TIME approach as it represents a uniform loading (pressure) for all ports across a given test. If a fixed quantity of packets are used, then the test can be very non-uniform. For instance, assume that half-duplex systems are being tested using bi-directional traffic and further assume that one port captures the wire. In this scenario, you actually have two separate uni-directional streams where the first stream transmits for the first half of the test and the other stream transmits for the second half of the test (after the first stream goes idle). In this scenario, the offered loads are not what you intended. Unless you make TIME a parameter that MUST be measured, you can not determine the offered load -- you will never know what happened during the test and you will be reporting bogus numbers. John -----Original Message----- From: Debby Stopp [mailto:dstopp@ixiacom.com] Sent: Thursday, November 15, 2001 4:13 PM To: Dawson, John [BLVWA:382:EXCH]; bmwg@ietf.org Subject: RE: [bmwg] bmwg-multicast document review See my comments below. > >> Section 3.1. Test Considerations > > >> Moreover, no provisions are > >> made for traffic-affecting factors, such as congestion control or > >> service differentiation mechanisms. Modifications to the > specified > >> collection procedures might need to be made to accommodate the > >> transmission media actually tested. These accommodations MUST be > >> presented with the test results. > I don't agree with this language. We have suffered for too > many years with throughput and performance > tests that allow congestion control and forward pressure > techniques to skew results. It is time to close > these holes. This paragraph essentially grants license to > change any collection procedures to accommodate > some designer agenda. If the procedures laid out in this RFC > can not be used to characterized a specific > media, then this RFC should not be used as a basis for > analyzing that technology. Although the authors > specify that any accommodations must be presented with the > test results, this action allows anyone to > re-engineer the test process and then leaves it to the > ingenuity of the reader to catch the tricks that may > be employed to mis-direct the less educated. > Are we agreeing to a process here? or are we granting > everyone free license to re-write their own test? This paragraph specifically worded to allow someone to use the methodology w/out regard to media. The intention is not to overload the receive ports & thus deal w/congestion & flow control , but rather to measure the ability to forward IPMC traffic in a particular scenario. Thus, I will add a sentence to this section stating that these tests MUST be run w/out flow control, etc, like this: ******** 3.1. Test Considerations The procedures outlined below are written without regard for specific physical layer or link layer protocols. The methodology further assumes a uniform medium topology. Issues regarding mixed transmission media, such as speed mismatch, headers differences, etc., are not specifically addressed. Flow control, QoS and other traffic-affecting mechanisms MUST be disabled. Modifications to the specified collection procedures might need to be made to accommodate the transmission media actually tested. These accommodations MUST be presented with the test results. ********* > >> Section 4.1 Mixed Class Throughput > >> Result > >> Parameters to be measured SHOULD include the frame loss > and percent > >> loss for each class of traffic per destination port. The ratio of > >> unicast traffic to multicast traffic MUST be reported. > Let's specify what must be measured, not what should be measured. I can reword this section <4.1> as follows: ******************************** Result Parameters to be measured SHOULD include the frame loss and percent loss for each class of traffic per destination port. The ratio of unicast traffic to multicast traffic MUST be reported. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. Parameters to be measured MUST include number of multicast frames offered, number of unicast frames offered, number of multicast frames received and number of unicast frames received. In addition, all other reporting parameters of mixed class throughput MUST be reflected in the results report, such as the transmitted packet size(s) and the granularity of the measurement process. Results for mixed class throughput MUST include the following parameters: the maximum aggregate offered frames per second from the source ports for both multicast and unicast frames at which no loss forwarding is observed and the ratio of unicast traffic to multicast traffic at each destination port. ******************************** Note - as the test duration must be included in all results reports, I can reword this section as follows: ******************************** 3.1.5. Trial Duration The duration of the test portion of each trial SHOULD be at least 30 seconds. This parameter MUST be included as part of the results reporting for each methodology. ******************************** I prefer to put this in a more generalized section, as I believe all tests should always report the test duration and I prefer not to cut and paste the same words throughout the document. For section 4.2, I have reworded as follows: ******************************** Results Parameters to be measured MUST include the offered load and forwarding rate as a function of the total number of multicast groups for each test iteration. The nature of the traffic stream contributing to the result MUST be reported, specifically number of source and destination ports within the multicast group. In addition, all other reporting parameters of the scaled group forwarding matrix methodology MUST be reflected in the results report, such as the transmitted packet size(s) and offered load of the packet stream for each source port. Result reports MUST include the following parameters for each iteration: the number of frames offered, number of frames received per each group, number of multicast groups and forwarding rate, in frames per second. Constructing a table that contains the forwarding rate vs. number of groups is desirable. ******** You had some comments re: the aggregated test; here's a revised version, see draft for revised section. ******** > >> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput this section has changed significantly based on Kevin's comments - please review again. > >> Section 5. Forwarding Latency This section has been reworded a bit thanks to Kevin's comments, see draft. Debby Stopp Principal Software Engineer IXIA 26601 W. Agoura Rd. Calabasas CA 91302 Tel: 818 871 1800 (US) +1 818 871 1800 (international) debby@ixiacom.com http://www.ixiacom.com ------_=_NextPart_001_01C16E3E.45FBEB90 Content-Type: text/html; charset="iso-8859-1" RE: [bmwg] bmwg-multicast document review

>> Parameters to be measured MUST include number of multicast frames offered,
>> number of unicast frames offered, number of multicast frames received and
>> number of unicast frames received.

You are assuming that test duration is a constant.  This is not true.  Test
duration (Time in seconds) is a measured quantity.  This is the differentiator
between OfferedLoad and IntendedLoad.  You can not report offered load unless
time is a measurement.  All methodology MUST include a measurement of the TIME
it takes to transmit the offered frames.

If you measure time at the receive side, then and only then is the forwarding
rate accurately known.

By including Time as a required measurement parameter, you will automatically
address the issues associated with flow control and forward/back-pressure.

I do not believe that it is appropriate to put test duration into the general
overhead section.  Test duration MUST be measured, but SHOULD also be reported.
If you choose not to report time directly, then time can be deduced by calculating
backwards from the Offered Load.  Time == FramesOffered/OfferedLoad.

It is also important to note that there are two methods that have been suggested
as a valid methodology for determining offered Load.  One is based upon transmitting
a fixed quantity of packets and measuring the time required to transmit the packets.
If multiple ports are transmitting, then TIME must be measured on each port.  The
other method, is to transmit packets for a fixed time.  The number of packets
transmitted by each port then becomes the measured quantity.

I prefer the fixed TIME approach as it represents a uniform loading (pressure) for
all ports across a given test.  If a fixed quantity of packets are used, then the
test can be very non-uniform.  For instance, assume that half-duplex systems are
being tested using bi-directional traffic and further assume that one port captures
the wire.  In this scenario, you actually have two separate uni-directional streams
where the first stream transmits for the first half of the test and the other stream
transmits for the second half of the test (after the first stream goes idle).  In
this scenario, the offered loads are not what you intended.

Unless you make TIME a parameter that MUST be measured, you can not determine the
offered load -- you will never know what happened during the test and you will be
reporting bogus numbers.

John



-----Original Message-----
From: Debby Stopp [mailto:dstopp@ixiacom.com]
Sent: Thursday, November 15, 2001 4:13 PM
To: Dawson, John [BLVWA:382:EXCH]; bmwg@ietf.org
Subject: RE: [bmwg] bmwg-multicast document review


See my comments below.

> >>  Section 3.1. Test Considerations
>   
> >> Moreover, no provisions are
> >> made for traffic-affecting factors, such as congestion control or
> >> service differentiation mechanisms.  Modifications to the
> specified
> >> collection procedures might need to be made to accommodate the
> >> transmission media actually tested.  These accommodations MUST be
> >> presented with the test results.
> I don't agree with this language.  We have suffered for too
> many years with throughput and performance
> tests that allow congestion control and forward pressure
> techniques to skew results.  It is time to close
> these holes.  This paragraph essentially grants license to
> change any collection procedures to accommodate
> some designer agenda.  If the procedures laid out in this RFC
> can not be used to characterized a specific
> media, then this RFC should not be used as a basis for
> analyzing that technology.  Although the authors
> specify that any accommodations must be presented with the
> test results, this action allows anyone to
> re-engineer the test process and then leaves it to the
> ingenuity of the reader to catch the tricks that may
> be employed to mis-direct the less educated.
> Are we agreeing to a process here? or are we granting
> everyone free license to re-write their own test?

This paragraph specifically worded to allow someone to use the methodology
w/out regard to media.  The intention is not to overload the receive ports &
thus deal w/congestion & flow control <there's already an RFC for that>, but
rather to measure the ability to forward IPMC traffic in a particular
scenario.

Thus, I will add a sentence to this section stating that these tests MUST be
run w/out flow control, etc, like this:

********
3.1.    Test Considerations

The procedures outlined below are written without regard for specific
physical layer or link layer protocols. The methodology further assumes a
uniform medium topology. Issues regarding mixed transmission media, such as
speed mismatch, headers differences, etc., are not specifically addressed.
Flow control, QoS and other traffic-affecting mechanisms MUST be disabled.
Modifications to the specified collection procedures might need to be made
to accommodate the transmission media actually tested.  These accommodations
MUST be presented with the test results.
*********


> >> Section 4.1 Mixed Class Throughput
> >> Result
> >> Parameters to be measured SHOULD include the frame loss
> and percent
> >> loss for each class of traffic per destination port.  The ratio of
> >> unicast traffic to multicast traffic MUST be reported.
> Let's specify what must be measured, not what should be measured.

I can reword this section <4.1> as follows:
********************************
Result

Parameters to be measured SHOULD include the frame loss and percent loss for
each class of traffic per destination port.  The ratio of unicast traffic to
multicast traffic MUST be reported.

The nature of the traffic stream contributing to the result MUST be
reported, specifically number of source and destination ports within the
multicast group. Parameters to be measured MUST include number of multicast
frames offered, number of unicast frames offered, number of multicast frames
received and number of unicast frames received.  In addition, all other
reporting parameters of mixed class throughput MUST be reflected in the
results report, such as the transmitted packet size(s) and the granularity
of the measurement process.

Results for mixed class throughput MUST include the following parameters:
the maximum aggregate offered frames per second from the source ports for
both multicast and unicast frames at which no loss forwarding is observed
and the ratio of unicast traffic to multicast traffic at each destination
port.

********************************

Note - as the test duration must be included in all results reports, I can
reword this section as follows:
********************************
3.1.5.  Trial Duration

The duration of the test portion of each trial SHOULD be at least 30
seconds.  This parameter MUST be included as part of the results reporting
for each methodology.
********************************

I prefer to put this in a more generalized section, as I believe all tests
should always report the test duration and I prefer not to cut and paste the
same words throughout the document.



For section 4.2, I have reworded as follows:
********************************
Results

Parameters to be measured MUST include the offered load and forwarding rate
as a function of the total number of multicast groups for each test
iteration.

The nature of the traffic stream contributing to the result MUST be
reported, specifically number of source and destination ports within the
multicast group.  In addition, all other reporting parameters of the scaled
group forwarding matrix methodology MUST be reflected in the results report,
such as the transmitted packet size(s) and offered load of the packet stream
for each source port.

Result reports MUST include the following parameters for each iteration: the
number of frames offered, number of frames received per each group, number
of multicast groups and forwarding rate, in frames per second.  Constructing
a table that contains the forwarding rate vs. number of groups is desirable.


********
You had some comments <as did others> re: the aggregated test; here's a
revised version, see draft for revised section.
********


> >> 4.4. Encapsulation/Decapsulation (Tunneling) Throughput

this section has changed significantly based on Kevin's comments - please
review again.


> >> Section  5. Forwarding Latency

This section has been reworded a bit thanks to Kevin's comments, see draft.
 

Debby Stopp
Principal Software Engineer


IXIA
26601 W. Agoura Rd.
Calabasas
CA 91302
Tel: 818 871 1800 (US) +1 818 871 1800 (international)

debby@ixiacom.com
http://www.ixiacom.com

------_=_NextPart_001_01C16E3E.45FBEB90-- _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From bmwg-admin@ietf.org Tue Nov 20 14:23:53 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00581 for ; Tue, 20 Nov 2001 14:23:53 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA01988; Tue, 20 Nov 2001 14:23:18 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA01955 for ; Tue, 20 Nov 2001 14:23:16 -0500 (EST) Received: from rip.psg.com (exim@rip.psg.com [147.28.0.39]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00509 for ; Tue, 20 Nov 2001 14:23:09 -0500 (EST) Received: from randy by rip.psg.com with local (Exim 3.33 #1) id 166GUC-0007HQ-00 for bmwg@ietf.org; Tue, 20 Nov 2001 11:23:12 -0800 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit To: bmwg@ietf.org Message-Id: Date: Tue, 20 Nov 2001 11:23:12 -0800 Content-Transfer-Encoding: 7bit Subject: [bmwg] draft-rai-test-auto-devices-00.txt Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org Content-Transfer-Encoding: 7bit check out draft-rai-test-auto-devices-00.txt _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From daemon@ns.ietf.org Tue Nov 20 14:23:55 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00597 for ; Tue, 20 Nov 2001 14:23:55 -0500 (EST) Received: (from daemon@localhost) by optimus.ietf.org (8.9.1a/8.9.1) id OAA02038 for bmwg-archive@odin.ietf.org; Tue, 20 Nov 2001 14:23:58 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA01988; Tue, 20 Nov 2001 14:23:18 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id OAA01955 for ; Tue, 20 Nov 2001 14:23:16 -0500 (EST) Received: from rip.psg.com (exim@rip.psg.com [147.28.0.39]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00509 for ; Tue, 20 Nov 2001 14:23:09 -0500 (EST) Received: from randy by rip.psg.com with local (Exim 3.33 #1) id 166GUC-0007HQ-00 for bmwg@ietf.org; Tue, 20 Nov 2001 11:23:12 -0800 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit To: bmwg@ietf.org Message-Id: Date: Tue, 20 Nov 2001 11:23:12 -0800 Content-Transfer-Encoding: 7bit Subject: [bmwg] draft-rai-test-auto-devices-00.txt Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org Content-Transfer-Encoding: 7bit check out draft-rai-test-auto-devices-00.txt _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From bmwg-admin@ietf.org Thu Nov 29 11:14:40 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17456 for ; Thu, 29 Nov 2001 11:14:39 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA18054; Thu, 29 Nov 2001 11:12:34 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA18025 for ; Thu, 29 Nov 2001 11:12:32 -0500 (EST) Received: from magenta.juniper.net (natint.juniper.net [207.17.136.129]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17269 for ; Thu, 29 Nov 2001 11:12:29 -0500 (EST) Received: from juniper.net (ssh.juniper.net [207.17.136.39]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id fATGC2141114 for ; Thu, 29 Nov 2001 08:12:02 -0800 (PST) (envelope-from kdubray@juniper.net) Message-ID: <3C065E51.DA8CCEE1@juniper.net> Date: Thu, 29 Nov 2001 11:12:02 -0500 From: Kevin Dubray X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: bmwg@ietf.org Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Subject: [bmwg] 52nd IETF - Benchmarking Methodology WG (bmwg) Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org Content-Transfer-Encoding: 7bit Draft agenda for the BMWG at the 52nd IETF. -Kevin ------------------------------------------------- Benchmarking Methodology WG (bmwg) TUESDAY, December 11, 2001, 0900-1130 ===================================== AGENDA: 1. Issues regarding the specification of a unified benchmarking methodology for Basic BGP convergence. ftp://ftp.ietf.org/internet-drafts/draft-ietf-bmwg-bgpbas-00.txt ftp://ftp.ietf.org/internet-drafts/draft-ietf-bmwg-conterm-00.txt Please help contribute to a successful meeting by reading the above I-D(s) before the meeting. To offer comments on BMWG work in progress or the agenda itself, please send email to: bmwg@ietf.org Alternatively, to offer potential agenda items, please email: kdubray@juniper.net _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg From daemon@optimus.ietf.org Thu Nov 29 11:14:41 2001 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17471 for ; Thu, 29 Nov 2001 11:14:41 -0500 (EST) Received: (from daemon@localhost) by optimus.ietf.org (8.9.1a/8.9.1) id LAA18133 for bmwg-archive@odin.ietf.org; Thu, 29 Nov 2001 11:14:43 -0500 (EST) Received: from optimus.ietf.org (localhost [127.0.0.1]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA18054; Thu, 29 Nov 2001 11:12:34 -0500 (EST) Received: from ietf.org (odin [132.151.1.176]) by optimus.ietf.org (8.9.1a/8.9.1) with ESMTP id LAA18025 for ; Thu, 29 Nov 2001 11:12:32 -0500 (EST) Received: from magenta.juniper.net (natint.juniper.net [207.17.136.129]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17269 for ; Thu, 29 Nov 2001 11:12:29 -0500 (EST) Received: from juniper.net (ssh.juniper.net [207.17.136.39]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id fATGC2141114 for ; Thu, 29 Nov 2001 08:12:02 -0800 (PST) (envelope-from kdubray@juniper.net) Message-ID: <3C065E51.DA8CCEE1@juniper.net> Date: Thu, 29 Nov 2001 11:12:02 -0500 From: Kevin Dubray X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: bmwg@ietf.org Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Subject: [bmwg] 52nd IETF - Benchmarking Methodology WG (bmwg) Sender: bmwg-admin@ietf.org Errors-To: bmwg-admin@ietf.org X-Mailman-Version: 1.0 Precedence: bulk List-Id: Benchmarking Methodology Working Group X-BeenThere: bmwg@ietf.org Content-Transfer-Encoding: 7bit Draft agenda for the BMWG at the 52nd IETF. -Kevin ------------------------------------------------- Benchmarking Methodology WG (bmwg) TUESDAY, December 11, 2001, 0900-1130 ===================================== AGENDA: 1. Issues regarding the specification of a unified benchmarking methodology for Basic BGP convergence. ftp://ftp.ietf.org/internet-drafts/draft-ietf-bmwg-bgpbas-00.txt ftp://ftp.ietf.org/internet-drafts/draft-ietf-bmwg-conterm-00.txt Please help contribute to a successful meeting by reading the above I-D(s) before the meeting. To offer comments on BMWG work in progress or the agenda itself, please send email to: bmwg@ietf.org Alternatively, to offer potential agenda items, please email: kdubray@juniper.net _______________________________________________ bmwg mailing list bmwg@ietf.org http://www1.ietf.org/mailman/listinfo/bmwg