Subject: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg)

From: The IESG <[email-masked]>
Date: Thu, 12 Jun 2014 11:00:54 -0700

The Benchmarking Methodology (bmwg) working group in the Operations and
Management Area of the IETF has been rechartered. For additional
information please contact the Area Directors or the WG Chairs.

Benchmarking Methodology (bmwg)
------------------------------------------------
Current Status: Active WG

Chairs:
  Sarah Banks <[email-masked]>
  Al Morton <[email-masked]>

Assigned Area Director:
  Joel Jaeggli <[email-masked]>

Mailing list
  Address: [email-masked]
  To Subscribe: [email-masked]
  Archive: http://www.ietf.org/mail-archive/web/bmwg/

Charter:

The Benchmarking Methodology Working Group (BMWG) will continue to
produce a series of recommendations concerning the key performance
characteristics of internetworking technologies, or benchmarks for
network devices, systems, and services. Taking a view of networking
divided into planes, the scope of work includes benchmarks for the
management, control, and forwarding planes.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

The set of relevant benchmarks will be developed with input from the
community of users (e.g., network operators and testing organizations)
and from those affected by the benchmarks when they are published
(networking and test equipment manufacturers). When possible, the
benchmarks and other terminologies will be developed jointly with
organizations that are willing to share their expertise. Joint review
requirements for a specific work area will be included in the detailed
description of the task, as listed below.

To better distinguish the BMWG from other measurement initiatives in the
IETF, the scope of the BMWG is limited to the characterization of
implementations of various internetworking technologies
using controlled stimuli in a laboratory environment. Said differently,
the BMWG does not attempt to produce benchmarks for live, operational
networks. Moreover, the benchmarks produced by this WG shall strive to
be vendor independent or otherwise have universal applicability to a
given technology class.

Because the demands of a particular technology may vary from deployment
to deployment, a specific non-goal of the Working Group is to define
acceptance criteria or performance requirements.

An ongoing task is to provide a forum for development and
advancement of measurements which provide insight on the
capabilities and operation of implementations of inter-networking
technology.

Ideally, BMWG should communicate with the operations community 
through organizations such as NANOG, RIPE, and APRICOT.

The BMWG is explicitly tasked to develop benchmarks and methodologies 
for the following technologies:

BGP Control-plane Convergence Methodology (Terminology is complete):
With relevant performance characteristics identified, BMWG will prepare
a Benchmarking Methodology Document with review from the Routing Area
(e.g., the IDR working group and/or the RTG-DIR). The Benchmarking
Methodology will be Last-Called in all the groups that previously
provided input, including another round of network operator input during
the last call.

SIP Networking Devices: Develop new terminology and methods to
characterize the key performance aspects of network devices using
SIP, including the signaling plane scale and service rates while
considering load conditions on both the signaling and media planes. This
work will be harmonized with related SIP performance metric definitions
prepared by the PMOL working group.

Traffic Management: Develop the methods to characterize the capacity 
of traffic management features in network devices, such as
classification, 
policing, shaping, and active queue management. Existing terminology 
will be used where appropriate. Configured operation will be verified 
as a part of the methodology. The goal is a methodology to assess the 
maximum forwarding performance that a network device can sustain without 
dropping or impairing packets, or compromising the accuracy of multiple 
instances of traffic management functions. This is the benchmark for 
comparison between devices. Another goal is to devise methods that 
utilize flows with congestion-aware transport as part of the traffic 
load and still produce repeatable results in the isolated test
environment.

IPv6 Neighbor Discovery: Large address space in IPv6 subnets presents 
several networking challenges, as described in RFC 6583. Indexes to 
describe the performance of network devices, such as the number of 
reachable devices on a sub-network, are useful benchmarks to the 
operations community. The BMWG will develop the necessary 
terminology and methodologies to measure such benchmarks.

In-Service Software Upgrade: Develop new methods and benchmarks to 
characterize the upgrade of network devices while in-service, 
considering both data and control plane operations and impacts. 
These devices are generally expected to maintain control plane session 
integrity, including routing connections. Quantification of upgrade 
impact will include packet loss measurement, and other forms of recovery 
behavior will be noted accordingly. The work will produce a definition 
of ISSU, which will help refine the scope. Liaisons will be established 
as needed.

Data Center Benchmarking: This work will define additional terms, 
benchmarks, and methods applicable to data center performance
evaluations. 
This includes data center specific congestion scenarios, switch buffer 
analysis, microburst, head of line blocking, while also using a wide mix 
of traffic conditions. Some aspects from BMWG's past work are not 
meaningful when testing switches that implement new IEEE specifications 
in the area of data center bridging. For example, throughput as defined 
in RFC 1242 cannot be measured when testing devices that implement three 
new IEEE specifications: priority-based flow control (802.1Qbb); 
priority groups (802.1Qaz); and congestion notification (802.1Qau).
This work will update RFC1242, RFC2544, RFC2889 (and other key RFCs), 
and exchange Liaisons with relevant SDOs, especially at WG Last Call.

VNF and Related Infrastructure Benchmarking: Benchmarking Methodologies 
have reliably characterized many physical devices. This work item extends

and enhances the methods to virtual network functions (VNF) and their 
unique supporting infrastructure. A first deliverable from this activity 
will be a document that considers the new benchmarking space to ensure
that common issues are recognized from the start, using background 
materials from industry and SDOs (e.g., IETF, ETSI NFV).
Benchmarks for platform capacity and performance characteristics of 
virtual routers, switches, and related components will follow, including 
comparisons between physical and virtual network functions. In many
cases,
the traditional benchmarks should be applicable to VNFs, but the lab 
set-ups, configurations, and measurement methods will likely need to 
be revised or enhanced.

Milestones:
  Jun 2014 - Basic BGP Convergence Benchmarking Methodology to IESG
Review
  Jul 2014 - Terminology for SIP Device Benchmarking to IESG Review
  Jul 2014 - Methodology for SIP Device Benchmarking to IESG Review
  Aug 2014 - Draft on Traffic Management Benchmarking to IESG Review
  Dec 2014 - Draft on IPv6 Neighbor Discovery to IESG Review
  Mar 2015 - Draft on In-Service Software Upgrade Benchmarking to IESG
Review
  Aug 2015 - Draft on VNF Benchmarking Considerations to IESG Review
  Dec 2015 - Drafts on Data Center Benchmarking to IESG Review

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 12 Jun 2014 14:21:05 -0400

We made it this far!  Everything we asked to do is in there.
Let's get to work and meet our milestones.

Al
bmwg co-chair

> -----Original Message-----
> From: bmwg [mailto:[email-masked]] On Behalf Of The IESG
> Sent: Thursday, June 12, 2014 2:01 PM
> To: IETF-Announce
> Cc: bmwg WG
> Subject: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg)
> 
> of ISSU, which will help refine the scope.??Liaisons will be established
> as needed.
>

---

From: Barry Constantine <[email-masked]>
Date: Thu, 12 Jun 2014 18:39:58 +0000

Hi Al,

This was good news to see, thanks for you and Sarah's perseverance.

It was funny, I essentially clicked the save button on the next version of the Traffic Management work and then this charter email popped up.

Thank you,
Barry Constantine

JDSU Network and Service Enablement
Principal Member Technical Staff
301-325-7069

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: Thursday, June 12, 2014 2:21 PM
To: bmwg WG
Subject: Re: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg)

Al
bmwg co-chair

> The Benchmarking Methodology (bmwg) working group in the Operations 
> and Management Area of the IETF has been rechartered. For additional 
> are pertinent to that class; clearly identify a set of metrics that 
> aid in the description of those characteristics; specify the 
> methodologies required to collect said metrics; and lastly, present 
> the requirements for the common, unambiguous reporting of benchmarking results.
> 
> To better distinguish the BMWG from other measurement initiatives in 
> the IETF, the scope of the BMWG is limited to the characterization of 
> implementations of various internetworking technologies using 
> controlled stimuli in a laboratory environment. Said differently, the 
> BMWG does not attempt to produce benchmarks for live, operational 
> Because the demands of a particular technology may vary from 
> deployment to deployment, a specific non-goal of the Working Group is 
> to define acceptance criteria or performance requirements.
> 
> An ongoing task is to provide a forum for development and advancement 
> of measurements which provide insight on the capabilities and 
> operation of implementations of inter-networking technology.
> 
> Ideally, BMWG should communicate with the operations community through 
> organizations such as NANOG, RIPE, and APRICOT.
> 
> With relevant performance characteristics identified, BMWG will 
> prepare a Benchmarking Methodology Document with review from the 
> Routing Area (e.g., the IDR working group and/or the RTG-DIR). The 
> Benchmarking Methodology will be Last-Called in all the groups that 
> previously provided input, including another round of network operator 
> input during the last call.
> 
> characterize the key performance aspects of network devices using SIP, 
> including the signaling plane scale and service rates while 
> considering load conditions on both the signaling and media planes. 
> This work will be harmonized with related SIP performance metric 
> definitions prepared by the PMOL working group.
> 
> classification, policing, shaping, and active queue management. 
> Existing terminology will be used where appropriate. Configured 
> operation will be verified as a part of the methodology. The goal is a 
> methodology to assess the maximum forwarding performance that a 
> network device can sustain without dropping or impairing packets, or 
> compromising the accuracy of multiple instances of traffic management 
> functions. This is the benchmark for comparison between devices. 
> Another goal is to devise methods that utilize flows with 
> congestion-aware transport as part of the traffic load and still 
> produce repeatable results in the isolated test environment.
> 
> operations community. The BMWG will develop the necessary terminology 
> and methodologies to measure such benchmarks.
> 
> impact will include packet loss measurement, and other forms of 
> recovery behavior will be noted accordingly. The work will produce a 
> definition of ISSU, which will help refine the scope.??Liaisons will 
> be established as needed.
> 
> analysis, microburst, head of line blocking, while also using a wide 
> mix of traffic conditions. Some aspects from BMWG's past work are not 
> meaningful when testing switches that implement new IEEE 
> specifications in the area of data center bridging. For example, 
> throughput as defined in RFC 1242 cannot be measured when testing 
> devices that implement three new IEEE specifications: priority-based 
> flow control (802.1Qbb); priority groups (802.1Qaz); and congestion notification (802.1Qau).
> VNF and Related Infrastructure Benchmarking: Benchmarking 
> Methodologies have reliably characterized many physical devices. This 
> work item extends
> 
> unique supporting infrastructure. A first deliverable from this 
> activity will be a document that considers the new benchmarking space 
> to ensure that common issues are recognized from the start, using 
> background materials from industry and SDOs (e.g., IETF, ETSI NFV).
> virtual routers, switches, and related components will follow, 
> including comparisons between physical and virtual network functions. 
> In many cases, the traditional benchmarks should be applicable to 
> VNFs, but the lab set-ups, configurations, and measurement methods 
> will likely need to be revised or enhanced.
> 
> Milestones:

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Banks, Sarah" <[email-masked]>
Date: Thu, 12 Jun 2014 13:44:30 -0500

Barry, BMWG,
	Barry, that's fantastic news. Thanks for your kind words, and effort! I
look forward to your next update :)
	Al, you beat me to the draw :) Thanks for all your help with this!
	
	BMWG, I'm excited by what's on our charter, and where that might lead
moving forward! Please note, if you plan on attending the upcoming F2F
meeting in Toronto, or having a draft reviewed, the cut off date is very
American this year - it's July 4th. Toronto is also my home town; let me
know if you have any questions. :)

Kind regards,
Sarah

On 6/12/14, 11:39 AM, "Barry Constantine" <[email-masked]>
wrote:

>It was funny, I essentially clicked the save button on the next version
>of the Traffic Management work and then this charter email popped up.
>
>Thank you,
>Barry Constantine
>