Subject: Fwd: FW: [bmwg] FW: IETF BMWG Work Item Proposal for SIP PerformanceBenchmarking

From: "Craig White" <[email-masked]>
Date: Fri, 23 Jun 2006 11:47:18 -0400

Greetings,

As always, I do agree that this is a fine item of work for the bmwg.
However, I have some reservations about what we call the effort and what the
real scope is.

At this point, everything I have seen would indicate that this is a
benchmark about a particular (and in my mind, corner case) of SIP, (and
RTP), mainly, a particular "VoIP" implementation as it is sold by current
vendors and/or implemented by the current service providers. This fact,
while alluded to in each section of the work proposal, is never stated which
tends to give the illusion of a benchmarking effort that is more than what
is proposed.

Will we cover the more "simple" cases of SIP usage and then continue to
build on these so that the tests and testbeds become more and more focused
until eventully we come to the current VoIP technical solution space, or do
we simply start there?

Would this include benchmarking SIP parameters dealing with presence? or
multicast? or is it only about "VoIP calls"?

I am intentionally staying away from RTP (media) issues in this posting, but
I would like to mention that sonet over packet, was originally proposed in
AVT, and I have  follow-on drafts about how to use SIP to signal these
"circuits" :)

I am simply suggesting that some work be put into the "name" of this effort
so as not to confuse. I would suggest something along the lines of
"Benchmark for evaluation of SIP to signal VoIP calls using session border
controllers, security gateways, and controllers"

We can cover the other issues as the methodologies and terminologies are
proposed and discussed.

-craig

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, May 23, 2006 5:50 PM
To: [email-masked]
Subject: [bmwg] FW: IETF BMWG Work Item Proposal for SIP
PerformanceBenchmarking
Importance: High

Fellow BMWG-ers,

Please find this IETF Work Item Proposal for your review and comment.
The
authors will be submitting a first draft Terminology and Methodology for
the
Montreal meeting.

Scott

>  -----Original Message-----
> From:         Poretsky, Scott
> Sent: Thursday, May 11, 2006 12:44 PM
> To:   [email-masked]'
> Subject:      IETF BMWG Work Item Proposal for SIP Performance
> Benchmarking
> Importance:   High
>
> Hi Al,
>
[snip]

> Thanks,
> Scott
>
> ########################
> BMWG Work Item Proposal
>
> TITLE: Session Initiation Protocol (SIP) Performance Benchmarking
>
> AUTHOR TEAM:
> Dr. Carol Davids, Illinois Institute of Technology
> Dr. Vijay Gurbani, Lucent Technologies
Scott Poretsky, Reef Point Systems
>
> GOALS
> The purpose of this work item is to provide a single terminology and
> methodology from which SIP equipment vendors and VoIP service
providers
> can measure performance benchmarking metrics for comparison and
selection.
> It is intended to develop terms, benchmarks, and methodologies that
can be
> applied to any type of IP device including SIP Servers, Session Border
> Controllers (SBCs), and Security Gateways (SEGs).
>
> MOTIVATION
> Service Providers are now planning VoIP and Multimedia network
deployments
> using the IETF developed Session Initiation Protocol (SIP).  Through
SIP,
> service providers will be able to have rich service offerings and many
> even plan to turn off their Public Switch Telephone Network within
five
> years.   VoIP has led to development of new networking devices
including
> SIP Servers, Session Border Controllers, and Security Gateways.  The
mix
> of voice and IP functions in this variety of devices has produced
> inconsistencies in vendor reported performance metrics and has caused
> confusion in the service provider community.
>
> SCOPE:
> This work item will provide terms, benchmarks, and methodologies for
> performance benchmarking the SIP control and media planes.  The
> methodologies can be used for benchmarking SIP performance of SIP
servers,
> SBCs, and SEGs.  The media used for the benchmarking will be the IETF
> defined Real-Time Protocol (RTP).  Test cases will allow testing of
VoIP
> or Multimedia over IP by varying the number of media streams per SIP
call
> from 1 to higher.  There will be at least one test case to measure the
> impact of a SIP DOS Attack on the performance metrics.  All SIP
traffic
> will be in the clear; no encryption using TLS or IPsec will be used.
> Similar benchmarks and methodologies using encryption is a potential
work
> item for the BMWG that could be considered in the future.
>
> There will be separate test cases in the methodology document for
> obtaining measurements for each of the Benchmarking Metrics defined in
the
> Terminology.  The benchmarking metrics to be measured for the SIP
Control
> Plane and SIP Media Plane are as follow:
>
>       Benchmark Metrics - SIP Control Plane
>       Standing Calls, maximum (calls)
>       Calls Per Second, maximum (CPS)
>       Call Attempts Per Second, maximum (CAPS)
>       Busy Hour Call Attempts, maximum (BHCA)
>       Busy Hour Call Connects, maximum (BHCC)
>       Call Completion Rate (%)
>       Call Setup Delay, average (msec)
>       Call Teardown Delay, average (msec)
>
>       Benchmark Metrics- SIP Media Plane
>       RTP Media Throughput, per Media Stream (pps)
>       RTP Media Throughput, Aggregate (pps)
>       RTP Packet Loss, average (pps)
>       RTP Packet Delay, average (msec)
>       RTP Packet Jitter, average (msec)
>
> In order to obtain measurements for the benchmarking metrics it will
be
> necessary to configure the test setup with certain test parameters.
Some
> of these test parameters may also be the benchmarking metric being
> measured for another test case.  The test parameters to be configured
for
> the SIP Control Plane and SIP Media Plane are as follow:
>
>       Test Parameters- SIP Control Plane
>       Call Duration (msec)
>       Call Per Second (CPS)
>       Call Attempts Per Second (CAPS)
>
>       Test Parameters - SIP Media Plane
>       RTP Media Streams per Call (streams per call)
>       RTP Packet Size (bytes)
>       RTP Media Offered Load, per Media Stream (pps)
>       RTP Media Offered Load, Aggregate (pps)
>
> The basic test topology to be used for this benchmarking is as
follows:
>
> Emulated Agents<--> SEG (optional) <--> SIP Proxy Server (or SBC) <-->
SIP
> Server (optional) <--> Emulated Agents
>
> PROPOSED MILESTONES
> 06/07 First Draft Terminology and Methodology
> 02/07 First WG Last Call
> 05/07 SIP WG and SIPPING WG Review
> 06/07 Final BMWG Last Call for Terminology and Methodology
> 08/07 Submittal for IESG review
>
>
>

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg