Subject: Re: [bmwg] SIP Benchmarking methodology and terminology - version 4 updates

From: Carol Davids <[email-masked]>
Date: Mon, 16 Jul 2012 13:23:06 -0500

Aamer,

Thank you very much for your discussion comments. We greatly appreciate
them.  Please see our responses below.

Comments and responses:
1. Can we please add a statement with regards to the relationship
between this work and rfc6076? (something more deeper than BMWG is in
the lab).

CD/VG: RFC6076 relates to end-to-end performance metrics. It includes
measurements of the percent of successful calls to total attempted calls,
for example. The current drafts relate to a steady-state measurement in
which no calls fail. We seek the maximum delivery rate that the system
can sustain without failure. Some of the metrics to be observed in
RFC6076 are ones that are set as input parameters to the tests defined in
our SIP Benchmarking Methodology draft. We added this information to the
new draft.

2. Have anybody checked to see if the metrics out of rfc6076 and this
work are comparable?

CD/VG: Yes, we did check. See 1) above.

3. "However, benchmarking SIP
presence is not a part of this work item.". Is there a reason for
this. Presence would be one of the more interesting cases to understand.

CD/VG: It could be done.  We thought the the application had too many
variables and
did not belong in a basic SIP benchmarking specification.  If the
Emulated Agent sends a SUBSCRIBE message to a presence server, the proxy
function within the DUT/SUT will forward that request to the presence
server, either statefully or statelessly. If the DUT/SUT contains the
presence server then of course it will work very hard. If it does not
contain the presence server then it will not work very hard. But if you
want to compare two different devices that both contain a presence server
there seemed to be too many variables.

To accurately benchmark presence requires a separate architecture,
processing elements, processing rules, and filtering rules as
required by the SIMPLE WG. Our aim in this particular work was
to get baseline SIP multimedia session setup benchmarked, with
a few simple extensions such as IM. Presence is far more
complex to be included in this work.

4. "Only session establishment is considered for the performance
benchmarks.  Session disconnect is not considered in the scope of
this work item." I would think that given connectivity churn
(links going up and down, power outages, etc), both establishment as
well as disconnect are interest items. Consider the cleanup work needed
after a disconnect... some of this might be a high workload than simply
maintaining a connection.

CD/VG: The cleanup work is taken into account in the measurement already.
We are
setting up and tearing down calls in order to find the rate at which we
can do so without any failures. So if we indicate that a call will last
for 5 seconds, that includes the fact that it will be disconnected after 5
seconds. The metric we are interested in is the session throughput. We
will add a sentence that explains this.

5. term-04, figure 12 seems misformatted, or it needs a better
explanation of how to read it...

CD/VG: Thanks. There appear to be a number of these. Our vertical lines
are not
lining up as intended. We will need to fix this.

6. For the case of video conferencing the SDP can get quite large. I am
wondering if this (SDP size) is an interesting axis to explore.

CD/VG: If the DUT/SUT processes the SDP it might be interesting. The size
of the
SDP will be affected by the choice of codec and the consequent number of
parameter lines. We could add the 'codec type' as a parameter of test.

Note that if the DUT is a user agent, then the benchmark
numbers derived from it already include the processing of SDP
since user agents (end points) must process the SDP to set up
sessions.

Best,

Carol

Carol Davids
Professor &  Director RTC Lab
Illinois Institute of Technology
Office: 630-682-6024
Mobile: 630-292-9417
Email: [email-masked]
Skype: caroldavids1
Web: rtc-lab.itm.iit.edu