Subject: [bmwg] SIP work feedback from Prague

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 22 Apr 2011 13:33:21 -0500

Oops, sorry for the resend.  I forgot to add Scott
Bradner in the To list since he is the "Scott" I
mention below.

Scott: Thank you for your comments on the SIP work during
BMWG meeting in Prague.  Here is an attempt to close
the open issues.

To wit, you had indicated the following issues:

1) The 50% step rate --- clarify that the decrease in certain
  test cases of 50% applies to decreasing the previous rate
  by 50%, not the base rate one starts with.

2) When testing, do not stop at the first error, and instead,
  use a time slice to conduct the test.  This allows other
  latent behaviour in the device to become manifest, that would
  otherwise not if the testing stopped at the first error.

3) Indicate that the idea of the SIP tests is to run the tests
with a middle box device from Vendor-A and repeat the tests
with a middle box device from Vendor-B to obtain a comparison
of performance between Vendors A and B.

Issue (1) is fairly cut and dry; indeed, we will do the changes
you suggest.  The same goes for (3).

Carol and I had a long chat about issue (2).  The intent of the SIP
work in bmwg is to discover the highest request rate that can be
successfully processed by a system or device before the load overwhelms
the system or device.  In this thinking, we are interested in knowing
the maximum number of SIP requests (registrations, invitations with
and without media, instant messages, etc.) that a system or device
can process before the load overwhelms the system or device such that
it is unable to process any more SIP requests.  In this context, we do
actually want to stop on the first error and recalibrate the
request rate appropriately.

The type of test you are referring to in (2) is a valid test, and
it is actually included in the current draft, albeit it is couched
in different terms (i.e., run the system or device under test with
the highest request rate for 100,000 requests).

What we propose is this: let's break apart the calibration
needed to figure out the highest request rate that can be successfully
processed by the system or device under test from the steady state
presentation of a load consisting of the highest request rate to
the system or device under test.  The former allows us to determine
the highest request rate number and the latter will allay your
concerns.

We plan to describe the above process in prose as well as pseudo-
code.  Here is the suggested text we plan to use.  Please do
go through the prose as well as the pseudo-code (we are sure they
match, but "given enough eyeballs ...")

Here is the prose description.

Our goal is to find the largest value of a SIP session-request-rate,
measured in sessions-per-second, which the DUT/SUT can process with
zero errors.  To discover that number, we first use an iterative
process to find a candidate for this rate.  Next, we subject the
DUT/SUT to an offered load whose arrival rate is set to that of the
candidate rate. This test is run for an extended period of time, which
we refer to as infinity, and which is, itself, a parameter of the test
labeled T in the pseudo-code).   We refer to this second phase of
testing as the steady-state phase. If errors are encountered during
this steady-state phase, then we reduce the candidate rate by a defined
percent, also a parameter of test, and  enter the steady-state phase
again until a final steady-state rate is achieved.

The iterative process is defined as follows.  We select a starting rate
of 100 sessions per second.  We run the test for the time period
identified by t in the pseudo-code below. If no failures occur, we
increase the rate to 150 sps and again test for time period t. We
continue to ramp up the attempt rate until we encounter a failure
before the end of the test time t.  Then we try an attempt rate that is
higher than the last successful attempt rate by a quantity equal to
half the difference between the rate at which failures occurred and the
last successful rate.  If this new attempt rate also results in errors
we again try an attempt rate that is higher than the last successful
attempt rate by a quantity equal to half the difference between the
rate at which failures occurred and the last successful rate.
Continuing in this way, we eventually find an attempt rate without
errors.

The pseudo-code corresponding to the above description follows:

; ---- Parameters of test, adjust as needed
t  := 5000   ; local maximum; used to figure out largest value
T  := 50000  ; global maximum; once largest value has been figured out,
              ; pump this many requests before calling the test a success
m  := {...}  ; other attributes that affect testing, such
              ; as media streams, etc.
s  := 100    ; Initial session attempt rate (in sessions/sec)
C  := 0.05   ; caliberation amount: How much to back down if we have
              ; found steady state and we cannot pump that much traffic
; ---- End of parameters of test

f  := false
F  := false
c  := 0

proc main
    find_largest_value  ; First, figure out the largest value.

    ; Now that the largest value (saved in s) has been figured out,
    ; use it for sending out s requests/s and send out T requests.

    do {
       send_traffic(s, m, T)    ; send_traffic not shown
       if (all requests succeeded) {
          F := true ; test is done
       } else f (even on request failed) {
          s := s - (0.05 * C)  ; Reduce s by caliberation amount
          steady_state
       }
    } while (F == false)
end proc

proc find_largest_value
    ; Iterative process to figure out the largest value we can
    ; handle with no failures
    do  {
        send_traffic(s, m, t) ; Send s request/s with m characteristics
                              ; until t requests have been sent
        if (all requests succeeded) {
           s' := s
           s  := s + (0.5 * s)
           if (c == 1) {
               f := true;
        } else if (even one request failed)  {
           c  := 1
           s  := s - (0.5 * (s - s'))
        }
    } while (f == false)
end proc

Please let me and Carol know if you have further questions.

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{bell-labs.com,acm.org} / [email-masked]
Web:   http://ect.bell-labs.com/who/vkg/
_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: [email-masked] (Scott O. Bradner)
Date: Fri, 22 Apr 2011 15:04:24 -0400 (EDT)

1 & 3 seem just fine

re #2 - this looks ok assuming the request process is single threaded
(send request, wait for response, delay to make sure rate is coreect, loop)

would this still work if the server is multi threaded and the response
latency is longer than the inter-request interval?

Scott

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 22 Apr 2011 14:36:09 -0500

On 04/22/2011 02:04 PM, Scott O. Bradner wrote:
> 1&  3 seem just fine

Scott: Thanks for a quick response.  I hope I have interpreted your
questions correctly and provided an answer in line with your
queries; if not, I apologize before hand and am afraid we'll have
to engage in a bit more email exchange.

That said, please see inline.

> (send request, wait for response, delay to make sure rate is coreect,

Most SIP test tools ("the request process" in your
response above) have the ability to send n requests/sec and
stop sending requests either after some number of seconds,
or when the maximum request limit has been reached (some
support both modes, the one we used supported sending
n req/s until you have sent N total requests, where
N > n).

Now, whether or not the request process is single
threaded or multi-threaded, or uses fork() to create
additional processes is probably not important as long as
the request process can send n req/sec and stop when
sending more requests after a certain --- well defined ---
point.

> would this still work if the server is multi threaded and the
> response latency is longer than the inter-request interval?

Yes, this works regardless of the relationship between response
latency and inter-request arrival interval*.  Most SIP servers
are multi-threaded; in fact, the most widely used open source
SIP server actually uses fork() to create multiple processes
per transport it serves.

* Clearly, the inter-request arrival interval at a SIP server
is inversely proportional to the number of requests issued
per time period.  That said, which response latency are you
worried about?  Provisional response or a final response?
It will take a longer time for a final response to be generated
compared to a provisional.  Final response latency will
usually be longer than inter-request arrival interval, but
the provisional response latency may less than the inter-request
arrival interval when the requests arriving per time period
are fairly low.

Thanks,

- vijay