Subject: RE: [Sipping] Comments on draft-malas-performance-metrics-05.txt

From: "Jean-Francois Mule" <[email-masked]>
Date: Tue, 7 Nov 2006 12:20:09 -0700

Daryl,

   Thank you for initiating this draft.

   It is quite a useful document, and at a minimum, it could bring
common terminology and methods to the long list of implementations that
measure some kinds of metrics like these and report those via various
mechanisms.

   Here are a couple of first comments, some of which we did chat about
yesterday.

--- Template for metrics to guide extensibility:
Given the wg acknowledged that the draft should only focus on the most
widely supported and useable metrics, and that many will come up with
other specific things, it would be useful to define how other metrics
should be defined by vendors, operators.
This could be a simple paragraph providing some kind of template: metric
name, 3 or 4 letter abbreviation, problem statement or motivation, what
is measured, how, in what units and optionally, if it can be averaged,
specify how.

--- IANA registry?
It might be useful to have some discussions on an iana registry for
these. This would allow protocol mechanisms used to report or correlate
these metrics to refer to a well-known registry and it could help
avoiding folks to overload a metric acronym.

--- Section 5.2, Data Collection Considerations
Replace:
< The information may also be transmitted through use of 
< SNMP traps as described in the work in progress SIP MIB draft
 ^     ^^^^^                     ^^^^^^^^^^^^^^^^        ^  
With:
> The information may also be transmitted through the use of 
> network management protocols like SNMP [RFC3410] and via future 
> extensions to the SIP MIB modules []

The reasons are:
 - SNMP notifications may not be the optimal way to fetch metrics in
general
- sure, they could be used if some metrics cross some configured
threshold to create an alarm but folks also use those kinds of metrics
with a statistical approach: over a period of hours/days, how are my
systems doing on call session attempts, etc. so poll operations (or push
at some well defined time intervals) may be the mainstream usage of
this.

Jean-Francois
[email-masked]

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 07 Nov 2006 15:02:40 -0600

> is measured, how, in what units and optionally, if it can be averaged,

There is complementary work in the bmwg WG that touches upon
other metrics.  Please see the following two companion
documents:

http://www.ietf.org/internet-drafts/draft-poretsky-sip-bench-term-02.txt
http://www.ietf.org/internet-drafts/draft-poretsky-sip-bench-term-02.txt

At least for right now, the bmwg drafts and the sipping drafts
are tracking each other closely.

Thanks.

- vijay
-- 
Vijay K. Gurbani  vkg@{lucent.com,research.bell-labs.com,acm.org}
Bell Laboratories, Lucent Technologies, Inc.
2701 Lucent Lane, Rm. 9F-546, Lisle, Illinois 60532 (USA)

---

From: "Reinaldo Penno" <[email-masked]>
Date: Tue, 7 Nov 2006 17:30:44 -0500

Hello Daryl,

This draft is a good idea and should be pursued forward. Here are my
comments:

"Time Begin (TB) - This is the time instant that starts a continuous
time interval running until the related* response is received.  TB
occurs when the designated request has been processed by the application
and last bit of the request packet has been sent from the proxy or UA
(and is externally observable at some physical interface)**."

* related to what?

** This is a little bit confusing. After the request is processed by the
application, it traverses the TCP/IP stack, which imposes a considerable
delay. Then it is put in the output interface and (most likely) sent
out. So, I would suggest you specify exactly where you start TB. This
would be probably after application processing, since you generally have
no way to passing interface counters back to the application.

"The first SIP message indicates the TB associated with the user or
application TB expectation associated with the request."

This phrase could use a rewrite to make it clear

"This metric is commonly calculated as an average.  The following
represents the calculation for this metric as an average:
		 SUM (Time of Final Response - Time of REGISTER Request)
ARRD = -------------------------------------------------------
				         SUM # of REGISTER Requests"

>From an implementaion perspective this seems a bit over the top. Some
customers want exponential moving average, other the mean, other
something else. We should separate the definition of metric, its
collection and its presentation. The draft should focus on the former
and mention the latters on some applicability statement.

I would suggest you leave the exact method or just say that average is
_one_ of the methods that could be used. 
This applies to all other formulas in the draft.

"3.1.2. Failed REGISTER Attempt RRD"

It seems there is a failed register attempt inside every successful
REGISTER completion when a 401 message is received. TS, in the case
above, should a Failed REGISTER attempt be incremented as well?

"seconds and/or milliseconds."

I would suggest you settle on one time measurement. This is the type of
thing that if is left open causes interoperability problems.

Most timers in RFC3261 expire in XX seconds, so I would suggest you go
with seconds for failed things and milliseconds for successful things,
but never both for the same measurement.

"In SIP, the message indicating status would be a non-100 Trying
provisional message received in response to an INVITE request.  In some
cases, a non-100 Trying provisional message is not received, but rather
a 200 message is received as the first status message instead.  "

Hummm...I'm not sure about that. It might make sense to consider a 183 a
successful setup, but clearly not a 181. A response of type 181 would be
similar to 302 below.

What happens if I send a CANCEL before the receipt of a 2xx response. Is
that counted as a successful session attempt or not?

"3.4.1. Successful session completion SDT
In a successful session completion, SDT is calculated as an average and
is defined as the duration of a dialog from receipt of a 200 OK"

In 3.2.1 you mention that a non-100 Provisional response establishes a
session and signals a TS event. So, wouldn't a non-100 response also be
counted as the TB in this case? What happens to the time between, for
example, a 180 and a 200 message? It seems it is not counted anywhere. 

General:

I believe the notion of "Session Establishment" needs to be well defined
in this doc. Again, here you consider a 200OK as the establishing the
session but not 180. If a session is established with a=sendonly or
a=inactive, etc

In 3.2.1, 180 is consider as establishing a session.

Thanks,

REinaldo

> -----Original Message-----
> From: Jean-Francois Mule [mailto:[email-masked]] 
> Sent: Tuesday, November 07, 2006 11:20 AM
> To: Malas, Daryl
> Cc: sipping
> Subject: RE: [Sipping] Comments on 
> draft-malas-performance-metrics-05.txt
> 
> Daryl,
> 
>    It is quite a useful document, and at a minimum, it could 
> bring common terminology and methods to the long list of 
> implementations that measure some kinds of metrics like these 
> and report those via various mechanisms.
> 
>    Here are a couple of first comments, some of which we did 
> chat about yesterday.
> 
> Given the wg acknowledged that the draft should only focus on 
> the most widely supported and useable metrics, and that many 
> will come up with other specific things, it would be useful 
> to define how other metrics should be defined by vendors, operators.
> This could be a simple paragraph providing some kind of 
> template: metric name, 3 or 4 letter abbreviation, problem 
> statement or motivation, what is measured, how, in what units 
> and optionally, if it can be averaged, specify how.
> 
> --- IANA registry?
> It might be useful to have some discussions on an iana 
> registry for these. This would allow protocol mechanisms used 
> to report or correlate these metrics to refer to a well-known 
> registry and it could help avoiding folks to overload a 
> metric acronym.
> 
> < The information may also be transmitted through use of < 
> SNMP traps as described in the work in progress SIP MIB draft
> > The information may also be transmitted through the use of network 
> > management protocols like SNMP [RFC3410] and via future 
> extensions to 
> > the SIP MIB modules []
> 
> The reasons are:
>  - SNMP notifications may not be the optimal way to fetch 
> metrics in general
> - sure, they could be used if some metrics cross some 
> configured threshold to create an alarm but folks also use 
> those kinds of metrics with a statistical approach: over a 
> period of hours/days, how are my systems doing on call 
> session attempts, etc. so poll operations (or push at some 
> well defined time intervals) may be the mainstream usage of this.
> 
> Jean-Francois
> [email-masked]
> 
> 
> Use [email-masked] for questions on current 
> sip Use [email-masked] for new developments of core SIP
>