Subject: Re: [PMOL] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01

From: Al Morton <[email-masked]>
Date: Mon, 21 Jul 2008 22:07:42 -0400

Hi Daryl,

At 10:48 PM 6/25/2008, Al Morton wrote:
>This message begins a WG Last Call on the following:
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-ietf-pmol-sip-perf-metrics-01.txt

my comments as a pmol participant follow,
Al

(I have several editorial suggestions which I will pass
to Daryl at the upcoming meeting.)

The major comment is the same one I made for the ccamp
GMPLS LSP metrics draft...
>-----------------
>There is a taxonomy of possible outcomes for set-up protocols.
>Each set-up attempt will have one of the following outcomes:
>
>1. Success
>2. Failure
>3. Incorrect Set-up (wrong destination)
>
>The same basic outcomes apply to release attempts.
>-----------------
...but here it can mostly be addressed
by changing the names of a few metrics.

In the current text, the set-up time metrics cover the
Successful outcome and the Failure outcome, but attempt
to do it with a *single* metric, AFAICT.
For Registration, its RRD (Registration Request Delay, sec 4.1),
and for Sessions, its SRD (Session Request Delay, in 4.2).
I don't think that Successful Delay and Failure Delay values
should be combined in the same distribution, and the draft seems
to allow that with only one _RD for either Registration or Session.

Later, the draft introduces the notion of an
Ineffective Session Attempts (ISA) when defining another important
metric, Session Establishment Rate (sec 4.6).
IMO, the ISA should be introduced as an alternative to the Successful SRD.
There should also be an Ineffective Registration Attempt (IRA).

Here's how the outline might change to do all this:

    4. SIP Performance Metrics

       4.1. Registration Request Metrics
          4.1.1. Successful REGISTER Delay (SRD)
                         ends with 200 OK
          4.1.2. Ineffective Registration Attempt (IRA)
                         count attempts with with non-200
          4.1.3. Failed REGISTER Delay (FRD)
                         ends with non-200, (some are just Timer F?)

       4.2. Session Request Metrics
          4.2.1. Successful Session-Setup Delay (SSD)
                         ends with 200 OK
          4.1.2. Ineffective Session-Setup Attempt (ISA)
                         count attempts with with non-200
          4.2.3. Failed Session-Setup Delay (FSD)
                         ends with non-200, (some are just Timer F?)
          4.2.X. Instant Messaging
(the possibilities here are similar to the above,
I think IM may deserve its own section.)

       4.3. Session Disconnect Delay
          4.3.1. Successful Session Disconnect Delay (SDD)
                         ends with 200 OK
          4.3.2. Ineffective (Session) Disconnect Attempt (IDA)
                         count attempts with with non-200
          4.3.2. Failed-Disconnect Delay (SFD)
                         ends with non-200, (some are just Timer F?)

       4.4. Session Time (ST)
(was Session Duration Time, SDT, the terms
Duration and Time are Redundant,
and the memo uses SD below)

          4.4.1. Session Time with successful completion ST
                        from 200 to BYE
          4.4.2. Session Time with failed completion SFT
                         ends with non-BYE, (just Timer F?)

       4.5. Hops per Request (HpR)
       4.6. Session Establishment Rate (SER)
(IMO, this should be Ratio,  and 1,$s /Rate/Ratio/g )

          4.6.1. Instant Messaging

       4.7. Session Establishment Efficiency Rate (SEER)
       4.8. Session Defects (SD)
       4.9. Ineffective Session Attempts (ISA)
(text moves to 4.2.2)
       4.10. Session Disconnect Failures (SDF)
(text moves to 4.3.2)

Other comments:
---------------

Section 2, Terminology, has the following definition:
>    End-to-End - This is described as two or more elements utilized for
>    initiating a request, receiving the request, and responding to the
>    request.  It encompasses elements as necessary to be involved in a
>    session dialog between the originating user agent client (UAC),
>    destination user agent server (UAS), and any interim proxies (may
>    also include back-to-back user agent's (B2BUA's)). This may be
>    relative to a single operator's set of elements or extend to
>    encompass all elements (if beyond a single operator's network)
>    associated with a session.

There are lots of definitions for "end-to-end" in standards,
so we should try to be consistent with them, and clear about
this particular context.  I suggest:

End-to-End - This phrase defines the scope of communications as
the logical extremes, covering two or more elements utilized for
initiating a request, receiving the request, and responding to the
request.  It encompasses elements necessarily involved in a
session dialog between the originating user agent client (UAC),
destination user agent server (UAS), and any interim proxies (may
also include back-to-back user agents (B2BUA's)). End-to-End may
denote a single operator's set of elements or extend to
encompass all elements (if beyond a single operator's network)
associated with a session dialog.

-----------------------

Section 4.

For all the  relevant metrics, it strikes me that the length
of the Request (INVITE) may affect the performance, especially if the
Request message is fragmented. It seems like the Request message length
and the MTU should be parameters for all relevant metrics.

-----------------------

Section 4.7 Session Establishment Efficiency Rate (SEER)
...
           # of INVITE Requests w/ associated 200OK, 480, 486, or 600
    SEER = --------------------------------------------------------------
      (Total # of INVITE Requests)-(# of INVITE Requests w/ 'a' Response)

This ratio has all the Requests that get a legitimate response
divided by the total requests, reduced by the "abnormal" responses.

I think I could understand this metric better if the denominator
was not reduced by the "abnormal" responses. That way it would be the
ratio of the legitimate responses to the total requests, and it would
be a ratio of 1 when signaling goes as planned, despite the effects
of the terminating UAS.  The metric would have the following ratio:

           # of INVITE Requests w/ associated 200OK, 480, 486, or 600
    SEER = --------------------------------------------------------------
                          (Total # of INVITE Requests)

----------------------

I might have some more comments, but don't want to hold
this message up any longer.

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 23 Jul 2008 14:26:00 +0200

Please find below my contributor comments. 

 1. idnits complains on a number of references problems:

  -- Looks like a reference, but probably isn't: 'RFC 1157' on line
     1149
'[RFC 1157] and via future extensions to the SIP Management...'

Please add [RFC1157] as reference

  == Unused Reference: '12' is defined on line 1282, but no explicit
     reference was found in the text
     '[12]  Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP
a...'

  == Unused Reference: '13' is defined on line 1286, but no explicit
     reference was found in the text
     '[13]  Crocker, D. and Overell, P.(Editors), "Augmented BNF for
Synta...'

Please take out [12] and [13] if not needed. 

  -- Possible downref: Non-RFC (?) normative reference: ref. '7'

  -- Possible downref: Non-RFC (?) normative reference: ref. '8'

  -- Possible downref: Non-RFC (?) normative reference: ref. '9'

These three are probably OK

  ** Downref: Normative reference to an Informational RFC: RFC 2330
(ref.
     '11')

Does this need to be a normative reference? 

  -- Obsolete informational reference (is this intentional?): RFC 2234
(ref.
     '13') (Obsoleted by RFC 4234)

Is this intentional, or can 2234 be replaced by RFC 4234. Of course if
[13] will be taken out (see above) no need to answer this. 

2. Introduction - 'responsiveness of the network' - is it really only
the network, or rather 'responsiveness of the network and application
servers'? 

3. same, why is DO NOT capitalized? 

4. section 4.4 - why does SDT need to be adjusted to indicate 'minutes
and seconds' and not just seconds? 

5. section 4.8 - the definition of SD seems broken. If the output value
of this metric is numerical and should be adjusted to indicate a
percentage of defective sessions, the formula seems to be SD = (# of
Defective Sessions) / (Total # of Session Requests). Am I missing
something? 

6. For consistency reasons should not metrics define in 4.8, 4.9, 4.10
be all named as Rate? 

7. RFC1157 is historic. Use RFC3411 to refer to SNMP

8. I suggest to add RAQMON (RFC4710) with possible future extensions to
the set of methods for transferring metrics information enumerated in
6.4

9. Sections 6.5 an 6.6 seem to deal with issues that are beyond the
scope of the SIP performance metrics. I think their place is not in the
normative part of this document, but maybe in the PMOL framework
document. I suggest to take out these sections completely or move them
to an Appendix to make clear that they are non-normative. 

Dan

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Al Morton
> Sent: Thursday, June 26, 2008 5:48 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [Sipping] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01
> 
> PMOL WG,
> cc SIPPING WG,
> 
>         Title		: SIP End-to-End Performance Metrics