Subject: RE: [Sipping] WGLC on RTCP Summary Draft

From: "Jean-Francois Mule" <[email-masked]>
Date: Fri, 11 Mar 2005 09:41:36 -0700

Hi,

   As you know, I support this draft on the SIP "RTCP Summary".
   Here are some comments for your consideration. 
   I have split them into 3 categories, general, editorial and
technical.

Thanks,
Jean-Francois.

--- Draft reviewed: draft-johnston-sipping-rtcp-summary-06.txt
--- part 1

--- 0. General comments
+ The name of the I-D is not reflecting the technical content
The name of the draft is: "SIP Service Quality Reporting Event". I
understand that they were some previous comments about the proper
naming of this draft but based on the I-D content (RTCP XR
VoIP metric block is the primary if not sole focus of section
 7), I don't understand how this draft is about "SIP service Quality
Reporting". I prefer the original name, or something closer to
the original name:
     Extended RTCP Metric Reporting via SIP
The notion of "SIP Service Quality" in the current name is way too
broad for the scope of a draft that starts with this first sentence:
"This document defines a new performance report SIP event package and
a new MIME type for VoIP performance metrics."
                    ^^^^^^^^^^^^^^^^^^^^^^^^
then further down in section 1, it also says:
  "The application/svcqual MIME type
   defined in this document is based on the information contained in a
   RTCP-XR VOIP block and is highly optimized and useful for voice
   session quality. "

I would argue that this draft has nothing to do with the "SIP Service
Quality" (unless the "SIP service" has some VoIP dsp/codec for which the
VoIP metric block can be reported). If it is about VoIP and the VoIP
metric block defined in rfc3611.

+ Terminology
You make use of various terms and you seem to use them like
synonymous. For the management folks, those have sometimes quite
 different meanings. For example, just in the title and introduction
paragraph, you use "service quality reporting event", "performance
reports", "VoIP performance metrics", "session quality", "quality
reports" without any clear distinction.
You may want to consider a smaller set of those terms and define them
in a terminology section where you could restrict their meaning scope
to what you mean with RTCP XR.
In general, I would recommend staying away from "performance" reports
without specific qualifier. The main reason is the metrics reported
here may have nothing to do with the actual device or DSP
"performance" per se, but more the combination of
device implementation of voice quality managemenent, network
conditions that can affect voice quality, etc. rfc 3611 does not
contain once the term performance but use specifically "voice quality
metrics".

+ Missing text on RFC 3611
Having some text that explains that this SIP event package allows VoIP
applications that support RFC 3611 to convey some of the metric block
data to SIP entities would be welcome.

+ no normative references?
this seems strange

+ FYI - similar effort with MGCP:
http://www.ietf.org/internet-drafts/draft-auerbach-mgcp-rtcpxr-00.txt
The above draft for MGCP is very rough and has all kinds of IDnits
that should be fixed but the technical content is closely related.
I would strongly encourage the authors of both to review each others'
draft. There is some aBNF grammar in both and while the use is
somewhat  different due to the protocol models, but modulo the
protocol  differences, having some "parity" of what can be reported in
either models may be important for some network elements that may want
to have one MGCP leg towards the MGCP endpoints, and one SIP leg...
and collect rtcp-xr metrics from the mgcp endpoint and pass them on to
a sip element...

for e.g., is it really necessary to have:
# in the sipping draft:
Quality  = ("QualityScores") EQ r-lq SP r-cg SP r-cq-ext SP m-lq SP m-cq
           r-lq     = ("r-lq") COLON 1*3DIGIT / notAvail
           r-cq     = ("r-cq") COLON 1*3DIGIT / notAvail
           r-cq-ext =  ("r-cq-ext") COLON 1*3DIGIT / notAvail
           m-lq     = ("mos-lq") COLON DIGIT ["." 1*2DIGIT] / notAvail
           m-cq     = ("mos-cq") COLON DIGIT ["." 1*2DIGIT] / notAvail

# in the mgcp draft:
                           / RFactor
                           / ExternalRFactor
                           / EstimatedMOS-LQ
                           / EstimatedMOS-CQ

I am not saying the grammar should be equivalent but the grammar
constructs (m-lq, EstimatedMOS-LQ, etc) could somewhat be closer
or aligned in some ways...

--- 1. Editorial Comments
+ p3: consider changing "SNMP" to "traditional element management
+     protocols like SNMP"
Based on your section 4, it does not matter whether it is SNMP, or
telnet or netconf. More on section 4 later.

+ p3: "performance quality collector"
 -> define what this is? (may be for terminology section). Are you
trying to define "roles" in the uml sense here?
 Apart from its first occurence (and the use of collector in the user
part of the URI in the message flow examples), it is not used once
more in the text. It would be nice to correlate it with other
paragraphs.

+p3: 
text says:
< "REQ-1: An authorized third party should be able to receive selected
<   performance reports on a near real time basis"
Is the "Authorized third party" the performance quality collector
talked about in the preceding paragraph?

<  REQ-2: The VoIP application should not have to store large amounts of
<  information.
Now you use "VoIP application" which is also vague. If the VoIP
application is something like {sip stack, sip app logic, RTP media
stack and some vocoder} - which is my interpretation, then it does not
store large amount of VoIP quality metrics - it is the element
management agent on that VoIP box that does some of that (SNMP agent
using in memory tables, etc.).

+ section 4
< The focus of SIP is applications and the performance management for
< those applications cannot rely on SNMP.
I agree with the point, not the wording. This text is what triggered
my comment on adding specific requirements that clearly justify the
event package you are defining.
Consider adding one or 2 requirements mentioned above and changing
this sentence to say something like "given REQ-x, SNMP is not an
optimal solution...."

< However, the information available in
< RTCP summary reports is of less interest to the management of the UA
< and more of interest to the VoIP service provider
arguable. For e.g., the entity that manages the *SIP UA* is often the
entity that also buys the phones which have DSP implementations and
RTCP - wouldn't this entity be interested in the quality of the device
it buys w.r.t. voice quality?
Also, couldn't the end user be interested in some data in those
reports? ["the call quality for your last VoIP call was better than
the average call quality on the traditional PSTN"]

+ section 5
< In this approach, a new SIP events package [6] is defined.  The
< intended methods to use for this event are PUBLISH [8] and
< SUBSCRIBE/NOTIFY.
                 ^ first citation, reference missing

< A VoIP application will send performance data
                               ^^^^^^^^^^^
prefer the use of the rfc3611 terminology: voice quality metrics
< using either of these methods to an entity which will make the
< information available to other applications.

--- 2. Technical Comments
+ p3: Requirements
  How are your requirements in section 2 met by your proposal.
Shouldn't the document validate them explicitely, somewhere in the
text?

+ p3:
<  REQ-4: The performance report information must be able to be
<  transferred securely.
   ^^^^^^^^^^^
Does that mean this is a requirement on the *transport* of the report
over SIP?

<  transferred securely.
               ^^^^^^^^
can you be more specific

+ p3: 
<  REQ-6: The reports will include or be associated with dialog
<  identifiers for correlation purposes.
I assume you meet this req with DialogID/CallID in the grammar, correct?
More text would be welcome in the draft for multi-party sessions (one
call-id, multiple dialogs with different parties when media mixing
happens at the SIP UA for e.g.). In some of these cases, you
have multiple dialog identifiers and also potentially
multiple RTCP XR remote report (i think). If this is true, given the
syntax you have in section 7, how does this work for the implementers?
If the scope of the ID is to make this work in the case of a 2 party
VoIP call, state it in some assumptions section/text. If not, please
elaborate on how this works so that implementers on both sides of the
subscribe/notify dialog can interoperate (and accurately correlate the
reports).

+p3: what technical requirements can't be met by using SNMP?
+p3: motivation section
I believe the list of requirements can all be met using SNMP (with an
appropriate SNMPv3 security model and the proper MIB object modeling).
That said, they are some significant technical requirements that this
solution is meeting that would be hard to meet with just SNMP. Those
do not seem to be documented:
  - have the ability to correlate the VoIP metric block information
    with the SIP signaling, for e.g. media codec changes via SIP UPDATEs
    (this facilitates the correlation of SIP dialog msg exchange with
     the media level session)
  - for AAA (accounting) purposes, this solution seems much more
    elegant: given some SIP network architectures, this allows SIP
    entities responsible for AAA on the SIP path to get the VoIP
    metric block and attach with some sort of CDR
  -  given that the call state is distributed and, more likely to be
     known by the SIP UA, having the ability to "push" VoIP reports
     when SIP transactions occur (INVITE, ACK, UPDATE, BYE) is
     something that would be hard to do with things like SNMP where
     the polling model from the voice metric collecting function (the
     use of snmp notifications for that would be ugly).
i'm sure there are better "motivations" than the ones above that could
go in the section 3 .

+ section 4 - why not SNMP?
Again, i think this ID is useful but SNMP is not the single "mgmt"
protocol talked about these days in IETF.
< Since this type of QoS monitoring seems related to management, SNMP
< could possibly be used to collect this type of data as well.
Should this section look at other protocols? i personally don't think
it is necessary as some of the requirements added above should explain
why the "binding" of the VoIP metric block with a SIP level dialog
is useful for some.

My second comment on section 4 is that some justifications for "not
SNMP" are plain wrong.
< SNMP is however primarily used for management of network devices as
< they relate to the infrastructure but is not typically used for
< management of applications on that infrastructure.
What does this mean when for the same co-authors of this ID develop an
SNMP MIB module in the AVT wg for RTCP-XR?
What about the network services mib (rfc2788) designed for monitoring
network applications?
I could go on but the point is this argument is just not right (and
I'm far from saying SNMP is good either for what you want to do).

+ p11:
< R -  Listening Quality
< Estimated listening call quality expressed in a score from
< 0 - 100, per ITU-T standard G.107.  

Are you forcing (MUST) this metric to be calculated per G.107?
I would really like to see more extensibility here, that is:
   - leave it more open here with loose language so that if someone
     comes up with a improved way of computing the R factor with
     another algorithm (refer to the work on the extended E-model in
     ITU), you don't have to change this SIP event package
   - or, you add another parameter that indicate what computational
     model was used to calculate the R factor (my preference).
     Note that a similar comment was made to Alan Clark and Amy in
IETF#61
     in the AVT group, in the context of the RTCP-XR MIB, see:
     http://www1.ietf.org/mail-archive/web/avt/current/msg04734.html
     (that is my preference).
I don't think the SIP event package should be updated in the future if
G.107 is extended.

--- end of part 1
that's all i could review for now, i may have more comments but it is
unclear when I will be able to send them to the authors or the list.

> > -----Original Message-----
> > From: Dean Willis [mailto:[email-masked]]
> > Sent: Monday, February 28, 2005 1:51 PM
> > To: 'sipping' WG
> > Cc: Alan Johnston; Rohan Mahy; Gonzalo Camarillo; Amy Pendleton
> > Subject: [Sipping] WGLC on RTCP Summary Draft
> >
> >
> > This should have happened about two months ago. I'm sorry.
> >
> > Your friendly but only marginally performant working group
> > chairs were
> > just reminded of the fact that the working group agreed in DC
> > to adopt
> > draft-johnston-sipping-rtcp-summary-06.txt as a working group
> > document
> > and proceed to WGLC. I believe we agreed that this would be an
> > informational track document.
> >
> > So, I'd like to announce a working group last call, starting
> > immediately, and closing March 16, 2005.
> >
> > The draft under discussion is available at:
> >
> http://www.ietf.org/internet-drafts/draft-johnston-sipping-rtcp-
> summary-06.txt
> 
> 
> As usual, please copy your comments to both the list and the authors.
> 
> And if we've forgotten anything else that you have the good fortune to
> be able to remember, please have pity and remind us gently.
> 
> --
> Dean
> 
> 
> _______________________________________________
> Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
> This list is for NEW development of the application of SIP
> Use [email-masked] for questions on current sip Use
> [email-masked] for new developments of core SIP

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Romascanu, Dan \(Dan\)" <[email-masked]>
Date: Tue, 15 Mar 2005 00:34:07 +0200

> Your friendly but only marginally performant working group chairs were  
> just reminded of the fact that the working group agreed in DC to adopt  
> draft-johnston-sipping-rtcp-summary-06.txt as a working group document  

Jean-Francois made a set of comments on this draft that I almost integrally agree with and will try not to repeat. Here are a few more.

1. Even if the document is informative, its technical content is strongly dependent on RTCP-XR, so I believe that RFC 3611 is a normative reference. 
2. The scope section should mention that the focus of the initial package as of the RTCP-XR bucket it is based upon is media monitoring specific. 
3. There is a REQ-5 missing in Section 2
4. Section 4 ('Why SNMP is not appropriate) should be taken out. It's simply wrong technically and politically :-) On the technical side it's wrong because as Jean Francois pointed, all requirements in Section 2 can actually be met by SNMPv3 with the proper security model in place. On the 'political' side - we need to work towards a comprehensive framework for VoIP performance monitoring, and there is room in this framework for SNMP in some management applications as RTCP-XR shows, there is a need for svcqual as this draft demonstrates. 
5. If you want an extra argument in favor of svcqual you should add scalability (maybe in the requirements list). By avoiding a centralized pull model, the SIP PUBLISH method allows for a broader scale deployment. 
6. Why do we need a re-definition of most if not all metrics in Section 7.7, rather than referring RFC 3611, Section 4.7?'  

Regards,

Dan

---

From: "Romascanu, Dan \(Dan\)" <[email-masked]>
Date: Thu, 17 Mar 2005 19:43:12 +0200

Amy,
 
I did not ask in my comments restricting the RTCP XR package for voice only. I am OK with the approach that you are taking, but I believe that it is not documented well enough, since my comments 1, 2, and to a certain extent 6. 
 
As the comments I submitted are WGLC comments submitted within the WGLC period, I believe that you should answer them one by one, and provide a suggestion for resolution before stepping ahead in editing a revised version of the Internet-Draft. This especially as you suggest to skip another WG draft, which implies consensus in resolving all WGLC comments. 

Regards,

Dan

-----Original Message-----
From: Amy Pendleton [mailto:[email-masked]]
Sent: 17 March, 2005 7:25 PM
To: Romascanu, Dan (Dan); [email-masked]; Alan Johnston; [email-masked]; [email-masked]; [email-masked]; Dean Willis; Rohan Mahy; Gonzalo Camarillo
Subject: RE: [Sipping] WGLC on RTCP Summary Draft

There were are few comments sent to me as well that I believe will need to be addressed for the draft.  These comments include:
 
- need to add an quality algorithm type parameter to align with related standards
- need to permit token extensions to the basic event definition
- need to clean up some minor issues with ABNF definitions
 
I recommend another draft cleanup but I don't believe it should require a new last call but I am looking for the advice from the WG chairs.
 
I do want to highlight, to respond to Jean Francois and Dan's comments about the generalization of the service reporting package versus restricting to RTCP XR for voice only.  This was discussed on the list previously and the conclusion was that it was not a good idea to restrict the package to voice only--it will force us to later create an entirely new package, or packages, for other media types.  The modification of the codec parameters definition to use SDP lines was meant to address that problem.  I don't see a reason to change the event definition back to "rtcp xr" just for the purpose of aligning with related efforts.  We are simply creating more work for the future.
 
Thanks,
Amy

-----Original Message-----
From: Romascanu, Dan (Dan) [mailto:[email-masked]] 
Sent: Monday, March 14, 2005 2:34 PM

Regards,

Dan

---

From: "Romascanu, Dan \(Dan\)" <[email-masked]>
Date: Thu, 24 Mar 2005 11:04:13 +0200

Dean,

Sorry for the editing mismatch. Dropping 'has' or adopting your proposed edit are both fine. 

Regards,

Dan

> Sent: 23 March, 2005 11:07 PM
> To: Amy Pendleton
> Cc: SIPPING WG; Romascanu, Dan (Dan)
> Subject: Re: [Sipping] WGLC on RTCP Summary Draft
> 
> 
> 
> On Mar 21, 2005, at 1:13 PM, Romascanu, Dan ((Dan)) wrote:
> > OLD:
> >
> > As VoIP is currently has the most immediate need  for the real time 
> > quality reports available with this package, it has
> >    been defined first.  However, the event package is 
> designed to be   
> > extensible -
> >  NEW:
> >  
> > As VoIP media monitoring  is currently has the most immediate 
> > need  for the real time quality reports available with this 
> package, 
> > it has been defined first.  However, the event package is 
> designed to 
> > be   extensible
> >
> 
> I'm not exactly sure what "is currently has" means in the preceding. 
> Perhaps there's an extra word in there somewhere.
> 
> Perhaps we are trying to say something like:
> 
> "Monitoring of the delivery quality for media associated with voice 
> traffic is believed to be the highest priority usage for this 
> mechanism. Therefore, this document defines only those elements of an 
> event package needed for voice traffic quality monitoring. 
> However, the 
> event package defined herein is designed to be extensible" . . .
> 
> Or we could just drop either the "is" or the "has".
> 
> --
> Dean
> 
>