Subject: Re: [manet] draft-nguyen-manet-ecds-mib and Performance Metrics (UNCLASSIFIED)

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 24 May 2013 10:26:10 +0100

Hi Benoit,
 
Thanks for cutting me in on this thread.
 
I do think this type of discussion should take place on WG mailing lists so I am
copying MANET on this reply.
 
It appears that the phrase "performance metric" causes alarm bells to ring in
several fire stations around the world. Although I strongly resent words or
phrases being captured for more specific and focused meaning than they deserve,
it is also clear that the pain caused by having a hundred firemen show up on
your doorstep at 3am is sufficient (unless you like firemen at 3am ;-) to
warrant taking Benoit's advice.
 
- Avoid the term "performance metric" unless you mean it in the
 sense of RFC 6390 (or be prepared to have a fight)
- Use the term "performance information" or maybe "performance
  counters"
 
<VBS>
 
Adrian
 
From: Benoit Claise [mailto:[email-masked]] 
Sent: 24 May 2013 09:33
To: Nguyen, James H CIV USARMY CERDEC (US)
Cc: Fred Baker; [email-masked]; me;
[email-masked]; [email-masked]; Adrian Farrel
Subject: Re: draft-nguyen-manet-ecds-mib and Performance Metrics (UNCLASSIFIED)
 
Hi James, draft-ietf-manet-smf-mib authors,

Agreed, RFC 6390 doesn't apply here.
See the email exchange regarding draft-ietf-manet-olsrv2-mib below, which I
reviewed part of the performance metric directorate.
Like Ulrich did for draft-ietf-manet-olsrv2-mib, it might better to use
"performance information" instead of "performance metrics" in your draft.
Benoit,
 
thank you very much for this review. I agree that using the term
"performance information" instead of "performance metrics" is a good
idea. We will make the change.
 
Best regards
Ulrich
 
On Mon, Apr 15, 2013 at 6:19 AM, Benoit Claise  <mailto:[email-masked]>
<[email-masked]> wrote:
Dear all,
 
I reviewed http://tools.ietf.org/html/draft-ietf-manet-olsrv2-mib-06, from a
performance metric directorate point of view.
 
This draft doesn't contain any reference to RFC6390, but contains
"performance metric". Hence this review was triggered. For details about the
directorate, see
http://www.ietf.org/iesg/directorate/performance-metrics.html

Definition of Managed Objects for the  Optimized Link State Routing
Protocol version 2
 
Abstract
 
   This document defines the Management Information Base (MIB) module
   for configuring and managing the Optimized Link State Routing
   protocol version 2 (OLSRv2).  The OLSRv2-MIB module is structured
   into state information, performance metrics, and notifications.  This
   additional state and performance information is useful to
   troubleshoot problems and performance issues of the routing protocol.
   Different levels of compliance allow implementers to use smaller
   subsets of all defined objects, allowing for this MIB module to be
   deployed on more constrained routers.

Basically, all performance metrics come from this table:
 
   o  olsrv2InterfacePerfTable - records performance counters for each
      active OLSRv2 interface on this device. selected path to each
      destination for which any such path is known.  This table has
      AUGMENTS { nhdpInterfacePerfEntry } and as such it is indexed via
      nhdpIfIndex from the NHDP-MIB.
 
NHDP-MIB is RFC 6779:
   NhdpInterfacePerfEntry ::=
      SEQUENCE {
         nhdpIfHelloMessageXmits
            Counter32,
         nhdpIfHelloMessageRecvd
            Counter32,
         nhdpIfHelloMessageXmitAccumulatedSize
            Counter64,
         nhdpIfHelloMessageRecvdAccumulatedSize
            Counter64,
         nhdpIfHelloMessageTriggeredXmits
            Counter32,
         nhdpIfHelloMessagePeriodicXmits
            Counter32,
         nhdpIfHelloMessageXmitAccumulatedSymmetricNeighborCount
            Counter32,
         nhdpIfHelloMessageXmitAccumulatedHeardNeighborCount
            Counter32,
         nhdpIfHelloMessageXmitAccumulatedLostNeighborCount
            Counter32
      }

This draft contains similar objects in olsrv2InterfacePerfTable :
 
    Olsrv2InterfacePerfEntry ::=
       SEQUENCE {
          olsrv2IfTcMessageXmits
             Counter32,
          olsrv2IfTcMessageRecvd
             Counter32,
          olsrv2IfTcMessageXmitAccumulatedSize
             Counter64,
          olsrv2IfTcMessageRecvdAccumulatedSize
             Counter64,
          olsrv2IfTcMessageTriggeredXmits
             Counter32,
          olsrv2IfTcMessagePeriodicXmits
             Counter32,
          olsrv2IfTcMessageForwardedXmits
             Counter32,
          olsrv2IfTcMessageXmitAccumulatedMPRSelectorCount
             Counter32
       }

Personally, I don't believe that those objects should be subject to the RFC
6390 template definition. (Performance Metric Definition Template, section
5.4.4, RFC 6390).
First reason: NhdpInterfacePerfEntry, from NHDP-MIB [RFC 6779] was not
subject to it
Second reason: these objects are not really performance metrics, but mainly
basic monitoring objects.
 
Since RFC 6779 uses the term performance information (in the abstract), I
would propose that draft-ietf-manet-olsrv2-mib also uses this term, and not
the "performance metric". That would avoid some confusion. However, keeping
the olsrv2InterfacePerfTable OID name is perfectly fine, for consistency
reason with RFC 6779.
 
Regards, Benoit

Regards, Benoit
Classification: UNCLASSIFIED
Caveats: NONE
 
Fred,
 
There are two counters that are defined for performance metrics.  Please see
below.  As we stated it in Section 9 "Applicability Statement," ECDS-MIB is
an extension of SMF-MIB.  Thus, ECDS-MIB's inherited SMF-MIB's performance
metrics.  
 
As I can see, ECDS-MIB does apply 6390's (i), (ii), (iii), and somewhat
(iv).  Please let me know if you have any suggestion to improve the draft.

      (i) the degree to which its absence would cause significant loss
      of information on the behavior or performance of the application
      or system being measured
 
      (ii) the correlation between the Performance Metric, the QoS, and
      the QoE delivered to the user (person or other application)
 
      (iii) the degree to which the Performance Metric is able to
      support the identification and location of problems affecting
      service quality
 
      (iv) the requirement to develop policies (Service Level Agreement,
      and potentially Service Level Contract) based on the Performance
      Metric

--
 -- E-CDS Performance Group
 --
 
 ecdsPerformanceGroup OBJECT IDENTIFIER ::= { ecdsMIBObjects 3 }
 
 ecdsInEcdsChange OBJECT-TYPE
         SYNTAX          Counter32
         MAX-ACCESS      read-only
         STATUS          current
         DESCRIPTION
                 "This object indicates how many times the current
                  node is configured to be in E-CDS."
 ::= { ecdsPerformanceGroup 1 }
 
 ecdsCurrentRtrPriValueChange OBJECT-TYPE
         SYNTAX          Counter32
         MAX-ACCESS      read-only
         STATUS          current
         DESCRIPTION
                 "This object indicates how many times the Router
                  Priority of the current node has been changed."
 ::= { ecdsPerformanceGroup 2 }

James Nguyen
US Army CERDEC S&TCD
Email: [email-masked]
Phone:  443-395-5628

-----Original Message-----
From: Fred Baker [mailto:[email-masked]] 
Sent: Thursday, May 23, 2013 11:12 AM
To: [email-masked]
Cc: [email-masked]
Subject: draft-nguyen-manet-ecds-mib and Performance Metrics
 
Hi:
 
I have a question for you. Your document mentions performance metrics.
Would you kindly take a look at RFC 6390 to see if any of its considerations
apply to it?  "No" is an acceptable response, of course; the point is to ask
the question.
 
6390 Guidelines for Considering New Performance Metric Development. A.
     Clark, B. Claise. October 2011. (Format: TXT=49930 bytes) (Also
     BCP0170) (Status: BEST CURRENT PRACTICE)

Classification: UNCLASSIFIED
Caveats: NONE

---

From: Benoit Claise <[email-masked]>
Date: Fri, 24 May 2013 12:02:34 +0200

Hi Adrian,

> I do think this type of discussion should take place on WG mailing 
> lists so I am copying MANET on this reply.
>
> It appears that the phrase "performance metric" causes alarm bells to 
> ring in several fire stations around the world. Although I strongly 
> resent words or phrases being captured for more specific and focused 
> meaning than they deserve, it is also clear that the pain caused by 
> having a hundred firemen show up on your doorstep at 3am is sufficient 
> (unless you like firemen at 3am ;-) to warrant taking Benoit's advice.
>
This is simply not correct!
Based on my two messages below (for the 2 drafts):

    the "performance metric". That would avoid some confusion.

    ...

    Like
    Ulrich did for draft-ietf-manet-olsrv2-mib, it might better to use
    "performance information" instead of "performance metrics" in your
    draft.

These are advice, and should be considered as such.

Regards, Benoit

> *From:*Benoit Claise [mailto:[email-masked]]
> *Sent:* 24 May 2013 09:33
> *To:* Nguyen, James H CIV USARMY CERDEC (US)
> *Cc:* Fred Baker; [email-masked]; me; 
> *Subject:* Re: draft-nguyen-manet-ecds-mib and Performance Metrics 
> (UNCLASSIFIED)
>
> See the email exchange regarding draft-ietf-manet-olsrv2-mib below, 
> which I reviewed part of the performance metric directorate.
> Like Ulrich did for draft-ietf-manet-olsrv2-mib, it might better to 
> use "performance information" instead of "performance metrics" in your 
> draft.
>
> Benoit,
>   
> On Mon, Apr 15, 2013 at 6:19 AM, Benoit Claise<[email-masked]>  <mailto:[email-masked]>  wrote:
> Dear all,
>   
> I reviewedhttp://tools.ietf.org/html/draft-ietf-manet-olsrv2-mib-06, from a
>     Email:[email-masked]  <mailto:[email-masked]>
>
>     Phone:   443-395-5628
>
>       
>
>       
>
>     To:[email-masked]  <mailto:[email-masked]>
>
>     Cc:[email-masked]  <mailto:[email-masked]>
>

---

From: "Cole, Robert G CIV USARMY CERDEC (US)" <[email-masked]>
Date: Fri, 24 May 2013 12:54:51 +0000

I agree that we should be more precise in our naming and descriptions.  This
question is showing up often in our MIB discussions.  I have made it a habit
of placing these counters within a PerformanceGroup instead of a
StatisticsGroup because I felt that Statistics implied we were collecting
statistical data (e.g., averages) instead of performance data (counts).  But
I agree that using the term Performance Metric goes too far and is not a
correct description of the data.  Going forward I'll try to be more
consistent and use the term 'Performance Information' as you all suggest and
as we did in RFC 6779.

Thanks,
Bob

Robert G. Cole
Comm:  443.395.8744
Email: [email-masked]

-----Original Message-----
From: Benoit Claise [mailto:[email-masked]] 
Sent: Friday, May 24, 2013 6:03 AM
To: [email-masked]
Cc: Nguyen, James H CIV USARMY CERDEC (US); 'Fred Baker';
[email-masked];
[email-masked]; [email-masked]; [email-masked]
Subject: Re: draft-nguyen-manet-ecds-mib and Performance Metrics
(UNCLASSIFIED)

Hi Adrian,

	Hi Benoit,

	It appears that the phrase "performance metric" causes alarm bells
to ring in several fire stations around the world. Although I strongly
resent words or phrases being captured for more specific and focused meaning
than they deserve, it is also clear that the pain caused by having a hundred
firemen show up on your doorstep at 3am is sufficient (unless you like
firemen at 3am ;-) to warrant taking Benoit's advice.

	Since RFC 6779 uses the term performance information (in the
abstract), I
	would propose that draft-ietf-manet-olsrv2-mib also uses this term,
and not
	the "performance metric". That would avoid some confusion.
	
	...
	
	Like 

Regards, Benoit

	  counters"

	<VBS>

	Adrian

	From: Benoit Claise [mailto:[email-masked]
<blockedmailto:[email-masked]> ] 
	Sent: 24 May 2013 09:33
	To: Nguyen, James H CIV USARMY CERDEC (US)
	Cc: Fred Baker; [email-masked]; me;
[email-masked]; [email-masked]; Adrian Farrel
	Subject: Re: draft-nguyen-manet-ecds-mib and Performance Metrics
(UNCLASSIFIED)

	Benoit,

	Best regards
	Ulrich
	 
	On Mon, Apr 15, 2013 at 6:19 AM, Benoit Claise <[email-masked]>
<blockedmailto:[email-masked]>  wrote:
	Dear all,
	 
	I reviewed http://tools.ietf.org/html/draft-ietf-manet-olsrv2-mib-06
<blockedhttp://tools.ietf.org/html/draft-ietf-manet-olsrv2-mib-06> , from a
	performance metric directorate point of view.
	 
	This draft doesn't contain any reference to RFC6390, but contains
	"performance metric". Hence this review was triggered. For details
about the
	directorate, see
	http://www.ietf.org/iesg/directorate/performance-metrics.html
<blockedhttp://www.ietf.org/iesg/directorate/performance-metrics.html> 

	Abstract
	 
	   This document defines the Management Information Base (MIB)
module
	   for configuring and managing the Optimized Link State Routing
	   protocol version 2 (OLSRv2).  The OLSRv2-MIB module is structured
	   into state information, performance metrics, and notifications.
This
	   additional state and performance information is useful to
	   troubleshoot problems and performance issues of the routing
protocol.
	   Different levels of compliance allow implementers to use smaller
	   subsets of all defined objects, allowing for this MIB module to
be
	   deployed on more constrained routers.

	   o  olsrv2InterfacePerfTable - records performance counters for
each
	      active OLSRv2 interface on this device. selected path to each
	      destination for which any such path is known.  This table has
	      AUGMENTS { nhdpInterfacePerfEntry } and as such it is indexed
via
	      nhdpIfIndex from the NHDP-MIB.
	 
	NHDP-MIB is RFC 6779:

	Personally, I don't believe that those objects should be subject to
the RFC
	6390 template definition. (Performance Metric Definition Template,
section
	5.4.4, RFC 6390).
	First reason: NhdpInterfacePerfEntry, from NHDP-MIB [RFC 6779] was
not
	subject to it
	Second reason: these objects are not really performance metrics, but
mainly
	basic monitoring objects.
	 
	Since RFC 6779 uses the term performance information (in the
abstract), I
	would propose that draft-ietf-manet-olsrv2-mib also uses this term,
and not
	the "performance metric". That would avoid some confusion. However,
keeping
	the olsrv2InterfacePerfTable OID name is perfectly fine, for
consistency
	reason with RFC 6779.
	 
	Regards, Benoit

	Regards, Benoit

		Fred,
		 
		There are two counters that are defined for performance
metrics.  Please see
		below.  As we stated it in Section 9 "Applicability
Statement," ECDS-MIB is
		an extension of SMF-MIB.  Thus, ECDS-MIB's inherited
SMF-MIB's performance
		metrics.  
		 
		As I can see, ECDS-MIB does apply 6390's (i), (ii), (iii),
and somewhat
		(iv).  Please let me know if you have any suggestion to
improve the draft.

		      (i) the degree to which its absence would cause
significant loss
		      of information on the behavior or performance of the
application
		      or system being measured
		 
		      (ii) the correlation between the Performance Metric,
the QoS, and
		      the QoE delivered to the user (person or other
application)
		 
		      (iii) the degree to which the Performance Metric is
able to
		      support the identification and location of problems
affecting
		      service quality
		 
		      (iv) the requirement to develop policies (Service
Level Agreement,
		      and potentially Service Level Contract) based on the
Performance
		      Metric

		 
		 ecdsPerformanceGroup OBJECT IDENTIFIER ::= { ecdsMIBObjects
3 }
		 
		 ecdsInEcdsChange OBJECT-TYPE
		         SYNTAX          Counter32
		         MAX-ACCESS      read-only
		         STATUS          current
		         DESCRIPTION
		                 "This object indicates how many times the
current
		                  node is configured to be in E-CDS."
		 ::= { ecdsPerformanceGroup 1 }
		 
		 ecdsCurrentRtrPriValueChange OBJECT-TYPE
		         SYNTAX          Counter32
		         MAX-ACCESS      read-only
		         STATUS          current
		         DESCRIPTION
		                 "This object indicates how many times the
Router
		                  Priority of the current node has been
changed."
		 ::= { ecdsPerformanceGroup 2 }

		James Nguyen
		US Army CERDEC S&TCD

		-----Original Message-----
		From: Fred Baker [mailto:[email-masked]
<blockedmailto:[email-masked]> ] 
		Sent: Thursday, May 23, 2013 11:12 AM
		To: [email-masked]
		Cc: [email-masked]
		Subject: draft-nguyen-manet-ecds-mib and Performance Metrics
		 
		Hi:
		 
		I have a question for you. Your document mentions
performance metrics.
		Would you kindly take a look at RFC 6390 to see if any of
its considerations
		apply to it?  "No" is an acceptable response, of course; the
point is to ask
		the question.
		 
		6390 Guidelines for Considering New Performance Metric
Development. A.
		     Clark, B. Claise. October 2011. (Format: TXT=49930
bytes) (Also
		     BCP0170) (Status: BEST CURRENT PRACTICE)