Subject: Re: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05

From: Benoit Claise <[email-masked]>
Date: Tue, 16 Nov 2010 10:15:35 +0100

Thanks Aamer for your review
>
> Hi Benoit,
>
> Please find my comments below. Overall I found the guidelines very 
> much on the right path – and (unsurprisingly) do mirror the internal 
> work discussions we have done in our definitions process.
>
> section 1.1
>
> The motivation does not seem to cleanly resolve whether the generation 
> of upper layer performance metrics needs to be separated from the 
> monitoring and transport frameworks (RMON, IPFIX). Unless this is 
> done, there may be overlapping efforts across the WGs. This is 
> somewhat talked about in section 6, but can we have clear guidance 
> between separation between metrics definition, transport, and 
> verification/validation. The verification/validation being something 
> perhaps we can extend in BMWGs charter.
>

In section 1.1, after this paragraph

    The IETF does have current and completed activities related to the
    reporting of application performance metrics: for example the Real-
    time Application Quality-of-Service Monitoring (RAQMON) FrameworkRFC  <http://tools.ietf.org/html/rfc4710>
    4710  <http://tools.ietf.org/html/rfc4710>  [RFC4710  <http://tools.ietf.org/html/rfc4710>], which extends the remote network monitoring (RMON)
    family of specifications to allow real-time quality-of-service (QoS)
    monitoring of various applications that run on devices such as IP
    phones, pagers, Instant Messaging clients, mobile phones, and various
    other handheld computing devices.

I would like to add:

    Another example is the IPFIX protocol [RFC5101], which exports flow
    records
    composed of IPFIX information elements, initially specified in the
    IPFIX Information Model [RFC5102].
    New IPFIX information elements can be requested to IPFIX IANA
    [IPFIX-IANA]. It is anticipated that
    some new information elements will contain performance metrics.

    In this document, the generation of upper layer performance metrics
    need
    to be separated from the monitoring and transport frameworks (such
    MIB variable
    with SNMP or information element with IPFIX).

Does it solve your issue?

Note: I'm not too sure how to translate this into a specification in 
section 5 or section 6.
>
> section 2.3
>
> "The quantitative component can be influenced by the
>
>    complete end-to-end system effects (network infrastructure).".
>
> Not clear what this means.
>
> What is the difference between application and network infrastructure? 
> Where do network transcoders sit?
>
> Is HD vs. SD video a qualitative or quantitative component?
>
If I recall correctly, Al wrote this one.
Al, can you shed some light.
>
> section 2.4
>
> "specific to an IETF-specified
>
>    protocol or specific to an application transported over an IETF-
>
>    specified protocol." then "database logging time"
>
> does this restrict the evaluation of database logging time to IETF 
> protocols (TCP, HTTP), and not any deeper?
>
Deeper is also allowed.
>
> section 3.
>
> Are active and passive measurements widely understood terms? Maybe 
> just a reference.
>
I was unable to find a good reference.
Let me propose

OLD:

    The framework in this document applies to performance metrics derived
    from both active and passive measurements.
NEW:
    The framework in this document applies to performance metrics derived
    from both active and passive measurements.
    Active measurements, also referred to as synthetic traffic, injects specific traffic
    in the network to order to measure its performance (example: one-way active measurement
    [RFC 4656] or two-way active measurements [RFC 5357], while passive measurements
    analyses the live traffic to measure its performance

> section 5.1
>
>                 "The key areas of variation between different metric 
> users..."
>
>                 Not sure if this covered by the exiting cases but 
> please add a statement regarding the performance context that is being 
> considered. For example, the network operator may find it convenient 
> to look at RTP flows as an indicator of network health rather than a 
> specific interest in the health of the RTP flow. In that regard, some 
> of the QoE aspects specific to the application may not matter at all.
>
Can you please propose some text.
Alternatively, a 5 minutes discussion would clarify.
>
> is time range part of the existing list (real-time, delayed reporting, 
> summarized reporting, etc)
>
Yes.
Proposal: add
    *   time range (real-time, delayed reporting, summarized reporting, 
etc)
>
> section 5.4.2
>
> "   Performance Metric names MUST be unique within the set of metrics
>
>    being defined and MAY be descriptive."
>
>    Unclear why a metric name would _not_ be descriptive. Perhaps an 
> example is needed?
>
I don't recall why we made that distinction.
Unless I hear otherwise, I will change it to

"   Performance Metric names MUST be unique within the set of metrics

    being defined and MUST be descriptive."
>
> section 5.4.2
>
>                 "If the measurement is specific to a measurement point 
> this SHOULD be
>
>    defined.  The measurement domain MAY also be defined.  Specifically,
>
>    if measurement points are spread across domains, the measurement
>
>    domain (intra-, inter-) is another factor to consider.
>
>    In some cases, the measurement requires multiple measurement points:
>
>    all measurement points SHOULD be defined, including the measurement
>
>    domain(s)."
>
>    It would be helpful to have references to more discussion on what 
> defines a measurement domain. Is a measurement domain an 
> administrative domain? Is the network portion of a flow leg a 
> measurement domain? What if a NAT is involved? Are there two domains then?
>
OLD:

    If the measurement is specific to a measurement point this SHOULD be
    defined.  The measurement domain MAY also be defined.  Specifically,
    if measurement points are spread across domains, the measurement
    domain (intra-, inter-) is another factor to consider.

NEW:

    If the measurement is specific to a measurement point this SHOULD be
    defined.  The measurement domain MAY also be defined.  The measurement
    domain is defined as a set of measurement point that operate under the
    the same administration. Specifically,
    if measurement points are spread across domains, the measurement
    domain (intra-, inter-) is another factor to consider.

> section 5.4.2
>
> Slightly related to measurement timing is the issue of granularity of 
> measurement needed to compute the metric. For example it is possible 
> to sample RTP flows to determine packet loss. However it is not 
> possible to sample packets from a single RTP flow to determine packet 
> loss.
>
Propose to add your sentence as a third paragraph to section 5.4.2 
"measurement timing"

Related to measurement timing is the issue of granularity of measurement 
needed to compute the metric. For example it is possible to sample RTP 
flows to determine packet loss. However it is not possible to sample 
packets from a single RTP flow to determine packet loss.

> section 6.2
>
>                 "repeatable"
>
>                 I think for application performance this might mean a 
> complete description of several workloads to be tested against. How 
> will this be done? ie are we talking about a pcap file-- if so what 
> would that pcap file need to look like? How would it be distributed?
>
Propose to add a paragraph at the end of section 6.2
Note that for the "repeatable" qualification, the performance metric 
might have to be qualified with some extra information related t othe 
environment during the measurement. For example, the workload, the 
existing flow, etc... in the measurement domain

Please let me know what you think
Thanks again for your review.

Regards, Benoit.
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf 
> Of *Benoit Claise (bclaise)
> *Sent:* Sunday, October 10, 2010 4:49 PM
> *To:* [email-masked]
> *Subject:* [PMOL] Fwd: New Version Notification 
> fordraft-ietf-pmol-metrics-framework-05
>
> Dear all,
>
> Here is a new version of the framework draft.
> What is new?
>     - improve the Performance Metric definition
>     - change the title to RFC5706
>     - PM Entity defined as a directorate
>
> No more open issues in the draft. This should be ready for WGLC.
>
> Regards, Benoit.
>
> -------- Original Message --------
>
> *Subject: *
>
> 	
>
> New Version Notification for draft-ietf-pmol-metrics-framework-05
>
> *Date: *
>
> 	
>
> Sun, 10 Oct 2010 13:38:40 -0700 (PDT)
>
> *From: *
>
> 	
>
> IETF I-D Submission Tool <[email-masked]> 
> <mailto:[email-masked]>
>
> *To: *
>
> 	
>
> [email-masked] <mailto:[email-masked]>
>
> *CC: *
>
> 	
>
> [email-masked] <mailto:[email-masked]>
>
> A new version of I-D, draft-ietf-pmol-metrics-framework-05.txt has been successfully submitted by Benoit Claise and posted to the IETF repository.
>   
> Filename:      draft-ietf-pmol-metrics-framework
> Revision:      05
> Title:         Guidelines for Considering New Performance Metric Development
> Creation_date:  2010-10-10
> WG ID:         pmol
> Number_of_pages: 21
>   
> Abstract:
> This document describes a framework and a process for developing
> performance metrics of protocols and applications transported over
> over IETF-specified protocols, and that can be used to characterize
> traffic on live networks and services.
>                                                                                    
>   
>   
> The IETF Secretariat.
>

---

From: Al Morton <[email-masked]>
Date: Wed, 17 Nov 2010 11:53:57 -0500

At 04:15 AM 11/16/2010, Benoit Claise wrote:
>>Aamer wrote:
>>What is the difference between application and network 
>>infrastructure? Where do network transcoders sit?
>>

I didn't write it, though I think I supplied the reference and text
from an existing Recommendation, E.800.

>Al, can you shed some light.

Part of the problem is that this definition wording does not
stand on its own beyond the context of the Recommendation
it comes from. Maybe we can clarify:

    NOTE 2 - QoE has two main components: quantitative and
    qualitative.  The quantitative component can be influenced by the
    complete end-to-end system effects (including user devices and
    network infrastructure).

where the "complete end-to-end system" is all communications equipment.
I'm avoiding the application/network split. As you pointed out,
transcoders would be part of network infrastructure.
HD vs. SD is quantitative in this view, a configuration option
of the e2e system.

This (QoE) has been a difficult term to define, it appears
to subsume part of the existing definition of QoS in some
people's minds, but it adds the opportunity to include context
more directly.  I don't intend to start a debate of the QoE
definition here, and if debate is wanted, it should take place
in the ITU-T Study Group responsible for this definition.

Al

---

From: Alan Clark <[email-masked]>
Date: Wed, 17 Nov 2010 14:59:01 -0500

I'd agree with the spirit of the QoE definition but the use of quantitative
vs qualitative could be stated a little differently.

QoE is a measure of the quality of an application as perceived by a person.
Externally observable behaviors of the application interact with any
psychoacoustic, psychovisual, memory and other effects and with the
conditioning and expectations of the person for the application, leading to
an expression of quality that is (or would be) stated by the person.
Observable behaviors of the application may be objectively measured or
inferred based on measurements of impairments that affect application
performance in a known way. QoE may be directly measured using subjective
testing or may be estimated through the use of appropriate perceptual
models. 

So:-
(i) if there is an (objectively measurable or quantitative) impairment, this
may be masked or enhanced by some psychoacoustic or psychovisual effect. For
example a macroblock error may be very visible in a video scene with smooth
panning but not noticeable in a video scene with high motion.
(ii) QoE is affected by the expectations of the user/ person.  SD video may
have an "excellent" QoE if the user's expectations are based on SD video and
there are no impairments other than image resolution. SD video may have a
"fair" QoE if the user's expectations are based on a mix of SD and HD video.
We find the same with narrowband vs wideband speech.  Historically
subjective tests were done with all narrowband or all wideband and hence the
results were relative to speech samples of the same type.  This leads to the
sometimes confusing result that a wideband speech sample may have lower MOS
than a narrowband sample, even though it sounds better.  There has been work
done in ITU using subjective tests with mixed narrowband and wideband
speech.

Alan

On 11/17/10 11:53 AM, "Al Morton" <[email-masked]> wrote:

> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol

---

From: Benoit Claise <[email-masked]>
Date: Mon, 22 Nov 2010 16:04:22 +0100

Al,
>    complete end-to-end system effects (including user devices and
I replaced the current NOTE 2 with the NOTE 2 above.
> in the ITU-T Study Group responsible for this definition.
Fully agree.

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Mon, 22 Nov 2010 17:06:02 +0100

Alan,

Actually, I would prefer to keep the E.800 definition as it is.
As Al suggested...

     > ...I don't intend to start a debate of the QoE

Regards, Benoit.
> On 11/17/10 11:53 AM, "Al Morton"<[email-masked]>  wrote:
>