Subject: [PMOL] Suggested updates to proposed charter wording

From: "Alan Clark" <[email-masked]>
Date: Tue, 25 Sep 2007 14:22:48 -0400

Al

A proposed rewording of the initial paragraph and some minor text cleanup

Regards

Alan

 

Performance Metrics at Other Layers WG (PMOL)

The successful implementation and operation of IP based applications often
depends on some underlying performance measurement infrastructure that helps
service operators or network managers to recognize when performance is
unsatisfactory and identify problems affecting service quality.
Standardized performance metrics add the desirable features of consistent
implementation, interpretation, and comparison.

The IETF has two Working Groups dedicated to the development of performance
metrics however each has strict limitations in their charters:

  - The Benchmarking Methodology WG has addressed a range of networking
technologies and protocols in their long history (such as IEEE 802.3, ATM,
Frame Relay, and Routing Protocols), but the charter strictly limits their
performance characterizations to the laboratory environment.

  - The IP Performance Metrics WG has the mandate to develop metrics
applicable to the performance of Internet data delivery, but it is
specifically prohibited from developing metrics that characterize traffic
(such as a VoIP stream).

The IETF also has current and completed activities related to the reporting
of application performance metrics (e.g. RAQMON and RTCP XR) and is also
actively involved in the development of reliable transport protocols which
would affect the relationship between IP performance and application
performance.

Thus there is a gap in the currently chartered coverage of IETF WGs:

development of performance metrics for IP-based applications that operate
over UDP, TCP, SCTP, DCCP, Forward Error Correction (FEC) and other robust
transport protocols, and that can be used to characterize traffic on live
networks.

The working group will focus on the completion of two RFCs:

1. A PMOL framework and guidelines memo that outlines the motivation

    for work to define performance metrics for applications transported

    over IETF-specified protocols, how that work fills a need and any gap

    in IETF-chartered work. The framework will describe the necessary

    elements of performance metric drafts and the various types of metrics

    that may be prepared in this work. The framework will also address the

    need to specify the intended audience and the motivation for the

    performance metrics. There will also be guidelines for a performance

    metric development process that includes entry criteria for

    new proposals (how a proposal might be evaluated for possible

    endorsement by a protocol development working group), and how a

    successful proposal will be developed by PMOL WG in cooperation with a

    protocol development WG.

2. A proof-of-concept RFC defining performance metrics for SIP, based on

    draft-malas-performance-metrics.  This memo would serve as an example of

    the framework and the PMOL development process in the IETF.

Discussion of new work proposals is strongly discouraged in the PMOL WG,
except to advise a protocol development WG when they are evaluating a new
work proposal for related performance metrics.

The PMOL WG will also be guided by a document describing how memos defining
performance metrics are intended to advance along the IETF Standards track
(draft-bradner-metrics-test).

Milestones

June 08  SIP Performance Metrics Draft to IESG Review 

Sept 08  PMOL Framework and Guidelines Draft to IESG Review

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

---

From: Henk Uijterwaal <[email-masked]>
Date: Wed, 26 Sep 2007 07:26:29 +0200

Al, others,

> The PMOL WG will also be guided by a document describing how memos 
> defining performance metrics are intended to advance along the IETF 
> Standards track (draft-bradner-metrics-test).

How can the PMOL group be guided by a draft that hasn't seen any work
for the last 5+ years and expired a long time ago?   I do agree
the metrics-test draft is useful, but it should be picked up by some
WG and finished before it serves as guidelines for PMOL.

Henk

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

# Lawyer: "Now sir, I'm sure you are an intelligent and honest man--"
# Witness: "Thank you. If I weren't under oath, I'd return the compliment."

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 26 Sep 2007 09:44:27 +0200

Actually Scott resubmitted his draft in June and then submitted a
revised version in August. Henk's observation is correct however to the
extent that this is still an Internet-Draft, so referring to it as
anything but as 'work-in-progress' is not appropriate. 

Dan

> -----Original Message-----
> From: Henk Uijterwaal [mailto:[email-masked]] 
> Sent: Wednesday, September 26, 2007 7:26 AM
> To: [email-masked]
> Subject: Re: [PMOL] Suggested updates to proposed charter wording
> 
> Al, others,
> 
> the metrics-test draft is useful, but it should be picked up 
> by some WG and finished before it serves as guidelines for PMOL.
> 
> Henk
> 
> 
> --

---

From: Henk Uijterwaal <[email-masked]>
Date: Wed, 26 Sep 2007 10:09:06 +0200

> anything but as 'work-in-progress' is not appropriate. 

Somebody correct me if I'm wrong but the June version was a resubmit of
an old draft, and the August version says near the end in difference
between the 02 and 03 versions: "correct author list".

Henk

---

From: Lars Eggert <[email-masked]>
Date: Wed, 26 Sep 2007 12:00:30 +0300

Hi,

a quick comment about draft-bradner-metrics-test: I had mentioned  
during the IPPM meeting that the draft is pretty complete, with the  
exception of a generally applicable definition what it would mean for  
two implementations of the same metric to generate "statistically  
equivalent" results. Help from metrics experts is surely welcome.

(This list is probably not the right place to discuss this - I  
suggest the IPPM list.)

Lars

---

From: Lars Eggert <[email-masked]>
Date: Wed, 26 Sep 2007 12:09:24 +0300

>     successful proposal will be developed by PMOL WG in cooperation  

Some of the content of this proposed WG deliverable is what we  
typically want to see already during the BOF proposal phase:

Specifically, the "motivation for work", "how that work fills a need  
and any gap in IETF-chartered work", "motivation for the performance  
metrics" seem pretty relevant already when asking for a BOF.

Likewise, information on "the various types of metrics that may be  
prepared in this work" seem important to define the scope for the  
planned WG. "Entry criteria for new proposals" is also something that  
should be in the proposed charter already.

Lars

---

From: Al Morton <[email-masked]>
Date: Wed, 26 Sep 2007 08:28:13 -0400

Henk,

Following-up here to close the loop on this,
because this also addresses the issue Lars raised.
I agree that further discussion could be
on the IPPM list...

>between the 02 and 03 versions: "correct author list".

There was a small group who discussed the draft with the authors
before the June release of 02.  One of the changes responded to
my suggestion for a test plan (where the accuracy/equivalency issue
would be dealt with up front, on a case-by-case basis),
and the new text in section 5 is partly based on Vern's reply, too:

    "...The implementation report must include a specific
    plan to test the specific metrics in the RFC in lab or real-world
    networks and reports of the tests performed with two or more
    implementations of the software.  The test plan should cover key
    parts of the specification, speak to the accuracy required for each
    aspect measured and thus define "statistically equivalent" for the
    specific metrics being tested.  Ideally, the test plan would co-
    evolve with the development of the metric, since that's when people
    have the most context in their thinking regarding the different
    subtleties that can arise."

IOW, it made more sense to me to defer this question of equivalency
to a specific metric, and allow the metricstest memo to move on
and define the process (which can be accomplished for a general case).

Lars -
This means that the open issue you identified would be addressed
in the test plan memo, and context of each metric.

There may be other differences, but this is one.

Al