Subject: [PMOL] Draft Charter for discussion

From: Al Morton <[email-masked]>
Date: Fri, 21 Sep 2007 09:23:49 -0400

Here's the initial draft of the charter.
Comments welcome, of course.

Al

Proposed Charter (0.0)

Performance Metrics at Other Layers WG (PMOL)

There are often uncertainties about the performance and
suitability of new technologies and applications for their intended
audience, and the Internet is no exception. Most uncertainties are
effectively addressed through quantified assessment of key performance
indicators.  Standardized performance metrics add the desirable features
of consistent implementation, interpretation, and comparison.

Although the IETF has two Working Groups dedicated to the development
of performance metrics, they each have strict limitations in their
charters:

  - The Benchmarking Methodology WG has addressed a range of networking
technologies and protocols in their long history (such as IEEE 802.3,
ATM, Frame Relay, and Routing Protocols), but the charter strictly
limits their performance characterizations to the laboratory environment.

  - The IP Performance Metrics WG has the mandate to develop metrics
applicable to live IP networks, but it is specifically prohibited from
developing metrics that characterize traffic (such as a VoIP stream).

The IETF also has current and completed activities related to the
reporting of application performance metrics (e.g. RAQMON) and is
also actively involved in the development of reliable transport
protocols which would affect the relationship between IP performance
and application performance.

Thus there is a gap in the currently chartered coverage of IETF WGs:
development of performance metrics for IP-based applications that
operate over UDP, TCP, SCTP, DCCP, Forward Error Correction (FEC)
and other robust transport protocols, and that can be used to
characterize traffic on live networks.

The working group will focus on the completion of two RFCs:

1. A PMOL framework and guidelines memo that includes the motivation
    of work to define performance metrics for applications transported
    on IETF-specified protocols, and how that work fills a need and a gap
    in IETF-chartered work. The framework will describe the necessary
    elements of performance metric drafts and the various types of metrics
    that may be prepared in this work. The framework will also address the
    need to specify the intended audience and the motivation for the
    performance metrics. There will also be guidelines for a performance
    metric development process that includes entry criteria for
    new proposals (how a proposal might be evaluated for possible
    endorsement by a protocol development working group), and how a
    successful proposal will be developed by PMOL WG in cooperation with a
    protocol development WG.

2. A proof-of-concept RFC defining performance metrics for SIP, based on
    draft-malas-performance-metrics.  This memo would serve as an example of
    the framework and the PMOL development process in the IETF.

Discussion of new work proposals is strongly discouraged in the PMOL
WG, except to advise a protocol development WG when they are evaluating
a new work proposal for related performance metrics.

The PMOL WG will also be guided by a document describing how memos
defining performance metrics are intended to advance along the IETF
Standards track (draft-bradner-metricstest).

Milestones
June 08  SIP Performance Metrics Draft to AD Review
Sept 08  PMOL Framework and Guidelines Draft to AD Review

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

---

From: Al Morton <[email-masked]>
Date: Fri, 21 Sep 2007 10:01:45 -0400

>Comments welcome, of course.

a post-press-send comment:
the charter needs a paragraph at the end on coordinating with
other standards development orgs. Something like

PMOL WG will take advantage of expertise and seek to avoid overlap with
other standards development organizations, such as ETSI STQ, ITU-T SG 12,
ATIS IIF, ATIS PRQC, and ...

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Fri, 21 Sep 2007 16:25:57 +0200

Hi Al,

Is the WG pmol created in the transport area?

To take in account the last IPPM meeting output, I will remove the passive metrics stuffs from http://tools.ietf.org/html/draft-ietf-ippm-multimetrics-04.  These performance metrics are performed on real applications traffic transported by IETF-specified protocols. Are they an emanation of PSAMP WG or of the (IETF-specified) protocol actually observed?

Regards
Emile

> -----Message d'origine-----
> De : Al Morton [mailto:[email-masked]]
> Envoyé : vendredi 21 septembre 2007 15:24
> À : [email-masked]
> Objet : [PMOL] Draft Charter for discussion
>

---

From: Al Morton <[email-masked]>
Date: Fri, 21 Sep 2007 10:37:34 -0400

Emile,  (dropping ippm and psamp lists from cc)

>Is the WG pmol created in the transport area?

The proposal is sponsored by the Ops and Mgmt Area.

>To take in account the last IPPM meeting output, I will remove the 
>passive metrics stuffs from 
>http://tools.ietf.org/html/draft-ietf-ippm-multimetrics-04.  These 
>performance metrics are performed on real applications traffic 
>transported by IETF-specified protocols. Are they an emanation of 
>PSAMP WG or of the (IETF-specified) protocol actually observed?

I suppose it could be either, and depends on how the proposed
metrics are constructed.

Al

---

From: Daryl Malas <[email-masked]>
Date: Fri, 21 Sep 2007 11:05:41 -0600

In addition to the list below, I think it should be noted the PMOL WG
will also take advantage of expertise in the relative IETF working
groups.

---

From: Al Morton <[email-masked]>
Date: Fri, 21 Sep 2007 13:51:42 -0400

>will also take advantage of expertise in the relative IETF working

I appreciate your point, and I think the emphasis on
cross-area review in the publication process covers
it to some degree (IOW, it's effectively in every charter).

Al

---

From: "Loki Jorgenson" <[email-masked]>
Date: Fri, 21 Sep 2007 13:49:32 -0700

Al - while I can appreciate the careful intention in charter wording, I
would point out that the relevance to "other protocols" to any specific
or general application is not addressed.

You make reference to "VoIP traffic" as an example of an application
case.  Without naming a vendor, a codec, or a context, I note that there
are possible various high-level characterizations about VoIP as an
application - e.g. the usual about loss and jitter sensitivity (and
conversely relative insensitivity to available capacity).  Similar
characterizations are possible for other broad categories of application
(e.g. data-intensive applications are very bandwidth and latency
sensitive - less sensitive to jitter).

Roughly following the ITU's lead (re: Y-1541), five major categories of
application can be identified:
      o Real-time (two sub-categories)
         o VoIP
         o Video
      o Transactional (e.g. banking)
      o Data (e.g. backup/recover)
      o Best Effort (e.g. email, Web)
A protocol like UDP could conceivably be associated with any of the
above (although not typical for all categories).  To address the
"performance of UDP" would seem less meaningful than the "performance of
<application category>".  Of course, the "performance of RTP" as a more
specific example narrows the implied application set.  In some cases,
the protocol is one-to-one with a particular application (e.g. NTP).  In
any case, simply addressing protocol overlooks some of the implications
of the type of application utilizing the protocol.

What would seem missing from the draft charter is a reference to some
sort of framework (instead of the implicit "set of L3 and higher network
protocols") that bridges metrics and performance with at least "types of
applications".

I am NOT recommending that PMOL be a replacement for any
application-specific performance initiative.  Rather that it naturally
bridge the gap between the level of IPPM (and related) and
application-specific groups such as AVT or related ITU groups.
Otherwise, the breadth of the perceived need, to which PMOL has been
conceived as a response, may not be fully addressed.

Loki Jorgenson
Apparent Networks
t   604 433 2333 ext 105
m   604 250-4642

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Friday, September 21, 2007 6:24 AM
To: [email-masked]
Subject: [PMOL] Draft Charter for discussion

Al

Proposed Charter (0.0)

---

From: Al Morton <[email-masked]>
Date: Fri, 21 Sep 2007 17:34:24 -0400

Loki,

At 04:49 PM 9/21/2007, Loki Jorgenson wrote:
>...What would seem missing from the draft charter is a reference to some
>applications". ...

I avoided an explicit framework because the intent is that this WG
be able to work on performance metric proposals both above
and below IP (if it continues beyond this initial charter).

For example, the latest version of  draft-xie-ccamp-lsp-dppm-
is a proposal on control plane performance below IP.
PMOL might be looking at that in a year or so, if it's successful.

In one sentence, this WG might cover vetted work that the other
charters don't, but perhaps that's too broad and blunt.

Al

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Sun, 23 Sep 2007 16:20:40 +0200

Hi Al, 

I have a couple of comments: 

- when describing the mandate of IPPM it would be more complete to
mention that IPPM explicitly limits its scope in the charter to a set of
metrics related to Internet data delivery 
- Guidelines should mention submission to the IESG rather than 'AD
review' and specify the target status for each of the two documents -
BCP for the framework and guidelines document, and Proposed Status for
the SIP metrics document. 

Also language-like it may be better to refer to 'applications and
sessions' rather than just 'applications' to make clear that with PMOL
we do not target only application layers metrics. 

Dan

> Sent: Friday, September 21, 2007 3:24 PM
> To: [email-masked]
> suitability of new technologies and applications for their 
> intended audience, and the Internet is no exception. Most 
> uncertainties are effectively addressed through quantified 
> assessment of key performance indicators.  Standardized 
> performance metrics add the desirable features of consistent 
> implementation, interpretation, and comparison.
> 
> Although the IETF has two Working Groups dedicated to the 
> development of performance metrics, they each have strict 
> limitations in their
> charters:
> 
>   - The Benchmarking Methodology WG has addressed a range of 
> networking technologies and protocols in their long history 
> (such as IEEE 802.3, ATM, Frame Relay, and Routing 
> Protocols), but the charter strictly limits their performance 
> characterizations to the laboratory environment.
> 
>   - The IP Performance Metrics WG has the mandate to develop 
> metrics applicable to live IP networks, but it is 
> specifically prohibited from developing metrics that 
> characterize traffic (such as a VoIP stream).
> 
> The IETF also has current and completed activities related to 
> the reporting of application performance metrics (e.g. 
> RAQMON) and is also actively involved in the development of 
> reliable transport protocols which would affect the 
> relationship between IP performance and application performance.
> 
> development of performance metrics for IP-based applications 
> that operate over UDP, TCP, SCTP, DCCP, Forward Error 
> Correction (FEC) and other robust transport protocols, and 
> that can be used to characterize traffic on live networks.
> 
> Discussion of new work proposals is strongly discouraged in 
> the PMOL WG, except to advise a protocol development WG when 
> they are evaluating a new work proposal for related 
> performance metrics.
> 
> The PMOL WG will also be guided by a document describing how 
> memos defining performance metrics are intended to advance 
> along the IETF Standards track (draft-bradner-metricstest).
> 
> Milestones
> June 08  SIP Performance Metrics Draft to AD Review Sept 08  
> PMOL Framework and Guidelines Draft to AD Review
> 
> 
> 
>

---

From: Al Morton <[email-masked]>
Date: Sun, 23 Sep 2007 23:40:39 -0400

At 10:20 AM 9/23/2007, Romascanu, Dan (Dan) wrote:
>review'

OK. I was thinking of "Publication Request" where
the first step is usually AD review, and IESG review follows.
Al

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 24 Sep 2007 09:16:07 +0200

It's the same but better name the WG action. AD review is already out of
the WG responsibility :-)

Dan

> Sent: Monday, September 24, 2007 5:41 AM
> To: Romascanu, Dan (Dan); [email-masked]
> Subject: RE: [PMOL] Draft Charter for discussion
> 
> OK. I was thinking of "Publication Request" where the first 
> step is usually AD review, and IESG review follows.
> Al
>