Subject: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13

From: Robert Sparks <[email-masked]>
Date: Tue, 15 May 2012 18:13:39 -0500

Summary: There are issues to resolve with a revised ID before 
progressing to IETF LC.

It's a struggle to see the _architecture_ this document is supposed to 
describe. I suspect most of the working group energy went into Section 5 
(the guidelines for writing XR block definitions) - the rest of the 
document needs a similar level of attention to detail.

There are two major issues that I've tagged with a (*). I followed a 
mostly document-order flow instead of grouping the major issues at the top.

(*) Given the document's title, it should be very easy to answer "What 
is the monitoring architecture?" for people who are reading this draft 
for the first time.
The key message is that the monitoring architecture consists of Monitors 
that exchange information using RTCP Metric Blocks. The text should say 
that. There is a lot of prose in section 3 and visual detail in the 
diagram in Figure 1 that makes that extremely difficult to see. 
Simplifying the figure by removing the details about applications and 
transports, or at least moving them to a second figure, making the 
Monitor boxes stand out, and making the types of information flows (the 
arrows) visually distinctive would help. Simplify the prose (much of it 
could be removed) to focus on that key message.

In the definition of Composed metrics, the sentence "An example is a 
metric calculated based on derived metrics that have been measured." 
makes little sense. Did you mean something like "An example is a metric 
derived by algorithmically combining measured metrics."?

The definitions of Interval, Cumulative, and Sampled metrics are not clear.

If the discussion at the end of section 3.1 is kept, the reference to 
RFC5760 should be a real reference, and RFC5760 should appear in the 
reference section.

The point of section 3.2 is not obvious. How does it contribute to the 
description of the architecture. It mixes talking about what report 
blocks are with occasionally needing to send them more frequently. Those 
points could be made separately and more simply.

Section 3.3 is not clear - it starts off saying that the location of the 
sender and receiver may change what metrics are meaningful. It's not 
clear what you mean by location. You certainly don't mean geospatial 
coordinates. Do you mean anything other than whether the element is 
trusted or untrusted by the operations network? If so, say that. The 
rest of the paragraph only speaks to what element collects the metrics. 
It's not clear that the set of meaningful metrics is changed at all - 
just where you collect them. If the set of meaningful metrics does in 
fact change, there needs to be additional discussion here.

Section 4's first paragraph: Is  "probably" the best the working group 
can say?

The last two sentences of the second bullet in section 4 do not parse. 
Please simplify them.

The first full sentence of the third bullet in section 4 does not make 
sense. The whole bullet could be clarified.

The fourth bullet in section 4 and section 5.5 should call out that 
RFC3611 already set Block Type 255 aside for future extensions, 
anticipating the potential need to extend the block type space.

In section 5, consider naming the subsections with the actual guideline 
instead of the problem the guideline is trying to address.
5.1. Metric blocks should contain a single metric
5.2. Include the payload type and format parameters in the metric block
5.3. Use RTCP SDES to correlate XR reports with non-RTP data
5.4. Don't repeat measurement information across metric blocks

This points out a structural problem with the section. Why is 5.5 here? 
It's not a guideline.

(*) And it raises a question about what the document is trying to 
achieve with section 5.4. That section claims the document is proposing 
a new XR block type to help avoid duplication, but it doesn't actually 
define the new block type. Is the guideline saying simply don't repeat 
measurements or its the guideline saying "use this new XR block type we 
haven't defined yet"? The section needs more clarity. If it really is 
trying to define a new type, there's more work to be done.

The Security Considerations section, particularly the third paragraph, 
needs to be clearer. What exactly do you mean by "monitoring should be 
prohibited", and how is the fact that third party monitors don't send 
RTCP relevant? Did you intend those to be separate points?

Editorial Nits and Suggestions:

The first paragraph of the introduction is overly complex, and won't age 
well. I suggest replacing it with: "Multimedia services using the 
Real-Time Protocol (RTP) are seeing increased use. Standard methods for 
gathering RTP performance metrics from these applications are needed to 
manage uncertainties in the behavior and availability of their 
services." Then replace "These rapidly emerging standards," with 
"Standards".

Introduction paragraph 3 sentence 2 should start "The RTCP Guidelines 
[RFC5968]". It would be even better to use the full title from 5968: 
'The "Guidelines for Extending the RTP Control Protocol (RTCP)" [RFC5968]'

This may become moot as section 3 is modified to make its main point 
more obvious, but RFC3550 does not characterize Monitors as "a source of 
information". That could be misunderstood. Please choose another word. 
Perhaps "repository"?

In bullet lists, starting an entry with "or" or "and" is unnecessary. It 
should be clear from the introduction to the list whether the individual 
entries should be combined or selected from.

In section 3.1 where you say "RTP may be used to multicast groups" did 
you mean "RTP may be sent to multicast groups" or "RTP may be used with 
multicast groups"?

In section 3.2 "The following are four use cases but are not limited 
to:" does not parse. The bullets aren't really use cases as much as they 
are mechanisms. Would it make more sense to say "Here are four 
mechanisms that have been created to address the need for more frequent 
reporting. This list is not intended to be exhaustive." The third bullet 
in that list should say "RTCP and RTCP XR are" instead of "RTCP or RTCP 
XR is".

The bullets in section 4 would be better as subsections with titles 
(taken from the first phrase of the bullet).

I suggest this alternative for the first full sentence in the first 
bullet of section 4 : "A compound metrics block is designed to contain a 
large number of parameters from different classes for a specific 
application in a single block."

Section 5's title should be "Guidelines for reporting" instead of 
"Guideline for reporting"

In section 5.1 I suggest "justify the overhead," instead of "justify the 
overheads,"

Section 7 - the last sentence of the first paragraph does not parse. The 
second paragraph (which is a single sentence) should be clarified - an 
MCU is not a topology.

The last two sentences of the first paragraph of the security 
considerations section add nothing to the document. I suggest deleting them.

---

From: Qin Wu <[email-masked]>
Date: Fri, 18 May 2012 18:37:56 +0800

Hi,Robert:
Thank for your valuable review. please see my replies inline belows.

Regards!
-Qin
----- Original Message ----- 
From: "Robert Sparks" <[email-masked]>
To: "IETF AVTCore WG" <[email-masked]>; <[email-masked]>
Sent: Wednesday, May 16, 2012 7:13 AM
Subject: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13

> that exchange information using RTCP Metric Blocks. 

[Qin]: Agree.

>The text should say 
> that.

[Qin]: Okay.

> There is a lot of prose in section 3 and visual detail in the 

[Qin]: Agree, I propose to remove the figure 1 and squeeze the section 3.1 as follows:

"

3.1.  Overview

   The RTP monitoring architecture comprises the following two key

   functional components shown below:

   o  RTP Monitor

   o  RTP Metric Block Structure

   RTP Monitor is the functional component defined in the Real-time

   Transport Protocol [RFC3550]. It acts as a source of information

   gathered for monitoring purposes and exchanges information with 

   the other RTP monitors using RTCP Metric Blocks.  According to 

   the definition of monitor in the

   RTP Protocol [RFC3550], the end system that runs an application

   program that sends or receives RTP data packets, an intermediate-

   system that forwards RTP packets to End-devices or a third party that

   observes the RTP and RTCP traffic but does not make itself visible to

   the RTP Session participants (i.e., the third party monitor depicted

   in figure 1) can act as the RTP monitor. The third party monitor should be

   placed on the RTP/RTCP paths between the sender, intermediate and the

   receiver.

   The RTP monitor also exposes real time Application QoS/QoE metric

   information in the appropriate report block format (e.g.,RTCP XR or 

   othe non-RTP means) to the management

   system.  Such information can be formulated as different metric types, 

   e.g.,application level metric, transport level metric, end system metric,

    interval metric, cumulative metric, sampled metric, direct metric, composed metric,etc.

   Both the RTCP or RTCP XR can be extended to convey these metrics.

   The details on transport protocols for these metric are described in

   Section 3.2.

   RTP may be used to multicast groups, both Any Source Multicast (ASM)

   and Source Specific Multicast (SSM).  These groups can be monitored

   using RTCP.  In the ASM case, the monitor is a member of the

   multicast group and listens to RTCP XR reports from all members of

   the ASM group.  In the SSM case, there is a unicast feedback target

   that receives RTCP feedback from receivers and distributes it to

   other members of the SSM group (see figure 1 of RFC5760).  The

   monitor will need to be co-located with the feedback target to

   receive all feedback from the receivers (this may also be an

   intermediate system).  In both ASM and SSM scenarios, receivers can

   send RTCP XR reports to enhance the reception quality reporting.

"

> makes little sense. 

[Qin] The derived metrics in the sentence should been corrected as "direct metric".

>Did you mean something like "An example is a metric 

[Qin]:I agree with your suggested text and like to adopt your suggestion.

> The definitions of Interval, Cumulative, and Sampled metrics are not clear.

[Qin]: I agree the definition for these three metrics need to be more cleaer, I proposes to do the following changes:

NEW Text:

"

   Interval metrics

      Metrics of which the reported values

      are measured in the most recent measurement interval duration between

      successive metrics reports. Such measurement interval is equal to one 

     RTCP report interval (RFC3550,Section 6.2). 

   Cumulative metrics

      Metrics of which the reported values

      apply to the accumulation period characteristic of cumulative

      measurements. The reported values are measured during the accumulation period. 

   The accumulation period spans over more than one RTCP report intervals.

   Sampled metrics

      Metrics of which the reported values only

      apply to the value of a continuously measured or calculated metric

      that has been sampled at any given instance of the RTCP report interval. 

     One example of such metric is the inter-arrival jitter described in the section 6.4.1 of RFC3550.

"

> RFC5760 should be a real reference, and RFC5760 should appear in the 

[Qin]: Okay. 

> points could be made separately and more simply.

[Qin]: Okay. I like to incorporate the key points in the section 3.2 into section 3.1

> rest of the paragraph only speaks to what element collects the metrics. 

[Qin] Section 3.3 is only used to describe where or from which RTP entity 

to gather the metrics (i.e., measurement point) in the physical network and

 is not focus of this document. Secondly, 

I think the definition of application level metrics has already clarified that 

application level metrics is normally gathered from user endpoint.

> fact change, there needs to be additional discussion here.

[Qin]: The metrics actually doesn't change.Since the definition of application level metrics 

Have already conveyed the meaning of what section 3.3 wants to describe, I propose to remove

this section.

> Section 4's first paragraph: Is  "probably" the best the working group 

[Qin]:No, will remove this word.

> The last two sentences of the second bullet in section 4 do not parse. 

[Qin]: Okay, how about change them as follows:

NEW Text:

"

     With these minimal number of

      RTCP statistics parameters mapped to non-RTCP

      measurements, it is hard to provide accurate

      measures of real time application quality,conduct 

     detailed data analysis and creates alerts timly to the

      users. Therefore correlation between RTCP XR and

      non-RTP data should be provided.

"

> sense. The whole bullet could be clarified.

[Qin]: Agree and will remove the first sentence.

> anticipating the potential need to extend the block type space.

[Qin]: Okay, How about revising the section 4,bullet 4 as follows:

"

Consumption of XR block code points.  The RTCP XR block namespace

      is limited by the 8-bit block type field in the RTCP XR header.

      Space exhaustion may be a concern in the future. Anticipating the

 potential need to extend the block type space, it is noted in [RFC3611]

 that Block Type 255 is reserved for future extensions, 

"

> 5.1. Metric blocks should contain a single metric

[Qin]: I suggest to change it as "Contain the single metrics in the metric block"

> 5.2. Include the payload type and format parameters in the metric block

[Qin]: Okay.

> 5.3. Use RTCP SDES to correlate XR reports with non-RTP data

[Qin]: Okay.

> 5.4. Don't repeat measurement information across metric blocks
[Qin]: It is not possible completely to avoid measurement information repeatition.

e.g., for packet loss robustness if the report blocks for the same interval

   span over more than one RTCP packet then each must have the

   same measurement identity information 

So I suggest to change it as:

"reduce measurement information across metric blocks"

> This points out a structural problem with the section. Why is 5.5 here? 

[Qin]: Agree and also as discussed,Space exhaustion is not big issue. How about remove this section?

> haven't defined yet"? 

[Qin]: Just to clarify, we have already defined one new XR Block 

type to help avoid or reduce duplication. It is measurement information 

block. The relevant draft is available at:

http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-meas-identity-06

>The section needs more clarity. If it really is 

[Qin]: In order to avoid such confusion, I proposed to do the following change to the section 5.4:

OLD Text:

"

This memo proposes to

   define a new XR Block that will be used to convey the common time

   period and the number of packets sent during this period.

"

New text:

"

This memo proposes to

   define a new XR Block (i.e., the Measurement

   information block[I.D-ietf-xrblock-rtcp-xr-meas-identity]) that will be used to convey the common time

   period and the number of packets sent during this period.

"

> RTCP relevant? Did you intend those to be separate points?

[Qin]: Yes, they are on the different port. Such monitor may receive RTP packet but does not send RTCP packet.

> services." Then replace "These rapidly emerging standards," with 

[Qin]: Okay.

> 'The "Guidelines for Extending the RTP Control Protocol (RTCP)" [RFC5968]'

[Qin]: Okay.

> information". That could be misunderstood. Please choose another word. 

[Qin]: Okay, repository looks good to me.

> entries should be combined or selected from.

[Qin]: Okay.

> you mean "RTP may be sent to multicast groups" or "RTP may be used with 

[Qin]: I think it should be the latter case,i.e.," RTP may be used with 
multicast groups".

> in that list should say "RTCP and RTCP XR are" instead of "RTCP or RTCP 

[Qin]: Okay, but I have incorporated the keypoints of section 3.2 into section 3.1.

> (taken from the first phrase of the bullet).

[Qin]: Okay.

> application in a single block."

[Qin]: Okay.

> "Guideline for reporting"

[Qin]: Okay.

> In section 5.1 I suggest "justify the overhead," instead of "justify the 

[Qin]: Okay.

> second paragraph (which is a single sentence) should be clarified - an 

[Qin]: Okay. How about revising them as follows:

OLD Text:

"

The topologies in this category

   are Topo-Video-Switch-MCU and Topo-RTCP-terminating-MCU.  Such

   topologies based on systems do not behave according to the RTP

   protocol [RFC3550].

   Considering the MCU and translator are two typical topologies in the

   two categories mentioned above, this document will take them as two

   typical examples to explain how RTCP XR report works in different

   RFC5117 topologies.

"

NEW Text:

"

The topologies in this category

   are Topo-Video-Switch-MCU and Topo-RTCP-terminating-MCU.  Such

   topologies based on systems (e.g.,MCU) do not behave according to the RTP

   protocol [RFC3550].

   Considering the MCU and translator are two typical intermediary systems in the

   two categories mentioned above, this document will take them as two

   typical examples to explain how RTCP XR report works in different

   RFC5117 topologies.

"

> considerations section add nothing to the document. I suggest deleting them.

[Qin]: Okay.

> _______________________________________________
> Audio/Video Transport Core Maintenance
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avt
>

---

From: Qin Wu <[email-masked]>
Date: Wed, 23 May 2012 09:30:57 +0800

>     One example of such metric is the inter-arrival jitter described in the section 6.4.1 of RFC3550.
[Qin]: One improvement to definitions of Interval Metrics, Cumulative Metrics and Sampled Metrics to get alignment with draft-ietf-xrblock-rtcp-xr-meas-identity
since based on draft-ietf-xrblock-rtcp-xr-meas-identity, in some case,the measurement interval for a metric is different from the RTCP
reporting interval, in such case, Measurement information block in the draft-ietf-xrblock-rtcp-xr-meas-identity can be used to specify such measurement interval.

The proposed improvement is described as follows:
"
Interval metrics:

Metrics calculated based on values measured over the course of a single reporting interval. This may be the most recent RTCP reporting interval ([RFC3550], section 6.2) or some other interval signalled using an RTCP Measurement Information XR Block [draft-ietf-xrblock-rtcp-xr-meas-identity]. An example interval metric is the count of the number of RTP packets lost over the course of the last RTCP reporting interval.

Cumulative metrics:

Metrics derived from cumulative measurements taken over several reporting intervals. The time period over which measurements are accumulated can be the complete RTP session, or some other interval signalled using an RTCP Measurement Information XR Block [draft-ietf-xrblock-rtcp-xr-meas-identity].  An example cumulative metric is the total number of RTP packets lost since the start of the RTP session.

Sampled metrics:

Metrics that are sampled at a particular time instant within a reporting interval (generally the value of some measurement as taken at the end of the reporting interval). An example is the inter-arrival jitter reported in RTCP SR and RR packets, which is continually updated as each RTP data packet arrives, but only reported based on a snapshot of the value which is sampled at the instant the reporting interval ends.

"

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Wed, 23 May 2012 12:29:34 -0700

Hi Qin,

The definitions section of the monarch draft needs to clearly define two
important concepts:
1) direct vs. composed metrics
2) interval vs. cumulative vs. sampled metrics
These are independent concepts, and the definitions need to be
independent as well. For that reason, I think it is important to avoid
any mixing of the two. Please see concerns regarding this inline.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Qin Wu
> Sent: Tuesday, May 22, 2012 6:31 PM
> To: Robert Sparks; IETF AVTCore WG; draft-ietf-avtcore-
> [email-masked]
> Subject: Re: [AVTCORE] Subject: AD review of draft-ietf-avtcore-
> monarch-13
> 
> >> The definitions of Interval, Cumulative, and Sampled metrics are
not
> clear.
> >
> > [Qin]: I agree the definition for these three metrics need to be
more
> cleaer, I proposes to do the following changes:
> >
> > NEW Text:
> >
> > "
> >
> >   Interval metrics
> >
> >
> >
> [Qin]: One improvement to definitions of Interval Metrics, Cumulative
> Metrics and Sampled Metrics to get alignment with draft-ietf-xrblock-
> rtcp-xr-meas-identity
> since based on draft-ietf-xrblock-rtcp-xr-meas-identity, in some
> case,the measurement interval for a metric is different from the RTCP
> reporting interval, in such case, Measurement information block in the
> draft-ietf-xrblock-rtcp-xr-meas-identity can be used to specify such
> measurement interval.
> 

All three definitions that follow need to use equivalent language such
that all three are clearly applicable to both direct and composed
metrics.

> "
> Interval metrics:
> 
> Metrics calculated based on values measured over the course of a
single
> reporting interval. This may be the most recent RTCP reporting
interval
> ([RFC3550], section 6.2) or some other interval signalled using an
RTCP
> Measurement Information XR Block [draft-ietf-xrblock-rtcp-xr-meas-
> identity]. An example interval metric is the count of the number of
RTP
> packets lost over the course of the last RTCP reporting interval.

How about replacing "Metrics calculated based on values measured over
the course ..." with "Metrics measured over the course ..."?
 
> Cumulative metrics:
> 
> Metrics derived from cumulative measurements taken over several
> reporting intervals. The time period over which measurements are
> accumulated can be the complete RTP session, or some other interval
> signalled using an RTCP Measurement Information XR Block [draft-ietf-
> xrblock-rtcp-xr-meas-identity].  An example cumulative metric is the
> total number of RTP packets lost since the start of the RTP session.
> 

How about replacing "Metrics derived from cumulative measurements taken
over several ..." with "Metrics measured over several ..."?

> Sampled metrics:
> 
> Metrics that are sampled at a particular time instant within a
> reporting interval (generally the value of some measurement as taken
at
> the end of the reporting interval). An example is the inter-arrival
> jitter reported in RTCP SR and RR packets, which is continually
updated
> as each RTP data packet arrives, but only reported based on a snapshot
> of the value which is sampled at the instant the reporting interval
> ends.
> 
> "

How about replacing "Metrics that are sampled at a particular time
instant ..." with "Metrics measured at a particular time instant..."?

This way the wording remains consistent across all three definitions
without mixing in any terms that may be confused to adding ties specific
to direct or composed metrics.

Cheers,
Charles

---

From: Qin Wu <[email-masked]>
Date: Fri, 25 May 2012 09:14:44 +0800

Hi, Charles:
I fully agree with your comments to distinct one concept from another. 
Please see my proposed change below.

Regards!
-Qin
----- Original Message ----- 
From: "Charles Eckel (eckelcu)" <[email-masked]>
To: "Qin Wu" <[email-masked]>; "Robert Sparks" <[email-masked]>; "IETF AVTCore WG" <[email-masked]>; <[email-masked]>
Sent: Thursday, May 24, 2012 3:29 AM
Subject: RE: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13

Hi Qin,

not
more

[Qin]:Okay. That's a good suggestion.

single
interval
RTCP
RTP

[Qin]: Okay, the proposed change to the definition of Interval metrics is described as follows:
NEW TEXT
"
Metrics measured over the course of a single reporting interval between two successive
report blocks. This may be the most recent RTCP reporting interval ([RFC3550], section 6.2) 
or some other interval signalled using an RTCP Measurement Information XR Block [MI]. 
An example interval metric is the count of the number of RTP packets lost over the course 
of the last RTCP reporting interval.

"

[Qin]: Okay, the proposed change to the definition of Interval metrics 
would be:
NEW TEXT
"
Metrics measured over several reporting intervals for accumulating statistics. 
The time period over which measurements are accumulated can be the
 complete RTP session, or some other interval signalled using an RTCP 
Measurement Information XR Block [MI].  An example cumulative metric 
is the total number of RTP packets lost since the start of the RTP session.

"

at
updated

[Qin]: Okay, the proposed change to the definition of Interval metrics 
would be:
NEW TEXT
"
Metrics measured at a particular time instant and sampled from the values of
a continuously measured or calculated metric within a reporting interval 
(generally the value of some measurement as taken at the end of the reporting interval).  
An example is the inter-arrival jitter reported in RTCP SR and RR packets, 
which is continually updated as each RTP data packet arrives, but only reported based
on a snapshot of the value which is sampled at the instant the reporting interval ends.
"

Cheers,
Charles

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Fri, 25 May 2012 13:08:45 -0700

Hi Qin,

This looks good to me.

Thanks,
Charles

> From: Qin Wu [mailto:[email-masked]]
> Sent: Thursday, May 24, 2012 6:15 PM
> To: Charles Eckel (eckelcu); Robert Sparks; IETF AVTCore WG; draft-
> [email-masked]
> To: "Qin Wu" <[email-masked]>; "Robert Sparks"
> <[email-masked]>; "IETF AVTCore WG" <[email-masked]>; <draft-ietf-
> [email-masked]>
> > >      successive metrics reports. Such measurement interval is
equal
> > to one
> > >      that has been sampled at any given instance of the RTCP
report
> > interval.
> > >
> > >     One example of such metric is the inter-arrival jitter
> described
> > in the section 6.4.1 of RFC3550.
> > >
> > > "
> is described as follows:
> Metrics measured over the course of a single reporting interval
between
> two successive