Subject: [AVT] Comments on draft-ietf-rmonmib-raqmon-framework-04.txt

From: "Alan Clark" <[email-masked]>
Date: Wed, 14 Jan 2004 14:18:25 -0500

I've copied this list of comments to AVT as some of them relate to the use
of RTCP for reporting purposes.

1. Scope
Although the framework makes general reference to "real time applications",
most of the examples give are VoIP related. It seems that the primary
intended use of RAQMON is VoIP performance monitoring and the umbrella "real
time applications" scope is something of a "red herring".

2. VoIP Media path QoS
The document suggests that RAQMON provides QoS statistics, including media
path stats, but does not refer to or incorporate RFC3611 - the scope of
which is specifically VoIP media path QoS.  The RFC3611 VoIP metrics form
the basis of a number of VoIP QoS reporting protocols, and seem to represent
a reasonable level of cross-industry consensus on what is "reasonable" for
VoIP QoS monitoring.   RAQMON stands alone in trying to argue that RTCP
SR/RR statistics are sufficient for VoIP (or Video) QoS monitoring when the
industry has moved on to other more appropriate metrics.

3. Reporting intervals
RAQMON proposes RTCP as one option for the transport of PDUs. RFC3550
provides guidance on the intervals at which RTCP frames should be sent.  The
RAQMON framework proposes an independent set of rules on RTCP transmission
timing - the AVT group is usually fairly insistent that the rules described
in RFC3550 are applied to RTCP and hence within the RAQMON draft the current
definition of reporting intervals should be replaced by references to the
appropriate parts of RFC3550.
RAQMON encourages the use of multiple consecutive RTCP frame transmissions
to send long RAQMON reports.  This would certainly be in conflict with the
recommendations of RFC3550.

4. Measurement methodology
RAQMON proposed that the measurement methodology used to generate the
reported metrics is "outside the scope" of RAQMON and should be left to the
implementer.  Whilst implementers can "get away with this" in certain closed
system applications it makes any form of open system/ multi-vendor network/
inter-domain QoS management almost impossible.  If the only application of
RAQMON is for use in closed, single-vendor applications then why make it a
standard?
Reference to using "IETF and ITU specificed methodologies" is insufficient,
for example there are many definitions of jitter within ITU and IETF
standards.

Some of the metrics specified are also impractical - for example one way
delay.  This requires an accurate synchronized time source at the two ends
of an IP connection - probes that implement this metric often use GPS to
obtain the level of timing accuracy needed, simply relying on NTP timers
(even with an NTP server) would be generally inadequate.

As another example, Section 5.11 defines Round Trip Delay as potentially
including or not including the end system delay.  Jitter buffers can
sometimes reach several hundred milliseconds in size and hence knowing if a
reported 300 millisecond round trip delay did or didn't include the end
system delay could make a large difference to the user perceived delay.

As another example, Section 5.14 describes the count of the number of
"application packets" as a metric that "could be measured in different ways,
including the methodology described in RFC3550".  Is an application packet a
CODEC frame, an RTP frame, UDP or IP packet?

5. Mis-understanding of loss and jitter
>From section 5.18 "Packet loss tracks persistent congestion while jitter
measures tracks transient congestion. Packet loss metric SHOULD indicate
loss associated to the network environment as well as local device losses
over time. Packet loss of underlying transport network can be measured using
various methodologies e.g. as described in [RFC2680], [RFC 3550] and local
device level packet losses should to be captured by the local device
specific algorithms."
5.a The first sentence is incorrect, also there is no apparent reason for
making the statement
5.b SHOULD leaves us not knowing if the reported loss number includes packet
discards or not.  If I have a call on which 2% of the packets were reported
as (lost, lost+discarded?) then do I know if this was a good call?
Obviously if the 2% were lost and an additional 5% were discarded then the
call would be much worse than if the 2% related to both lost and discarded
packets!!
5.c With reference to comment 4 above, "various methodologies" and "local
device specific algorithms" leaves us completely in the dark on what has
been reported.
6. References
Doesn't refer to RFC3611?
Regards
Alan Clark

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt