Subject: Re: [AVT] I-D ACTION:draft-ietf-avt-rtcpxr-video-02.txt

From: Colin Perkins <[email-masked]>
Date: Sun, 2 Dec 2007 08:20:13 -0800

On 19 Nov 2007, at 15:15, [email-masked] wrote:
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
> This draft is a work item of the Audio/Video Transport Working  
> Group of the IETF.
>
> 	Title		: RTCP XR - Video Metrics Report Blocks
> 	Author(s)	: A. Clark, A. Pendleton
> 	Filename	: draft-ietf-avt-rtcpxr-video-02.txt
> 	Pages		: 0
> 	Date		: 2007-11-19
> 	
> This document defines extensions to the RTCP XR extended report
>    packet type blocks to support the monitoring of video over IP
>    for IPTV and videoconferencing endpoint reporting.
>
> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-ietf-avt-rtcpxr-video-02.txt

There seems to be an assumption that video comprises I, P, and B  
frames. How valid is this, given the flexibility in modern codecs?

Mean video bit rate shouldn't be tied to MPEG transport. Also, what  
do you report for a stream using RTP header compression (the draft  
requires RTP headers to be included, but not IP/UDP headers, but  
header compression operates cross-layer).

Also, my comments on the rtcpxr-audio draft apply here.

-- 
Colin Perkins
http://csperkins.org/

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Stephan Wenger <[email-masked]>
Date: Sun, 2 Dec 2007 13:52:24 -0800

On Dec 2, 2007, at 8:20 AM, Colin Perkins wrote:

>> [...]
>> [...]
> [...]

Not very.  Certainly not for any modern services, YouTube and such.   
At present there may still be some relevance for the first generation  
of IP-TV, but I view even this to disappear rather quickly.

My two cents,
Stephan

---

From: "Even, Roni" <[email-masked]>
Date: Sun, 2 Dec 2007 23:56:59 +0200

Hi,
I agree with Colin's comments, examples are the H264-SVC and evbr audio
codec. The work should try to address both old and new codecs
I also would suggest having reference to metrics defined in the ITU or
other standard bodies
Roni Even

> -----Original Message-----
> From: Colin Perkins [mailto:[email-masked]]
> Sent: Sunday, December 02, 2007 6:20 PM
> To: AVT WG
> Subject: Re: [AVT] I-D ACTION:draft-ietf-avt-rtcpxr-video-02.txt
>

---

From: "Alan Clark" <[email-masked]>
Date: Mon, 3 Dec 2007 11:17:32 -0500

Colin, Roni

This draft does refer to the ATIS 080008 standard - which is the most
definitive set of IP video related metrics I'm aware of and which has been
used as a reference by the ITU-T IPTV FG, DSL Forum and other groups.  For
those on the list that aren't familiar with ATIS - it is the telecom carrier
equivalent of TIA and EIA. 

With regard to metrics defined in ITU - then any pointers to specific
metrics, ITU Recommendations or even Questions would be very useful.  I am
tracking work in VQEG, ITU-T SG9, ITU-T SG12 and the ITU-T IPTV Focus Group
however the ATIS standard appears to be the most advanced.

On the question of I/B/P frame statistics - these could either be defined in
terms of "I, B, P frames or slices" (which ATIS 080008 uses) or could be
defined in a way that is independent of the frame/slice structure used by
the encoder.  For example, using the term "video frame" to mean a decoded
frame intended for display:

 - Proportion of video frames impaired
 - Proportion of video frames impaired by error propagation from previous
frames

The least ambiguous definition of video bandwidth would appear to be the
bandwidth of the RTP payload, which would include MPEG Transport overhead.
MPEG Transport is not subject to RTP header compression and generally
represents a fixed overhead.

Regards

Alan

-----Original Message-----
From: Even, Roni [mailto:[email-masked]] 
Sent: Sunday, December 02, 2007 4:57 PM
To: Colin Perkins; AVT WG
Subject: RE: [AVT] I-D ACTION:draft-ietf-avt-rtcpxr-video-02.txt 

Hi,
I agree with Colin's comments, examples are the H264-SVC and evbr audio
codec. The work should try to address both old and new codecs I also would
suggest having reference to metrics defined in the ITU or other standard
bodies Roni Even

> > This draft is a work item of the Audio/Video Transport Working Group 
> > of the IETF.
> >
> Mean video bit rate shouldn't be tied to MPEG transport. Also, what do 
> you report for a stream using RTP header compression (the draft

---

From: <[email-masked]>
Date: Mon, 3 Dec 2007 20:48:32 +0200

Hi,

Here are some comments and questions on this draft. 

- On I, P and B frames. This differentiation in quality reporting is
probably OK for old video coding standards like MPEG-2 Video, but not
good enough for new standards like H.264/AVC and its extensions Scalable
Video Coding (SVC) and Multiview Video Coding (MVC). Traditional I and P
frames are always reference frames (hence the decoding will affect other
frames) while B frames are non-reference frames (hence the decoding will
not affect other frames). This has been changed in H.264/AVC and its
extensions. In H.264/AVC, there are IDR pictures, normal intra pictures,
reference and no-reference P or B pictures (though the spec itself
specifies "slices" instead of "pictures"). There may also be SP and/or
SI picture in H.264/AVC. In SVC, the scalability indications would
decide the relative importance of pictures. 

- Program ID (PID) is the header. But cow common would MPEG transport be
used in IPTV and video conconferencing?

- On Measurement Interval: when does the interval start and end?

- Aren't "proportion of impaired I frames" and "loss rate within I
frames" essentially the same? When are they different?

- On "mean inter-I-frame gap (MIIF)": A) Is this reported from the
sender to the receiver? Asked this because the sender has the knowledge.
What is the benefit of having this? B) The term "Groups of Pictures
(GoP)" needs to be defined, because you said "The I frames occur at the
start of Groups of Pictures (GoP)" and because the term is not defined
in all video coding standards. C) Could you please explain a bit of the
equation? 

- Why to have Max GoP Length (MGoP)?

- How exactly is Estimated PSNR (EPSNR) derived?

- On EPSNR Threshold: Is this to be sent from the sender to the
receiver? If from receiver to sender, how is decided by the receiver?

- On Mean Video Bit Rate: From sender to receiver or the other way
round? Why to have this?

- On Round Trip Delay: Defined as "The Round Trip Delay between the
originating and terminating ends of this RTP stream, expressed in
millseconds." According to this definition, it should be "media duration
time"? Why to have this?

- On A-V Delay: I guess this is related to jitter. How exactly is this
derived?

- On Video Playout Buffer Size: This includes only the RTP reception
buffer (i.e. exlcudes decoder buffer)?

- Finally, I think some of the QoE metrics defined in the 3GPP PSS (3GPP
TS 26.234) and MBMS (3GPP TS 26.346) worth a looking into to proceed
this work. 

BR, YK
http://research.nokia.com/people/ye_kui_wang 

>From: ext Alan Clark [mailto:[email-masked]] 
>Sent: Monday, December 03, 2007 6:18 PM
>To: 'Even, Roni'; 'Colin Perkins'; 'AVT WG'
>This draft does refer to the ATIS 080008 standard - which is 
>the most definitive set of IP video related metrics I'm aware 
>of and which has been used as a reference by the ITU-T IPTV 
>FG, DSL Forum and other groups.  For those on the list that 
>aren't familiar with ATIS - it is the telecom carrier 
>With regard to metrics defined in ITU - then any pointers to 
>specific metrics, ITU Recommendations or even Questions would 
>be very useful.  I am tracking work in VQEG, ITU-T SG9, ITU-T 
>SG12 and the ITU-T IPTV Focus Group however the ATIS standard 
>appears to be the most advanced.
>
>On the question of I/B/P frame statistics - these could either 
>be defined in terms of "I, B, P frames or slices" (which ATIS 
>080008 uses) or could be defined in a way that is independent 
>of the frame/slice structure used by the encoder.  For 
>example, using the term "video frame" to mean a decoded frame 
>intended for display:
>
> - Proportion of video frames impaired by error propagation 
>from previous frames
>
>The least ambiguous definition of video bandwidth would appear 
>to be the bandwidth of the RTP payload, which would include 
>MPEG Transport overhead.
>MPEG Transport is not subject to RTP header compression and 
>generally represents a fixed overhead.
>
>
>Regards
>
>Alan
>
>
>I agree with Colin's comments, examples are the H264-SVC and 
>evbr audio codec. The work should try to address both old and 
>new codecs I also would suggest having reference to metrics 
>defined in the ITU or other standard bodies Roni Even
>

---

From: "Alan Clark" <[email-masked]>
Date: Mon, 3 Dec 2007 15:19:51 -0500

> Here are some comments and questions on this draft. 

Thanks!!
 
> - On I, P and B frames. This differentiation in quality 
> reporting is probably OK for old video coding standards like 
> MPEG-2 Video, but not good enough for new standards like 
> H.264/AVC and its extensions Scalable Video Coding (SVC) and 
> Multiview Video Coding (MVC). Traditional I and P frames are 
> always reference frames (hence the decoding will affect other
> frames) while B frames are non-reference frames (hence the 
> decoding will not affect other frames). This has been changed 
> in H.264/AVC and its extensions. In H.264/AVC, there are IDR 
> pictures, normal intra pictures, reference and no-reference P 
> or B pictures (though the spec itself specifies "slices" 
> instead of "pictures"). There may also be SP and/or SI 
> picture in H.264/AVC. In SVC, the scalability indications 
> would decide the relative importance of pictures. 
> 

Presumably - this still results in some displayed frames being unimpaired,
some being impaired due to problems that impacted the data decoded to render
all or part of "this" frame and some being impaired due to problems that
impacted earlier reference frames.  

Do you think that if the metrics were defined in this way rather than I,B,P
that would be sufficient to address the various possibilities that H.264/AVC
and other future video codecs would raise?

> - Program ID (PID) is the header. But how common would MPEG 
> transport be used in IPTV and video conconferencing?

Very common in IPTV systems, which very typically carry a single video
stream and the associated audio streams in MPEG transport format (together
with PATs, PMTs etc)

> - On Measurement Interval: when does the interval start and end?

The assumption is that RTCP SR NTP timestamps can be used for the end time
of the interval

> - Aren't "proportion of impaired I frames" and "loss rate 
> within I frames" essentially the same? When are they different?

They are quite different - for example you could have 2% packet loss within
80% of the I frames - hence the proportion of impaired I frames would be 80%
and the packet loss rate within I frames would be 2%.

> - On "mean inter-I-frame gap (MIIF)": A) Is this reported 
> from the sender to the receiver? Asked this because the 
> sender has the knowledge.

The sender does have knowledge however with content being relayed from other
services (e.g. a video stream is created, sent in MPEG Transport over UDP to
the demarc point with a service provider, who then maps the video stream
onto IP/RTP and sends over their IP network to the user) the sending RTP
endpoint may not have knowledge of the internal structure of the encoded
video stream. (** for later reference)

> What is the benefit of having this? B) The term "Groups of 
> Pictures (GoP)" needs to be defined, because you said "The I 
> frames occur at the start of Groups of Pictures (GoP)" and 
> because the term is not defined in all video coding 
> standards. 

Maybe we need to define something more generic however for those systems
that do have a GoP (either fixed or variable length) it is very useful to
know.  Obviously the issues is that a short GoP reduces error propagation
but takes more bandwidth - the sending end knows what the GoP size is
however this may be adaptive/ variable and service providers would like to
understand what "is" happening.

C) Could you please explain a bit of the equation? 

Based on the notion that encoders may insert I frames (or their equivalent)
when there is a scene change or panning, it is useful to report.  An
unexpectedly high number of I frames can increase bandwidth and lead to
localized congestion.

> - Why to have Max GoP Length (MGoP)?

To give some idea of the maximum error propagation that may occur

> - How exactly is Estimated PSNR (EPSNR) derived?

Two methods - some decoders produce an estimated PSRN from analysis of the
proportion of macroblocks they need to repair/ replace.  Also ATIS is
developing a trial use standard for calculating this from the decoder input

> - On EPSNR Threshold: Is this to be sent from the sender to 
> the receiver? If from receiver to sender, how is decided by 
> the receiver?

This would typically be defined by a service provider and provisioned in the
STB, video endpoint.

> - On Mean Video Bit Rate: From sender to receiver or the 
> other way round? Why to have this?

Sender-Receiver - to give service providers an understanding of the
bandwidth usage vs their expectations.  They may not exactly know the sender
parameters **

> - On Round Trip Delay: Defined as "The Round Trip Delay 
> between the originating and terminating ends of this RTP 
> stream, expressed in millseconds." According to this 
> definition, it should be "media duration time"? Why to have this?

This is the sender-terminator "end" of the RTP session - not the start-end
of the session.  This is mainly useful in IP videoconferencing applications

> - On A-V Delay: I guess this is related to jitter. How 
> exactly is this derived?

Audio-video sync delay - should be measured on the output of the decoder.
This is a very well known and understood parameter, with standards defining
limits - I'll have to find a reference on measuring it.

> - On Video Playout Buffer Size: This includes only the RTP 
> reception buffer (i.e. exlcudes decoder buffer)?

The assumption is that this is the variable size buffer preceding the
decoder and would generally be the RTP buffer.

> - Finally, I think some of the QoE metrics defined in the 
> 3GPP PSS (3GPP TS 26.234) and MBMS (3GPP TS 26.346) worth a 
> looking into to proceed this work. 

Thanks, I'll take a look at those

Best Regards

Alan

---

From: <[email-masked]>
Date: Mon, 3 Dec 2007 23:07:49 +0200

Responses interleaved in below. 

>> probably OK for old video coding standards like
>H.264/AVC and 
>> its extensions Scalable Video Coding (SVC) and Multiview 
>Video Coding 
>> (MVC). Traditional I and P frames are always reference frames (hence 
>> the decoding will affect other
>> frames) while B frames are non-reference frames (hence the decoding 
>> will not affect other frames). This has been changed in 
>H.264/AVC and 
>> its extensions. In H.264/AVC, there are IDR pictures, normal intra 
>> pictures, reference and no-reference P or B pictures (though 
>the spec 
>> itself specifies "slices"
>> instead of "pictures"). There may also be SP and/or SI picture in 
>> H.264/AVC. In SVC, the scalability indications would decide the 
>> relative importance of pictures.
>> 
>
>Presumably - this still results in some displayed frames being 
>unimpaired, some being impaired due to problems that impacted 
>the data decoded to render all or part of "this" frame and 
>some being impaired due to problems that impacted earlier 
>reference frames.  
>
>Do you think that if the metrics were defined in this way 
>rather than I,B,P that would be sufficient to address the 
>various possibilities that H.264/AVC and other future video 
>codecs would raise?
>

If rough differentiation of two or three types is good enough, then I
think you can make 1) intra, 2) reference inter, and 3) non-reference
inter, which will be generic enough for all video codecs. 

>The assumption is that RTCP SR NTP timestamps can be used for 
>the end time of the interval
>

OK - then I think it should be spelled out explicitly. 

>They are quite different - for example you could have 2% 
>packet loss within 80% of the I frames - hence the proportion 
>of impaired I frames would be 80% and the packet loss rate 
>within I frames would be 2%.
>

In average, the allocation of macroblocks to slices in intra pictures
should be roughly even. For intra pictures, only the area covered by
lost or discarded slices will be impaired. Therefore, the two in average
should be roughly equivalent unless strange encoding scheme has been
applied. 

>The sender does have knowledge however with content being 
>relayed from other services (e.g. a video stream is created, 
>sent in MPEG Transport over UDP to the demarc point with a 
>service provider, who then maps the video stream onto IP/RTP 
>and sends over their IP network to the user) the sending RTP 
>endpoint may not have knowledge of the internal structure of 
>the encoded video stream. (** for later reference)
>

OK.

>> (GoP)" needs to be defined, because you said "The I frames occur at 
>> the start of Groups of Pictures (GoP)" and because the term is not 
>> defined in all video coding standards.
>
>Maybe we need to define something more generic however for 
>those systems that do have a GoP (either fixed or variable 
>length) it is very useful to know.  Obviously the issues is 
>that a short GoP reduces error propagation but takes more 
>bandwidth - the sending end knows what the GoP size is however 
>this may be adaptive/ variable and service providers would 
>like to understand what "is" happening.
>

OK. Note that there are more that would affect the compromise between
coding efficiency and error resilience of a video stream, e.g.
percentage of intra macroblocks. 

>Based on the notion that encoders may insert I frames (or 
>their equivalent) when there is a scene change or panning, it 
>is useful to report.  An unexpectedly high number of I frames 
>can increase bandwidth and lead to localized congestion.
>

I meant why the numbers 15 and 16 are in the equaltion, not anything
else. 

>To give some idea of the maximum error propagation that may occur

OK. However, I doubt that this can give any meaningful indication of the
error resilience property of a video stream. 

>Two methods - some decoders produce an estimated PSRN from 
>analysis of the proportion of macroblocks they need to repair/ 
>replace.  Also ATIS is developing a trial use standard for 
>calculating this from the decoder input
>

I meant that the precise derivation method, e.g. using equations, should
be provided for people to better understanding the metric, and it should
be specified clearly anyway in the final spec. 

>This would typically be defined by a service provider and 
>provisioned in the STB, video endpoint.
>

OK. Thanks. 

>Sender-Receiver - to give service providers an understanding 
>of the bandwidth usage vs their expectations.  They may not 
>exactly know the sender parameters **
>

I think during session negotiation the sender and the receiver should
have already exchanged the bitrate information, thus both should know?

>> millseconds." According to this definition, it should be "media 
>> duration time"? Why to have this?
>
>This is the sender-terminator "end" of the RTP session - not 
>the start-end of the session.  This is mainly useful in IP 
>videoconferencing applications
>

Could you clarify in more detail: when is "the originating end of this
RTP session", and when is "the terminating end of this RTP session"?
Sorry if these are common terms used in AVT. 

>exactly is this 
>This is a very well known and understood parameter, with 
>standards defining limits - I'll have to find a reference on 
>measuring it.
>

Sorry I don't know very well this well-know term, but we have tried to
make the language deadly clear in defining the jitter metric in 3GPP PSS
and MBMS. For a standard specification, I think it should be clearly
specified anyway. 

>The assumption is that this is the variable size buffer 
>preceding the decoder and would generally be the RTP buffer.
>

OK. Thanks. 

BR, YK

>> - Finally, I think some of the QoE metrics defined in the 3GPP PSS 
>> (3GPP TS 26.234) and MBMS (3GPP TS 26.346) worth a looking into to 
>> proceed this work.
>
>> >definitive set of IP video related metrics I'm aware of and 
>which has 
>> >been used as a reference by the ITU-T IPTV FG, DSL Forum and other 
>> >groups.  For those on the list that aren't familiar with 
>ATIS - it is 
>> >the telecom carrier equivalent of TIA and EIA.
>> >
>> >metrics, ITU Recommendations or even Questions would be 
>very useful.  
>> >I am tracking work in VQEG, ITU-T SG9, ITU-T
>> >SG12 and the ITU-T IPTV Focus Group however the ATIS 
>standard appears 
>> >to be the most advanced.
>> >
>> >On the question of I/B/P frame statistics - these could either be 
>> >defined in terms of "I, B, P frames or slices" (which ATIS
>> >080008 uses) or could be defined in a way that is 
>independent of the 
>> >frame/slice structure used by the encoder.  For example, using the 
>> >term "video frame" to mean a decoded frame intended for display:
>> >
>> > - Proportion of video frames impaired by error propagation from 
>> >previous frames
>> >
>> >The least ambiguous definition of video bandwidth would 
>appear to be 
>> >the bandwidth of the RTP payload, which would include MPEG 
>Transport 
>> >overhead.
>> >I agree with Colin's comments, examples are the H264-SVC and evbr 
>> >audio codec. The work should try to address both old and 
>new codecs I 
>> >also would suggest having reference to metrics defined in 
>the ITU or 
>> >other standard bodies Roni Even
>> >
>> >> Audio/Video Transport Working Group [email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Mon, 03 Dec 2007 18:41:14 -0500

>limits - I'll have to find a reference on measuring it.

We'll need some type of definition/equation - most-recent, average,
maximum, over what period, what sort of aging of data, what exactly it
means, etc.  There are "standards" for DVD's, etc that set maximums.  Of
course, I've seen (many times) LCD TV's connected via HDMI to a cable
settop with a 2 second(!!!!) lip-sync delay.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: "Alan Clark" <[email-masked]>
Date: Tue, 4 Dec 2007 13:59:01 -0500

> >> (3GPP TS 26.234) and MBMS (3GPP TS 26.346) worth a looking into to 

Ye-Kui

I've reviewed the two references - these appear to define the following
metrics

(i) Corruption duration
(ii) Rebuffering duration
(iii) Initial buffering duration
(iv) Successive RTP loss
(v) Frame rate deviation
(vi) Jitter duration

I assume that the most relevant of these to yesterday's IBP frame discussion
was the Corruption Duration metric.  

I also note that Nokia appears to have several patent applications that
"may" be relevant to these metrics?  I've been trying to keep the RTCP XR
video/ audio/ transport payloads free of IPR issues as far as possible hence
IMHO early disclosure of any IPR interests on specific metrics that are
being proposed would be helpful (although it is obviously a WG decision on
the IPR vs technical merit tradeoff).

Best Regards

Alan

---

From: <[email-masked]>
Date: Wed, 5 Dec 2007 01:11:50 +0200

Hi Alan,

You are fast! 

I believe Nokia is one of the forerunners of timely IPR declarations
according to the IETF patent policy. 

BR, YK 

>Sent: Tuesday, December 04, 2007 8:59 PM
>To: Wang Ye-Kui (Nokia-NRC/Tampere); [email-masked]; 
>[email-masked]; [email-masked]
>3GPP PSS 
>I've reviewed the two references - these appear to define the 
>following metrics
>
>(i) Corruption duration
>I assume that the most relevant of these to yesterday's IBP 
>frame discussion was the Corruption Duration metric.  
>
>I also note that Nokia appears to have several patent 
>applications that "may" be relevant to these metrics?  I've 
>been trying to keep the RTCP XR video/ audio/ transport 
>payloads free of IPR issues as far as possible hence IMHO 
>early disclosure of any IPR interests on specific metrics that 
>are being proposed would be helpful (although it is obviously 
>a WG decision on the IPR vs technical merit tradeoff).
>
>Best Regards
>
>Alan
>
>