Subject: Re: [rmcat] Video Quality for RMCAT

From: Varun Singh <[email-masked]>
Date: Tue, 16 Oct 2012 15:34:40 +0300

Hi Stefan,

changing the email subject and snipping away the rest of the message.

Comments inline.

>> >
>> > 2) for what concerns the video quality indices, it is well known
>> > that PSNR is not well related to the QoE. (See for instance
>> >
>> > http://www.pscr.gov/about_pscr/press/video/video_quality_experts_interview_sigmm-122009.pdf)
>> > Are we sure we want to cite PSNR?
>> >
>>
>> I believe we need some objective metric for video either
>> PSNR or something else. I am happy to include or
>> update the document based on that discussion.
>> (Note: I have not read the above document but
>> will go through it.)
>>
>> However, when choosing these quality metrics we should be aware
>> of their defined operating range (some models are limited to specific
>> range of video resolutions or bit rates, etc.).
>>
>
> I agree that PSNR isn't likely to be a good choice, I would assume that we
> at least need a metric which takes both the video quality and the delay into
> account? The experienced video quality depends on several aspects other than

Apart from delay there is also balancing throughput and loss.
In the scenario of a single RTP flow we could come up with a
simple metric but with competing traffic it may be more difficult.
By introducing a video quality metric in those scenarios,
I was intending to quantify the balance of throughput, loss and delay.

> the congestion control, such as encoder settings, transmitted spatial and
> temporal resolution, video jitter buffer algorithm at the receiver, etc.

Since we are talking about interactive real-time media, we could *agree* on
running a baseline setup with preset parameters and reference video
sequences. :)

> Therefore I'm not sure it's a good idea to evaluate our congestion control
> algorithms based on video quality. If possible I would rather see us
> concentrate on simpler metrics as those you described earlier in your draft,
> which are more related to how the algorithm actually adapts to changes in
> bandwidth.

In my opinion, if we can come up with a good QoE metric then it
would be nice to have in addition to the throughput, loss and delay metrics.
OTOH if we come up with a quality metric that is based on the relationship
of the three then we may have a single metric to compare each algorithm.
I am not sure which of these two methods would be better, but I am happy
to hear more arguments for and against QoE, also other QoE methods
before we decide on a quality metric.

Regards,
Varun

---

From: Michael Welzl <[email-masked]>
Date: Tue, 16 Oct 2012 15:01:30 +0200

Sorry if this is stupid, perhaps I'm missing something - but:

even if we could agree on a baseline test setup, standard movies etc.: isn't the very idea of a QoE metric inevitably bound to the codec, which then binds evaluating congestion control mechanisms to a debate that this really shouldn't be a part of?

Cheers,
Michael

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 17 Oct 2012 17:31:27 -0700

> I was intending to quantify the balance of throughput, loss and delay.

If you want to measure video quality in the presence of changing 
bitrates, loss, and delay, you have a difficult problem on your hands. 
Not the least of which is that PSNR (or similar metrics) require a 
reference to compare against, which starts to get meaningless once you 
change resolution or framerate in the encoder (with dropping frames 
being a very crude form of changing framerate), all of which are things 
the encoder can do to adapt to changes in available bandwidth. I 
discussed some of this in more detail in my submission to the workshop: 
<http://www.tschofenig.priv.at/cc-workshop/irtf_iab-ccirtcpaper26.pdf>

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 17 Oct 2012 17:38:41 -0700

> even if we could agree on a baseline test setup, standard movies etc.: isn't the very idea of a QoE metric inevitably bound to the codec, which then binds evaluating congestion control mechanisms t…

Yes. I think there are two separable problems here:

1) Estimating the channel capacity
2) Allocating that capacity among multiple flows constrained by the same 
bottleneck (or multiple, multiplexed streams on the same flow, if you like).

The only interaction between problem 1 and the codec(s) is how quickly 
they can adapt to the changing estimates that are produced. I'm not sure 
that requires looking at the quality of the resulting content at all, 
however.

For problem 2, actual video/audio quality becomes much more interesting, 
but I think is something this is a problem that has to be solved by the 
application itself, not the congestion control layer. Only the 
application knows things like "I'm using G.711: the only way I can adapt 
its rate is to drop the packets, so all reductions must come from my 
video stream," or "I'm transmitting two video streams, but one of them 
is much easier to encode than the other, so when the rate drops, I 
should steal more bits from that one."

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 17 Oct 2012 23:14:21 -0400

On 10/17/2012 8:38 PM, Timothy B. Terriberry wrote:
>> even if we could agree on a baseline test setup, standard movies etc.:
>> isn't the very idea of a QoE metric inevitably bound to the codec,
>> which then binds evaluating congestion control mechanisms to a debate
>> that this really shouldn't be a part of?
> 

I totally agree with this.  The elasticity of the codecs matters, and
the congestion controller will benefit from an understanding of the
envelopes they can operate within, but it's the job of the codecs to
ensure quality within their envelopes of elasticity, and not at all the
congestion controller's job.

The CC may say "you now have N kbps to work within" to each codec, and
this will be based on what it estimates is safe for the network given
the current ensemble of codecs and their stated operating envelopes,
*not* on any explicit attempt by the CC to maximize user experience.

-- 
Wes Eddy
MTI Systems

---

From: Michael Welzl <[email-masked]>
Date: Thu, 18 Oct 2012 09:45:26 +0200

> congestion controller's job.

Exactly, I also totally agree. Do we then have a consensus to kick out the QoE metric?   :-)
I think that would be a good thing.

> *not* on any explicit attempt by the CC to maximize user experience.

Agreed. Note, what you describe here is an "interaction between applications and RTP flows", which is a quote from the charter, a part of a milestone due in December... is anyone working on a draft for this?

This is where "packet importance" to be handed over to the congestion control should also fit... (that was discussed as a part of the sender- vs. receiver-based thread) IF we decide to have that.

Cheers,
Michael

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 18 Oct 2012 15:32:04 +0200

On 10/18/2012 02:31 AM, Timothy B. Terriberry wrote:
> being a very crude form of changing framerate), all of which are 
> things the encoder can do to adapt to changes in available bandwidth. 
> I discussed some of this in more detail in my submission to the 
> workshop: 
Wouldn't a simple (simplistic?) way of measuring quality in the presence 
of packet drops be to always use a test rig where one end looped back 
the encoded video signal to the other end?

Thus, the one that injects the signal could compare the pre-encoding 
video source to the post-decoding video sink, which I suppose is what 
PSNR measurements need.

This is inappropriate for measurements on a live service, but might be 
appropriate for generating benchmarks.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 19 Oct 2012 00:48:47 -0400

I won't bother quoting people, but I'll say that trying to evaluate a CC 
algorithm (even one meant for media) via a video QoE test is a really 
bad idea.  It's way, way too subjective - the problem is far *harder* 
than it is for pre-recorded sequences, and there's no great agreement 
there on how to weight all the factors. (From what I hear).

It may be *interesting* to look at, but I'd never try to use it as any 
sort of definitive test.

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Fri, 19 Oct 2012 00:52:58 -0400

> Agreed. Note, what you describe here is an "interaction between applications and RTP flows", which is a quote from the charter, a part of a milestone due in December... is anyone working on a draft…

I have made a proposal in the W3 bugzilla and on the list.  There was a 
small amount of discussion; none since.  I'd suggest starting from 
there.  It may be worthwhile to re-examine the API in light of the stats 
proposals to avoid conflicting/overlapping data.

> This is where "packet importance" to be handed over to the congestion control should also fit... (that was discussed as a part of the sender- vs. receiver-based thread) IF we decide to have that.

Well, packet importance would (generally) be in the internals, not at 
the application/JS layer, except maybe to provide stream-level priorities.

-- 
Randell Jesup
[email-masked]

---

From: Michael Welzl <[email-masked]>
Date: Fri, 19 Oct 2012 09:56:17 +0200

On 19. okt. 2012, at 06:52, Randell Jesup wrote:

> I have made a proposal in the W3 bugzilla and on the list.  There was a small amount of discussion; none since.  I'd suggest starting from there.  It may be worthwhile to re-examine the API in light of the stats proposals to avoid conflicting/overlapping data.

I think I've asked you before, and I think you probably answered => so I'm sorry for being too chaotic, but could you maybe give us that link again?

> Well, packet importance would (generally) be in the internals, not at the application/JS layer, except maybe to provide stream-level priorities.

Yes, but to me, "application" in the rmcat context is not the JS application but the codec. This is at least where that "interactions" deliverable originated - to address comments like this one:

from: http://www.ietf.org/proceedings/84/minutes/minutes-84-rmcat
" - Stephan Wenger suggested that the discussions have been behind the
   state of the art in joint source-channel coding. The group should
   consider that not all packets are equal in importance when designing
   congestion control algorithms. "

Cheers,
Michael

---

From: Varun Singh <[email-masked]>
Date: Fri, 19 Oct 2012 13:26:15 +0300

On Thu, Oct 18, 2012 at 4:32 PM, Harald Alvestrand <[email-masked]> wrote:
>> If you want to measure video quality in the presence of changing bitrates,
>> loss, and delay, you have a difficult problem on your hands. Not the least
>> of which is that PSNR (or similar metrics) require a reference to compare
>> against, which starts to get meaningless once you change resolution or
>> framerate in the encoder (with dropping frames being a very crude form of
>> changing framerate), all of which are things the encoder can do to adapt to
>> changes in available bandwidth. I discussed some of this in more detail in
>> my submission to the workshop:
> Wouldn't a simple (simplistic?) way of measuring quality in the presence of
> packet drops be to always use a test rig where one end looped back the
> encoded video signal to the other end?
>
> Thus, the one that injects the signal could compare the pre-encoding video
> source to the post-decoding video sink, which I suppose is what PSNR
> measurements need.
>

I was thinking about a similar setup when writing the document.
Perhaps the Quality Metrics should not be used to compare
between congestion control algorithms but compare the
behaviour of the congestion control (all metrics not only quality)
in different evaluation scenarios.

-- 
http://www.netlab.tkk.fi/~varun/

---

From: Michael Welzl <[email-masked]>
Date: Fri, 19 Oct 2012 12:31:25 +0200

> in different evaluation scenarios.

I can't parse that. Not compare between but compare the behaviour?

Anyway, I still think that a quality metric makes no sense here...  you can't get rid of the fact that it's bound to the codec?

Cheers,
Michael

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 19 Oct 2012 15:09:31 +0200

> Anyway, I still think that a quality metric makes no sense here...  you can't get rid of the fact that it's bound to the codec?
And that makes some sense.... algorithms need to be defined without 
reference to codecs, and some of their evaluation criteria 
(slef-fairness, lack of congestion collapse) need to be evaluated 
independent of the codec, but what impact they have on media streams 
encoded with existing codecs matters for real life deployments.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 19 Oct 2012 12:16:52 -0400

On 10/19/2012 3:56 AM, Michael Welzl wrote:
>>> Agreed. Note, what you describe here is an "interaction between 
>>> applications and RTP flows", which is a quote from the charter, a 
>>> part of a milestone due in December... is anyone working on a draft 
>>> for this?
>>
>> I have made a proposal in the W3 bugzilla and on the list.  There was 
>> a small amount of discussion; none since.  I'd suggest starting from 
>> there.  It may be worthwhile to re-examine the API in light of the 
>> stats proposals to avoid conflicting/overlapping data.
>
> I think I've asked you before, and I think you probably answered => so 
> I'm sorry for being too chaotic, but could you maybe give us that link 
> again?

https://www.w3.org/Bugs/Public/show_bug.cgi?id=15861

It was also discussed on the lists (rtcweb or public-webrtc more likely)

-- 
Randell Jesup
[email-masked]

---

From: Michael Welzl <[email-masked]>
Date: Fri, 19 Oct 2012 22:23:22 +0200

On Oct 19, 2012, at 6:16 PM, Randell Jesup wrote:

>>>> part of a milestone due in December... is anyone working on a  
>>>> draft for this?
>>>
>>> I have made a proposal in the W3 bugzilla and on the list.  There  
>>> was a small amount of discussion; none since.  I'd suggest  
>>> starting from there.  It may be worthwhile to re-examine the API  
>>> in light of the stats proposals to avoid conflicting/overlapping  
>>> data.
>>
>> I think I've asked you before, and I think you probably answered =>  
>> so I'm sorry for being too chaotic, but could you maybe give us  
>> that link again?
>

Thanks!!

And: well, I took a look - that seems really unrelated to the type of  
communication needed between a congestion controller and a codec??

Cheers,
Michael