Subject: [aqm] FW: Objective Video Quality Assessment

From: "Robinson, Dave C (Dave)" <[email-masked]>
Date: Fri, 7 Mar 2014 10:22:52 +0000

All,

During the AQM WG session on Tuesday I mentioned the need to have an objective measure of the QoE for a video session when testing AQMs.  A couple of years back internally we were experimenting with scheduling algorithms in LTE eNodeB (base station) and their impact on HTTP Adaptive Streaming (HAS) content. The only sane way to compare two simulation runs was to have an objective way to measure the QoE.

There are many things which impact QoE. However, there is a simplification we can make with HAS. With HAS, content is encoded in multiple video quality levels (say 6). The content is then divided into short segments or chunks (2-10 seconds depending on the HAS variant in use). The client pulls chunks using HTTP GET. By sensing the available bandwidth, it selects the best video quality level for the chunk to maximize QoE. To do this, it knows the mean bit rate for each video quality level. As scene complexity is not constant, there will be some variation in bit rate but the bit rates in each video quality level are typically disjoint.

A useful assumption is that the encoder parameters should be set so any one video quality gives constant MOS throughout the duration. This means that only knowing the video quality level, you know the MOS (i.e. QoE) a viewer would report if they actually watched the stream. Second assumption is that the increments in MOS between video quality levels should be constant. This means we do not need batteries of viewers to score the quality of any run.

First, a stream SHALL NOT UNDERFLOW. The QoE of streams which rebuffer is known to be bad and should be avoided. Provided the adaptive stream can maintain at least the lowest Video Quality level, this is preferable.

Second, knowing only the Video Quality level requested in HTTP GETs, its possible to get a very close approximation to the MOS. In a paper given at IEEE/IFIP conference in 2013 http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6573179&tag=1  we described a simple linear relationship of the mean and standard deviation of the Video Quality level to predict the MOS. This work was part of an experiment of the Next Generation Mobile Network (NGMN) consortium looking for KPIs to assess the quality of video streamed over LTE networks. From other internal work, we are confident the principle also applies to HAS delivered over broadband networks as well.

FYI in 3G and LTE operator voice ( VoIP and VoLTE) is send on a separate queue so is not impacted by bulky data transfers including web page downloads, HTTP streams and FTP transfers. And the low latency targets needed for voice communications can be met.

I'm happy to discuss this further on the AQM list or off as the chair decides.

Regards
   Dave

Dr Dave Robinson
IP Rouitng and Transport
Alcatel-Lucent
+44 7801 878952(m)

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 8 Mar 2014 11:45:04 +0100 (CET)

On Fri, 7 Mar 2014, Robinson, Dave C (Dave) wrote:

> Second, knowing only the Video Quality level requested in HTTP GETs, its 
> possible to get a very close approximation to the MOS. In a paper given

How do you know this?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Robinson, Dave C (Dave)" <[email-masked]>
Date: Sat, 8 Mar 2014 21:05:56 +0000

It's in the URL of the HTTP GET request.

-----Original Message-----
From: Mikael Abrahamsson [mailto:[email-masked]] 
Sent: 08 March 2014 10:45
To: Robinson, Dave C (Dave)
Cc: [email-masked]
Subject: Re: [aqm] FW: Objective Video Quality Assessment

On Fri, 7 Mar 2014, Robinson, Dave C (Dave) wrote:

> Second, knowing only the Video Quality level requested in HTTP GETs, 
> its possible to get a very close approximation to the MOS. In a paper 
> given

How do you know this?

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 8 Mar 2014 22:35:26 +0100 (CET)

> It's in the URL of the HTTP GET request.

So when (not if) most traffic is HTTPS (TLS), this doesn't work at all, 
right? Because what you're talking about requires middle boxes to do DPI.

---

From: "Robinson, Dave C (Dave)" <[email-masked]>
Date: Sun, 9 Mar 2014 17:11:22 +0000

I raised the matter for predicting MOS of a HAS stream in the context of testing AQMs. I'd assume that our labs / simulations would not need to used TLS. And even if end-to-end encryption was included, we'd be able to read the ACCESS LOGs of the web servers. 

-----Original Message-----
From: Mikael Abrahamsson [mailto:[email-masked]] 
Sent: 08 March 2014 21:35
To: Robinson, Dave C (Dave)
Cc: [email-masked]
Subject: RE: [aqm] FW: Objective Video Quality Assessment

> It's in the URL of the HTTP GET request.

So when (not if) most traffic is HTTPS (TLS), this doesn't work at all, right? Because what you're talking about requires middle boxes to do DPI.