Subject: [rmcat] Where? Re: [rtcweb] Discussion on RMCAT traffic model in Vancouver (03 Nov, 1300-1500)

From: Dave Taht <[email-masked]>
Date: Sun, 3 Nov 2013 12:54:59 -0800

So did this ever get a room?

On Nov 3, 2013 12:45 PM, "Dave Taht" <[email-masked]> wrote:
>
>
> On Nov 3, 2013 9:38 AM, "Harald Alvestrand" <[email-masked]> wrote:
> >
> > On 10/29/2013 11:32 AM, Michael Welzl wrote:
> >>
> >> Okay,
> >>
> >> I'm not convinced, but I will not argue the point any longer. I raised
my concerns but maybe I'm wrong, no problem.
> >
> >
> > I think we need to do both....
> >
> > Arguing by analogy (always dangerous):
> >
> > in another debate, evaluating the performance of a codec was likened to
evaluating the performance of a car; by taking out the engine and mounting
it on a bench, you get very precise information about how much force the
engine is putting out, what its power/consumption curves are and so on.
> >
> > But if the question is "how well does the car drive in a race?",
there's no substitute for mounting the engine in the car and driving a few
laps around the track.
> >
> > I agree with Michael that when we model congestion, we have to make the
offered load fit some simple model. Otherwise, we have no idea whether a
simple tweak to the congestion control's constants was actually beneficial,
but happened to randomly interact with some property of the load.
> >
> > And I agree with Colin that in order to evaluate the whole system, we
have to run the whole system.
> >
> > One should be done, and the other not be neglected.
> > Expensive, but necessary!
>
> +100!
>
> I am hauling enough gear along to be able to evaluate the main aqm
technologies we're fiddling with against some webrtc loads...
>
> I tend to argue that experiments tend to uncover variables you didn't
even know existed.
>
> >
> >
> >
> >>
> >> Cheers,
> >> Michael
> >>
> >>
> >> On 29. okt. 2013, at 10:31, Colin Perkins wrote:
> >>
> >>> Michael,
> >>>
> >>> Your concern is why I think we need to test the codec and the
congestion control together. The dynamics of how, and importantly when, the
codec can change its rate will be critical to the dynamics of the
congestion control. This is especially true if we want to keep the latency
low by avoiding sender-side buffering to smooth the codec output to match
the desired sending rate. Testing congestion control without a codec will
give us something that doesn't respect the QoE concerns, and runs the risk
of producing an algorithm that is safe and responsive, but not useful for
interactive video.
> >>>
> >>> Colin
> >>>
> >>>
> >>> On 29 Oct 2013, at 08:30, Michael Welzl <[email-masked]> wrote:
> >>>>
> >>>> So - looking at this thread, it seems that consensus is against me
 :-)
> >>>>
> >>>> Hence I'd like to clarify - my concern was that we're making the
system too complex by involving a codec that reacts to the congestion
controller **when evaluating congestion control mechanisms**. I mean, once
I've done a test, I'd like to know why I see the effects that I see, and
which mechanism is to blame for them. This will get harder with this setup.
> >>>>
> >>>> Let me give an example to explain my concern:
> >>>>
> >>>> Say, we create a test with a reactive codec over a congestion
controller, and in some network scenarios, we see oscillations: the
congestion controller increased its rate, allowing the codec to switch to a
higher encoding rate; this higher rate was too much, and so quite fast the
congestion controller reduces its rate again, hence the codec reacts and
switches to a lower rate, and so forth.
> >>>>
> >>>> Now, if that is the test setup for evaluating a congestion
controller, a conclusion from such a test could be that the congestion
controller has to be stabilized. I would, however, argue that a
congestion controller should be as quick as it possibly can at seeking out
available bandwidth, and hence such fluctuations are natural (*), and it's
rather the job of the codec to dampen its rate switching.
> >>>>
> >>>> Either way, in this example test, we face a bad result, and two
screws to tune to fix the problem - one on the side of the congestion
control, and one on the side of the codec.
> >>>>
> >>>> This is why I would prefer to evaluate congestion controls (again,
not the whole system, to me that's a different evaluation, and of course a
better congestion control should lead to a better total system) with a
static traffic profile.
> >>>>
> >>>> Is my concern unfounded? Is it a non-problem I'm envisioning?   (if
so, see below for 2 more concerns  :-)   )
> >>>>
> >>>> Cheers,
> >>>> Michael
> >>>>
> >>>> PS:  Concern #2: we can expect a static traffic profile feeding into
a congestion controller to be relatively quickly done, but creating a test
setup with a reactive codec is going to be much harder and take more time.
Will we ever see it? When? Will it be easy to use?
> >>>> PPS: Concern #3: the parameter space for tests might expand quite
significantly when we involve reactive codecs... this will make any
evaluation harder, making the results harder to compare, resulting in more
time necessary to come to a conclusion about which controller does a better
job.
> >>>>
> >>>>
> >>>> (*) TFRC is an argument to the contrary, but I haven't seen a single
evaluation result that shows that its smooth rate is indeed beneficial. On
the contrary, we now seem to have some agreement that TFRC isn't the best
mechanism.
> >>>>
> >>>>
> >>>> On 29. okt. 2013, at 01:53, Mo Zanaty (mzanaty) wrote:
> >>>>
> >>>>> Agreed. The base layer has no flexibility just like a flat
non-scalable stream.
> >>>>>
> >>>>>
> >>>>> On 10/28/13, 7:45 PM, Bernard Aboba <[email-masked]>
wrote:
> >>>>>
> >>>>> You're right that there is flexibility in enhancement layers. But
because of the layering structure, the base is not flexible - since the
enhancement layers depend on it, it is typically protected via FEC or RTX.
This affects the viability of "slow start" schemes as well other aspects -
no point sending video at all if you cannot support the base layer, at
least.
> >>>>>
> >>>>> On Oct 28, 2013, at 3:10 PM, "Mo Zanaty (mzanaty)" <
[email-masked]> wrote:
> >>>>>
> >>>>>> This is true for simulcast layers, but not necessarily scalable
layers. For the latter, arbitrary rates can be achieved within a limited
range (like down to half the bitrate) by selectively dropping some packets
of the highest enhancement layer with minimal visual quality impact. If the
stream is highly scalable (many layers, which is rare), the limited range
expands (like down to maybe a third, quarter, or less of the bitrate). If
the stream is coarsely scalable (e.g. two layers, which is common),
dropping below the limited range incurs changing encoding parameters, at
which point arbitrary rates are again possible.
> >>>>>>
> >>>>>> Mo
> >>>>>>
> >>>>>>
> >>>>>> On 10/28/13, 5:58 PM, Bernard Aboba <[email-masked]>
wrote:
> >>>>>>
> >>>>>> Colin said:
> >>>>>>
> >>>>>> “Do you mean preparing the same video sequence at multiple rates
and the
> >>>>>> congestion controller picking the appropriate media rate that fits
its
> >>>>>> estimate? ie rate switch between different media rates of the same
> >>>>>> sequence?
> >>>>>> ”
> >>>>>>
> >>>>>> [BA] I like this approach, because it somewhat approximates the
situation of simulcast/layered coding.  Previous work has indicated a
number of considerations that arise in those implementations (such as the
need for rapid rampup, so as to be able to send at least one video stream
as well as issues with TFRFC) that would be likely to be missed in a model
in which an encoder has fully smooth rate control.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I agree with Colin. We should be focused on live encoding where the
> >>>>>> encoder rate control is highly granular (often fully smooth) and
coupled
> >>>>>> to the network congestion control, not pre-encoded streaming of a
small
> >>>>>> set of fixed bitrates like DASH/HLS.
> >>>>>>
> >>>>>> A more formal approach to the problem would be to decompose the
factors
> >>>>>> contributing to the final rate placed onto the network.
> >>>>>> 1. Video content / source statistics, using a large set of
representative
> >>>>>> sequences. An estimate of the entropy per frame could simply be the
> >>>>>> encoded frame size using constant quantization, i.e. no encoder
rate
> >>>>>> control.
> >>>>>> 2. Video encoder rate control behavior, which modulates the source
> >>>>>> statistics to fit into network constraints over rather large time
> >>>>>> intervals (>0.5s).
> >>>>>> 3. Network transmission pacing / shaper, which smooths out bursts
in
> >>>>>> encoder output to fit into network constraints over fine time
intervals
> >>>>>> (<0.5s).
> >>>>>>
> >>>>>> It should be possible to model all three in an abstract way to
arrive at
> >>>>>> some statistical model that can then be used for all tests without
any
> >>>>>> real video content or encoding (to avoid introducing noise in
comparisons
> >>>>>> if there was any variability in any of the factors).
> >>>>>>
> >>>>>> Mo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 10/28/13, 1:49 PM, Colin Perkins <[email-masked]> wrote:
> >>>>>>
> >>>>>> On 27 Oct 2013, at 11:50, Michael Welzl <[email-masked]> wrote:
> >>>>>>
> >>>>>> On 27. okt. 2013, at 12:33, Varun Singh <[email-masked]>
wrote:
> >>>>>>>
> >>>>>>> Now, next, what if we'd have a set of such sequences, and then
stack
> >>>>>>> them over a codec that produces quite arbitrary bitrates,
assuming that
> >>>>>>> the network can always follow it -
> >>>>>>
> >>>>>>
> >>>>>> Do you mean preparing the same video sequence at multiple rates
and the
> >>>>>> congestion controller picking the appropriate media rate that fits
its
> >>>>>> estimate? ie rate switch between different media rates of the same
> >>>>>> sequence?
> >>>>>>
> >>>>>> I think that could be a very good idea!
> >>>>>>
> >>>>>>
> >>>>>> I'd be more interested in a standard set of *uncompressed* video
test
> >>>>>> sequences, that we can use to test the combination of video codec
and
> >>>>>> congestion control. If we're building something for interactive
video
> >>>>>> congestion control we should assume live encoding, with the codec
> >>>>>> potentially having the flexibility to react to the network, and our
> >>>>>> testbed ought to reflect that.
> >>>>>>
> >>>>>> --
> >>>>>> Colin Perkins
> >>>>>> http://csperkins.org/
> >>>
> >>>
> >>
> >
> >
> > --
> > Surveillance is pervasive. Go Dark.