Subject: [rmcat] Discussion on RMCAT traffic model in Vancouver (03 Nov, 1300-1500)

From: Varun Singh <[email-masked]>
Date: Sun, 27 Oct 2013 12:57:13 +0200

Hi all,

In RMCAT WG there has been discussion surrounding the use
of a media traffic generator and to facilitate discussion,
RMCAT is hosting a side meeting on Sunday, 03 November
at 1300-1500. I suppose the room details will be announced
shortly.

Below is a list of questions that will help design or choose
an appropriate traffic generator for Interactive multimedia traffic.
The list is an initial set and mainly to foster discussion both
on the mailing list and at the meeting.

Mirja and Lars are not available that day, hence on their behalf,
the meeting is hosted by Colin and I .

Cheers,
Varun
---

Traffic generator: to model the intrinsic variability of video:

- what target bit rates can be achieved? (max?, min?)
- how much variability in media bit rate for a given target bit rate
- how to model motion? capturing high motion leads to bursty video
  (higher than target bit rate), but by how much? can this even be done?

Traffic generator: on responsiveness to a new target bitrate:

- how quickly can the codec generate a lower bit rate?
- immediately? does this depend on the new or requested bit rate?
- what is immediately? is it next frame or a few frames later, until then
  does it generates at the current rate?
- if not immediately, what bitrates (and duration) will it generate before
  meeting the target bitrate.

- how quickly can the codec generate a higher bit rate?
- immediately in the next frame? does this also depend on the new
   or requested bit rate?
- not immediately but in steps of intermediate bitrates, i.e., intermediate
  bitrate for a short duration then an increase in step, repeating until
  reaching the target bit rate.

- If the target bit rate is achieved in steps, can they be announced to the
  congestion controller, i.e., the congestion controller knows the ramp up
  curve to meet the target bit rate.

- If the media bit rate has not yet reached the requested target bit rate and
  the congestion control requests a new target bit rate, how does the codec
  respond?

Application design related:
- should the congestion control be able to change the frame rate (video) or
  packetization time (audio) or is this something that the application controls.
- should the congestion control be able to change video resolution?
- should error-repair: NACKs, FEC be considered part of congestion control or
  outside it, i.e., it is something that the application does and not
  congestion control
- If FEC is controlled by the congestion control, how quickly can the sending
  rate be adapted by reducing/adding redundancy (FEC)? is something link this
  done? Does this makes sense at all?
- Apart from loss and RTT, what other congestion cues are currently used by
  congestion control algorithms? e.g., Decoding rate/goodput, application
  decode error rate, ECN, PCN, etc.
- If the codec is data limited, i.e., producing a sending rate smaller than the
  one calculated by the congestion control, can the congestion control probe
  the bandwidth by sending additional data packets in a certain pattern (e.g.,
  PathChirp)? would somehting link this help?
- application input on prioritization: audio vs video (vs data)

-- 
http://www.netlab.tkk.fi/~varun/

---

From: Michael Welzl <[email-masked]>
Date: Sun, 27 Oct 2013 12:08:26 +0100

[removing the other lists except rmcat]

Great initiative, count me in!  (sorry for saying this only so late)

Re: discussion on the mailing list, here's a question I have, since you mention "how to model motion". Does a list of standard test sequences (like these http://media.xiph.org/video/derf/ ) for interactive use exist? If not, maybe a subset of those could be picked (I'd imagine that foreman is rather representative for interactive use, but many others are not  :)  ).

Now, next, what if we'd have a set of such sequences, and then stack them over a codec that produces quite arbitrary bitrates, assuming that the network can always follow it - as a sort of extreme case test? Then we'd have standard test sequences of bit rates that could be used to check how good a particular cc. is at following along. I'm not saying that "follow along" is all a controller's supposed to do, indeed a codec can react to feedback from the network, but couldn't this be a useful way of separating out at least one side of the story and having something to test?

I also wonder about the related work in this regard... there must be tons of papers out there on this problem?

Cheers,
Michael

On 27. okt. 2013, at 11:57, Varun Singh <[email-masked]> wrote:

> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Varun Singh <[email-masked]>
Date: Sun, 27 Oct 2013 13:25:04 +0200

> Re: discussion on the mailing list, here's a question I have, since you mention "how to model motion". Does a list of standard test sequences (like these http://media.xiph.org/video/derf/ ) for int…

There is also the "news" sequence which is quite popular for mimicking
heads and shoulders but it has a pirouetting ballerina in the
background.

---

From: Michael Welzl <[email-masked]>
Date: Sun, 27 Oct 2013 12:33:07 +0100

> heads and shoulders but it has a pirouetting ballerina in the

Indeed I think there are several that could quite easily be picked from e.g.
http://media.xiph.org/video/derf/
as being representative for interactive use.

e.g. also carphone...  a good handful of fitting ones I'd say.

---

From: Varun Singh <[email-masked]>
Date: Sun, 27 Oct 2013 13:33:37 +0200

Hi Michael,

> Now, next, what if we'd have a set of such sequences, and then stack them over a codec that produces quite arbitrary bitrates, assuming that the network can always follow it -

Do you mean preparing the same video sequence at multiple rates and
the congestion controller picking the appropriate media rate that fits
its estimate? ie rate switch between different media rates of the same
sequence?

> as a sort of extreme case test? Then we'd have standard test sequences of bit rates that could be used to check how good a particular cc. is at following along. I'm not saying that "follow along" is all a controller's supposed to do, indeed a codec can react to feedback from the network, but couldn't this be a useful way of separating out at least one side of the story and having something to test?
>

---

From: Varun Singh <[email-masked]>
Date: Sun, 27 Oct 2013 13:35:26 +0200

> On 27.10.2013, at 13.33, Michael Welzl <[email-masked]> wrote:
>
>

I'm all for picking a handful of sequences. :)

We used foreman and news for our circuit breaker,  rrtcc and FEC paper.

---

From: Michael Welzl <[email-masked]>
Date: Sun, 27 Oct 2013 12:50:58 +0100

> its estimate? ie rate switch between different media rates of the same

I think that could be a very good idea!

I was actually thinking of the opposite, but I'm not sure if that makes sense: by their nature, if you assume no congestion control and just send data following these videos with a fixed frame rate, such that the data has to arrive at the receiver in time, the videos will translate into rate fluctuations, and an "ideal" congestion controller dealing with a non-reactive codec would then follow along with these videos.

Or are these fluctuations too small to play a role, and changing the encoding rate of these videos is the only real effect that you would really "feel" on the network side?

In this case, my proposal doesn't make sense I guess, but yours certainly does?

Cheers,
Michael

---

From: Varun Singh <[email-masked]>
Date: Sun, 27 Oct 2013 18:06:38 +0200

Hi Michael,

comments inline.

Cheers,
Varun

> I was actually thinking of the opposite, but I'm not sure if that makes sense: by their nature, if you assume no congestion control and just send data following these videos with a fixed frame rate…

Would require that the sending buffer would have to pace out packets
(or drop them) based on the congestion controller estimate?

> In this case, my proposal doesn't make sense I guess, but yours certainly does?

I am not particularly fond of the rate-switching idea, because it depends on the
number of target rates per sequence, i.e., if too few rates are used, switching
would be coarse grained leading to too slow ramp-up (or sometimes not
even possible), the ramp-downs may undershoot by too much. The alternative
would be to have sequences at several intermediate rates.

The other issue is, if we would like to playback the received video
(to observe)
the rate-switching between the different sequences would have to occur
at particular
instances (e.g., at key frame boundaries?).

These techniques are used for in video streaming, therefore, if we go ahead
with such a technique, it should be possible to carefully select a set
of target rates
for evaluating the RMCAT congestion control mechanisms.

---

From: Michael Welzl <[email-masked]>
Date: Sun, 27 Oct 2013 17:30:47 +0100

> (or drop them) based on the congestion controller estimate?

I was not considering a congestion controller at all, but imagining that packets are sent out from the sender buffer at a rate of FS / T, where FS = the size of the next frame in the buffer, and T =  a fixed time interval over which the frame is sent (some fixed parameter).

Wouldn't this give us the rates at which a congestion controller would have to drain the send buffer, assuming:
- a codec that doesn't react to anything
- that every frame arrives on the receiving end within a certain fixed time interval  (of course assuming a perfect network)

Maybe there's a flaw somewhere here, but that was my thinking... to produce some kind of workload that can be used to test congestion controllers.

> for evaluating the RMCAT congestion control mechanisms.

Argh, yes, too many problems on the table.

Maybe we have to separate two things here. If I consider a non-reactive application that produces data at rates which fluctuate, like I'm imagining, then I can create some sort of "worst case" scenario for a congestion controller from this, and see if my congestion controller deals well in the presence of such workload.

As far as I'm concerned, as a congestion control designer, once I have investigated how my control operates:
1) with a bulk data transfer (i.e. always-full buffer)
2) with occasionally stopping traffic (i.e. buffer running empty)
3) with a buffer that is filled at varying speeds, and that my controller is supposed to drain at varying speeds
... then my work is done?!

I mean, then we do have a good idea of how good the control is. What we don't know is if it would be a better system design to give the controller a workload of type 1 above and let the codec react to make that happen, or a workload of type 3 above and not react, or exactly where in between we should place ourselves. This has a big effect on the quality experienced by the user, of course - but I think this decision will probably be a function of how well a congestion control deals with cases 1, 2, 3 above (plus other things, but regarding the congestion control, this is probably all we need to know?!). So maybe this is where the line between "congestion control design" and "overall system design" goes?

If so, maybe we don't really need to consider the case of a codec that reacts to the congestion control when evaluating a congestion controller at all. I guess so?? Or am I being really stupid right now?

If I'm not, this separation could perhaps simplify things a great deal...

Cheers,
Michael

---

From: Varun Singh <[email-masked]>
Date: Sun, 27 Oct 2013 19:46:09 +0200

> On 27. okt. 2013, at 17:06, Varun Singh <[email-masked]> wrote:
[snip!]
> - that every frame arrives on the receiving end within a certain fixed time interval  (of course assuming a perfect network)

Agree, a non-reactive codec will produce one of the two behaviours:

If the FS/T is less than or matches the available end-to-end capacity,
it should arrive in link latency time.
If FS/T is more than the end-to-end capacity, the network will
add queuing delay.

> Maybe there's a flaw somewhere here, but that was my thinking... to produce some kind of workload that can be used to test congestion controllers.
i.e., model the workload such that at some instances it exceeds the
capacity and some other instance it doesn't and observe how the
sending buffer (controlled by the cc) handles this situation?

> 1) with a bulk data transfer (i.e. always-full buffer)

but the size of the buffer would vary depending on the rate generated
by the codec,
I am not sure if for a variable sized buffer we can say it is full or
not. Perhaps
the controller can only say if it can drain it as fast as it is filling up.

If we have a fixed size buffer, what happens when it overflows? did
the controller fail?

> 3) with a buffer that is filled at varying speeds, and that my controller is supposed to drain at varying speeds

Agree...

> I mean, then we do have a good idea of how good the control is. What we don't know is if it would be a better system design to give the controller a workload of type 1 above and let the codec react…

This is a good discussion to have :)

> If so, maybe we don't really need to consider the case of a codec that reacts to the congestion control when evaluating a congestion controller at all. I guess so?? Or am I being really stupid righ…

We really should discuss this in the context of  **evaluating** RMCAT
congestion control, while in the real-world we'd expect some coupling
between the codec and the congestion control.

Cheers,
Varun

---

From: Dave Taht <[email-masked]>
Date: Sun, 27 Oct 2013 10:54:58 -0700

Given that this is mostly a peer to peer technology, would it be
possible to just setup a bunch of laptops and fiddle?

I'll bring a router.

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: Michael Welzl <[email-masked]>
Date: Sun, 27 Oct 2013 19:49:16 +0100

> sending buffer (controlled by the cc) handles this situation?

The way I envision it there is no link capacity in this model, which would make it more generally applicable. The link capacity is unlimited, you don't care about the link, you just send with the rate the codec tells you, and so you simply get a sequence of application-desired rates.

Add a link with a given capacity to this, and you get what you describe - but just having a set of rates would be more general I think.

> the controller can only say if it can drain it as fast as it is filling up.

The only constraint of  concern for item #1 is that whenever the controller wants to send, there is something available.

> If we have a fixed size buffer, what happens when it overflows? did

No, it that happens then the application failed to react to the congestion controller's speed of draining the buffer, and again it's a larger system problem and we're not really evaluating a congestion controller anymore.

> between the codec and the congestion control.

Yes, I'm only trying to separate these two here for simplicity. I think we're making the "evaluate congestion control" bit too hard for ourselves by considering interactions with applications beyond 1) and 2) and 3) above.

Cheers,
Michael

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 28 Oct 2013 12:31:56 +0100

Hi Varun,

just one comment on one point only:

[snip]

> I am not particularly fond of the rate-switching idea, because it depends
> on the number of target rates per sequence, i.e., if too few rates are
> used, switching would be coarse grained leading to too slow ramp-up (or
> sometimes not even possible), the ramp-downs may undershoot by too much.
> The alternative would be to have sequences at several intermediate rates.
What do you me by 'too slow ramp-up'? Of the congestion control? Again my 
proposal that the transport should potentially probe for available rate and 
by this maybe even be able to 'jump' to a higher rate (potentially not to the 
maximum measure available, but safely to something below that).

Mirja

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Varun Singh <[email-masked]>
Date: Mon, 28 Oct 2013 13:52:07 +0200

Hi Mirja,

> What do you me by 'too slow ramp-up'? Of the congestion control? Again my

Clarifying what I meant:
If the congestion control recommends a new estimate for which there is no
media file, the system has to approximate what to do, either pick the file
with a higher rate or nearest lower rate.

For example:
media files with 200, 500, 1000, 1500, 2000 Kbps exist
and if the the estimate by the controller is 800 kbps, it can either pick
500 or 1000 kbps. if it picks 500 kbps, it is going to ramp-up more slowly
compared to a set of sequences which perhaps has an additional
media file for a target bit rate of 750 kbps.

In the above example, I arbitrarily picked a set of 5 target rates, if
I'd picked
3 or 7 the evaluation would be different. Picking the right number of
sequences and/or target rate would affect how quickly the media flow
ramps up and down.

> maximum measure available, but safely to something below that).

Agree, a congestion controller should be capable of probing when the
application is data limited (i.e., media rate is lower than estimate).
I suppose this could be done, as long as the probing does not affect
the timeliness of the media stream?

-Varun

---

From: Colin Perkins <[email-masked]>
Date: Mon, 28 Oct 2013 17:49:52 +0000

On 27 Oct 2013, at 11:50, Michael Welzl <[email-masked]> wrote:
>> Do you mean preparing the same video sequence at multiple rates and the congestion controller picking the appropriate media rate that fits its estimate? ie rate switch between different media rates of the same sequence?
> 

I'd be more interested in a standard set of *uncompressed* video test sequences, that we can use to test the combination of video codec and congestion control. If we're building something for interactive video congestion control we should assume live encoding, with the codec potentially having the flexibility to react to the network, and our testbed ought to reflect that.

-- 
Colin Perkins
http://csperkins.org/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 28 Oct 2013 18:39:05 +0000

I agree with Colin. We should be focused on live encoding where the
encoder rate control is highly granular (often fully smooth) and coupled
to the network congestion control, not pre-encoded streaming of a small
set of fixed bitrates like DASH/HLS.

A more formal approach to the problem would be to decompose the factors
contributing to the final rate placed onto the network.
1. Video content / source statistics, using a large set of representative
sequences. An estimate of the entropy per frame could simply be the
encoded frame size using constant quantization, i.e. no encoder rate
control.
2. Video encoder rate control behavior, which modulates the source
statistics to fit into network constraints over rather large time
intervals (>0.5s).
3. Network transmission pacing / shaper, which smooths out bursts in
encoder output to fit into network constraints over fine time intervals
(<0.5s).

It should be possible to model all three in an abstract way to arrive at
some statistical model that can then be used for all tests without any
real video content or encoding (to avoid introducing noise in comparisons
if there was any variability in any of the factors).

Mo

On 10/28/13, 1:49 PM, Colin Perkins <[email-masked]> wrote:

On 27 Oct 2013, at 11:50, Michael Welzl <[email-masked]> wrote:
>>> Now, next, what if we'd have a set of such sequences, and then stack
>>>them over a codec that produces quite arbitrary bitrates, assuming that
>>>the network can always follow it -
>> 
>> Do you mean preparing the same video sequence at multiple rates and the
>>congestion controller picking the appropriate media rate that fits its
>>estimate? ie rate switch between different media rates of the same
>>sequence?
> 

I'd be more interested in a standard set of *uncompressed* video test
sequences, that we can use to test the combination of video codec and
congestion control. If we're building something for interactive video
congestion control we should assume live encoding, with the codec
potentially having the flexibility to react to the network, and our
testbed ought to reflect that.

-- 
Colin Perkins
http://csperkins.org/

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Mon, 28 Oct 2013 20:15:07 +0000

Agree with Colin and Mo.

For a statistical video traffic model, I would propose to break down the 2nd point (encoder rate control) in Mo's list further, to three sub-components:

a) delayed response to a rate change request (e.g., > 0.5s)
b) deviations from the requested rate (maybe as randomly fluctuating around the target?)
c) the different frame types or coding modes inside the encoder will yield some encoded
frames much larger than others. This will lead to burstiness on time scale of frame intervals
(typically 33ms), and later smoothed out by the pacing/shaper.

Also, if we go for the approach of using uncompressed test video sequence for testing the combination of video codec and congestion control, we should also recommend a reference codec and default configurations. So that focus of the evaluations remains on the congestion control module.  What do folks think of x264 (http://www.videolan.org/developers/x264.html)?

Best,
Xiaoqing

On Oct 28, 2013, at 1:39 PM, Mo Zanaty (mzanaty) <[email-masked]<mailto:[email-masked]>> wrote:

Mo

On 10/28/13, 1:49 PM, Colin Perkins <[email-masked]<mailto:[email-masked]>> wrote:

On 27 Oct 2013, at 11:50, Michael Welzl <[email-masked]<mailto:[email-masked]>> wrote:
On 27. okt. 2013, at 12:33, Varun Singh <[email-masked]<mailto:[email-masked]>> wrote:
Now, next, what if we'd have a set of such sequences, and then stack
them over a codec that produces quite arbitrary bitrates, assuming that
the network can always follow it -

--
Colin Perkins
http://csperkins.org/

---

From: "Van Der Auwera, Geert" <[email-masked]>
Date: Mon, 28 Oct 2013 20:43:47 +0000

Also agree. For initial evaluations of CC proposals and to find consensus quickly, a statistical video traffic model should be sufficient. However, proponents should be allowed and perhaps encouraged from producing additional results with a complete live video encoding testbed of their choice (video quality?). Agreeing on a particular codec at this point may be a bit hard. Regarding the uncompressed test video sequences (VT type), Qualcomm could provide a set of long sequences (> 1min).

Best regards,

Geert

From: [email-masked] [mailto:[email-masked]] On Behalf Of Xiaoqing Zhu (xiaoqzhu)
Sent: Monday, October 28, 2013 1:15 PM
To: Mo Zanaty (mzanaty)
Cc: rmcat WG; Colin Perkins; Michael Welzl; Varun Singh
Subject: Re: [rmcat] [rtcweb] Discussion on RMCAT traffic model in Vancouver (03 Nov, 1300-1500)

Agree with Colin and Mo.

Best,
Xiaoqing

Mo

--
Colin Perkins
http://csperkins.org/

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 28 Oct 2013 14:58:32 -0700

Colin said: 

"Do you mean preparing the same video sequence at multiple rates and the
congestion controller picking the appropriate media rate that fits its
estimate? ie rate switch between different media rates of the same
sequence?

"

[BA] I like this approach, because it somewhat approximates the situation of
simulcast/layered coding.  Previous work has indicated a number of
considerations that arise in those implementations (such as the need for
rapid rampup, so as to be able to send at least one video stream as well as
issues with TFRFC) that would be likely to be missed in a model in which an
encoder has fully smooth rate control.   

Mo

On 10/28/13, 1:49 PM, Colin Perkins <[email-masked]
<mailto:[email-masked]> > wrote:

On 27 Oct 2013, at 11:50, Michael Welzl <[email-masked]
<mailto:[email-masked]> > wrote:

On 27. okt. 2013, at 12:33, Varun Singh <[email-masked]
<mailto:[email-masked]> > wrote:

-- 
Colin Perkins
http://csperkins.org/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 28 Oct 2013 22:10:31 +0000

This is true for simulcast layers, but not necessarily scalable layers. For the latter, arbitrary rates can be achieved within a limited range (like down to half the bitrate) by selectively dropping some packets of the highest enhancement layer with minimal visual quality impact. If the stream is highly scalable (many layers, which is rare), the limited range expands (like down to maybe a third, quarter, or less of the bitrate). If the stream is coarsely scalable (e.g. two layers, which is common), dropping below the limited range incurs changing encoding parameters, at which point arbitrary rates are again possible.

Mo

On 10/28/13, 5:58 PM, Bernard Aboba <[email-masked]<mailto:[email-masked]>> wrote:

Colin said:

“Do you mean preparing the same video sequence at multiple rates and the
congestion controller picking the appropriate media rate that fits its
estimate? ie rate switch between different media rates of the same
sequence?
”

[BA] I like this approach, because it somewhat approximates the situation of simulcast/layered coding.  Previous work has indicated a number of considerations that arise in those implementations (such as the need for rapid rampup, so as to be able to send at least one video stream as well as issues with TFRFC) that would be likely to be missed in a model in which an encoder has fully smooth rate control.

Mo

--
Colin Perkins
http://csperkins.org/

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 28 Oct 2013 16:45:21 -0700

You're right that there is flexibility in enhancement layers. But because of the layering structure, the base is not flexible - since the enhancement layers depend on it, it is typically protected via FEC or RTX. This affects the viability of "slow start" schemes as well other aspects - no point sending video at all if you cannot support the base layer, at least.

> On Oct 28, 2013, at 3:10 PM, "Mo Zanaty (mzanaty)" <[email-masked]> wrote:
> 
> On 10/28/13, 5:58 PM, Bernard Aboba <[email-masked]> wrote:
> 
> Colin said:
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 29 Oct 2013 00:53:52 +0000

Agreed. The base layer has no flexibility just like a flat non-scalable stream.

On 10/28/13, 7:45 PM, Bernard Aboba <[email-masked]<mailto:[email-masked]>> wrote:

On Oct 28, 2013, at 3:10 PM, "Mo Zanaty (mzanaty)" <[email-masked]<mailto:[email-masked]>> wrote:

Mo

Colin said:

Mo

--
Colin Perkins
http://csperkins.org/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 29 Oct 2013 02:27:09 +0000

I’m glad you removed rtcweb before recommending a reference codec!
Seriously, we should wait on the rtcweb codec decision next week.
And in the (likely) case of no decision, use both vp8 and x264.

Mo

On 10/28/13, 4:15 PM, Xiaoqing Zhu (xiaoqzhu) <[email-masked]<mailto:[email-masked]>> wrote:

Agree with Colin and Mo.

Best,
Xiaoqing

Mo

--
Colin Perkins
http://csperkins.org/

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 28 Oct 2013 21:03:32 -0700

Heh, good point about removing RTCWEB.  No need to start a riot :)
But now that we've removed them, we can speak frankly... the VP8 vs. H.264 battle is "so 2012".   The industry has moved on to VP9 and H.265, both of which are likely to be deployed in mixed spatial simulcast/temporal SVC modes. 
So while it's certainly convenient to experiment with VP8 and H.264 because the tools are available... we should be thinking about the future because, as they say in Plan 9 "that is where we expect to spend the rest of our lives." 

From: [email-masked]
To: [email-masked]
Date: Tue, 29 Oct 2013 02:27:09 +0000
CC: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [rmcat] [rtcweb] Discussion on RMCAT traffic model in Vancouver (03 Nov, 1300-1500)

Mo

On 10/28/13, 4:15 PM, Xiaoqing Zhu (xiaoqzhu) <[email-masked]> wrote:

Agree with Colin and Mo. 

Also, if we go for the approach of using uncompressed test video sequence for testing the combination of video codec and congestion control, we should also recommend a reference codec and default configurations. So that focus of the evaluations remains
 on the congestion control module.  What do folks think of x264 (http://www.videolan.org/developers/x264.html)? 

Best,
Xiaoqing

On Oct 28, 2013, at 1:39 PM, Mo Zanaty (mzanaty) <[email-masked]> wrote:

control.

intervals (>0.5s).

(<0.5s).

Mo

sequence?

-- 

Colin Perkins

http://csperkins.org/

---

From: Michael Welzl <[email-masked]>
Date: Tue, 29 Oct 2013 09:30:38 +0100

So - looking at this thread, it seems that consensus is against me  :-)

Hence I'd like to clarify - my concern was that we're making the system too complex by involving a codec that reacts to the congestion controller **when evaluating congestion control mechanisms**. I mean, once I've done a test, I'd like to know why I see the effects that I see, and which mechanism is to blame for them. This will get harder with this setup.

Let me give an example to explain my concern:

Say, we create a test with a reactive codec over a congestion controller, and in some network scenarios, we see oscillations: the congestion controller increased its rate, allowing the codec to switch to a higher encoding rate; this higher rate was too much, and so quite fast the congestion controller reduces its rate again, hence the codec reacts and switches to a lower rate, and so forth.

Now, if that is the test setup for evaluating a congestion controller, a conclusion from such a test could be that the congestion controller has to be stabilized. I would, however, argue that a congestion controller should be as quick as it possibly can at seeking out available bandwidth, and hence such fluctuations are natural (*), and it's rather the job of the codec to dampen its rate switching.

Either way, in this example test, we face a bad result, and two screws to tune to fix the problem - one on the side of the congestion control, and one on the side of the codec.

This is why I would prefer to evaluate congestion controls (again, not the whole system, to me that's a different evaluation, and of course a better congestion control should lead to a better total system) with a static traffic profile.

Is my concern unfounded? Is it a non-problem I'm envisioning?   (if so, see below for 2 more concerns  :-)   )

Cheers,
Michael

PS:  Concern #2: we can expect a static traffic profile feeding into a congestion controller to be relatively quickly done, but creating a test setup with a reactive codec is going to be much harder and take more time. Will we ever see it? When? Will it be easy to use?
PPS: Concern #3: the parameter space for tests might expand quite significantly when we involve reactive codecs... this will make any evaluation harder, making the results harder to compare, resulting in more time necessary to come to a conclusion about which controller does a better job.

(*) TFRC is an argument to the contrary, but I haven't seen a single evaluation result that shows that its smooth rate is indeed beneficial. On the contrary, we now seem to have some agreement that TFRC isn't the best mechanism.

On 29. okt. 2013, at 01:53, Mo Zanaty (mzanaty) wrote:

> On 10/28/13, 7:45 PM, Bernard Aboba <[email-masked]> wrote:
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 29 Oct 2013 09:31:06 +0000

Michael,

Your concern is why I think we need to test the codec and the congestion control together. The dynamics of how, and importantly when, the codec can change its rate will be critical to the dynamics of the congestion control. This is especially true if we want to keep the latency low by avoiding sender-side buffering to smooth the codec output to match the desired sending rate. Testing congestion control without a codec will give us something that doesn't respect the QoE concerns, and runs the risk of producing an algorithm that is safe and responsive, but not useful for interactive video.

Colin

---

From: Michael Welzl <[email-masked]>
Date: Tue, 29 Oct 2013 11:32:04 +0100

Okay,

I'm not convinced, but I will not argue the point any longer. I raised my concerns but maybe I'm wrong, no problem.

Cheers,
Michael

---

From: Varun Singh <[email-masked]>
Date: Tue, 29 Oct 2013 13:31:41 +0200

Hi,

> Also, if we go for the approach of using uncompressed test video sequence
> for testing the combination of video codec and congestion control, we should
> also recommend a reference codec and default configurations. So that focus
> of the evaluations remains on the congestion control module.  What do folks
> think of x264 (http://www.videolan.org/developers/x264.html)?

In [1], we used gstreamer to set and get the encoding bit rate. We chose to
use gstreamer instead of using x264 directly was the idea to easily swap
it out with libvpx. We haven't tested this yet but I suppose it should
be possible.

In our experiments, we just set the application constraints (e.g.,
video resolution,
frame rate, etc) and let the codec parameters set as default. I suppose the same
would be true if we just swapped out x264 for vpx?

[1] http://arxiv.org/pdf/1310.1582v1.pdf See section 6.1, we describe
the interaction
between the APIs between

---

From: Varun Singh <[email-masked]>
Date: Tue, 29 Oct 2013 13:36:24 +0200

Hi Xiaoqing,

A couple of comments inline.

On Mon, Oct 28, 2013 at 10:15 PM, Xiaoqing Zhu (xiaoqzhu)
<[email-masked]> wrote:
> Agree with Colin and Mo.
>
> For a statistical video traffic model, I would propose to break down the 2nd
> point (encoder rate control) in Mo's list further, to three sub-components:
>

The original email lists questions for modeling video traffic, if you have
concrete suggestions on values, could you put them together in an email?

> frames much larger than others. This will lead to burstiness on time scale

Any idea on how to model this? i.e., how large (in size) should these
packets be to the others?

> of frame intervals

Agree, with the pacing/shaper.

---

From: Varun Singh <[email-masked]>
Date: Tue, 29 Oct 2013 13:39:25 +0200

Clarification for SVC: does this mean that the base layer's target
encoding bit rate is fixed and cannot be modified?

On Tue, Oct 29, 2013 at 2:53 AM, Mo Zanaty (mzanaty) <[email-masked]> wrote:
> You're right that there is flexibility in enhancement layers. But because of
> the layering structure, the base is not flexible - since the enhancement
> layers depend on it, it is typically protected via FEC or RTX. This affects
> the viability of "slow start" schemes as well other aspects - no point
> sending video at all if you cannot support the base layer, at least.
>
> This is true for simulcast layers, but not necessarily scalable layers. For
> the latter, arbitrary rates can be achieved within a limited range (like
> down to half the bitrate) by selectively dropping some packets of the
> highest enhancement layer with minimal visual quality impact. If the stream
> is highly scalable (many layers, which is rare), the limited range expands
> (like down to maybe a third, quarter, or less of the bitrate). If the stream
> is coarsely scalable (e.g. two layers, which is common), dropping below the
> limited range incurs changing encoding parameters, at which point arbitrary
> rates are again possible.
>
> Mo
>
>

---

From: Varun Singh <[email-masked]>
Date: Tue, 29 Oct 2013 13:54:58 +0200

HI Geert,

On Mon, Oct 28, 2013 at 10:43 PM, Van Der Auwera, Geert
<[email-masked]> wrote:
> Also agree. For initial evaluations of CC proposals and to find consensus
> quickly, a statistical video traffic model should be sufficient. However,
> proponents should be allowed and perhaps encouraged from producing
> additional results with a complete live video encoding testbed of their

The cc-eval document encourages proponents to use a video stream,
though this is encouraged as a second step (Sec 4.3), but it should
not dissuade proponents from pursuing this path.

> choice (video quality?). Agreeing on a particular codec at this point may be

In Berlin (IETF87), it was decided to leave video quality as an open
issue and add it to the appendix, i.e., to reopen it when we have more
information, or proponents conducting experiments with live encoded
streams.

> a bit hard. Regarding the uncompressed test video sequences (VT type),
> Qualcomm could provide a set of long sequences (> 1min).
>

+1

Are these are similar to the ones available at media.xiph.org/video/derf/
Usually I loop the test video sequences to make them longer.

---

From: Paul Coverdale <[email-masked]>
Date: Tue, 29 Oct 2013 13:33:59 +0100

I should have jumped in earlier but I agree with Michael's concern. By using
a codec which behaves adaptively it will be difficult to figure out what the
congestion controller is really doing. At least we should include a static
profile as a baseline reference.

...Paul

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Michael Welzl
Sent: Tuesday, October 29, 2013 11:32 AM
To: Colin Perkins
Cc: rmcat WG
Subject: Re: [rmcat] [rtcweb] Discussion on RMCAT traffic model in Vancouver
(03 Nov, 1300-1500)

Okay,

I'm not convinced, but I will not argue the point any longer. I raised my
concerns but maybe I'm wrong, no problem.

Cheers,

Michael

Michael,

Your concern is why I think we need to test the codec and the congestion
control together. The dynamics of how, and importantly when, the codec can
change its rate will be critical to the dynamics of the congestion control.
This is especially true if we want to keep the latency low by avoiding
sender-side buffering to smooth the codec output to match the desired
sending rate. Testing congestion control without a codec will give us
something that doesn't respect the QoE concerns, and runs the risk of
producing an algorithm that is safe and responsive, but not useful for
interactive video.

Colin

Hence I'd like to clarify - my concern was that we're making the system too
complex by involving a codec that reacts to the congestion controller **when
evaluating congestion control mechanisms**. I mean, once I've done a test,
I'd like to know why I see the effects that I see, and which mechanism is to
blame for them. This will get harder with this setup.

Say, we create a test with a reactive codec over a congestion controller,
and in some network scenarios, we see oscillations: the congestion
controller increased its rate, allowing the codec to switch to a higher
encoding rate; this higher rate was too much, and so quite fast the
congestion controller reduces its rate again, hence the codec reacts and
switches to a lower rate, and so forth.

Now, if that is the test setup for evaluating a congestion controller, a
conclusion from such a test could be that the congestion controller has to
be stabilized. I would, however, argue that a congestion controller should
be as quick as it possibly can at seeking out available bandwidth, and hence
such fluctuations are natural (*), and it's rather the job of the codec to
dampen its rate switching.

Either way, in this example test, we face a bad result, and two screws to
tune to fix the problem - one on the side of the congestion control, and one
on the side of the codec.

This is why I would prefer to evaluate congestion controls (again, not the
whole system, to me that's a different evaluation, and of course a better
congestion control should lead to a better total system) with a static
traffic profile.

Is my concern unfounded? Is it a non-problem I'm envisioning?   (if so, see
below for 2 more concerns  :-)   )

Cheers,

Michael

PS:  Concern #2: we can expect a static traffic profile feeding into a
congestion controller to be relatively quickly done, but creating a test
setup with a reactive codec is going to be much harder and take more time.
Will we ever see it? When? Will it be easy to use?

PPS: Concern #3: the parameter space for tests might expand quite
significantly when we involve reactive codecs... this will make any
evaluation harder, making the results harder to compare, resulting in more
time necessary to come to a conclusion about which controller does a better
job.

(*) TFRC is an argument to the contrary, but I haven't seen a single
evaluation result that shows that its smooth rate is indeed beneficial. On
the contrary, we now seem to have some agreement that TFRC isn't the best
mechanism.

Agreed. The base layer has no flexibility just like a flat non-scalable
stream.

On Oct 28, 2013, at 3:10 PM, "Mo Zanaty (mzanaty)" <[email-masked]>
wrote:

Mo

Colin said: 

"

Mo

-- 
Colin Perkins
http://csperkins.org/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 29 Oct 2013 12:45:07 +0000

Live encoding parameters can always be modified even for base layers. The original point was that even pre-encoded scalable streams can transmit arbitrarily smooth rates within a limited range (down to the base layer rate) without modifying encoding parameters by simply dropping (pre-)encoded packets from enhancement layers.

Mo

On Oct 29, 2013, at 7:39 AM, "Varun Singh" <[email-masked]> wrote:

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Tue, 29 Oct 2013 13:32:42 +0000

FWIW - I also agree with Michael (Welzl, as I usually agree with myself).

I think we are going down a dangerous road and one in which, if we went down previously, the Internet as we know it would not exist.

When did we - in the effort to prove sufficiency, convergence and/or stability of a lower layer - demand that the constraints and DESIRED OPERATION of the upper layer (such as it can only change rate every so often) MUST be met?

Michael Welzl is right, if we need the codec to "shut up" (stop producing data for a while) or to quickly slowdown in order to have a stable transport layer - then we need it to shut up or slow down. This goes to the "upper layer API" issue that Mirja brought up a few weeks ago.

The only "desired operation" of the upper layer we desire for success is low delay (and "reasonable share of bandwidth")  - and even that is qualified for "when low delay is possible".

That is why I argued that for the base stability tests that we must do what TCP assumed for stability, the ability to send an infinite amount of data.

We need to live with the (predominantly TCP and best-effort/tail-drop) world as we know it and live within those constraints.

Once we know what is possible, then codec designers will differentiate themselves on how best to render what we (i.e., RMCAT or similar) can provide them. Good codec designs are NOT STATIC, but rather are improved as the constraints presented to them are known. For example, if I frame bursts are "too big", then you find codec manufacturers building gradual I-frame refresh capabilities. Designing the rmcat transport assuming that codec capabilities are known and are forever static is like TCP having been designed with the assumption that hosts would forever have sub 200 MHz processors in them.

[Aside: Although at data center speeds TCP did have its challenges, it was pretty darn good from hundreds of bits per second to a few megabits per second with few changes to its underlying mechanics. But I digress.]

There is only so much the rmcat layer can do regarding QoE. Codec manufactures may desire, for example, not to change to a higher resolution until they are "very sure" that they will not have to change downward anytime soon - because changing resolutions too often does not yield a better QoE. Let them design algorithms on when and how to shift based on whatever information the "rmcat layer" provides them.

Off Soapbox,

Michael Ramalho

From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Welzl
Sent: Tuesday, October 29, 2013 6:32 AM
To: Colin Perkins
Cc: rmcat WG
Subject: Re: [rmcat] [rtcweb] Discussion on RMCAT traffic model in Vancouver (03 Nov, 1300-1500)

Okay,

Cheers,
Michael

Michael,

Colin

On 29 Oct 2013, at 08:30, Michael Welzl <[email-masked]<mailto:[email-masked]>> wrote:
So - looking at this thread, it seems that consensus is against me  :-)

Cheers,
Michael

Mo

Colin said:

Mo

--
Colin Perkins
http://csperkins.org/

---

From: Michael Welzl <[email-masked]>
Date: Tue, 29 Oct 2013 14:41:59 +0100

True, especially when considering FEC! You're right, that was the original point!

This makes me think - agree with it or not, but: to satisfy what *I* wanted to have (different static traffic profiles as inputs to the congestion controller), is it perhaps enough to just consider a min and max rate?????

Now THAT could make things easy at least in a first step?!

Cheers,
Michael

---

From: John Leslie <[email-masked]>
Date: Tue, 29 Oct 2013 10:27:30 -0400

Michael Ramalho (mramalho) <[email-masked]> wrote:
>... 
> When did we - in the effort to prove sufficiency, convergence and/or
> stability of a lower layer - demand that the constraints and DESIRED
> OPERATION of the upper layer (such as it can only change rate every
> so often) MUST be met?

   I had to read that three times to catch your meaning. :^(

   (You do mean that it's a bad idea to demand that a transport "send"
without regard to network conditions, right?)

> The only "desired operation" of the upper layer we desire for success
> is low delay (and "reasonable share of bandwidth") - and even that is
> qualified for "when low delay is possible".

   There is certainly a desire for low delay -- I would argue that it
trumps "reasonable share of bandwidth" when it gets above 100 msec.

> That is why I argued that for the base stability tests that we must
> do what TCP assumed for stability, the ability to send an infinite
> amount of data.

   That strikes me as a bad model: we want to send at a _rate_ which
is fairly consistent. The TCP model of constantly probing for more
available bandwidth _needs_to_ be avoided.

> We need to live with the (predominantly TCP and best-effort/tail-drop)
> world as we know it and live within those constraints.

   We need to _function_in_ that world for a while, yes; but the nature
of our traffic is entirely too different to "live within" the constraints
of TCP.

> Once we know what is possible, then codec designers will differentiate
> themselves on how best to render what we (i.e., RMCAT or similar) can
> provide them.

   "What is possible" changes too fast for that.

   Codec designers must devote most of their attention to balancing
bandwidth and delay -- it's always possible to reduce bandwidth by
increasing delay through the codec.

> Good codec designs are NOT STATIC, but rather are improved as the
> constraints presented to them are known. For example, if I-frame
> bursts are "too big", then you find codec manufacturers building
> gradual I-frame refresh capabilities.

   That's worth doing anyway. (Further, I doubt that "bursts too big"
has ever been an unambiguous constraint.)

> Designing the rmcat transport assuming that codec capabilities are
> known and are forever static is like TCP having been designed with
> the assumption that hosts would forever have sub 200 MHz processors
> in them.

   Nonetheless, there are limits to what a standardized transport can
adapt to. We need to build into the first version spec what sort of
"slow down" signals we can return to the codecs.

> There is only so much the rmcat layer can do regarding QoE. Codec
> manufacturers may desire, for example, not to change to a higher
> resolution until they are "very sure" that they will not have to
> change downward anytime soon - because changing resolutions too often
> does not yield a better QoE.

   Yes.

> Let them design algorithms on when and how to shift based on whatever
> information the "rmcat layer" provides them.

   But we have to design that feedback path into our first spec.

--
John Leslie <[email-masked]>

---

From: Stefan Holmer <[email-masked]>
Date: Tue, 29 Oct 2013 15:32:25 +0100

I also agree that we should try to evaluate the different components (CC,
pacing, rate control) separately. But I also think that doing full system
evaluation has some value for the reasons mentioned by Colin. In the end we
want to have a congestion controller which behaves in a way which is good
for real-time media quality. I'm not sure how useful it is to evaluate for
all types of streams/rate control we can come up with though (svc,
simulcast, one fully flexible stream). Maybe we have to if we are also
designing how to probe between simulcast/svc layers?

When it comes to pacing, which was mentioned earlier in this thread, I want
to highlight the issue of pacing at the rate suggested by the congestion
controller. If the CC is limited by media bitrate we would introduce big
delays for users who have "infinite" bandwidth if we would pace the packets
at the CC rate. I don't think it's unreasonable to allow pacing at a higher
bitrate than the CC suggests and rely on the codec to, on average, stay
within its target. We have to make sure we measure both packets lost and
delay incurred when evaluating this component.

Related to this, are we considering the encoder and its rate control as a
component that we can trust, or should the CC queue/drop packets if the
encoder produces too much over some window?

On Tue, Oct 29, 2013 at 2:32 PM, Michael Ramalho (mramalho) <
[email-masked]> wrote:

>  FWIW – I also agree with Michael (Welzl, as I usually agree with myself).
> ****
>
> ** **
>
> I think we are going down a dangerous road and one in which, if we went
> down previously, the Internet as we know it would not exist.****
>
> ** **
>
> When did we – in the effort to prove sufficiency, convergence and/or
> stability of a lower layer – demand that the constraints and DESIRED
> OPERATION of the upper layer (such as it can only change rate every so
> often) MUST be met? ****
>
> ** **
>
> Michael Welzl is right, if we need the codec to “shut up” (stop producing
> data for a while) or to quickly slowdown in order to have a stable
> transport layer – then we need it to shut up or slow down. This goes to the
> “upper layer API” issue that Mirja brought up a few weeks ago.****
>
> ** **
>
> The only “desired operation” of the upper layer we desire for success is
> low delay (and “reasonable share of bandwidth”)  – and even that is
> qualified for “when low delay is possible”. ****
>
> ** **
>
> That is why I argued that for the base stability tests that we must do
> what TCP assumed for stability, the ability to send an infinite amount of
> data.****
>
> ** **
>
> world as we know it and live within those constraints.****
>
> ** **
>
> provide them. Good codec designs are NOT STATIC, but rather are improved as
> the constraints presented to them are known. For example, if I frame bursts
> are “too big”, then you find codec manufacturers building gradual I-frame
> refresh capabilities. Designing the rmcat transport assuming that codec
> capabilities are known and are forever static is like TCP having been
> designed with the assumption that hosts would forever have sub 200 MHz
> processors in them.****
>
> ** **
>
> [Aside: Although at data center speeds TCP did have its challenges, it was
> pretty darn good from hundreds of bits per second to a few megabits per
> second with few changes to its underlying mechanics. But I digress.]****
>
> ** **
>
> manufactures may desire, for example, not to change to a higher resolution
> until they are “very sure” that they will not have to change downward
> anytime soon – because changing resolutions too often does not yield a
> better QoE. Let them design algorithms on when and how to shift based on
> whatever information the “rmcat layer” provides them.****
>
> ** **
>
> Off Soapbox,****
>
> ** **
>
> Michael Ramalho****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Michael Welzl
> *Sent:* Tuesday, October 29, 2013 6:32 AM
>
> *To:* Colin Perkins
> *Cc:* rmcat WG
> *Subject:* Re: [rmcat] [rtcweb] Discussion on RMCAT traffic model in
> Vancouver (03 Nov, 1300-1500)****
>
>  ** **
>
> Okay,****
>
> ** **
>
> concerns but maybe I'm wrong, no problem.****
>
> ** **
>
> Cheers,****
>
> Michael****
>
> ** **
>
>  ****
>
> On 29. okt. 2013, at 10:31, Colin Perkins wrote:****
>
>
>
> ****
>
> Michael,****
>
> ** **
>
> interactive video.****
> ** **
> Colin****
> ** **
> ** **
> On 29 Oct 2013, at 08:30, Michael Welzl <[email-masked]> wrote:****
>
>  So - looking at this thread, it seems that consensus is against me  :-)**
> **
>
> ** **
>
> Hence I'd like to clarify - my concern was that we're making the system
> too complex by involving a codec that reacts to the congestion controller
> **when evaluating congestion control mechanisms**. I mean, once I've done a
> test, I'd like to know why I see the effects that I see, and which
> mechanism is to blame for them. This will get harder with this setup.****
>
> ** **
>
> Let me give an example to explain my concern:****
>
> ** **
>
> switches to a lower rate, and so forth.****
>
> ** **
>
> be as quick as it possibly can at seeking out available bandwidth, and
> hence such fluctuations are natural (*), and it's rather the job of the
> codec to dampen its rate switching.****
>
> ** **
>
> tune to fix the problem - one on the side of the congestion control, and
> one on the side of the codec.****
>
> ** **
>
> traffic profile.****
> ** **
> Is my concern unfounded? Is it a non-problem I'm envisioning?   (if so,
> see below for 2 more concerns  :-)   )****
>
> ** **
>
> Cheers,****
>
> Michael****
>
> ** **
>
> Will we ever see it? When? Will it be easy to use?****
>
> job.****
> ** **
> ** **
> mechanism.****
> ** **
> ** **
> On 29. okt. 2013, at 01:53, Mo Zanaty (mzanaty) wrote:****
>
>
>
> ****
>
> stream.****
> ** **
> ** **
> On 10/28/13, 7:45 PM, Bernard Aboba <[email-masked]> wrote:****
>
> ** **
>
> You're right that there is flexibility in enhancement layers. But because
> of the layering structure, the base is not flexible - since the enhancement
> sending video at all if you cannot support the base layer, at least.****
>
>
> wrote:****
>  This is true for simulcast layers, but not necessarily scalable layers.
> For the latter, arbitrary rates can be achieved within a limited range
> (like down to half the bitrate) by selectively dropping some packets of the
> (like down to maybe a third, quarter, or less of the bitrate). If the
> stream is coarsely scalable (e.g. two layers, which is common), dropping
> below the limited range incurs changing encoding parameters, at which point
> arbitrary rates are again possible.****
>
> ** **
>
> Mo****
>
> ** **
>
> ** **
>
> On 10/28/13, 5:58 PM, Bernard Aboba <[email-masked]> wrote:****
>
> ** **
>
> Colin said: ****
>
>  ****
>
> sequence?****
> ”****
>  ****
> [BA] I like this approach, because it somewhat approximates the situation
> of simulcast/layered coding.  Previous work has indicated a number of
> encoder has fully smooth rate control.   ****
>
>  ****
>
> On 27 Oct 2013, at 11:50, Michael Welzl <[email-masked]> wrote:****
>
> On 27. okt. 2013, at 12:33, Varun Singh <[email-masked]> wrote:****
>
> the network can always follow it -****
>
>
> sequence?****
> I think that could be a very good idea!****
>
>
> http://csperkins.org/****
>
>     ** **
>
> ** **
>

---

From: Ted Hardie <[email-masked]>
Date: Tue, 29 Oct 2013 08:38:48 -0700

> producing an algorithm that is safe and responsive, but not useful for
Hi Colin,

Not to challenge your point on testing methodology, but to uplevel a bit
for the benefit of
those not as involved in WebRTC as you are:  Please remember that there is
no
single codec that will provide a comprehensive test.  The WebRTC framework
is set up
for codec agility, and we certainly expect different codecs to be
negotiated in different
circumstances; the decision next week is for a codec sufficient to avoid
negotiation
failure, not one that replaces that negotiation.

So, please assume the congestion control mechanisms need to serve VP8 and
H.264
and VP9 and H.265 and whatever else someone chooses.  This system will not
make
a single choice.

regards,

Ted Hardie

> mechanism is to blame for them. This will get harder with this setup.
>
> codec to dampen its rate switching.
>
> one on the side of the codec.
>
> see below for 2 more concerns  :-)   )
>
> Cheers,
> Michael
>
>   You're right that there is flexibility in enhancement layers. But
> because of the layering structure, the base is not flexible - since the
> enhancement layers depend on it, it is typically protected via FEC or RTX.
> This affects the viability of "slow start" schemes as well other aspects -
> no point sending video at all if you cannot support the base layer, at
> least.
>
> arbitrary rates are again possible.
>
>  Mo
>
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 29 Oct 2013 16:02:56 +0000

Hi Ted,

On 29 Oct 2013, at 15:38, Ted Hardie <[email-masked]> wrote:
> Not to challenge your point on testing methodology, but to uplevel a bit for the benefit of those not as involved in WebRTC as you are:  Please remember that there is no single codec that will provide a comprehensive test.  The WebRTC framework is set up for codec agility, and we certainly expect different codecs to be negotiated in different
> circumstances; the decision next week is for a codec sufficient to avoid negotiation failure, not one that replaces that negotiation.
> 
> So, please assume the congestion control mechanisms need to serve VP8 and H.264 and VP9 and H.265 and whatever else someone chooses.  This system will not make a single choice.

Absolutely agree.

That said, I tend to suspect that a congestion control algorithm that can react to the dynamics of one modern codec will likely do a reasonable job of adapting to another codec (possibly excepting layered vs. non-layered codecs), and that most of the benefit comes from testing with some codec, and paying attention to the impact of the rate variation on QoE, rather than just testing the impact on the network.

-- 
Colin Perkins
http://csperkins.org/

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Tue, 29 Oct 2013 16:48:07 +0000

Thanks for the pointer, Varun.  This is helpful in putting together a complete
systems (CC + live encoding) for more realistic evaluations.  Will give that a 
try on my end …

Best,
Xiaoqing

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Tue, 29 Oct 2013 16:52:27 +0000

Hi Varun, 

Regarding your comments, we will share a concrete example of how we modeled
the video traffic in our NADA simulations at Vancouver.  Once I put together the 
slides/write-up (working on it right now), can also share it on the mailing list first
to solicit comments. 
 
Thanks,
Xiaoqing

---

From: Dave Taht <[email-masked]>
Date: Tue, 29 Oct 2013 11:29:38 -0700

>>> testbed ought to reflect that.

I would be very happy if also along the way, while exploring e2e cc
techniques for webrtc, that folk also tried the various aqm and packet
scheduling techniques being developed and deployed, other than drop
tail, notably wred, fq_codel, sqf & pie.

That's a two way street as I'd also very much like the aqm developers
paying attention here also... (I did some early work with webrtc +
fq_codel earlier this year but it was too buggy to get anything
publishable.)

-- 
Dave Täht

---

From: Varun Singh <[email-masked]>
Date: Wed, 30 Oct 2013 15:49:33 +0200

Hi all,

Practical details for the meeting on **RMCAT Traffic models**

Date: 03.11. 2013
Time: 12:30 to 14:30.
Assigned Room: Regency A

Cheers,
Varun

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 30 Oct 2013 11:59:10 -0700

These are good modeling ideas/parameters. I think it would be good to hae a plan first
though how this can be executed (which will depnd on who is able to contribute work
to the process).

Eg: Are there going to be a range of actual codec developers that would be able
and willing to run with their real codec some combined reference-video+codec+
reference congestion control method to derive the models codec-parameters ? WOuld they
be able to claim codec parameters (as outlined by Mo/Xianqing) even without running
such tests ?

Alternatively: Don't bother about a wider range of (likely proprietary) production
codecs, but just pick one reference h264 and one reference VP8 codec and do all
modelling based on using the parameters in thoe codecs - and potentially fudging/improving
those reference codecs to play beter ball with congestion control.

These are of course not the only choices for execution, but i think they give an
idea about the type of execution details that IMHO we should know to be feasiable
before finalizing on a modelling approach.

I would also like to chime into the origianl approach that would much more take the
intra-codec rate-control loop out of the picture:

I was hoping RMcat would also work in the presence of switched video conferences. A
switching MCU can react to congestion by shaping (increasing delay), switch streams
or dropping (low priority) video frames. This might be easier to model first, and it
also creates a sender that can potentially react faster than a codec (that may need
to have a longer sequence of frames in flight before being able to impact new encoding
bitrate).

Cheers
    Toerless

-- 
---
Toerless Eckert, [email-masked]
Cisco NSSTG Systems & Technology Architecture
SDN: Let me play with the network, mommy!