Subject: Re: [rmcat] Comments on NADA algorithm update and test case evaluations

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 1 Dec 2014 14:53:48 +0100

Hi Ingemar,

quick comment on the level of detail question:

The draft definitely needs enough details for other people to be able to re-implement your algorithm; so (pseudo)-code is definitely useful. However, the draft should also give a more higher level overview of the algorithm (for people as a starting point or if someone doesn’t not want to re-implement it but understand and use it). It’s often hard to find the right balance here. I personally would recommend the following: try to write a (short) overview section (2-3 pages) where the basic principles are explained and maybe even some kind of simplified pseudo code (only a few lines at once) is given. Then have the detailed description with some potentially pseudo code but also everything explained in textual description and, finally, put more detailed code (if still needed) in the appendix (without much additional textual description).

Hope that helps,
Mirja

> Am 12.11.2014 um 07:54 schrieb Ingemar Johansson S <[email-masked]>:
> 
> Ok, thanks for the response. 
> 
> I guess others need to chime in but I am not sure that it is sufficient to only mention the necessity of de-noising. 
> I agree that de-noising can be done in many ways, problem as I see it is that in most of the cases de-noising adds a certain degree of low-pass filtering that will remove the "edge" of the derivating function that is supposed to give a prompt indication of congestion, in other words congestion detection may become slower.
> For that reason I would believe that at least an example algorithm that can work well with the basic test cases would be good to have.
> 
> But I guess this all boils down to what level of detail is required in the drafts. SCReAM goes into a lot of detail, perhaps too much detail. Some advice from the WG about the level of detail would be welcome here.
> 
> /Ingemar 
> 
>> -----Original Message-----
>> From: Xiaoqing Zhu (xiaoqzhu) [mailto:[email-masked]]
>> Sent: den 12 november 2014 01:06
>> To: Ingemar Johansson S; rmcat WG ([email-masked])
>> Subject: Re: [rmcat] Comments on NADA algorithm update and test case
>> evaluations
>> 
>> Hi Ingemar,
>> 
>> Thanks much for looking into the equations in greater details. I agree with
>> you that if we were to apply eq (2) directly on the ³raw² delay samples, it may
>> not very well in the presence of high jitter, or any other source of delay
>> spikes (e.g., system hiccup).
>> 
>> In practical implementations some proper de-noising procedure needs to be
>> in place,  before one can trustfully rely on delay values as congestion signals. I
>> don¹t think we want to stick with any specific method of de-noising, but
>> rather to emphasize the necessity of such components.
>> Based on your input, I will also add some explicit discussions on that in the
>> next version of the draft.
>> 
>> As mentioned in the earlier email, I will report back to the mailing list after
>> we¹ve tried out with some tests with added network jitter.
>> 
>> Best,
>> Xiaoqing
>> 
>> 
>> On 11/11/14, 2:41 AM, "Ingemar Johansson S"
>> <[email-masked]> wrote:
>> 
>>> Hi
>>> Resend in plain text to see if the attachments are kept...
>>> 
>>> As a followup to the jitter discussion  to furher stress the importance
>>> of trying out the algorithms with additional jitter.
>>> 
>>> I had a look at equation 2 in your draft
>>> 
>>>               x_n - x_(n-k)
>>>       x'_n = ---------------.           (2)
>>>                 delta
>>> 
>>> I guess the idea is here to be able to promptly react to increasing one
>>> way delays. The risk with the derivation is that the impact of noise
>>> (jitter) is increased.
>>> I extracted the one way IP packet delays from our LTE simulations with
>>> a fixed 300kbps video source. i.e no rate adaptation whatsoever.
>>> Attached are two examples where x and x´are computed based on this
>>> data. The first example is DL (video direction towards LTE terminal), the
>> second is UL.
>>> The load level is the lowest possible, corresponding to 2 users/cell in
>>> DL and 1 user per cell in UL, which means that there are not really any
>>> serious issues with congestion.
>>> As you can observe, the jitter, especially in the UL case, causes x´ to
>>> reach quite high values and this I believe will cause xhat computed as
>>> ..
>>>   x_hat = x_n + eta*tao_o* x'_n       (5)
>>> 
>>> ..to get very high values too with the possible consequence that the
>>> bitrate is reduced alternatively that the bitrate will stay put on low
>>> values, even for moderate amounts of jitter, not saying that this will
>>> happen, just that there is a certain risk that it will happen.
>>> 
>>> 
>>> /Ingemar
>>> 
>>> 
>>> From: Xiaoqing Zhu (xiaoqzhu) [mailto:[email-masked]]
>>> Sent: den 11 november 2014 01:17
>>> To: Ingemar Johansson S; rmcat WG ([email-masked])
>>> Subject: Re: [rmcat] Comments on NADA algorithm update and test case
>>> evaluations
>>> 
>>> Thanks for your comments, Ingemar.  I believe we've covered some of
>>> this during the presentation.  More inline below.
>>> 
>>> Best,
>>> Xiaoqing
>>> 
>>> 
>>> From: Ingemar Johansson S <[email-masked]>
>>> Date: Sunday, November 9, 2014 at 10:03 AM
>>> To: "rmcat WG ([email-masked])" <[email-masked]>
>>> Cc: Xiaoqing Zhu <[email-masked]>
>>> Subject: [rmcat] Comments on NADA algorithm update and test case
>>> evaluations
>>> 
>>> Hi
>>> 
>>> Thanks for the illustrative slide set, it helped me to understand the
>>> concept.
>>> 
>>> A few email questions on your presentation
>>> 
>>> +Non-linear warping of delay:
>>> I read one of the background papers behind this some time ago, one
>>> thing that crossed my mind is that there is a risk that one
>>> accidentially may run into the loss/marking based region in cases where
>>> several users share a common transmission medium, which is the case e.g
>>> with LTE and WiFi access. To take the LTE example with default bearers
>>> which is the part I know the best. In cases where new users enter a
>>> cell (either because of handover or just because the flows are just
>>> started).  New users typically get a higher scheduling weight in cases
>>> where proportional fair schedulers are used. This gives already
>>> existing users a lower throughput which will manifest itself as a temporary
>> larger delay.
>>> The outcome may then be that the loss/marking region is enterered even
>>> though there is in actually no competing flow on the bearer, rather it
>>> is a case of competing bearers.
>>> Once there it can be difficult to get back to the delay based region. I
>>> have not been able to try out how serious this potential may be.
>>> 
>>> The concern you mentioned is a valid one, in general. Indeed we have
>>> not yet tested with cellular test cases to see how the scheme works there.
>>> However, what we do know is that the scheme is fairly robust to
>>> individual delay outliers so long as the receiver is equipped with some
>>> standard outlier rejection mechanism (e.g., median filtering) before
>>> reporting on the delay values. In the draft, for simplicity, we
>>> currently do not spell out those specific operations. In the next
>>> version I'll add some discussions on that.
>>> 
>>> Even if occasionally the reported delay is outrageously large, since
>>> the rate update equation itself is _gradual_.  So long as the reported
>>> delays are not consistently long for a substantial period for
>>> non-congestion reasons, the algorithm should not be locked in
>>> loss-based mode by accidental large delay values.
>>> 
>>> And, of course, I also plan to test more extensively (e.g., with a mix
>>> of NADA/TCP flows) to further validate effectiveness of this
>>> loss-resilient version of NADA.
>>> 
>>> 
>>> +Evaluation:
>>> + Is this evaluation run  with some additional network jitter ?,
>>> + reason
>>> why I ask is that cellular access may add some extra jitter mainly due
>>> to scheduling.
>>> 
>>> Right now, no delay jitter. Adding that is certainly a good suggestion.
>>> Will give it a try.
>>> 
>>> 
>>> + I see that the graphs indicate the send rate. i.e the output rate
>>> + from
>>> the rate shaping buffer. What about the delay figures, do they depict
>>> the network delay (socket to socket) or is it the video frame delay ?
>>> 
>>> These are all packet level socket-to-socket delays. I could add the
>>> video frame delay stats too but right now, since we are currently
>>> experimenting with an ideal synthetic traffic source, I don't think it
>>> will much new information.  Things will be more interesting once we
>>> move to tests driven by synthetic video-like traffic sources.
>>> 
>>> 
>>> 
>>> Regards
>>> /Ingemar
>>> =================================
>>> Ingemar Johansson  M.Sc.
>>> Senior Researcher
>>> 
>>> Ericsson AB
>>> Wireless Access Networks
>>> Labratoriegränd 11
>>> 971 28, Luleå, Sweden
>>> Phone +46-1071 43042
>>> SMS/MMS +46-73 078 3289
>>> [email-masked]
>>> www.ericsson.com
>>> 
>>> "The difference between stupidity and
>>> genius is that genius has its limits."
>>> Albert Einstein
>>> =================================
>>> 
>>>

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 10 Dec 2014 18:03:54 +0000

Thanks Mirja for the details. Will update the draft and submit it after the holidays. Not sure how to do with the pseudo code, that may have to wait until a later version.

/Ingemar

-----Original Message-----
From: Mirja Kühlewind [mailto:[email-masked]] 
Sent: den 1 december 2014 14:54
To: Ingemar Johansson S
Cc: Xiaoqing Zhu (xiaoqzhu); rmcat WG ([email-masked])
Subject: Re: [rmcat] Comments on NADA algorithm update and test case evaluations

Hi Ingemar,

The draft definitely needs enough details for other people to be able to re-implement your algorithm; so (pseudo)-code is definitely useful. However, the draft should also give a more higher level overview of the algorithm (for people as a starting point or if someone doesn't not want to re-implement it but understand and use it). It's often hard to find the right balance here. I personally would recommend the following: try to write a (short) overview section (2-3 pages) where the basic principles are explained and maybe even some kind of simplified pseudo code (only a few lines at once) is given. Then have the detailed description with some potentially pseudo code but also everything explained in textual description and, finally, put more detailed code (if still needed) in the appendix (without much additional textual description).

Hope that helps,
Mirja

>> Thanks much for looking into the equations in greater details. I 
>> agree with you that if we were to apply eq (2) directly on the ³raw² 
>> delay samples, it may not very well in the presence of high jitter, 
>> or any other source of delay spikes (e.g., system hiccup).
>> 
>> In practical implementations some proper de-noising procedure needs 
>> to be in place,  before one can trustfully rely on delay values as 
>> congestion signals. I don¹t think we want to stick with any specific 
>> method of de-noising, but rather to emphasize the necessity of such components.
>> Based on your input, I will also add some explicit discussions on 
>> that in the next version of the draft.
>> 
>> As mentioned in the earlier email, I will report back to the mailing 
>> list after we¹ve tried out with some tests with added network jitter.
>> 
>> Best,
>> Xiaoqing
>> 
>> 
>>> As a followup to the jitter discussion  to furher stress the 
>>> importance of trying out the algorithms with additional jitter.
>>> 
>>> I guess the idea is here to be able to promptly react to increasing 
>>> one way delays. The risk with the derivation is that the impact of 
>>> noise
>>> (jitter) is increased.
>>> I extracted the one way IP packet delays from our LTE simulations 
>>> with a fixed 300kbps video source. i.e no rate adaptation whatsoever.
>>> The load level is the lowest possible, corresponding to 2 users/cell 
>>> in DL and 1 user per cell in UL, which means that there are not 
>>> really any serious issues with congestion.
>>> As you can observe, the jitter, especially in the UL case, causes x´ 
>>> to reach quite high values and this I believe will cause xhat 
>>> computed as ..
>>> bitrate is reduced alternatively that the bitrate will stay put on 
>>> low values, even for moderate amounts of jitter, not saying that 
>>> this will happen, just that there is a certain risk that it will happen.
>>> 
>>> 
>>> /Ingemar
>>> 
>>> 
>>> Thanks for the illustrative slide set, it helped me to understand 
>>> the concept.
>>> 
>>> accidentially may run into the loss/marking based region in cases 
>>> where several users share a common transmission medium, which is the 
>>> case e.g with LTE and WiFi access. To take the LTE example with 
>>> default bearers which is the part I know the best. In cases where 
>>> new users enter a cell (either because of handover or just because 
>>> the flows are just started).  New users typically get a higher 
>>> scheduling weight in cases where proportional fair schedulers are 
>>> used. This gives already existing users a lower throughput which 
>>> will manifest itself as a temporary
>> larger delay.
>>> The outcome may then be that the loss/marking region is enterered 
>>> even though there is in actually no competing flow on the bearer, 
>>> rather it is a case of competing bearers.
>>> Once there it can be difficult to get back to the delay based 
>>> region. I have not been able to try out how serious this potential may be.
>>> 
>>> individual delay outliers so long as the receiver is equipped with 
>>> some standard outlier rejection mechanism (e.g., median filtering) 
>>> before reporting on the delay values. In the draft, for simplicity, 
>>> we currently do not spell out those specific operations. In the next 
>>> the rate update equation itself is _gradual_.  So long as the 
>>> reported delays are not consistently long for a substantial period 
>>> for non-congestion reasons, the algorithm should not be locked in 
>>> And, of course, I also plan to test more extensively (e.g., with a 
>>> mix of NADA/TCP flows) to further validate effectiveness of this 
>>> +reason
>>> why I ask is that cellular access may add some extra jitter mainly 
>>> due to scheduling.
>>> 
>>> the rate shaping buffer. What about the delay figures, do they 
>>> depict the network delay (socket to socket) or is it the video frame delay ?
>>> 
>>> experimenting with an ideal synthetic traffic source, I don't think 
>>> it will much new information.  Things will be more interesting once 
>>> we move to tests driven by synthetic video-like traffic sources.
>>> 
>>> 
>>> 
>>> Regards
>>> /Ingemar
>>> "The difference between stupidity and genius is that genius has its 
>>> limits."
>>> Albert Einstein