Subject: Re: [rmcat] Review of draft-ietf-rmcat-scream-cc-02

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 12 Jan 2016 12:12:15 +0000

Hi

Thanks for the review
Answers inline below, marked [IJ]
/Ingemar

From: Xiaoqing Zhu (xiaoqzhu) [mailto:[email-masked]]
Sent: den 23 december 2015 22:37
To: [email-masked]
Subject: [rmcat] Review of draft-ietf-rmcat-scream-cc-02

Hi,

Here are my review comments for SCReAM draft.  Overall this a well-structured draft. It has explained the basic design principles of the SCReAM, and has provided sufficient details regarding several key modules (network congestion control, media rate control, etc.) at the sender.  It is expected that the authors will update the draft with further details on feedback message from the receiver.

First, some general, high-level comments:

  *   The introduction section seems to motivate the self-clocked algorithm of SCReAM by presenting challenges from wireless LTE-like access links.  However, I am missing the logic behind why a self-clocked algorithm will work better in a wireless environment? It is not clear from the design of SCReAM that it is specifically tailored for transmission over wireless.   Also, does it make more sense to move some of the discussions on challenges of wireless access links (e.g., LTE) to the requirement document instead, since they need to be considered by all RMCAT solutions?
[IJ] I don't think that the benefits with self-clocking is limited to only wireless (LTE) access, it is however possible that it manifests itself more easily as the statistical multiplexing over LTE bearers is generally quite small. If statistical multiplexing is high, then there is a possibility that a rate based congestion control can push back other (self-clocked) flows without a too seriously increased delay (or packet loss). This luxury is not available if the RMCAT flow is the only flow in the LTE bearer, once the radio scheduler has reduced the resource allocation for the bearer the flow needs to reduce the sending rate as soon as possible. Self-clocking has the benefit that it provides a fine-grained congestion control and that is what makes reaction to reduced throughput prompt, this is elaborated upon in  Soo-Hyun Choi´s and Mark Handley´s paper (see TFWC reference in the SCReAM draft).
It is possible that the discussion can be done in the requirements document, not sure.

  *   While the draft has explained in detail operations of the "network congestion control" and "media rate control" modules, it has not explained how "sender transmission control" should be carried out.  For instance, it is unclear how it "limits the output of data" and how exactly it paces outgoing packets.  Will be nice to have a subsection to cover this module as well.   Will also be nice to point out explicitly how the sending window size, as calculated by the network congestion control module (in Sec. 4.1.2.4) relates to transmission control.
[IJ] This part (sender transmission control) has been moved in and out of the draft. In short it is something like
if (bytes_in_flight < cwnd) then transmit_packet_if_available . It was removed earlier for some reason before but if it is Ok to have it moved back then I am happy to add it.

  *   Currently, discussions on how SCReAM takes characteristics of realtime media in to account are spread out throughout the draft. Sometimes that disrupts the flow of explanation of how the algorithm proceeds. Example: end of 4.1.3.  Is there a way to move them into the discussion section instead?
[IJ] OK, can make a try to clean up this and collect this, it is however a balance to be struck here,  as it is sometimes better to explain a given function along with the pseudo code.

  *   Now that I've given this draft a thorough read, it seems to me that the collection of modules in Fig. 1 corresponds well with the overview figure in the NADA draft.  Will be nice for us to align the key terminologies used by both drafts. Might be something to consider for the next meeting :)
[IJ] Yes, it can be good to have a common terminology
Now, some more specific, miscellaneous comments/questions:

  *   Sec. 1.2:  I am not so sure about the statement that "A rate based congestion control has only one mechanism to adjust ...." Fundamentally, both window-based and rate-based congestion control needs to decide on how often and how much data to pump to the network. So why would a window-based scheme have "more" mechanisms to adjust?
[IJ] The reason is the self-clocking, which makes sure that no more data is clocked out than what leaves the network in the other end. This provides with a fast control loop that avoids that too much data is queued up in the network. The slower control loops are the adjustment of the CWND and the media rate control, it is necessary to make the latter loops slower in order to avoid overreaction to delay spikes and jitter. Traditionally rate based schemes require an integration time to become reasonably robust to jitter, one can shorten  this integration time but I would then say that you have ended up with a window based schema.
·  Sec. 3:  typo:  "Addition of media a rate control function" => "Addition of a media rate control function"
[IJ] OK

  *   Sec. 3.1 (and the use of OWD throughout this draft):  both in the LEDBAT draft (also conventionally), the term one-way-delay refers to the end-to-end delay that includes both path propagation delay and queuing delay.  In this draft, if I understood it correctly, OWD is used to refer only the queuing delay portion, which is really one-way-delay minus base delay.  Will be good to avoid such confusion, and maybe pick an alternative term such as "relative one-way-delay", or simply "queuing delay".
[IJ] You are right, this is a long term mistake by yours truly. Ideally this should probably be renamed to queuing delay (queuing_delay) in the draft but I believe that it could be better to just specify clearly that the OWD mentioned in this draft is the queuing delay i.e. with the propagation delay removed.
  *   Sec. 4.1.1.1:  Why is it that for "PRE_CONGESTION_GUARD" and "TX_QUEUE_SIZE",  a range of values are given as opposed to a default value like others?  What default value should an implementer pick for these constants? Missing default value for "T_RESUME_FAST_INCREASE"
[IJ] The rationale was to allow these values to be modified based on media coder behavior. It seems though like at least OpenH264 and VP8 can use the same default settings. Will add T_RESUME_FAST_INCREASE to the list.
  *   Sec. 4.1.1.2:  looks like this list is fairly incomplete.  For instance, it is missing owd_fraction although it's being referred to in the rest of the list.  It is also missing variables that showed up later in equations, such as "gain", "off_target", "current_rate".    Typo under "send_wnd (0)": "when CWND is updated" => "when cwnd is updated"
[IJ] I will have an extra look, I believe it was argued that temporary variables can be omitted from the list
  *   Sec. 4.1.2:   It says in the text (page 12) that "The OWD fraction is sampled every 50ms ...".  On the other hand, the value of ow_fraction is only updated upon the arrival of a feedback message, the interval of which can range between 10s of ms to 200ms (according to Sec.5).  So, what happens when the feedback interval is 100ms?  Would the same owd_fraction value show up twice in the list, or the span of 20 samples exceed 1 sec?  Please add some clarifications in the draft.
[IJ] Yes, this is a simplification and can well mean that the same OWD is sampled twice or more. In practice it does not matter much as the 20 sample vector is only used for computation of an OWD trend
  *   Sec 4.1.2.2:  page 15:  instead of hard-coding the value of "0.2" in pseudo-code, will be better to assign it a constant and add to the list in Sec. 4.1.1.1.
[IJ] OK, will see if I come up with a good reasonably short name
  *   Sec 4.1.2.3:  typo:  ".... by adjusting the owr_target" => "... By adjusting the owd_target".
[IJ] OK
  *   Sec 4.1.2.3:  page 16:  again, some hard-coded values (0.26, 20, 100) in the pseudo-code
[IJ] OK,
  *   Sec. 4.1.2:  it will be helpful to add a paragraph or a flow diagram to explain how the algorithm switches from one mode of operation (e.g., fast increase, reacting to loss events) to another.
  *   Sec 4.1.3: Based on the pseudo code on page 20, calculation of the media target rate does not depend on the current rate from network congestion control when the sender is in fast increase mode. Would this cause overflow/underflow at the RTP queue?
[IJ] Possible and one can see it happen on occasions. It is in practice necessary that the media coder push a little extra in order to provide with enough data to probe the link. So, yes, the RTP queue can sometimes overshoot slightly, most often it is just a matter of milliseconds and depends on how much the bitrate from the media encoder fluctuates.
  *   Sec 4.1.3 (page 21):  typo:  "The vary reader ..." => "The very reader ..."
[IJ] Should in  fact be "the vary reader", meaning a "reader with both eyes open" :) , realize though that it is probably not proper language so I will change this
  *   Sec. 4.2:  one suggestion/request for the feedback message description is to specify the # of bytes used for each field.  One related question is whether the field of total_loss_count and total_ECN_count will encounter wrap-around issues when the conference call lasts for a long time.  Is that something to worry about?
[IJ] OK, will add the requirements. The wraparound of the total_loss_count and total_ECN_count will be an issue if the counters wrap around to exactly the same value as when the last feedback was received, if which case a loss event or ECN event was missed (a 1 in 256 probability). A wrap around of total_loss_count means a  massive loss of packets, so that means that we have other more serious issues and the self-clocking has probably stalled then too.
  *   Sec. 6.2.: broken reference on SREaAM_Cplusplus_Implementation
[IJ] Will fix this, thanks

Thanks,
Xiaoqing

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 12 Jan 2016 15:10:48 +0000

Hi

And thanks for the review.

New year ☺, hopefully comprehensive answers/comments inline below, marked [IJ]

/Ingemar

From: Stefan Holmer [mailto:[email-masked]]
Sent: den 31 december 2015 14:50
To: Karen E. E. Nielsen; Xiaoqing Zhu (xiaoqzhu); Varun Singh; Mo Zanaty (mzanaty)
Cc: Mirja Kühlewind; Ingemar Johansson S; Zaheduzzaman Sarker
Subject: Re: Review of draft-ietf-rmcat-scream-cc-02

These are my comments, and it's still 2015. :)

Overall I think it was possible to understand most of it, although there may be some things that need more detail in order for me to understand why things were designed the way it was. I may also have to be convinced about the benefits of some of the complexity.

Let me know if some of them doesn't make sense. I put my nits last.

/Stefan

"In addition, the grace time, i.e. allowed reaction time
   from the time that the congestion is detected until a reaction in
   terms of a rate reduction is effected, is generally very short, in
   the order of one RTT (Round Trip Time)."

holmer: I don't understand this sentence. What is it exactly that is short? Does the network not queue packets more than one RTT?
[IJ] The queuing delay in the network can be as large as buffers and lack of AQM allows. What I refer to is that (in LTE access): Once the scheduler has reduced the resource allocation for a bearer, an RMCAT flow needs to reduce the sending rate quite quickly (in one RTT)  in order to avoid excessive queuing delay or packet loss.

Self-clocked congestion control algorithm provides with a benefit
   over the rate based counterparts in that the former consists of two
   parts; the congestion window computation that evolves over a longer timescale (several RTTs) especially when the congestion window
   evolution is dictated by estimated delay and;

holmer: Why does estimated delay cause cwnd to evolve slower?
[IJ] CWND modification cannot be tuned to sensitive to delay as the resulting algorithm would then be too vulnerable to jitter.

A rate based congestion control has only one mechanism to adjust the
   sending rate and that makes it more problematic to reach the goal of
   prompt reaction to congestion and also high throughput when channel
   conditions are good.

holmer: Loss and delay are two mechanisms that I can think of? I guess what you want to say is that a rate-based algorithm must detect congestion (which can take time) to be able to adjust the rate. I think this can be communicated more clearly.
[IJ] Yes, need to work on the words better here.

Congestion window validation techniques.  These are similar in
      action as the method described in [I-D.ietf-tcpm-newcwv].  The
      allowed idle period in this draft is shorter than in the
      reference, this to avoid excessive delays in the cases where e.g.
      wireless throughput has decreased during a period where the output
      bitrate has been low.

holmer: Would it make sense to describe shortly what these are here? Currently I would have to open the reference to understand what the following comments about idle period etc. actually mean.
[IJ] Yes, will add a more explanatory text.

Packet pacing is used to mitigate issues with ACK
   compression that may cause increased jitter and/or packet loss in the
   media traffic.

holmer: I think it would be good with a longer explanation of this, or a reference.
[IJ] Can add a better explanation, not sure if there as any good reference, there is work around packet pacing for TCP, and lately QUIC introduced packet pacing.

RTCP packets are received
   (6) and the information about bytes in flight and congestion window
   is exchanged between the network congestion control and the sender
   transmission control (7).

holmer: Is it covered somewhere how feedback from multiple streams are handled?
[IJ] Not directly, the draft was recently trimmed down to concentrate on a one stream case. The program code supports multiple stream handling however. In short the sender keeps a list of transmitted RTP sizes and SSRC, and transmission times. When a feedback is received it gets mapped against the list, this makes it possible to count the bytes in flight and also the transmission rate for each stream

   OWD_TARGET_HI (0.4s)
     Target value for the maximum OWD

holmer: I think it would make sense to explain what the difference is between high and low. Why is there not a single target?
[IJ] This is to set an upper limit to the OWD target value (default 100ms) on case a competing loss based flow over the same bottleneck increases the queuing delay. A fixed OWD target would starve the RCMAT flow, to avoid this, the OWD target to allowed to increase, in this case to a max value of 400ms

   BETA_R (0.9)
     Target rate scale factor due to loss event

holmer: Why does loss affect the target rate? Doesn't seem to follow the logic explained in the previous section.
[IJ] This is to avoid an (unnecessary) increase in RTP queue delay. A loss event means a known reduction in the sending rate, rather than wait the extra time for the media rate control to react to the increased RTP queuing delay, the media rate is here reduced directly. That said, even though it makes sense to reduce the media coder target rate due to detected loss event, I have a hard time to see the difference with and without the rate scaling with BETA_R

 BYTES_IN_FLIGHT_SLACK (10%)
     Additional slack to the congestion window

holmer: Is this constant the same as MAX_BYTES_IN_FLIGHT_HEAD_ROOM (=10%)?
[IJ] No they are not the same, the former allows a little extra to be transmitted beyond the congestion window when little congestion is detected. The latter allows the congestion window to be slightly higher than the number of bytes in flight, the rationale is to avoid unnecessary transmission limitations when the media coder bitrate fluctuates. Note that both both values are set to zero in the code (MAX_BYTES_IN_FLIGHT_HEAD_ROOM=1.0) it is possible that these parameters are no longer needed.

  RATE_ADJUST_INTERVAL (0.2s)
     Interval between media bitrate adjustments

holmer: Is there a reason why the rate only is allowed to change every 200 ms?
[IJ] It is the result of earlier experimentation, a more frequent rate change did not give any benefit, what was see also was that frequent changes seemed to make the video coder rate control loop unstable. This was not studied in detail though but as 200ms was good enough it was selected

   owd_fraction_avg (0.0)
     EWMA filtered owd_fraction

holmer: Doesn't seem to be defined here?
[IJ] Not sure I understand.

   owd_trend_mem (0.0)
     Low pass filtered version of owd_trend

holmer: Does this mean it's not EWMA filtered as owd_fraction_avg?
[IJ] It is computed as a kind of peak-hold VU meter, this means that it keeps a memory over past congestion events to avoid a too aggressive rate increase.

   owd_norm_hist[100] ({0,..,0})
     Vector of the last 100 owd_norm

holmer: owd_norm is not defined here?
[IJ] own_norm is a local variable, calculation is defined in the coder

   rate_rtp (0.0 bps)
     Measured bitrate from the media encoder

holmer: "_rtp" indicates that this is including rtp headers. Is that the case?
[IJ] Not necessarily, should perhaps be renamed to rate_media ?.
Depends on the implementation if RTP headers are included. In OpenWebRTC it is possible to get the size of the RTP payload, thus the rate_rtp is media rate without the RTP headers accounted for.

       # R is an autocorrelation function of owd_fraction_hist
       #  at lag K
       a = R(owd_fraction_hist,1)/R(owd_fraction_hist,0)

holmer: What does 1 and 0 mean here?
[IJ] Explained in section A.2, it is possible that it is not sufficiently explained, please let me know if this is the case.

       #calculate OWD trend
       owd_trend = a*owd_fraction_avg

holmer: Can owd_trend be negative here?
[IJ] Yes, it can but it is clamped to the range [0.0 1.0],

     on loss(owd):
       in_fast_increase = false
       cwnd_last_max = cwnd
       cwnd = max(min_cwnd,cwnd*BETA_LOSS)
       ...

holmer: Seems like on loss(owd) and on ECN(owd) are very similar. Could we reduce the duplication there?
[IJ] Yes, it should be possible

Packets that are lost are also included, which means that even
   though e.g packet N+2 was lost, its size is still included in the
   update of bytes_newly_acked.

holmer: I think this could be explained earlier too, for instance when bytes_newly_acked is defined. It is a bit confusing to say that these bytes are acked, even though they weren't even received.
[IJ] Yes, this has been among the more tricky parts to explain, the closest similarity is the FACK (forward ACK) algorithm in TCP.

holmer: After reading section 4.1.2 I was under the impression that the algorithm was based on packets and not on bytes:
" Unlike TCP, SCReAM is not a byte oriented protocol, rather it is an
RTP packet oriented protocol. Thus a list of transmitted RTP packets
and their respective transmission times (wall-clock time) is kept for
further calculation." But here I get the impression that it's based on bytes?
[IJ] Yes, the algorithm is RTP packet based, however as the congestion window is computed in bytes it becomes necessary to use the term bytes in flight computed as the sum of the sizes of the RTP packets that are in flight.

   update_cwnd(bytes_newly_acked):
     # additional scaling factor to slow down closer to target
     # The min scale factor is 0.2 to avoid that the congestion window
     #  growth is stalled
     scale = max(0.2,min(1.0,(abs(cwnd-cwnd_last_max)/cwnd_i*4)^2))

holmer: Is 0.2 given a name somewhere? Is it the same as for owd_trend >= 0.2 below?
[IJ] No these are not the same, in this particular case the value is to avoid that the congestion window growth is stalled when cwnd==cwnd_last_max, also noted a type above.. should be cwnd_last_max istf cwnd_i
holmer: nit: Move *4 to before abs( to make it clearer that this shouldn't be (cwnd_i*4)
[IJ] OK

     # adapt only increase based on scale
     if (off_target > 0)
       gain *= (1 - owd_trend/ 0.2) * scale

holmer: I don't understand the comment, and can owd_trend be negative here? Is it allowed to be > 0.2?
[IJ] Hmm, could be a bug in the draft. Should actually be  gain *= max(0.0, (1 - owd_trend/ 0.2)) * scale

  SCReAM uses the terminology "Bytes in flight (bytes_in_flight)" which
   is computed as the sum of the sizes...

holmer: I think it would have been useful to have this a bit earlier in the document.
[IJ] OK

The condition to update
   owd_target is fulfilled if owd_norm_var < 0.16 (indicating that the
   standard deviation is less than 0.4).

holmer: It would be good to explain what this is supposed to capture. Do you mean to classify whether the link is congested or not?
[IJ] It indicates that the OWD does not change rapidly, we want to avoid the case that the OWD target is increased due to self-congestion , still we wish to be able to increase the OWD target if the OWD continues to be high. This is a simple function with a certain risk of both false positives and negatives but it manages competing FTP flows reasonably well at the same time as it avoids accidential increases of the target OWD in simulated LTE test cases.

   owd_norm is the OWD divided by OWD_TARGET_LO. owd_norm_mean_sh is the
   short term (last 20 samples) average of owd_norm.  owd_norm_var is
   the variance of owd_norm over the last 100 samples.

holmer: Why is the average computed based on 20 samples and the variance based on 100 samples?
[IJ] This is given by experimentation

A strict rule as the one given above
      will have the effect that the media bitrate will have difficulties
      to increase as the congestion window puts a too hard restriction
      on the media frame size variation.

holmer: I don't follow this. If the frame size is too large for one or two frames, doesn't that just mean that your rtp queue grows? How can this limit the growth of cwnd?
[IJ] It is possible that parts of this text can be modified as BYTES_IN_FLIGHT_SLACK=0.0f in the code and it still works fine

          cwnd_s = max(cwnd*x_cwnd, cwnd+MSS)

holmer: Could you elaborate on why the cwnd_s is at least MSS greater than cwnd? Seems like you never actually enforce the cwnd? Is this where you increase your cwnd on an ack?
[IJ] There are two parts here, one part that applies when the OWD is below the target, in which case the transmission condition is more relaxed. In the other case it gets more strict. This speeds up the media rate. This extra code is not necessary in TCP, difference is that TCP generally deals with large send buffers which SCReAM has to work with rate limited sources.

In addition,
   owd_trend is also considered in the media rate control, this to
   reduce the amount of induced network jitter.

holmer: It would be interesting to see a comparison with and without this feature. My feeling is that this kind of fine tuning may not help much due to the intertia of the encoder. Have you done any experiments?
[IJ] Yes, it is actually possible to remove the dependency on owd_trend. The effect is slightly more jitter, not sure if one will notice the additional jitter though.

The target bitrate can vary between a minimum value
   (target_bitrate_min) and a maximum value (target_bitrate_max).

holmer: What happens if the cwnd is too small to send at target_bitrate_min?
[IJ] RTP packets start to queue up, finally the RTP queue is discarded in its entirety if the delay in the RTP queue becomes too large, ideally the video encoder is informed of this so that the Video coder states can be kept consistent . In general this should be possible to handle better than only a target_bitrate_min , the video encoder can for instance change to a low frame rate. SCReAM is quite agnostic to frame rate, it simply put takes RTP packets and transmits them so there are likely more or less smart things that can be done here.

The rationale behind the use of
   rate_ack in addition to rate_transmit is that rate_transmit is
   affected also by the amount of data that is available to transmit,
   thus a lack of data to transmit can be seen as reduced throughput
   that may itself cause an unnecessary rate reduction.  To overcome
   this shortcoming; rate_ack is used as well.  This gives a more stable
   throughput estimate.

holmer: Maybe note here that this can be the case anyway, for instance if the input video is static for a longer time than ~one RTT.
[IJ] Yes, but this is a slightly different aspect. Especially OpenH264 has the feature that the video rate drops to really low bitrates when input is static. The effect was that the bitrate becomes too limited when input to the video encoder becomes more dynamic. To solve this a median rtp rate is computed over a longer time and this is used to set a more suitable max bitrate (look for rate_rtp_limit)

   The rate change behavior depends on whether a loss event has
   occurred, and if the congestion control is in fast increase or not.

holmer: I find this a bit strange. You are taking the packet spread caused by the network into account here, but not lost packets?
[IJ] Perhaps a problem with the wording, a loss event indicates one or more lost RTP packets in one RTT.

 In the fast increase phase an allowed increment is computed based on
   the congestion level and the relation to target_bitrate_last_max and
   the target_bitrate is reduced further if congestion is detected.

holmer: Maybe mention what variable "congestion level" refers to here? How is it measured?
[IJ] Refers to owd_trend, will try to clarify

A multiplier is applied to
   rate_rtp_limit, depending on congestion history. The target_bitrate
   is then limited by this rate_rtp_limit.

holmer: Explain why a multiplier is needed here.
[IJ] This is to allow the media bitrate to increase quickly in cases when it is long since congestion occurred. In cases where congestion has occurred not too long ago, then the rate_rtp_limit is restricted more to avoid that the media bitrate overshoots. All this is actually to handle the case that the input to the video encoder can change between static and dynamic and was something that was designed as SCReAM was integrated into OpenWebRTC, it is generally nothing that one will see in the simulations.

Thanks for the nits below, will take care of these
Nits:

"reaction in terms of a rate reduction is effected"
holmer: "is in effect"?

"This information
   is then used to determine the amount of bytes can be transmitted at
   any given time instant."

holmer: "that can be transmitted"

"... the e2e latency is kept low."
holmer: "end-to-end latency"

"The fast increase state can be however be resumed if..."
holmer: -be

   o  Addition of media a rate control function.

holmer: "of a media rate control function"

"The SCReAM congestion control method, uses LEDBAT"

holmer: -,

   owd_target (OWD_TARGET_LO)
     OWD target

holmer: Maybe write "One-way delay target" instead?

 The wall clock timestamp corresponding to the received RTP packet
      with he highest sequence number.

holmer: "...with the highest sequence number."

       # R is an autocorrelation function of owd_fraction_hist
       #  at lag K

holmer: What is K?

"This vector is used in the
   computation of an OWD trend"

holmer: I _think_ this should be "of a OWD trend"?

   o  [RFC6817] specifies that the CWND increased is limited by an

holmer: increase

"SCReAM takes care of such situations by adjusting the owr_target."

holmer: owd_target

       owd_norm_var = VARIATION(owd_norm_history(100))

holmer: VARIANCE()?
[IJ] Correct

On Thu, Nov 19, 2015 at 12:31 PM Karen Elisabeth Egede Nielsen <[email-masked]<mailto:[email-masked]>> wrote:
Hi Xiaoging, Stefan, Varun, Mo

This is a for the record email reminding you that
you have promised to provided comments on draft-ietf-rmcat-scream-cc-02
before the end of this year. I.e., comments should be provided
to the authors and the RMCAT list latest by December 31.

Note:
The document version in question does not include a separate section
uniquely dedicated to describing and discussing
the requirements for the feedback message, but the information should  generally be available in the document.
It is recognized that the next version of the document will include a separate section for collecting the information given
on this topic. You may take this into consideration when doing the review.

BR Karen/Mirja