Subject: [rmcat] Review of draft-ietf-rmcat-eval-test-02

From: Safiqul Islam <[email-masked]>
Date: Wed, 27 Jan 2016 15:47:20 +0000

Hi all,

Here are my comments for eval-test draft. The draft provides sufficient details of the basic test scenarios to evaluate the candidate algorithms.

General comments:

  *   The abstract can clarify the scope of the test scenarios.
  *   The document uses on the following terms interchangeably to represent RMCAT flows. I would like to suggest using only one of them in the document to make it consistent:

     *    Media flow
     *    RMCAT flow
     *    RMCAT media flow

  *   The document mentions “it is expected to log enough information, e.g., Page 14, section 5.3,  “To evaluate the performance of the candidate algorithms it is expected to log enough information “
     *   what is “it” ? It needs to be clarified.
  *   The document suggests to use bit rate range 150 kbps - 1.5 Mbps. I think the upper limit should be increased to match with  real-life applications.

Specific, miscellaneous comments/questions:

  *   Page 3, section 3, Expected behavior:,  describe -> describes
  *   I suggest using either behaviour or behavior to make it consistent.
  *   Page 3, last line, The following basic structure of test case -> of a test case
  *   Page 4. line 2, I suggest using a different verb instead of “flow" to avoid confusion with “flows"
  *   Page 4, line 11, The following sentence needs to be rephrased.

It should be noted that depending on the test cases it is possible to have different path characteristics in of the either directions.

  *   Page 7, section 4.1,  metircs -> metrics
  *   Page 9, section 4.3,  The following sentence needs to be revised.

"Also it is possible that the media traffic generator used in a particular  simulator or testbed if not capable of generating higher bitrate.”

  *   Page 10,  line 2, unless otherwise specified in the test case -> unless otherwise specified in the test case(s)
  *   Page 10, an optimal startup  bitrate values -> an optimal startup  bitrate value
  *   Page 10, section 5.1, maximum Media Bit Rate is Greater than Link Capacity -> maximum media bit rate is greater than link capacity
  *   Page 11, paragraph 2, the following sentence needs to be rephrased to enhance clarity.

        The oscillations occur when the media flow(s) attempts to reach its maximum bit rate, overshoots the usage of the available bottleneck capacity, to rectify it reduces the bit rate and starts to ramp up again.

  *   Page 12, the following sentence can be broken into smaller sentences to enhance clarity.

When using background non-adaptive UDP traffic to induce time-varying bottleneck for the RMCAT flow, the physical path capacity is 4Mbps and the UDP traffic source rate changes over  time as (4-x)Mbps, where x is the bottleneck capacity specified in Table 1

  *   Page 13, at a fine enough time granularity: -> at a fine enough time granularity.
  *   Page 14, i.e, when the backward channel has no impairments -> i.e, when the backward channel has no impairments.
  *   Page 14, section 5.3, at a fine-grained time intervals: -> at a fine-grained time interval.
  *   Page 16, section 5.4, more than one RMCAT media flow shares the -> more than one RMCAT media flow share the

  *   Page 17,  connected to respective R1, R2, R3. -> connected to their respective destinations R1, R2, R3.

  *   Page 19,  It is expected that the competing flows will converge to bit rates to ->  It is expected that the competing flows will converge to optimum bit rates to
  *   Page 19, at a fine enough time granularity: -> at a fine enough time granularity.
  *   Page 19,  One-Way propagation delay for each flow: 10ms, 25ms, 50ms, 100ms, 150ms. -> One-Way propagation delay for each flow: 10ms, 25ms, 50ms, 100ms, and 150ms respectively.
  *   Page 20, section 5.6, at least one long lived TCP flows -> at least one long lived TCP flow
  *   Page 21, connected to corresponding media sink, R1 -> connected to its corresponding media sink, R1
  *   Page 21,  The TCP traffic goes over the forward path from, S_tcp with acknowledgement packets flowing along the backward path from, R_tcp ->  The TCP traffic goes over the forward path from S_tcp, and corresponding acknowledgement packets are transmitted over the backward path from R_tcp.
  *   Page 22, one (1), long-lived TCP -> one (1) and long-lived TCP
  *   Page 22, Congestion control: Default TCP congestion control. -> Congestion control: default TCP congestion control.
  *   Page 22, Test Specific Information: None -> Test Specific Information: none
  *   Page 23, This scenario shows the performance of the multimedia application -> This scenario shows the performance of a multimedia application
  *   Page 23, Media Source:  - > Media source:
  *   Page 24,  The following sentence needs to be revised:
     *   Not all short TCPs start at the same time, 2 start in the ON state while 8 start in an OFF stats.
  *   Page 25, section 5.8, At these stage one of the media flow is -> At this stage, one of the media flows is
  *   Page 25, section 5.8, flows and as they -> flows as they
  *   Page 25. section 5.8,  The following sentence s

When the paused media flow will resume it would no longer have the same bandwidth share on the link.  It has to make  it's way through the other existing flows in the link to achieve a fair share of the link capacity.

  *   Page 25, section 5.8, The general description of the testbed parameters are -> The general description of the testbed parameters is
  *   Page 26, section 6.1, the candidate algorithms does not fail -> the candidate algorithms do not fail
  *   Page 26, section 6.2, In this test case one RMCAT flow, S1->R2 traverse a path -> In this test case one RMCAT flow, S1->R2 traverses a path
  *   Page 27, connected to respective destinations R1, R2, and R3. -> connected to their respective destinations R1, R2, and R3.

  *   I would like to suggest the following two additional test-cases under section 6 (other potential test cases).

6.3 Prioritization test

In this test case, three RMCAT media flows started  out with a priority of 1 each. At around 50 s, the priorities of flow 2 and flow 3 were decreased to 1/3 and 2/3.  A flow i  is assigned the available capacity based on its priority p_{i} as follows:

b_{i} = (p_{i} / Sum(p_{k}) ) * BW   ;  k = {1.. 3}

This test case ensures that flows are getting their assigned portions of the available capacity based on their assigned priorities. It can be useful for [I-D.ietf-rmcat-coupled-cc]. It addresses the requirement 3A in  [I-D.ietf-rmcat-cc-requirements]

Testbed topology: Same as test case defined in section 5.4

Testbed Attribute:

The general description of the testbed parameters is same as Section 5.4 with changes in the test specific setup as  below-

  o  Other test specific setup:

      *  RMCAT media flow timeline:

         +  Flow ID: One (1)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Priority: no change

     *  RMCAT media flow timeline:

         +  Flow ID: Two (2)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Priority:  change to 1/3 after 50s

      *  RMCAT media flow timeline:

         +  Flow ID: Three (3)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Priority: change to 2/3 after  50s

6.4 RMCAT flows competing with one or more long TCP flows

This test case is an extension of the test case defined in section 5.6. It is important for [I-D.ietf-rmcat-coupled-cc], where it requires more than one RMCAT flows in order to couple them.

The general description of the testbed parameters is same as section 5.6.

Regards,
Safiqul

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 28 Jan 2016 14:29:17 +0100

Hi Safiqul,

Thanks for your review of the documents.

please see inline below.

BR

Zahed

On 2016-01-27 16:47, Safiqul Islam wrote:
> Hi all,
>
> Here are my comments for eval-test draft. The draft provides sufficient
> details of the basic test scenarios to evaluate the candidate algorithms.
>
> General comments:
>

could you explain a but more what needs to be clarified?

>   * The document uses on the following terms interchangeably to
>     represent RMCAT flows. I would like to suggest using only one of
>     them in the document to make it consistent:
>
>      1.   Media flow
>      2.   RMCAT flow
>      3.   RMCAT media flow

Good point, I think we need a terminology section to clarify this.

>
>   * The document mentions “it is expected to log enough information,
>     e.g., Page 14, section 5.3,  “To evaluate the performance of the
>     candidate algorithms it is expected to log enough information “
>       o what is “it” ? It needs to be clarified.

I can also see the doc mentions "to visualize the metrics described in 
Section 4.1". So the document points to number of important metrics. As 
the document does not dictate in what environment the tests will be 
performed (simulator or testbed, real implementation or simulation), it 
is very hard to say how and what can be logged. I believe that is an 
implementation details that we don't need to describe here. But the text 
can be improved to clarify this.

>   * The document suggests to use bit rate range 150 kbps - 1.5 Mbps. I
>     think the upper limit should be increased to match with  real-life
>     applications.

I think we have discussed this in the past and settled with this bitrate 
range for the test cases. in section 4.3 the text explains this.

> It should be noted that depending on the test cases it is possible to
> have different path characteristics in of the either directions.
>
> "Also it is possible that the media traffic generator used in a
> particular  simulator or testbed if not capable of generating higher
> bitrate.”
>
>   * Page 10,  line 2, unless otherwise specified in the test case ->
>     unless otherwise specified in the test case(s)
>   * Page 10, section 5.1, maximum Media Bit Rate is Greater than Link
>     Capacity -> maximum media bit rate is greater than link capacity
> The oscillations occur when the media flow(s) attempts to reach its
> maximum bit rate, overshoots the usage of the available bottleneck
> capacity, to rectify it reduces the bit rate and starts to ramp up again.

ok

> When using background non-adaptive UDP traffic to induce time-varying
> bottleneck for the RMCAT flow, the physical path capacity is 4Mbps and
> the UDP traffic source rate changes over  time as (4-x)Mbps, where x is
> the bottleneck capacity specified in Table 1

will try.

>   * Page 14, i.e, when the backward channel has no impairments -> i.e,
>     when the backward channel has no impairments.
>   * Page 16, section 5.4, more than one RMCAT media flow shares the
>     -> more than one RMCAT media flow share the
>
>   * Page 19,  It is expected that the competing flows will converge to
>     bit rates to ->  It is expected that the competing flows will
>     converge to optimum bit rates to
>   * Page 19,  One-Way propagation delay for each flow: 10ms, 25ms, 50ms,
>     100ms, 150ms. -> One-Way propagation delay for each flow: 10ms,
>     25ms, 50ms, 100ms, and 150ms respectively.
>   * Page 21,  The TCP traffic goes over the forward path from, S_tcp
>     with acknowledgement packets flowing along the backward path from,
>     R_tcp ->  The TCP traffic goes over the forward path from S_tcp, and
>     corresponding acknowledgement packets are transmitted over the
>     backward path from R_tcp.
>   * Page 22, Congestion control: Default TCP congestion control.
>     -> Congestion control: default TCP congestion control.
>   * Page 23, This scenario shows the performance of the multimedia
>     application -> This scenario shows the performance of a multimedia
>     application
>       o Not all short TCPs start at the same time, 2 start in the ON
>         state while 8 start in an OFF stats.
>   * Page 25, section 5.8, At these stage one of the media flow is
>     -> At this stage, one of the media flows is

?? what is your comment here?
>
> When the paused media flow will resume it would no longer have the same
> bandwidth share on the link.  It has to make  it's way through the other
> existing flows in the link to achieve a fair share of the link capacity.

>
>   * Page 25, section 5.8, The general description of the testbed
>     parameters are -> The general description of the testbed parameters is
>   * Page 26, section 6.2, In this test case one RMCAT flow, S1->R2
>     traverse a path -> In this test case one RMCAT flow, S1->R2
>     traverses a path
>   * Page 27, connected to respective destinations R1, R2, and R3. ->
>     connected to their respective destinations R1, R2, and R3.
>
>   * I would like to suggest the following two additional test-cases
>     under section 6 (other potential test cases).

The test cases in the draft were designed for end-point adaptation 
algorithm without coupled congestion control in mind i.e. no need for 
shared bottleneck detection and flow priority. In the last IETF meeting 
we talked about the possibility of extending the test cases to include 
coupled congestion control algorithm.

Now if we extend the test cases then I don't think only adding the two 
proposed test cases will be sufficient. We will need to identify all the 
test cases those are suitable for running with coupled congestion 
control. Like 5.2 can be run with priority per flow. By doing this we 
can mention (in each test case or in one place in the document) that one 
can run tests with flow priority and perhaps give some means to evaluate 
the available capacity distribution with some equation like the one 
mentioned here in the proposed test case 6.3. However, then this 
document also needs to include description for shared bottleneck 
detection. In the simplest form, we can perhaps assume that the 
identified test cases have single bottleneck point.

I think we need to discuss what is best possible way to extend the test 
cases if we agreed to do that.

But before that I would like to know if the WG is OK with extending the 
test cases for coupled congestion control.

And will it be a compulsory (candidate algorithms MUST present results) 
or optional (candidate algorithms MAY present results) part of the test 
case.

>
> 6.3 Prioritization test
>
> In this test case, three RMCAT media flows started  out with a priority
> of 1 each. At around 50 s, the priorities of flow 2 and flow 3 were
> decreased to 1/3 and 2/3.  A flow i  is assigned the available capacity
> based on its priority p_{i} as follows:
>
> This test case ensures that flows are getting their assigned portions of
> the available capacity based on their assigned priorities. It can be
> useful for [I-D.ietf-rmcat-coupled-cc]. It addresses the requirement 3A
> in  [I-D.ietf-rmcat-cc-requirements]
>
> The general description of the testbed parameters is same as Section 5.4
> with changes in the test specific setup as  below-
>
> This test case is an extension of the test case defined in section 5.6.
> It is important for [I-D.ietf-rmcat-coupled-cc], where it requires more
> than one RMCAT flows in order to couple them.
>

-- 
Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Services, Media and Network Features
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com

==================================================

---

From: Safiqul Islam <[email-masked]>
Date: Thu, 28 Jan 2016 16:53:35 +0000

Hi Zahed,

Comments inline.

On 28. jan. 2016, at 14.29, Zaheduzzaman Sarker <[email-masked]<mailto:[email-masked]>> wrote:

Hi Safiqul,

please see inline below.

BR

Zahed

General comments:

if you mention that the test cases cover the basic usage scenarios and can easily be extended, it will add value for the reader.

    1.   Media flow
    2.   RMCAT flow
    3.   RMCAT media flow

I can also see the doc mentions "to visualize the metrics described in Section 4.1". So the document points to number of important metrics. As the document does not dictate in what environment the tests will be performed (simulator or testbed, real implementation or simulation), it is very hard to say how and what can be logged. I believe that is an implementation details that we don't need to describe here. But the text can be improved to clarify this.

I think we have discussed this in the past and settled with this bitrate range for the test cases. in section 4.3 the text explains this.

OK.

 * Page 3, section 3, Expected behavior:,  describe -> describes
 * I suggest using either behaviour or behavior to make it consistent.
 * Page 3, last line, The following basic structure of test case -> of
   a test case
 * Page 4. line 2, I suggest using a different verb instead of “flow"
   to avoid confusion with “flows"
 * Page 4, line 11, The following sentence needs to be rephrased.

 * Page 10,  line 2, unless otherwise specified in the test case ->
   unless otherwise specified in the test case(s)
 * Page 10, an optimal startup  bitrate values -> an optimal startup
     bitrate value
 * Page 10, section 5.1, maximum Media Bit Rate is Greater than Link
   Capacity -> maximum media bit rate is greater than link capacity
 * Page 11, paragraph 2, the following sentence needs to be rephrased
   to enhance clarity.

ok

 * Page 12, the following sentence can be broken into smaller sentences
   to enhance clarity.

will try.

 * Page 13, at a fine enough time granularity: -> at a fine enough time
   granularity.
 * Page 14, i.e, when the backward channel has no impairments -> i.e,
   when the backward channel has no impairments.
 * Page 14, section 5.3, at a fine-grained time intervals: -> at a
   fine-grained time interval.
 * Page 16, section 5.4, more than one RMCAT media flow shares the
   -> more than one RMCAT media flow share the

 * Page 17,  connected to respective R1, R2, R3. -> connected to their
   respective destinations R1, R2, R3.

 * Page 19, at a fine enough time granularity: -> at a fine enough time
   granularity.
 * Page 20, section 5.6, at least one long lived TCP flows -> at least
   one long lived TCP flow
 * Page 21, connected to corresponding media sink, R1 -> connected
   to its corresponding media sink, R1
 * Page 22, Test Specific Information: None -> Test Specific
   Information: none

So sorry, I forgot to complete the sentence. The following sentences should be revised to enhance clarity.

 * Page 25, section 5.8, The general description of the testbed
   parameters are -> The general description of the testbed parameters is
 * Page 26, section 6.1, the candidate algorithms does not fail -> the
   candidate algorithms do not fail
 * Page 26, section 6.2, In this test case one RMCAT flow, S1->R2
   traverse a path -> In this test case one RMCAT flow, S1->R2
   traverses a path
 * Page 27, connected to respective destinations R1, R2, and R3. ->
   connected to their respective destinations R1, R2, and R3.

The test cases in the draft were designed for end-point adaptation algorithm without coupled congestion control in mind i.e. no need for shared bottleneck detection and flow priority. In the last IETF meeting we talked about the possibility of extending the test cases to include coupled congestion control algorithm.

Now if we extend the test cases then I don't think only adding the two proposed test cases will be sufficient. We will need to identify all the test cases those are suitable for running with coupled congestion control. Like 5.2 can be run with priority per flow. By doing this we can mention (in each test case or in one place in the document) that one can run tests with flow priority and perhaps give some means to evaluate the available capacity distribution with some equation like the one mentioned here in the proposed test case 6.3. However, then this document also needs to include description for shared bottleneck detection.

I think showing prioritization results with test-case 5.4 should be sufficient.

In the simplest form, we can perhaps assume that the identified test cases have single bottleneck point.

+1

I think we need to discuss what is best possible way to extend the test cases if we agreed to do that.

But before that I would like to know if the WG is OK with extending the test cases for coupled congestion control.

And will it be a compulsory (candidate algorithms MUST present results) or optional (candidate algorithms MAY present results) part of the test case.

The two test cases are written explicitly to highlight the features of coupled congestion control where we needed to play around with more than one flow and priorities.
In my opinion, the rest of the test cases can remain as they are, and have no direct effect from coupled congestion control. That’s why I recommended to
add them in section 6 (other potential test cases).

6.3 Prioritization test

Testbed Attribute:

         +  Flow ID: One (1)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Priority: no change

         +  Flow ID: Two (2)

         +  Start time: 0s

         +  Flow duration: 119s

         +  Flow ID: Three (3)

         +  Start time: 0s

         +  Flow duration: 119s

Regards,
Safiqul

Cheers,
Safiqul