Subject: [rmcat] Sections 5.1 and 5.2 of draft-ietf-rmcat-eval-test-01

From: Sergio Mena <[email-masked]>
Date: Thu, 13 Aug 2015 14:29:59 +0200

Zahed,

/[CCed the author list of the rmcat test cases draft, as well as the 
rmcat mailing list]/

There was a discussion triggered at rmcat main session in Prague on how 
to simulate bottleneck-link bandwidth changes. Two options were 
discussed (1) changing the physical link throughput (currently specified 
in the draft), and (2) using an insensitive CBR UDP flow as background 
traffic.

My understanding from the discussion during the session was that none of 
these two options can simulate the other and, therefore, we somehow need 
to test how candidate algorithms adapt in both cases.

You and I then had an offline discussion on the subject in which you 
suggested me to review sections 5.1 and 5.2 of the test cases draft, and 
propose an alternate text to extend these two test cases to the 
background traffic option.

I have reviewed sections 5.1 and 5.2 and come up with a proposal for 
extending them to using insensitive background traffic, in addition to 
the current option in the text (physical link throughput change).

Please find below what I would like to propose as updated text for 5.1 
and 5.2. [Text in square brackets are instructions to locate and replace 
the original text].

Thanks,

Sergio Mena

PS Proposed changes:

===========
Section 5.1
===========

[In the three-bullet list after the first paragraph, change the text in 
the second bullet from:]

     o change in available capacity (e.g., due to interface change,
       routing change).

[to]

     o change in available capacity (e.g., due to interface change,
       routing change, sharp change in background traffic).

---

[Replace the paragraph immediately after the three-bullet list mentioned 
above, the one that starts with]

It should be noted that the exact variation...

[with the following]

The test case defined in this section (5.1) is divided into two subcases 
5.1a and 5.1b. The main
difference is how the bottleneck-link capacity is changed in the 
testbed. In 5.1a, the goal is to
simulate a modification in the underlying physical properties, such as 
routing change or interface
change. In 5.1b, the goal is to simulate a sharp change in background 
traffic to which the candidate
algorithm is to adapt.

---

[Change the last item of testbed attributes from]

       *  Competing traffic:

          +  Number of sources : Zero (0)

[to]

       *  Competing traffic:

          +  In 5.1a

             - Number of sources : Zero (0)

          +  In 5.1b

             - Number of sources : One (1)

             - Type of sources : CBR traffic

             - Sending bitrate : See test specific information below.

             - Traffic direction : Forward

             - Congestion control : None (insensitive traffic)

             - Traffic timeline:

               - Start time : 0s

               - End time : 100s

---

[Change 2nd bullet of test-specific information from]

       *  This test uses bottleneck path capacity variation as listed in
          Table 1

[to]

       *  This test uses bottleneck path capacity variation as listed in
          Table 1.

          - In 5.1a, the path capacity transitions are achieved by 
instantly changing the
            physical throughput of the bottleneck link. For simplicity, 
when changing
            the bottleneck link throughput, other parameters such as 
queue length in
            packets will be left unchanged, even if it implies that the 
new queue has a
            different length in milliseconds.

          - In 5.1b, the path capacity transitions are achieved by 
instantly changing the
            sending bitrate of the competing CBR flow. The new sending 
bitrate must be
            the difference between the bottleneck link-capacity (4 Mbps, 
as specified in
            4.2) and the capacity specified in Table 1.

            The sharp bitrate change in the CBR flow might imply the 
losses on both the
            application-related traffic and the CBR flow. As a result, 
the reaction of
            the candidate algorithm might be different in 5.1a and in 
5.1b, hence the
            distinction between the two sub-cases.

===========
Section 5.2
===========

[Insert this after the first paragraph of the section]

Analogously to Section 5.1, the present test case (5.2) is divided into 
two subcases 5.2a and 5.2b.
The main difference is the same: 5.2a simulates a change in the 
underlying physical network properties,
such as routing change or interface change; whereas 5.2b simulates a 
sharp change in background
competing traffic.

---

[Change last paragraph from]

    Test Specific Information: This test uses path capacity variation as
    listed in Table 2 with a corresponding end time of 125 seconds.

[to]

    Test Specific Information: This test uses path capacity variation as
    listed in Table 2 with a corresponding end time of 125 seconds. The
    two media flows stop at time 124.

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Mon, 24 Aug 2015 11:45:49 +0000

Hi Sergio,

Thanks for your effort.

I think the proposed changes looks good.

As we have plan to update the variation of path capacity relative to initial path capacity for test case 5.1 and test case 5.2, we need to change the respective text in the 5.1b and 5.2b to be aligned with that.

One clarification question : in 5.1b what does the “insensitive traffic” means in the new text?

BR

Zahed

From: Sergio Mena [mailto:[email-masked]]
Sent: den 13 augusti 2015 14:30
To: Zaheduzzaman Sarker
Cc: [email-masked]; [email-masked]
Subject: Sections 5.1 and 5.2 of draft-ietf-rmcat-eval-test-01

Zahed,

[CCed the author list of the rmcat test cases draft, as well as the rmcat mailing list]

There was a discussion triggered at rmcat main session in Prague on how to simulate bottleneck-link bandwidth changes. Two options were discussed (1) changing the physical link throughput (currently specified in the draft), and (2) using an insensitive CBR UDP flow as background traffic.

My understanding from the discussion during the session was that none of these two options can simulate the other and, therefore, we somehow need to test how candidate algorithms adapt in both cases.

You and I then had an offline discussion on the subject in which you suggested me to review sections 5.1 and 5.2 of the test cases draft, and propose an alternate text to extend these two test cases to the background traffic option.

I have reviewed sections 5.1 and 5.2 and come up with a proposal for extending them to using insensitive background traffic, in addition to the current option in the text (physical link throughput change).

Please find below what I would like to propose as updated text for 5.1 and 5.2. [Text in square brackets are instructions to locate and replace the original text].

Thanks,

Sergio Mena

PS Proposed changes:

===========
Section 5.1
===========

[In the three-bullet list after the first paragraph, change the text in the second bullet from:]

[to]

---

[Replace the paragraph immediately after the three-bullet list mentioned above, the one that starts with]

[with the following]

The test case defined in this section (5.1) is divided into two subcases 5.1a and 5.1b. The main
difference is how the bottleneck-link capacity is changed in the testbed. In 5.1a, the goal is to
simulate a modification in the underlying physical properties, such as routing change or interface
change. In 5.1b, the goal is to simulate a sharp change in background traffic to which the candidate
algorithm is to adapt.

---

      *  Competing traffic:

[to]

      *  Competing traffic:

         +  In 5.1a

         +  In 5.1b

            - Traffic timeline:

              - Start time : 0s

              - End time : 100s

---

[to]

         - In 5.1a, the path capacity transitions are achieved by instantly changing the
           physical throughput of the bottleneck link. For simplicity, when changing
           the bottleneck link throughput, other parameters such as queue length in
           packets will be left unchanged, even if it implies that the new queue has a
           different length in milliseconds.

         - In 5.1b, the path capacity transitions are achieved by instantly changing the
           sending bitrate of the competing CBR flow. The new sending bitrate must be
           the difference between the bottleneck link-capacity (4 Mbps, as specified in
           4.2) and the capacity specified in Table 1.

           The sharp bitrate change in the CBR flow might imply the losses on both the
           application-related traffic and the CBR flow. As a result, the reaction of
           the candidate algorithm might be different in 5.1a and in 5.1b, hence the
           distinction between the two sub-cases.

===========
Section 5.2
===========

Analogously to Section 5.1, the present test case (5.2) is divided into two subcases 5.2a and 5.2b.
The main difference is the same: 5.2a simulates a change in the underlying physical network properties,
such as routing change or interface change; whereas 5.2b simulates a sharp change in background
competing traffic.

---

[to]

---

From: Sergio Mena <[email-masked]>
Date: Wed, 26 Aug 2015 09:35:37 +0200

Hi Zahed,

First of all, your question. By "insensitive traffic" I meant that the 
flow is CBR, so it won't back off whether other traffic is present in 
the bottleneck link or not. Maybe we can find a better name for this...

On the other hand, in the last days, I have been working on an ns3 
implementation of some of the test cases in the draft. I realized that 
it would be coherent (and useful, IMO) for 5.3 to also have a "b" 
version where both forward and backward paths are changed by a CBR, 
similarly to 5.1 and 5.2. Let me know your thoughts on this.

Regards,

Sergio

On 24/08/15 13:45, Zaheduzzaman Sarker wrote:
> As we have plan to update the variation of path capacity relative to 
> initial path capacity for test case 5.1 and test case 5.2, we need to 
> change the respective text in the 5.1b and 5.2b to be aligned with that.
>
> *From:*Sergio Mena [mailto:[email-masked]]
> *Sent:* den 13 augusti 2015 14:30
> *To:* Zaheduzzaman Sarker
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Sections 5.1 and 5.2 of draft-ietf-rmcat-eval-test-01
>
> Zahed,
>
> There was a discussion triggered at rmcat main session in Prague on 
> how to simulate bottleneck-link bandwidth changes. Two options were 
> discussed (1) changing the physical link throughput (currently 
> specified in the draft), and (2) using an insensitive CBR UDP flow as 
> background traffic.
>
> My understanding from the discussion during the session was that none 
> of these two options can simulate the other and, therefore, we somehow 
> need to test how candidate algorithms adapt in both cases.
>
> suggested me to review sections 5.1 and 5.2 of the test cases draft, 
> and propose an alternate text to extend these two test cases to the 
> and 5.2. [Text in square brackets are instructions to locate and 
> replace the original text].
>
> Thanks,
>
> Sergio Mena
>
>
> PS Proposed changes:
>
> ===========
> Section 5.1
> ===========
>
> [Replace the paragraph immediately after the three-bullet list 
> mentioned above, the one that starts with]
>

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Wed, 26 Aug 2015 15:08:27 +0000

I concur with Sergio’s suggestions.

One thought: instead of stating variation within each sub-section, we can state that there exists two options of changing bandwidth in the beginning of Sec 5. This will also be consistent with the option/flexibility we now offer for folks to pick different initial capacities and scale rest of the BW values accordingly.

Best,
Xiaoqing

From: "Sergio Mena de la Cruz (semena)" <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, August 26, 2015 at 12:35 AM
To: Zaheduzzaman Sarker <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: Sections 5.1 and 5.2 of draft-ietf-rmcat-eval-test-01

Hi Zahed,

First of all, your question. By "insensitive traffic" I meant that the flow is CBR, so it won't back off whether other traffic is present in the bottleneck link or not. Maybe we can find a better name for this...

On the other hand, in the last days, I have been working on an ns3 implementation of some of the test cases in the draft. I realized that it would be coherent (and useful, IMO) for 5.3 to also have a "b" version where both forward and backward paths are changed by a CBR, similarly to 5.1 and 5.2. Let me know your thoughts on this.

Regards,

Sergio

Thanks for your effort.

BR

Zahed

From: Sergio Mena [mailto:[email-masked]]
Sent: den 13 augusti 2015 14:30
To: Zaheduzzaman Sarker
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Sections 5.1 and 5.2 of draft-ietf-rmcat-eval-test-01

Zahed,

Thanks,

Sergio Mena

PS Proposed changes:

===========
Section 5.1
===========

[to]

---

[with the following]

---

      *  Competing traffic:

[to]

      *  Competing traffic:

         +  In 5.1a

         +  In 5.1b

            - Traffic timeline:

              - Start time : 0s

              - End time : 100s

---

[to]

===========
Section 5.2
===========

---

[to]

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 27 Aug 2015 10:42:55 +0200

> the bottleneck link or not. Maybe we can find a better name for this...
ok, understood. To me "flow is CBR" is good enough.
> similarly to 5.1 and 5.2. Let me know your thoughts on this.
is it possible to share your findings so that we can also have a look at 
the characteristics the new test option (path capacity limitation by 
introducing CBR) is revealing (applicable for 5.1 and 5.2 as well)? I 
think it is better to discuss and learn more about the test case 
characteristics before we add the new option.

For test case 5.3, it is important that the path capacity change in the 
backward path happens at the right time so that we can see the effect of 
lost or delayed feedback in the forward path. have you seen any change 
required at the time one need to start the CBR traffic to introduce the 
capacity change at correct time? How easy will it be to do this in a lab 
network?

For 5.1, 5.2 (and 5.3), how do you introduce the competing CBR traffic? 
if there is a single bottleneck node between A and B then the competing 
CBR traffic can share the same path as the media traffic. However, if 
there are multiple nodes between A and B (can happen in the lab 
network), then sharing the same path can create multiple bottlenecks, 
right? I haven't tried that yet. Even if that is the case then I think 
this can be solved by some additional text in the draft but it will 
require different topology than the current assumptions.

BR
-- 
Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Services, Media and Network Features
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com

==================================================

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 27 Aug 2015 10:48:16 +0200

Hi all,

just a quick thought in addition (as an individual): if you use CBR cross traffic, it would potentially also be interesting to add another evaluation metric and that’s the loss rate of the CBR traffic because this tells you how 'aggressive' your impact on cross traffic is.

Mirja

> Am 27.08.2015 um 10:42 schrieb Zaheduzzaman Sarker <[email-masked]>:
> 
> 
> 
> is it possible to share your findings so that we can also have a look at the characteristics the new test option (path capacity limitation by introducing CBR) is revealing (applicable for 5.1 and 5.2 as well)? I think it is better to discuss and learn more about the test case characteristics before we add the new option.
> 
> For test case 5.3, it is important that the path capacity change in the backward path happens at the right time so that we can see the effect of lost or delayed feedback in the forward path. have you seen any change required at the time one need to start the CBR traffic to introduce the capacity change at correct time? How easy will it be to do this in a lab network?
> 
> For 5.1, 5.2 (and 5.3), how do you introduce the competing CBR traffic? if there is a single bottleneck node between A and B then the competing CBR traffic can share the same path as the media traffic. However, if there are multiple nodes between A and B (can happen in the lab network), then sharing the same path can create multiple bottlenecks, right? I haven't tried that yet. Even if that is the case then I think this can be solved by some additional text in the draft but it will require different topology than the current assumptions.
> 
> BR
> -- 
> Zahed
>

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 27 Aug 2015 12:51:40 +0200

On 2015-08-27 10:48, Mirja Kühlewind wrote:
> Hi all,
>
> just a quick thought in addition (as an individual): if you use CBR
> cross traffic, it would potentially also be interesting to add
> another evaluation metric and that’s the loss rate of the CBR traffic
> because this tells you how 'aggressive' your impact on cross traffic
> is.
I think this is a very good point. Along with loss rate we might want to 
look at the goodput of the CBR traffic.

BR
-- 
Zahed

Ericsson AB

---

From: Sergio Mena <[email-masked]>
Date: Thu, 27 Aug 2015 14:05:15 +0200

Hi Zahed,

Please see inline [SM]

Regards,

Sergio

On 27/08/15 10:42, Zaheduzzaman Sarker wrote:
> is it possible to share your findings so that we can also have a look 
> at the characteristics the new test option (path capacity limitation 
> by introducing CBR) is revealing (applicable for 5.1 and 5.2 as well)? 
> I think it is better to discuss and learn more about the test case 

[SM] The main reasons why I raised the question (during rmcat session in 
Prague, and later on in this mailing list) of using CBR traffic to limit 
the bottleneck capacity are the following.

  * When one changes the nominal (i.e., physical) line rate of a network 
link (the bottleneck link in our case), there are other important 
parameters that are affected. For instance, packet transmission delay 
changes. Also, importantly enough for CC, the queue sizes in ms will 
also change (since queue sizes are usually specified in terms of packets 
or bytes in real networks).

  * Another important reason is that, in the wild, it is likely that a 
CC algorithm needs to react to abrupt changes in background traffic, 
roughly as likely as it needs to react to changes in topology. And, IMO, 
the test cases you guys are proposing should cover most use cases that 
are likely to occur in real life.

>
> For test case 5.3, it is important that the path capacity change in 
> the backward path happens at the right time so that we can see the 
> effect of lost or delayed feedback in the forward path. have you seen 
> any change required at the time one need to start the CBR traffic to 
> introduce the capacity change at correct time? How easy will it be to 
> do this in a lab network?

[SM] So far, I did not need to change the specified times for the 
backward path in 5.3, in order to expose the features of the CC 
algorithm that this test case is supposed to check (mainly congested 
feedback paths). Bear in mind that the CBR does not ramp up: it starts 
right at the CBR rate. So I think the times specified in the draft are 
fine as they are. Of course, my implementation of the tests is work in 
progress, so I'll let you know quickly if I observe anything else.

I think it would be relatively easy to implement CBR in a real testbed 
(it is simply a process that starts at a given time and sends UDP 
packets at the specified rate right from the beginning). So it takes in 
the order of 100s of ms (1/2 the rtt) for the whole path to "feel" it; 
this is not a problem for 5.3 where times are specified in the order of 
10s of seconds.

>
> For 5.1, 5.2 (and 5.3), how do you introduce the competing CBR 
> traffic? if there is a single bottleneck node between A and B then the 
> competing CBR traffic can share the same path as the media traffic. 
> However, if there are multiple nodes between A and B (can happen in 
> the lab network), then sharing the same path can create multiple 
> bottlenecks, right? I haven't tried that yet. Even if that is the case 
> then I think this can be solved by some additional text in the draft 
> but it will require different topology than the current assumptions.

[SM] The way I understand it: if there are other hops in the real 
testbed we use to run the rmcat tests, the non-bottleneck links should 
have a capacity so high that the particular value of the capacity does 
not matter. Otherwise, there may be losses/queuing delay at places other 
than the bottleneck, /even in the absence of CBR traffic/. If that was 
the case, then the full topology should be specified in the draft, so 
that the results are reproducible (but I do not think that 5.1 5.2 5.3 
should have more than one bottleneck link per direction).