Subject: [rmcat] test cases for coupled cc [was: Re: Review of draft-ietf-rmcat-eval-test-02]

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 9 Feb 2016 15:51:00 +0100

Hi Zahed,

see below.

> Am 28.01.2016 um 14:29 schrieb Zaheduzzaman Sarker <[email-masked]>:
> 
>> * I would like to suggest the following two additional test-cases
>>    under section 6 (other potential test cases).
> 
> The test cases in the draft were designed for end-point adaptation algorithm without coupled congestion control in mind i.e. no need for shared bottleneck detection and flow priority. In the last IETF meeting we talked about the possibility of extending the test cases to include coupled congestion control algorithm.

I believe we decided already at the last meeting that we need to adapt this draft to include uses cases for coupled congestion (and that’s why we asked Safiqul to propose some).

> 
> Now if we extend the test cases then I don't think only adding the two proposed test cases will be sufficient.

Yes, that might be true but adding these use cases below would probably be a good starting point.

> We will need to identify all the test cases those are suitable for running with coupled congestion control.

Not sure, if you only have one flow that test case is simply not suitable and that’s fine.

> Like 5.2 can be run with priority per flow. By doing this we can mention (in each test case or in one place in the document) that one can run tests with flow priority and perhaps give some means to evaluate the available capacity distribution with some equation like the one mentioned here in the proposed test case 6.3.

Yes.

> However, then this document also needs to include description for shared bottleneck detection.

Why?

> In the simplest form, we can perhaps assume that the identified test cases have single bottleneck point.

Yes.

> 
> I think we need to discuss what is best possible way to extend the test cases if we agreed to do that.

As I said I think we agreed already. Please following with Safiqul on this!

> 
> But before that I would like to know if the WG is OK with extending the test cases for coupled congestion control.

Any further feedback on this from the group is of course welcome! Please speak up now!

> 
> And will it be a compulsory (candidate algorithms MUST present results) or optional (candidate algorithms MAY present results) part of the test case.

I guess only candidates that use coupled cc..?

Mirja

> 
>> 
>> 6.3 Prioritization test
>> 
>> In this test case, three RMCAT media flows started  out with a priority
>> of 1 each. At around 50 s, the priorities of flow 2 and flow 3 were
>> decreased to 1/3 and 2/3.  A flow i  is assigned the available capacity
>> based on its priority p_{i} as follows:
>> 
>> b_{i} = (p_{i} / Sum(p_{k}) ) * BW   ;  k = {1.. 3}
>> 
>> This test case ensures that flows are getting their assigned portions of
>> the available capacity based on their assigned priorities. It can be
>> useful for [I-D.ietf-rmcat-coupled-cc]. It addresses the requirement 3A
>> in  [I-D.ietf-rmcat-cc-requirements]
>> 
>> Testbed topology: Same as test case defined in section 5.4
>> 
>> Testbed Attribute:
>> 
>> The general description of the testbed parameters is same as Section 5.4
>> with changes in the test specific setup as  below-
>> 
>>   o  Other test specific setup:
>> 
>>       *  RMCAT media flow timeline:
>> 
>>          +  Flow ID: One (1)
>> 
>>          +  Start time: 0s
>> 
>>          +  Flow duration: 119s
>> 
>>          +  Priority: no change
>> 
>>      *  RMCAT media flow timeline:
>> 
>>          +  Flow ID: Two (2)
>> 
>>          +  Start time: 0s
>> 
>>          +  Flow duration: 119s
>> 
>>          +  Priority:  change to 1/3 after 50s
>> 
>>       *  RMCAT media flow timeline:
>> 
>>          +  Flow ID: Three (3)
>> 
>>          +  Start time: 0s
>> 
>>          +  Flow duration: 119s
>> 
>>          +  Priority: change to 2/3 after  50s
>> 
>> 
>> 6.4 RMCAT flows competing with one or more long TCP flows
>> 
>> This test case is an extension of the test case defined in section 5.6.
>> It is important for [I-D.ietf-rmcat-coupled-cc], where it requires more
>> than one RMCAT flows in order to couple them.
>> 
>> The general description of the testbed parameters is same as section 5.6.

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Wed, 10 Feb 2016 10:24:37 +0100

Hi Mirja,

please see inline....

On 2016-02-09 15:51, Mirja Kühlewind wrote:
>>
>> The test cases in the draft were designed for end-point adaptation
>> algorithm without coupled congestion control in mind i.e. no need
>> for shared bottleneck detection and flow priority. In the last IETF
>> meeting we talked about the possibility of extending the test cases
>> to include coupled congestion control algorithm.
>
> I believe we decided already at the last meeting that we need to
> adapt this draft to include uses cases for coupled congestion (and
> that’s why we asked Safiqul to propose some).

That is what I can also recall but didn't see anything in the meeting
minutes so was not sure what we agreed on. May be missed it in the 
meeting minutes.

>
>>
>> Now if we extend the test cases then I don't think only adding the
>> two proposed test cases will be sufficient.
>

If I understand shafiqul's last mail, it is fine with only having test 
results with priority for 5.4 test case.

> suitable and that’s fine.
Yes, that is one identification criteria. But may be we dont need 
results for all the test cases who has more than one flow. see my 
previous comment.
>
>> Like 5.2 can be run with priority per flow. By doing this we can
>> mention (in each test case or in one place in the document) that
>> one can run tests with flow priority and perhaps give some means to
>> evaluate the available capacity distribution with some equation
>> like the one mentioned here in the proposed test case 6.3.
>
> Yes.
>
Bcos I dont think the document now even mention "shared bottleneck" and 
to me coupled congestion control does not make sense without shared 
bottleneck.
>> But before that I would like to know if the WG is OK with extending
>> the test cases for coupled congestion control.
>
>> And will it be a compulsory (candidate algorithms MUST present
>> results) or optional (candidate algorithms MAY present results)
>> part of the test case.
>

hmm...

I am a bit confused now. Are we talking about test case(s) to evaluate 
the Coupled CC or the use of Coupled CC in the candidate CC?

BR

-- 
Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Services, Media and Network Features
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com

==================================================

---

From: Safiqul Islam <[email-masked]>
Date: Wed, 10 Feb 2016 14:36:08 +0000

HI Zahed, Hi Mirja,

Please see inline

/Safiqul

On 10. feb. 2016, at 10.24, Zaheduzzaman Sarker <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mirja,

please see inline....

That is what I can also recall but didn't see anything in the meeting
minutes so was not sure what we agreed on. May be missed it in the meeting minutes.

Yes, that might be true but adding these use cases below would
probably be a good starting point.

If I understand shafiqul's last mail, it is fine with only having test results with priority for 5.4 test case.

For simplicity, I suggested to use only test case 5.4, but I am not against using other cases.
I can add some recommendations for other cases too.

We will need to identify all the test cases those are suitable for
running with coupled congestion control.

Not sure, if you only have one flow that test case is simply not
suitable and that’s fine.
Yes, that is one identification criteria. But may be we dont need results for all the test cases who has more than one flow. see my previous comment.

+1

Yes.

However, then this document also needs to include description for
shared bottleneck detection.

Why?
Bcos I dont think the document now even mention "shared bottleneck" and to me coupled congestion control does not make sense without shared bottleneck.

The document covers the basic scenarios where flows share a common bottleneck.  We can simply add a line saying “these additional coupled-cc test-cases are used for flows sharing a bottleneck”.

In the simplest form, we can perhaps assume that the identified
test cases have single bottleneck point.

Yes.

I think we need to discuss what is best possible way to extend the
test cases if we agreed to do that.

As I said I think we agreed already. Please following with Safiqul on
this!

Any further feedback on this from the group is of course welcome!
Please speak up now!

hmm...

I am a bit confused now. Are we talking about test case(s) to evaluate the Coupled CC or the use of Coupled CC in the candidate CC?

Without making it compulsory, I suggested to add these in the other cases. We can perhaps say:

“When flows share a common bottleneck, combining their congestion controllers can be beneficial.
These additional test cases can be used to evaluate a congestion control mechanism when using a method to couple flows, such as [I-D.ietf-rmcat-coupled-cc]."

Does this put it better?

Regards,
Safiqul

BR

--
Zahed

Ericsson AB
Services, Media and Network Features
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]<mailto:[email-masked]>
www.ericsson.com

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Tue, 16 Feb 2016 13:14:33 +0100

Hi,

Please see inline.

BR

Zahed

On 02/10/2016 03:36 PM, Safiqul Islam wrote:
> HI Zahed, Hi Mirja,
>
> Please see inline
>
> /Safiqul
>
>
>> On 10. feb. 2016, at 10.24, Zaheduzzaman Sarker 
>> <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Hi Mirja,
>>
>> please see inline....
>>
>> If I understand shafiqul's last mail, it is fine with only having 
>> test results with priority for 5.4 test case.
>
A section in 5.4 describing the required changed to run with couple CC 
should be enough.
>> Bcos I dont think the document now even mention "shared bottleneck" 
>> and to me coupled congestion control does not make sense without 
>> shared bottleneck.
>
> The document covers the basic scenarios where flows share a common 
> bottleneck.  We can simply add a line saying “these additional 
> coupled-cc test-cases are used for flows sharing a bottleneck”.
yes, something like that.
>> I am a bit confused now. Are we talking about test case(s) to 
>> evaluate the Coupled CC or the use of Coupled CC in the candidate CC?
>
>
> These additional test cases can be used to evaluate a congestion 
> control mechanism when using a method to couple flows, such as 
> [I-D.ietf-rmcat-coupled-cc]."
>
> Does this put it better?
I dont think this document to referent to any candidate solution.

I think the test cases should be defined so that one can evaluate other 
alternatives if there is any. Now, if there is more than one coupled CC 
solution and the candidate algorithms perform differently with different 
solution, how that will help us to evaluate coupled Congestion control 
algorithms. How does the WG plans to solve such situation?

---

From: Michael Welzl <[email-masked]>
Date: Tue, 16 Feb 2016 06:42:05 -1000

Hi,

My 2 cents about one bit here, cutting away the rest:

> On 16. feb. 2016, at 02.14, Zaheduzzaman Sarker <[email-masked]> wrote:
> (..)
(..)

> I think the test cases should be defined so that one can evaluate other alternatives if there is any. Now, if there is more than one coupled CC solution and the candidate algorithms perform differently with different solution, how that will help us to evaluate coupled Congestion control algorithms. How does the WG plans to solve such situation?

The sentence Safiqul proposed mentions the coupled-cc draft as an example - not more, not less. Remove it if you want, and nothing has changed: this isn’t about evaluating coupled CC. algorithms, it’s about evaluating how congestion control algorithms work when they are combined with a method to couple them. It’s not mandatory either - but if you want to test how e.g. SCReAM operates when coupled, these test cases probable make sense. That’s all this says.

Cheers,
Michael

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 24 Feb 2016 12:01:49 +0100

Hi Zahed,

quickly on this point:

> minutes so was not sure what we agreed on. May be missed it in the

I guess there was no official decision of the wg to include text for 
coupled-cc, however, Safiqul volunteered to provide some input here and I 
don't recall that there have been any objections to include this. I also 
don't see how this could be a problem.

I agree with you and Safiqul that we don't need to consider coupled-cc for 
all test cases; only where it makes sense (e.g. test cases with one flow do 
not make any sense...)

If anybody has a different opinion, please speak up now!

Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 24 Feb 2016 12:06:14 +0100

Hi Zahed,

I agree with Michael. My view is here that we need to evaluate each 
combination of a certain scheme to couple things using a certain congestion 
control algorithm separately. That means, if someone proposes a new (generic) 
scheme for coupling, we as a group would only advise to use this scheme with 
a certain cc algorithm if we have seen results for this specific combination. 
Does this make sense to you?

I hope this also reflects the view of the wg. If not, please let us now and 
we would need to discuss this again.

Mirja

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Wed, 24 Feb 2016 21:20:02 +0000

Hi, 

I would like to provide my input on this discussion.  

My understanding is that the test case draft specifies scenarios over which we would like to evaluate candidate algorithms (oh well, stating the obvious.) As such, Sec 5.4 (Competing Flows with Same RMCAT Algorithm) is a good place to add one flavor/variation of the test set up, whereby all senders share the same physical node so that one can test how a coupled CC solution works.  

I don't see why the test case draft need to say anything additional (e.g., whether a proposed CC solution should be combined with which flavor of the candidate algorithm) beyond describing that sub-test case. 

Thanks,
Xiaoqing

________________________________________
From: rmcat <[email-masked]> on behalf of Mirja Kühlewind <[email-masked]>
Sent: Wednesday, February 24, 2016 3:06 AM
To: Zaheduzzaman Sarker
Cc: [email-masked]; Michael Welzl
Subject: Re: [rmcat] test cases for coupled cc [was: Re: Review of draft-ietf-rmcat-eval-test-02]

Hi Zahed,

Mirja

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 25 Feb 2016 09:31:47 +0100

On 2016-02-24 12:01, Mirja Kühlewind wrote:
> coupled-cc, however, Safiqul volunteered to provide some input here and
> I don't recall that there have been any objections to include this. I
> also don't see how this could be a problem.
I didn't see this as a problem however I had some confusions and 
questions those needed to be clarified.

>
> I agree with you and Safiqul that we don't need to consider coupled-cc
> for all test cases; only where it makes sense (e.g. test cases with one
> flow do not make any sense...)
>

-- 
Zahed

Ericsson AB

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 25 Feb 2016 10:03:20 +0100

Hi Mirja,

Please see inline below.

BR

Zahed

On 02/24/2016 12:06 PM, Mirja Kühlewind wrote:
> combination of a certain scheme to couple things using a certain 
> congestion control algorithm separately. 
Fine. but again what exactly are we supposed to evaluate? the 
performance of the congestion control algorithm or the performance of 
the coupled congestion control algorithm? or something else.
> That means, if someone proposes a new (generic) scheme for coupling, 
> we as a group would only advise to use this scheme with a certain cc 
> algorithm if we have seen results for this specific combination. Does 
> this make sense to you?
Kind of, this means it is fine to have a coupled congestion control that 
may not work will all the combinations even if RMCAT proposes more that 
one congestion control algorithms.

I didnt get the "generic"  part though. How can the WG call a coupled 
congestion "generic" even if that fails to show results with all the 
RMCAT CC algorithms (assuming RMCAT has proposed more than one CC 
algorithms)?
>
>
> I hope this also reflects the view of the wg. If not, please let us 
> now and we would need to discuss this again.
>
> Mirja
>
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 25 Feb 2016 17:12:46 +0100

Are your questions now clarified? If not, what exactly are the questions?

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 25 Feb 2016 17:20:23 +0100

Hi Zahed,

see inline.

> the coupled congestion control algorithm? or something else.

The performance of the combined system of the coupling algo and the selected 
cc aglo.

> one congestion control algorithms.

Yes (as long as this is clearly stated in the doc).

> RMCAT CC algorithms (assuming RMCAT has proposed more than one CC

That's only a matter of how it is described in the doc. Basically you say, we 
describe this in a generic way, but it's only tested together with these 
algos. If you want use it with others, it might be possible, but you have to 
(adapt and) test it on your own. Important is that you have both in the same 
doc: the generic description and how it is applied to certain algos. Does 
that make sense? Or it this a problem for any of your work?

I believe that is at least the approach we are currently taking for the 
coupled-cc draft that we have. Micheal, please speak up if you have a 
different view here!

Mirja

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Fri, 26 Feb 2016 09:03:18 +0100

Hi Mirja,

I had my questions asked in the beginning of the this email thread (kind 
of repeated in other mails).

Things got lot better now.

Thanks.

BR

Zahed

-- 
Zahed

Ericsson AB

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Fri, 26 Feb 2016 09:15:44 +0100

On 2016-02-25 17:20, Mirja Kühlewind wrote:

> The performance of the combined system of the coupling algo and the
> selected cc aglo.
ok, the current draft says "This document describes the test cases
    to be used in the performance evaluation of those candidate
    algorithms."
as now we are talking about combined system, this needs to be changed I 
guess. right? but we need to clarify what this combined system means, I 
hope we are not leaning towards the whole RTC system evaluation.
> Yes (as long as this is clearly stated in the doc).
ok. but the eval-test draft doesn't need to clarify that, right?
> That's only a matter of how it is described in the doc. Basically you
> say, we describe this in a generic way, but it's only tested together
> with these algos. If you want use it with others, it might be possible,
> but you have to (adapt and) test it on your own. Important is that you
> have both in the same doc: the generic description and how it is applied
> to certain algos. Does that make sense? Or it this a problem for any of
> your work?
Ok so here "generic" does not mean the algorithm is generic, it points 
to the generic description of the coupled CC.

No problem for any work as long as the above clarification text is not 
part of the eval-test draft.

BR
-- 
Zahed

Ericsson AB