Subject: [rmcat] Wireless test cases

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 16 Feb 2016 18:08:50 +0100

Hi all,

quick questions on the wireless test cases: 

First, who did already run all or some of the test cases, and which? 

And, second, should all the wireless test cases (both mobile and wifi) be part of the set of 'base' test cases for which we as a group would like to see results before moving a candidate document forward or do we need to even pick a subset of those?

Mirja

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Fri, 19 Feb 2016 20:27:11 +0000

Hi Mirja, 

For evaluation of NADA, we have previously run NS-3-based simulation test on the Wi-Fi test cases as described in Sections 4.1 and 4.2 of the draft.  

So far we've only tested with a fixed number of RMCAT flows only. The test scenarios with TCP background flows yet and varying number of flows are in the pipeline.   Results were presented earlier at IETF-93:

http://www.ietf.org/proceedings/93/slides/slides-93-rmcat-1.pdf

As for Sec. 3 on Cellular Network Specific Test Cases, would appreciate more detailed information on how to set those up, or relevant pointers/code on cellular network simulators.  

Thanks,
Xiaoqing
________________________________________
From: rmcat <[email-masked]> on behalf of Mirja Kühlewind <[email-masked]>
Sent: Tuesday, February 16, 2016 11:08 AM
To: rmcat WG ([email-masked])
Cc: Anna Brunström
Subject: [rmcat] Wireless test cases

Hi all,

Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 24 Feb 2016 12:33:26 +0100

Hi Xiaoqing,

On 19.02.2016 21:27, Xiaoqing Zhu (xiaoqzhu) wrote:
> As for Sec. 3 on Cellular Network Specific Test Cases, would appreciate
> more detailed information on how to set those up, or relevant pointers/code
> on cellular network simulators.

Can you let us/Zahed know what exactly you need/is missing?

Thanks,
Mirja

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Wed, 24 Feb 2016 16:50:00 +0000

Hi Mirja, all, 

Zahed and I have indeed discussed about this in Yokohama. Maybe I did not make it clear in my previous email, trying again. 

Basically, to carry out tests in Sec 3 (Cellular Network Specific Test Cases), one needs to run it over a module that simulates packet behavior over cellular links -- just like for the Wi-Fi portion of the test cases, we relied on the Wi-Fi module in NS3 simulators to do so.  

What is missing is the cellular network simulator, preferably as a plug-in module for NS3, over which we can run rate adaptation tests.  

Thanks,
Xiaoqing

________________________________________
From: Mirja Kühlewind <[email-masked]>
Sent: Wednesday, February 24, 2016 3:33 AM
To: Xiaoqing Zhu (xiaoqzhu); rmcat WG ([email-masked])
Cc: Anna Brunström; Zaheduzzaman Sarker
Subject: Re: [rmcat] Wireless test cases

Hi Xiaoqing,

Thanks,
Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 24 Feb 2016 18:02:51 +0100

Hi Xiaoqing,

> Am 24.02.2016 um 17:50 schrieb Xiaoqing Zhu (xiaoqzhu) <[email-masked]>:
> 
> Hi Mirja, all, 
> 

I though that this is available in the mean time. 

Zahed, do you have link or is this already included in the latest version of ns-3?

Mirja

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 25 Feb 2016 08:45:45 +0100

Yes,

NS-3 has necessary modules to run LTE simulations.

https://www.nsnam.org/docs/models/html/lte.html

BR

Zahed

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Thu, 25 Feb 2016 17:42:35 +0000

Thanks for the pointer, Zahed.  That's a good place to start.  Would it be possible for you to share your NS-3 code/script for testing SCReAM over LTE? 

BTW, you may want to update the corresponding link in the wireless-test-case document. Currently, the reference on [LTE-simulator] points to the NS-3 manual.

Best,
Xiaoqing

________________________________________
From: Zaheduzzaman Sarker <[email-masked]>
Sent: Wednesday, February 24, 2016 11:45 PM
To: Mirja Kühlewind; Xiaoqing Zhu (xiaoqzhu)
Cc: rmcat WG ([email-masked]); Anna Brunström
Subject: Re: [rmcat] Wireless test cases

Yes,

BR

Zahed

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Fri, 26 Feb 2016 07:51:29 +0000

Hi,

We didn't used NS-3 simulator to evaluation SCReAM over LTE, we used our proprietary simulator.
However, I have looked into the NS-3 simulator and talked with the LTE module implementer of NS-3 (LUNA). My conclusion is, the draft can be realized in the NS-3 simulator.

BR

Zahed 

> -----Original Message-----
> From: Xiaoqing Zhu (xiaoqzhu) [mailto:[email-masked]]
> Sent: den 25 februari 2016 18:43
> To: Zaheduzzaman Sarker; Mirja Kühlewind
> Thanks for the pointer, Zahed.  That's a good place to start.  Would it be
> possible for you to share your NS-3 code/script for testing SCReAM over LTE?
> 
> BTW, you may want to update the corresponding link in the wireless-test-case
> document. Currently, the reference on [LTE-simulator] points to the NS-3
> manual.
> 
> Best,
> Xiaoqing
> 
> >> Zahed and I have indeed discussed about this in Yokohama. Maybe I did not
> make it clear in my previous email, trying again.
> >>
> >> Basically, to carry out tests in Sec 3 (Cellular Network Specific Test Cases),
> one needs to run it over a module that simulates packet behavior over cellular
> links -- just like for the Wi-Fi portion of the test cases, we relied on the Wi-Fi
> module in NS3 simulators to do so.
> >>
> >> What is missing is the cellular network simulator, preferably as a plug-in
> module for NS3, over which we can run rate adaptation tests.
> >>> As for Sec. 3 on Cellular Network Specific Test Cases, would
> >>> appreciate more detailed information on how to set those up, or
> >>> relevant pointers/code on cellular network simulators.

---

From: Stefan Holmer <[email-masked]>
Date: Fri, 26 Feb 2016 08:55:29 +0000

As a first step we'll probably look into at least some of the wifi test
cases, but I don't know how soon we can have those done. Our current focus
is on improving the loss-based part of GCC, and after that we can focus on
further tests.

If it's reasonably simple to run lte tests then it sounds good to run some
of those too. If we can share code to set up NS3 for wifi and lte tests,
that would be great.

/Stefan

On Fri, Feb 26, 2016 at 8:51 AM Zaheduzzaman Sarker <
[email-masked]> wrote:

> However, I have looked into the NS-3 simulator and talked with the LTE
> module implementer of NS-3 (LUNA). My conclusion is, the draft can be
> realized in the NS-3 simulator.
>
> BR
>
> Zahed
>
> > > Zahed, do you have link or is this already included in the latest
> version of ns-
> > 3?
> > >
> > > Mirja
> > >
> > >
> > >> Thanks,
> > >> Xiaoqing
> > >>
> > >>

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Fri, 26 Feb 2016 16:50:30 +0000

​I will look into sharing the NS3 code/script for WiFi tests.

As for LTE test cases:  it's good to know that it can be done.  But, since there is no precedence to follow, this may take substantial time and effort. So we may not get around to tackle that soon.

Thanks,

Xiaoqing

________________________________
From: Stefan Holmer <[email-masked]>
Sent: Friday, February 26, 2016 12:55 AM
To: Zaheduzzaman Sarker; Xiaoqing Zhu (xiaoqzhu); Mirja Kühlewind
Cc: rmcat WG ([email-masked]); Anna Brunström
Subject: Re: [rmcat] Wireless test cases

As a first step we'll probably look into at least some of the wifi test cases, but I don't know how soon we can have those done. Our current focus is on improving the loss-based part of GCC, and after that we can focus on further tests.

If it's reasonably simple to run lte tests then it sounds good to run some of those too. If we can share code to set up NS3 for wifi and lte tests, that would be great.

/Stefan

On Fri, Feb 26, 2016 at 8:51 AM Zaheduzzaman Sarker <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

BR

Zahed

> From: Xiaoqing Zhu (xiaoqzhu) [mailto:[email-masked]<mailto:[email-masked]>]
> Cc: rmcat WG ([email-masked]<mailto:[email-masked]>); Anna Brunström
> From: Zaheduzzaman Sarker <[email-masked]<mailto:[email-masked]>>
> Cc: rmcat WG ([email-masked]<mailto:[email-masked]>); Anna Brunström
> >> Am 24.02.2016 um 17:50 schrieb Xiaoqing Zhu (xiaoqzhu)
> <[email-masked]<mailto:[email-masked]>>:
> >>
> >> Hi Mirja, all,
> >>
> >> From: Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>
> >> To: Xiaoqing Zhu (xiaoqzhu); rmcat WG ([email-masked]<mailto:[email-masked]>)

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 26 Feb 2016 18:10:32 +0100

Hi all,

thanks for you feedback.

This brings me back to my original question: Would the working group require to see evaluation results to the mobile and wireless test cases before moving to publication for experimental or are the basic test cases (in draft-ietf-rmcat-eval-test-02) sufficient to move on and classify the candidates to be safe for experimentation in the Internet?

Mirja

> Am 26.02.2016 um 17:50 schrieb Xiaoqing Zhu (xiaoqzhu) <[email-masked]>:
> 
> On Fri, Feb 26, 2016 at 8:51 AM Zaheduzzaman Sarker <[email-masked]> wrote:
> Hi,
> 
> > <[email-masked]>:

---

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 27 Feb 2016 09:02:35 +0000

Hi

For what it´s worth.
I will run the LTE test cases and the basic test cases with SCReAM in our proprietary  system simulator. I will possibly also run some WiFi testcase in our simulator.
I will not run in NS3 however, simple reason is that I don't have time to delve into that platform.

/Ingemar

> From: Mirja Kühlewind [mailto:[email-masked]]
> Sent: den 26 februari 2016 18:11
> To: Xiaoqing Zhu (xiaoqzhu)
> Cc: rmcat WG ([email-masked]); Zaheduzzaman Sarker; Stefan Holmer; Anna
> Brunström
> This brings me back to my original question: Would the working group require
> to see evaluation results to the mobile and wireless test cases before moving
> to publication for experimental or are the basic test cases (in draft-ietf-
> rmcat-eval-test-02) sufficient to move on and classify the candidates to be
> safe for experimentation in the Internet?
> 
> Mirja
> 
> 
> > As for LTE test cases:  it's good to know that it can be done.  But, since there
> is no precedence to follow, this may take substantial time and effort. So we
> may not get around to tackle that soon.
> >
> > Thanks,
> > Xiaoqing
> >
> > As a first step we'll probably look into at least some of the wifi test cases,
> but I don't know how soon we can have those done. Our current focus is on
> improving the loss-based part of GCC, and after that we can focus on further
> tests.
> >
> > If it's reasonably simple to run lte tests then it sounds good to run some of
> those too. If we can share code to set up NS3 for wifi and lte tests, that
> would be great.
> >
> > /Stefan
> >
> > On Fri, Feb 26, 2016 at 8:51 AM Zaheduzzaman Sarker
> <[email-masked]> wrote:
> > Hi,
> >
> > > Thanks for the pointer, Zahed.  That's a good place to start.  Would
> > > it be possible for you to share your NS-3 code/script for testing SCReAM
> over LTE?
> > >
> > > BTW, you may want to update the corresponding link in the
> > > wireless-test-case document. Currently, the reference on
> > > [LTE-simulator] points to the NS-3 manual.
> > >
> > > Best,
> > > Xiaoqing
> > >
> > > one needs to run it over a module that simulates packet behavior
> > > over cellular links -- just like for the Wi-Fi portion of the test
> > > cases, we relied on the Wi-Fi module in NS3 simulators to do so.
> > > >>

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 29 Feb 2016 13:38:38 +0000

I don't think we should require mobile/wireless evaluation results before
experimental publication. I think we have proof enough from the base test
cases, especially with the extra artificial jitter and fast capacity
changes.

/Stefan

On Fri, Feb 26, 2016 at 6:10 PM Mirja Kühlewind <
[email-masked]> wrote:

> Hi all,
>
> thanks for you feedback.
>
> This brings me back to my original question: Would the working group
> require to see evaluation results to the mobile and wireless test cases
> before moving to publication for experimental or are the basic test cases
> (in draft-ietf-rmcat-eval-test-02) sufficient to move on and classify the
> candidates to be safe for experimentation in the Internet?
>
> Mirja
>
>
> > As for LTE test cases:  it's good to know that it can be done.  But,
> since there is no precedence to follow, this may take substantial time and
> effort. So we may not get around to tackle that soon.
> >
> > Thanks,
> > Xiaoqing
> >
> > If it's reasonably simple to run lte tests then it sounds good to run
> some of those too. If we can share code to set up NS3 for wifi and lte
> tests, that would be great.
> >
> > /Stefan
> >
> it be
> wireless-test-case

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Mon, 29 Feb 2016 22:43:09 +0000

​This is my view too.

I still believe that it's important to understand candidate solution behavior over wireless. My colleagues and I are actively working on that.  I just don't think it's realistic to tie the progress of the candidate solution drafts to such evaluations, for two reasons:

a)  the wireless test case draft itself is still evolving;

b)  folks have not yet converged on a common platform (testbed or simulation) for carrying out wireless tests. That makes it hard to compare across candidate solutions.

Thanks,

Xiaoqing

________________________________
From: Stefan Holmer <[email-masked]>
Sent: Monday, February 29, 2016 7:38 AM
To: Mirja Kühlewind; Xiaoqing Zhu (xiaoqzhu)
Cc: Zaheduzzaman Sarker; rmcat WG ([email-masked]); Anna Brunström
Subject: Re: [rmcat] Wireless test cases

I don't think we should require mobile/wireless evaluation results before experimental publication. I think we have proof enough from the base test cases, especially with the extra artificial jitter and fast capacity changes.

/Stefan

On Fri, Feb 26, 2016 at 6:10 PM Mirja Kühlewind <[email-masked]<mailto:[email-masked]>> wrote:
Hi all,

thanks for you feedback.

Mirja

> Am 26.02.2016 um 17:50 schrieb Xiaoqing Zhu (xiaoqzhu) <[email-masked]<mailto:[email-masked]>>:
>
> From: Stefan Holmer <[email-masked]<mailto:[email-masked]>>