Subject: [rmcat] for discussion: candidate solution evaluation

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 5 Mar 2014 14:35:53 +0000

Hi,

I wanted to put my proposal from the session into writing, so we can discuss it here.

I think we've made some good progress on the evaluation criteria and scenarios, to where we can begin to discuss the properties of some of the proposed solution candidate algorithms (on the list and at the next meeting.)

My proposal would be that the proponents of said candidates discuss among themselves and with Varun to pick a small number of them for initial evaluation. Varun & co-authors will focus their efforts on fleshing those out in more detail, and meanwhile, the solution proponents begin evaluation of the candidates in (at least) those scenarios.

In the next session, we'd begin by having Varun walk the group through the details of those scenarios, and then see evaluation results for the various candidates.

Because I want to encourage folks to implement and evaluate each others work* (instead of just their own proposal), I'd like to "reward" those teams that can show results for at least one competitor - in addition to their own candidate - with more session time in Toronto.

Thoughts?

Lars

* The reason being that this will result in more directly comparable numbers, while at the same time improving the quality of the specifications of the various candidates, because we will verify that they actually can be implemented based on the textual description.

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Wed, 5 Mar 2014 17:21:01 +0100

On 2014-03-05 15:35, Eggert, Lars wrote:

>
> My proposal would be that the proponents of said candidates discuss
> among themselves and with Varun to pick a small number of them for
> initial evaluation. Varun & co-authors will focus their efforts on
> fleshing those out in more detail, and meanwhile, the solution
> proponents begin evaluation of the candidates in (at least) those
> scenarios.

The proponents may pick small number of the testcases but they must 
select the same test cases.

But I would like to get input and comments on all the test cases on the 
list. And we(=the authors) will try to address the comments and concerns 
in all those test cases in the next version. Otherwise, we run a risk of 
loosing the non-picked testcases as for some reason those were down 
prioritized. I personally didn't have any preference when those were 
written and the indexing of the test case does not show any priority 
either. I would be interesting to know what the proponents are 
considering when selecting the test cases.

 From my experience I can say, if somebody setup the testbed and agree 
on the attribute values specified in the respective test cases, then 
running all the test cases and as small number of them should not make 
much of a difference.

>
> In the next session, we'd begin by having Varun walk the group
> through the details of those scenarios, and then see evaluation
> results for the various candidates.
In this meeting the focus was on the structure and the list of test 
cases but in future we definitely have to discuss the details and 
attribute values in each of the test cases.

BR

Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Services, Media and Network Features
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com

==================================================

---

From: Varun Singh <[email-masked]>
Date: Wed, 5 Mar 2014 17:40:34 +0000

Hi all,

comments inline.

>> among themselves and with Varun to pick a small number of them for

Those interested in this should drop me a note at [email-masked].

> The proponents may pick small number of the testcases but they must select
> the same test cases.

I concur, whichever tests are picked we should make sure that they are
the same.
However, I would expect the proponents to run test with different set
of attributes.
For example, different values of RTT

> list. And we(=the authors) will try to address the comments and concerns in
> all those test cases in the next version. Otherwise, we run a risk of
> prioritized. I personally didn't have any preference when those were written
> and the indexing of the test case does not show any priority either. I would
> be interesting to know what the proponents are considering when selecting
> the test cases.
>
> From my experience I can say, if somebody setup the testbed and agree on the
> attribute values specified in the respective test cases, then running all
> the test cases and as small number of them should not make much of a
> difference.
>
>
>>
> In this meeting the focus was on the structure and the list of test cases
> but in future we definitely have to discuss the details and attribute values
> in each of the test cases.
>

The attribute values are not set in stone, they can be changed. We
just have to make sure that all the proponents use roughly the same
values.

Cheers,
Varun

-- 
http://www.netlab.tkk.fi/~varun/

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 5 Mar 2014 18:32:05 +0000

Hi,

On 2014-3-5, at 16:21, Zaheduzzaman Sarker <[email-masked]> wrote:
> The proponents may pick small number of the testcases but they must select the same test cases.

yep.

> But I would like to get input and comments on all the test cases on the list. And we(=the authors) will try to address the comments and concerns in all those test cases in the next version. Otherwise, we run a risk of loosing the non-picked testcases as for some reason those were down prioritized. I personally didn't have any preference when those were written and the indexing of the test case does not show any priority either. I would be interesting to know what the proponents are considering when selecting the test cases.

Also agreed. I didn't mean to say that all other activity should stop, and we will obviously give session time to our other milestones and individual drafts as needed.

Lars