Subject: Re: [rmcat] Please review draft-ietf-rmcat-eval-criteria-01

From: "Karen E. Egede Nielsen" <[email-masked]>
Date: Sun, 13 Jul 2014 16:18:23 +0200

Hi Varun,

The draft also references the Jain-fairness index (Open Issues 2) how does
this relate to open issue 1 (below).

Editorial - I think ? - In Section 3 you start by using a bulleted format
but then after item 5., the bullet format is no longer used. Is that
intentionally ?

Thanks.

BR, Karen

> -----Original Message-----
> From: Varun Singh [mailto:[email-masked]]
> Sent: 12. juni 2014 13:28
> To: rmcat WG
> Cc: Eggert, Lars; Karen E. Egede Nielsen; Jörg Ott; Mirja Kühlewind
> Subject: Re: Please review draft-ietf-rmcat-eval-criteria-01
>
> Hi all,
>
> There was one one open issue that needs to be discussed and it is related
> to
> fairness :)
>
> This has been discussed at length in the design group meetings and also at
> the IETF meetings, but I am not sure if the text in the open issue had
> working
> group agreement.
>
> The summary of the discussion so far is:
> - The eval-criteria needs to define a guideline that helps in evaluating
> an
> RMCAT proposal
> - The individual test case should define its expected fairness criteria,
> and it
> may vary between scenarios.
>
> NEW TEXT to replace the open issue:
>
>  Fairness or Unfairness: Experiments testing the performance of an
>    RMCAT proposal against any cross-traffic must define its expected
>    criteria for fairness.
>
>    The "unfairness" test guideline to assist in defining such a criteria
> (measured at 1s intervals) is:
>    1.  Does not trigger the circuit breaker.
>    2.  When the competing streams with identical RMCAT streams having
> similar RTTs,
>    no stream achieves over 3 times the min RMCAT stream or less than
> 1/3 times the
>    throughput of the max RMCAT stream.
>    3.  When competing with identical RMCAT streams having similar RTTs,
>    no RTT measurements for an RMCAT stream is over 3 times delay compared
> to RTT
>    measurements performed before starting the RMCAT flow.
>
>
> Next step:
> Is the guideline helpful and should be in the draft?, or suggestion, and
> counter proposals are welcome.
>
> Cheers,
> Varun
>
>
> On Mon, Mar 31, 2014 at 5:02 PM, Mirja Kühlewind
> <[email-masked]> wrote:
> > Hi group,
> >
> > draft-ietf-rmcat-eval-criteria-01 was updated after the last meeting
> > addressing all comments from the meeting. We would really need people
> > to review this document now! Please do so!
> >
> > https://datatracker.ietf.org/doc/draft-ietf-rmcat-eval-criteria/
> >
> > Mirja
> >
> >
>
>
>
> --
> http://www.netlab.tkk.fi/~varun/

---

From: Varun Singh <[email-masked]>
Date: Mon, 14 Jul 2014 17:37:54 +0300

Hi Michael,

Missed this email from earlier, comments inline.

Cheers,
Varun

On Fri, Jun 13, 2014 at 4:07 PM, Michael Welzl <[email-masked]> wrote:
> Hi,
>
> On 12. juni 2014, at 13:28, Varun Singh wrote:
>
>> Hi all,
>>
>> There was one one open issue that needs to be discussed and it is
>> related to fairness :)
>>
>> This has been discussed at length in the design group meetings and
>> also at the IETF meetings, but I am not sure if the text in the open
>> issue had working group agreement.
>>
>> - The eval-criteria needs to define a guideline that helps in
>> evaluating an RMCAT proposal
>> - The individual test case should define its expected fairness
>> criteria, and it may vary between scenarios.
>>
>>   The "unfairness" test guideline to assist in defining such a
>> criteria (measured at 1s intervals) is:
> In principle, what this new text up to here is trying to say (if I interpret it correctly) sounds good to me.
> However, the English is pretty broken, please fix.
>

2. When competing with identical RMCAT streams having similar RTTs,
no RMCAT stream achieves
more than 3 times the average throughput of the RMCAT stream with the
lowest average throughput,
or less than 1/3 times the average throughput of the RMCAT stream with
the highest average throughput.

>>   no RTT measurements for an RMCAT stream is over 3 times delay compared to RTT
> Huh? "Compared to measurements performed before starting the RMCAT flow" => which one? All, including the background flows, or only the foreground flow? The foreground flow must not make the RTT grow by a factor of 3, is that it? I think that's not a fairness criterion, it's a delay-minimizing criterion, and a weird one too. I wonder if this could just be removed.

My interpretation of the proposal was that the RTT should not grow by
a factor of 3 for the existing flows when a new flow is added. I can
remove this, if there is consensus on removing this requirement.

I do not remember correctly who made the original proposal (was
perhaps Kevin Gross), but was in a design meeting from a year ago.

> Michael

---

From: Varun Singh <[email-masked]>
Date: Mon, 14 Jul 2014 17:46:27 +0300

Hi Karen,

Thank you for the feedback, comments in line.

- Varun.

> this relate to open issue 1 (below).
I think it is still TBD, the assumption last year was that we need
more evaluation results before we can comment on the list of metrics.
Since, there are more evaluations in the upcoming meeting, I believe
we might be getting closer to answering the question.

> but then after item 5., the bullet format is no longer used. Is that

These items were editor notes before, which may explain why they were
outside the < list >.
They should be bullet items. Will be fixed in the next version.

---

From: Michael Welzl <[email-masked]>
Date: Wed, 16 Jul 2014 20:05:52 +0200

Hi,

In line:

> the highest average throughput.

Looks fine to me

> remove this, if there is consensus on removing this requirement.

Ah, got it. I'm fine with the requirement (although I stick with my observation that it's kinda weird  :)  ). Anyway I'd rephrase it as you wrote it here, which is much easier to understand than your original text.

Cheers,
Michael

---

From: Varun Singh <[email-masked]>
Date: Thu, 17 Jul 2014 19:46:25 +0300

Hi Michael,

Snipping away the original email, comment inline.

>> the highest average throughput.

Will add this in the next update.

> Ah, got it. I'm fine with the requirement (although I stick with my observation that it's kinda weird  :)  ). Anyway I'd rephrase it as you wrote it here, which is much easier to understand than yo…

I can fix the text as suggest above.

Alternatively, I am wondering if this should be expressed as a
function of queuing delay? i.e., QD = RTT_measured - RTT_base.
RTT_base is the configured value in the testbed/simulator. I can't
think of a good enough value here either. Suggestions are welcome.

I suppose another way to measure this is if the endpoint starts to
discard late arriving packets, then we may not have to worry about the
delay metric. This however implies that the simulator/testbed has a
playout buffer or some model to simulate it.

---

From: Michael Welzl <[email-masked]>
Date: Thu, 17 Jul 2014 19:27:11 +0200

> think of a good enough value here either. Suggestions are welcome.

I'd keep it simple and not go there. Anyway the 3x thing is pretty dubious - depends on queue lengths and whatnot; thus, best not to dig into the details, this is only asking for trouble  :-)

Cheers,
Michael

---

From: Varun Singh <[email-masked]>
Date: Wed, 23 Jul 2014 09:55:46 -0400

Hi all,

If everyone is in agreement, I will generate a new version of the
eval-criteria document incorporating the feedback in the next couple
of days.

Cheers,
Varun.

---

From: Anna Brunstrom <[email-masked]>
Date: Thu, 24 Jul 2014 05:21:19 +0200

Hi Varun,

On 2014-07-23 15:55, Varun Singh wrote:
[snip]
>>> the highest average throughput.

Unless I am missing something, the above text states the same criteria 
twice. Is this intended/needed?

Cheers,
Anna