Subject: [rmcat] WGLC comments draft-ietf-rmcat-eval-test-06

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 21 Aug 2018 06:08:06 +0000

Hi
A few comments on draft-ietf-rmcat-eval-test-06

In general, the draft outlines a number of test cases but does not outline a pass or fail criterion. I guess that is OK as it is yet an extra level of complexity and given that the energy level in the group is not very high it is probably sensible to skip the pass or fail criteria. In any case it may be good to state this clearly in the draft.

A few editorial nits :

Page 4 : Whole paragraph starting with "In a testbed environment.." The last sentence is loaded with double negations and there is also a grammatical nit. Perhaps rewrite it as follows ?
"In a testbed environment with real equipment, there may exist a significant amount
  of unwanted traffic on portions of the network path between the endpoints.
  Some of this traffic may be generated by other processes on the
  endpoints themselves (e.g., discovery protocols) or by other
  endpoints not presently under test.  Such unwanted traffic should be removed or avoided to the
  greatest extent possible"

Page 7:
"test case which are" --> "test case that is"

Section 5.3 :
 "It is expected... " should be prepended by "Expected behavior: " to make the layout similar to previous test cases.

Page 16 last bullet before table 3 : " where x is the bottleneck capacity specified in Table 4 ", I guess this should also apply to values in table 3 if a UDP load is used for the forward direction.

Section 5.5 :
"It is expected... " should be prepended by "Expected behavior: " to make the layout similar to previous test cases.

Section 5.7 :
"Expected behavior: ...." missing

Section 5.8 :
"Expected behavior: ...." missing

Section 6.1 : Last paragraph. Another alternative is to use a weighted priority scheduler, used e.g. in SCReAM (RFC8298)

Regards
/Ingemar

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 23 Aug 2018 09:06:32 +0000

Hi Ingemar,

Thanks for your review, really appreciate it. Please see in-line.

BR

Zahed

On 2018-08-21, 08:08, "Ingemar Johansson S" <[email-masked]<mailto:[email-masked]>> wrote:

Hi

I remember to discuss this at the very early state of this draft. We ended up having describing “Expected behaviour” rather than pass/fail criteria. It appeared that we needed some absolute values on reaction time and recovery time, throughput and goodput to define some pass/fail criteria. As we are dealing with a wide range of media characteristics over RTP those value will vary depending on media type and characteristics, hence it was not a feasible solution. We later on used this test cases for comparing candidate algorithms and judges or showed relative results. During this time, the pass/fail criterial was also not needed/missed.

We now have “expected behavior”, which should be good enough for evaluating candidate algorithms.

A few editorial nits :

Page 4 : Whole paragraph starting with “In a testbed environment..” The last sentence is loaded with double negations and there is also a grammatical nit. Perhaps rewrite it as follows ?
“In a testbed environment with real equipment, there may exist a significant amount
  of unwanted traffic on portions of the network path between the endpoints.
  Some of this traffic may be generated by other processes on the
  endpoints themselves (e.g., discovery protocols) or by other
  endpoints not presently under test.  Such unwanted traffic should be removed or avoided to the
  greatest extent possible”

your proposal LGTM.

Page 7:
“test case which are” --> “test case that is”

Section 5.3 :
 “It is expected… “ should be prepended by “Expected behavior: ” to make the layout similar to previous test cases.
Right.

Page 16 last bullet before table 3 : “ where x is the bottleneck capacity specified in Table 4 “, I guess this should also apply to values in table 3 if a UDP load is used for the forward direction.
I need we need to correct the formula and text accordingly here. See my comments to Sergio’s review.

Section 5.5 :
“It is expected… “ should be prepended by “Expected behavior: ” to make the layout similar to previous test cases.
Yes.

Section 5.7 :
“Expected behavior: ….” missing

Section 5.8 :
“Expected behavior: ….” missing

Thanks for pointing out. I dont really remember now why is that I will check with other authors if we had any reason for that, if it is unintentional then we need to add it.

I think we can add this alternative too.

Regards
/Ingemar

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 24 Aug 2018 12:04:45 +0000

Hi
A suggested addition to section 6.1 is a add something like
===
Another alternative is to implement credit based scheduling, this approach is further explained in section 4.1.2.8 in [RFC8298] and also in [REF].
===
Where is [REF] is
https://www.hindawi.com/journals/wcmc/2018/3142496/
/Ingemar

From: Zaheduzzaman Sarker
Sent: den 23 augusti 2018 11:07
To: Ingemar Johansson S <[email-masked]>; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: WGLC comments draft-ietf-rmcat-eval-test-06

Hi Ingemar,

BR

Zahed

Hi

A few editorial nits :

your proposal LGTM.

Page 7:

Section 5.3 :

Section 5.5 :

Section 5.7 :

Section 5.8 :

Regards
/Ingemar