Subject: [rmcat] Ongoing working group last calls

From: Colin Perkins <[email-masked]>
Date: Mon, 6 Aug 2018 16:52:27 +0100

RMCAT working group,

As noted at the meeting in Montreal, the following drafts are in working group last call:

- draft-ietf-rmcat-eval-criteria-07
- draft-ietf-rmcat-eval-test-06
- draft-ietf-rmcat-video-traffic-model-04

All three working group last calls were due to conclude at that meeting, but we have not yet received sufficient reviews of these drafts to do so. 

If we are to progress these drafts, we need additional reviews from working group members. Please, do read the drafts. If you have no comments and think the drafts are ready to progress, then please send a note to that effect. If you have comments, please do send them to the list.

Thanks,
Colin (as WG co-chair)

-- 
Colin Perkins
https://csperkins.org/

---

From: Sergio Mena <[email-masked]>
Date: Tue, 21 Aug 2018 00:45:12 +0200

RMCAT working group,

I could finally carve out some time to review 
*draft-ietf-rmcat-eval-test-06*.

The last time read this draft thoroughly was version -01. So what 
follows is based on reviewing the text differences between version -01 
and version -06.

I have classified my feedback into minor feedback (mostly editorial), 
and "less minor".

  * Minor/editorial:
      o Page 3. Last paragraph. "These sources generate media traffic
        and use either congestion control algorithm under
        investigation". Consider changing "either" to "one of the", or
        "any"; as there are (or might be) more than two algorithms.
      o Page 4. Beginning of paragraph just below Figure 1. "In a
        testbed environment where real equipments are used to create a
        laboratory,". This statement's wording sounds strange to me...
        did you mean to say something like the sentence below ?
        "In a testbed environment where real equipment is set up in a
        laboratory network, "
      o Page 4. Same paragraph as previous bullet. Consider changing
        "test bed" to "testbed"
      o Page 8. "A.  End-to-end delay for the congestion controlled
        media flow.". Consider changing "flow" by "flow(s)" for
        consistency with E below
      o Page 17. First paragraph. Consider changing "more than one media
        flows" to "two or more media flows"... alternatively, consider
        changing "flows" to its singular form.
      o Page 19. 4th paragraph. I think mentioning the propagation
        delays here is redundant, as they are specified a few lines below.
      o Page 25. Consider changing "start in an OFF stats" by "start in
        an OFF state" (or maybe I misread something)
      o Page 27. Second last paragraph. I don't really understand what
        "flow S1->R2" means... did you mean S1->R1 ?
  * Less minor:
      o Page 12. "One-way propagation delay: [50 ms, 100 ms]. on the
        forward path direction". One should assume then that the one-way
        propagation delays will be different for forward and backwards
        paths, as the backward path's propagation delay is not mentioned
        (so doesn't change)... is this intended? A clarification would
        be good
      o Page 12. Last paragraph. I see Table 1 has been updated to
        contain capacity factors, rather than absolute values. I'm fine
        with that. Bearing that in mind, I agree with the formula
        (4-x)Mbps; however, the explanation of x still looks like it
        denotes an absolute value (not a factor). I would propose
        something like "(4-x)Mbps, where x is the path capacity ratio
        specified in Table 1"
      o Page 14. Second paragraph. "The reference bottleneck capacity is
        2Mbps [...] source rate changes over time as (4-x)Mbps". I think
        that is inconsistent, as it should read (4-2*x)Mbps, given that
        the reference bottleneck capacity is 2Mbps in this test case.
        Also, same comment about the explanation of x applies here (it
        gives the impression x is an absolute value).
      o Page 16. Same comment on explanation of 'x'
      o Page 19. First paragraph. "end-to-end path latency": I don't
        seem to find this term defined anywhere in the draft. If think I
        should be clearly defined, to avoid different interpretations of
        that it is (e.g., as I interpret it: the delay of a packet
        traveling end to end in the absence of any other traffic).
        Moreover, a few lines below the term "One-Way propagation delay"
        is used. I would unify those two terms (if I didn't
        misunderstand the text, they denote the same concept).
      o Page 22. Section 5.6. Consider making the test duration higher
        (e.g. 300 seconds, just as test case 5.7). In the last months
        (or years), we have tested NADA extensively against TCP using
        this test case, and we found 120 seconds to be too short to
        fully appreciate the behavior of the algorithms (TCP and
        candidate) competing for the bandwidth. We ended up increasing
        the test duration every time we wanted to analyze the results of
        this test case.
      o Page 22. Section 5.6. Consider removing bottleneck queue size of
        1000ms from the test, as 1-second queuing delay while competing
        with TCP (i.e., loss-based) renders the real time media flow
        unusable (unmanageable delay for the user), even if the
        algorithm manages to get its fair share of the bottleneck bandwidth.
      o Page 25. Section 5.7. Consider modifying the test case so that
        the short lived flows stop some time before the end of the test
        (e.g., 60 seconds before the end). This modification permits the
        test to show whether the congestion controlled flows are able to
        come back to normal behavior after all short-lived TCP flows
        have stopped (note that the short-lived TCP flows overshoot the
        bottleneck quite a lot with their current parameters). Quite
        frankly, this is one of the points that makes this test case a
        meaningful one. I remember the rmcat group members discussing
        about the usefulness of this test case in the past. With the
        modification I'm proposing here, it's definitely useful to me.

That's all I have. I hope this feedback helps improve the quality of the 
draft.

Thanks,

Sergio

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 23 Aug 2018 08:21:50 +0000

Hi Sergio,

Thanks for your detailed review. Really appreciate that. Please see inline below.

BR

Zahed

On 2018-08-21, 00:45, "Sergio Mena" <semena=[email-masked]<mailto:semena=[email-masked]>> wrote:

RMCAT working group,

I could finally carve out some time to review draft-ietf-rmcat-eval-test-06.

The last time read this draft thoroughly was version -01. So what follows is based on reviewing the text differences between version -01 and version -06.

I have classified my feedback into minor feedback (mostly editorial), and "less minor".
·         Minor/editorial:
o    Page 3. Last paragraph. "These sources generate media traffic and use either congestion control algorithm under investigation". Consider changing "either" to "one of the", or "any"; as there are (or might be) more than two algorithms.
 Changed to  ”and uses the congestion control algorithm under investigation”. Should be fine now.
o    Page 4. Beginning of paragraph just below Figure 1. "In a testbed environment where real equipments are used to create a laboratory,". This statement's wording sounds strange to me... did you mean to say something like the sentence below ?
"In a testbed environment where real equipment is set up in a laboratory network, "
Yes.
o    Page 4. Same paragraph as previous bullet. Consider changing "test bed" to "testbed"
Done.
o    Page 8. "A.  End-to-end delay for the congestion controlled media flow.". Consider changing "flow" by "flow(s)" for consistency with E below
Done.
o    Page 17. First paragraph. Consider changing "more than one media flows" to "two or more media flows"... alternatively, consider changing "flows" to its singular form.
Ok.
o    Page 19. 4th paragraph. I think mentioning the propagation delays here is redundant, as they are specified a few lines below.
Right.
o    Page 25. Consider changing "start in an OFF stats" by "start in an OFF state" (or maybe I misread something)
Nope, you are right. It should be  ”state”
o    Page 27. Second last paragraph. I don't really understand what "flow S1->R2" means... did you mean S1->R1 ?
Yes.
·         Less minor:
o    Page 12. "One-way propagation delay: [50 ms, 100 ms]. on the forward path direction". One should assume then that the one-way propagation delays will be different for forward and backwards paths, as the backward path's propagation delay is not mentioned (so doesn't change)... is this intended? A clarification would be good
Here we are interested in propagation delay on the path media traffic is traversing. If you note, in the previous section (application related) the media is only on the forward path. Hence, the propagation delay is only mentioned for forward path.
o    Page 12. Last paragraph. I see Table 1 has been updated to contain capacity factors, rather than absolute values. I'm fine with that. Bearing that in mind, I agree with the formula (4-x)Mbps; however, the explanation of x still looks like it denotes an absolute value (not a factor). I would propose something like "(4-x)Mbps, where x is the path capacity ratio specified in Table 1"
Good catch. Thanks. However, if we use path capacity ratio in the text shouldn’t the formula looks like (4 - (4 x Y)), where Y is the path capacity ratio?
o    Page 14. Second paragraph. "The reference bottleneck capacity is 2Mbps [...] source rate changes over time as (4-x)Mbps". I think that is inconsistent, as it should read (4-2*x)Mbps, given that the reference bottleneck capacity is 2Mbps in this test case. Also, same comment about the explanation of x applies here (it gives the impression x is an absolute value).
See my previous comment for the formula. If that is correct then text need to be changed accordingly.
o    Page 16. Same comment on explanation of 'x'
o    Page 19. First paragraph. "end-to-end path latency": I don't seem to find this term defined anywhere in the draft. If think I should be clearly defined, to avoid different interpretations of that it is (e.g., as I interpret it: the delay of a packet traveling end to end in the absence of any other traffic). Moreover, a few lines below the term "One-Way propagation delay" is used. I would unify those two terms (if I didn't misunderstand the text, they denote the same concept).
Again, good catch. I think we should use the “One-Way propagation delay “
o    Page 22. Section 5.6. Consider making the test duration higher (e.g. 300 seconds, just as test case 5.7). In the last months (or years), we have tested NADA extensively against TCP using this test case, and we found 120 seconds to be too short to fully appreciate the behavior of the algorithms (TCP and candidate) competing for the bandwidth. We ended up increasing the test duration every time we wanted to analyze the results of this test case.
So far, in the WG evaluation we have used the test duration as it is. I think as this stage we can stick to that. The test duration however represents the minimum amount of time duration the test must be run to show results to the WG. Algorithm designers are free to use longer version of the test cases.
o    Page 22. Section 5.6. Consider removing bottleneck queue size of 1000ms from the test, as 1-second queuing delay while competing with TCP (i.e., loss-based) renders the real time media flow unusable (unmanageable delay for the user), even if the algorithm manages to get its fair share of the bottleneck bandwidth.
The test case is very hash to media traffic due to the intention of the test case as described. The expected behavior is that the media algorithm avoids congestion collapse and can hold some ground even when going to minimum bitrate. Again, we have been seen results with this configuration, and if not completely broken we should stick to it.
o    Page 25. Section 5.7. Consider modifying the test case so that the short lived flows stop some time before the end of the test (e.g., 60 seconds before the end). This modification permits the test to show whether the congestion controlled flows are able to come back to normal behavior after all short-lived TCP flows have stopped (note that the short-lived TCP flows overshoot the bottleneck quite a lot with their current parameters). Quite frankly, this is one of the points that makes this test case a meaningful one. I remember the rmcat group members discussing about the usefulness of this test case in the past. With the modification I'm proposing here, it's definitely useful to me.
Personally, I don’t see any issue with adding a timeline for short-lived flow in the test case as you propose. AFAIK, as we have number of test cases where the “back to normal behavior” is already shown, the extra time for media traffic was skipped in the particular test case. The intention of this test case is to observe how the candidate algorithm dose during the short-term bursts, not the recovery after the short-term flows are gone.

That's all I have. I hope this feedback helps improve the quality of the draft.

Thanks,

Sergio

RMCAT working group,

Thanks,

Colin (as WG co-chair)

---

From: Sergio Mena <[email-masked]>
Date: Fri, 24 Aug 2018 16:12:57 +0200

Hi Zahed,

Please see inline [SM]

Thanks,

Sergio

On 23/08/18 10:21, Zaheduzzaman Sarker wrote:
> On 2018-08-21, 00:45, "Sergio Mena" <semena=[email-masked] 
> <mailto:semena=[email-masked]>> wrote:
>
> RMCAT working group,
>
> ·Minor/editorial:
> oPage 3. Last paragraph. "These sources generate media traffic and use 
> either congestion control algorithm under investigation". Consider 
> changing "either" to "one of the", or "any"; as there are (or might 
> be) more than two algorithms.
>
> oPage 4. Beginning of paragraph just below Figure 1. "In a testbed 
> environment where real equipments are used to create a laboratory,". 
> This statement's wording sounds strange to me... did you mean to say 
> something like the sentence below ?
> oPage 4. Same paragraph as previous bullet. Consider changing "test 
> bed" to "testbed"
>
> Done.
>
> oPage 8. "A.  End-to-end delay for the congestion controlled media 
> flow.". Consider changing "flow" by "flow(s)" for consistency with E 
> below
>
> Done.
>
> oPage 17. First paragraph. Consider changing "more than one media 
> oPage 19. 4th paragraph. I think mentioning the propagation delays 
> here is redundant, as they are specified a few lines below.
>
> Right.
>
> oPage 25. Consider changing "start in an OFF stats" by "start in an 
> OFF state" (or maybe I misread something)
>
> oPage 27. Second last paragraph. I don't really understand what "flow 
> S1->R2" means... did you mean S1->R1 ?
>
> Yes.
>
> ·Less minor:
>
> oPage 12. "One-way propagation delay: [50 ms, 100 ms]. on the forward 
> path direction". One should assume then that the one-way propagation 
> delays will be different for forward and backwards paths, as the 
> backward path's propagation delay is not mentioned (so doesn't 
> change)... is this intended? A clarification would be good
>
> Here we are interested in propagation delay on the path media traffic 
> is traversing. If you note, in the previous section (application 
> related) the media is only on the forward path. Hence, the propagation 
> delay is only mentioned for forward path.
>
[SM] I see. I still think the text should say the backward path's 
propagation delay doesn't change. That would help an implementer that is 
setting up the testbed for this test case. Also, bear in mind that the 
backward path's propagation delay affects the time at which the feedback 
message will be received, and acted upon.
>
> oPage 12. Last paragraph. I see Table 1 has been updated to contain 
> capacity factors, rather than absolute values. I'm fine with that. 
> Bearing that in mind, I agree with the formula (4-x)Mbps; however, the 
> explanation of x still looks like it denotes an absolute value (not a 
> factor). I would propose something like "(4-x)Mbps, where x is the 
> path capacity ratio specified in Table 1"
>
> Good catch. Thanks. However, if we use path capacity ratio in the text 
> shouldn’t the formula looks like (4 - (4 x Y)), where Y is the path 
> capacity ratio?
>
[SM] Mmmh, I don't understand the meaning of the inner 4. Since the 
Reference bottleneck capacity is 1 Mbps, I would imagine something like 
"(4 - (Y x r)), where Y is the path capacity ratio, and r is the 
Reference bottleneck capacity in Mbps (that is, r=1)". Let me know if 
you don't agree.
>
> oPage 14. Second paragraph. "The reference bottleneck capacity is 
> 2Mbps [...] source rate changes over time as (4-x)Mbps". I think that 
> is inconsistent, as it should read (4-2*x)Mbps, given that the 
> reference bottleneck capacity is 2Mbps in this test case. Also, same 
> comment about the explanation of x applies here (it gives the 
> impression x is an absolute value).
>
[SM] Along the lines of my previous comment, in this case I'd propose 
the same formula, but now r=2
>
> oPage 16. Same comment on explanation of 'x'
>
> oPage 19. First paragraph. "end-to-end path latency": I don't seem to 
> find this term defined anywhere in the draft. If think I should be 
> clearly defined, to avoid different interpretations of that it is 
> (e.g., as I interpret it: the delay of a packet traveling end to end 
> in the absence of any other traffic). Moreover, a few lines below the 
> term "One-Way propagation delay" is used. I would unify those two 
> terms (if I didn't misunderstand the text, they denote the same concept).
>
> oPage 22. Section 5.6. Consider making the test duration higher (e.g. 
> 300 seconds, just as test case 5.7). In the last months (or years), we 
> have tested NADA extensively against TCP using this test case, and we 
> found 120 seconds to be too short to fully appreciate the behavior of 
> the algorithms (TCP and candidate) competing for the bandwidth. We 
> ended up increasing the test duration every time we wanted to analyze 
> the results of this test case.
>
> So far, in the WG evaluation we have used the test duration as it is. 
> I think as this stage we can stick to that. The test duration however 
> represents the minimum amount of time duration the test must be run to 
> show results to the WG. Algorithm designers are free to use longer 
> version of the test cases.
>
[SM] Fair enough
>
> oPage 22. Section 5.6. Consider removing bottleneck queue size of 
> 1000ms from the test, as 1-second queuing delay while competing with 
> TCP (i.e., loss-based) renders the real time media flow unusable 
> (unmanageable delay for the user), even if the algorithm manages to 
> get its fair share of the bottleneck bandwidth.
>
> The test case is very hash to media traffic due to the intention of 
> the test case as described. The expected behavior is that the media 
> algorithm avoids congestion collapse and can hold some ground even 
> when going to minimum bitrate. Again, we have been seen results with 
> this configuration, and if not completely broken we should stick to it.
>
[SM] I see your point. Moreover, I realize I hadn’t though that we would 
want to make sure that, at least, the candidate algorithm doesn't cause 
the TCP flow to starve in such situation, which would be unacceptable.
>
> oPage 25. Section 5.7. Consider modifying the test case so that the 
> short lived flows stop some time before the end of the test (e.g., 60 
> seconds before the end). This modification permits the test to show 
> whether the congestion controlled flows are able to come back to 
> normal behavior after all short-lived TCP flows have stopped (note 
> that the short-lived TCP flows overshoot the bottleneck quite a lot 
> with their current parameters). Quite frankly, this is one of the 
> points that makes this test case a meaningful one. I remember the 
> rmcat group members discussing about the usefulness of this test case 
> in the past. With the modification I'm proposing here, it's definitely 
> useful to me.
>
> Personally, I don’t see any issue with adding a timeline for 
> short-lived flow in the test case as you propose. AFAIK, as we have 
> number of test cases where the “back to normal behavior” is already 
> shown, the extra time for media traffic was skipped in the particular 
> test case. The intention of this test case is to observe how the 
> candidate algorithm dose during the short-term bursts, not the 
> recovery after the short-term flows are gone.
>
[SM] This is the test case where the competing traffic (TCP) overshoots 
the bottleneck the most, so I still think it is the best place to 
observe whether the candidate algorithms are able to come back to normal 
operation after the overshooting ceases. But anyway, that's just my 
opinion; I understand those changes are not trivial to make at this 
stage of the process.

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Tue, 28 Aug 2018 09:48:21 +0000

Hi,

Please see inline below.

BR

Zahed

·         Less minor:
o    Page 12. "One-way propagation delay: [50 ms, 100 ms]. on the forward path direction". One should assume then that the one-way propagation delays will be different for forward and backwards paths, as the backward path's propagation delay is not mentioned (so doesn't change)... is this intended? A clarification would be good
Here we are interested in propagation delay on the path media traffic is traversing. If you note, in the previous section (application related) the media is only on the forward path. Hence, the propagation delay is only mentioned for forward path.
[SM] I see. I still think the text should say the backward path's propagation delay doesn't change. That would help an implementer that is setting up the testbed for this test case.
OK.
Also, bear in mind that the backward path's propagation delay affects the time at which the feedback message will be received, and acted upon.

o    Page 12. Last paragraph. I see Table 1 has been updated to contain capacity factors, rather than absolute values. I'm fine with that. Bearing that in mind, I agree with the formula (4-x)Mbps; however, the explanation of x still looks like it denotes an absolute value (not a factor). I would propose something like "(4-x)Mbps, where x is the path capacity ratio specified in Table 1"
Good catch. Thanks. However, if we use path capacity ratio in the text shouldn’t the formula looks like (4 - (4 x Y)), where Y is the path capacity ratio?
[SM] Mmmh, I don't understand the meaning of the inner 4. Since the Reference bottleneck capacity is 1 Mbps, I would imagine something like "(4 - (Y x r)), where Y is the path capacity ratio, and r is the Reference bottleneck capacity in Mbps (that is, r=1)". Let me know if you don't agree.
The Inner 4 is the reference bottleneck, your version is much clear. Thanks.

o    Page 14. Second paragraph. "The reference bottleneck capacity is 2Mbps [...] source rate changes over time as (4-x)Mbps". I think that is inconsistent, as it should read (4-2*x)Mbps, given that the reference bottleneck capacity is 2Mbps in this test case. Also, same comment about the explanation of x applies here (it gives the impression x is an absolute value).
See my previous comment for the formula. If that is correct then text need to be changed accordingly.
[SM] Along the lines of my previous comment, in this case I'd propose the same formula, but now r=2

o    Page 22. Section 5.6. Consider removing bottleneck queue size of 1000ms from the test, as 1-second queuing delay while competing with TCP (i.e., loss-based) renders the real time media flow unusable (unmanageable delay for the user), even if the algorithm manages to get its fair share of the bottleneck bandwidth.
The test case is very hash to media traffic due to the intention of the test case as described. The expected behavior is that the media algorithm avoids congestion collapse and can hold some ground even when going to minimum bitrate. Again, we have been seen results with this configuration, and if not completely broken we should stick to it.
[SM] I see your point. Moreover, I realize I hadn’t though that we would want to make sure that, at least, the candidate algorithm doesn't cause the TCP flow to starve in such situation, which would be unacceptable.

o    Page 25. Section 5.7. Consider modifying the test case so that the short lived flows stop some time before the end of the test (e.g., 60 seconds before the end). This modification permits the test to show whether the congestion controlled flows are able to come back to normal behavior after all short-lived TCP flows have stopped (note that the short-lived TCP flows overshoot the bottleneck quite a lot with their current parameters). Quite frankly, this is one of the points that makes this test case a meaningful one. I remember the rmcat group members discussing about the usefulness of this test case in the past. With the modification I'm proposing here, it's definitely useful to me.
Personally, I don’t see any issue with adding a timeline for short-lived flow in the test case as you propose. AFAIK, as we have number of test cases where the “back to normal behavior” is already shown, the extra time for media traffic was skipped in the particular test case. The intention of this test case is to observe how the candidate algorithm dose during the short-term bursts, not the recovery after the short-term flows are gone.
[SM] This is the test case where the competing traffic (TCP) overshoots the bottleneck the most, so I still think it is the best place to observe whether the candidate algorithms are able to come back to normal operation after the overshooting ceases. But anyway, that's just my opinion; I understand those changes are not trivial to make at this stage of the process.

Thanks,

Sergio

RMCAT working group,

Thanks,

Colin (as WG co-chair)