Subject: [rmcat] Updates for Test Case 5.5 and 5.6 in draft-ietf-rmcat-eval-test

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Thu, 3 Sep 2015 20:22:49 +0000

Hi all,

We are about to update the test case draft to keep it alive by September 11, 2015. And, as co-author, I would like to poll the mailing list regarding two proposed changes for Test Case 5.5 and 5.6.

  *   Test Case 5.5 (RTT Fairness):  currently all flows start gradually, and that may lead to late-coming flows settling at higher rates if they rely on queuing delay measurements.  Since the intention of this test case is to measure fairness w.r.t RTT, would it make more sense to start all flows at the same time, or roughly the same time (e.g., random start within 5 seconds) to avoid flow synchronization effects?  Notice that if any candidate scheme has an issue with late-comer advantage, that should have already been exposed by Test Case 5.4.
  *   Test Case 5.6 (RMCAT vs. Long TCP): currently 3 bottleneck queue sizes are specified:  20ms, 300ms, 1000ms. Given that most AQM schemes (CoDel and PIE) are striving to contain queuing delay with a target of 20ms, while allowing for a larger physical queue size to accommodate instantaneous bursts, a value of 20m for a drop-tail queue depth seems rather out of place.  Should we simply remove the value of 20ms for now?   If we want testers to play with a lower queue depth, we may start with 100ms of queuing, which corresponds to the bandwidth-delay product along that path (50ms propagation each direction).  On a related note: the queue depth for QoS aware in the eval-criteria draft (Sec 5.3) is specified as 70ms. Just wondering how that number came about?

Basically, I would like to get a sense of how much debate is needed regarding possible revisions of the above two points. So as to decide whether we should make changes to the draft now, or defer till after more discussions on the list or in IETF-94.

Thanks,
Xiaoqing

---

From: Michael Welzl <[email-masked]>
Date: Fri, 4 Sep 2015 09:21:17 +0200

> On 03 Sep 2015, at 22:22, Xiaoqing Zhu (xiaoqzhu) <[email-masked]> wrote:
> 
> Hi all, 
> 
> 	• Test Case 5.5 (RTT Fairness):  currently all flows start gradually, and that may lead to late-coming flows settling at higher rates if they rely on queuing delay measurements.  Since the intention of this test case is to measure fairness w.r.t RTT, would it make more sense to start all flows at the same time, or roughly the same time (e.g., random start within 5 seconds) to avoid flow synchronization effects?  Notice that if any candidate scheme has an issue with late-comer advantage, that should have already been exposed by Test Case 5.4. 

I'd say no to this one: I don't understand how starting at the same time or roughly at the same time would avoid flow synchronization effects?  RTT Fairness should clearly work if flows join later or all at roughly the same time. Also note that randomness only makes sense to introduce if there is a minimum number of repetitions and we consider the distribution of output data, i.e. it's not particularly useful when showing plots over time as is the common thing here.

> 	• Test Case 5.6 (RMCAT vs. Long TCP): currently 3 bottleneck queue sizes are specified:  20ms, 300ms, 1000ms. Given that most AQM schemes (CoDel and PIE) are striving to contain queuing delay with a target of 20ms, while allowing for a larger physical queue size to accommodate instantaneous bursts, a value of 20m for a drop-tail queue depth seems rather out of place.  Should we simply remove the value of 20ms for now?   If we want testers to play with a lower queue depth, we may start with 100ms of queuing, which corresponds to the bandwidth-delay product along that path (50ms propagation each direction).  On a related note: the queue depth for QoS aware in the eval-criteria draft (Sec 5.3) is specified as 70ms. Just wondering how that number came about? 

I have no strong opinion on this one; though, in a way I'd say it's a good thing that the 20ms case highlights what happens when the queue is below the target, we should at least know what controllers do under such circumstances?  But then this may just twist the result strangely if it's part of a test case, not a test case on its own, so I can understand that it seems out of place, as you say...

Cheers,
Michael

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Fri, 4 Sep 2015 20:18:56 +0000

Hi Michael, 

Thanks for your input.  A few more comments below starting with [XZ]

Best,
Xiaoqing

On 9/4/15, 2:21 AM, "Michael Welzl" <[email-masked]> wrote:

>> We are about to update the test case draft to keep it alive by
>>September 11, 2015. And, as co-author, I would like to poll the mailing
>>list regarding two proposed changes for Test Case 5.5 and 5.6.
>> 	€ Test Case 5.5 (RTT Fairness):  currently all flows start gradually,
>>and that may lead to late-coming flows settling at higher rates if they
>>rely on queuing delay measurements.  Since the intention of this test
>>case is to measure fairness w.r.t RTT, would it make more sense to start
>>all flows at the same time, or roughly the same time (e.g., random start
>>within 5 seconds) to avoid flow synchronization effects?  Notice that if
>>any candidate scheme has an issue with late-comer advantage, that should
>>have already been exposed by Test Case 5.4.
>
>I'd say no to this one: I don't understand how starting at the same time
>or roughly at the same time would avoid flow synchronization effects?
>RTT Fairness should clearly work if flows join later or all at roughly
>the same time. Also note that randomness only makes sense to introduce if
>there is a minimum number of repetitions and we consider the distribution
>of output data, i.e. it's not particularly useful when showing plots over
>time as is the common thing here.

[XZ] Agree that RTT fairness certainly works even if flows gradually
start. Just that instead of observing same rate per flow (regardless of
RTT), we would observe later flows at higher rate due to late-comer
advantage.  Basically the fairness evaluation will be ³muddled² by a
separate factor independent of path RTT.

[XZ] I like your comment on how experiments involving randomness should be
evaluated.  So, maybe a different approach is to repeat the test case but
with random ordering of the flow start times multiple times and compare
the average rate per flow.  Then we don¹t need to start all flows at the
same time. The change to the description of test case will be minimal.
Will people like this approach better than starting all flows at same
time?    

>
>
>> 	€ Test Case 5.6 (RMCAT vs. Long TCP): currently 3 bottleneck queue
>>sizes are specified:  20ms, 300ms, 1000ms. Given that most AQM schemes
>>(CoDel and PIE) are striving to contain queuing delay with a target of
>>20ms, while allowing for a larger physical queue size to accommodate
>>instantaneous bursts, a value of 20m for a drop-tail queue depth seems
>>rather out of place.  Should we simply remove the value of 20ms for now?
>>  If we want testers to play with a lower queue depth, we may start with
>>100ms of queuing, which corresponds to the bandwidth-delay product along
>>that path (50ms propagation each direction).  On a related note: the
>>queue depth for QoS aware in the eval-criteria draft (Sec 5.3) is
>>specified as 70ms. Just wondering how that number came about?
>
>I have no strong opinion on this one; though, in a way I'd say it's a
>good thing that the 20ms case highlights what happens when the queue is
>below the target, we should at least know what controllers do under such
>circumstances?  But then this may just twist the result strangely if it's
>part of a test case, not a test case on its own, so I can understand that
>it seems out of place, as you say...

[XZ] Not sure if I understand what you mean by ³when the queue is below
the target²? How would source congestion control behave if we indeed have
a bottleneck queue depth of 20ms? I¹d say people are welcome to test out
that scenario. Just that the setup itself does not seem to be motivated by
a practical deployment scenario.

[XZ] In any case I suppose we share the same viewpoint on this one.

---

From: Michael Welzl <[email-masked]>
Date: Fri, 4 Sep 2015 23:21:36 +0200

> On 4. sep. 2015, at 22.18, Xiaoqing Zhu (xiaoqzhu) <[email-masked]> wrote:
> 
> Hi Michael, 
> 

Hm, I didn’t suggest to do this, I just tried to explain why I think randomness is not a good idea to add.
This sounds to me like making it more complicated. If you introduce random ordering of flow start times, you’ll need to make a choice for the interval over which randomness applies. On what basis? If it gets large, you can be back at flows gradually starting again.

All in all, I’m not sure I agree if it makes sense to say that flows gradually starting “muddles” the behavior - you could also say that starting at the same time is a special case…

> the target²?

I meant, what if e.g. you have a 10ms queue but congestion control mechanisms try to operate at 20ms?
(20 / 20 is a special case at the borderline between this (“below the target”) and “normal”)

> How would source congestion control behave if we indeed have

Yes

Cheers,
Michael

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Fri, 4 Sep 2015 22:43:33 +0000

Trying to clarify further with responses below inline.

On 9/4/15, 4:21 PM, "Michael Welzl" <[email-masked]> wrote:

>This sounds to me like making it more complicated. If you introduce
>random ordering of flow start times, you’ll need to make a choice for the
>interval over which randomness applies. On what basis? If it gets large,
>you can be back at flows gradually starting again.
>
>All in all, I’m not sure I agree if it makes sense to say that flows
>gradually starting “muddles” the behavior - you could also say that
>starting at the same time is a special case…

[XZ] Yes, starting at the same time is a special case. But that’s the case
that does not give any non-RTT-related bias for any flows. Since we care
about RTT-fairness in Test Case 5.5, I thought it’s important to design
the test case to avoid those biases.  However, we do not need to stick
with starting at same time.  We could also use random starting older to
avoid such bias.   

[XZ] Let’s go with the concrete example with 5 flows with RTT of 20ms,
50ms, 100ms, 200ms, and 300ms. If they start sequentially, 10 seconds
apart (as specified in the current draft), then the flow with RTT of 300ms
will get a higher rate than the first flow with 20ms even if the
underlying CC algorithm is completely RTT fair. If, instead, we run the
test several times with different starting *order* yet deterministic
starting time (e.g., flow w/ RTT=300ms starting first, flow w/ RTT=20ms
starting second) then that effect will not show up in the final averaged
rate per flow. 

>(20 / 20 is a special case at the borderline between this (“below the

[XZ]  Thanks for clarifying this. Yes, will be interesting to find out. As
an extension, will also be interesting to see how CC schemes with a target
queue delay (e.g., 50ms) interact with an AQM that tries to keep the
queuing delay at 20ms.