Subject: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-wireless-tests-09: (with DISCUSS and COMMENT)

From: Roman Danyliw via Datatracker <[email-masked]>
Date: Wed, 04 Mar 2020 07:16:07 -0800

Roman Danyliw has entered the following ballot position for
draft-ietf-rmcat-wireless-tests-09: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-rmcat-wireless-tests/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Please let me know if I've misunderstood the test execution protocol
incorrectly:

Section 6.  Per the paragraph “The evaluation of the test cases are intended to
carry out in a controlled lab environment … It is important to take appropriate
caution to avoid leaking non-responsive traffic from unproven congestion
avoidance techniques onto the open Internet”, this is good guidance in general
case.  However, in the case of this document how applicable is it?  Didn’t
Section 3 (“We, therefore, recommend that a cellular network simulator is used
for the test cases defined in this document …” and practically establish it
can’t be done without simulation with the scenario of the underground mine) and
Section 4 (“We recommend to carry out the test cases as defined in this
document using a simulator, such as [NS-2] or [NS-3]).   If all the testing is
supposed to be in a simulator how is it leaking out onto the internet?  As far
as I can tell, this helpful text is common in RMCAT document, but in this case
could it please be tailored for the proposed testing regime.

Perhaps something on the order adding text on the order of “Given the
difficulty of deterministic wireless testing, it is RECOMMENDED and expected
that the tests described in this document would be done via simulation. 
However, <in the case of not doing it that way> <leave the existing language>”

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

** Section 3.1.  Per “In this test case, each of the user/UE in the media
session is an RMCAT compliant endpoint”, what is a “RMCAT compliant endpoint”? 
Could this be cited please.

** Section 3.1.  Per “At the beginning of the simulation, there should be
enough time to warm-up the network”, intuitively the notion of “warm[ing]-up
the network” makes sense.  However, is more precision required to side-by-side
analysis of test runs of when the network is “warm enough”?

** Section 4.  Per “Statistics collected from enterprise Wi-Fi networks show
that the two dominant physical modes are 802.11n and 802.11ac, accounting for
41% and 58% of connected devices”, it is would be valuable to cite this value
and provide a timestamp.  This distribution will certainly change as this
document ages.

** Section 4.  Per “Unless otherwise mentioned, test cases in this section are
described using the underlying PHY- and MAC-layer parameters based on the IEEE
802.11n Standard.”, this focus on only 802.11n is surprising, since the next
sentence establishes that 802.11.n is already less than half (41%) of the Wi-Fi
traffic (and likely will continue to shrink).  Why not ac?

** There appear to be some differences between the description of the Cellular
(Section 3) and Wifi (Section 4) test. -- Section 4.1.2 and 4.2.2 are called
“Test setup”, but Section 3.1.2 and 3.2.2. are called “Simulation setup”.  Was
this intentional?

-- Section 4.*.4 discusses the expected test behavior, but Section 3.* does
not?  Was that explicit?

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 4 Mar 2020 16:50:18 +0100

Hi Roman,

Just on your discuss point below.

> On 4. Mar 2020, at 16:16, Roman Danyliw via Datatracker <[email-masked]> wrote:
> 

I would say that the proposed test cases are rather independent of the method used for evaluation: this could be simulation, emulation (where you plug a part of the simulated network into a real network), or testbed-based. Yes the author recommend simulation because having a real system would be really complicated (but not impossible…) and not necessarily easily provide reproducible results. However, as you note correctly there are no issues with potentially overloading a real network when you use simulations, that's why this case is not discussed in the security considerations section.

I guess we could clarify this in the text, however, note that this is the same text we also have in draft-ietf-rmcat-eval-test which is waiting in the RFC editor queue for this document.

Mirja

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Fri, 13 Mar 2020 19:31:22 +0000

Hi Roman, 

Many thanks for your thorough review of this draft.  We have updated it to version -11: 
https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-wireless-tests-11

Please also see our detailed responses, tagged [authors] inline below. 

Best regards,
Xiaoqing (on behalf of all authors)

On 3/4/20, 9:16 AM, "Roman Danyliw via Datatracker" <[email-masked]> wrote:

    

    

[authors] Thanks for the suggestion.  We have now revised the paragraph (now in Sec. 5)
to begin with: 

	Given the difficulty of deterministic wireless testing, it is
   	recommended and expected that the tests described in this document
	 would be done via simulations.  However, in the case where these test
   	cases are carried out in a testbed setting, the evaluation should
   	take place in a controlled lab environment.

   
[authors] We realize that the term RMCAT will no longer be relevant after all works of the 
WG are completed. We have therefore revised the sentence (now in Sec. 2.1 and 2.2) to:

	In this test case, each user/UE in the media session is an endpoint following RTP-based
   	congestion control.

[authors] Good point. In the test case, the evaluation time is specified to be 30 seconds
after the start of the simulation. We've added the following as clarifications to the beginning
of both Sections 2.1 and 2.2 (previously 3.1 and 3.2):

	Typically, the evaluation period starts 30 seconds after test initialization.

[authors] These statistics are collected from our internal analytics dashboard
based from enterprise customer deployment. Unfortunately, we cannot provide
a public reference link to that data source. We have searched for alternative
sources of similar information (such as a white paper) but haven't yet found a
good substitute. If you are aware of any pointers that's very much welcome.

[authors] This draft was initially developed in 2015. At that time 11n was still the
dominant technology for endpoints and 11ac was not supported in ns-3.
We later updated the statistics on respective portions of 11n vs. 11ac clients in the draft.
On the other hand, since we haven't yet got the time/resource to update all ns3-based
test case implementations to be based on 11ac, our test case specifications are stuck
with the 11n mode.  Instead, we've added some text right after that sentence to encourage
practitioners to evaluate using up-to-date versions of Wi-Fi:

	As Wi-Fi standards evolve over time -- for instance, with the introduction of the
	emerging Wi-Fi 6 (based on IEEE 802.11ax) products -- the PHY- and MAC-layer
	test case specifications need to be updated accordingly to reflect such changes. 

[authors] This is an artificat due to the merge of two separate drafts. Since
our validation efforts for cellular is in simulation only whereas for WiFi we've
implemented both simulation-based and testbed-based tests, we've updated the 
headings for Wi-Fi tests to be ' Test/simulation setup' instead. 

    
 [authors]. Again, this inconsistency in style is due to the draft merging. The draft originally
contained some inline texts on the expected behavior of cellular test cases.  Per your comment,
we have moved them out to be stand-along subsections (Sec. 2.1.3 and 2.2.3), which reads:

	The investigated congestion control algorithms should result in
   	maximum possible network utilization and stability in terms of rate
   	variations, lowest possible end to end frame latency, network latency
   	and Packet Loss Rate (PLR) at different cell load levels.