Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

From: Christoph Paasch <[email-masked]>
Date: Tue, 04 Jan 2022 16:23:30 -0800

Hello Al,

thanks a lot for your feedback! Please see inline:

> On Dec 17, 2021, at 3:50 PM, MORTON JR., AL <[email-masked]> wrote:
> 
> Hi authors and ippm-chairs,
>  
> Thanks for writing this-up!
>  
> I took one pass through, and have the following comments during Adoption call for draft-cpaasch-ippm-responsiveness:
>  
> TL;DR:
> Many previously undefined terms were used here, and a more direct description using the term “saturation” seems possible, IMO.

I fully agree with you. We are not very good at describing the "working conditions"/"saturation" we are aiming for, why we use these and what the right approach is.

The type of "working conditions" is crucial as to what the measurement result will be
For example, flooding the network with UDP traffic will saturate the network pretty well, but it is far away from a realistic working condition.

What we are aiming for is near worst-case scenario while still being realistic. At least, that is the intention and it may be good to have an open discussion about this.

> IPPM has used a template for metric drafts, and use of the hierarchy of singleton, sample, and statistic metrics from RFC 2330 will help with clarity/answer many of my questions.
>  
> regards (I’m off-line for a while now, so enjoy the holidays),
> Al
>  
> From the Abstract:
>  
>    This document specifies the "RPM Test" for measuring responsiveness.
>    It uses common protocols and mechanisms to measure user experience
>    especially when the network is fully loaded ("responsiveness under
>    working conditions".)  The measurement is expressed as "Round-trips
>    Per Minute" (RPM) and should be included with throughput (up and
>    down) and idle latency as critical indicators of network quality.
>  
> “fully loaded” and “working conditions” aren’t necessarily the same, to me.  I’ll be looking for better definitions.

Agree'd.

>  
> 3 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-3>.  Goals
>  
>    The algorithm described here defines an RPM Test that serves as a
>    good proxy for user experience.  This means:
>  
>    1.  Today's Internet traffic primarily uses HTTP/2 over TLS.  Thus,
>        the algorithm should use that protocol.
>  
>        As a side note: other types of traffic are gaining in popularity
>        (HTTP/3) and/or <???? UDP ???> are already being used widely (RTP).
>  
> There are many measurement stability challenges when TCP is involved, see section 4 of RFC8337:  https://datatracker.ietf.org/doc/html/rfc8337#section-4 <https://datatracker.ietf.org/doc/html/rfc8337#section-4>
> RFC8337 intentionally broke the TCP control loop to make measurements in the face of these challenges.

Yes, we are aware of these kind of stability challenges. And actually sometimes observe them as results can vary to some degree across different runs.

The goal is to get as close as possible to a stable measurement result, while still using the protocols the end-users use on a day-to-day basis.

>  
> 4.1 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-4.1>.  Working Conditions
>  
>    For the purpose of this methodology, typical "working conditions"
>    represent a state of the network in which the bottleneck node is
>    experiencing ingress and egress flows similar to those created by
>    humans in the typical day-to-day pattern.
>  
>    While a single HTTP transaction might briefly put a network into
>    working conditions, making reliable measurements requires maintaining
>    the state over sufficient time.
>  
>    The algorithm must also detect when the network is in a persistent
>    working condition, also called "saturation".
>  
>    Desired properties of "working condition":
>  
>    o  Should not waste traffic, since the person may be paying for it
>  
>    o  Should finish within a short time to avoid impacting other people
>       on the same network, to avoid varying network conditions, and not
>       try the person's patience.
>  
> These seem like reasonable goals for the traffic that loads the network.
> New terms needing definition were introduced:
> “persistent working condition = saturation”,
> which is different from
> “ingress and egress flows similar to those created by humans in the typical day-to-day pattern”
>  
> Later in 4.1.1, terms like “saturate a path”  and “fill the pipe” appear, and
>  
>    The goal of the RPM Test is to keep the network as busy as possible
>    in a sustained and persistent way.  It uses multiple TCP connections
>    and gradually adds more TCP flows until saturation is reached.
>  
> The terms “busy as possible”, and “typical day-to-day pattern”, or
> “saturation” and “working conditions” indicate different load levels to me.
>  
> @@@@ Suggestion: I think it would help to simplify the terminology in this draft. You intend to measure a saturated path, so just say that. No “typical”, no “working conditions”, etc., in these early sections. 
>  
> The sentence beginning “The goal...” should really appear in Section 3. Goals
>  
> Also, you have defined a measurement method in the sentence, “It uses...” above. This method of adding connections has been observed in other measurement systems, but it isn’t typical of user traffic, especially when each connection has an ~infinite amount of data to send during the test.

From your comments I see that we definitely need a longer explanation of the tradeoffs that are being made of measuring this near worst-case, but realistic scenario. As you correctly point out, we are mixing confusing and sometimes contradictory terms. This needs to be cleaned up.

For all of the above points, I filed: https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/34 <https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/34>

Please feel free to add to the github issue.

>  
> 4.1.2 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-4.1.2>.  Parallel vs Sequential Uplink and Downlink
>  
> ...
>    To measure responsiveness under working conditions, the algorithm
>    must saturate both directions.
>  
> Bi-directional saturation is really atypical of usage. I don’t think the benefit of “more data” pays off.
>  
> ...
>  
>    However, a number of caveats come with measuring in parallel:
>  
>    o  Half-duplex links may not permit simultaneous uplink and downlink
>       traffic.  This means the test might not saturate both directions
>       at once.
>  
>    o  Debuggability of the results becomes harder: During parallel
>       measurement it is impossible to differentiate whether the observed
>       latency happens in the uplink or the downlink direction.
>  
>    o  Consequently, the test should have an option for sequential
>       testing.
>  
> @@@@ Suggestion: IMO, tests/results with Downlink saturation OR Uplink saturation would be more straightforward, and can be understood by users (especially those who have tested in the past). Avoid the pitfalls and make Sequential testing the preferred option.

I tend to agree with you. We can leave "Parallel" as an interesting extension to the test that can expose other types of characteristics of the network.

> 4.1.3 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-4.1.3>.  Reaching saturation
>  
>    The RPM Test gradually increases the number of TCP connections and
>    measures "goodput" - the sum of actual data transferred across all
>    connections in a unit of time.  When the goodput stops increasing, it
>    means that saturation has been reached.
> ...
>  
>    Filling buffers at the bottleneck depends on the congestion control
>    deployed on the sender side.  Congestion control algorithms like BBR
>    may reach high throughput without causing queueing because the
>    bandwidth detection portion of BBR effectively seeks the bottleneck
>    capacity.
>  
>    RPM Test clients and servers should use loss-based congestion
>    controls like Cubic to fill queues reliably.
>  
> With the evolution of Congestion control algorithms seeking to avoid filling buffers, does it make sense to require a full buffer at the bottleneck to achieve saturation?
> In fact, the definition above, “When the goodput stops increasing,...” does not require full buffers; it requires maximizing a delivery rate measurement instead.

The above paragraph on BBR vs Cubic should probably be changed. With our goal being to measure "realistic" usage patterns, the recommendation should be to use the congestion control that is currently most widely deployed. If the majority of the Internet switches to BBR, then that's what should be measured.

> In 4.1.4, the final steps of the algorithm were not clear to me:
>  
>       *  Else, network reached saturation for the current flow count.
> @@@@ This wording implies it to be the final step, but there are further conditions to test.
>      Maybe this step is “Else, Candidate for stable saturation”?

Sounds good!

>  
>          +  If new flows added and for 4 seconds the moving average
>             throughput did not change: network reached stable saturation
> @@@@ Maybe: 
>          +  If the 4 second moving average of "instantaneous aggregate goodput" with no new 
>             flows added did not change 
>             (defined as: moving average = "previous" moving average +/- 5%),
>             then the network reached stable saturation

That's better!

> ----------------------------------------------------------------------------------------------------
>  
>          +  Else, add four more flows
> @@@ ??? and return to start?

Yes, the entire thing is evaluated every 1-second interval. I will make that explicit.

>  
>  
> Finally, in 4.1.4, the Note explains:
>  
>    Note: It is tempting to envision an initial base RTT measurement and
>    adjust the intervals as a function of that RTT.  However, experiments
>    have shown that this makes the saturation detection extremely
>    unstable in low RTT environments.  In the situation where the
>    "unloaded" RTT is in the single-digit millisecond range, yet the
>    network's RTT increases under load to more than a hundred
>    milliseconds, the intervals become much too low to accurately drive
>    the algorithm.
>  
> Well, TCP senders/control-loops are involved here, and likely play a
> role in behavior categorized as “difficult to measure”.
>  
> By the time we get to
>  
> 4.2 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-4.2>.  Measuring Responsiveness
>  
>    Once the network is in a consistent working conditions, the RPM Test
>    must "probe" the network multiple times to measure its
>    responsiveness.
>  
>    Each RPM Test probe measures:
>  
> You previously started at least four TCP connections with infinitely large files.
> The “create connection” RPM probes establish additional connections, DNS, TCP, etc.
> Is each new connection an RPM probe? or is the set of connection tests a single probe?
> (later we learn it is the set of
> What if one of the set of connections fails/times-out?
>  
> I take it that the “load-bearing” connections are driving the path to saturation.
> Maybe “load-generating connections” is more clear?

"load-generating" is indeed a better term!

For all of the above "minor" edits, I filed https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/35 <https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/35> 

> 4.2.1 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-4.2.1>.  Aggregating the Measurements
>  
>    The algorithm produces sets of 5 times for each probe, namely: DNS
>    handshake, TCP handshake, TLS handshake, HTTP/2 request/response on
>    separate (idle) connections, HTTP/2 request/response on load bearing
>    connections.  This fine-grained data is useful, but not necessary for
>    creating a useful metric.
>  
> @@@@ So, only ONE of the load-generating connections runs the 1-byte GET ?  (it says connections, and there are at least 4) The various handshakes result in 4 RT measurements.
>  
> But, do you mean *5 repeated measurement sets*? Each set with:
> DNS HS, TCP HS, TLS HS, HTTP/2 idle GET, and the potentially much longer GET on the 
> load generating connections.
>  
>    To create a single "Responsiveness" (e.g., RPM) number, this first
>    iteration of the algorithm gives an equal weight to each of these
>    values.  That is, it sums the five time values for each probe, and
>    divides by the total number of probes to compute an average probe
>    duration.  The reciprocal of this, normalized to 60 seconds, gives
>    the Round-trips Per Minute (RPM).
>  
> @@@@ I’m missing a step, I think:
> Are the “time values for each probe” the sum of handshake or response times for
> DNS HS, TCP HS, TLS HS, HTTP/2 idle GET and load generating connections?
> The processing doesn’t seem to include this preliminary calculation to produce 
> “five time values”.  
>  
>  
> In Section 5, “no new protocol is defined”, but
>  
>    The client begins the responsiveness measurement by querying for the
>    JSON configuration.  This supplies the URLs for creating the load
>    bearing connections in the upstream and downstream direction as well
>    as the small object for the latency measurements.
>  
> The client needs to know how the server response will be organized, down to key: value, right?
> Some client and server agreements needed...

Yes! With "protocol" we mean nothing new at HTTP/TCP/... layer. Just standard JSON. I don't know if that qualifies as a new "protocol" ;-)

Thanks a lot,
Christoph

>  
>  
> From: ippm <[email-masked]> On Behalf Of Marcus Ihlar
> Sent: Monday, December 6, 2021 10:53 AM
> To: [email-masked]
> Subject: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness
>  
> Hi IPPM,
>  
> This email starts an adoption call for draft-cpaasch-ippm-responsiveness, "Responsiveness under Working Conditions”. This document specifies the “RPM Test” for measuring user experience when the network is fully loaded. The intended status of the document is Experimental.   
>  
> https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/ <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/__;!!BhdT!zI6d5je1i8cafA6NXByD5tvxHFKKPMjYgtM6t2aLUHFPsyPz-XwPFguwa1HS$>
> https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01 <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01__;!!BhdT!zI6d5je1i8cafA6NXByD5tvxHFKKPMjYgtM6t2aLUHFPsyPz-XwPFq67PfWg$>
>  
> This adoption call will last until Monday, December 20. Please review the document, and reply to this email thread to indicate if you think IPPM should adopt this document.
>  
> BR,
> Marcus 
>  
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 04 Jan 2022 16:32:03 -0800

Hello Matt,

> On Dec 19, 2021, at 7:38 PM, Matt Mathis <mattmathis=[email-masked]> wrote:
> 
> I support this draft, and I have even offered to be a co-author.

thanks for the support and indeed you will be on the next version of the draft.

> Thank you  Al. for the extensive comments.   I only have a moment for some high level responses:
> 
> In some sense this document  is opposite of RFC8337.  Responsiveness is about measuring the how well/early the bottleneck signals congestion to the transport without incurring excessive queueing.   This not only depends on the loop behavior of the bottleneck and the measurement stream, it also depends on the loop behavior, population and RTT's of the cross traffic.   I do not expect any of the theory to be mature enough to be able to do any of this as abstractly as IPPM does with most metrics.   At this time I expect a responsiveness metric to require fingerprinting the hardware, OS and application software, and include a warning that results may be sensitive to the details of the implementation.

Yes - things can be very implementation-dependent. Different service deployments can have different ways of generating the "working conditions". A service with QUIC deployment using BBR may expose entirely different responsiveness metrics than a TCP-based deployment.

I wonder if we can abstract away the definition of "working conditions" enough so that it becomes independent of the implementation details. It's gonna be a tradeoff between how much we want different implementations to measure similar results vs leaving room for implementations to make their own choices.

> WiFi is a poorly modeled half duplex shared broadcast channel, that is a critical (dominant?) part of the responsiveness for typical Internet users.     Bidirectional traffic matters a lot, but I fear the scope might be too broad.

Agree'd. It complicates things by quite a bit.

Christoph

> 
> Thanks,
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
> 
> We must not tolerate intolerance;
>        however our response must be carefully measured: 
>             too strong would be hypocritical and risks spiraling out of control;
>             too weak risks being mistaken for tacit approval.
> 
> 
> On Fri, Dec 17, 2021 at 3:51 PM MORTON JR., AL <[email-masked] <mailto:[email-masked]>> wrote:
> Many previously undefined terms were used here, and a more direct description using the term “saturation” seems possible, IMO. IPPM has used a template for metric drafts, and use of the hierarchy of singleton, sample, and statistic metrics from RFC 2330 will help with clarity/answer many of my questions.
> 
>  
> 
> From: ippm <[email-masked] <mailto:[email-masked]>> On Behalf Of Marcus Ihlar
> To: [email-masked] <mailto:[email-masked]>
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/ippm <https://www.ietf.org/mailman/listinfo/ippm>

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 04 Jan 2022 16:55:11 -0800

And I just realize I jumped over some parts of your comments. Please see more inline:

> On Jan 4, 2022, at 4:23 PM, Christoph Paasch <cpaasch=[email-masked]> wrote:

[snip]

>> What if one of the set of connections fails/times-out?

Yes, the set will be factored into the final RPM number.

>> load generating connections.

You are right, this is not very clear. I am working on testing a refinement of the methodology and how the numbers are aggregated.

I'm thinking of the following:

Each of the RPM-measuring separate connections should get latency numbers for DNS, TCP, TLS and H2 req/response. Additionally, we have the measurement on the load-generating connections (one probe on each of these).

This gives us 5 sets of numbers:
DNS: [42ms, 39ms, 43ms, ...] -> 75th percentile = V
TCP: [..., ..., ...] -> 75th percentile = W
TLS: [... ,..., ...] -> ... = X
H2: [..., ..., ...] ->  ... = Y
H2 on load-generating: [..., ..., ...] -> ... = Z

Then, RPM is the average (normalized to 60 seconds) of these values.

The tricky part is to choose 75th percentile or 90th percentile,... as well as how many probes are needed.

Does this make it clearer on how the probes are measured and create the RPM-value ?

Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 05 Jan 2022 10:52:13 -0800

Hello Greg,

thanks for your comments. Please see inline:

> On Dec 22, 2021, at 11:43 PM, Greg Mirsky <[email-masked]> wrote:
> 
> Dear Marcus, Authors, et al,
> apologies for the belated response.
> I've read the draft and have some comments to share with you:
> as I understand it, the proposed new responsiveness metric is viewed as the single indicator of a bufferbloat condition in a network. As I recall, the discussion at Measuring Network Quality for End-Users workshop and on the mailing list <https://mailarchive.ietf.org/arch/browse/network-quality-workshop/?gbt=1&index=cuW_1lh4DD22V28EvlPFB_NjjZY> indicated, that there’s no consensus on what behaviors, symptoms can reliably signal the bufferbloat.
We are not trying for this responsiveness metric to be "the single indicator of bufferbloat". Bufferbloat can be measured in many different number of ways. And each of these will produce a correct, but a different result. Thus, "bufferbloat" is whatever the methodology tries to detect.

Let me give an example of two methodologies that are both correct but both will produce entirely different numbers :

If we would decide to generate the load by flooding the network with UDP traffic from a specific 4-tuple and measure latency with parallel ICMP pings. Then, on a over-buffered FIFO queue we would measure huge latencies (thus correctly expose bufferbloat), while on a FQ-codel queue we would not measure any bufferbloat.

If on the other hand, the load-generating traffic is changing the source-port for every single UDP-packet, then in both the FIFO-queue and the FQ-codel queue we will measure huge amounts of bufferbloat.

Thus, these two methods both produced correct results but with hugely different numbers in the FQ-codel case. [1]

Now, while both methods measure some variant of bufferbloat, they both don't measure a realistic usage of the network.

That is why the "Responsiveness under working conditions" tries to clearly specify how the load is generated and how the latency is being measured. And it does not measure "bufferbloat" but it measures "responsiveness under working conditions" based on the methodology that is being used (using HTTP/2 or HTTP/3, multiple flows, ...). It does expose bufferbloat which can happen in the network. It also exposes certain server-side behaviors that can cause (huge amounts of) additional latency - those behaviors are typically not called "bufferbloat".

> It seems that it would be reasonable to first define what is being measured, characterized by the responsiveness metric. Having a document that discusses and defines the bufferbloat would be great.
I agree that there is a lack of definition for what "bufferbloat" really is.

The way we look at "responsiveness under working conditions" is that it measures the latency in conditions that may realistically happen in worst-case scenarios with end-users/implementations that are non-malicious (non-malicious to exclude the UDP-flooding scenario).

Thus, I assume we should make a better job at explaining this. The lack of a formal definition of "bufferbloat" doesn't help and thus we are indeed using this term a bit freely in the current draft. We will improve the Introduction to better set the stage (https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31 <https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31>).

> It seems like in the foundation of the methodology described in the draft lies the assumption that without adding new flows the available bandwidth is constant, does not change. While that is mostly the case, there are technologies that behave differently and may change bandwidth because of the outside conditions. Some of these behaviors of links with variable discrete bandwidth are discussed in, for example, RFC 8330 <https://datatracker.ietf.org/doc/rfc8330/> and RFC 8625 <https://datatracker.ietf.org/doc/rfc8625/>.
I'm not sure I entirely understand your comment. But let me explain why we are gradually adding new flows:

1. TCP-implementations have usually a fixed limit for the upper bound of the receive window. In some networks that upper bound is lower than the BDP of the network. Thus, the only way to reach full capacity is by having multiple flows.
2. Having multiple connections allows to quicker achieve full capacity in high-RTT networks and thus speeds up the test-duration.
3. In some networks with "random" packet-loss, congestion-control may come in the way of achieving full capacity. Again, multiple flows will work around that.
> Then, I find the motivation not to use time units to express the responsiveness metric not convincing:
>    "Latency" is a poor measure of responsiveness, since it can be hard
>    for the general public to understand.  The units are unfamiliar
>    ("what is a millisecond?") and counterintuitive ("100 msec - that
>    sounds good - it's only a tenth of a second!").

Can you expand on what exactly is not convincing to you? Do you think that people will mis-understand the metric or that milli-seconds is the right way to communicate responsiveness to the general public?

Thanks a lot,
Christoph

[1] And there are many networks that prioritize ICMP pings, thus we could observe even more different results based on what protocol is used to measure the latency.

> 
> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar <marcus.ihlar=[email-masked] <mailto:[email-masked]>> wrote:
> Hi IPPM,
> 
>  
> 
> https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/ <https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/>
> https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01>
>  
>

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 06 Jan 2022 13:47:29 -0800

Hello Ruediger,

thanks for your feedback! Please see inline:

> On Dec 22, 2021, at 11:16 PM, [email-masked] wrote:
> 
> Hi Marcus,
> 
> Noting that some more work needs to be added, I support adoption.
> 
> Some comments:
> - I'd appreciate if the draft was covering expectable operational scenarios, namely
>  * There may be no bottleneck buffer - the test/metric needs to detect that and act appropriate.

I'm not sure what you mean ? Every flow is limited by some bottleneck that sits somewhere. Sometimes, it's the last mile, sometimes it's the Wi-Fi connectivity, sometimes flows become CPU-limited. But there always is a bottleneck somewhere that is limiting things at which point a buffer is building up (even though that "buffer" may be just 1 packet large)

>  * One might be interested in the metric while the test adds an insignificant load only. While receiver tries to consume service X, what's the metric seen from Y? 
>     (may help to clarify, whether the access causes trouble or another network section).

That would indeed be interesting for trouble-shooting and debugging. I believe this is out-of-scope though.

> - Measuring the maximum RPM as seen from the test server is part of the test or it isn't? That's not clear to me by now.
>  I'm favouring to make measuring the maximum RPM part of the test, this may help to scale results.

The methodology defines to only measure from the client as we are interested in responsiveness as a metric that is (as close as possible) representative of an end-user experience.

When you say "maximum RPM", do you mean that on an "idle" network?

> - The test set-up should prevent application as DoS attack tool.

The json-fetch at the beginning of the test allows server-side CDN deployments to "rate-limit" clients by simply delaying the response to the json-fetch.

What else do you have in mind regarding a DoS attack tool?

Thanks,
Christoph

> Regards,
> 
> Ruediger
> 
> 
> From: Marcus Ihlar <[email-masked] >
> Date: December 6, 2021 at 7:52:58 AM PST
> To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
> List-Id: IETF IP Performance Metrics Working Group 
> 
> Hi IPPM,
> 
> This email starts an adoption call for draft-cpaasch-ippm-responsiveness, "Responsiveness under Working Conditions". This document specifies the "RPM Test" for measuring user experience when the network is fully loaded. The intended status of the document is Experimental.
> 
> https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/
> 
> https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01
> 
> [email-masked]<mailto:[email-masked]>
>

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 6 Jan 2022 15:00:06 -0800

Hi Christoph,
a happy and healthy New Year to you and All!
Thank you for your kind consideration of my notes and detailed responses.
Please find my follow-up notes in-line below under the GIM>> tag.

Regards,
Greg

On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <[email-masked]> wrote:

>    - as I understand it, the proposed new responsiveness metric is viewed
>    as the single indicator of a bufferbloat condition in a network. As I
>    recall, the discussion at Measuring Network Quality for End-Users workshop
>    and on the mailing list
>    <https://mailarchive.ietf.org/arch/browse/network-quality-workshop/?gbt=1&index=cuW_1lh4DD22V28EvlPFB_NjjZY>
>    indicated, that there’s no consensus on what behaviors, symptoms can
>    reliably signal the bufferbloat.
>
> We are not trying for this responsiveness metric to be "the single
> indicator of bufferbloat". Bufferbloat can be measured in many different
> number of ways. And each of these will produce a correct, but a different
> result. Thus, "bufferbloat" is whatever the methodology tries to detect.
>
> Let me give an example of two methodologies that are both correct but both
> will produce entirely different numbers :
>
> If we would decide to generate the load by flooding the network with UDP
> traffic from a specific 4-tuple and measure latency with parallel ICMP
> pings. Then, on a over-buffered FIFO queue we would measure huge latencies
> (thus correctly expose bufferbloat), while on a FQ-codel queue we would not
> measure any bufferbloat.
>
> If on the other hand, the load-generating traffic is changing the
> source-port for every single UDP-packet, then in both the FIFO-queue and
> the FQ-codel queue we will measure huge amounts of bufferbloat.
>
> Thus, these two methods both produced correct results but with hugely
> different numbers in the FQ-codel case. [1]
>
> Now, while both methods measure some variant of bufferbloat, they both
> don't measure a realistic usage of the network.
>
GIM>> Thank you for the insights. It seems to me that what the method can
demonstrate is rather the level of efficiency of the AQM in the network for
a particular class of applications.

>
>
> That is why the "Responsiveness under working conditions" tries to clearly
> specify how the load is generated and how the latency is being measured.
> And it does not measure "bufferbloat" but it measures "responsiveness under
> working conditions" based on the methodology that is being used (using
> HTTP/2 or HTTP/3, multiple flows, ...). It does expose bufferbloat which
> can happen in the network. It also exposes certain server-side behaviors
> that can cause (huge amounts of) additional latency - those behaviors are
> typically not called "bufferbloat".
>
GIM>> Thank you for pointing out that the result of the RTT measurement has
two contributing factors - network and server. It seems worth enhancing the
method to localize each contribution and measure them separately.

>
>
>
>
>    - It seems that it would be reasonable to first define what is being
>    measured, characterized by the responsiveness metric. Having a document
>    that discusses and defines the bufferbloat would be great.
>
> The way we look at "responsiveness under working conditions" is that it
> measures the latency in conditions that may realistically happen in
> worst-case scenarios with end-users/implementations that are non-malicious
> (non-malicious to exclude the UDP-flooding scenario).
>
> Thus, I assume we should make a better job at explaining this. The lack of
> a formal definition of "bufferbloat" doesn't help and thus we are indeed
> using this term a bit freely in the current draft. We will improve the
> Introduction to better set the stage (
> https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31
> ).
>
>
>    - It seems like in the foundation of the methodology described in the
>    draft lies the assumption that without adding new flows the
>    available bandwidth is constant, does not change. While that is mostly the
>    case, there are technologies that behave differently and may change
>    bandwidth because of the outside conditions. Some of these behaviors of
>    links with variable discrete bandwidth are discussed in, for example, RFC
>    8330 <https://datatracker.ietf.org/doc/rfc8330/> and RFC 8625
>    <https://datatracker.ietf.org/doc/rfc8625/>.
>
> 1. TCP-implementations have usually a fixed limit for the upper bound of
> the receive window. In some networks that upper bound is lower than the BDP
> of the network. Thus, the only way to reach full capacity is by having
> multiple flows.
> 2. Having multiple connections allows to quicker achieve full capacity in
> high-RTT networks and thus speeds up the test-duration.
> 3. In some networks with "random" packet-loss, congestion-control may come
> in the way of achieving full capacity. Again, multiple flows will work
> around that.
>
GIM>> I might have asked several questions at once. Let me clarify what I
am looking for:

   - As I understand the method of creating the "working conditions in a
   network" is based on certain assumptions. First, seems is that the
   bandwidth is symmetrical between the measurement points. Second, that the
   bandwidth doesn't change for the duration of the measurement session.
   AFAIK, in the access networks, both are not necessarily always the case.
   - On the other hand, I might have missed how the method of creating the
   "working conditions" guarantees a symmetrical load between the measurement
   points.

>
>    - Then, I find the motivation not to use time units to express the
>    responsiveness metric not convincing:
>
> Can you expand on what exactly is not convincing to you? Do you think that
> people will mis-understand the metric or that milli-seconds is the right
> way to communicate responsiveness to the general public?
>
GIM>> Let me try. We know packet delay requirements for AR, VR
applications. I believe that gamers are familiar with these numbers too.
The same is likely the case for the industrial automation use cases served,
for example, by Deterministic Networking.

>
>
> Thanks a lot,
> Christoph
>
> [1] And there are many networks that prioritize ICMP pings, thus we could
> observe even more different results based on what protocol is used to
> measure the latency.
>
>
> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar <marcus.ihlar=
> [email-masked]> wrote:
>
>> Hi IPPM,
>>
>>
>>
>> This email starts an adoption call for draft-cpaasch-ippm-responsiveness,
>> "Responsiveness under Working Conditions”. This document specifies the “RPM
>> Test” for measuring user experience when the network is fully loaded. The
>> intended status of the document is Experimental.
>>
>>
>>
>> This adoption call will last until *Monday, December 20*. Please review
>> the document, and reply to this email thread to indicate if you think IPPM
>> should adopt this document.
>>
>>
>>
>> BR,
>>
>> Marcus
>>
>>

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 06 Jan 2022 16:42:08 -0800

Hello Greg,

> On Jan 6, 2022, at 3:00 PM, Greg Mirsky <[email-masked]> wrote:
> 
> Hi Christoph,

Happy New Year to you as well!

> Thank you for your kind consideration of my notes and detailed responses. Please find my follow-up notes in-line below under the GIM>> tag.

Thanks for your replies. Please see inline:

> On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
> Hello Greg,
> 
>> On Dec 22, 2021, at 11:43 PM, Greg Mirsky <[email-masked] <mailto:[email-masked]>> wrote:
>> 
> GIM>> Thank you for the insights. It seems to me that what the method can demonstrate is rather the level of efficiency of the AQM in the network for a particular class of applications.

Yes, that is a good description. It is for a "particular class of applications" and we are trying to make this class of applications representative of a "typical user-scenario". (admittedly, we can debate forever on what kind of applications are representative and I would love to have that debate :-)).

On the point of "efficiency of the AQM". I would go even further that it's not only AQM but also the client- and server-side implementations of these applications (as noted further below).

> GIM>> Thank you for pointing out that the result of the RTT measurement has two contributing factors - network and server.

Yes, servers contribute as do the client-side implementations. It's all three (client, network, server) that need to work "correctly" to achieve good responsiveness. Btw., as we are now gathering more experience with our methodology in different environments we find that the biggest portions of latency actually come from the server-side. We see several seconds of latency introduced by the HTTP/2 and TCP implementations.

> It seems worth enhancing the method to localize each contribution and measure them separately.

With the latency measuring probes being sent on load-bearing connections and separate connections and with the separate connections serving to measure DNS/TCP/... individually, the different data-points actually allow to localize to some extend.

However, I would be reluctant to dive too deep into localization/trouble-shooting/debugging of networks as part of this I-D. As this opens a whole new can of worms. We could then start thinking about sending latency-probes while playing with the IP TTL to find which router is introducing the latency,... It's an entirely different research-topic IMO :-) Dave Taht was thinking of starting something along these lines (https://github.com/dtaht/wtbb <https://github.com/dtaht/wtbb>).

> GIM>> I might have asked several questions at once. Let me clarify what I am looking for:
> As I understand the method of creating the "working conditions in a network" is based on certain assumptions. First, seems is that the bandwidth is symmetrical between the measurement points. Second, that the bandwidth doesn't change for the duration of the measurement session. AFAIK, in the access networks, both are not necessarily always the case.
We don't have the assumption that bandwidth is symmetrical (assuming, you mean uplink/downlink symmetry - please clarify otherwise).

The load-generating algorithm runs independently for uplink and downlink traffic. And it is perfectly fine when both have huge asymmetry.

Regarding the stability of the bandwidth:
You are making a good point indeed that we assume that the bandwidth is to some extend stable while ramping up the flows to "working conditions". Admittedly that assumption does not always hold, and that is one of the reasons why we try hard for the test to not take too long.
I'm not sure how we could adjust the algorithm for varying bandwidth without introducing too much complexity. I'm open for suggestions :-)
> On the other hand, I might have missed how the method of creating the "working conditions" guarantees a symmetrical load between the measurement points.
As mentioned above, we don't assume a symmetrical load. Can you show us where in the draft we give that impression, so we can fix that?

> GIM>> Let me try. We know packet delay requirements for AR, VR applications. I believe that gamers are familiar with these numbers too. The same is likely the case for the industrial automation use cases served, for example, by Deterministic Networking.

I can understand that for a technical audience, milli-seconds is easy and familiar. A non-technical audience might be more open to accepting a new "higher-is-better" metric. Responsiveness is something new and abstract so, it's kind of natural that it comes with a new unit.

But I fully recognize that that's a controversial topic and can be discussed at length :)

Cheers,
Christoph

---

From: <[email-masked]>
Date: Fri, 7 Jan 2022 08:03:15 +0000

Hi Christoph,

Replies marked [RG]...

You wrote:

----- -----

Hello Ruediger,

[RG] 1 packet or 0 packets in the case of policing. There may be drops, but no added delay. I'm sure, that this will occur someplace in one direction, whether this condition may affect up- and downstream of the same success, I can't tell.

[RG] If this draft isn't about trouble-shooting and debugging, what is it about? Please mention that capturing above metric is out-of scope in the draft.

[RG] Thanks for asking. Likely "idle network": the maximum number of RPM measured from the client if there's no load on the access. Once these are known, one can relate the minimum number to be measured by design to them (that's a ratio, if my English doesn't leave me).

[RG] Spoofed addresses in any direction, unexpected messages, undesired automated activation in the background of a client.... ideally, there should be no possibility to activate such a test, if the client doesn't want that explicitly. I've also proposed an "emergency stop" for load tests for the capacity measurement metric, and I think such a function may be worth a thought for all kind of tests aiming to fully consume the capacity of a bottleneck.

Regards,

Ruediger