Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

From: Greg Mirsky <[email-masked]>
Date: Tue, 8 Feb 2022 13:10:25 -0800

Hi Christoph,
apologies for the belated response and thank you for sharing interesting
details of you using the measurement method. I think that if the
measurement method can not only provide the Round-trip Per Minute (RPM)
metric but expose the network propagation and residential components of the
round-trip delay, then it seems to me, the scope of the draft to be aligned
with the charter of the IPPM WG and I'll be in favor of the WG adoption of
the work.
What do you think? What is the opinion of the authors and the WG?

Regards,
Greg

On Thu, Jan 6, 2022 at 4:42 PM Christoph Paasch <[email-masked]> wrote:

> Hello Greg,
>
> On Jan 6, 2022, at 3:00 PM, Greg Mirsky <[email-masked]> wrote:
>
> Hi Christoph,
> a happy and healthy New Year to you and All!
>
>
> Happy New Year to you as well!
>
> Thank you for your kind consideration of my notes and detailed responses.
> Please find my follow-up notes in-line below under the GIM>> tag.
>
>
> Thanks for your replies. Please see inline:
>
> On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <[email-masked]>
> wrote:
>
>> Hello Greg,
>>
>> thanks for your comments. Please see inline:
>>
>> On Dec 22, 2021, at 11:43 PM, Greg Mirsky <[email-masked]> wrote:
>>
>> Dear Marcus, Authors, et al,
>> apologies for the belated response.
>> I've read the draft and have some comments to share with you:
>>
>>    - as I understand it, the proposed new responsiveness metric is
>>    viewed as the single indicator of a bufferbloat condition in a network. As
>>    I recall, the discussion at Measuring Network Quality for End-Users
>>    workshop and on the mailing list
>>    <https://mailarchive.ietf.org/arch/browse/network-quality-workshop/?gbt=1&index=cuW_1lh4DD22V28EvlPFB_NjjZY>
>>    indicated, that there’s no consensus on what behaviors, symptoms can
>>    reliably signal the bufferbloat.
>>
>> We are not trying for this responsiveness metric to be "the single
>> indicator of bufferbloat". Bufferbloat can be measured in many different
>> number of ways. And each of these will produce a correct, but a different
>> result. Thus, "bufferbloat" is whatever the methodology tries to detect.
>>
>> Let me give an example of two methodologies that are both correct but
>> both will produce entirely different numbers :
>>
>> If we would decide to generate the load by flooding the network with UDP
>> traffic from a specific 4-tuple and measure latency with parallel ICMP
>> pings. Then, on a over-buffered FIFO queue we would measure huge latencies
>> (thus correctly expose bufferbloat), while on a FQ-codel queue we would not
>> measure any bufferbloat.
>>
>> If on the other hand, the load-generating traffic is changing the
>> source-port for every single UDP-packet, then in both the FIFO-queue and
>> the FQ-codel queue we will measure huge amounts of bufferbloat.
>>
>> Thus, these two methods both produced correct results but with hugely
>> different numbers in the FQ-codel case. [1]
>>
>> Now, while both methods measure some variant of bufferbloat, they both
>> don't measure a realistic usage of the network.
>>
> GIM>> Thank you for the insights. It seems to me that what the method can
> demonstrate is rather the level of efficiency of the AQM in the network for
> a particular class of applications.
>
>
> Yes, that is a good description. It is for a "particular class of
> applications" and we are trying to make this class of applications
> representative of a "typical user-scenario". (admittedly, we can debate
> forever on what kind of applications are representative and I would love to
> have that debate :-)).
>
> On the point of "efficiency of the AQM". I would go even further that it's
> not only AQM but also the client- and server-side implementations of these
> applications (as noted further below).
>
>
>
>>
>> That is why the "Responsiveness under working conditions" tries to
>> clearly specify how the load is generated and how the latency is being
>> measured. And it does not measure "bufferbloat" but it measures
>> "responsiveness under working conditions" based on the methodology that is
>> being used (using HTTP/2 or HTTP/3, multiple flows, ...). It does expose
>> bufferbloat which can happen in the network. It also exposes certain
>> server-side behaviors that can cause (huge amounts of) additional latency -
>> those behaviors are typically not called "bufferbloat".
>>
> GIM>> Thank you for pointing out that the result of the RTT measurement
> has two contributing factors - network and server.
>
>
> Yes, servers contribute as do the client-side implementations. It's all
> three (client, network, server) that need to work "correctly" to achieve
> good responsiveness. Btw., as we are now gathering more experience with our
> methodology in different environments we find that the biggest portions of
> latency actually come from the server-side. We see several seconds of
> latency introduced by the HTTP/2 and TCP implementations.
>
> It seems worth enhancing the method to localize each contribution and
> measure them separately.
>
>
> With the latency measuring probes being sent on load-bearing connections
> and separate connections and with the separate connections serving to
> measure DNS/TCP/... individually, the different data-points actually allow
> to localize to some extend.
>
> However, I would be reluctant to dive too deep into
> localization/trouble-shooting/debugging of networks as part of this I-D. As
> this opens a whole new can of worms. We could then start thinking about
> sending latency-probes while playing with the IP TTL to find which router
> is introducing the latency,... It's an entirely different research-topic
> IMO :-) Dave Taht was thinking of starting something along these lines (
> https://github.com/dtaht/wtbb).
>
>
>>
>>
>>    - It seems that it would be reasonable to first define what is being
>>    measured, characterized by the responsiveness metric. Having a document
>>    that discusses and defines the bufferbloat would be great.
>>
>> I agree that there is a lack of definition for what "bufferbloat" really
>> is.
>>
>> The way we look at "responsiveness under working conditions" is that it
>> measures the latency in conditions that may realistically happen in
>> worst-case scenarios with end-users/implementations that are non-malicious
>> (non-malicious to exclude the UDP-flooding scenario).
>>
>> Thus, I assume we should make a better job at explaining this. The lack
>> of a formal definition of "bufferbloat" doesn't help and thus we are indeed
>> using this term a bit freely in the current draft. We will improve the
>> Introduction to better set the stage (
>> https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31
>> ).
>>
>>
>>    - It seems like in the foundation of the methodology described in the
>>    draft lies the assumption that without adding new flows the
>>    available bandwidth is constant, does not change. While that is mostly the
>>    case, there are technologies that behave differently and may change
>>    bandwidth because of the outside conditions. Some of these behaviors of
>>    links with variable discrete bandwidth are discussed in, for example, RFC
>>    8330 <https://datatracker.ietf.org/doc/rfc8330/> and RFC 8625
>>    <https://datatracker.ietf.org/doc/rfc8625/>.
>>
>> I'm not sure I entirely understand your comment. But let me explain why
>> we are gradually adding new flows:
>>
>> 1. TCP-implementations have usually a fixed limit for the upper bound of
>> the receive window. In some networks that upper bound is lower than the BDP
>> of the network. Thus, the only way to reach full capacity is by having
>> multiple flows.
>> 2. Having multiple connections allows to quicker achieve full capacity in
>> high-RTT networks and thus speeds up the test-duration.
>> 3. In some networks with "random" packet-loss, congestion-control may
>> come in the way of achieving full capacity. Again, multiple flows will work
>> around that.
>>
> GIM>> I might have asked several questions at once. Let me clarify what I
> am looking for:
>
>    - As I understand the method of creating the "working conditions in a
>    network" is based on certain assumptions. First, seems is that the
>    bandwidth is symmetrical between the measurement points. Second, that the
>    bandwidth doesn't change for the duration of the measurement session.
>    AFAIK, in the access networks, both are not necessarily always the case.
>
> We don't have the assumption that bandwidth is symmetrical (assuming, you
> mean uplink/downlink symmetry - please clarify otherwise).
>
> The load-generating algorithm runs independently for uplink and downlink
> traffic. And it is perfectly fine when both have huge asymmetry.
>
>
> Regarding the stability of the bandwidth:
> You are making a good point indeed that we assume that the bandwidth is to
> some extend stable while ramping up the flows to "working conditions".
> Admittedly that assumption does not always hold, and that is one of the
> reasons why we try hard for the test to not take too long.
> I'm not sure how we could adjust the algorithm for varying bandwidth
> without introducing too much complexity. I'm open for suggestions :-)
>
>
>    - On the other hand, I might have missed how the method of creating
>    the "working conditions" guarantees a symmetrical load between the
>    measurement points.
>
> As mentioned above, we don't assume a symmetrical load. Can you show us
> where in the draft we give that impression, so we can fix that?
>
>
>>
>>    - Then, I find the motivation not to use time units to express the
>>    responsiveness metric not convincing:
>>
>>    "Latency" is a poor measure of responsiveness, since it can be hard
>>    for the general public to understand.  The units are unfamiliar
>>    ("what is a millisecond?") and counterintuitive ("100 msec - that
>>    sounds good - it's only a tenth of a second!").
>>
>>
>> Can you expand on what exactly is not convincing to you? Do you think
>> that people will mis-understand the metric or that milli-seconds is the
>> right way to communicate responsiveness to the general public?
>>
> GIM>> Let me try. We know packet delay requirements for AR, VR
> applications. I believe that gamers are familiar with these numbers too.
> The same is likely the case for the industrial automation use cases served,
> for example, by Deterministic Networking.
>
>
> I can understand that for a technical audience, milli-seconds is easy and
> familiar. A non-technical audience might be more open to accepting a new
> "higher-is-better" metric. Responsiveness is something new and abstract so,
> it's kind of natural that it comes with a new unit.
>
> But I fully recognize that that's a controversial topic and can be
> discussed at length :)
>
>
> Cheers,
> Christoph
>
>
>>
>> Thanks a lot,
>> Christoph
>>
>> [1] And there are many networks that prioritize ICMP pings, thus we could
>> observe even more different results based on what protocol is used to
>> measure the latency.
>>
>>
>> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar <marcus.ihlar=
>> [email-masked]> wrote:
>>
>>> Hi IPPM,
>>>
>>>
>>>
>>> This email starts an adoption call for
>>> draft-cpaasch-ippm-responsiveness, "Responsiveness under Working
>>> Conditions”. This document specifies the “RPM Test” for measuring user
>>> experience when the network is fully loaded. The intended status of the
>>> document is Experimental.
>>>
>>>
>>>
>>> https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/
>>>
>>>
>>> https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01
>>>
>>>
>>>
>>> This adoption call will last until *Monday, December 20*. Please review
>>> the document, and reply to this email thread to indicate if you think IPPM
>>> should adopt this document.
>>>
>>>
>>>
>>> BR,
>>>
>>> Marcus
>>>
>>>
>>> _______________________________________________
>>> ippm mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/ippm
>>>
>> _______________________________________________
>> ippm mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/ippm
>>
>>
>>
>

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 15 Feb 2022 16:11:06 -0800

Hello Greg,

> On Feb 8, 2022, at 1:10 PM, Greg Mirsky <[email-masked]> wrote:
> 
> Hi Christoph,
> apologies for the belated response and thank you for sharing interesting details of you using the measurement method. I think that if the measurement method can not only provide the Round-trip Per Minute (RPM) metric but expose the network propagation and residential components of the round-trip delay, then it seems to me, the scope of the draft to be aligned with the charter of the IPPM WG and I'll be in favor of the WG adoption of the work.

I am assuming that with "residential components" you mean the server/client-side contribution to the measured latency, right?

In that case, yes the method does allow to separate these, as latency-probes are sent on both the load-generating connections and on separate connections. The difference between the two represents the "server-side contribution" to the latency.

I think what would be helpful would be a section in the draft that explains the different sources of latency (network, server, client) and how they affect the final RPM-number and how one can separate out these two components. It is also important to understand that the results are highly implementation-dependent. And explaining that in this section should help, I believe.

Would that be in line with what you are looking for?

Thanks,
Christoph

> 
> Regards,
> Greg
> 
> On Thu, Jan 6, 2022 at 4:42 PM Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
> Hello Greg,
> 
>> On Jan 6, 2022, at 3:00 PM, Greg Mirsky <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Christoph,
>> Thank you for your kind consideration of my notes and detailed responses. Please find my follow-up notes in-line below under the GIM>> tag.
> 
>> On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
>> Hello Greg,
>> 
>>> On Dec 22, 2021, at 11:43 PM, Greg Mirsky <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> as I understand it, the proposed new responsiveness metric is viewed as the single indicator of a bufferbloat condition in a network. As I recall, the discussion at Measuring Network Quality for End-Users workshop and on the mailing list <https://mailarchive.ietf.org/arch/browse/network-quality-workshop/?gbt=1&index=cuW_1lh4DD22V28EvlPFB_NjjZY> indicated, that there’s no consensus on what behaviors, symptoms can reliably signal the bufferbloat.
>> We are not trying for this responsiveness metric to be "the single indicator of bufferbloat". Bufferbloat can be measured in many different number of ways. And each of these will produce a correct, but a different result. Thus, "bufferbloat" is whatever the methodology tries to detect.
>> 
>> Let me give an example of two methodologies that are both correct but both will produce entirely different numbers :
>> 
>> If we would decide to generate the load by flooding the network with UDP traffic from a specific 4-tuple and measure latency with parallel ICMP pings. Then, on a over-buffered FIFO queue we would measure huge latencies (thus correctly expose bufferbloat), while on a FQ-codel queue we would not measure any bufferbloat.
>> 
>> If on the other hand, the load-generating traffic is changing the source-port for every single UDP-packet, then in both the FIFO-queue and the FQ-codel queue we will measure huge amounts of bufferbloat.
>> 
>> Thus, these two methods both produced correct results but with hugely different numbers in the FQ-codel case. [1]
>> 
>> Now, while both methods measure some variant of bufferbloat, they both don't measure a realistic usage of the network.
>> GIM>> Thank you for the insights. It seems to me that what the method can demonstrate is rather the level of efficiency of the AQM in the network for a particular class of applications.
> 
> Yes, that is a good description. It is for a "particular class of applications" and we are trying to make this class of applications representative of a "typical user-scenario". (admittedly, we can debate forever on what kind of applications are representative and I would love to have that debate :-)).
> 
> On the point of "efficiency of the AQM". I would go even further that it's not only AQM but also the client- and server-side implementations of these applications (as noted further below).
> 
> 
>> 
>> 
>> That is why the "Responsiveness under working conditions" tries to clearly specify how the load is generated and how the latency is being measured. And it does not measure "bufferbloat" but it measures "responsiveness under working conditions" based on the methodology that is being used (using HTTP/2 or HTTP/3, multiple flows, ...). It does expose bufferbloat which can happen in the network. It also exposes certain server-side behaviors that can cause (huge amounts of) additional latency - those behaviors are typically not called "bufferbloat".
>> GIM>> Thank you for pointing out that the result of the RTT measurement has two contributing factors - network and server.
> 
> Yes, servers contribute as do the client-side implementations. It's all three (client, network, server) that need to work "correctly" to achieve good responsiveness. Btw., as we are now gathering more experience with our methodology in different environments we find that the biggest portions of latency actually come from the server-side. We see several seconds of latency introduced by the HTTP/2 and TCP implementations.
> 
>> It seems worth enhancing the method to localize each contribution and measure them separately.
> 
> With the latency measuring probes being sent on load-bearing connections and separate connections and with the separate connections serving to measure DNS/TCP/... individually, the different data-points actually allow to localize to some extend.
> 
> However, I would be reluctant to dive too deep into localization/trouble-shooting/debugging of networks as part of this I-D. As this opens a whole new can of worms. We could then start thinking about sending latency-probes while playing with the IP TTL to find which router is introducing the latency,... It's an entirely different research-topic IMO :-) Dave Taht was thinking of starting something along these lines (https://github.com/dtaht/wtbb <https://github.com/dtaht/wtbb>).
> 
>>> It seems that it would be reasonable to first define what is being measured, characterized by the responsiveness metric. Having a document that discusses and defines the bufferbloat would be great.
>> I agree that there is a lack of definition for what "bufferbloat" really is.
>> 
>> The way we look at "responsiveness under working conditions" is that it measures the latency in conditions that may realistically happen in worst-case scenarios with end-users/implementations that are non-malicious (non-malicious to exclude the UDP-flooding scenario).
>> 
>> Thus, I assume we should make a better job at explaining this. The lack of a formal definition of "bufferbloat" doesn't help and thus we are indeed using this term a bit freely in the current draft. We will improve the Introduction to better set the stage (https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31 <https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31>).
>> 
>>> It seems like in the foundation of the methodology described in the draft lies the assumption that without adding new flows the available bandwidth is constant, does not change. While that is mostly the case, there are technologies that behave differently and may change bandwidth because of the outside conditions. Some of these behaviors of links with variable discrete bandwidth are discussed in, for example, RFC 8330 <https://datatracker.ietf.org/doc/rfc8330/> and RFC 8625 <https://datatracker.ietf.org/doc/rfc8625/>.
>> I'm not sure I entirely understand your comment. But let me explain why we are gradually adding new flows:
>> 
>> 1. TCP-implementations have usually a fixed limit for the upper bound of the receive window. In some networks that upper bound is lower than the BDP of the network. Thus, the only way to reach full capacity is by having multiple flows.
>> 2. Having multiple connections allows to quicker achieve full capacity in high-RTT networks and thus speeds up the test-duration.
>> 3. In some networks with "random" packet-loss, congestion-control may come in the way of achieving full capacity. Again, multiple flows will work around that.
>> GIM>> I might have asked several questions at once. Let me clarify what I am looking for:
>> As I understand the method of creating the "working conditions in a network" is based on certain assumptions. First, seems is that the bandwidth is symmetrical between the measurement points. Second, that the bandwidth doesn't change for the duration of the measurement session. AFAIK, in the access networks, both are not necessarily always the case.
> We don't have the assumption that bandwidth is symmetrical (assuming, you mean uplink/downlink symmetry - please clarify otherwise).
> 
> The load-generating algorithm runs independently for uplink and downlink traffic. And it is perfectly fine when both have huge asymmetry.
> 
> 
> You are making a good point indeed that we assume that the bandwidth is to some extend stable while ramping up the flows to "working conditions". Admittedly that assumption does not always hold, and that is one of the reasons why we try hard for the test to not take too long.
> I'm not sure how we could adjust the algorithm for varying bandwidth without introducing too much complexity. I'm open for suggestions :-)
>> On the other hand, I might have missed how the method of creating the "working conditions" guarantees a symmetrical load between the measurement points.
> As mentioned above, we don't assume a symmetrical load. Can you show us where in the draft we give that impression, so we can fix that?
> 
>>> 
>>> Then, I find the motivation not to use time units to express the responsiveness metric not convincing:
>> Can you expand on what exactly is not convincing to you? Do you think that people will mis-understand the metric or that milli-seconds is the right way to communicate responsiveness to the general public?
>> GIM>> Let me try. We know packet delay requirements for AR, VR applications. I believe that gamers are familiar with these numbers too. The same is likely the case for the industrial automation use cases served, for example, by Deterministic Networking.
> 
> I can understand that for a technical audience, milli-seconds is easy and familiar. A non-technical audience might be more open to accepting a new "higher-is-better" metric. Responsiveness is something new and abstract so, it's kind of natural that it comes with a new unit.
> 
> But I fully recognize that that's a controversial topic and can be discussed at length :)
> 
> 
> Cheers,
> Christoph
> 
>> 
>> 
>> Thanks a lot,
>> Christoph
>> 
>> [1] And there are many networks that prioritize ICMP pings, thus we could observe even more different results based on what protocol is used to measure the latency.
>> 
>>> 
>>> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar <marcus.ihlar=[email-masked] <mailto:[email-masked]>> wrote:
>>> Hi IPPM,
>>> 
>>>  
>>> 
>>> This email starts an adoption call for draft-cpaasch-ippm-responsiveness, "Responsiveness under Working Conditions”. This document specifies the “RPM Test” for measuring user experience when the network is fully loaded. The intended status of the document is Experimental.   
>>> 
>>>  
>>> 
>>> https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/ <https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/>
>>> https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01>
>>>  
>>> 
>>> This adoption call will last until Monday, December 20. Please review the document, and reply to this email thread to indicate if you think IPPM should adopt this document.
>>> 
>>>  
>>> 
>>> BR,
>>> 
>>> Marcus
>>> 
>>>  
>>> 
>>> [email-masked] <mailto:[email-masked]>
>>> https://www.ietf.org/mailman/listinfo/ippm <https://www.ietf.org/mailman/listinfo/ippm>
>>> [email-masked] <mailto:[email-masked]>
>>> https://www.ietf.org/mailman/listinfo/ippm <https://www.ietf.org/mailman/listinfo/ippm>
>> 
>

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 15 Feb 2022 16:33:58 -0800

Hi Christoph,
thank you for putting your thought into my comments. Your understanding is
absolutely correct and the new section, as you've outlined it, would make
the document even more useful to a reader. With that plan in place, I
support the adoption of the draft and would help with review and comments.

Regards,
Greg

On Tue, Feb 15, 2022 at 4:11 PM Christoph Paasch <[email-masked]> wrote:

> I am assuming that with "residential components" you mean the
> server/client-side contribution to the measured latency, right?
>
> In that case, yes the method does allow to separate these, as
> latency-probes are sent on both the load-generating connections and on
> separate connections. The difference between the two represents the
> "server-side contribution" to the latency.
>
> I think what would be helpful would be a section in the draft that
> explains the different sources of latency (network, server, client) and how
> they affect the final RPM-number and how one can separate out these two
> components. It is also important to understand that the results are highly
> implementation-dependent. And explaining that in this section should help,
> I believe.
>
>> On the point of "efficiency of the AQM". I would go even further that
>> it's not only AQM but also the client- and server-side implementations of
>> these applications (as noted further below).
>>
>>
>>
>>>
>>>    - It seems like in the foundation of the methodology described in
>>>    the draft lies the assumption that without adding new flows the
>>> 2. Having multiple connections allows to quicker achieve full capacity
>>> in high-RTT networks and thus speeds up the test-duration.
>> You are making a good point indeed that we assume that the bandwidth is
>> to some extend stable while ramping up the flows to "working conditions".
>>> [1] And there are many networks that prioritize ICMP pings, thus we
>>> could observe even more different results based on what protocol is used to
>>> measure the latency.
>>>
>>>
>>>> This adoption call will last until *Monday, December 20*. Please
>>>> review the document, and reply to this email thread to indicate if you
>>>> think IPPM should adopt this document.
>>>>
>>>>
>>>>
>>>> BR,
>>>>
>>>> Marcus
>>>>
>>>>

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 15 Feb 2022 16:42:34 -0800

Thanks a lot, Greg! I filed https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/36 <https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/36> and will be working on an updated I-D for the coming weeks.

Christoph

> On Feb 15, 2022, at 4:33 PM, Greg Mirsky <[email-masked]> wrote:
> 
> Hi Christoph,
> thank you for putting your thought into my comments. Your understanding is absolutely correct and the new section, as you've outlined it, would make the document even more useful to a reader. With that plan in place, I support the adoption of the draft and would help with review and comments.
> 
> Regards,
> Greg
> 
> On Tue, Feb 15, 2022 at 4:11 PM Christoph Paasch <[email-masked] <mailto:[email-masked]>> wrote:
> Hello Greg,
> 
>> On Feb 8, 2022, at 1:10 PM, Greg Mirsky <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Christoph,

---

From: Dave Taht <[email-masked]>
Date: Tue, 15 Feb 2022 17:00:26 -0800

> In that case, yes the method does allow to separate these, as latency-probes are sent on both the load-generating connections and on separate connections. The difference between the two represents…

I remember vividly how "neat" it seemed that QUIC had adopted the idea
of an inband "ping" in version Q018 in 2012 (or so).

https://docs.google.com/document/d/1WJvyZflAO2pq77yOLbp9NsGjC1CHetAXV8I0fQe-B_U/edit
Q018: Added a PING frame

What I don't remember or understand at the moment is I'm under the
impression that's in the SSL layer as of http2.0? There a good ref?

>> On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <[email-masked]> wrote:
>>>
>>> Hello Greg,
>>>
>>> as I understand it, the proposed new responsiveness metric is viewed as the single indicator of a bufferbloat condition in a network. As I recall, the discussion at Measuring Network Quality for End-Users workshop and on the mailing list indicated, that there’s no consensus on what behaviors, symptoms can reliably signal the bufferbloat.
>>>
>> However, I would be reluctant to dive too deep into localization/trouble-shooting/debugging of networks as part of this I-D. As this opens a whole new can of worms. We could then start thinking about sending latency-probes while playing with the IP TTL to find which router is introducing the latency,... It's an entirely different research-topic IMO :-) Dave Taht was thinking of starting something along these lines (https://github.com/dtaht/wtbb).
>>
>>> Thus, I assume we should make a better job at explaining this. The lack of a formal definition of "bufferbloat" doesn't help and thus we are indeed using this term a bit freely in the current draft. We will improve the Introduction to better set the stage (https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31).
>>>
>>> It seems like in the foundation of the methodology described in the draft lies the assumption that without adding new flows the available bandwidth is constant, does not change. While that is mostly the case, there are technologies that behave differently and may change bandwidth because of the outside conditions. Some of these behaviors of links with variable discrete bandwidth are discussed in, for example, RFC 8330 and RFC 8625.
>>>
>>> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar <marcus.ihlar=[email-masked]> wrote:
>>>>
>>>> Hi IPPM,
>>>>
>>>>
>>>>

--
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

---

From: "MORTON JR., AL" <[email-masked]>
Date: Wed, 16 Feb 2022 16:07:54 +0000

Hi Christoph,

> I think what would be helpful would be a section in the draft that explains the different sources of latency (network, server, client) and how they affect the final RPM-number and how one can separ…

when you say, "...the results are highly implementation-dependent.", 
which implementations are you referring to:
 
Network path? Server? Client? "Working Load" conditions? 

All of the above?

Network implementation should influence the results, the others not so much (with a standardized metric, we should have equivalent results across implementations, at least that's what we said in RFC6576/BCP176). And it can be rather difficult to separate the (I count at least 4) results-influencing factors.

more stuff to consider,
Al

> -----Original Message-----
> From: ippm <[email-masked]> On Behalf Of Dave Taht
> Sent: Tuesday, February 15, 2022 8:00 PM
> To: Christoph Paasch <cpaasch=[email-masked]>
> Cc: Marcus Ihlar <marcus.ihlar=[email-masked]>; IETF IPPM WG
> <[email-masked]>
> Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness
> 
> details of you using the measurement method. I think that if the measurement
> method can not only provide the Round-trip Per Minute (RPM) metric but expose
> the network propagation and residential components of the round-trip delay,
> then it seems to me, the scope of the draft to be aligned with the charter of
> the IPPM WG and I'll be in favor of the WG adoption of the work.
> > I am assuming that with "residential components" you mean the server/client-
> side contribution to the measured latency, right?
> >
> > In that case, yes the method does allow to separate these, as latency-probes
> are sent on both the load-generating connections and on separate connections.
> The difference between the two represents the "server-side contribution" to
> the latency.
> 
> https://urldefense.com/v3/__https://docs.google.com/document/d/1WJvyZflAO2pq77
> yOLbp9NsGjC1CHetAXV8I0fQe-B_U/edit__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPeUiZuj5$
> Q018: Added a PING frame
> 
> > I think what would be helpful would be a section in the draft that explains
> the different sources of latency (network, server, client) and how they affect
> the final RPM-number and how one can separate out these two components. It is
> also important to understand that the results are highly implementation-
> dependent. And explaining that in this section should help, I believe.
> >
> >>> as I understand it, the proposed new responsiveness metric is viewed as
> the single indicator of a bufferbloat condition in a network. As I recall, the
> discussion at Measuring Network Quality for End-Users workshop and on the
> mailing list indicated, that there’s no consensus on what behaviors, symptoms
> can reliably signal the bufferbloat.
> >>>
> >>> Let me give an example of two methodologies that are both correct but both
> will produce entirely different numbers :
> >>>
> traffic from a specific 4-tuple and measure latency with parallel ICMP pings.
> Then, on a over-buffered FIFO queue we would measure huge latencies (thus
> correctly expose bufferbloat), while on a FQ-codel queue we would not measure
> any bufferbloat.
> >>>
> >>> If on the other hand, the load-generating traffic is changing the source-
> port for every single UDP-packet, then in both the FIFO-queue and the FQ-codel
> queue we will measure huge amounts of bufferbloat.
> >>>
> demonstrate is rather the level of efficiency of the AQM in the network for a
> particular class of applications.
> >>
> >>
> >>> That is why the "Responsiveness under working conditions" tries to clearly
> specify how the load is generated and how the latency is being measured. And
> it does not measure "bufferbloat" but it measures "responsiveness under
> working conditions" based on the methodology that is being used (using HTTP/2
> or HTTP/3, multiple flows, ...). It does expose bufferbloat which can happen
> in the network. It also exposes certain server-side behaviors that can cause
> (huge amounts of) additional latency - those behaviors are typically not
> called "bufferbloat".
> >>
> >> GIM>> Thank you for pointing out that the result of the RTT measurement has
> two contributing factors - network and server.
> >>
> >>
> three (client, network, server) that need to work "correctly" to achieve good
> responsiveness. Btw., as we are now gathering more experience with our
> latency actually come from the server-side. We see several seconds of latency
> introduced by the HTTP/2 and TCP implementations.
> >>
> and separate connections and with the separate connections serving to measure
> DNS/TCP/... individually, the different data-points actually allow to localize
> to some extend.
> >>
> >> However, I would be reluctant to dive too deep into localization/trouble-
> shooting/debugging of networks as part of this I-D. As this opens a whole new
> can of worms. We could then start thinking about sending latency-probes while
> playing with the IP TTL to find which router is introducing the latency,...
> It's an entirely different research-topic IMO :-) Dave Taht was thinking of
> starting something along these lines
> (https://urldefense.com/v3/__https://github.com/dtaht/wtbb__;!!BhdT!yrleDfru-
> 45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPRghwEjg$ ).
> >>
> >>> It seems that it would be reasonable to first define what is being
> measured, characterized by the responsiveness metric. Having a document that
> discusses and defines the bufferbloat would be great.
> >>>
> measures the latency in conditions that may realistically happen in worst-case
> scenarios with end-users/implementations that are non-malicious (non-malicious
> to exclude the UDP-flooding scenario).
> >>>
> >>> Thus, I assume we should make a better job at explaining this. The lack of
> a formal definition of "bufferbloat" doesn't help and thus we are indeed using
> this term a bit freely in the current draft. We will improve the Introduction
> to better set the stage
> (https://urldefense.com/v3/__https://github.com/network-quality/draft-cpaasch-
> ippm-responsiveness/issues/31__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPU0pYBOv$ ).
> >>>
> >>> It seems like in the foundation of the methodology described in the draft
> lies the assumption that without adding new flows the available bandwidth is
> constant, does not change. While that is mostly the case, there are
> technologies that behave differently and may change bandwidth because of the
> outside conditions. Some of these behaviors of links with variable discrete
> bandwidth are discussed in, for example, RFC 8330 and RFC 8625.
> >>>
> the receive window. In some networks that upper bound is lower than the BDP of
> the network. Thus, the only way to reach full capacity is by having multiple
> flows.
> >>> 3. In some networks with "random" packet-loss, congestion-control may come
> in the way of achieving full capacity. Again, multiple flows will work around
> that.
> >>
> >> As I understand the method of creating the "working conditions in a
> network" is based on certain assumptions. First, seems is that the bandwidth
> is symmetrical between the measurement points. Second, that the bandwidth
> doesn't change for the duration of the measurement session. AFAIK, in the
> access networks, both are not necessarily always the case.
> >>
> >> On the other hand, I might have missed how the method of creating the
> "working conditions" guarantees a symmetrical load between the measurement
> points.
> >>
> >>> Can you expand on what exactly is not convincing to you? Do you think that
> people will mis-understand the metric or that milli-seconds is the right way
> to communicate responsiveness to the general public?
> >>
> applications. I believe that gamers are familiar with these numbers too. The
> same is likely the case for the industrial automation use cases served, for
> example, by Deterministic Networking.
> >>
> >>
> observe even more different results based on what protocol is used to measure
> the latency.
> >>>
> >>>
> >>> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar
> <marcus.ihlar=[email-masked]> wrote:
> >>>>
> >>>> Hi IPPM,
> >>>>
> >>>>
> >>>>
> >>>> This email starts an adoption call for draft-cpaasch-ippm-responsiveness,
> "Responsiveness under Working Conditions”. This document specifies the “RPM
> Test” for measuring user experience when the network is fully loaded. The
> intended status of the document is Experimental.
> >>>>
> >>>>
> >>>>
> >>>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-
> cpaasch-ippm-responsiveness/__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPU3zpKI3$
> >>>>
> >>>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-
> cpaasch-ippm-responsiveness-01__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPf9N-E8h$
> >>>>
> >>>>
> >>>>
> >>>> This adoption call will last until Monday, December 20. Please review the
> document, and reply to this email thread to indicate if you think IPPM should
> adopt this document.
> >>>>
> >>>>
> >>>>
> >>>> BR,
> >>>>
> >>>> Marcus
> >>>>
> >>>>
> >>>>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$
> >>>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$
> >>>
> >>>
> >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$
> 
> 
> 
> --
> https://urldefense.com/v3/__https://wayforward.archive.org/?site=https*3A*2F*2
> Fwww.icei.org__;JSUl!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPXLNE64R$
> 
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 17 Feb 2022 16:23:58 -0800

Hello Al,

> On Feb 16, 2022, at 8:07 AM, MORTON JR., AL <[email-masked]> wrote:
> 
> Hi Christoph,
> 
> you wrote:

Network path, Server and Client. Because the layer at which things are being measured is right above the HTTP layer. Thus, implementation-details all the way from the PHY up to HTTP matter and can influence the results.

> Network implementation should influence the results, the others not so much (with a standardized metric, we should have equivalent results across implementations, at least that's what we said in RF…

I think it is impossible to only have network implementation influence the results. For example, if UDP is used to measure latency the entire UDP-stack, IP-stack,... of the sender and receiver will influence the results.

In some way, sender and receive UDP/IP/Driver/... stack is part of the "network implementation" here.

For the responsiveness it is exactly the same. We use HTTP/2, and thus this is part of the "network implementation".

Does that make sense?

Thanks,
Christoph

>> From: ippm <[email-masked] <mailto:[email-masked]>> On Behalf Of Dave Taht
>> To: Christoph Paasch <cpaasch=[email-masked] <mailto:cpaasch=[email-masked]>>
>> Cc: Marcus Ihlar <marcus.ihlar=[email-masked] <mailto:marcus.ihlar=[email-masked]>>; IETF IPPM WG
>> <[email-masked] <mailto:[email-masked]>>
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1WJvyZflAO2pq77 <https://urldefense.com/v3/__https://docs.google.com/document/d/1WJvyZflAO2pq77>
>> (https://urldefense.com/v3/__https://github.com/dtaht/wtbb__;!!BhdT!yrleDfru- <https://urldefense.com/v3/__https://github.com/dtaht/wtbb__;!!BhdT!yrleDfru->
>> (https://urldefense.com/v3/__https://github.com/network-quality/draft-cpaasch- <https://urldefense.com/v3/__https://github.com/network-quality/draft-cpaasch->
>> <marcus.ihlar=[email-masked] <mailto:marcus.ihlar=[email-masked]>> wrote:
>>>>>> 
>>>>>> Hi IPPM,
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft- <https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft->
>>>>>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft- <https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft->
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
>> https://urldefense.com/v3/__https://wayforward.archive.org/?site=https*3A*2F*2 <https://urldefense.com/v3/__https://wayforward.archive.org/?site=https*3A*2F*2>
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd>

---

From: "MORTON JR., AL" <[email-masked]>
Date: Sun, 20 Feb 2022 17:37:04 +0000

Hi Christoph,
Thanks for continuing this discussion. Allow me to reply and add my view below,
Al

From: Christoph Paasch <[email-masked]>
Sent: Thursday, February 17, 2022 7:24 PM
To: MORTON JR., AL <[email-masked]>
Cc: Dave Taht <[email-masked]>; Marcus Ihlar <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

Hello Al,

On Feb 16, 2022, at 8:07 AM, MORTON JR., AL <[email-masked]<mailto:[email-masked]>> wrote:

Hi Christoph,

you wrote:

All of the above?

I think it is impossible to only have network implementation influence the results. For example, if UDP is used to measure latency the entire UDP-stack, IP-stack,... of the sender and receiver will influence the results.
[acm]
Agreed, although the one-to-one correspondence of packets on the wire and UDP datagrams is an advantage with UDP measurements. Many measurement systems use UDP today, and it has been straightforward to calibrate the delay introduced by the host stack and remove that component when necessary (within a tolerance of variation appropriate for network measurements, it may be nearly deterministic).

In some way, sender and receive UDP/IP/Driver/... stack is part of the "network implementation" here.
[acm]
We have performed the calibrations of the stack so that we can continue to use the notion of “wire-time” (that separate stack from network).

For the responsiveness it is exactly the same. We use HTTP/2, and thus this is part of the "network implementation".
[acm]
I would like to see more on this, because:
I haven’t measured or attempted to calibrate delays this high in the stack myself. But I suspect there may be less determinism and more host processing time/variation compared to UDP delay measurements. This is one place that new forms of implementation dependence might creep-in, in addition to the aspect of Working-Load generation (another possible implementation-dependency mentioned above).

Does that make sense?

Thanks,
Christoph

more stuff to consider,
Al

-----Original Message-----
From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Dave Taht
Sent: Tuesday, February 15, 2022 8:00 PM
To: Christoph Paasch <cpaasch=[email-masked]<mailto:cpaasch=[email-masked]>>
Cc: Marcus Ihlar <marcus.ihlar=[email-masked]<mailto:marcus.ihlar=[email-masked]>>; IETF IPPM WG
<[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

On Tue, Feb 15, 2022 at 4:11 PM Christoph Paasch
<cpaasch=[email-masked]<mailto:cpaasch=[email-masked]>> wrote:

Hello Greg,

On Feb 8, 2022, at 1:10 PM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hi Christoph,

https://urldefense.com/v3/__https://docs.google.com/document/d/1WJvyZflAO2pq77<https://urldefense.com/v3/__https:/docs.google.com/document/d/1WJvyZflAO2pq77>
yOLbp9NsGjC1CHetAXV8I0fQe-B_U/edit__;!!BhdT!yrleDfru-45Lar4Qe5-
4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPeUiZuj5$
Q018: Added a PING frame

Thanks,
Christoph

Regards,
Greg

On Thu, Jan 6, 2022 at 4:42 PM Christoph Paasch <[email-masked]<mailto:[email-masked]>> wrote:

Hello Greg,

On Jan 6, 2022, at 3:00 PM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hi Christoph,

On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <[email-masked]<mailto:[email-masked]>> wrote:

Hello Greg,

On Dec 22, 2021, at 11:43 PM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

(https://urldefense.com/v3/__https://github.com/dtaht/wtbb__;!!BhdT!yrleDfru-<https://urldefense.com/v3/__https:/github.com/dtaht/wtbb__;!!BhdT!yrleDfru->

(https://urldefense.com/v3/__https://github.com/network-quality/draft-cpaasch-<https://urldefense.com/v3/__https:/github.com/network-quality/draft-cpaasch->

I'm not sure I entirely understand your comment. But let me explain why we
are gradually adding new flows:

Then, I find the motivation not to use time units to express the
responsiveness metric not convincing:

Cheers,
Christoph

Thanks a lot,
Christoph

On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar
<marcus.ihlar=[email-masked]<mailto:marcus.ihlar=[email-masked]>> wrote:

Hi IPPM,

https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft->
cpaasch-ippm-responsiveness/__;!!BhdT!yrleDfru-45Lar4Qe5-
4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPU3zpKI3$

https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft->
cpaasch-ippm-responsiveness-01__;!!BhdT!yrleDfru-45Lar4Qe5-
4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPf9N-E8h$

BR,

Marcus

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$

--
I tried to build a better future, a few times:
https://urldefense.com/v3/__https://wayforward.archive.org/?site=https*3A*2F*2<https://urldefense.com/v3/__https:/wayforward.archive.org/?site=https*3A*2F*2>
Fwww.icei.org__;JSUl!!BhdT!yrleDfru-45Lar4Qe5-
4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPXLNE64R$

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 23 Feb 2022 11:43:56 -0800

Hello Al,

please see inline:

> On Feb 20, 2022, at 9:37 AM, MORTON JR., AL <[email-masked]> wrote:
> 
> Hi Christoph,
> From: Christoph Paasch <[email-masked] <mailto:[email-masked]>> 
> To: MORTON JR., AL <[email-masked] <mailto:[email-masked]>>
> Cc: Dave Taht <[email-masked] <mailto:[email-masked]>>; Marcus Ihlar <[email-masked] <mailto:[email-masked]>>; IETF IPPM WG <[email-masked] <mailto:[email-masked]>>
> On Feb 16, 2022, at 8:07 AM, MORTON JR., AL <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi Christoph,
> 
> you wrote:
> 

The methodology does allow to some extend factor out network-latency from end-host latency by sending probes on load-generating and separate connections. So, that should allow to calibrate things to some extend.

Now, with these discussion I am wondering about one thing:

Why would one want to calibrate out the end-host latency from the measurement? Sure, when trying to debug things to identify the sources of bufferbloat. But for debugging the method described in the draft is a very poor tool. When I'm debugging bufferbloat, I resort back to traceroute, tcptraceroute, ping, iperf, curl,...

So, I don't fully understand why one would want to calibrate the end-host latency away from the measurement. Because, at the end that is what the end-users are experiencing and so I want to measure it entirely end-to-end. Responsiveness really is a metric that is focused on the end-user and not the network in isolation.

Christoph

> <cpaasch=[email-masked] <mailto:cpaasch=[email-masked]>> wrote:
> 
> 
> Hello Greg,
> 
> https://urldefense.com/v3/__https://docs.google.com/document/d/1WJvyZflAO2pq77 <https://urldefense.com/v3/__https:/docs.google.com/document/d/1WJvyZflAO2pq77>
> (https://urldefense.com/v3/__https://github.com/dtaht/wtbb__;!!BhdT!yrleDfru- <https://urldefense.com/v3/__https:/github.com/dtaht/wtbb__;!!BhdT!yrleDfru->
> (https://urldefense.com/v3/__https://github.com/network-quality/draft-cpaasch- <https://urldefense.com/v3/__https:/github.com/network-quality/draft-cpaasch->
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft- <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft->
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft- <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft->
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>
> https://urldefense.com/v3/__https://wayforward.archive.org/?site=https*3A*2F*2 <https://urldefense.com/v3/__https:/wayforward.archive.org/?site=https*3A*2F*2>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!Bhd>

---

From: Marcus Ihlar <[email-masked]>
Date: Thu, 24 Feb 2022 15:07:21 +0000

Hi,

We've seen good discussion and good support for taking on this work. Thanks to everyone who has contributed.

Several concerns have been raised around how to separate  the various components that contribute to the latency metric.

Based on this we would declare that there is working group consensus for adopting this document and that the working group must focus on describing the various sources of latency involved in this metric and how those latency components can be separated.

This email ends the adoption call.

BR

Marcus

From: ippm <[email-masked]> On Behalf Of Marcus Ihlar
Sent: den 6 december 2021 16:53
To: [email-masked]
Subject: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

Hi IPPM,

This email starts an adoption call for draft-cpaasch-ippm-responsiveness, "Responsiveness under Working Conditions". This document specifies the "RPM Test" for measuring user experience when the network is fully loaded. The intended status of the document is Experimental.

BR,

Marcus

---

From: "MORTON JR., AL" <[email-masked]>
Date: Sun, 27 Feb 2022 19:58:45 +0000

Hi Christoph,
Thanks for your reply.
I gave your questions (at the end) some thought before replying.
I typed-up what I came-up-with so far in the reply below.
Al

Sent: Wednesday, February 23, 2022 2:44 PM

Hello Al,

please see inline:

On Feb 20, 2022, at 9:37 AM, MORTON JR., AL <[email-masked]<mailto:[email-masked]>> wrote:

Hi Christoph,

From: Christoph Paasch <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 17, 2022 7:24 PM
To: MORTON JR., AL <[email-masked]<mailto:[email-masked]>>
Cc: Dave Taht <[email-masked]<mailto:[email-masked]>>; Marcus Ihlar <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

Hello Al,

Hi Christoph,

you wrote:

All of the above?

Why would one want to calibrate out the end-host latency from the measurement? Sure, when trying to debug things to identify the sources of bufferbloat. But for debugging the method described in the draft is a very poor tool. When I'm debugging bufferbloat, I resort back to traceroute, tcptraceroute, ping, iperf, curl,...
[acm]
I think that you might want to characterize the Working-Load in terms of metrics on the network: send and receive rates, RT and one-way delay, delay variation, loss, etc.
I agree with you that the method in the draft is not good for this characterization.
I think it will help to supply the context for responsiveness measurements and recommend to make the network measurements separately, under Working-Load.

So, I don't fully understand why one would want to calibrate the end-host latency away from the measurement. Because, at the end that is what the end-users are experiencing and so I want to measure it entirely end-to-end. Responsiveness really is a metric that is focused on the end-user and not the network in isolation.
[acm]
We are discussing implementation dependencies, and that the user host and server host stack implementation will influence the responsiveness results. (A question in my mind is *how much* the stack implementation influences the results, and that’s an interesting side experiment.)
I think that the responsiveness measurement might be useful for users in a relative way:
+ Today, my responsiveness is higher than yesterday, and that’s not good.
+ When I switch from WiFi access to Cellular access, my responsiveness is about the same.
+ When I use different test servers with different network paths, I can see the results change.

The (imperfect) analogy I found so far is the “steps per day” metric. Many people use this metric. They can easily see their relative performance, and understand better or worse performance comparisons. We can communicate with others that we achieved, exceeded, or fell short of our daily goal (10k steps). But we can only begin to make scientific comparisons between different user measurements when we know the device making the measurement (smart watch or fit-bit and its HW/SW), the measured stride of each user, the walking conditions (stairs, slope or flat: analogy to the network in this factor), etc.  IOW, the steps-per-day metric becomes a more qualitative metric beyond the single user context (even though there is a quantitative result), due to measurement implementation, the path, and even the user herself.

Christoph

Does that make sense?

Thanks,
Christoph

more stuff to consider,
Al

-----Original Message-----
From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Dave Taht
Sent: Tuesday, February 15, 2022 8:00 PM
To: Christoph Paasch <cpaasch=[email-masked]<mailto:cpaasch=[email-masked]>>
Cc: Marcus Ihlar <marcus.ihlar=[email-masked]<mailto:marcus.ihlar=[email-masked]>>; IETF IPPM WG

Hello Greg,

Hi Christoph,

Thanks,
Christoph

Regards,
Greg

Hello Greg,

Hi Christoph,

Hello Greg,

Cheers,
Christoph

Thanks a lot,
Christoph

Hi IPPM,

BR,

Marcus