Subject: [ippm] Fwd: How should capacity measurement interact with shaping?

From: Matt Mathis <[email-masked]>
Date: Thu, 19 Sep 2019 14:17:58 -0700

Ok, moving the thread to IPPM

Some background, we (Measurement Lab) are testing a new transport (TCP)
performance measurement tool, based on BBR-TCP.   I'm not ready to talk
about results yet (well ok, it looks pretty good).    (BTW the BBR
algorithm just happens to resemble the algorithm described
in draft-morton-ippm-capcity-metric-method-00.)

Anyhow we noticed some interesting performance features for number of ISPs
in the US and Europe and I wanted to get some input for how these cases
should be treated.

One data point, a single trace saw ~94.5 Mbit/s for ~4 seconds,
fluctuating performance ~75 Mb/s for ~1 second and then stable
performance at ~83Mb/s for the rest of the 10 second test.    If I were to
guess this is probably a policer (shaper?) with a 1 MB token bucket and
a ~83Mb/s token rate (these numbers are not corrected for header overheads,
which actually matter with this tool).  What is weird about it is that
different ingress interfaces to the ISP (peers or serving locations)
exhibit different parameters.

Now the IPPM measurement question:   Is the bulk transport capacity of this
link ~94.5 Mbit/s or ~83Mb/s?   Justify your answer....?

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of
control;
            too weak risks being mistaken for tacit approval.

Forwarded Conversation
Subject: How should capacity measurement interact with shaping?
------------------------

From: Matt Mathis <[email-masked]>
Date: Thu, Aug 15, 2019 at 8:55 AM
To: MORTON, ALFRED C (AL) <[email-masked]>

We are seeing shapers  with huge bucket sizes, perhaps as larger or larger
than 100 MB.

These are prohibitive to test by default, but can have a huge impact in
some common situations.  E.g. downloading software updates.

An unconditional pass is not good, because some buckets are small.  What
counts as large enough to be ok, and what "derating" is ok?

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of
control;
            too weak risks being mistaken for tacit approval.

----------
From: MORTON, ALFRED C (AL) <[email-masked]>
Date: Mon, Aug 19, 2019 at 5:08 AM
To: Matt Mathis <[email-masked]>
Cc: CIAVATTONE, LEN <[email-masked]>, [email-masked] <
[email-masked]>

Hi Matt, currently cruising between Crete and Malta,

with about 7 days of vacation remaining – Adding my friend Len.

You know Rüdiger. It appears I’ve forgotten how to typs in 2 weeks

given the number of typos I’ve fixed so far...

We’ve seen big buffers on a basic DOCSIS cable service (downlink >2 sec)

but,

  we have 1-way delay variation or RTT variation limits

  when searching for the max rate, that don’t many packets

  queue in the buffer

  we want the status messages that result in rate adjustment to return

 in a reasonable amount of time (50ms + RTT)

  we usually search for 10 seconds, but if we go back and test with

  a fixed rate, we can see the buffer growing if the rate is too high.

  There will eventually be a discussion on the thresholds we use

  in the search // load rate control algorithm. The copy of

  Y.1540 I sent you has a simple one, we moved beyond that now

  (see the slides I didn’t get to present at IETF).

  There is value in having some of this discussion on IPPM-list,

  so we get some **agenda time at IETF-106**

We measure rate and performance, with some performance limits

built-in.  Pass/Fail is another step, de-rating too (made sense

with MBM “target_rate”).

Al

----------
From: <[email-masked]>
Date: Mon, Aug 26, 2019 at 12:05 AM
To: <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>

Hi Al,

thanks for keeping me involved. I don’t have a precise answer and doubt,
that there will be a single universal truth.

If the aim is only to determine the IP bandwidth of an access, then we
aren’t interested in filling a buffer. Buffering events may occur, some of
which are useful and to be expected, whereas others are not desired:

   - Sender shaping behavior may matter (is traffic at the source CBR or is
   it bursty)
   - Random collisions should be tolerated at the access whose bandwidth is
   to be measured.
   - Limiting packet drop due to buffer overflow is a design aim or an
   important part of the algorithm, I think.
   - Shared media might create bursts. I’m not an expert in the area, but
   there’s an “is bandwidth available” check in some cases between a central
   sender using a shared medium and the receivers connected. WiFi and may be
   other wireless equipment buffers packets also to optimize wireless resource
   optimization.
   - It might be an idea to mark some flows by ECN, once there’s a guess on
   a sending bitrate when to expect no or very little packet drop. Today, this
   is experimental. CE marks by an ECN capable device should be expected
   roughly once queuing starts.

Practically, the set-up should be configurable with commodity hard- and
software and all metrics should be measurable at the receiver. Burstiness
of traffic and a distinction between queuing events which are to be
expected and (undesired) queue build up are the events to be distinguished.
I hope that can be done with commodity hard- and software. I at least am
not able to write down a simple metric distinguishing queues to be expected
from (undesired) queue build up causing congestion. The hard- and software
to be used should be part of the solution, not part of the problem (bursty
source traffic and timestamps with insufficient accuracy to detect queues
are what I’d like to avoid).

I’d suggest to move discussion to the list.

Regards,

Rüdiger

----------
From: MORTON, ALFRED C (AL) <[email-masked]>
Date: Thu, Sep 19, 2019 at 7:01 AM
To: [email-masked] <[email-masked]>
Cc: CIAVATTONE, LEN <[email-masked]>, [email-masked] <
[email-masked]>

I’m catching-up with this thread again, but before I reply:

*** Any objection to moving this discussion to IPPM-list ?? ***

@Matt – this is a question to you at this point...

thanks,

Al

*From:* [email-masked] [mailto:[email-masked]]
*Sent:* Monday, August 26, 2019 3:05 AM
*To:* MORTON, ALFRED C (AL) <[email-masked]>
*Cc:* CIAVATTONE, LEN <[email-masked]>; [email-masked]
*Subject:* AW: How should capacity measurement interact with shaping?

Hi Al,

thanks for keeping me involved. I don’t have a precise answer and doubt,
that there will be a single universal truth.

If the aim is only to determine the IP bandwidth of an access, then we
aren’t interested in filling a buffer. Buffering events may occur, some of
which are useful and to be expected, whereas others are not desired:

-        Sender shaping behavior may matter (is traffic at the source CBR
or is it bursty)

-        Random collisions should be tolerated at the access whose
bandwidth is to be measured.

-        Limiting packet drop due to buffer overflow is a design aim or an
important part of the algorithm, I think.

-        Shared media might create bursts. I’m not an expert in the area,
but there’s an “is bandwidth available” check in some cases between a
central sender using a shared medium and the receivers connected. WiFi and
may be other wireless equipment buffers packets also to optimize wireless
resource optimization.

-        It might be an idea to mark some flows by ECN, once there’s a
guess on a sending bitrate when to expect no or very little packet drop.
Today, this is experimental. CE marks by an ECN capable device should be
expected roughly once queuing starts.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 19 Sep 2019 22:34:52 +0000

Thanks Matt!  This is an interesting trace to consider,
and an important discussion to share with the group.

When I look at the equation for BBR:
https://cacm.acm.org/magazines/2017/2/212428-bbr-congestion-based-congestion-control/fulltext

both BBR and Maximum IP-layer Capacity Metric seek the
Max over some time interval. The window seems smaller for
BBR: 6 to 10 RTTs, where we’ve been using parameters that
result in a rate measurement once a second and take the max
of the 10 one-second measurements.

We’ve also evaluate several performance metrics when
adjusting load, and that determines how high the sending
rate will go (based on feedback from the receiver).
https://tools.ietf.org/html/draft-morton-ippm-capcity-metric-method-00#section-4.3

So, it seems that the MAX delivered rate for the 10 second test,  we
can all see is 94.5 Mbps. This rate was sustained for more
than a trivial amount of time, too. But if you are concerned that this
rate was somehow inflated by a large buffer and a large
burst tolerance in the shaper – that’s where the additional
metrics and slightly different sending rate control
that we described in the draft (and the slides) might help.
https://datatracker.ietf.org/meeting/105/materials/slides-105-ippm-metrics-and-methods-for-ip-capacity-00

IOW, it might well be that Max IP Capacity, measured as we designed
and parameterized it, measures 83 Mbps for this path
(assuming the 94.5 is the result of big overshoot at sender, and the
fluctuating performance afterward seems to support that).

When I was looking for background on BBR, I saw a paper comparing
BBR and CUBIC during drive tests.
http://web.cs.wpi.edu/~claypool/papers/driving-bbr/
One pair of plots seemed to indicate that BBR sent lots of Bytes
early-on, and grew the RTT pretty high before settling down
(Figure 5, a & b).
This looks a bit like the case you described below,
except 94.5 Mbps is a Received Rate – we don’t know
what came out of the network, just what went in and filled
a buffer before crashing down in the drive test.

So, I think I did more investigation than justification
for my answers, but I conclude the parameters like the
individual measurement intervals and overall time interval
from which the Max is drawn, plus the rate control algorithm
itself, play a big role here.

regards,
Al

From: Matt Mathis [mailto:[email-masked]]
Sent: Thursday, September 19, 2019 5:18 PM
To: MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: Fwd: How should capacity measurement interact with shaping?

Some background, we (Measurement Lab) are testing a new transport (TCP) performance measurement tool, based on BBR-TCP.   I'm not ready to talk about results yet (well ok, it looks pretty good).    (BTW the BBR algorithm just happens to resemble the algorithm described in draft-morton-ippm-capcity-metric-method-00.)

Anyhow we noticed some interesting performance features for number of ISPs in the US and Europe and I wanted to get some input for how these cases should be treated.

One data point, a single trace saw ~94.5 Mbit/s for ~4 seconds, fluctuating performance ~75 Mb/s for ~1 second and then stable performance at ~83Mb/s for the rest of the 10 second test.    If I were to guess this is probably a policer (shaper?) with a 1 MB token bucket and a ~83Mb/s token rate (these numbers are not corrected for header overheads, which actually matter with this tool).  What is weird about it is that different ingress interfaces to the ISP (peers or serving locations) exhibit different parameters.

Now the IPPM measurement question:   Is the bulk transport capacity of this link ~94.5 Mbit/s or ~83Mb/s?   Justify your answer....?

Thanks,
--MM--

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of control;
            too weak risks being mistaken for tacit approval.

Forwarded Conversation

From: Matt Mathis <[email-masked]<mailto:[email-masked]>>
Date: Thu, Aug 15, 2019 at 8:55 AM
To: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>

We are seeing shapers  with huge bucket sizes, perhaps as larger or larger than 100 MB.

These are prohibitive to test by default, but can have a huge impact in some common situations.  E.g. downloading software updates.

An unconditional pass is not good, because some buckets are small.  What counts as large enough to be ok, and what "derating" is ok?

Thanks,
--MM--

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of control;
            too weak risks being mistaken for tacit approval.

----------
From: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>
Date: Mon, Aug 19, 2019 at 5:08 AM
To: Matt Mathis <[email-masked]<mailto:[email-masked]>>
Cc: CIAVATTONE, LEN <[email-masked]<mailto:[email-masked]>>, [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>

  There is value in having some of this discussion on IPPM-list,
  so we get some *agenda time at IETF-106*

Al

----------
From: <[email-masked]<mailto:[email-masked]>>
Date: Mon, Aug 26, 2019 at 12:05 AM
To: <[email-masked]<mailto:[email-masked]>>
Cc: <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>

Hi Al,

thanks for keeping me involved. I don’t have a precise answer and doubt, that there will be a single universal truth.

If the aim is only to determine the IP bandwidth of an access, then we aren’t interested in filling a buffer. Buffering events may occur, some of which are useful and to be expected, whereas others are not desired:

  *   Sender shaping behavior may matter (is traffic at the source CBR or is it bursty)
  *   Random collisions should be tolerated at the access whose bandwidth is to be measured.
  *   Limiting packet drop due to buffer overflow is a design aim or an important part of the algorithm, I think.
  *   Shared media might create bursts. I’m not an expert in the area, but there’s an “is bandwidth available” check in some cases between a central sender using a shared medium and the receivers connected. WiFi and may be other wireless equipment buffers packets also to optimize wireless resource optimization.
  *   It might be an idea to mark some flows by ECN, once there’s a guess on a sending bitrate when to expect no or very little packet drop. Today, this is experimental. CE marks by an ECN capable device should be expected roughly once queuing starts.

Practically, the set-up should be configurable with commodity hard- and software and all metrics should be measurable at the receiver. Burstiness of traffic and a distinction between queuing events which are to be expected and (undesired) queue build up are the events to be distinguished. I hope that can be done with commodity hard- and software. I at least am not able to write down a simple metric distinguishing queues to be expected from (undesired) queue build up causing congestion. The hard- and software to be used should be part of the solution, not part of the problem (bursty source traffic and timestamps with insufficient accuracy to detect queues are what I’d like to avoid).

Regards,

Rüdiger

----------
From: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>
Date: Thu, Sep 19, 2019 at 7:01 AM
To: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Cc: CIAVATTONE, LEN <[email-masked]<mailto:[email-masked]>>, [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>

thanks,
Al

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, August 26, 2019 3:05 AM
To: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>
Cc: CIAVATTONE, LEN <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: AW: How should capacity measurement interact with shaping?

Hi Al,

thanks for keeping me involved. I don’t have a precise answer and doubt, that there will be a single universal truth.

If the aim is only to determine the IP bandwidth of an access, then we aren’t interested in filling a buffer. Buffering events may occur, some of which are useful and to be expected, whereas others are not desired:

-        Sender shaping behavior may matter (is traffic at the source CBR or is it bursty)
-        Random collisions should be tolerated at the access whose bandwidth is to be measured.
-        Limiting packet drop due to buffer overflow is a design aim or an important part of the algorithm, I think.
-        Shared media might create bursts. I’m not an expert in the area, but there’s an “is bandwidth available” check in some cases between a central sender using a shared medium and the receivers connected. WiFi and may be other wireless equipment buffers packets also to optimize wireless resource optimization.
-        It might be an idea to mark some flows by ECN, once there’s a guess on a sending bitrate when to expect no or very little packet drop. Today, this is experimental. CE marks by an ECN capable device should be expected roughly once queuing starts.

---

From: Matt Mathis <[email-masked]>
Date: Thu, 19 Sep 2019 18:43:19 -0700

I am actually more interested in the philosophical questions about how this
should be reported, and what should the language be about non-stationary
available capacity.   One intersecting issue: BBR converges on both the
initial and final rate in under 2 seconds (this was a long path, so startup
took more than a second).   Do users want a quick (and relatively cheap)
test that takes 2 seconds or a longer test that is more likely to discover
the token bucket?  How long?  If we want to call them different names, what
should they be?

On the pure technical issues: BBR is still quite a moving target.   I have
a paper in draft that will shed some light on this.  It is due to be
unembargoed sometime in October.
BBRv1 (released slightly after the cacm paper you mention) measures the
max_BW every 8 RTT.  BBRv2 measures the max_BW on a sliding schedule that
loosely matches CUBIC.  (In both, min_RTT is measured every 10 seconds, in
the absence of organic low RTT samples).   BBRv2 uses additional signals
and does a much better job of avoiding overshoot at the startup.

In any case the best (most stable) BBR based metric seems to be
delta(snd.una)/elapsed_time, which is the progress as seen by upper
layers.  If you look at short time slices (we happen to be using 0.25
seconds) you see a mostly crisp square wave.  If you average from the
beginning of the connection to now, the peak rate happens at the moment the
bucket runs out of tokens, and falls towards the toke rate after that.

Thanks,
--MM--

On Thu, Sep 19, 2019 at 3:35 PM MORTON, ALFRED C (AL) <[email-masked]>
wrote:

> *From:* Matt Mathis [mailto:[email-masked]]
> *Sent:* Thursday, September 19, 2019 5:18 PM
> *To:* MORTON, ALFRED C (AL) <[email-masked]>;
> [email-masked]
> *Cc:* [email-masked]
> *Subject:* Fwd: How should capacity measurement interact with shaping?
>
>
>
> Now the IPPM measurement question:   Is the bulk transport capacity of
> this link ~94.5 Mbit/s or ~83Mb/s?   Justify your answer....?
>
>
>
> Thanks,
>
> --MM--
> *Forwarded Conversation*
> *Subject: How should capacity measurement interact with shaping?*
> ------------------------
>
>
>
> From: *Matt Mathis* <[email-masked]>
> From: *MORTON, ALFRED C (AL)* <[email-masked]>
>    - It might be an idea to mark some flows by ECN, once there’s a guess
>    on a sending bitrate when to expect no or very little packet drop. Today,
>    this is experimental. CE marks by an ECN capable device should be expected
> From: *MORTON, ALFRED C (AL)* <[email-masked]>
> -        Limiting packet drop due to buffer overflow is a design aim or
> an important part of the algorithm, I think.
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 21 Sep 2019 15:34:16 +0000

Hi Matt,

I had another thought about the 94 > 75 > 83 Mbps trace:
This might well be a service provider’s feature, where
they favor new flows with a high-rate for a short amount
of time.  I remember some ISPs offering a “speed boost”
to load web pages fast, but settling to a lower rate for
longer transfers. The same strategy could help reduce
the initial buffering time for video streams, perhaps with
different time intervals and rates. This might be implemented
by changing the token rate, or through some other means.

From: Matt Mathis [mailto:[email-masked]]
Sent: Thursday, September 19, 2019 9:43 PM
To: MORTON, ALFRED C (AL) <[email-masked]>
Cc: [email-masked]; [email-masked]; CIAVATTONE, LEN <[email-masked]>
Subject: Re: How should capacity measurement interact with shaping?

I am actually more interested in the philosophical questions about how this should be reported, and what should the language be about non-stationary available capacity.   One intersecting issue: BBR converges on both the initial and final rate in under 2 seconds (this was a long path, so startup took more than a second).   Do users want a quick (and relatively cheap) test that takes 2 seconds or a longer test that is more likely to discover the token bucket?  How long?  If we want to call them different names, what should they be?
[acm]
So, if your trace is revealing a bimodal form of service rates,
then it ought to be characterized with that in mind and
allow for two modes of operation when reporting:
94 initial peak Capacity, 83 sustained Capacity
*when this behavior is demonstrated and repeatable”.

Thanks for more insights about BBR, too.
Al

On the pure technical issues: BBR is still quite a moving target.   I have a paper in draft that will shed some light on this.  It is due to be unembargoed sometime in October.
BBRv1 (released slightly after the cacm paper you mention) measures the max_BW every 8 RTT.  BBRv2 measures the max_BW on a sliding schedule that loosely matches CUBIC.  (In both, min_RTT is measured every 10 seconds, in the absence of organic low RTT samples).   BBRv2 uses additional signals and does a much better job of avoiding overshoot at the startup.

In any case the best (most stable) BBR based metric seems to be delta(snd.una)/elapsed_time, which is the progress as seen by upper layers.  If you look at short time slices (we happen to be using 0.25 seconds) you see a mostly crisp square wave.  If you average from the beginning of the connection to now, the peak rate happens at the moment the bucket runs out of tokens, and falls towards the toke rate after that.

Thanks,
--MM--

On Thu, Sep 19, 2019 at 3:35 PM MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>> wrote:
Thanks Matt!  This is an interesting trace to consider,
and an important discussion to share with the group.

When I look at the equation for BBR:
https://cacm.acm.org/magazines/2017/2/212428-bbr-congestion-based-congestion-control/fulltext<https://urldefense.proofpoint.com/v2/url?u=https-3A__cacm.acm.org_magazines_2017_2_212428-2Dbbr-2Dcongestion-2Dbased-2Dcongestion-2Dcontrol_fulltext&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=r9ZlRUrf3rt7ZNsIpoSXARp3_ljQn3CHbrDZ7zMojow&e=>

We’ve also evaluate several performance metrics when
adjusting load, and that determines how high the sending
rate will go (based on feedback from the receiver).
https://tools.ietf.org/html/draft-morton-ippm-capcity-metric-method-00#section-4.3<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dmorton-2Dippm-2Dcapcity-2Dmetric-2Dmethod-2D00-23section-2D4.3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=CJxf89ZTabkMAiPj-b3_gyTKJfK2Ob3z9z1_Gwvzq-Q&e=>

https://datatracker.ietf.org/meeting/105/materials/slides-105-ippm-metrics-and-methods-for-ip-capacity-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_meeting_105_materials_slides-2D105-2Dippm-2Dmetrics-2Dand-2Dmethods-2Dfor-2Dip-2Dcapacity-2D00&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=8Fc9gs6FeJKe_bVYX3ROnlZNUfJxvV2MW8qvgNAWShU&e=>

http://web.cs.wpi.edu/~claypool/papers/driving-bbr/<https://urldefense.proofpoint.com/v2/url?u=http-3A__web.cs.wpi.edu_-7Eclaypool_papers_driving-2Dbbr_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=eOYze9z2Nr5bSmDtcvs7xwy8Rn7IFzUnigeTS9ul0W8&e=>

regards,
Al

From: Matt Mathis [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, September 19, 2019 5:18 PM
To: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Fwd: How should capacity measurement interact with shaping?

Thanks,
--MM--

Forwarded Conversation

Thanks,
--MM--

----------

Al

----------

Hi Al,

Regards,

Rüdiger

----------

thanks,
Al

Hi Al,

---

From: Matt Mathis <[email-masked]>
Date: Sat, 21 Sep 2019 11:32:58 -0700

Yes, exactly, I am sure this is a provider's feature.   I have a receiver
side pcap, and it is quite a bit more complicated than I thought:
- Zero losses in the entire trace.  It is dynamically shaped at a
bottleneck with a long queue that is pacing packets.
- The initial part is really straight (looks like a hard limit)
- The rate (and packet headway) smoothly wanders irregularly all over the
place in the latter part of the trace, from a low of about 1 Mb/s to peaks
close to the max rate.   My earlier data was from BBR max_rate, so the
fluctuating rate apparently has stable peaks.

By smooth: it looked like a spline fit, suggesting the instantaneous packet
headway was determined by a differential equation....

This behavior is not an accident, but the result of a very sophisticated
controller.   And we have seen other bottlenecks like it elsewhere in the
US and Europe.

As people may know, I do believe that shaping is the most appropriate way
to deal with heavy hitters.  And that we (IPPM) really need a way to
characterize shaping.  I do care about the asymptotic rate, and how quickly
I fall out of the initial rate.

Thanks,
--MM--

On Sat, Sep 21, 2019 at 8:34 AM MORTON, ALFRED C (AL) <[email-masked]>
wrote:

> *Sent:* Thursday, September 19, 2019 9:43 PM
> *Cc:* [email-masked]; [email-masked]; CIAVATTONE, LEN <
> [email-masked]>
> *Subject:* Re: How should capacity measurement interact with shaping?
>
>
>
> I am actually more interested in the philosophical questions about how
> this should be reported, and what should the language be about
> non-stationary available capacity.   One intersecting issue: BBR converges
> on both the initial and final rate in under 2 seconds (this was a long
> path, so startup took more than a second).   Do users want a quick (and
> relatively cheap) test that takes 2 seconds or a longer test that is more
> likely to discover the token bucket?  How long?  If we want to call them
> different names, what should they be?
>
> *[acm] *
>
> On Thu, Sep 19, 2019 at 3:35 PM MORTON, ALFRED C (AL) <
> [email-masked]> wrote:
>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cacm.acm.org_magazines_2017_2_212428-2Dbbr-2Dcongestion-2Dbased-2Dcongestion-2Dcontrol_fulltext&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=r9ZlRUrf3rt7ZNsIpoSXARp3_ljQn3CHbrDZ7zMojow&e=>
>
>
>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dmorton-2Dippm-2Dcapcity-2Dmetric-2Dmethod-2D00-23section-2D4.3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=CJxf89ZTabkMAiPj-b3_gyTKJfK2Ob3z9z1_Gwvzq-Q&e=>
>
>
>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_meeting_105_materials_slides-2D105-2Dippm-2Dmetrics-2Dand-2Dmethods-2Dfor-2Dip-2Dcapacity-2D00&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=8Fc9gs6FeJKe_bVYX3ROnlZNUfJxvV2MW8qvgNAWShU&e=>
>
>
>
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__web.cs.wpi.edu_-7Eclaypool_papers_driving-2Dbbr_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=hc7clxrgBJnSNsskiiOadU_kTq28Yt__Nwwdg4C9zms&s=eOYze9z2Nr5bSmDtcvs7xwy8Rn7IFzUnigeTS9ul0W8&e=>
>

---

From: <[email-masked]>
Date: Mon, 23 Sep 2019 08:19:52 +0000

Hi Matt,

Thanks. My expertise is more with fixed network access. I don’t know how access bandwidth is shared between LTE flows at a mobile access.

On the fixed network side, I’d expect a policy control point to be connected to some access line concentrator (be it optical, or copper based – coax I can’t tell). To my understanding the configurations having an impact at the fixed access are:

  *   The access shaper rate (settable)
  *   The access shaper burst size (likely a default value, depending on phys. line card bandwidth)
  *   The queue depth (settable)
  *   AQM parameters (settable to some extent, if deployed)

A burst which is below or equal to shaper burst size with a temporal pacing allowing to the shaper to recover for new tokens will likely see line speed at that bottleneck. That should hold in general.

The shaper kicks in, if more than default shaper burst size packets arrive within the given time frame. Then the queue-limit has an impact, if more than access shaper rate packets arrive. If the latency added by the building queue starts to control the measurement flow rate fast enough, loss should be avoided. But then you should be able to measure the shaper rate, I’d expect.

While I doubt that this fits to what you’ve observed, it may still help to design a test which either measures available bandwidth or access bandwidth in the presence of a shaper:

  *   An initial burst may pass a shaper at line speed
  *   A measurement sending up to shaper rate CBR spaced packets which consists of more than “default shaper burst size” packets  should see no added delay and be forwared by shaper rate (it would of course pass with up to “default shaper burst size” packets at that rate too, but then the rate measurement isn’t sustainable, if enough tokens are added between separate measurement streams) .
  *   If the CBR spaced measurement flow sends more than default shaper burst size packets with more than shaper rate, then a queue should build at the shaper.

You mentioned a token bucket of 1 MB. I’d expect the shaper default burst size to depend on the phys. line card bandwidth of the interface, where this shaper is configured. I’m not sure, whether to expect a single constant value for different implementations. I’d like to describe the default shaper bandwidth by a strawman “1ms@phys-interface-bandwidth” token bucket. Interfaces may be 1GE, 10GE or 100GE. I’d however not be surprised, if different generations of linecards from the same vendor support different values for the same physical port bandwidth, and different vendors may have different design philosophies.

Regards,

Ruediger

Von: Matt Mathis <[email-masked]>
Gesendet: Donnerstag, 19. September 2019 23:18
An: ALFRED C (AL) <[email-masked]>; Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Fwd: How should capacity measurement interact with shaping?

Thanks,
--MM--

Forwarded Conversation

Thanks,
--MM--

----------

Al

----------

Hi Al,

Regards,

Rüdiger

----------

thanks,
Al

Hi Al,

---

From: Matt Mathis <[email-masked]>
Date: Mon, 23 Sep 2019 11:36:05 -0700

This was actually DOCSIS.  I am guessing that the shaping is done at the
CMTS, based on load signals from elsewhere in the network, because the
behaviors vary depending on where the sender is located.   This has all
sorts of interesting implications from a measurement perspective.

For one thing, it is potentially "below the horizon" for many current
techniques.

Thanks,
--MM--

On Mon, Sep 23, 2019 at 1:19 AM <[email-masked]> wrote:

> Hi Matt,
>
>
>
> Thanks. My expertise is more with fixed network access. I don’t know how
> access bandwidth is shared between LTE flows at a mobile access.
>
>
>
> On the fixed network side, I’d expect a policy control point to be
> connected to some access line concentrator (be it optical, or copper based
> – coax I can’t tell). To my understanding the configurations having an
> impact at the fixed access are:
>
>    - The access shaper rate (settable)
>    - The access shaper burst size (likely a default value, depending on
>    phys. line card bandwidth)
>    - The queue depth (settable)
>    - AQM parameters (settable to some extent, if deployed)
>
>
>
> A burst which is below or equal to shaper burst size with a temporal
> pacing allowing to the shaper to recover for new tokens will likely see
> line speed at that bottleneck. That should hold in general.
>
>
>
> The shaper kicks in, if more than default shaper burst size packets arrive
> within the given time frame. Then the queue-limit has an impact, if more
> than access shaper rate packets arrive. If the latency added by the
> building queue starts to control the measurement flow rate fast enough,
> loss should be avoided. But then you should be able to measure the shaper
> rate, I’d expect.
>
>
>
> While I doubt that this fits to what you’ve observed, it may still help to
> design a test which either measures available bandwidth or access bandwidth
> in the presence of a shaper:
>
>    - An initial burst may pass a shaper at line speed
>    - A measurement sending up to shaper rate CBR spaced packets which
>    consists of more than “default shaper burst size” packets  should see no
>    added delay and be forwared by shaper rate (it would of course pass with up
>    to “default shaper burst size” packets at that rate too, but then the rate
>    measurement isn’t sustainable, if enough tokens are added between separate
>    measurement streams) .
>    - If the CBR spaced measurement flow sends more than default shaper
>    burst size packets with more than shaper rate, then a queue should build at
>    the shaper.
>
>
>
> You mentioned a token bucket of 1 MB. I’d expect the shaper default burst
> size to depend on the phys. line card bandwidth of the interface, where
> this shaper is configured. I’m not sure, whether to expect a single
> constant value for different implementations. I’d like to describe the
> default shaper bandwidth by a strawman “1ms@phys-interface-bandwidth”
> token bucket. Interfaces may be 1GE, 10GE or 100GE. I’d however not be
> surprised, if different generations of linecards from the same vendor
> support different values for the same physical port bandwidth, and
> different vendors may have different design philosophies.
>
>
>
> Regards,
>
>
>
> Ruediger
>
>
>
>
>
>
>
> *Von:* Matt Mathis <[email-masked]>
> *Gesendet:* Donnerstag, 19. September 2019 23:18
> *An:* ALFRED C (AL) <[email-masked]>; Geib, Rüdiger <
> *Betreff:* Fwd: How should capacity measurement interact with shaping?
>
>
>

---

From: Joachim Fabini <[email-masked]>
Date: Tue, 24 Sep 2019 11:19:58 +0200

Matt,

access technology was the first question I wanted to ask yesterday, you
answered it in your last email.
Some more comments, questions, and observations:
- On-demand capacity allocation over time was/is common for mobile
cellular access technologies. Superposition of algorithms at various
layers (cellular access capacity management, buffering, users in the
cell and CCA) generates challenging traces.
- Some details on the measurement architecture could help: where did you
measure BTC: at the receiver, at the sender, at intermediate nodes? Time
granularity/resolution?
- Al made several good points in his replies; the overshooting of
BBR in Fig.5a of
http://web.cs.wpi.edu/~claypool/papers/driving-bbr/paper-final.pdf is a
hint that *if* there are buffers on the path, BBR will attempt to use
them. It's the sequence cache-memory-network: first you fill up all
local buffers, then the ones at intermediate nodes on the path, and only
later on you can measure the network capacity. Still, 4 seconds for the
initial 94.5 Mbit/s burst means truly large buffers...

You asked for an answer and justification - here we go:
Boiling down to the essence, your question (which capacity to prefer,
94.5 or 83Mb/s) imo addresses a weakness of RFC3148. In particular a
more accurate definition and refinement of "long term" could help.
Quoting RFC 3148: "The intuitive definition of BTC is the expected *long
term* average data rate (bits per second) of a single ideal TCP
implementation over the path in question". Is *long term* 4 seconds?
10s? 1000s?

My interpretation is that you meant a "time interval sufficiently large
for CCA to converge and reach a steady state" (D1). However, this
definition considers E2E control/CCA *only* and falls short of the
pitfall that Al and I attempted to address in RFC 7312 for delay
measurements in mobile network measurements: today's (access) networks
are stateful (the copper wire abstraction fails). This stateful network
policing interacts with the end-to-end CCA and originates artificial,
non-representative and non-repeatable measurement results. In DOCSIS
(that I did not measure) it seems that buffering is the main issue.

Some more reasoning on your BTC measurements:
- In your measurements, when BTC is measured at the sender, the informal
definition (D1) seems to hold true for a flow that lasts for slightly
less than 4 seconds. But do BTC measurements at the receiver's end yield
the same results? They may or may not.
- If the flow duration exceeds 4 seconds, the convergence criterion (as
seen by the sender) is no longer fulfilled: conditions change.
- If I'm asked if 94.5Mb/s or 83Mb/s is "the" path BTC, my decision
depends on the receiver view: (a) if some buffers on the network path
fill up, but the data rate at the receiver is only 83Mb/s, then the
network path's BTC is 83Mb/s. If there's a shaper on the path that
grants an initial "data credit" to the sender at full speed and only
later on shapes, i.e, the receiver can measure 94.5Mb/s for the first 4
seconds, then I'd agree to a BTC of 94.5Mb/s. However, only for the
first 4 second interval or for the first y Mbytes of data.

My point is that (unfortunately) the measurement of BTC for a network
path must be tied to use cases to be meaningful. I'm truly convinced
that with today's networks it's an illusion to have one single,
representative value "BTC is x Mb/s". Decoupling BTC from the network
path's state and its evolution with time may not be feasible.

Reasoning on objective BTC:
First, the measurement of stateful network BTC mandate to take into
consideration parameters like (a) the network path implementation,
configuration and behavior, (b) the measurement point (sender, receiver,
intermediate point), and (c) the measurement flow characteristics
(duration, total amount of data? Any parameter that influences on the
network state wrt the flow?). These are just examples, it may make sense
to structure and categorize these influencing parameters.

Second, the definition (D1) in RFC3148 could be refined. The *long term*
should be made explicit, covering two mandatory components: (a) CCA
(end-to-end control) reaching a steady state and (b) the network path
under observation reaching a steady state over (c) a well-specified
interval.

In order to be meaningful (and scientifically sound), BTC as an
intuitive metric should imho reflect a network path's ability to offer a
sustained transfer capacity under (almost) constant conditions for a
specified time interval (and/or for a specified amount of data?). Still,
architecture and implementation of today's (access) networks make it
challenging to identify those intervals during which we have constant
conditions.

So the key question is if one can provide a BTC definition that is
accurate, representative and simple/usable for the broad mass. (*)
Al and his co-authors have included some key aspect in the
capacity-metric draft, some aspects of this thread may be added there,
too. But BTC imo is different and even more complex to measure.

regards Joachim
PS: (*) Leaving aside uncertainty that we will never be able to handle
in measurements because of the capacity-sharing in today's access
networks (users-in-a-cell, cell usage).

On 19/09/2019 23:17, Matt Mathis wrote:
> Anyhow we noticed some interesting performance features for number of
> ISPs in the US and Europe and I wanted to get some input for how these
> cases should be treated.
> 
> performance at ~83Mb/s for the rest of the 10 second test.    If I were
> to guess this is probably a policer (shaper?) with a 1 MB token bucket
> and a ~83Mb/s token rate (these numbers are not corrected for header
> overheads, which actually matter with this tool).  What is weird about
> it is that different ingress interfaces to the ISP (peers or
> serving locations) exhibit different parameters.  
> 
> From: *Matt Mathis* <[email-masked] <mailto:[email-masked]>>
> From: *MORTON, ALFRED C (AL)* <[email-masked]
> <mailto:[email-masked]>>
> To: Matt Mathis <[email-masked] <mailto:[email-masked]>>
> Cc: CIAVATTONE, LEN <[email-masked] <mailto:[email-masked]>>,
> [email-masked] <mailto:[email-masked]>
> <[email-masked] <mailto:[email-masked]>>
> 
> 
> Hi Matt, currently cruising between Crete and Malta,____
> 
> with about 7 days of vacation remaining – Adding my friend Len.____
> 
> You know Rüdiger. It appears I’ve forgotten how to typs in 2 weeks____
> 
> given the number of typos I’ve fixed so far...____
> 
> __ __
> 
> We’ve seen big buffers on a basic DOCSIS cable service (downlink >2 sec)____
> 
> but,____
> 
>   we have 1-way delay variation or RTT variation limits____
> 
>   when searching for the max rate, that don’t many packets ____
> 
>   queue in the buffer____
> 
> __ __
> 
>   we want the status messages that result in rate adjustment to return____
> 
>  in a reasonable amount of time (50ms + RTT)____
> 
> __ __
> 
>   we usually search for 10 seconds, but if we go back and test with ____
> 
>   a fixed rate, we can see the buffer growing if the rate is too high.____
> 
> __ __
> 
>   There will eventually be a discussion on the thresholds we use____
> 
>   in the search // load rate control algorithm. The copy of ____
> 
>   Y.1540 I sent you has a simple one, we moved beyond that now____
> 
>   (see the slides I didn’t get to present at IETF).____
> 
> __ __
> 
>   There is value in having some of this discussion on IPPM-list,____
> 
>   so we get some **agenda time at IETF-106** ____
> 
> __ __
> 
> We measure rate and performance, with some performance limits____
> 
> built-in.  Pass/Fail is another step, de-rating too (made sense ____
> 
> with MBM “target_rate”).____
> 
> __ __
> 
> Al____
> 
> 
> 
> ----------
> From: <[email-masked] <mailto:[email-masked]>>
> To: <[email-masked] <mailto:[email-masked]>>
> Cc: <[email-masked] <mailto:[email-masked]>>, <[email-masked]
> <mailto:[email-masked]>>
> 
> 
> Hi Al,____
> 
> __ __
> 
> that there will be a single universal truth. ____
> 
> __ __
> 
> aren’t interested in filling a buffer. Buffering events may occur, some
> of which are useful and to be expected, whereas others are not desired:____
> 
> __ __
> 
>   * Sender shaping behavior may matter (is traffic at the source CBR or
>     is it bursty)____
>   * Random collisions should be tolerated at the access whose bandwidth
>     is to be measured.____
>   * Limiting packet drop due to buffer overflow is a design aim or an
>     important part of the algorithm, I think.____
>   * Shared media might create bursts. I’m not an expert in the area, but
>     there’s an “is bandwidth available” check in some cases between a
>     central sender using a shared medium and the receivers connected.
>     WiFi and may be other wireless equipment buffers packets also to
>     optimize wireless resource optimization.____
>   * It might be an idea to mark some flows by ECN, once there’s a guess
>     on a sending bitrate when to expect no or very little packet drop.
>     Today, this is experimental. CE marks by an ECN capable device
>     should be expected roughly once queuing starts.____
> 
> __ __
> 
> software and all metrics should be measurable at the receiver.
> Burstiness of traffic and a distinction between queuing events which are
> to be expected and (undesired) queue build up are the events to be
> distinguished. I hope that can be done with commodity hard- and
> software. I at least am not able to write down a simple metric
> distinguishing queues to be expected from (undesired) queue build up
> causing congestion. The hard- and software to be used should be part of
> the solution, not part of the problem (bursty source traffic and
> timestamps with insufficient accuracy to detect queues are what I’d like
> to avoid).____
> 
> __ __
> 
> I’d suggest to move discussion to the list. ____
> 
> __ __
> 
> Regards,____
> 
> __ __
> 
> Rüdiger____
> 
> 
> 
> ----------
> From: *MORTON, ALFRED C (AL)* <[email-masked]
> <mailto:[email-masked]>>
> To: Ruediger.Geib@telekom..de <mailto:[email-masked]>
> <[email-masked] <mailto:[email-masked]>>
> Cc: CIAVATTONE, LEN <[email-masked] <mailto:[email-masked]>>,
> [email-masked] <mailto:[email-masked]>
> <[email-masked] <mailto:[email-masked]>>
> 
> 
> I’m catching-up with this thread again, but before I reply:____
> 
> __ __
> 
> *** Any objection to moving this discussion to IPPM-list ?? ***____
> 
> __ __
> 
> @Matt – this is a question to you at this point...____
> 
> __ __
> 
> thanks,____
> 
> Al____
> 
> __ __
> 
> *From:[email-masked] <mailto:[email-masked]>
> [mailto:[email-masked] <mailto:[email-masked]>]
> *To:* MORTON, ALFRED C (AL) <[email-masked]
> <mailto:[email-masked]>>
> *Cc:* CIAVATTONE, LEN <[email-masked] <mailto:[email-masked]>>;
> [email-masked] <mailto:[email-masked]>
> *Subject:* AW: How should capacity measurement interact with shaping?____
> 
> __ __
> 
> Hi Al,____
> 
> __ __
> 
> that there will be a single universal truth. ____
> 
> __ __
> 
> aren’t interested in filling a buffer. Buffering events may occur, some
> of which are useful and to be expected, whereas others are not desired:____
> 
> __ __
> 
> __-        __Sender shaping behavior may matter (is traffic at the
> source CBR or is it bursty)____
> 
> __-        __Random collisions should be tolerated at the access whose
> bandwidth is to be measured.____
> 
> __-        __Limiting packet drop due to buffer overflow is a design aim
> or an important part of the algorithm, I think.____
> 
> __-        __Shared media might create bursts. I’m not an expert in the
> area, but there’s an “is bandwidth available” check in some cases
> between a central sender using a shared medium and the receivers
> connected. WiFi and may be other wireless equipment buffers packets also
> to optimize wireless resource optimization.____
> 
> __-        __It might be an idea to mark some flows by ECN, once there’s
> a guess on a sending bitrate when to expect no or very little packet
> drop. Today, this is experimental. CE marks by an ECN capable device
> should be expected roughly once queuing starts.____
> 
> 
> 
> 
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>

---

From: <[email-masked]>
Date: Tue, 24 Sep 2019 10:14:17 +0000

Hi Joachim,

thanks for adding a discussion related to mobile access behavior. I've read some "streaming QoE over LTE" related publications. These are not BTC related, but LTE and CCA interaction also impacts the packet and flow related measurements performed. One conclusion of these publications is, that throughput fluctuations characterize an LTE access network better, than just average throughput values.

Your discussion whether to consider an access showing 94.5 Mbit/s throughput for a short time and 83 Mbit/s for a longer interval may indicate, that singleton, short interval BTC measurements used as samples to build a (or some) BTC statistics_to_be_defined might be useful approach.  

The above QoE experts use receiver statistics, as you suggest too, however their activities are passive, whereas BTC measurements discussed here are active. I add this to indicate, that a mobile access IPPM BTC metric discussion will have to produce own approaches.

I agree that defining "long term average data rate" in a way which is useful to an implementors use case is desirable (the same holds for guidance on a short term singleton BTC metric).

To start a discussion: a BTC singleton could consist of back-to-back packets of a length corresponding to an important or expected bottleneck burst tolerance (and/or buffer depth). These singletons should be spaced in a way that a loss and may be queue free long term BTC results. 

Regards,

Ruediger 

-----Ursprüngliche Nachricht-----
Von: Joachim Fabini <[email-masked]> 
Gesendet: Dienstag, 24. September 2019 11:20
An: Matt Mathis <mattmathis=[email-masked]>; ALFRED C (AL) <[email-masked]>; Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: [ippm] Fwd: How should capacity measurement interact with shaping?

Matt,

access technology was the first question I wanted to ask yesterday, you answered it in your last email.
Some more comments, questions, and observations:
- On-demand capacity allocation over time was/is common for mobile cellular access technologies. Superposition of algorithms at various layers (cellular access capacity management, buffering, users in the cell and CCA) generates challenging traces.
- Some details on the measurement architecture could help: where did you measure BTC: at the receiver, at the sender, at intermediate nodes? Time granularity/resolution?
- Al made several good points in his replies; the overshooting of BBR in Fig.5a of http://web.cs.wpi.edu/~claypool/papers/driving-bbr/paper-final.pdf is a hint that *if* there are buffers on the path, BBR will attempt to use them. It's the sequence cache-memory-network: first you fill up all local buffers, then the ones at intermediate nodes on the path, and only later on you can measure the network capacity. Still, 4 seconds for the initial 94.5 Mbit/s burst means truly large buffers...

You asked for an answer and justification - here we go:
Boiling down to the essence, your question (which capacity to prefer,
94.5 or 83Mb/s) imo addresses a weakness of RFC3148. In particular a more accurate definition and refinement of "long term" could help.
Quoting RFC 3148: "The intuitive definition of BTC is the expected *long
term* average data rate (bits per second) of a single ideal TCP implementation over the path in question". Is *long term* 4 seconds?
10s? 1000s?

My interpretation is that you meant a "time interval sufficiently large for CCA to converge and reach a steady state" (D1). However, this definition considers E2E control/CCA *only* and falls short of the pitfall that Al and I attempted to address in RFC 7312 for delay measurements in mobile network measurements: today's (access) networks are stateful (the copper wire abstraction fails). This stateful network policing interacts with the end-to-end CCA and originates artificial, non-representative and non-repeatable measurement results. In DOCSIS (that I did not measure) it seems that buffering is the main issue.

Some more reasoning on your BTC measurements:
- In your measurements, when BTC is measured at the sender, the informal definition (D1) seems to hold true for a flow that lasts for slightly less than 4 seconds. But do BTC measurements at the receiver's end yield the same results? They may or may not.
- If the flow duration exceeds 4 seconds, the convergence criterion (as seen by the sender) is no longer fulfilled: conditions change.
- If I'm asked if 94.5Mb/s or 83Mb/s is "the" path BTC, my decision depends on the receiver view: (a) if some buffers on the network path fill up, but the data rate at the receiver is only 83Mb/s, then the network path's BTC is 83Mb/s. If there's a shaper on the path that grants an initial "data credit" to the sender at full speed and only later on shapes, i.e, the receiver can measure 94.5Mb/s for the first 4 seconds, then I'd agree to a BTC of 94.5Mb/s. However, only for the first 4 second interval or for the first y Mbytes of data.

My point is that (unfortunately) the measurement of BTC for a network path must be tied to use cases to be meaningful. I'm truly convinced that with today's networks it's an illusion to have one single, representative value "BTC is x Mb/s". Decoupling BTC from the network path's state and its evolution with time may not be feasible.

Reasoning on objective BTC:
First, the measurement of stateful network BTC mandate to take into consideration parameters like (a) the network path implementation, configuration and behavior, (b) the measurement point (sender, receiver, intermediate point), and (c) the measurement flow characteristics (duration, total amount of data? Any parameter that influences on the network state wrt the flow?). These are just examples, it may make sense to structure and categorize these influencing parameters.

Second, the definition (D1) in RFC3148 could be refined. The *long term* should be made explicit, covering two mandatory components: (a) CCA (end-to-end control) reaching a steady state and (b) the network path under observation reaching a steady state over (c) a well-specified interval.

In order to be meaningful (and scientifically sound), BTC as an intuitive metric should imho reflect a network path's ability to offer a sustained transfer capacity under (almost) constant conditions for a specified time interval (and/or for a specified amount of data?). Still, architecture and implementation of today's (access) networks make it challenging to identify those intervals during which we have constant conditions.

So the key question is if one can provide a BTC definition that is accurate, representative and simple/usable for the broad mass. (*) Al and his co-authors have included some key aspect in the capacity-metric draft, some aspects of this thread may be added there, too. But BTC imo is different and even more complex to measure.

regards Joachim
PS: (*) Leaving aside uncertainty that we will never be able to handle in measurements because of the capacity-sharing in today's access networks (users-in-a-cell, cell usage).

On 19/09/2019 23:17, Matt Mathis wrote:
> Some background, we (Measurement Lab) are testing a new transport 
> (TCP) performance measurement tool, based on BBR-TCP.   I'm not ready 
> to talk about results yet (well ok, it looks pretty good).    (BTW the 
> BBR algorithm just happens to resemble the algorithm described in 
> draft-morton-ippm-capcity-metric-method-00.)
> 
> performance at ~83Mb/s for the rest of the 10 second test.    If I 
> were to guess this is probably a policer (shaper?) with a 1 MB token 
> bucket and a ~83Mb/s token rate (these numbers are not corrected for 
> header overheads, which actually matter with this tool).  What is 
> weird about it is that different ingress interfaces to the ISP (peers 
> or serving locations) exhibit different parameters.
> 
> These are prohibitive to test by default, but can have a huge impact 
> in some common situations.  E.g. downloading software updates.
> 
> An unconditional pass is not good, because some buckets are small.  
> What counts as large enough to be ok, and what "derating" is ok?
> 
> Thanks,
> --MM--
> ____
> thanks for keeping me involved. I don’t have a precise answer and 
> doubt, that there will be a single universal truth. ____
> 
> __ __
> 
> aren’t interested in filling a buffer. Buffering events may occur, 
> some of which are useful and to be expected, whereas others are not 
> desired:____
> 
> __ __
> 
> Practically, the set-up should be configurable with commodity hard- 
> and software and all metrics should be measurable at the receiver.
> Burstiness of traffic and a distinction between queuing events which 
> are to be expected and (undesired) queue build up are the events to be 
> causing congestion. The hard- and software to be used should be part 
> of the solution, not part of the problem (bursty source traffic and 
> timestamps with insufficient accuracy to detect queues are what I’d 
> like to avoid).____
> 
> __ __
> 
> thanks for keeping me involved. I don’t have a precise answer and 
> doubt, that there will be a single universal truth. ____
> 
> __ __
> 
> aren’t interested in filling a buffer. Buffering events may occur, 
> some of which are useful and to be expected, whereas others are not 
> desired:____
> 
> __ __
> 
> __-        __Limiting packet drop due to buffer overflow is a design 
> aim or an important part of the algorithm, I think.____
> 
> __-        __Shared media might create bursts. I’m not an expert in 
> the area, but there’s an “is bandwidth available” check in some cases 
> connected. WiFi and may be other wireless equipment buffers packets 
> also to optimize wireless resource optimization.____
> 
> __-        __It might be an idea to mark some flows by ECN, once 
> there’s a guess on a sending bitrate when to expect no or very little 
> packet drop. Today, this is experimental. CE marks by an ECN capable 
> device should be expected roughly once queuing starts.____
> 
> 
> 
>

---

From: Joachim Fabini <[email-masked]>
Date: Wed, 25 Sep 2019 11:55:20 +0200

Hi Ruediger,

thank you for your comments, answers inline.

> thanks for adding a discussion related to mobile access behavior. I've read some "streaming QoE over LTE" related publications. These are not BTC related, but LTE and CCA interaction also impacts t…

Did these publications use isolated LTE networks or public ones?
Whenever (access) network capacity is shared between users (which is the
case for most cellular users in a cell) this is a potential source of
non-determinism in terms of measurements that is {hard|impossible} to
control. Measurement results depend to a substantial extent on the
traffic profile and timing of competing devices and users.

> Your discussion whether to consider an access showing 94.5 Mbit/s throughput for a short time and 83 Mbit/s for a longer interval may indicate, that singleton, short interval BTC measurements used…

Yes, I like your proposal. A BTC singleton could be measured after
detecting stable conditions and for an interval that is sufficiently
short to increase the likelihood of almost constant parameters. The need
to reflect the "true" path BTC (i.e., the *receiver's* perspective) and
detect/eliminate intermediate node buffering is challenging but imo
crucial for such experiments.

> The above QoE experts use receiver statistics, as you suggest too, however their activities are passive, whereas BTC measurements discussed here are active. I add this to indicate, that a mobile ac…

The good news about BTC in mobile networks is that (at least in the
past, when I did measurements) the scheduling/capacity algorithm
predictability and determinism increased for greedy ("the more you ask
for, the more you get") users. And BTC measurements are greedy by
definition.

> I agree that defining "long term average data rate" in a way which is useful to an implementors use case is desirable (the same holds for guidance on a short term singleton BTC metric).

Yes, but again: in white box measurements (without exact knowledge on
the measurement path implementation and configuration) imo there is an
imperative need for the *receiver's* perspective. The *sender* itself
can't differentiate between stateful policing, shaping and buffering
that may happen on the network path.

> To start a discussion: a BTC singleton could consist of back-to-back packets of a length corresponding to an important or expected bottleneck burst tolerance (and/or buffer depth). 

Hmmh. If buffering is known to happen on the path, I consider the BTC
singleton that you mention to be a *prerequisite* for the actual
measurement BTC singleton: The active measurement must send sufficient
data to fill all buffers within the network path under test. Only
subsequent packets allow inference on the path's true BTC.

As a side note: an identical approach ("preload") is used for active
delay measurements in mobile networks: transferring a large data burst
over an idle 3G/4G link makes the radio scheduler to allocate
substantial radio resources (i.e. high capacity) to this link. Sending a
delay measurement packet immediately after the data burst yields delay
values that are by orders of magnitude lower compared to sending the
same packets on an almost idle link.

I mentioned this detail because of two aspects:
1. The same mechanism could be used to create adequate measurement
prerequisites for mobile cellular networks and for fixed access
networks, for BTC and for delay measurements. The packets serve distinct
purpose - force capacity allocation in cellular networks vs. filling
buffers in fixed network paths - but the same method may work for both
cases.
2. Measurement results nowadays depend on the use case (i.e., the
context in which they have been acquired). They reflect the performance
of the network in a particular state that we can (sometimes)
artificially enforce as measurement prerequisite. If the state is the
one encountered in typical operation, then the results are
representative (which will be typically true for BTC). Otherwise they
may be misleading.

> These singletons should be spaced in a way that a loss and may be queue free long term BTC results. 

Sorry, I don't get this last sentence (something missing?).

Regards
Joachim

---

From: <[email-masked]>
Date: Wed, 25 Sep 2019 10:57:35 +0000

Hi Joachim,

the QoE measurement campaigns I mentioned were performed on both, laboratory and public LTE networks. The researchers wanted to measure results which relate to consumer experience. I regard QoE measurements as a source giving me general guidance on what's done on a subject also related to QoS, but don't expect them to serve as a blueprint for QoS measurements.  

Pre-loading may be a useful approach to reach "stable" mobile network conditions. Hopefully, there are choices to specify "stable" in a way allowing separate, but compatible implementations of a metric.

My closing statement below  should express, that a metric determining a stable BTC should consider packet loss and queue build up (i.e., increasing one-way latency during "stable" conditions) as indicators of congestion. The highest sending rate not causing congestion seems to be a metric useful for a BTC metric to me. Another one might be a burst size which can be transmitted causing congestion by a queue build up, but no packet drop.

I fully agree that receiver based measurements or receiver perspective, respectively, should be taken into account.

Regards, Ruediger

-----Ursprüngliche Nachricht-----
Von: Joachim Fabini <[email-masked]> 
Gesendet: Mittwoch, 25. September 2019 11:55
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: AW: [ippm] Fwd: How should capacity measurement interact with shaping?

Hi Ruediger,

Did these publications use isolated LTE networks or public ones?
Whenever (access) network capacity is shared between users (which is the case for most cellular users in a cell) this is a potential source of non-determinism in terms of measurements that is {hard|impossible} to control. Measurement results depend to a substantial extent on the traffic profile and timing of competing devices and users.

> Your discussion whether to consider an access showing 94.5 Mbit/s throughput for a short time and 83 Mbit/s for a longer interval may indicate, that singleton, short interval BTC measurements used…

Yes, I like your proposal. A BTC singleton could be measured after detecting stable conditions and for an interval that is sufficiently short to increase the likelihood of almost constant parameters. The need to reflect the "true" path BTC (i.e., the *receiver's* perspective) and detect/eliminate intermediate node buffering is challenging but imo crucial for such experiments.

> The above QoE experts use receiver statistics, as you suggest too, however their activities are passive, whereas BTC measurements discussed here are active. I add this to indicate, that a mobile ac…

The good news about BTC in mobile networks is that (at least in the past, when I did measurements) the scheduling/capacity algorithm predictability and determinism increased for greedy ("the more you ask for, the more you get") users. And BTC measurements are greedy by definition.

> I agree that defining "long term average data rate" in a way which is useful to an implementors use case is desirable (the same holds for guidance on a short term singleton BTC metric).

Yes, but again: in white box measurements (without exact knowledge on the measurement path implementation and configuration) imo there is an imperative need for the *receiver's* perspective. The *sender* itself can't differentiate between stateful policing, shaping and buffering that may happen on the network path.

> To start a discussion: a BTC singleton could consist of back-to-back packets of a length corresponding to an important or expected bottleneck burst tolerance (and/or buffer depth). 

Hmmh. If buffering is known to happen on the path, I consider the BTC singleton that you mention to be a *prerequisite* for the actual measurement BTC singleton: The active measurement must send sufficient data to fill all buffers within the network path under test. Only subsequent packets allow inference on the path's true BTC.

As a side note: an identical approach ("preload") is used for active delay measurements in mobile networks: transferring a large data burst over an idle 3G/4G link makes the radio scheduler to allocate substantial radio resources (i.e. high capacity) to this link. Sending a delay measurement packet immediately after the data burst yields delay values that are by orders of magnitude lower compared to sending the same packets on an almost idle link.

I mentioned this detail because of two aspects:
1. The same mechanism could be used to create adequate measurement prerequisites for mobile cellular networks and for fixed access networks, for BTC and for delay measurements. The packets serve distinct purpose - force capacity allocation in cellular networks vs. filling buffers in fixed network paths - but the same method may work for both cases.
2. Measurement results nowadays depend on the use case (i.e., the context in which they have been acquired). They reflect the performance of the network in a particular state that we can (sometimes) artificially enforce as measurement prerequisite. If the state is the one encountered in typical operation, then the results are representative (which will be typically true for BTC). Otherwise they may be misleading.

Regards
Joachim

> An: Matt Mathis <mattmathis=[email-masked]>; ALFRED C 
> (AL) <[email-masked]>; Geib, Rüdiger <[email-masked]>
> Cc: [email-masked]
>> to talk about results yet (well ok, it looks pretty good).    (BTW 
>> the BBR algorithm just happens to resemble the algorithm described in
>> ISPs in the US and Europe and I wanted to get some input for how 
>> these cases should be treated.
>>
>> Now the IPPM measurement question:   Is the bulk transport capacity 
>> of this link ~94.5 Mbit/s or ~83Mb/s?   Justify your answer....?
>>
>> Thanks,
>> --MM--
>> If the aim is only to determine the IP bandwidth of an access, then 
>> we aren’t interested in filling a buffer. Buffering events may occur, 
>> are to be expected and (undesired) queue build up are the events to 
>> be distinguished. I hope that can be done with commodity hard- and 
>> If the aim is only to determine the IP bandwidth of an access, then 
>> we aren’t interested in filling a buffer. Buffering events may occur, 
>> __-        __Random collisions should be tolerated at the access 
>> whose bandwidth is to be measured.____
>>

---

From: J Ignacio Alvarez-Hamelin <[email-masked]>
Date: Wed, 25 Sep 2019 19:20:39 -0300

Dear Matt,

I use to discuss with Al and Rüdiger about some issues about measurements in IP networks. 
I would like to introduce another point of view (sorry to get the "entropy" larger…), In my opinion one part of the problem is that one would like to establish how significant is the Internet connection, and it depends not only on the channel technology but in a lot of other parameters (like "buckets", etc.). My question is: why one wants to measure the "real" access if this one could not be available 100% of the time at that specific speed? It is clearly not realistic that an ISP provides 100Mbps all the time for every customer; therefore, another kind of measurement could be considered. 
Form the other optic, BBR is a great technique to achieve the best possible capacity incredibly quick, and here the problem is another. (By the way, I think that the BBR difficulty is potentially related to  the interactions with several concurrent flows; which, even if there are some studies, I think it is not entirely understood.) 
Then, there are situations, from the user-application, that it is needed to use a high capacity during a  time (i.e., security software updates), and this could be interesting to provide protocols, like BBR, to ensure that.
What end you pursuit? Understand what is happening on the network to bring a solution to this kind of applications, or you would like to measure the properties? 
For my point of view (sorry Al to repeat this one more time), if we like to produce some information about the network status for the end-user, we need to work on parameters that express the "quality": quite hard. 
On idea on this path is to know if my network could carry some "bursty" traffic in *every* time; which means to measure continuously, i.e., we cannot saturate the link all the time. This fact leads to another idea: to develop an active and low impact measurement to asses to quality: can I send or receive some busty traffic? I understand that how long and how much "bursty" is not defined here, but we can figure out (or try some reasonable numbers at least ). I got a couple of ideas using RTTs and my experience measuring RTTs on IP networks.
The central part here is that performing measurement using bottleneck limit trigger different mechanisms, like one that you described, but potentially others more complex. (By the way, avoiding lost packets but controlling the delay is an exciting way to influence BBR).

Best,

	Ignacio

_______________________________________________________________

Dr. Ing. José Ignacio Alvarez-Hamelin
CONICET and Facultad de Ingeniería, Universidad de Buenos Aires
Av. Paseo Colón 850 - C1063ACV - Buenos Aires - Argentina
+54 (11) 5285 0716 / 5285 0705
e-mail: [email-masked]
web: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/
_______________________________________________________________

> On 21 Sep 2019, at 15:32, Matt Mathis <mattmathis=[email-masked]> wrote:
> 
> Yes, exactly, I am sure this is a provider's feature.   I have a receiver side pcap, and it is quite a bit more complicated than I thought:
> - Zero losses in the entire trace.  It is dynamically shaped at a bottleneck with a long queue that is pacing packets.
> - The rate (and packet headway) smoothly wanders irregularly all over the place in the latter part of the trace, from a low of about 1 Mb/s to peaks close to the max rate.   My earlier data was from BBR max_rate, so the fluctuating rate apparently has stable peaks.
> 
> By smooth: it looked like a spline fit, suggesting the instantaneous packet headway was determined by a differential equation....
> 
> This behavior is not an accident, but the result of a very sophisticated controller.   And we have seen other bottlenecks like it elsewhere in the US and Europe.
> 
> As people may know, I do believe that shaping is the most appropriate way to deal with heavy hitters.  And that we (IPPM) really need a way to characterize shaping.  I do care about the asymptotic rate, and how quickly I fall out of the initial rate.
> 
> Thanks,
> --MM--
> On Sat, Sep 21, 2019 at 8:34 AM MORTON, ALFRED C (AL) <[email-masked]> wrote:
> Hi Matt,
> 
>  
> 
> On Thu, Sep 19, 2019 at 3:35 PM MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> Cc: CIAVATTONE, LEN <[email-masked]>, [email-masked] <[email-masked]>
> 
>  
> 
> 	• Sender shaping behavior may matter (is traffic at the source CBR or is it bursty)
> 	• Random collisions should be tolerated at the access whose bandwidth is to be measured.
> 	• Limiting packet drop due to buffer overflow is a design aim or an important part of the algorithm, I think.
> 	• Shared media might create bursts. I’m not an expert in the area, but there’s an “is bandwidth available” check in some cases between a central sender using a shared medium and the receivers connected. WiFi and may be other wireless equipment buffers packets also to optimize wireless resource optimization.
> 	• It might be an idea to mark some flows by ECN, once there’s a guess on a sending bitrate when to expect no or very little packet drop. Today, this is experimental. CE marks by an ECN capable device should be expected roughly once queuing starts.
>  
> 
> Cc: CIAVATTONE, LEN <[email-masked]>, [email-masked] <[email-masked]>
> 
>  
> 
> From: [email-masked] [mailto:[email-masked]] 
> Cc: CIAVATTONE, LEN <[email-masked]>; [email-masked]

---

From: <[email-masked]>
Date: Thu, 26 Sep 2019 06:50:30 +0000

Dear Ignacio,

[IAH] My question is: why one wants to measure the "real" access if this one could not be available 100% of the time at that specific speed?

[RG] Regulators busy an industry with so called access "speed tests". This also involves providers which are subjected to regulation. Some of these are interested in standardizing IP access bandwidth measurements.

[RG] Some text of your message seems to be concerned with detection of the available bottleneck bandwidth (e.g., in the presence of background traffic). That's an interesting subject too. We should separate discussion, I think. 

[RG] Contributions to the detection of congestion indicated by a queue build up are welcome. RTT is one potential method and the merits and flaws of it can become part of the draft, I think.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: ippm <[email-masked]> Im Auftrag von J Ignacio Alvarez-Hamelin
Gesendet: Donnerstag, 26. September 2019 00:21
An: Matt Mathis <mattmathis=[email-masked]>
Cc: MORTON, ALFRED C (AL) <[email-masked]>; [email-masked]
Betreff: Re: [ippm] How should capacity measurement interact with shaping?

Dear Matt,

Form the other optic, BBR is a great technique to achieve the best possible capacity incredibly quick, and here the problem is another. (By the way, I think that the BBR difficulty is potentially related to  the interactions with several concurrent flows; which, even if there are some studies, I think it is not entirely understood.) Then, there are situations, from the user-application, that it is needed to use a high capacity during a  time (i.e., security software updates), and this could be interesting to provide protocols, like BBR, to ensure that.

Best,

	Ignacio

CONICET and Facultad de Ingeniería, Universidad de Buenos Aires Av. Paseo Colón 850 - C1063ACV - Buenos Aires - Argentina

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 30 Sep 2019 11:38:45 +0000

Hi Ignacio and Rüdiger,
please one reply below.

> -----Original Message-----
> From: ippm [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Thursday, September 26, 2019 2:51 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [ippm] How should capacity measurement interact with shaping?
> 
> Dear Ignacio,
> 
> [IAH] My question is: why one wants to measure the "real" access if this
> one could not be available 100% of the time at that specific speed?
> 
> [RG] Regulators busy an industry with so called access "speed tests". This
> also involves providers which are subjected to regulation. Some of these
> are interested in standardizing IP access bandwidth measurements.
> 
> [RG] Some text of your message seems to be concerned with detection of the
> available bottleneck bandwidth (e.g., in the presence of background
> traffic). That's an interesting subject too. We should separate
> discussion, I think.
> 
> [RG] Contributions to the detection of congestion indicated by a queue
> build up are welcome. RTT is one potential method and the merits and flaws
> of it can become part of the draft, I think.
[acm] 
On this last point, it could be part of the draft where Ignacio has already
provided some discussion:
https://tools.ietf.org/html/draft-ietf-ippm-route-05#section-5
since this section aims to identify properties of paths, IMO.

Al

> I would like to introduce another point of view (sorry to get the
> "entropy" larger…), In my opinion one part of the problem is that one
> would like to establish how significant is the Internet connection, and it
> depends not only on the channel technology but in a lot of other
> parameters (like "buckets", etc.). My question is: why one wants to
> measure the "real" access if this one could not be available 100% of the
> time at that specific speed? It is clearly not realistic that an ISP
> provides 100Mbps all the time for every customer; therefore, another kind
> of measurement could be considered.
> Form the other optic, BBR is a great technique to achieve the best
> possible capacity incredibly quick, and here the problem is another. (By
> the way, I think that the BBR difficulty is potentially related to  the
> interactions with several concurrent flows; which, even if there are some
> studies, I think it is not entirely understood.) Then, there are
> situations, from the user-application, that it is needed to use a high
> capacity during a  time (i.e., security software updates), and this could
> be interesting to provide protocols, like BBR, to ensure that.
> What end you pursuit? Understand what is happening on the network to bring
> a solution to this kind of applications, or you would like to measure the
> properties?
> For my point of view (sorry Al to repeat this one more time), if we like
> to produce some information about the network status for the end-user, we
> need to work on parameters that express the "quality": quite hard.
> On idea on this path is to know if my network could carry some "bursty"
> traffic in *every* time; which means to measure continuously, i.e., we
> cannot saturate the link all the time. This fact leads to another idea: to
> develop an active and low impact measurement to asses to quality: can I
> send or receive some busty traffic? I understand that how long and how
> much "bursty" is not defined here, but we can figure out (or try some
> reasonable numbers at least ). I got a couple of ideas using RTTs and my
> experience measuring RTTs on IP networks.
> The central part here is that performing measurement using bottleneck
> limit trigger different mechanisms, like one that you described, but
> potentially others more complex. (By the way, avoiding lost packets but
> controlling the delay is an exciting way to influence BBR).
> 
> Best,
> 
> 	Ignacio
> 
> 
> CONICET and Facultad de Ingeniería, Universidad de Buenos Aires Av. Paseo
> Colón 850 - C1063ACV - Buenos Aires - Argentina
> web: https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__cnet.fi.uba.ar_ignacio.alvarez-2Dhamelin_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=fBHgaolKIJm5xN0KzOYdpvuxBfPZb7a3jwA5PaxNJo8&e=
> > On 21 Sep 2019, at 15:32, Matt Mathis
> <mattmathis=[email-masked]> wrote:
> >
> > Yes, exactly, I am sure this is a provider's feature.   I have a
> receiver side pcap, and it is quite a bit more complicated than I thought:
> > - The rate (and packet headway) smoothly wanders irregularly all over
> the place in the latter part of the trace, from a low of about 1 Mb/s to
> peaks close to the max rate.   My earlier data was from BBR max_rate, so
> the fluctuating rate apparently has stable peaks.
> >
> > By smooth: it looked like a spline fit, suggesting the instantaneous
> packet headway was determined by a differential equation....
> >
> > As people may know, I do believe that shaping is the most appropriate
> way to deal with heavy hitters.  And that we (IPPM) really need a way to
> characterize shaping.  I do care about the asymptotic rate, and how
> quickly I fall out of the initial rate.
> >
> > Thanks,
> > --MM--
> this should be reported, and what should the language be about non-
> stationary available capacity.   One intersecting issue: BBR converges on
> both the initial and final rate in under 2 seconds (this was a long path,
> so startup took more than a second).   Do users want a quick (and
> > On the pure technical issues: BBR is still quite a moving target.   I
> have a paper in draft that will shed some light on this.  It is due to be
> beginning of the connection to now, the peak rate happens at the moment
> the bucket runs out of tokens, and falls towards the toke rate after that.
> >
> >
> >
> > Thanks,
> >
> > --MM--
> > https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__cacm.acm.org_magazines_2017_2_212428-2Dbbr-2Dcongestion-2Dbased-
> 2Dcong&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=0U4RvwRX4FczhzosE-0W0YYHtDbDgumnInzLZKbJDzQ&e=
> > estion-control/fulltext
> >
> >
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dmorton-2Dippm-2Dcapcity-2Dmetric-2Dmethod-
> 2D00&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=clXXsk_Qf4IKTrlTErZavqIalw01TstgIbM6ekDiULg&e=
> > #section-4.3
> >
> >
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_meeting_105_materials_slides-2D105-2Dippm-
> 2Dmet&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=_1Y3qxWv-oQptQVtwk42-OFsfbkf9ckKFWFZ5Bs7dbI&e=
> > rics-and-methods-for-ip-capacity-00
> >
> >
> >
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__web.cs.wpi.edu_-
> 7Eclaypool_papers_driving-2Dbbr_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=sW-j7eyAkkqcrmPgkTdVlScf4546Gz3HCtf1AQldc5Q&e=
> >
> algorithm just happens to resemble the algorithm described in draft-
> morton-ippm-capcity-metric-method-00.)
> >
> >
> >
> fluctuating performance ~75 Mb/s for ~1 second and then stable performance
> at ~83Mb/s for the rest of the 10 second test.    If I were to guess this
> is probably a policer (shaper?) with a 1 MB token bucket and a ~83Mb/s
> token rate (these numbers are not corrected for header overheads, which
> actually matter with this tool).  What is weird about it is that different
> ingress interfaces to the ISP (peers or serving locations) exhibit
> different parameters.
> >
> >
> >
> > 	• Limiting packet drop due to buffer overflow is a design aim or an
> > 	• Shared media might create bursts. I’m not an expert in the area,
> > 	• It might be an idea to mark some flows by ECN, once there’s a
> expected and (undesired) queue build up are the events to be
> distinguished. I hope that can be done with commodity hard- and software.
> I at least am not able to write down a simple metric distinguishing queues
> to be expected from (undesired) queue build up causing congestion. The
> hard- and software to be used should be part of the solution, not part of
> the problem (bursty source traffic and timestamps with insufficient
> accuracy to detect queues are what I’d like to avoid).
> >
> >
> >
> > -        Shared media might create bursts. I’m not an expert in the
> area, but there’s an “is bandwidth available” check in some cases between
> a central sender using a shared medium and the receivers connected. WiFi
> and may be other wireless equipment buffers packets also to optimize
> wireless resource optimization.
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_ippm&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=gcL4ff0SGxbLqvJzUshWBxIyvEfPeenVE7LsB_MJBxE&e=
> 
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_ippm&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=gcL4ff0SGxbLqvJzUshWBxIyvEfPeenVE7LsB_MJBxE&e=
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_ippm&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=Q01OpB9liGLHD2NujOQz2ZsZstWhJ
> mYmq9QDzuytvFM&s=gcL4ff0SGxbLqvJzUshWBxIyvEfPeenVE7LsB_MJBxE&e=