Subject: [iccrg] Evidence for non-congestive loss

From: Bob Briscoe <[email-masked]>
Date: Mon, 13 Nov 2017 15:09:03 +0800

Neal,

Following up on my question just now from the mic in iccrg "Where was 
the evidence that loss is not a good estimator of congestion?"

We all know that all loss is not congestion, but what we don't know is 
how prevalent the different types of loss are.

When Google (Nandita) introduced IW10, it was backed up by an impressive 
amount of measurement evidence to support it. BBR would (still) benefit 
from a similar degree of diligence.

Cheers

Bob

PS. some causes of intermittent loss:

  * tail loss
  * AQM
  * rate policing
  * radio interference
  * packet processer exhaustion

Tail loss or AQM loss could be due to

  * self-induced loss-seeking (traditional CCs)
  * sudden reduction in available capacity due to radio rate selection
  * sudden reduction in available capacity due to competing flow-starts

---

From: Yuchung Cheng <[email-masked]>
Date: Sun, 12 Nov 2017 23:21:19 -0800

Just to clarify (I wanted to chime but Michael already cut the line):

First, please note that Neal's slide says

loss **alone** is not a good signal for congestion.

I am concerned that people are missing the keyword "alone" and think BBR
developers consider loss is a useless signal. We do not.

Like you and Stewart pointed out, loss is more than often caused by
congestion, and it is really hard to tell if it is not precisely. Equally
difficult is that loss is caused by the flow experiencing it.

So no evidence is needed, b/c it's not a fact we believe.

On Sun, Nov 12, 2017 at 11:09 PM, Bob Briscoe <[email-masked]> wrote:

> Neal,
>
> Following up on my question just now from the mic in iccrg "Where was the
> evidence that loss is not a good estimator of congestion?"
>
> We all know that all loss is not congestion, but what we don't know is how
> prevalent the different types of loss are.
>    - tail loss
>    - AQM
>    - rate policing
>    - radio interference
>    - packet processer exhaustion
>
>    - self-induced loss-seeking (traditional CCs)
>    - sudden reduction in available capacity due to radio rate selection
>    - sudden reduction in available capacity due to competing flow-starts
>
>
>
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg
>
>

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 13 Nov 2017 03:32:27 -0500

Hi Bob,

Thanks for your note. I mainly just want to echo Yuchung's remark. It's
3:30am here in NYC now, so I'm sorry if I wasn't clear, or mis-spoke while
talking, but I meant to say something more narrow and closer to what the
slide says, which is "Packet loss alone is not a good proxy to detect
congestion". :-)

It sounds like we agree that not all loss is congestion. I suspect we also
agree that not all congestion results in loss (i.e. that buffer-bloat can
happen). (And I think we agree on the importance of measurement evidence.)

We are certainly not intending to say that congestion control algorithms
should ignore loss. Loss is often a hugely useful signal, and clearly is
often correlated with congestion. Even BBR v1 responds to loss to some
extent. And today we described at ICCRG how we intend for BBR v2 to use a
packet loss signal to a considerably greater extent, and described the
kinds of improvements we are seeing from that approach.

But I think an important piece of all this is that where loss is due to
congestion, there are important questions that the Reno and CUBIC
approaches do not grapple with, which we are trying to be conscious of in
the BBR project:

(a) What is the time scale of the congestion? For example, if the buffer is
1ms and a flow's RTT is 100ms, there can easily be several bursts of
1ms-scale packet loss on every round-trip of that flow when more bandwidth
is available (the link may even be 90% idle). So Reno- or CUBIC-style
responses will have a hard time utilizing such links.

(b) How bursty or consistent is the congestion, and what is the time scale
of the back-off and probing response? For example, if a bottleneck sees
bursty cross-traffic that slow-starts and fills the buffer and causes loss
every N seconds, and the congestion control algorithm run by long-lived
flows on the link only probes the link beyond the flows' pre-congestion
rate at intervals greater than N seconds, then the CC algorithm will have a
hard time utilizing the pipe well. (As you illustrated nicely in a recent
IETF, CUBIC can take quite a long time to reprobe for bandwidth.)

thanks,
neal

ps: For folks who haven't sen them, the BBR slides for today's ICCRG talk
are here:

https://datatracker.ietf.org/meeting/100/materials/slides-100-iccrg-a-quick-bbr-update-bbr-in-shallow-buffers/

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 15 Nov 2017 10:57:15 +0800

Yuchung,

OK, I'll also refine the criticism:

The commonly accepted criterion for CC over the last 3 decades has been 
that not all losses are due to congestion (i.e. due to capacity-seeking 
behaviour). However, it has not been possible to tell which sort of loss 
each loss is (literally hundreds of research papers on this). So has 
been considered necessary to behave conservatively and assume each loss 
might be due to congestion. If a heuristic can be used to determine it 
was probably not congestion, it might then be possible to rewind the 
previous decision.

There have also been hundreds of research papers (often rejected many 
times before they are accepted in some rubbish conference) that just 
assume away loss as probably due to interference, and (unsurprisingly) 
show really good performance. These papers rarely show evidence that 
other flows have not been harmed.

BBR v1 is little different to these papers, with the exception that it 
has been deployed for a significant portion of the traffic on the 
planet. You may not realize (both being remote from the IETF) that BBR 
is being considered the big joke of the moment.

The originating authors of BBR all have access to machinery that can 
gather evidence far more easily than the rest of us. I am asking you to 
switch your focus from a) measuring self-performance of your deployment 
to b) finding, using and reporting evidence that will help distinguish 
the different types of loss.

Bob

On 13/11/2017 15:21, Yuchung Cheng wrote:
> I am concerned that people are missing the keyword "alone" and think 
> BBR developers consider loss is a useless signal. We do not.
>
> congestion, and it is really hard to tell if it is not precisely. 
> Equally difficult is that loss is caused by the flow experiencing it.
>
> On Sun, Nov 12, 2017 at 11:09 PM, Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Neal,
>
>     Following up on my question just now from the mic in iccrg "Where
>     was the evidence that loss is not a good estimator of congestion?"
>
>     We all know that all loss is not congestion, but what we don't
>     know is how prevalent the different types of loss are.
>
>     When Google (Nandita) introduced IW10, it was backed up by an
>     impressive amount of measurement evidence to support it. BBR would
>     (still) benefit from a similar degree of diligence.
>
>     Cheers
>
>
>     Bob
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.irtf.org/mailman/listinfo/iccrg>
>
>
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 15 Nov 2017 11:44:45 +0800

Neal,

Yes, this is the direction I was asking for. Thank you.

Self-correlation of losses used to be considered (in the 1990s) a way to 
distinguish tail loss from interference loss. Then there were a couple 
of papers in the 2000s that found a different trend.

Here's some references on burst loss temporal correlation pasted from my 
PhD dissertation (2009):

We can find only two Internet measurement studies of temporal loss 
correlation where the year of data collection was after RED could have 
been deployed—after the IRTF published its strong recommendation to use 
RED [BCC+98] in 1998. Zhang et al [ZD01] (1999–2001 data) found nothing 
to contradict earlier studies [Pax99,YMKT99] showing losses were not iid 
but arrived in strongly correlated bursts. But Brosh et al [BLSS05, §IV] 
(2002 data) claimed that the distribution of about 150,000 measured loss 
burst lengths was consistent with iid packet loss, contradicting earlier 
findings of loss correlation.^28 Brosh et al ventured the explanation 
that more widespread RED deployment may have been the cause, or perhaps 
there were now more flows at tail drop queue

______________________
^28 Unfortunately, this finding was incidental to the primary focus of 
the paper so the data analysis was not published.

*References**
*
[BCC+98] B. Braden, D. Clark, J. Crowcroft, B. Davie, S. Deering, D. 
Estrin, S. Floyd, V. Jacobson, G. Minshall, C. Partridge, L. Peterson, 
K. Ramakrishnan, S. Shenker, J. Wroclawski, and L. Zhang. 
Recommendations on queue management and congestion avoidance in the 
Internet. Request for comments 2309, Internet Engineering Task Force, 
April 1998

[ZD01]Yin Zhang and Nick Duffield. On the constancy of Internet path 
properties. In Proc. 1st SIGCOMM Workshop on Internet Measurement (IMW 
’01), pages 197–211, New York, NY, USA, 2001. ACM

[Pax99] Vern Paxson. End-to-end Internet packet dynamics. IEEE/ACM 
Transactions on Networking, 7(3):227–292, June 1999

[YMKT99] Maya Yajnik, Sue B. Moon, James F. Kurose, and Donald F. 
Towsley. Measurement and modeling of the temporal dependence in packet 
loss. In Proc. IEEE Conference on Computer Communications (Infocom’99), 
pages 345–352. IEEE, 1999.

[BLSS05] Eli Brosh, Galit Lubetzky-Sharon, and Yuval Shavitt. 
Spatial-temporal analysis of passive TCP measurements. In Proc. IEEE 
Conference on Computer Communications (Infocom’05), pages 949–959. IEEE, 
March 2005.

Bob

On 13/11/2017 16:32, Neal Cardwell wrote:
> Hi Bob,
>
> Thanks for your note. I mainly just want to echo Yuchung's remark. 
> It's 3:30am here in NYC now, so I'm sorry if I wasn't clear, or 
> mis-spoke while talking, but I meant to say something more narrow and 
> closer to what the slide says, which is "Packet loss alone is not a 
> good proxy to detect congestion". :-)
>
> It sounds like we agree that not all loss is congestion. I suspect we 
> also agree that not all congestion results in loss (i.e. that 
> buffer-bloat can happen). (And I think we agree on the importance of 
> measurement evidence.)
>
> We are certainly not intending to say that congestion control 
> algorithms should ignore loss. Loss is often a hugely useful signal, 
> and clearly is often correlated with congestion. Even BBR v1 responds 
> to loss to some extent. And today we described at ICCRG how we intend 
> for BBR v2 to use a packet loss signal to a considerably greater 
> extent, and described the kinds of improvements we are seeing from 
> that approach.
>
> But I think an important piece of all this is that where loss is due 
> to congestion, there are important questions that the Reno and CUBIC 
> approaches do not grapple with, which we are trying to be conscious of 
> in the BBR project:
>
> (a) What is the time scale of the congestion? For example, if the 
> buffer is 1ms and a flow's RTT is 100ms, there can easily be several 
> bursts of 1ms-scale packet loss on every round-trip of that flow when 
> more bandwidth is available (the link may even be 90% idle). So Reno- 
> or CUBIC-style responses will have a hard time utilizing such links.
>
> (b) How bursty or consistent is the congestion, and what is the time 
> scale of the back-off and probing response? For example, if a 
> bottleneck sees bursty cross-traffic that slow-starts and fills the 
> buffer and causes loss every N seconds, and the congestion control 
> algorithm run by long-lived flows on the link only probes the link 
> beyond the flows' pre-congestion rate at intervals greater than N 
> seconds, then the CC algorithm will have a hard time utilizing the 
> pipe well. (As you illustrated nicely in a recent IETF, CUBIC can take 
> quite a long time to reprobe for bandwidth.)
>
> thanks,
> neal
>
> ps: For folks who haven't sen them, the BBR slides for today's ICCRG 
> talk are here:
> On Mon, Nov 13, 2017 at 2:21 AM, Yuchung Cheng <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I am concerned that people are missing the keyword "alone" and
>     think BBR developers consider loss is a useless signal. We do not.
>
>     Like you and Stewart pointed out, loss is more than often caused
>     by congestion, and it is really hard to tell if it is not
>     precisely. Equally difficult is that loss is caused by the flow
>     experiencing it.
>
>         Following up on my question just now from the mic in iccrg
>         "Where was the evidence that loss is not a good estimator of
>         congestion?"
>
>         impressive amount of measurement evidence to support it. BBR
>         would (still) benefit from a similar degree of diligence.
>
>         Cheers
>
>
>         Bob
>

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 15 Nov 2017 23:06:09 +0800

You may not realize (both being remote from the IETF) that BBR is being
considered the big joke of the moment.

(chair hat off)

This is unwarranted and really not necessary. I was in the room, spoke with
folks after, and this is nowhere near my read of people's response to BBR.
It's being considered seriously by most, and folks are watching with keen
interest. I found much of the engagement useful and helpful, including
yours, and I'm hoping that we can continue to have constructive
conversation.

- jana

The originating authors of BBR all have access to machinery that can gather
evidence far more easily than the rest of us. I am asking you to switch
your focus from a) measuring self-performance of your deployment to b)
finding, using and reporting evidence that will help distinguish the
different types of loss.

Bob

_______________________________________________
iccrg mailing [email-masked]://www.irtf.org/mailman/listinfo/iccrg

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 16 Nov 2017 15:29:21 -0500

Bob,

Thank you for the pointers to references. I can't speak for my co-authors,
but some of those publications I was aware of, and others I had not run
into.

>From my perspective, a big focus of BBR v2 is essentially related to this
question you raise about distinguishing different types of loss.

Your point about evidence is well-taken. For the roll-out of BBR v1 we
accumulated a a fair amount of data, and we are hoping to do what we can to
see if we can share more of that data (as well as data related to v2, when
the time comes).

cheers,
neal

On Tue, Nov 14, 2017 at 10:44 PM, Bob Briscoe <[email-masked]> wrote:

> distinguish tail loss from interference loss. Then there were a couple of
> papers in the 2000s that found a different trend.
>
> correlation where the year of data collection was after RED could have been
> deployed—after the IRTF published its strong recommendation to use RED
> [BCC+98] in 1998. Zhang et al [ZD01] (1999–2001 data) found nothing to
> contradict earlier studies [Pax99,YMKT99] showing losses were not iid but
> arrived in strongly correlated bursts. But Brosh et al [BLSS05, §IV]
> findings of loss correlation.28 Brosh et al ventured the explanation that
> more widespread RED deployment may have been the cause, or perhaps there
> were now more flows at tail drop queue
>
> ______________________
> 28 Unfortunately, this finding was incidental to the primary focus of the
> paper so the data analysis was not published.
>
> *References*
>
> Estrin, S. Floyd, V. Jacobson, G. Minshall, C. Partridge, L. Peterson, K.
> Ramakrishnan, S. Shenker, J. Wroclawski, and L. Zhang. Recommendations on
> queue management and congestion avoidance in the Internet. Request for
> comments 2309, Internet Engineering Task Force, April 1998
>
> ’01), pages 197–211, New York, NY, USA
> <https://maps.google.com/?q=197%E2%80%93211,+New+York,+NY,+USA&entry=gmail&source=g>,
> 2001. ACM
>
> [YMKT99] Maya Yajnik, Sue B. Moon, James F. Kurose, and Donald F. Towsley.
> Measurement and modeling of the temporal dependence in packet loss. In
> Proc. IEEE Conference on Computer Communications (Infocom’99), pages
> 345–352. IEEE, 1999.
>
> (a) What is the time scale of the congestion? For example, if the buffer
> is 1ms and a flow's RTT is 100ms, there can easily be several bursts of
>   https://datatracker.ietf.org/meeting/100/materials/slides-
> 100-iccrg-a-quick-bbr-update-bbr-in-shallow-buffers/
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 17 Nov 2017 11:24:53 +0800

Neal,

It's going to be hard to identify AQM in real-time given there are 
potentially different signatures (and they will change over time). This 
paper might contain helpful pointers - obviously it is trying to 
distinguish between different AQMs, but that is a step on the way to 
distinguishing between other causes of loss from loss due to any AQM.

Demonstration of TADA: A tool for automatic detection of AQM 
<http://ieeexplore.ieee.org/document/7562227/?reload=true>

Bob

On 17/11/2017 04:29, Neal Cardwell wrote:
> Bob,
>
> Thank you for the pointers to references. I can't speak for my 
> co-authors, but some of those publications I was aware of, and others 
> I had not run into.
>
> From my perspective, a big focus of BBR v2 is essentially related to 
> this question you raise about distinguishing different types of loss.
>
> accumulated a a fair amount of data, and we are hoping to do what we 
> can to see if we can share more of that data (as well as data related 
> to v2, when the time comes).
>
> cheers,
> neal
>
> On Tue, Nov 14, 2017 at 10:44 PM, Bob Briscoe <[email-masked] 
>     Self-correlation of losses used to be considered (in the 1990s) a
>     way to distinguish tail loss from interference loss. Then there
>     were a couple of papers in the 2000s that found a different trend.
>
>     Here's some references on burst loss temporal correlation pasted
>     from my PhD dissertation (2009):
>
>     correlation where the year of data collection was after RED could
>     have been deployed—after the IRTF published its strong
>     recommendation to use RED [BCC+98] in 1998. Zhang et al [ZD01]
>     (1999–2001 data) found nothing to contradict earlier studies
>     [Pax99,YMKT99] showing losses were not iid but arrived in strongly
>     correlated bursts. But Brosh et al [BLSS05, §IV] (2002 data)
>     claimed that the distribution of about 150,000 measured loss burst
>     lengths was consistent with iid packet loss, contradicting earlier
>     findings of loss correlation.^28 Brosh et al ventured the
>     explanation that more widespread RED deployment may have been the
>     cause, or perhaps there were now more flows at tail drop queue
>
>     ______________________
>     ^28 Unfortunately, this finding was incidental to the primary
>     focus of the paper so the data analysis was not published.
>
>     *References**
>     *
>     [BCC+98] B. Braden, D. Clark, J. Crowcroft, B. Davie, S. Deering,
>     D. Estrin, S. Floyd, V. Jacobson, G. Minshall, C. Partridge, L.
>     Peterson, K. Ramakrishnan, S. Shenker, J. Wroclawski, and L.
>     Zhang. Recommendations on queue management and congestion
>     avoidance in the Internet. Request for comments 2309, Internet
>     Engineering Task Force, April 1998
>
>     [ZD01]Yin Zhang and Nick Duffield. On the constancy of Internet
>     path properties. In Proc. 1st SIGCOMM Workshop on Internet
>     Measurement (IMW ’01), pages 197–211, New York, NY, USA
>     Towsley. Measurement and modeling of the temporal dependence in
>     packet loss. In Proc. IEEE Conference on Computer Communications
>     (Infocom’99), pages 345–352. IEEE, 1999.
>
>     Spatial-temporal analysis of passive TCP measurements. In Proc.
>     IEEE Conference on Computer Communications (Infocom’05), pages
>     949–959. IEEE, March 2005.
>
>
>     Bob
>
>
>>     Thanks for your note. I mainly just want to echo Yuchung's
>>     remark. It's 3:30am here in NYC now, so I'm sorry if I wasn't
>>     clear, or mis-spoke while talking, but I meant to say something
>>     more narrow and closer to what the slide says, which is "Packet
>>     loss alone is not a good proxy to detect congestion". :-)
>>
>>     It sounds like we agree that not all loss is congestion. I
>>     suspect we also agree that not all congestion results in loss
>>     (i.e. that buffer-bloat can happen). (And I think we agree on the
>>     importance of measurement evidence.)
>>
>>     algorithms should ignore loss. Loss is often a hugely useful
>>     signal, and clearly is often correlated with congestion. Even BBR
>>     v1 responds to loss to some extent. And today we described at
>>     ICCRG how we intend for BBR v2 to use a packet loss signal to a
>>     considerably greater extent, and described the kinds of
>>     improvements we are seeing from that approach.
>>
>>     But I think an important piece of all this is that where loss is
>>     due to congestion, there are important questions that the Reno
>>     and CUBIC approaches do not grapple with, which we are trying to
>>     be conscious of in the BBR project:
>>
>>     buffer is 1ms and a flow's RTT is 100ms, there can easily be
>>     several bursts of 1ms-scale packet loss on every round-trip of
>>     that flow when more bandwidth is available (the link may even be
>>     90% idle). So Reno- or CUBIC-style responses will have a hard
>>     time utilizing such links.
>>
>>     (b) How bursty or consistent is the congestion, and what is the
>>     time scale of the back-off and probing response? For example, if
>>     a bottleneck sees bursty cross-traffic that slow-starts and fills
>>     the buffer and causes loss every N seconds, and the congestion
>>     control algorithm run by long-lived flows on the link only probes
>>     the link beyond the flows' pre-congestion rate at intervals
>>     greater than N seconds, then the CC algorithm will have a hard
>>     time utilizing the pipe well. (As you illustrated nicely in a
>>     recent IETF, CUBIC can take quite a long time to reprobe for
>>     bandwidth.)
>>
>>     thanks,
>>     neal
>>
>>     ps: For folks who haven't sen them, the BBR slides for today's
>>     ICCRG talk are here:
>>     <https://datatracker.ietf.org/meeting/100/materials/slides-100-iccrg-a-quick-bbr-update-bbr-in-shallow-buffers/>
>>
>>
>>         I am concerned that people are missing the keyword "alone"
>>         and think BBR developers consider loss is a useless signal.
>>         We do not.
>>
>>         Like you and Stewart pointed out, loss is more than often
>>         caused by congestion, and it is really hard to tell if it is
>>         not precisely. Equally difficult is that loss is caused by
>>         the flow experiencing it.
>>
>>         On Sun, Nov 12, 2017 at 11:09 PM, Bob Briscoe
>>         <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>             Neal,
>>
>>             Following up on my question just now from the mic in
>>             iccrg "Where was the evidence that loss is not a good
>>             estimator of congestion?"
>>
>>             We all know that all loss is not congestion, but what we
>>             don't know is how prevalent the different types of loss are.
>>
>>             When Google (Nandita) introduced IW10, it was backed up
>>             by an impressive amount of measurement evidence to
>>             support it. BBR would (still) benefit from a similar
>>             degree of diligence.
>>
>>             Cheers
>>
>>
>>             Bob
>>

---

From: Neal Cardwell <[email-masked]>
Date: Sun, 19 Nov 2017 20:55:30 -0500

Hi Bob,

Thanks for the pointer to the TADA paper.

cheers,
neal

On Thu, Nov 16, 2017 at 10:24 PM, Bob Briscoe <[email-masked]> wrote:

>> findings of loss correlation.28 Brosh et al ventured the explanation
>> 28 Unfortunately, this finding was incidental to the primary focus of
>> also agree that not all congestion results in loss (i.e. that buffer-bloat
>> can happen). (And I think we agree on the importance of measurement
>> evidence.)
>>
>> scale of the back-off and probing response? For example, if a bottleneck
>> sees bursty cross-traffic that slow-starts and fills the buffer and causes
>> loss every N seconds, and the congestion control algorithm run by
>> long-lived flows on the link only probes the link beyond the flows'
>> pre-congestion rate at intervals greater than N seconds, then the CC
>> algorithm will have a hard time utilizing the pipe well. (As you
>> illustrated nicely in a recent IETF, CUBIC can take quite a long time to
>> reprobe for bandwidth.)
>>
>> thanks,
>> neal
>>
>>   https://datatracker.ietf.org/meeting/100/materials/slides-10
>> 0-iccrg-a-quick-bbr-update-bbr-in-shallow-buffers/
>>
>>
>>>> impressive amount of measurement evidence to support it. BBR would (still)
>>>> benefit from a similar degree of diligence.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> Bob
>>>>