Subject: [AVTCORE] Mirja Kühlewind's Discuss on draft-ietf-avtcore-rtp-circuit-breakers-15: (with DISCUSS and COMMENT)

From: "Mirja Kuehlewind" <[email-masked]>
Date: Tue, 03 May 2016 05:03:21 -0700

Mirja Kühlewind has entered the following ballot position for
draft-ietf-avtcore-rtp-circuit-breakers-15: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-avtcore-rtp-circuit-breakers/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

This is about one point in section 7 (ECN) that I think is wrong but I
would like to get some feedback from the authors:
"then ECN-CE marked packets SHOULD
   be treated as if they were lost when calculating if the congestion-
   based RTP circuit breaker"
(also section 5: "The count of ECN-CE marked packets
   contained in those ECN feedback reports is counted towards the number
   of lost packets reported")

We are currently discussion mechanisms where the AQM in the congested
network node sends  much more CE markings than one would see loss (when
using TCP) for the same level of congestion. When treating ECN-CE similar
to loss, such a different behavior could trigger the circuit breaker
unnecessarily. Potentially ECN-CE might not need to be considered here at
all, because as long as there are (only) ECN-CE marks (and no loss) all
data is transmitted correctly to the receiver and therefore there is no
need to trigger a circuit breaker.

Further also in section 7: "ECN-CE marked packets SHOULD be treated as if
they were lost for the
   purposes of congestion control"

This document should not impose any SHOULDs for congestion control as
this doc is only about circuit breaker sand therefore the sentence above
should be removed.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Few more minor comments:

1) reference [I-D.ietf-tsvwg-circuit-breaker] should be normative
2) How is the loss rate in 4.3 calculated if some (but no all) RR are
lost?
3) I don't think that most of the text on congestion control in section 2
(as well as the abstract) is necessary for this doc (but it also don't
really hurt)

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 4 May 2016 12:52:58 +0200

Hi Colin,

see below. 

> Am 04.05.2016 um 12:32 schrieb Colin Perkins <[email-masked]>:
> 
> Hi,
> 
>> On 3 May 2016, at 13:03, Mirja Kuehlewind <[email-masked]> wrote:
>> 
> This is echoing the language in RFCs 3168 and 6679. My understanding is that both require CE marks to be treated the same as lost packets for congestion control, and the circuit breaker is essentially a form of congestion control.

There is currently a draft under discussion in tcpm to update rfc6138 and remove this statement.  So this statement should definitely not be picked up in this draft.

Further I disagree that a circuit breaker is a form of congestion control. It’s a last resort but there seem to be not much of a control loop involved here, making two very different thing with different requirements.

> Not sure I agree. The circuit breaker needs to treat lost and CE-marked packets the same as a congestion control algorithm, and the ECN specifications say that the response to CE marks needs to be the same as the response to loss. 

I disagree, given the explanation above that ECN-CE marking indicate that all traffic have been transmitted correctly. It’s only an input for a control mechanism/loop and should not be used as input for a circuit breaker, at least not the same way as loss it used.

You could calculate the loss and ecn ratio separately and e.g. lower the k value if there is also a high level of CE marks. However, only CE marks without loss should not trigger a circuit breaker. This is also what ietf-tsvwg-circuit-breaker says:
"If Explicit
      Congestion Notification (ECN) is enabled [RFC3168], an egress
      meter MAY also count the number of ECN congestion marks/event per
      measurement interval, but even if ECN is used, loss MUST still be
      measured, since this better reflects the impact of persistent
      congestion.“

> This was discussed in relation to Ben’s AD review. I think you can implement the RTP circuit breaker without reading the TSV draft, so informative seems correct. However, I don’t much care either way.

I will double-check this; was not aware of any previous discussions here. I still think it should be normative...

> There’s no obvious way for the receiver of the RRs to know that some were lost, so the calculation will proceed as if the reporting interval was longer. 

Hm… what’s about using the (difference of )total number of losses instead?

> I agree it’s not necessary, but I tend to think motivation text is helpful in specifications.

Don’t have strong feeling about this. However as explain above, I don’t agree that circuit breaks are a kind of congestion control and there it would be much cleaner from my point of view to have those two things clearly separated.

Mirja

> Cheers,
> Colin
> -- 
> Colin Perkins
> https://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Wed, 4 May 2016 11:32:35 +0100

Hi,

Cheers,
Colin

-- 
Colin Perkins
https://csperkins.org/

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 4 May 2016 13:42:31 +0200

Sorry, one clarification: updating 3168 is work in tsvwg (not tcpm).

> Am 04.05.2016 um 12:52 schrieb Mirja Kuehlewind (IETF) <[email-masked]>:
> 
> Hi Colin,
> 
> see below. 
> 
>

---

From: Varun Singh <[email-masked]>
Date: Thu, 5 May 2016 23:43:24 +0300

Hi Mirja,

see inline.

Regards,
Varun

>       measured, since this better reflects the impact of persistent

First, I believe the draft assumes ECN-CE should be considered as lost
packets in addition to the lost packets.

Second, I am not sure the text from the tsvg-circuit-breaker applies
in this case. Because the calculation in section 4.3 takes place at
the sender. And the  sender receives the ECN-CE counter from
the receiver in an RTCP XR report (RFC6679).

In RFC6679: the receiver reports the cumulative lost packets and
cumulative ECN counters. So at the sender, there may not be
sufficient information to say if the ECN-CE packets were
accompanied by loss or not.

> Hm… what’s about using the (difference of )total number of losses instead?

As Colin indicated there is no way to know that the report was lost.

In (RFC3550), the fraction lost is defined to be the number of packets
lost divided by the number of packets expected since the last report.
And the formula in Section 4.3 does not rely on all receiving all
reports. Furthermore, the RTT and fraction lost plugged into the
formula are based on the same report.

Is the concern that the fraction lost may not be sufficiently
indicative if the preceding RTCP RR was lost?

-- 
Founder, CEO, callstats.io
http://www.callstats.io
Analytics and Optimizations for WebRTC.

We are hiring: www.callstats.io/jobs/

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 11:18:41 -0400

Hi Varun,

I would say that parts of tsvg-circuit-breaker are so general that they also apply for this doc. 

However, in any case my point it that (while congestion control may or may not threat ECN-CE marks similar as loss), circuit breaker MUST NOT do that because only ECN-CE marks without loss clearly shows that everything is working right and there is not need for a circuit breaker to react. That is really important to get right!

Mirja

> Am 05.05.2016 um 16:43 schrieb Varun Singh <[email-masked]>:
> 
> Hi Mirja,
> 
> see inline.
> 
> Regards,
> Varun
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 16:34:03 +0100

Hi Mirja,

Persistant ECN-CE marks indicates a flow is causing persistent congestion, no? 

Colin

> On 10 May 2016, at 16:18, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 
> Hi Varun,
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 16:32:47 +0100

> On 4 May 2016, at 11:52, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 
> Hi Colin,
> 
> see below. 
> 
> 

This would also have to update RFC 6679. I haven’t see it raised in AVTCORE or RMCAT.

> Further I disagree that a circuit breaker is a form of congestion control. It’s a last resort but there seem to be not much of a control loop involved here, making two very different thing with dif…

The circuit breaker has a control loop: it reacts to congestion feedback by reducing the offered load. I agree it’s much slower than typical congestion control, but it responds to the same type of congestion signal.

>      measured, since this better reflects the impact of persistent

I agree that makes sense in a TCP-like congestion control, where loss is the primary congestion signal. I’m not sure it’s right for real-time flows, where one of the primary goals is keeping latency down. Real-time flows that are causing persistent congestion ought to be stopped before they cause packet loss, if running in a network that uses ECN as the primary congestion signal.

> Hm… what’s about using the (difference of )total number of losses instead?

I think Varun answered this.

-- 
Colin Perkins
https://csperkins.org/

---

From: Varun Singh <[email-masked]>
Date: Tue, 10 May 2016 18:55:08 +0300

Hi Mirja,

> However, in any case my point it that (while congestion control may or may not threat ECN-CE marks similar as loss), circuit breaker MUST NOT do that because only ECN-CE marks without loss clearly…

I am slightly confused by the above statement: is the guidance above
meant for a media sender or media receiver?

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 16:21:11 +0100

Hi Mirja,

Which draft? 

Colin

> On 4 May 2016, at 12:42, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 12:07:07 -0400

Actually no, because the stable state of operation might cause a constant high level of markings (whereas you don’t want to have a high constant rate of loss).

Constantly increasing marking level might be an indication for congestion.

Mirja

> Am 10.05.2016 um 11:34 schrieb Colin Perkins <[email-masked]>:
> 
> Hi Mirja,
>

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 12:14:34 -0400

Currently, it's the second part of 

https://datatracker.ietf.org/doc/draft-welzl-irtf-iccrg-tcp-in-udp/

However, this was discussed at the last tsvwg meeting:

https://www.ietf.org/proceedings/95/slides/slides-95-tsvwg-13.pdf

Maybe also check the minutes. There was no clear decision for this but I believe the plan is to submit a separate draft for the updating part and then discuss adoption. There was definitely interest in this work!

Mirja

> Am 10.05.2016 um 11:21 schrieb Colin Perkins <[email-masked]>:
> 
> Hi Mirja,
> 
> Which draft? 
> 
> Colin
> 
> 
> 
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 17:54:50 +0100

Do you mean draft-khademi-alternativebackoff-ecn?

The ABE work makes a lot of sense for TCP, but it’s tied to the way TCP congestion control interacts with small buffers, and I don’t recall any evaluation for other congestion control algorithms. Did I miss something? That draft doesn’t reference RFC 6679, or any of the RMCAT algorithms.

Colin

> On 10 May 2016, at 17:14, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 13:00:16 -0400

Sorry for the confusion. Yes currently draft-khademi-alternativebackoff-ecn. And I’m talking only about the second part - section 3. This will go into a separate doc. I also expect that the proposed text will change a lot than what’s currently in there as there was already quite a bit discuss in tsvwg last time.

Mirja

> Am 10.05.2016 um 12:54 schrieb Colin Perkins <[email-masked]>:
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 18:04:38 +0100

But that’s TCP specific, and we’re talking about non-TCP flows using ECN. 

Colin

> On 10 May 2016, at 18:00, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 13:25:14 -0400

Actually no. It’s an update to RFC3168 which is also about ECN in general over IP. I just remembered that the the change of the following sentence is missing in the draft (but it’s in the slides) - section 5 in RFC3168:

"Upon the receipt by an ECN-Capable transport of a single CE packet,
   the congestion control algorithms followed at the end-systems MUST be
   essentially the same as the congestion control response to a *single*
   dropped packet.“

That’s what we want to change.

Mirja

> Am 10.05.2016 um 13:04 schrieb Colin Perkins <[email-masked]>:
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 22:10:11 +0100

Mirja,

The ABE work motivates making that change for TCP. The RMCAT work on SCReAM and NADA might also do so for RTP flows, but draft-khademi-alternative-backoff-ecn doesn’t, and it’s not clear that TSVWG has considered the broader issues. Changing this would require an update to RFC 6679, in particular section 7.3.3 which imposes requirements on future changes to the ECN response for RTP flows, as well as RFC 3168.

It’s also not clear how such a change would affect the RTP circuit breaker draft. It says that ECN-CE marked packets “SHOULD BE treated as lost”. That seems like the right default, since this needs to work with proprietary congestion control that treats ECN-CE like loss, and since IETF hasn’t yet published any RTP congestion control algorithms that treat ECN-CE differently. I guess we could say “SHOULD BE treated as lost unless an alternative response is specified in some future Standards Track congestion control algorithm that updates this memo”, but that seems more something that should be specified in that future specification than here.

Colin

> On 10 May 2016, at 18:25, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 23:08:51 +0100

Hi,

> On 10 May 2016, at 22:52, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 
> I thought that rfc6679 was already written or at least reviewed with the notion that CE marks and loss are not equivalent (even though threaten like this most often). A quick look at the doc  also gives me the impression that there is no update of rfc6679 needed. However, this should probably be discussed on the tsvwg list. Feel free to raise the issue there.

Section 7.3.3 of RFC 6679 says:

   The reception of RTP packets with ECN-CE marks in the IP header is a
   notification that congestion is being experienced.  The default
   reaction on the reception of these ECN-CE-marked packets MUST be to
   provide the congestion control algorithm with a congestion
   notification that triggers the algorithm to react as if packet loss
   had occurred.  There should be no difference in congestion response
   if ECN-CE marks or packet drops are detected.

   Other reactions to ECN-CE may be specified in the future, following
   IETF Review.  Detailed designs of such alternative reactions MUST be
   specified in a Standards Track RFC and be reviewed to ensure they are
   safe for deployment under any restrictions specified.  A potential
   example for an alternative reaction could be emergency communications
   (such as that generated by first responders, as opposed to the
   general public) in networks where the user has been authorised.  A
   more detailed description of these other reactions, as well as the
   types of congestion control algorithms used by end-nodes, is outside
   the scope of this document.

I read that as allowing other reactions, but such need to be documented and updated RFC 6679.

> I still think, actually independent of what we do with rfc3168, that "SHOULD BE treated as lost“ is the wrong thing to say. Not because congestion control might change in future but because the way AQMs signal CE is even less well defined. Further as I said, there is a different between circuit breakers and low time-scale congestion control and for circuit breaker and therefore there must be a difference between loss and CE marks (as reflected correctly in ietf-tsvwg-circuit-breaker).

This forces queue overflows and high latency before the circuit breaker triggers, rather than allowing it to trigger on persistent congestion signals from the AQM that can avoid inducing latency for other flows. I’m not sure that’s the right trade-off, especially if the AQM behaviour is not well defined. 

-- 
Colin Perkins
https://csperkins.org/

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 17:52:38 -0400

Hi Colin,

Mirja

> Am 10.05.2016 um 17:10 schrieb Colin Perkins <[email-masked]>:
> 
> Mirja,
>

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 18:26:57 -0400

Hi Colin,

> Am 10.05.2016 um 18:08 schrieb Colin Perkins <[email-masked]>:
> 
> Hi,
> 

An update might be needed or not (also depends on what we finally end up with). However, this needs to go on the tsvwg list.

> This forces queue overflows and high latency before the circuit breaker triggers, rather than allowing it to trigger on persistent congestion signals from the AQM that can avoid inducing latency fo…

The term we ended up using in ietf-tsvwg-circuit-breaker is "persistent excessive congestion“ and further "This is a safety measure to prevent starvation of network resources
   denying other flows from access to the Internet.  [...]  Avoiding persistent excessive
   congestion is important to reduce the potential for "Congestion
   Collapse" [RFC2914].“

So persistent congestion in not the problem but excessive congestion (that might even have an increasing trend). And further for ECN the persistent congestion level can be quite high, even without causing the queue to fill up but maintaining a low queuing delay level. Therefore ECN needs clearly to be treated differently than loss.

Mirja

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 May 2016 23:49:17 +0100

> On 10 May 2016, at 23:26, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 
> Hi Colin,
> 
>   Collapse” [RFC2914].“

I well understand this.

> So persistent congestion in not the problem but excessive congestion (that might even have an increasing trend).

This is clearly stated in the circuit breaker draft.

> And further for ECN the persistent congestion level can be quite high, even without causing the queue to fill up but maintaining a low queuing delay level. Therefore ECN needs clearly to be treated differently than loss.

If you require the circuit breaker to only trigger on loss, not on ECN marks, this forces the queue to overflow before the problematic flow is stopped. That disrupts other traffic sharing the queue. The point of using ECN is to be able to stop flows causing persistent excessive congestion without the latency penalty of the queue overflow.

Colin

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 18:58:41 -0400

Hi Colin,

> Am 10.05.2016 um 18:49 schrieb Colin Perkins <[email-masked]>:
> 

I’m not objecting to use ECN as input signal (I personally still think it might not be needed, but, yes can be used). However, it must be treated differently than loss. You are free to propose a mechanism how to take CE into account (differently than counting it as loss), e.g. counting it separately and using a even higher threshold...

Mirja

---

From: Colin Perkins <[email-masked]>
Date: Wed, 11 May 2016 00:26:00 +0100

> On 10 May 2016, at 23:58, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 
> Hi Colin,
> 

I'm sorry, but I'm still not seeing the justification for this. The ABE work you cited earlier is TCP specific, and I don't recall any other discussion in TSV or RMCAT around alternative back-off for non-TCP flows. What am I missing?

Colin

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Tue, 10 May 2016 18:58:41 -0400

Hi Colin,

Mirja

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 11 May 2016 07:12:19 -0400

Hi Colin,

> Am 10.05.2016 um 19:26 schrieb Colin Perkins <[email-masked]>:
> 

This is not about the congestion control in the endpoint, this is about how you set the CE marking in there router and that’s the same for all flows that use IP.

Further, if your argument is that a high CE rate causes high delay (but no potentially no loss), you should not use the number of CE marking as an input for you circuit breaker but the measure end-to-end delay.

A high CE rate can be seen in ‚normaL‘ operation without loss and low delay if you use an AQM than has a low marking threshold and a high marking probability. You naturally would try to avoid such a config for loss-based AQM as a high loss rate is not helpful but there is no reason to not use such a config for ECN. Loss and CE marks are not the same!

Mirja

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 18 May 2016 11:03:53 +0200

Mirja and Colin,

We need to make progress on resolving this Discuss.

To my understanding where we stand are that Mirja thinks that Circuit 
breaker should not state that ECN CE marks should not result in the same 
response as a packet loss provides. Colin disagree and base that on the 
current in force Standards Track document (RFC3168).

 From my view, I do understand Mirja's view that this is likely not 
where we want to be in the future when it comes to response to ECN CE 
marks. However, at the same time, there is not yet a even a WG item for 
addressing this and changing the current ECN specification. There 
doesn't yet exist even a WG consensus on what the desired response to 
ECN CE marks are. Blocking the publication of the document for 
referencing what it the current in force specification is not a 
reasonable discuss in this case.

The suggestion that there should be another specified behaviour for ECN 
CE marks for the algorithm in circuit breakers is hard to perform 
without a IETF consensus for what the desired response to ECN CE marks is.

I also note that if circuit breakers would like to define another 
behaviour to ECN-CE marks, it would need to update the existing MUST in 
ECN specification. We can't simply ignore the MUST in RFC3168 simply 
because it doesn't suit us. Thus requiring the circuit breaker document 
to achieve the IETF consensus on the ECN change. This may very likely 
cause a minimal 1 year delay, more likely 2 years before we have an IETF 
consensus on this question. This for a document that is already have a 
significant missref queue behind it.

What I think is reasonable is to pave the way as good as possible for 
the future change. Do a much by reference to the current in force 
specification. But, I fear that we will not be able to avoid doing an 
update on circuit breaker when the new ECN-CE behaviour has been agreed.

Does this sound like a reasonable way forward?

Cheers

Magnus Westerlund
AVTCORE WG chair and document shepherd.

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 18 May 2016 11:59:29 +0200

Hi Magnus,

there are two places in RFC3168 where this is specified. One is in Section 6.1.2 "The TCP Sender“ which is solely about TCP and therefore does not apply in this case. The other one is in section 5. "Explicit Congestion Notification in IP“:

This is explicitly about congestion control. I argue that there is a big difference between circuit breaker and congestion control. Therefore from my point of view this does also not apply either.

However, there is draft-fairhurst-tsvwg-circuit-breaker that is already approved and in the publication process which says:

"An egress meter can also count ECN [RFC3168] congestion marks as
        a part of measurement of congestion, but in this case, loss MUST
        also be measured to provide a complete view of the level of
        congestion.“

This document is about Network Circuit Breakers where Colin and Varun argued that is also doesn’t apply. Btw. this doc also explains nicely why there is a big difference between congestion control and circuit breaker and givens reason why the reaction should be different here. To me this doc is much closer to closer to the rtp-circiut and I would recommend to apply the same assumption here.

However, assuming that none of these documents apply, I can only says that is it simply wrong to treat ECN and loss the same in a circuit breaker, because having only ECN markings and no loss (and no large additional delay) simply means everything works fine and there should for sure not be a circuit breaker stopping the transmission.

Mirja

P.S.: In any case the sentence 

"ECN-CE marked packets SHOULD be treated as if they were lost for the
   purposes of congestion control“

must be removed because this doc is about circuit breaker and should not specify any SHOULDs or MUSTs for congestion control. (This would also not work well with the current work in rmcat.)

> Am 18.05.2016 um 11:03 schrieb Magnus Westerlund <[email-masked]>:
> 
> Mirja and Colin,
> 
> To my understanding where we stand are that Mirja thinks that Circuit breaker should not state that ECN CE marks should not result in the same response as a packet loss provides. Colin disagree and base that on the current in force Standards Track document (RFC3168).
> 
> From my view, I do understand Mirja's view that this is likely not where we want to be in the future when it comes to response to ECN CE marks. However, at the same time, there is not yet a even a WG item for addressing this and changing the current ECN specification. There doesn't yet exist even a WG consensus on what the desired response to ECN CE marks are. Blocking the publication of the document for referencing what it the current in force specification is not a reasonable discuss in this case.
> 
> The suggestion that there should be another specified behaviour for ECN CE marks for the algorithm in circuit breakers is hard to perform without a IETF consensus for what the desired response to ECN CE marks is.
> 
> I also note that if circuit breakers would like to define another behaviour to ECN-CE marks, it would need to update the existing MUST in ECN specification. We can't simply ignore the MUST in RFC3168 simply because it doesn't suit us. Thus requiring the circuit breaker document to achieve the IETF consensus on the ECN change. This may very likely cause a minimal 1 year delay, more likely 2 years before we have an IETF consensus on this question. This for a document that is already have a significant missref queue behind it.
> 
> What I think is reasonable is to pave the way as good as possible for the future change. Do a much by reference to the current in force specification. But, I fear that we will not be able to avoid doing an update on circuit breaker when the new ECN-CE behaviour has been agreed.
>

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 18 May 2016 13:27:46 +0200

Hi Mirja,

Thanks for the clarifications on your view of the status of the discussion.

Den 2016-05-18 kl. 11:59, skrev Mirja Kuehlewind (IETF):
> Hi Magnus,
>
> there are two places in RFC3168 where this is specified. One is in
> Section 6.1.2 "The TCP Sender“ which is solely about TCP and
> therefore does not apply in this case. The other one is in section 5.
> "Explicit Congestion Notification in IP“:
>
> the congestion control algorithms followed at the end-systems MUST
> be essentially the same as the congestion control response to a
> *single* dropped packet.“
>
> This is explicitly about congestion control. I argue that there is a
> big difference between circuit breaker and congestion control.
> Therefore from my point of view this does also not apply either.

I don't share your view on this. I think circuit breaker is a term we 
created for a very coarse grained congestion control mechanism. I also 
note that the above defines the know semantics of what signal an ECN-CE 
marked packet provides. So from my perspective it does apply also to the 
CBs.

>
> However, there is draft-fairhurst-tsvwg-circuit-breaker that is
> already approved and in the publication process which says:
>
> "An egress meter can also count ECN [RFC3168] congestion marks as a
> part of measurement of congestion, but in this case, loss MUST also
> be measured to provide a complete view of the level of congestion.“

To my understanding this requirement (3) in Section 4 is fulfilled by 
the RTP CB. When using the RTP ECN mechanism you will on reporting 
interval level get both CE and loss indications. Thus both are metered 
by the egress, and that information is feed to the ingress. Thus, I 
don't consider part of Varun's argument to be valid.

I do note that nothing in that requirement discuss the relative 
weighting of the CE vs loss.

>
> This document is about Network Circuit Breakers where Colin and Varun
> argued that is also doesn’t apply. Btw. this doc also explains nicely
> why there is a big difference between congestion control and circuit
> breaker and givens reason why the reaction should be different here.
> To me this doc is much closer to closer to the rtp-circiut and I
> would recommend to apply the same assumption here.

 From my perspective, I do think it applies. However, although the TSVWG 
CB document is clear about the general need for longer timescales than 
1-RTT and higher trigger thresholds, I have failed to find anything that 
discusses the ECN vs loss difference that you are going for. Like the 
below text from Bullet 3 in Section 4:

         For tunnels,
         [ID-ietf-tsvwg-tunnel-congestion-feedback] describes a way to
         measure both loss and ECN-marking; these measurements could be
         used on a relatively short timescale to drive a congestion
         control response and/or aggregated over a longer timescale with
         a higher trigger threshold to drive a CB.  Subsequent bullet
         items in this section discuss the necessity of using a longer
         timescale and a higher trigger threshold.

I interpret this as general discussion of the difference between CC and 
CB, not loss vs ECN-CE.

>
> However, assuming that none of these documents apply, I can only says
> that is it simply wrong to treat ECN and loss the same in a circuit
> breaker, because having only ECN markings and no loss (and no large
> additional delay) simply means everything works fine and there should
> for sure not be a circuit breaker stopping the transmission.

The problem here is that based on current set of IETF specifications, I 
would argue that level of 90% ECN-CE marking over minute time-scales 
would be an indication of persistent congestion. Motivating that the 
circuit breaker would trigger. The reason for it to trigger is that the 
RTP flow with the CB implemented would get a very unfair rate compared 
to CE reacting flows.

I do understand that there are ongoing discussion and desire to develop 
the ECN bits to have a different meaning. But until that semantics are 
agreed on, I don't know what design criteria to have when defining a 
different weighting between CE and loss.

> "ECN-CE marked packets SHOULD be treated as if they were lost for
> the purposes of congestion control“
>
> must be removed because this doc is about circuit breaker and should
> not specify any SHOULDs or MUSTs for congestion control. (This would
> also not work well with the current work in rmcat.)
>

I agree with the reason for addressing the above sentence.

I think the sentence that actually are the heart of the Discuss is the 
following from circuit breaker:

    If an RTP sender has negotiated ECN
    support for an RTP session, and has successfully initiated ECN use on
    the path to the receiver [RFC6679], then ECN-CE marked packets SHOULD
    be treated as if they were lost when calculating if the congestion-
    based RTP circuit breaker (Section 4.3) has been met, unless the RTP
    implementation can determine that the ECN-CE marking on this path is
    not reliable.

Which is the one that applies to the CB actual treatment of ECN-CE marks.

I think there might be a point of scheduling a call about how to resolve 
this discuss.

Cheers

Magnus Westerlund

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 18 May 2016 14:17:52 +0200

Hi Magnus,

see one point below.

> Am 18.05.2016 um 13:27 schrieb Magnus Westerlund <[email-masked]>:
> 
> Hi Mirja,
> 
> I don't share your view on this. I think circuit breaker is a term we created for a very coarse grained congestion control mechanism. I also note that the above defines the know semantics of what signal an ECN-CE marked packet provides. So from my perspective it does apply also to the CBs.
> 
>> 
> To my understanding this requirement (3) in Section 4 is fulfilled by the RTP CB. When using the RTP ECN mechanism you will on reporting interval level get both CE and loss indications. Thus both are metered by the egress, and that information is feed to the ingress. Thus, I don't consider part of Varun's argument to be valid.
> 
> I do note that nothing in that requirement discuss the relative weighting of the CE vs loss.
> 
>> 
> From my perspective, I do think it applies. However, although the TSVWG CB document is clear about the general need for longer timescales than 1-RTT and higher trigger thresholds, I have failed to find anything that discusses the ECN vs loss difference that you are going for. Like the below text from Bullet 3 in Section 4:
> 
>        For tunnels,
> I interpret this as general discussion of the difference between CC and CB, not loss vs ECN-CE.
> 
>> 
> The problem here is that based on current set of IETF specifications, I would argue that level of 90% ECN-CE marking over minute time-scales would be an indication of persistent congestion. Motivating that the circuit breaker would trigger. The reason for it to trigger is that the RTP flow with the CB implemented would get a very unfair rate compared to CE reacting flows.

My personal opinion is that I would not consider ECN at all. However my Discuss this that the doc simply should not say 

"ECN-CE marked packets SHOULD
   be treated as if they were lost when calculating if the congestion-
   based RTP circuit breaker“

There is currently no document that says that both signal have to be treated differently but there is also no document that defines that they have to be the same for circuit breakers. This doc would be the first one doing so. Given the currently knowledge we have and the current work we are doing, this is simply wrong.

However, I did not say that you cannot use ECN-CE as an input for a circuit breaker at all (if you really want to), just treat it not the same but different.

From my personal point of view there are two options here:

1) have a separate counter for ECN-CE with a much higher threshold than for loss, or

2) make sure that there are not only ECN-CE markings but also loss (maybe with a certain minimum rate).

> 
> I do understand that there are ongoing discussion and desire to develop the ECN bits to have a different meaning. But until that semantics are agreed on, I don't know what design criteria to have when defining a different weighting between CE and loss.
> 
>> 
>> Mirja
>> 
> I think the sentence that actually are the heart of the Discuss is the following from circuit breaker:
> 
> I think there might be a point of scheduling a call about how to resolve this discuss.

If none of the two solutions (or a third one that you come up with) is acceptable for you, then a call would be good. Unfortunately, I’m more than booked the next 1.5 weeks. I could maybe do tomorrow after the telechat (whenever that is) or on Monday May 30…

Mirja

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 18 May 2016 14:55:37 +0200

Hi Mirja,

I will let the authors and anyone else who want to provide input do so 
now. From my perspective we clarified the positions and possibilities a 
bit more by this exchange.

Den 2016-05-18 kl. 14:17, skrev Mirja Kuehlewind (IETF):

> My personal opinion is that I would not consider ECN at all. However
> my Discuss this that the doc simply should not say
>
> "ECN-CE marked packets SHOULD be treated as if they were lost when
> calculating if the congestion- based RTP circuit breaker“
>
> There is currently no document that says that both signal have to be
> treated differently but there is also no document that defines that
> they have to be the same for circuit breakers. This doc would be the
> first one doing so. Given the currently knowledge we have and the
> current work we are doing, this is simply wrong.

One question that will arise if one remove ECN-CE marks as input, that 
is if the RTP-CB is at all applicable to be used in RTP sessions where 
one have negotiated ECN Capable Transport (ECT). I guess you would say 
yes, based on the arguments so far has brought up. From my perspective, 
I would have to say questionable, as this could result in that the RTP 
flow would have the possibility in some cases to cause persistent 
congestion, without ever having the possibility to trigger for this.

So a possibility could be to remove ECN-CE reaction and at the same time 
reduce the applicability of the specification. Then address this in the 
future when we know what the updated ECN-CE semantics is.

>
> However, I did not say that you cannot use ECN-CE as an input for a
> circuit breaker at all (if you really want to), just treat it not the
> same but different.

And I don't see how we can arrive at such an answer prior to updated 
semantics for the ECN-CE has been defined. Then you can determine what 
is reasonable response for a CB. From my perspective we can't have this 
circuit breaker wait for such work.

> (maybe with a certain minimum rate).

The issue is to set the parameters for these in a correct (or even 
mostly working) way. We have no way of knowing if these would work well 
with the future semantics.

Cheers

Magnus Westerlund

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 18 May 2016 16:49:21 +0200

Hi Magnus,

I have to add one more point here. See below.

> Am 18.05.2016 um 14:55 schrieb Magnus Westerlund <[email-masked]>:
> 
> Hi Mirja,
> 
> I will let the authors and anyone else who want to provide input do so now. From my perspective we clarified the positions and possibilities a bit more by this exchange.
> 
> One question that will arise if one remove ECN-CE marks as input, that is if the RTP-CB is at all applicable to be used in RTP sessions where one have negotiated ECN Capable Transport (ECT). I guess you would say yes, based on the arguments so far has brought up. From my perspective, I would have to say questionable, as this could result in that the RTP flow would have the possibility in some cases to cause persistent congestion, without ever having the possibility to trigger for this.
> 
> So a possibility could be to remove ECN-CE reaction and at the same time reduce the applicability of the specification. Then address this in the future when we know what the updated ECN-CE semantics is.

Removing ECN, would basically mean you cannot use ECN with most RTP application because you really should implement a CB. That would be not a great solution from point of view.

However, there good arguments for e.g. using a higher threshold for ECN. As a side note I have to say that the value for loss (k=10) is also rather random, there the value itself for ECN can only be random as well (but should be higher).

Then first of all, all networks try to minimize loss as loss is basically lost transmission capacity. There is no such an incentive to minimize ECN marks. The usually assumption we can make today is that most loss-based queues are droptail, while whenever you decide to enable ECN you have to use an AQM scheme with a lower threshold then the maximum queue size. In this situation you will see more marking than you have seen losses with droptail.

Then even if you use the same AQM for loss and ECN, you usually see more ECN markings than losses, because we a packet is dropped the queue shrinks, while a marking keeps the queue growing and causes more markings. This is less a problem for congestion control, as it only reacts once per RTT. However this doc says that the actual rate of losses or ECN marks should be considered.

So just given the way ECN works, it’s simply different from loss. This difference is not that important for congestion control. But the way it is used in this doc, it is.

Finally, I really would like to avoid a situation, where, as ECN is basically not enable in the network right now, an operator decides to enable ECN and suddenly all WebRTC connections crash. This would be a strong disincentive to deploy ECN while we are working very hard on finally deploying it, especially as services like WebRTC could really benefit from it.

Mirja

> And I don't see how we can arrive at such an answer prior to updated semantics for the ECN-CE has been defined. Then you can determine what is reasonable response for a CB. From my perspective we can't have this circuit breaker wait for such work.
> 
>> 
> The issue is to set the parameters for these in a correct (or even mostly working) way. We have no way of knowing if these would work well with the future semantics.
> 
> Cheers
> 
> Magnus Westerlund
>

---

From: Colin Perkins <[email-masked]>
Date: Wed, 18 May 2016 17:44:31 +0200

Apologies for the slow response - travel and exam season.

> On 18 May 2016, at 11:59, Mirja Kuehlewind (IETF) <[email-masked]> wrote:
> 
> Hi Magnus,
> 

There is also RFC 6679 section 7.3.3:

> This is explicitly about congestion control. I argue that there is a big difference between circuit breaker and congestion control. Therefore from my point of view this does also not apply either.

I’m sorry, but I do not agree with this viewpoint. The circuit breaker has been explicitly designed to respond to the same signals as used by the congestion control algorithms, acting as a very coarse-grained form of congestion control.

> This document is about Network Circuit Breakers where Colin and Varun argued that is also doesn’t apply. Btw. this doc also explains nicely why there is a big difference between congestion control…

The RTP circuit breaker does also measure loss, so I don’t understand the issue here.

> However, assuming that none of these documents apply, I can only says that is it simply wrong to treat ECN and loss the same in a circuit breaker, because having only ECN markings and no loss (and…

I’m not sure I agree. Persistent excessive congestion reported by ECN-CE marks ought to trigger the RTP circuit breaker. Otherwise, we require the network to be driven into overload, with queue overflow and high latency, and the implied latency hit to other flows sharing the queue, before we can stop a persistently misbehaving flow.

> must be removed because this doc is about circuit breaker and should not specify any SHOULDs or MUSTs for congestion control. (This would also not work well with the current work in rmcat.)

Firstly, I disagree that this document is not about congestion control.

Secondly, there are already RFCs that give requirements for congestion response to ECN-CE marks. This document is consistent with those RFCs.

Finally, it’s not clear what other reaction this document could specify, since there is no consensus on how RTP congestion response to an ECN-CE mark should differ from response to loss. The TSV drafts you referenced earlier are specific to TCP, and don’t apply to RTP flows, and the RMCAT candidate congestion control algorithms differ in how they respond to CE marks, and motivate their choice by QoE requirements, not by differences in congestion marking behaviour.

The RTP circuit breaker is consistent with RFCs 3168 and 6679, there are no working group drafts in progress to update those RFCs, and there is no consensus on what any future update would say. Getting consensus on a new recommendation to update RFCs 3168, 6679, and the RMCAT drafts, will take some considerable time. I don’t think it appropriate to block this draft until then. Rather, this draft should be published, and updated along with the other documents if there is a future consensus on an alternate response.

Colin

-- 
Colin Perkins
https://csperkins.org/

---

From: "Ben Campbell" <[email-masked]>
Date: Wed, 18 May 2016 20:54:21 -0500

On 18 May 2016, at 7:17, Mirja Kuehlewind (IETF) wrote:

> If none of the two solutions (or a third one that you come up with) is 
> acceptable for you, then a call would be good. Unfortunately, I’m 
> more than booked the next 1.5 weeks. I could maybe do tomorrow after 
> the telechat (whenever that is) or on Monday May 30…

As the shepherding AD, I strongly encourage a call. We really need to 
converge on this.

Unfortunately, I am booked for both of the times Mirja proposes (I have 
another call after the telechat, and May 30 is a US holiday.) But if 
others can make one of those times, please proceed without me.

Ben.

---

From: Alissa Cooper <[email-masked]>
Date: Wed, 18 May 2016 20:10:48 -0700

> On May 18, 2016, at 6:54 PM, Ben Campbell <[email-masked]> wrote:
> 
> As the shepherding AD, I strongly encourage a call. We really need to converge on this.
> 
> Unfortunately, I am booked for both of the times Mirja proposes (I have another call after the telechat, and May 30 is a US holiday.) But if others can make one of those times, please proceed without me.

Why not use the next informal telechat slot?

Alissa

> 
> Ben.
> 
> _______________________________________________
> Audio/Video Transport Core Maintenance
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avt

---

From: "Ben Campbell" <[email-masked]>
Date: Wed, 18 May 2016 22:43:59 -0500

On 18 May 2016, at 22:10, Alissa Cooper wrote:

>>> If none of the two solutions (or a third one that you come up with) 
>>> is acceptable for you, then a call would be good. Unfortunately, 
>>> I’m more than booked the next 1.5 weeks. I could maybe do tomorrow 
>>> after the telechat (whenever that is) or on Monday May 30…
>>
>> Unfortunately, I am booked for both of the times Mirja proposes (I 
>> have another call after the telechat, and May 30 is a US holiday.) 
>> But if others can make one of those times, please proceed without me.
>

Ah, excellent idea. Mirja: do you plan to be on the next informal?
Colin, Magnus: could you make that? (Thu, May 26 1000 us/eastern)

---

From: Colin Perkins <[email-masked]>
Date: Thu, 19 May 2016 09:15:25 +0200

> On 19 May 2016, at 05:43, Ben Campbell <[email-masked]> wrote:

I can’t do that time, but I think we’ve converged on a call on 8 June.

-- 
Colin Perkins
https://csperkins.org/

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Thu, 19 May 2016 09:46:13 +0200

> Am 19.05.2016 um 09:15 schrieb Colin Perkins <[email-masked]>:
> 

I will also not be in the next informal...