Subject: [tsvwg] Experience and Status of ECN in QUIC work

From: Magnus Westerlund <[email-masked]>
Date: Thu, 5 Jul 2018 14:40:58 +0200

TSVWG,

ECN support was added to QUIC in the latest draft versions (-13). The 
relevant documents that include ECN related text are:

https://datatracker.ietf.org/doc/draft-ietf-quic-transport/
https://datatracker.ietf.org/doc/draft-ietf-quic-recovery/

The inclusion of ECN in QUIC can be summarized as. Each sender performs 
initial verification when the connection is established or migrated by 
setting ECT on the packets it send. The receiver will report if it sees 
ECN marks, i.e. any value other than not-ECT, using a new 
acknowledgement frame type (ACK_ECN) that includes counters of how many 
packets with the various markings the receiver have seen since 
connection establishment. If the receiver gets not-ECT or can't read ECN 
marks then it uses the ACK frame to report these packets so that the 
sender can learn of the lack of ECT capability and turn its marking off. 
Marking consistency are continuously verified. ECN black hole mitigation 
is discussed and a possible mechanism is that any RTO will result in 
retransmission without ECT marks.

The inclusion of ECN in QUIC has so far been fairly well discussed. 
Initially by the design team. Then at the QUIC interim meeting in June, 
followed by a lot of discussion of the actual text proposal on github. 
Thanks to everyone that engaged. Based on these discussions there are 
some questions and experience that I have noticed that is worth 
spreading and discussing. This is also the topics coming in the TSVWG 
session for my agenda item.

Also I don't believe the last word is said about details of how ECN in 
QUIC is realized. There are currently three different github issues 
related to ECN:

- Improve ACK_ECN frame encoding (e.g., use bit-vector) 
<https://github.com/quicwg/base-drafts/issues/1439> 
(https://github.com/quicwg/base-drafts/issues/1439)
- Fragility in ECN counters with lost acks 
<https://github.com/quicwg/base-drafts/issues/1481> 
(https://github.com/quicwg/base-drafts/issues/1481)
- Detect adversarial ECN reporting 
<https://github.com/quicwg/base-drafts/issues/1426> 
(https://github.com/quicwg/base-drafts/issues/1426)

So the things I want to bring to attention are:

A. Suppression of ECN values in packet duplicates

In QUIC the drafts currently says that a receiver should suppress the 
ECN values received in any packet duplicates. The main reason for that 
is to mitigate a potential active attack by a off-path attacker that has 
some capability to receive a copy of the packets of the target flow. We 
call it an on-the-side attack. By taking a copy of a legit packet, and 
changing the ECN field to ECN-CE and then forward the modified copy to 
the intended receiver one can perform a denial of service attack by 
driving down the congestion window. To mitigate this, the recommendation 
is to care about only the first arrived packet. This forces an 
on-the-side attacker to be able to race their attack packets with the 
original packet.

The downside of this is of course that an actual network duplication 
where none-first packets gets marked as ECN-CE after the duplication we 
ignore the congestion signal. My personal standpoint is that this is 
okay. If not the first packet was marked then congestion may not be so 
bad that it is a problem if the duplication don't happens. Secondly, if 
it is any persistent congestion then this flow will be marked in a later 
packet. There was some disagreement on this so I think it is worth more 
discussion.

B. Will ECT(0) and ECT(1) be mixed in one packet flow?

In the discussions around optimizing how the ECN information reported 
from receiver to sender this question arose. If one can make assumption 
that within a connection one will use only one of the ECT codepoints 
then we can optimize the reporting based on that assumption. So how safe 
is this assumption? And in this case, what is the importance of being 
able to detect ECT codepoint remarking, i.e. changing between 0 and 1 in 
either direction?

C. Detecting Cheating Receivers

As there are some risks in certain deployment situations that the 
receiver would have an incentive to lie about if an ECN-CE mark was 
received or not to gain throughput. A sender could detect this case by 
sending packets marked as ECN-CE to verify that receiver reports these 
packets are being marked. As the sender will ignore the ECN-CE marks it 
self generated there is a risk that these marks hide real CE marks. 
Thus, the general question is how often, at most, should a sender mark 
with ECN-CE?

D. Delayed Acknowledgement and ECN

We also have an interesting discussion around ECN, Delayed 
Acknowledgement and the interaction with recovery period. QUIC supports 
delayed acknowledgement. So far there are quite a lot of implementer 
freedom of how to use this. What is currently specified is that when 
receiving an ECN-CE mark the receiver sends an immediate ACK, so get 
rapid response to the congestion signal. When the ACK_ECN reaches the 
sender it will go into Recovery until a packet sent after entering 
recovery has been acknowledged. During recovery as the congestion window 
already is reduced and frozen the reception of additional ECN-CE marks 
(at least for the current congestion control mechanism) is not that 
important. What is important is if any packets are ECN-CE marked when 
exiting recovery. Marks on packets sent after recovery should result in 
a new congestion event and new recovery period. To resolve this the 
current requirement is to immediately acknowledge each ECN-CE marks. 
This will result in some additional acknowledgements.

There has been some discussion if one want to optimize this case or if 
it is not worth it. One direction was to ensure that even with delayed 
ACKs one get frequent enough acknowledgements. This resulted in 
discussion in the direction of ensuring that the longest delay before 
acknowledging is caped at a quarter of the RTT. That way one achieve no 
worse than quarter RTT resolution on the ECN-CE marks. Another potential 
direction is explicit indication of which packet numbers that would 
resolve this uncertainty. Then also exists the question if the receiver 
needs any indication of recovery periods to optimize the acknowledgements.

E. Utility of detailed CE information

As there has been discussion about explicitly indicate which packet 
numbers was marked as CE, it is also good to understand if there are any 
significant utility of having this information for the sender compared 
to what the counters provide. I assume this goes into the potential for 
congestion control mechanism evolution. But, does it matter for example 
for L4S?

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Network Architecture & Protocols, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Black, David" <[email-masked]>
Date: Fri, 6 Jul 2018 22:54:41 +0000

Hi Magnus,
<WG Chair Hat = OFF>

> In the discussions around optimizing how the ECN information reported from receiver to sender this question arose.
> If one can make assumption that within a connection one will use only one of the ECT codepoints then we can optimize
> the reporting based on that assumption. So how safe is this assumption? And in this case, what is the importance of
> being able to detect ECT codepoint remarking, i.e. changing between 0 and 1 in either direction?
RFC 8311 currently prohibits use of ECT(1) outside of experiments.  The L4S experiment is the current known use of ECT(1), and it only uses one ECT codepoint per flow.

If that QUIC optimization is valuable, it may be reasonable to go ahead with it, leaving its consequences for possible future experiments to be dealt with by the possible future experimenters.

In addition ...

> C. Detecting Cheating Receivers
This is not a new concern - RFC 8311's discussion of the ECN Nonce (RFC 3540) and the rationale for obsoleting it is relevant to this topic.

</WG Chair Hat = OFF>

Thanks, --David

From: tsvwg [mailto:[email-masked]] On Behalf Of Magnus Westerlund
Sent: Thursday, July 5, 2018 8:41 AM
To: tsvwg IETF list
Subject: [tsvwg] Experience and Status of ECN in QUIC work

TSVWG,

ECN support was added to QUIC in the latest draft versions (-13). The relevant documents that include ECN related text are:

The inclusion of ECN in QUIC can be summarized as. Each sender performs initial verification when the connection is established or migrated by setting ECT on the packets it send. The receiver will report if it sees ECN marks, i.e. any value other than not-ECT, using a new acknowledgement frame type (ACK_ECN) that includes counters of how many packets with the various markings the receiver have seen since connection establishment. If the receiver gets not-ECT or can't read ECN marks then it uses the ACK frame to report these packets so that the sender can learn of the lack of ECT capability and turn its marking off. Marking consistency are continuously verified. ECN black hole mitigation is discussed and a possible mechanism is that any RTO will result in retransmission without ECT marks.

The inclusion of ECN in QUIC has so far been fairly well discussed. Initially by the design team. Then at the QUIC interim meeting in June, followed by a lot of discussion of the actual text proposal on github. Thanks to everyone that engaged. Based on these discussions there are some questions and experience that I have noticed that is worth spreading and discussing. This is also the topics coming in the TSVWG session for my agenda item.

Also I don't believe the last word is said about details of how ECN in QUIC is realized. There are currently three different github issues related to ECN:

- Improve ACK_ECN frame encoding (e.g., use bit-vector)<https://github.com/quicwg/base-drafts/issues/1439> (https://github.com/quicwg/base-drafts/issues/1439)
- Fragility in ECN counters with lost acks<https://github.com/quicwg/base-drafts/issues/1481> (https://github.com/quicwg/base-drafts/issues/1481)
- Detect adversarial ECN reporting<https://github.com/quicwg/base-drafts/issues/1426> (https://github.com/quicwg/base-drafts/issues/1426)

In QUIC the drafts currently says that a receiver should suppress the ECN values received in any packet duplicates. The main reason for that is to mitigate a potential active attack by a off-path attacker that has some capability to receive a copy of the packets of the target flow. We call it an on-the-side attack. By taking a copy of a legit packet, and changing the ECN field to ECN-CE and then forward the modified copy to the intended receiver one can perform a denial of service attack by driving down the congestion window. To mitigate this, the recommendation is to care about only the first arrived packet. This forces an on-the-side attacker to be able to race their attack packets with the original packet.

The downside of this is of course that an actual network duplication where none-first packets gets marked as ECN-CE after the duplication we ignore the congestion signal. My personal standpoint is that this is okay. If not the first packet was marked then congestion may not be so bad that it is a problem if the duplication don't happens. Secondly, if it is any persistent congestion then this flow will be marked in a later packet. There was some disagreement on this so I think it is worth more discussion.

In the discussions around optimizing how the ECN information reported from receiver to sender this question arose. If one can make assumption that within a connection one will use only one of the ECT codepoints then we can optimize the reporting based on that assumption. So how safe is this assumption? And in this case, what is the importance of being able to detect ECT codepoint remarking, i.e. changing between 0 and 1 in either direction?

As there are some risks in certain deployment situations that the receiver would have an incentive to lie about if an ECN-CE mark was received or not to gain throughput. A sender could detect this case by sending packets marked as ECN-CE to verify that receiver reports these packets are being marked. As the sender will ignore the ECN-CE marks it self generated there is a risk that these marks hide real CE marks. Thus, the general question is how often, at most, should a sender mark with ECN-CE?

We also have an interesting discussion around ECN, Delayed Acknowledgement and the interaction with recovery period. QUIC supports delayed acknowledgement. So far there are quite a lot of implementer freedom of how to use this. What is currently specified is that when receiving an ECN-CE mark the receiver sends an immediate ACK, so get rapid response to the congestion signal. When the ACK_ECN reaches the sender it will go into Recovery until a packet sent after entering recovery has been acknowledged. During recovery as the congestion window already is reduced and frozen the reception of additional ECN-CE marks (at least for the current congestion control mechanism) is not that important. What is important is if any packets are ECN-CE marked when exiting recovery. Marks on packets sent after recovery should result in a new congestion event and new recovery period. To resolve this the current requirement is to immediately acknowledge each ECN-CE marks. This will result in some additional acknowledgements.

There has been some discussion if one want to optimize this case or if it is not worth it. One direction was to ensure that even with delayed ACKs one get frequent enough acknowledgements. This resulted in discussion in the direction of ensuring that the longest delay before acknowledging is caped at a quarter of the RTT. That way one achieve no worse than quarter RTT resolution on the ECN-CE marks. Another potential direction is explicit indication of which packet numbers that would resolve this uncertainty. Then also exists the question if the receiver needs any indication of recovery periods to optimize the acknowledgements.

As there has been discussion about explicitly indicate which packet numbers was marked as CE, it is also good to understand if there are any significant utility of having this information for the sender compared to what the counters provide. I assume this goes into the potential for congestion control mechanism evolution. But, does it matter for example for L4S?

Cheers

Magnus Westerlund

SE-164 80 Stockholm, Sweden | mailto: [email-masked]<mailto:[email-masked]>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 13 Jul 2018 12:25:50 -0500

Responsible AD hat is even more off than David's WG chair hat in his
response ...

On Thu, Jul 5, 2018 at 7:41 AM Magnus Westerlund <
[email-masked]> wrote:

> setting ECT on the packets it send. The receiver will report if it sees ECN
> marks, i.e. any value other than not-ECT, using a new acknowledgement frame
> type (ACK_ECN) that includes counters of how many packets with the various
> markings the receiver have seen since connection establishment. If the
> receiver gets not-ECT or can't read ECN marks then it uses the ACK frame to
> report these packets so that the sender can learn of the lack of ECT
> capability and turn its marking off. Marking consistency are continuously
> verified. ECN black hole mitigation is discussed and a possible mechanism
> is that any RTO will result in retransmission without ECT marks.
>
> Thanks to everyone that engaged. Based on these discussions there are some
> questions and experience that I have noticed that is worth spreading and
> discussing. This is also the topics coming in the TSVWG session for my
> agenda item.
>
I'd like to thank the QUIC-ECN design team for surfacing this in TSVWG at
this time.

> QUIC is realized. There are currently three different github issues related
> to ECN:
>
> <https://github.com/quicwg/base-drafts/issues/1439> (
> https://github.com/quicwg/base-drafts/issues/1439)
> <https://github.com/quicwg/base-drafts/issues/1481> (
> https://github.com/quicwg/base-drafts/issues/1481)
> <https://github.com/quicwg/base-drafts/issues/1426> (
> https://github.com/quicwg/base-drafts/issues/1426)
>
> In QUIC the drafts currently says that a receiver should suppress the ECN
> values received in any packet duplicates. The main reason for that is to
> mitigate a potential active attack by a off-path attacker that has some
> capability to receive a copy of the packets of the target flow. We call it
> an on-the-side attack. By taking a copy of a legit packet, and changing the
> ECN field to ECN-CE and then forward the modified copy to the intended
> receiver one can perform a denial of service attack by driving down the
> congestion window. To mitigate this, the recommendation is to care about
> only the first arrived packet. This forces an on-the-side attacker to be
> able to race their attack packets with the original packet.
>
> The downside of this is of course that an actual network duplication where
> none-first packets gets marked as ECN-CE after the duplication we ignore
> the congestion signal. My personal standpoint is that this is okay. If not
> the first packet was marked then congestion may not be so bad that it is a
> problem if the duplication don't happens. Secondly, if it is any persistent
> congestion then this flow will be marked in a later packet. There was some
> disagreement on this so I think it is worth more discussion.
>
>
> In the discussions around optimizing how the ECN information reported from
> receiver to sender this question arose. If one can make assumption that
> within a connection one will use only one of the ECT codepoints then we can
> optimize the reporting based on that assumption. So how safe is this
> assumption? And in this case, what is the importance of being able to
> detect ECT codepoint remarking, i.e. changing between 0 and 1 in either
> direction?
>
>
> As there are some risks in certain deployment situations that the receiver
> would have an incentive to lie about if an ECN-CE mark was received or not
> to gain throughput. A sender could detect this case by sending packets
> marked as ECN-CE to verify that receiver reports these packets are being
> marked. As the sender will ignore the ECN-CE marks it self generated there
> is a risk that these marks hide real CE marks. Thus, the general question
> is how often, at most, should a sender mark with ECN-CE?
>
In my mind, A and C are related - how perfectly does a sender need to react
perfectly to ECN-CE markings?

In A, I think that may be where Magnus is coming from - the sender is
already reacting to ECN-CE, and if the sender hasn't responded
sufficiently, the flow will continue to be marked in later packets, and the
sender will have an opportunity to react again, until it has reacted
sufficiently. Is that the question?

In C, I'm trying to understand how much a receiver can benefit by cheating.
ISTM that if a receiver sees ECN-CE markings and doesn't reflect that back
to the sender, one of two things will happen.

   - Either the sender really doesn't need to slow down (so no further
   ECN-CE marks arrive at the receiver), or
   - The sender really does need to slow down (so the receiver keeps
   cheating until actual packets get lost, which the receiver really can't
   disguise), and is finally slowing down based on packet losses

I understand that CoDel and PIE are Experimental, and I understand that L4S
is still in TSV, but does a receiver cheating now, and going forward, make
as much sense as it did when
https://tools.ietf.org/html/rfc3168#section-18(*) was published in 2001,
with congestion control mechanisms available at that time? It seems like
that's just begging for bufferbloat, followed by losses, followed by poor
performance during recovery.

Curiously,

Spencer, with no hats

(*) I understand that https://tools.ietf.org/html/rfc3168#section-18 is
discussing misbehaving routers, not misbehaving receivers, but that was a
handy reference for "what happens if the sender can't trust ECN signals" ...

> We also have an interesting discussion around ECN, Delayed Acknowledgement
> and the interaction with recovery period. QUIC supports delayed
> acknowledgement. So far there are quite a lot of implementer freedom of how
> to use this. What is currently specified is that when receiving an ECN-CE
> mark the receiver sends an immediate ACK, so get rapid response to the
> congestion signal. When the ACK_ECN reaches the sender it will go into
> Recovery until a packet sent after entering recovery has been acknowledged.
> During recovery as the congestion window already is reduced and frozen the
> reception of additional ECN-CE marks (at least for the current congestion
> control mechanism) is not that important. What is important is if any
> packets are ECN-CE marked when exiting recovery. Marks on packets sent
> after recovery should result in a new congestion event and new recovery
> period. To resolve this the current requirement is to immediately
> acknowledge each ECN-CE marks. This will result in some additional
> acknowledgements.
>
> There has been some discussion if one want to optimize this case or if it
> is not worth it. One direction was to ensure that even with delayed ACKs
> one get frequent enough acknowledgements. This resulted in discussion in
> the direction of ensuring that the longest delay before acknowledging is
> caped at a quarter of the RTT. That way one achieve no worse than quarter
> RTT resolution on the ECN-CE marks. Another potential direction is explicit
> indication of which packet numbers that would resolve this uncertainty.
> Then also exists the question if the receiver needs any indication of
> recovery periods to optimize the acknowledgements.
>
> significant utility of having this information for the sender compared to
> what the counters provide. I assume this goes into the potential for
> congestion control mechanism evolution. But, does it matter for example for
> L4S?
>
>
> Cheers
>
> Magnus Westerlund
>

---

From: "Black, David" <[email-masked]>
Date: Fri, 13 Jul 2018 20:05:21 +0000

WG chair hat still off …

> I'm trying to understand how much a receiver can benefit by cheating. ISTM that if a receiver sees ECN-CE markings and doesn't reflect that back to the sender, one of two things will happen.

  *   Either the sender really doesn't need to slow down (so no further ECN-CE marks arrive at the receiver), or
  *   The sender really does need to slow down (so the receiver keeps cheating until actual packets get lost, which the receiver really can't disguise), and is finally slowing down based on packet losses
It helps to think about behavior at the bottleneck queue that is generating the ECN-CE marks.   If all receivers cheat and ignore the marks, nobody wins because the bottleneck condition persists.   The cheating receiver wins if other receivers don’t cheat, and the resulting backoff of other flows causes the bottleneck queue to no longer be a bottleneck, i.e., it stops generating ECN-CE marks.   At that point the cheating receiver’s flow has more share of the formerly bottlenecked queue’s throughput than that flow should.

Thanks, --David

From: tsvwg [mailto:[email-masked]] On Behalf Of Spencer Dawkins at IETF
Sent: Friday, July 13, 2018 1:26 PM
To: Magnus Westerlund
Cc: [email-masked]
Subject: Re: [tsvwg] Experience and Status of ECN in QUIC work

Responsible AD hat is even more off than David's WG chair hat in his response ...

On Thu, Jul 5, 2018 at 7:41 AM Magnus Westerlund <[email-masked]<mailto:[email-masked]>> wrote:

TSVWG,

The inclusion of ECN in QUIC has so far been fairly well discussed. Initially by the design team. Then at the QUIC interim meeting in June, followed by a lot of discussion of the actual text proposal on github. Thanks to everyone that engaged. Based on these discussions there are some questions and experience that I have noticed that is worth spreading and discussing. This is also the topics coming in the TSVWG session for my agenda item.
I'd like to thank the QUIC-ECN design team for surfacing this in TSVWG at this time.

As there are some risks in certain deployment situations that the receiver would have an incentive to lie about if an ECN-CE mark was received or not to gain throughput. A sender could detect this case by sending packets marked as ECN-CE to verify that receiver reports these packets are being marked. As the sender will ignore the ECN-CE marks it self generated there is a risk that these marks hide real CE marks. Thus, the general question is how often, at most, should a sender mark with ECN-CE?
In my mind, A and C are related - how perfectly does a sender need to react perfectly to ECN-CE markings?

In A, I think that may be where Magnus is coming from - the sender is already reacting to ECN-CE, and if the sender hasn't responded sufficiently, the flow will continue to be marked in later packets, and the sender will have an opportunity to react again, until it has reacted sufficiently. Is that the question?

In C, I'm trying to understand how much a receiver can benefit by cheating. ISTM that if a receiver sees ECN-CE markings and doesn't reflect that back to the sender, one of two things will happen.

  *   Either the sender really doesn't need to slow down (so no further ECN-CE marks arrive at the receiver), or
  *   The sender really does need to slow down (so the receiver keeps cheating until actual packets get lost, which the receiver really can't disguise), and is finally slowing down based on packet losses
I understand that CoDel and PIE are Experimental, and I understand that L4S is still in TSV, but does a receiver cheating now, and going forward, make as much sense as it did when https://tools.ietf.org/html/rfc3168#section-18(*) was published in 2001, with congestion control mechanisms available at that time? It seems like that's just begging for bufferbloat, followed by losses, followed by poor performance during recovery.

Curiously,

Spencer, with no hats

(*) I understand that https://tools.ietf.org/html/rfc3168#section-18 is discussing misbehaving routers, not misbehaving receivers, but that was a handy reference for "what happens if the sender can't trust ECN signals" ...

Cheers

Magnus Westerlund

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 13 Jul 2018 16:09:14 -0500

Hi, David,

AD hat still off as well ...

On Fri, Jul 13, 2018 at 3:05 PM Black, David <[email-masked]> wrote:

> WG chair hat still off …
>
>
>
> > I'm trying to understand how much a receiver can benefit by cheating.
> It helps to think about behavior at the bottleneck queue that is
> generating the ECN-CE marks.   If all receivers cheat and ignore the marks,
> nobody wins because the bottleneck condition persists.   The cheating
> receiver wins if other receivers don’t cheat, and the resulting backoff of
> other flows causes the bottleneck queue to no longer be a bottleneck, i.e.,
> it stops generating ECN-CE marks.   At that point the cheating receiver’s
> flow has more share of the formerly bottlenecked queue’s throughput than
> that flow should.
>

That is helpful. Thank you.

I'm still curious about how long this advantage lasts, but I can easily
imagine that's not an answerable question in the general case.

I'll keep watching discussion on the mailing list, and during TSVWG next
week.

Spencer

>
>
> Thanks, --David
>
>
>
> *From:* tsvwg [mailto:[email-masked]] *On Behalf Of *Spencer
> Dawkins at IETF
> *Sent:* Friday, July 13, 2018 1:26 PM
> *To:* Magnus Westerlund
> *Cc:* [email-masked]
> *Subject:* Re: [tsvwg] Experience and Status of ECN in QUIC work
>
>
>
> In C, I'm trying to understand how much a receiver can benefit by
> cheating. ISTM that if a receiver sees ECN-CE markings and doesn't reflect
> that back to the sender, one of two things will happen.
>
> I understand that CoDel and PIE are Experimental, and I understand that
> L4S is still in TSV, but does a receiver cheating now, and going forward,
> make as much sense as it did when