Subject: Re: Adding ECN to Transport and Recovery

From: Magnus Westerlund <[email-masked]>
Date: Fri, 8 Jun 2018 16:51:36 +0200

WG,

The pull request has been updated so that each received CE mark will 
result in an immediate ACK. This will prevent the uncertainty of CE 
marks in related to the recovery period assuming no ACK loss, and having 
a fair chance to avoid the issue also at some ACK loss.

Secondly, I have added a security consideration section for ECN related 
issues.

My understanding of the views at the interim was that eliminating the 
additional ACK_ECN frame was desired. I will next update the pull 
request to use an updated ACK frame with a signaling bit for the present 
or absence of the ECN counters. The plan is to steal a bit in the ACK 
Block count and make that into a fixed 7-bit field rather than a 
variable length encoding. Limiting an ACK Frame to 128 ACK blocks.

Cheers

Magnus

Den 2018-05-28 kl. 22:29, skrev Magnus Westerlund:
> WG,
>
> I have been working on a Pull Request 
> (https://github.com/quicwg/base-drafts/pull/1372) that implements the 
> ECN proposal from the design team 
> (https://github.com/quicwg/base-drafts/wiki/ECN-in-QUIC) into 
> something reasonably well integrated into the current drafts. Thus, I 
> think it is time that people review and consider this proposal so that 
> any issues can be addressed.
>
> I would very much like to present at the interim this proposal, some 
> of the choices made, and determine if there are additional changes to 
> be made before merging this.
>
> Cheers
>
> Magnus Westerlund
>
> ----------------------------------------------------------------------
> Network Architecture & Protocols, Ericsson Research
> ----------------------------------------------------------------------
> Ericsson AB                 | Phone  +46 10 7148287
> Torshamnsgatan 23           | Mobile +46 73 0949079
> SE-164 80 Stockholm, Sweden | mailto: [email-masked]
> ----------------------------------------------------------------------
>

-- 

Magnus Westerlund

----------------------------------------------------------------------
Network Architecture & Protocols, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 8 Jun 2018 11:08:35 -0400

On Fri, Jun 08, 2018 at 04:51:36PM +0200, Magnus Westerlund wrote:
> additional ACK_ECN frame was desired.

That is my understanding as well.

> I will next update the pull request to use an updated ACK frame with a
> signaling bit for the present or absence of the ECN counters. The plan
> is to steal a bit in the ACK Block count and make that into a fixed
> 7-bit field rather than a variable length encoding.  Limiting an ACK
> Frame to 128 ACK blocks.

Even though I don't have strong feelings about this, I wonder if there
is another way.  The ACK frame (Section 7.15 of
[draft-ietf-quic-transport-12]) is all varints.  Changing a field to a
fixed encoding would go against the "flavor of the frame," if one may
say so.  Perhaps we could keep the varint encoding and interpret the
decoded value as an ORed block count with the ECN bit (the low bit)?

Introducing an artifial limit to where there was none is a separate
discussion.  Perhaps 128 blocks is really enough for all practical
purposes.  And yet: at some point the block limit was removed as we
went from gQUIC to IETF QUIC...

  - Dmitri.

---

From: Ian Swett <[email-masked]>
Date: Fri, 8 Jun 2018 12:40:35 -0400

As Dmitri said, the num blocks is currently a varint, it's not a single
byte like it was in GQUIC, so stealing a bit from it is a bit awkward and
would not change the limit on the number of ack blocks from 256 to 128, and
instead it would decrease it from 2^62 to 2^61.

Can you make the least significant bit of the ACK frame type indicate
whether the ECN section is present, such as by using the 0x0d and 0x0e code
points?  This is basically equivalent to defining two types, admittedly,
but by making the least significant bit the indicator, one could write up
the text as a single frame if people prefer that.

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 9 Jun 2018 01:17:14 +0300

2018-06-08 19:40 GMT+03:00 Ian Swett <ianswett=[email-masked]>:
> As Dmitri said, the num blocks is currently a varint, it's not a single byte
> like it was in GQUIC, so stealing a bit from it is a bit awkward and would
> not change the limit on the number of ack blocks from 256 to 128, and
> points?  This is basically equivalent to defining two types, admittedly, but
> by making the least significant bit the indicator, one could write up the
> text as a single frame if people prefer that.

+1 to using different frame types. We do not want to introduce
complexity for conserving the frame types. That was the conclusion of
the extension discussion, wasn't it?

I would prefer having different frame types, unless we find a very
clean way of encoding ACK and ACK_ECN using a single frame type.
Stealing bits from a varint encoding is a haphazard approach that
comes with a complexity.

Personally, I doubt if we will ever run out of frames types,
considering how we have done so far with TLS ContentType and HTTP/2
frame type. And if we do run out, we have the luxury of renumbering
unlike TCP or TLS, because QUIC encrypts the frame types.

>
> On Fri, Jun 8, 2018 at 11:08 AM Dmitri Tikhonov
> <[email-masked]> wrote:
>>

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Fri, 8 Jun 2018 17:36:28 -0700

> unlike TCP or TLS, because QUIC encrypts the frame types.

+1. It is much simpler to have two frame types. And in fact is would be
better to not try to be too smart, Ian's suggestion of allocating two
consecutive types differing by one bit is cute, but it will create
incompatibility with the previous versions for which 0xE means "path
challenge". Not a very big deal but still annoying, especially if one
attempts to support multiple versions while testing the new one.

-- Christian Huitema

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Sat, 9 Jun 2018 13:25:23 +0200

+1 to two frame types; IIRC the suggestion was to add ECN data to the "normal" ACK frame, then have a diminished ACK frame for situations where there is no ECN information to report. The concern here is that having an ACK and an ACK-ECN frame reinforces the incorrect idea that ECN is somehow optional to implement for QUIC, which it should not be.

Half-serious suggestions for the smaller ACK frame included SMACK (small ACK), SHACK (short ACK), or ACK-ICN (Implicit Congestion Notification).

Cheers,

Brian

> On 9 Jun 2018, at 02:36, Christian Huitema <[email-masked]> wrote:
> 
>>> instead it would decrease it from 2
>>> 62 to 261
>>> .
>>> 
> +1. It is much simpler to have two frame types. And in fact is would be better to not try to be too smart, Ian's suggestion of allocating two consecutive types differing by one bit is cute, but it will create incompatibility with the previous versions for which 0xE means "path challenge". Not a very big deal but still annoying, especially if one attempts to support multiple versions while testing the new one.
> 
> -- Christian Huitema

---

From: Ian Swett <[email-masked]>
Date: Sat, 9 Jun 2018 11:27:45 -0400

QUIC is complex enough, I think we should just call them ACK and ACK_ECN.
A few implementations sending at least one ACK_ECN frame per connection
will dispel any belief that ACK_ECN is optional.  A small amount of use
will be a lot more valuable than clever naming.

On Sat, Jun 9, 2018 at 7:25 AM Brian Trammell (IETF) <[email-masked]>
wrote:

> +1 to two frame types; IIRC the suggestion was to add ECN data to the
> "normal" ACK frame, then have a diminished ACK frame for situations where
> there is no ECN information to report. The concern here is that having an
> ACK and an ACK-ECN frame reinforces the incorrect idea that ECN is somehow
> optional to implement for QUIC, which it should not be.
>
> Half-serious suggestions for the smaller ACK frame included SMACK (small
> ACK), SHACK (short ACK), or ACK-ICN (Implicit Congestion Notification).
>
> Cheers,
>
> Brian
>
> >> 2018-06-08 19:40 GMT+03:00 Ian Swett <ianswett=
> [email-masked]>
> >> :
> >>

---

From: Christian Huitema <[email-masked]>
Date: Sat, 9 Jun 2018 08:38:48 -0700

> On Jun 9, 2018, at 8:27 AM, Ian Swett <[email-masked]> wrote:
> 
> QUIC is complex enough, I think we should just call them ACK and ACK_ECN.  A few implementations sending at least one ACK_ECN frame per connection will dispel any belief that ACK_ECN is optional.  A small amount of use will be a lot more valuable than clever naming.

Also, ECN is just one of the extensions that could be planned for ACK. Timestamps would be nice too, for those who want one-way-delay based congestion control. 

-- Christian Huitema

---

From: Jana Iyengar <[email-masked]>
Date: Sun, 10 Jun 2018 10:14:30 -0700

Sounds like there's general agreement that modifying the varint to contain
a flag bit will add unnecessary complexity.
Let's try to get the PR in as it is now, and if anyone really wants to go
merge this and the ACK frame or rename things, let's paint that bikeshed
separately.

On Sat, Jun 9, 2018 at 8:39 AM Christian Huitema <[email-masked]>
wrote:

> > QUIC is complex enough, I think we should just call them ACK and
> ACK_ECN.  A few implementations sending at least one ACK_ECN frame per
> connection will dispel any belief that ACK_ECN is optional.  A small amount
> of use will be a lot more valuable than clever naming.
>
> Also, ECN is just one of the extensions that could be planned for ACK.
> Timestamps would be nice too, for those who want one-way-delay based
> congestion control.
>
> -- Christian Huitema
>

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 11 Jun 2018 13:15:28 +0200

Hi,

I agree with Jana, as there are not a clear consensus lets take any 
further modifications of the ACK frames as a separate issue. The PR have 
from my perspective no open technical content issues. I will await to 
see if the editors or any of you have any editorial issues. But, 
otherwise I expect this PR to be merged quite soon.

I will now start working on a new section for a separate PR regarding 
the ECN black hole mitigation.

Cheers

Magnus

Den 2018-06-10 kl. 19:14, skrev Jana Iyengar:
> Sounds like there's general agreement that modifying the varint to 
> contain a flag bit will add unnecessary complexity.
> Let's try to get the PR in as it is now, and if anyone really wants to 
> go merge this and the ACK frame or rename things, let's paint that 
> bikeshed separately.
>
>
>
> On Sat, Jun 9, 2018 at 8:39 AM Christian Huitema <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>
>     > On Jun 9, 2018, at 8:27 AM, Ian Swett <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >
>     ACK_ECN.  A few implementations sending at least one ACK_ECN frame
>     per connection will dispel any belief that ACK_ECN is optional.  A
>     small amount of use will be a lot more valuable than clever naming.
>
>     Also, ECN is just one of the extensions that could be planned for
>     ACK. Timestamps would be nice too, for those who want
>     one-way-delay based congestion control.
>
>     -- Christian Huitema
>

-- 

Magnus Westerlund

---

From: "Lubashev, Igor" <[email-masked]>
Date: Mon, 11 Jun 2018 12:18:11 +0000

I do not think I have seen this in the PR, so I am wondering what your recommendation is for defending against an on-a-side attacker, who sets CE bits on every EC(*) packet and races that packet to the destination? Should that be a part of this PR?

- Igor

-----Original Message-----
From: Magnus Westerlund [[email-masked]]
Received: Monday, 11 Jun 2018, 1:15PM
To: Jana Iyengar [[email-masked]]; Christian Huitema [[email-masked]]
CC: Brian Trammell (IETF) [[email-masked]]; Ian Swett [[email-masked]]; QUIC WG [[email-masked]]
Subject: Re: Adding ECN to Transport and Recovery

Hi,

I agree with Jana, as there are not a clear consensus lets take any further modifications of the ACK frames as a separate issue. The PR have from my perspective no open technical content issues. I will await to see if the editors or any of you have any editorial issues. But, otherwise I expect this PR to be merged quite soon.

I will now start working on a new section for a separate PR regarding the ECN black hole mitigation.

Cheers

Magnus

Den 2018-06-10 kl. 19:14, skrev Jana Iyengar:
Sounds like there's general agreement that modifying the varint to contain a flag bit will add unnecessary complexity.
Let's try to get the PR in as it is now, and if anyone really wants to go merge this and the ACK frame or rename things, let's paint that bikeshed separately.

On Sat, Jun 9, 2018 at 8:39 AM Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

> On Jun 9, 2018, at 8:27 AM, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:
>

-- Christian Huitema

--

Magnus Westerlund

SE-164 80 Stockholm, Sweden | mailto: [email-masked]<mailto:[email-masked]>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 12 Jun 2018 10:22:33 +0200

Hi,

So, if the on-a-side attack manages to race its attack packet with CE 
set so that it arrives prior to the original packet with an ECT 
code-point it will affect the congestion window. The same way that if an 
attack can cause periodic network drop of packets for a QUIC connection 
the connection can't really distinguish this attack from a legit 
congestion indication signal.

So the PR's current security consideration addition says:

12.7.  Explicit Congestion Notification Attacks

    The ECN bits [RFC3168] are an unauthenticated signal from the
    network.  An on-path attacker may manipulate the value of the field.
    Thus, affecting the congestion avoidance behavior of the sender.  By
    clearing any CE marks the connection can help drive a bottle neck
    queue into a loss regime.  By setting the ECN field to CE marking it
    can drive down the senders congestion window thus resulting in
    reduced throughput.  The later could equally be accomplished by
    dropping packets for the connection.  Section 18 and 19 of [RFC3168]
    discusses the effects of undesired manipulation of the ECN field in
    more details.

    If a receiver would not have packet duplication detection and not
    discard any duplicates an off-path attacker that can receive copies
    of the connection's packets can manipulate the senders congestion
    avoidance state.  If packet duplicates are dropped, the off-path
    attacker will need to race the original packet to be successful in
    this attack.

If there is a proper on-path attack, then it can accomplish this already
to day by dropping packets, so ECN does not create a new vulnerability.
The requirement for an on-the-side attacker to successfully race its modified
packets make the attack harder to perform, the right situation must exist
where the normal route is longer than the route for the attacker, i.e.
tap -> attacker -> receiver. And the attacker must be able to tap the flow.
 From my perspective, that makes this an attack with limited applicability.

Mitigating the attack in either of the QUIC endpoints are also extremely
hard. How does a receiver or sender determine that the congestion signals
are manipulated? For an attacker that changes an ECT to ECN-CE I don't see
there being any reasonable detection mechanism. One would need to generate
parallel flow and statistically compare the behaviour of the flows to determine
that one flow was manipulated. But, such a flow is likely to be picked up by
an attacker as if I am attacking one I would tap the flows using a single
source or destination IP as flow template for the traffic that I as an
attacker are interested in.

For the case where an attacker removes the CE marks, that would be possible
to detect by sender side probes injecting CE marked packets, as discussed
in this issue: https://github.com/quicwg/base-drafts/issues/1426

My view is that the PR has the necessary text noting the issue and
referencing the far more extensive discussion in RFC 3168. We have an
issue to continue to discuss additional mitigation. Such mitigation should
be written up in a separate PR in my view.

Cheers

Magnus

Den 2018-06-11 kl. 14:18, skrev Lubashev, Igor:
> I do not think I have seen this in the PR, so I am wondering what your 
> recommendation is for defending against an on-a-side attacker, who 
> sets CE bits on every EC(*) packet and races that packet to the 
> destination? Should that be a part of this PR?
>
> - Igor
>
> *From:* Magnus Westerlund [[email-masked]]
> *Received:* Monday, 11 Jun 2018, 1:15PM
> *To:* Jana Iyengar [[email-masked]]; Christian Huitema 
> [[email-masked]]
> *CC:* Brian Trammell (IETF) [[email-masked]]; Ian Swett 
> [[email-masked]]; QUIC WG [[email-masked]]
> *Subject:* Re: Adding ECN to Transport and Recovery
>
> Hi,
>
> further modifications of the ACK frames as a separate issue. The PR 
> have from my perspective no open technical content issues. I will 
> await to see if the editors or any of you have any editorial issues. 
> But, otherwise I expect this PR to be merged quite soon.
>
>> Let's try to get the PR in as it is now, and if anyone really wants 
>> to go merge this and the ACK frame or rename things, let's paint that 
>> bikeshed separately.
>>
>>
>>
>>     > QUIC is complex enough, I think we should just call them ACK
>>     and ACK_ECN.  A few implementations sending at least one ACK_ECN
>>     frame per connection will dispel any belief that ACK_ECN is
>>     optional.  A small amount of use will be a lot more valuable than
>>     clever naming.
>>
> SE-164 80 Stockholm, Sweden | mailto:[email-masked]

-- 

Magnus Westerlund

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 12 Jun 2018 12:17:27 +0000

Thanks. So what you are saying is that a requirement for a on-a-side attacker to race cooked packets is what make this attack hard? I can buy this, especially since such an attacker could interfere with the initial version negotiation, too. The only thing that makes an attack on the congestion window nefarious vs an attack on version negotiation is that it leaves the user with an impression that the service is poor vs that the network is interfering with him connecting to the service.

-----Original Message-----
From: Magnus Westerlund [[email-masked]]
Received: Tuesday, 12 Jun 2018, 4:23AM
To: Lubashev, Igor [[email-masked]]; [email-masked] [[email-masked]]; [email-masked] [[email-masked]]
CC: [email-masked] [[email-masked]]; [email-masked] [[email-masked]]; [email-masked] [[email-masked]]
Subject: Re: Adding ECN to Transport and Recovery

Hi,

So, if the on-a-side attack manages to race its attack packet with CE set so that it arrives prior to the original packet with an ECT code-point it will affect the congestion window. The same way that if an attack can cause periodic network drop of packets for a QUIC connection the connection can't really distinguish this attack from a legit congestion indication signal.

For the case where an attacker removes the CE marks, that would be possible
to detect by sender side probes injecting CE marked packets, as discussed
in this issue: https://github.com/quicwg/base-drafts/issues/1426<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_base-2Ddrafts_issues_1426&d=DwMD-g&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=fCJNH4_GrpJCEuL__VxRjfVtL0Cs4z66XI5BrJPbYi4&s=2_OCqkf4LQgo0ms4jC0p4xPnRUznLuTGsDfIhwcrcYY&e=>

Cheers

Magnus

- Igor

-----Original Message-----
From: Magnus Westerlund [[email-masked]<mailto:[email-masked]>]
Received: Monday, 11 Jun 2018, 1:15PM
To: Jana Iyengar [[email-masked]<mailto:[email-masked]>]; Christian Huitema [[email-masked]<mailto:[email-masked]>]
CC: Brian Trammell (IETF) [[email-masked]<mailto:[email-masked]>]; Ian Swett [[email-masked]<mailto:[email-masked]>]; QUIC WG [[email-masked]<mailto:[email-masked]>]
Subject: Re: Adding ECN to Transport and Recovery

Hi,

Cheers

Magnus

-- Christian Huitema

--

Magnus Westerlund

--

Magnus Westerlund

---

From: "Eggert, Lars" <[email-masked]>
Date: Tue, 12 Jun 2018 12:27:01 +0000

On 2018-6-12, at 14:17, Lubashev, Igor <ilubashe=[email-masked]> wrote:
> The only thing that makes an attack on the congestion window nefarious vs an attack on version negotiation is that it leaves the user with an impression that the service is poor vs that the network is interfering with him connecting to the service.

Right, but that is not specific to QUIC.

There are possibilities to add mitigations at a later stage (QUIC v2 or v1.1), but I personally don't think these need to be part of v1.

Lars

---

From: Ian Swett <[email-masked]>
Date: Tue, 12 Jun 2018 08:33:17 -0400

I'd be shocked if anyone tried to use this attack in practice, and as Lars
said, this is in no way specific to QUIC.  The one case I can imagine
someone wanting to inject ECN is an on path middlebox, in which case that's
what ECN is for.

I don't think this needs to be part of v1.

On Tue, Jun 12, 2018 at 8:27 AM Eggert, Lars <[email-masked]> wrote:

> > The only thing that makes an attack on the congestion window nefarious
> vs an attack on version negotiation is that it leaves the user with an
> impression that the service is poor vs that the network is interfering with
> him connecting to the service.
>
> There are possibilities to add mitigations at a later stage (QUIC v2 or
> v1.1), but I personally don't think these need to be part of v1.
>
> Lars
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 12 Jun 2018 14:41:29 +0000

I think we are in an agreement here.

A more generic conclusion from this discussion is that defense against man-on-a-side DoS attacks is out of scope for this WG (at least for v1).

  *   I

From: Ian Swett [mailto:ianswett=[email-masked]]
Sent: Tuesday, June 12, 2018 8:33 AM
To: Eggert, Lars <[email-masked]>
Cc: Lubashev, Igor <[email-masked]>; Magnus Westerlund <[email-masked]>; Jana Iyengar <[email-masked]>; Christian Huitema <[email-masked]>; Brian Trammell <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Adding ECN to Transport and Recovery

I'd be shocked if anyone tried to use this attack in practice, and as Lars said, this is in no way specific to QUIC.  The one case I can imagine someone wanting to inject ECN is an on path middlebox, in which case that's what ECN is for.

On Tue, Jun 12, 2018 at 8:27 AM Eggert, Lars <[email-masked]<mailto:[email-masked]>> wrote:

Lars

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 13 Jun 2018 16:03:28 +0200

Hi,

I have just added one more paragraph to the pull request to deal with an 
issue that arises if there is persistent ACK loss such that the sender 
never receives an acknowledgement for some packet that was actually 
received by receiver and counted in the ECN counters. To make 
implementers aware of the issue I have added this paragraph in Section 
6.6.1. Continuous Verification of ECN:

    If the acknowledgements from the receiver are lost such that one or
    more packet are received by the receiver, but never acknowledged to
    the sender an insensitivity to bleaching will be created.  In this
    situation the ECN counters reported will have increase, but the
    sender side total for acknowledged packets will not have increased.
    Thus, a number of bleached packets equal to the number of packets
    that failed to be acknowledged can be received before triggering the
    continuous verification.  To address this issue the sender detect the
    case when the ECN counters grows more than number of acknowledged
    packets when a ACK_ECN frame is received.  In such cases a new
    comparison point is created by storing the current number of totally
    acknowledged packets and latest ECN counters.  Then comparison are
    done by subtracting these stored values from the respective counters
    prior to the comparison.  Note that any out-of-order ACK_ECN frames
    can't be used for determining any loss of acknowledgements.

Please provide feedback on this addition.

Cheers

Magnus

-- 

Magnus Westerlund