Subject: Re: [tcpm] Does this fix sufficiently de-ossify overstrict ECN negotiation by Linux TCP servers?

From: Bob Briscoe <[email-masked]>
Date: Sat, 16 Mar 2019 11:38:20 +0000

Folks,

After offline discussion with the Linux maintainers, we have agreed on 
option #2, which is to make overstrict ECN test more specific to RFC3168 
by only disabling ECN if a SYN has a non-zero IP-ECN field and the first 
six TCP flags in the main header are 0b000011.

This will be a simple 1-line patch that maximizes the likelihood that it 
will be back-ported as far back as possible, hopefully back to the 
introduction of the over-strict test in May 2012. This should fix a 
large proportion of the installed base of Linux servers reasonably quickly.

==Logging==
Separately we will submit a patch to add logging of each event that 
disables ECN, but only for future kernel versions, not for back-porting. 
However, there is resistance to any logging, so this might not be 
accepted. The proposal is
* at default logging level, increment an SNMP count when ECN disabled 
for a connection due to a protocol violation
* at increased logging level, on each event also write rate-limited 
kernel messages that record the client IP.

Then, if a sysadmin wants to, they can get any network mangling fixed.

Bob

On 20/01/2019 23:31, Bob Briscoe wrote:
> TCPM folks,
>
> RFC8311 and draft-ietf-tcpm-generalized-ecn enable ECN-capable SYNs. 
> However, if a client sends an ECN-capable SYN, existing Linux servers 
> disable ECN for the rest of the connection.
>
> To fix as many existing Linux servers as possible, as soon as 
> possible, I would recommend short-term solution #2 (see 'Three Fixes' 
> below). An alternative would be short-term solution #3.
>
> Anyone got a better fix? Anyone disagree with recommending #2? If not, 
> pls confirm your agreement.
>
>
>       Summary of Pros and Cons
>
> #2 pro: No longer ossifies non-ECN SYN
> #2 pro: If a path does "unbleach" IP-ECN, protects the connection by 
> disabling ECN
> #2 con: A residual degree of ossification remains
>
> #3 pro: No longer ossifies non-ECN SYN
> #3 pro: No ossification at all
> #3 con: If a path does "unbleach" IP-ECN, connection still attempts to 
> use ECN even though the path is mangling the ECN field.
>
> #1 is the full solution, but it should proceed in parallel as a 
> solution for the future, not to fix the past.
>
>
>       Background
>
> To negotiate ECN for a connection the client sets flag bits 8-9 in the 
> TCP header to 11 [RFC3168 Section 6.1.1] 
> <https://tools.ietf.org/html/rfc3168#section-6.1.1>. For your 
> convenience, I've included a picture of the TCP header flags at the end.
>
> In 2012 an additional test was patched into Linux servers. It disables 
> ECN for the connection unless the IP-ECN field of the SYN is also zero 
> (I'll call this the "overstrict ECN test").
>
> This was because Section 6.1.1. of RFC3168 also said "A host MUST NOT 
> set ECT on SYN or SYN-ACK packets." Unfortunately, the overstrict ECN 
> test ossifies this requirement into TCP.
>
> The Linux patch was intended to protect a connection against the 
> network mangling the IP-ECN field to a non-zero value. Unfortunately, 
> it also disables ECN completely if a client tries to set ECT in the 
> IP-ECN field of the SYN.
>
> RFC8311 has now updated RFC3168 to allow ECT on a SYN, in conjunction 
> with the ECN++ experiment. But few developers are going to set ECT on 
> a SYN if it causes about 84% of servers to completely disable ECN (see 
> Section 4.2.2. of the ECN++ draft 
> <https://tools.ietf.org/html/draft-ietf-tcpm-generalized-ecn-03#section-4.2.2.2> 
> for further details and the source of these numbers).
>
>
>       Three Fixes
>
> I can think of three ways to fix existing Linux servers:
>
>  1. Deploy Accurate ECN
>     <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn> on
>     existing Linux servers.
>       * AccECN inherently replaces the overstrict ECN test with a pair
>         of proper two-ended tests for network mangling, in which each
>         end feeds back the arriving IP ECN field in its TCP feedback.
>         In contrast, the overstrict ECN test is one-ended, with the
>         server only assuming what the client set.
>       * Altho AccECN will solve the problem, it should not be relied
>         on to fix the last six years of Linux server deployment.
>         AccECN is a non-trivial update so it is unlikely to get
>         back-ported and auto-deployed to most existing Linux servers
>         any time soon.
>  2. Make the overstrict ECN test more specific to RFC3168, and add
>     logging.
>       * Currently, the only proposal for allowing ECT on a SYN
>         requires the client to also request Accurate ECN feedback,
>         which requires TCP header bits 7-9 to be 111.
>       * Therefore, the overstrict ECN test in a Linux server could be
>         altered to solely apply if bits 7-9 == 011 (rather than just
>         testing if bits 8-9 == 11).
>       * However, I am concerned that this ossifies around the solution
>         based on ECN++ and AccECN. What if ECN++ needs to be
>         superseded by some future protocol that allows ECT on a SYN
>         when some other field is different (e.g. one of TCP's three
>         remaining reserved flags, or a new TCP option)?
>       * Unfortunately, we cannot test for the absence of a
>         yet-to-be-invented TCP option. Nonetheless, I suggest *the
>         overstrict ECN test in a Linux server should at least be
>         altered to solely apply if TCP header bits 4-9 == 000011*.
>       * I suggest Linux also *logs whenever it disables ECN due to
>         this test*, so that broken paths can be identified and fixed.
>       * An extremely simple patch like this would be much more likely
>         than #1 (AccECN) to be back-ported and deployed to most
>         existing servers via regular Linux auto-updates.
>  3. As #2, but do not disable ECN when the test fails, just log those
>     paths suspected of mangling ECN.
>       * Our most recent measurements of millions of Internet paths
>         found no occurrences where zero IP-ECN on a SYN was changed to
>         non-zero ("unbleaching"). So we could just remove the test.
>       * However, no occurrences found does not mean none exist. Altho
>         we tested millions of paths, we only tested from a few dozen
>         vantage points, albeit we did test both mobile and fixed
>         networks (see Section 4.2.2. of the ECN++ draft
>         <https://tools.ietf.org/html/draft-ietf-tcpm-generalized-ecn-03#section-4.2.2.2>
>         for summary of tests).
>       * If the number of problematic paths is small, it would still
>         make sense to run the ECN test as proposed under item #2, but
>         not disable ECN if it fails. Instead, just log any negative
>         results, so the broken network elements can be identified and
>         fixed.
>
> Cheers
>
>
> Bob
>
> PS. For convenience, here's the TCP header flags (assuming the 
> experimental AccECN protocol is approved as RFC):
>         0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>       |               |           | A | C | E | U | A | P | R | S | F |
>       | Header Length | Reserved  | E | W | C | R | C | S | S | Y | I |
>       |               |           |   | R | E | G | K | H | T | N | N |
>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Sat, 16 Mar 2019 15:32:02 +0100

Hi Bob,

This seems eminently reasonable. I’m curious, would there be equivalent pushback on _only_ adding the SNMP counter? One could always use observation to debug once the counter starts going up...

Cheers,

Brian

Sent from my iPhone

> On 16 Mar 2019, at 12:38, Bob Briscoe <[email-masked]> wrote:
> 
> Folks,
> 
> After offline discussion with the Linux maintainers, we have agreed on option #2, which is to make overstrict ECN test more specific to RFC3168 by only disabling ECN if a SYN has a non-zero IP-ECN field and the first six TCP flags in the main header are 0b000011.
> 
> This will be a simple 1-line patch that maximizes the likelihood that it will be back-ported as far back as possible, hopefully back to the introduction of the over-strict test in May 2012. This should fix a large proportion of the installed base of Linux servers reasonably quickly.
> 
> ==Logging==
> Separately we will submit a patch to add logging of each event that disables ECN, but only for future kernel versions, not for back-porting. However, there is resistance to any logging, so this might not be accepted. The proposal is 
> * at default logging level, increment an SNMP count when ECN disabled for a connection due to a protocol violation
> * at increased logging level, on each event also write rate-limited kernel messages that record the client IP.
> 
>> RFC8311 and draft-ietf-tcpm-generalized-ecn enable ECN-capable SYNs. However, if a client sends an ECN-capable SYN, existing Linux servers disable ECN for the rest of the connection.
>> 
>> To fix as many existing Linux servers as possible, as soon as possible, I would recommend short-term solution #2 (see 'Three Fixes' below). An alternative would be short-term solution #3.
>> 
>> Anyone got a better fix? Anyone disagree with recommending #2? If not, pls confirm your agreement.
>> 
>> Summary of Pros and Cons
>> 
>> #2 pro: If a path does "unbleach" IP-ECN, protects the connection by disabling ECN
>> #3 con: If a path does "unbleach" IP-ECN, connection still attempts to use ECN even though the path is mangling the ECN field.
>> 
>> #1 is the full solution, but it should proceed in parallel as a solution for the future, not to fix the past.
>> Background
>> 
>> To negotiate ECN for a connection the client sets flag bits 8-9 in the TCP header to 11 [RFC3168 Section 6.1.1]. For your convenience, I've included a picture of the TCP header flags at the end.
>> 
>> In 2012 an additional test was patched into Linux servers. It disables ECN for the connection unless the IP-ECN field of the SYN is also zero (I'll call this the "overstrict ECN test").
>> 
>> This was because Section 6.1.1. of RFC3168 also said "A host MUST NOT set ECT on SYN or SYN-ACK packets." Unfortunately, the overstrict ECN test ossifies this requirement into TCP. 
>> 
>> The Linux patch was intended to protect a connection against the network mangling the IP-ECN field to a non-zero value. Unfortunately, it also disables ECN completely if a client tries to set ECT in the IP-ECN field of the SYN.
>> 
>> RFC8311 has now updated RFC3168 to allow ECT on a SYN, in conjunction with the ECN++ experiment. But few developers are going to set ECT on a SYN if it causes about 84% of servers to completely disable ECN (see Section 4.2.2. of the ECN++ draft for further details and the source of these numbers).
>> Three Fixes
>> 
>> Deploy Accurate ECN on existing Linux servers.
>> AccECN inherently replaces the overstrict ECN test with a pair of proper two-ended tests for network mangling, in which each end feeds back the arriving IP ECN field in its TCP feedback. In contrast, the overstrict ECN test is one-ended, with the server only assuming what the client set.
>> Altho AccECN will solve the problem, it should not be relied on to fix the last six years of Linux server deployment. AccECN is a non-trivial update so it is unlikely to get back-ported and auto-deployed to most existing Linux servers any time soon. 
>> Make the overstrict ECN test more specific to RFC3168, and add logging.
>> Currently, the only proposal for allowing ECT on a SYN requires the client to also request Accurate ECN feedback, which requires TCP header bits 7-9 to be 111.
>> Therefore, the overstrict ECN test in a Linux server could be altered to solely apply if bits 7-9 == 011 (rather than just testing if bits 8-9 == 11).
>> However, I am concerned that this ossifies around the solution based on ECN++ and AccECN. What if ECN++ needs to be superseded by some future protocol that allows ECT on a SYN when some other field is different (e.g. one of TCP's three remaining reserved flags, or a new TCP option)? 
>> Unfortunately, we cannot test for the absence of a yet-to-be-invented TCP option. Nonetheless, I suggest the overstrict ECN test in a Linux server should at least be altered to solely apply if TCP header bits 4-9 == 000011. 
>> I suggest Linux also logs whenever it disables ECN due to this test, so that broken paths can be identified and fixed.
>> An extremely simple patch like this would be much more likely than #1 (AccECN) to be back-ported and deployed to most existing servers via regular Linux auto-updates.
>> As #2, but do not disable ECN when the test fails, just log those paths suspected of mangling ECN.
>> Our most recent measurements of millions of Internet paths found no occurrences where zero IP-ECN on a SYN was changed to non-zero ("unbleaching"). So we could just remove the test.
>> However, no occurrences found does not mean none exist. Altho we tested millions of paths, we only tested from a few dozen vantage points, albeit we did test both mobile and fixed networks (see Section 4.2.2. of the ECN++ draft for summary of tests).
>> If the number of problematic paths is small, it would still make sense to run the ECN test as proposed under item #2, but not disable ECN if it fails. Instead, just log any negative results, so the broken network elements can be identified and fixed.
>> Cheers
>> 
>> 
>> Bob
>> 
>> PS. For convenience, here's the TCP header flags (assuming the experimental AccECN protocol is approved as RFC):
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 17 Mar 2019 02:05:27 +0000

Brian,
I've asked - I'll let you know once I hear.

Bob

On 16/03/2019 14:32, Brian Trammell (IETF) wrote:
> Hi Bob,
>
> This seems eminently reasonable. I’m curious, would there be 
> equivalent pushback on _only_ adding the SNMP counter? One could 
> always use observation to debug once the counter starts going up...
>
> Cheers,
>
> Brian
>
> Sent from my iPhone
>
> On 16 Mar 2019, at 12:38, Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>> Folks,
>>
>> After offline discussion with the Linux maintainers, we have agreed 
>> on option #2, which is to make overstrict ECN test more specific to 
>> RFC3168 by only disabling ECN if a SYN has a non-zero IP-ECN field 
>> and the first six TCP flags in the main header are 0b000011.
>>
>> This will be a simple 1-line patch that maximizes the likelihood that 
>> it will be back-ported as far back as possible, hopefully back to the 
>> disables ECN, but only for future kernel versions, not for 
>> back-porting. However, there is resistance to any logging, so this 
>> might not be accepted. The proposal is
>>> However, if a client sends an ECN-capable SYN, existing Linux 
>>> servers disable ECN for the rest of the connection.
>>>
>>> possible, I would recommend short-term solution #2 (see 'Three 
>>> Fixes' below). An alternative would be short-term solution #3.
>>>
>>> #3 con: If a path does "unbleach" IP-ECN, connection still attempts 
>>> to use ECN even though the path is mangling the ECN field.
>>>
>>> To negotiate ECN for a connection the client sets flag bits 8-9 in 
>>> the TCP header to 11 [RFC3168 Section 6.1.1] 
>>> In 2012 an additional test was patched into Linux servers. It 
>>> disables ECN for the connection unless the IP-ECN field of the SYN 
>>> is also zero (I'll call this the "overstrict ECN test").
>>>
>>> This was because Section 6.1.1. of RFC3168 also said "A host MUST 
>>> NOT set ECT on SYN or SYN-ACK packets." Unfortunately, the 
>>> overstrict ECN test ossifies this requirement into TCP.
>>>
>>> network mangling the IP-ECN field to a non-zero value. 
>>> Unfortunately, it also disables ECN completely if a client tries to 
>>> set ECT in the IP-ECN field of the SYN.
>>>
>>> RFC8311 has now updated RFC3168 to allow ECT on a SYN, in 
>>> conjunction with the ECN++ experiment. But few developers are going 
>>> to set ECT on a SYN if it causes about 84% of servers to completely 
>>> disable ECN (see Section 4.2.2. of the ECN++ draft 
>>>       * AccECN inherently replaces the overstrict ECN test with a
>>>         pair of proper two-ended tests for network mangling, in
>>>         which each end feeds back the arriving IP ECN field in its
>>>         TCP feedback. In contrast, the overstrict ECN test is
>>>         one-ended, with the server only assuming what the client set.
>>>       * Therefore, the overstrict ECN test in a Linux server could
>>>         be altered to solely apply if bits 7-9 == 011 (rather than
>>>         just testing if bits 8-9 == 11).
>>>       * However, I am concerned that this ossifies around the
>>>         solution based on ECN++ and AccECN. What if ECN++ needs to
>>>         be superseded by some future protocol that allows ECT on a
>>>         SYN when some other field is different (e.g. one of TCP's
>>>         three remaining reserved flags, or a new TCP option)?
>>>       * An extremely simple patch like this would be much more
>>>         likely than #1 (AccECN) to be back-ported and deployed to
>>>         most existing servers via regular Linux auto-updates.
>>>  3. As #2, but do not disable ECN when the test fails, just log
>>>     those paths suspected of mangling ECN.
>>>         found no occurrences where zero IP-ECN on a SYN was changed
>>>         to non-zero ("unbleaching"). So we could just remove the test.
>>>       * However, no occurrences found does not mean none exist.
>>>         Altho we tested millions of paths, we only tested from a few
>>>         dozen vantage points, albeit we did test both mobile and
>>>         fixed networks (see Section 4.2.2. of the ECN++ draft
>>>         make sense to run the ECN test as proposed under item #2,
>>>         but not disable ECN if it fails. Instead, just log any
>>>         negative results, so the broken network elements can be
>>>         identified and fixed.
>>>
>>> Cheers
>>>
>>>
>>> Bob
>>>
>> [email-masked] <mailto:[email-masked]>

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 18 Mar 2019 00:16:53 +0000

Brian,

Looks like adding only an SNMP counter could be acceptable. We'll submit 
a patch for net-next on that basis (as well as a patch without logging 
for net-stable).

Bob

On 17/03/2019 02:05, Bob Briscoe wrote:
>>> This will be a simple 1-line patch that maximizes the likelihood 
>>> that it will be back-ported as far back as possible, hopefully back 
>>> to the introduction of the over-strict test in May 2012. This should 
>>> fix a large proportion of the installed base of Linux servers 
>>> reasonably quickly.
>>>
>>> ==Logging==
>>> * at default logging level, increment an SNMP count when ECN 
>>> disabled for a connection due to a protocol violation
>>>> RFC8311 and draft-ietf-tcpm-generalized-ecn enable ECN-capable 
>>>> SYNs. However, if a client sends an ECN-capable SYN, existing Linux 
>>>>       * Altho AccECN will solve the problem, it should not be
>>>>         relied on to fix the last six years of Linux server
>>>>         deployment. AccECN is a non-trivial update so it is
>>>>         unlikely to get back-ported and auto-deployed to most
>>>>         existing Linux servers any time soon.
>>>>         Altho we tested millions of paths, we only tested from a
>>>>         few dozen vantage points, albeit we did test both mobile
>>>>         and fixed networks (see Section 4.2.2. of the ECN++ draft