Subject: Self-healing Networking with Flow Label

From: Vasilenko Eduard <[email-masked]>
Date: Sun, 1 Aug 2021 16:46:42 +0000

Hi  Alexander,

Have I understood your presentation right?

The client SHOULD change IPv6 flow label after SYN RTO to have a chance to be moved to the working path inside DC fabric (if DC fabric supports flow label for hash calculation)

But at the same time

The client SHOULD NOT change the IPv6 flow label after SYN RTO to avoid being switched to a different TCP proxy engine.

Looks like a deadlock, especially if both things should happen for the same traffic:

it should reach DC fabric

and

it should be hash load-balanced between different TCP proxy engines (or applications) inside DC Fabric.

I see one bad solution ("Disable Flow Label"):

Routers up to TCP proxy engine SHOULD be configured not to use flow label (by the way these are all routers on the Internet),

TCP flow engines SHOULD be outside of the DC Fabric (CLOS) - probably in front of it.

Routers/Switches inside DC Fabric SHOULD use flow labels.

I see another bad solution ("Disable Anycast"):

Disable anycast on routers in principle, use only stateful LB.

It has been commented in the chat that Anycast is not possible in principle for stateful connection. It is too general a statement.

Anycast is just not compatible with Flow Label. It is not a problem for IPv4 anycast even if the connection is stateful (TCP) because 5-tuple for hash would not change.

Hence, IPv6 anycast has become dead at the time when Flow Label change has been added in LINUX for active TCP session.

Among 3 thins:

-          Anycast

-          Flow Label load balancing (basic Flow Label functionality)

-          Flow Label change on the active session for application to be more active in new path search

You have to choose which one to kill - all 3 are not compatible with each other at the same.

I vote to disable Flow Label change in LINUX. Then wait till the network would fix itself.

We have so many fancy TE tools our days. A broken link or a broken node could be excluded from routing for 50ms.

PS: I am not subscribed to the RTGWG alias, please keep me on a copy of this thread.
Best Regards
Eduard Vasilenko
Senior Architect
Europe Standardization & Industry Development Department
Tel: +7(985) 910-1105, +7(916) 800-5506

---

From: Mark Smith <[email-masked]>
Date: Mon, 2 Aug 2021 08:13:02 +1000

What are TCP proxy engines?

What are TCP flow engines?

In which RFCs are they described?

On Mon, 2 Aug 2021, 02:47 Vasilenko Eduard, <[email-masked]>
wrote:

> The client SHOULD change IPv6 flow label after SYN RTO to have a chance to
> be moved to the working path inside DC fabric (if DC fabric supports flow
> label for hash calculation)
>
> But at the same time
>
> The client SHOULD NOT change the IPv6 flow label after SYN RTO to avoid
> being switched to a different TCP proxy engine.
>
>
>
> I see one bad solution (“Disable Flow Label”):
>
> Routers up to TCP proxy engine SHOULD be configured not to use flow label
> (by the way these are all routers on the Internet),
>
> TCP flow engines SHOULD be outside of the DC Fabric (CLOS) – probably in
> front of it.
>
> I see another bad solution (“Disable Anycast”):
>
> It has been commented in the chat that Anycast is not possible in
> principle for stateful connection. It is too general a statement.
>
> Anycast is just not compatible with Flow Label. It is not a problem for
> IPv4 anycast even if the connection is stateful (TCP) because 5-tuple for
> hash would not change.
>
> Hence, IPv6 anycast has become dead at the time when Flow Label change has
> been added in LINUX for active TCP session.
>
>
>
> Among 3 thins:
>
> -          Anycast
>
> You have to choose which one to kill – all 3 are not compatible with each
> other at the same.
>
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg
>

---

From: Jeff Tantsura <[email-masked]>
Date: Sun, 1 Aug 2021 20:56:28 -0700

Eduard,

The issue is present not in link/device case, if well implemented - fast rehash takes care of updating forwarding within a number of ms. The problem is with  “gray” failures,  when the link in question is UP from routing/forwarding prospective but drops traffic (mostly occasionally and when a HW bug occurs has a distinct flow attributes).

In many large DC fabrics, the majority of the traffic is east-west, between end-points that aren’t anycast. In such deployments - the solution solves  issues rather elegantly and without any interventions from the operator.
The issues/side effects are well understood and will be documented.

The best way to receive RTGWG emails is well… to subscribe to RTGWG ;-)

Cheers,
Jeff

> On Aug 1, 2021, at 09:47, Vasilenko Eduard <[email-masked]> wrote:
> 
> 
> Hi  Alexander,
>  
> TCP flow engines SHOULD be outside of the DC Fabric (CLOS) – probably in front of it.
> You have to choose which one to kill – all 3 are not compatible with each other at the same.

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 2 Aug 2021 08:12:42 +0000

Mark, sorry, I am just using terminology from the slide.
Look slides 21-29 from https://datatracker.ietf.org/meeting/111/materials/slides-111-rtgwg-sessb-3-selfhealing-network-01
It is easy to guess what it is: a device that terminates a TCP session, initiates a new one, and stitches both. Firewall, stateful load balancer.
Stateful devices are not very scalable, hence, anycast distribution of the load in front of the stack of such devices is common practice.
IMHO: It is not the bad terminology that Alexander used. It is easy to guess what is “TCP proxy”. I do not see the need for RFC.
Eduard
From: Mark Smith [mailto:[email-masked]]
Sent: Monday, August 2, 2021 1:13 AM
To: Vasilenko Eduard <[email-masked]>
Cc: [email-masked]; routing WG <[email-masked]>
Subject: Re: Self-healing Networking with Flow Label

In which RFCs are they described?
On Mon, 2 Aug 2021, 02:47 Vasilenko Eduard, <[email-masked]<mailto:[email-masked]>> wrote:

Hi  Alexander,

But at the same time

and

Among 3 thins:

-          Anycast

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 2 Aug 2021 08:48:34 +0000

Hi Jeff,
The situation when Control Plane does not understand what the Forwarding pane doing is a bug.
Yes, RTO in TCP helps to find a work-around for this bug. And yes, Anycast is typically absent inside DC – it does not create the problem in the DC environment.

But the same LINUX is used outside DC. RTO Flow Label change here would create even more problems if Anycast would happen on the traffic path (not much predictable for client).
Do we need separate LINUX distribution for DC and separate distribution for other environments?
Or should we rely on the proper non-default configuration for different environments? (Admin should not forget to change)
What if Anycast would become needed in DC?

RTO flow label recalculation is mutually exclusive with Anycast on the traffic part.
What is more valuable for the public?

IMHO: It is better to fight the problem of such type of a bug with iOAM than to cancel Anycast.

IMHO: It is better to have Flow Label recalculation on RTO as “off” by default.
DC Admin has the higher qualification to activate it in a controlled environment than every client worldwide that should not forget to disable it.

Eduard
From: Jeff Tantsura [mailto:[email-masked]]
Sent: Monday, August 2, 2021 6:56 AM
To: Vasilenko Eduard <[email-masked]>
Cc: [email-masked]; routing WG <[email-masked]>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

On Aug 1, 2021, at 09:47, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi  Alexander,

But at the same time

and

Among 3 thins:

-          Anycast

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 2 Aug 2021 10:44:54 +0000

It is the key in this presentation “This behavior MUST be switched off by default”
It has been shown on slides 7-10 that flow label change on RTO is enabled by default for BSD and LINUX.
It needs regulation. It needs a standard RFC. Because it kills Anycast otherwise.

The story of how to use RTO to work-around “silent drop” vendor’s bugs could be a good informational RFC.
My be people developing iOAM would pay more attention to this use case.

IMHO: these are 2 separate drafts.
Eduard
From: Alexander Azimov [mailto:[email-masked]]
Sent: Monday, August 2, 2021 1:20 PM
To: Vasilenko Eduard <[email-masked]>; Jeff Tantsura <[email-masked]>
Cc: routing WG <[email-masked]>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

Please see the quote from the slide 28. My suggestion was:

Client – sends SYN, Server – responds with SYN&ACK

  *   In case of SYN_RTO or RTO events Server SHOULD recalculate its TCP socket hash, thus change Flow Label. This behavior MAY be switched on by default;
  *   In case of SYN_RTO or RTO events Client MAY recalculate its TCP socket hash, thus change Flow Label. This behavior MUST be switched off by default;
This looks like a safe default behavior, that saves the part of the improvements, but makes the work with stateful anycast services safe.

And yes, IMO it's ok to have a knob to enable it in the controlled environment. If you ask how to enable it in the presence of internal anycast services - there was also a suggestion in the slides: eBPF. It gives a good way to make this kind of separation.

02.08.2021, 11:48, "Vasilenko Eduard" <[email-masked]<mailto:[email-masked]>>:

Hi Jeff,

Eduard

From: Jeff Tantsura [mailto:[email-masked]]
Sent: Monday, August 2, 2021 6:56 AM
To: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; routing WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

Cheers,

Jeff

Hi  Alexander,

But at the same time

and

Among 3 thins:

-          Anycast

Best Regards

Eduard Vasilenko

Senior Architect

--
Best regards,
Alexander Azimov

---

From: Alexander Azimov <[email-masked]>
Date: Mon, 2 Aug 2021 14:00:54 +0300

Eduard,

> It needs regulation. It needs a standard RFC. Because it kills Anycast
As I'm partially responsible for the key points of the presentation, I can
stress that it is a bit different.

   - We have an opportunity for self-healing TCP on top of IPv6, it should
   be preserved;
   - The Linux defaults should be changed to a safe mode to prevent session
   timeouts;
   - The hash recalculation behavior should be documented;

I'm not sure what you mean by the term 'regulation'.

> IMHO: these are 2 separate drafts.
I'm not sure about it, we'll try to provide -00 before the next IETF
meeting, let's see how it progresses.

> Eduard
>
> *From:* Alexander Azimov [mailto:[email-masked]]
> *Sent:* Monday, August 2, 2021 1:20 PM
> *To:* Vasilenko Eduard <[email-masked]>; Jeff Tantsura <
> [email-masked]>
> *Cc:* routing WG <[email-masked]>
> *Subject:* Re: Self-healing Networking with Flow Label
>
>
>
> Eduard,
>
>
>
>    - In case of SYN_RTO or RTO events Server SHOULD recalculate its TCP
>    socket hash, thus change Flow Label. This behavior MAY be switched on by
>    default;
>    - In case of SYN_RTO or RTO events Client MAY recalculate its TCP
>    socket hash, thus change Flow Label. This behavior MUST be switched off by
>    default;
>
> This looks like a safe default behavior, that saves the part of the
> improvements, but makes the work with stateful anycast services safe.
>
>
>
> And yes, IMO it's ok to have a knob to enable it in the controlled
> environment. If you ask how to enable it in the presence of internal
> anycast services - there was also a suggestion in the slides: eBPF. It
> gives a good way to make this kind of separation.
>
>
>
> 02.08.2021, 11:48, "Vasilenko Eduard" <[email-masked]>:
>
> Hi Jeff,
>
> Yes, RTO in TCP helps to find a work-around for this bug. And yes, Anycast
> is typically absent inside DC – it does not create the problem in the DC
> environment.
>
>
>
> But the same LINUX is used outside DC. RTO Flow Label change here would
> create even more problems if Anycast would happen on the traffic path (not
> much predictable for client).
>
> Or should we rely on the proper non-default configuration for different
> environments? (Admin should not forget to change)
>
> DC Admin has the higher qualification to activate it in a controlled
> environment than every client worldwide that should not forget to disable
> it.
>
>
>
> Eduard
>
> *From:* Jeff Tantsura [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Monday, August 2, 2021 6:56 AM
> *To:* Vasilenko Eduard <[email-masked]>
> *Cc:* [email-masked]; routing WG <[email-masked]>
> *Subject:* Re: Self-healing Networking with Flow Label
>
>
>
> Eduard,
>
>
>
> The issue is present not in link/device case, if well implemented - fast
> rehash takes care of updating forwarding within a number of ms. The problem
> is with  “gray” failures,  when the link in question is UP from
> routing/forwarding prospective but drops traffic (mostly occasionally and
> when a HW bug occurs has a distinct flow attributes).
>
>
>
> In many large DC fabrics, the majority of the traffic is east-west,
> between end-points that aren’t anycast. In such deployments - the solution
> solves  issues rather elegantly and without any interventions from the
> operator.
>

-- 
Best regards,
Alexander Azimov

---

From: Jeff Tantsura <[email-masked]>
Date: Mon, 2 Aug 2021 15:08:09 -0700

Eduard,

The idea of the draft to come is to explain what to do - when and how.
The goal is not to regulate (we really don’t) but to provide, similarly to RFC7938 a set of guidelines that community can use to build better and more resilient networks.

Cheers,
Jeff

> On Aug 2, 2021, at 04:01, Alexander Azimov <[email-masked]> wrote:
> 
> 
> Eduard, 
> 
> As I'm partially responsible for the key points of the presentation, I can stress that it is a bit different.
> We have an opportunity for self-healing TCP on top of IPv6, it should be preserved;
> The Linux defaults should be changed to a safe mode to prevent session timeouts;
> The hash recalculation behavior should be documented;
> I'm not sure about it, we'll try to provide -00 before the next IETF meeting, let's see how it progresses.
>  
>> Eduard
>> 
>> In case of SYN_RTO or RTO events Server SHOULD recalculate its TCP socket hash, thus change Flow Label. This behavior MAY be switched on by default;
>> In case of SYN_RTO or RTO events Client MAY recalculate its TCP socket hash, thus change Flow Label. This behavior MUST be switched off by default;

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 3 Aug 2021 09:03:43 +0000

Hi all,
Not many people worldwide read this alias and understand
That RTO could be leveraged to fight “silent drops” in the DC environment.
It is a good use case to publish/document (with more details that it was in the presentation).
I hope that in the future OAM would be used for this purpose – it is better from architecture point of view.
Eduard
From: Jeff Tantsura [mailto:[email-masked]]
Sent: Tuesday, August 3, 2021 1:08 AM
To: Alexander Azimov <[email-masked]>
Cc: Vasilenko Eduard <[email-masked]>; Alexander Azimov <[email-masked]>; routing WG <[email-masked]>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

Cheers,
Jeff

On Aug 2, 2021, at 04:01, Alexander Azimov <[email-masked]<mailto:[email-masked]>> wrote:

Eduard,

пн, 2 авг. 2021 г. в 13:45, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>:

  *   We have an opportunity for self-healing TCP on top of IPv6, it should be preserved;
  *   The Linux defaults should be changed to a safe mode to prevent session timeouts;
  *   The hash recalculation behavior should be documented;
I'm not sure what you mean by the term 'regulation'.

Eduard
From: Alexander Azimov [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, August 2, 2021 1:20 PM
To: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Jeff Tantsura <[email-masked]<mailto:[email-masked]>>
Cc: routing WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

Hi Jeff,

Eduard

Eduard,

Cheers,

Jeff

Hi  Alexander,

But at the same time

and

Among 3 thins:

-          Anycast

Best Regards

Eduard Vasilenko

Senior Architect

--
Best regards,
Alexander Azimov

--
Best regards,
Alexander Azimov

---

From: Alexander Azimov <[email-masked]>
Date: Mon, 02 Aug 2021 13:20:04 +0300

Eduard,

Hi Jeff,

Eduard

Eduard,

Cheers,

Jeff

Hi  Alexander,

But at the same time

and

Among 3 thins:

-          Anycast

Best Regards

Eduard Vasilenko

Senior Architect

--

Best regards,

Alexander Azimov

---

From: Mark Smith <[email-masked]>
Date: Tue, 17 Aug 2021 20:46:23 +1000

> gives a good way to make this kind of separation.

I think there is a broader and significant issue to consider.

The flow label field is, per the FL RFC, a supplemental informational
field. It's additional information about the traffic flow that the network
might use to provide better service than just the packets' DA.

What this proposal is doing is changing it to being a field that is trying
to actively steer traffic independently of what the network and network
operator's goals and policy might be.

When to you have two independent parties trying to achieve the same or a
similar goal, in an uncoordinated manner, those parties can end up working
against each other. This will result in a worse rather than better outcome.
(This is also the point of RFC5704.)

So here we have a proposal that hosts try to actively steer traffic by
manipulating the flow label, independently of how the network might steer
the traffic. The result may be worse rather than better network service for
the host that is actively trying to manipulate the network.

This proposal needs to go to 6man WG for consideration, since the Flow
Label field's semantics and use is theirs to define.

(RFC 6436 describes rational for updating the Flow Label field to its
current spec. See Appendix A for discussion about the idea of a local or
controlled domain use.)

Regards,
Mark.

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 17 Aug 2021 11:08:15 +0000

+1
I did have the temptation to say the same.
In my mind it is “race conditions” – the result is very dependent on who is faster.
I remember how miserable was goodput of TCP inside X.25 because both have windows for flow control that they operate independently.

In this particular case traffic switching to a different path may immediately follow the next switch to a different path because topology would change.
2 disruptions for the ordinary link fault with different latency, jitter, reordering.

Vendor’s bugs are better to solve my proper monitoring tools.

Eduard
From: Mark Smith [mailto:[email-masked]]
Sent: Tuesday, August 17, 2021 1:46 PM
To: Alexander Azimov <[email-masked]>
Cc: Vasilenko Eduard <[email-masked]>; Jeff Tantsura <[email-masked]>; routing WG <[email-masked]>
Subject: Re: Self-healing Networking with Flow Label

On Tue, 17 Aug 2021, 16:46 Alexander Azimov, <[email-masked]<mailto:[email-masked]>> wrote:
Eduard,

The flow label field is, per the FL RFC, a supplemental informational field. It's additional information about the traffic flow that the network might use to provide better service than just the packets' DA.

What this proposal is doing is changing it to being a field that is trying to actively steer traffic independently of what the network and network operator's goals and policy might be.

When to you have two independent parties trying to achieve the same or a similar goal, in an uncoordinated manner, those parties can end up working against each other. This will result in a worse rather than better outcome. (This is also the point of RFC5704.)

So here we have a proposal that hosts try to actively steer traffic by manipulating the flow label, independently of how the network might steer the traffic. The result may be worse rather than better network service for the host that is actively trying to manipulate the network.

This proposal needs to go to 6man WG for consideration, since the Flow Label field's semantics and use is theirs to define.

(RFC 6436 describes rational for updating the Flow Label field to its current spec. See Appendix A for discussion about the idea of a local or controlled domain use.)

Regards,
Mark.

Hi Jeff,

Eduard

Eduard,

Cheers,

Jeff

Hi  Alexander,

But at the same time

and

Among 3 thins:

-          Anycast

Best Regards

Eduard Vasilenko

Senior Architect

--
Best regards,
Alexander Azimov