Subject: [multipathtcp] Supporting ECMP and load balancers with Multipath TCP

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 9 Sep 2015 23:18:46 +0200

Hello,

Let me continue the discussion about load balancing with a slightly 
different viewpoint than in the previous email. In some deployments 
several load balancers are used and ECMP is deployed to balance the load 
between them. A simplified architecture would be

ECMP ------ LB1 ---- server1
   |          +------ server2
   |
   +-------- LB2 ----- server3
              +------- server4

In this architecture, each LB exposes the same public IP address

This architecture poses a challenge for Multipath TCP since the 
different subflows, typically coming from different source addresses, 
that belong to the same Multipath TCP connection will not be delivered 
to the same LB and thus could reach different servers.

I think that the only way to support this kind of deployment would be to 
attach a public address to each server. By "address", I mean either an 
IP address (in IPv6, this would be easy) or a combination IP address + 
port (in an IPv4 environment having limited number of available public 
addresses). If each server has its own public address, then one 
possibility would be to operate as follows.

configuration

- each server has a public IP address
- all the servers behind one LB belong to the same prefix
- each LB advertises the prefix that contains all its attached servers

a. The SYN with MP_CAPABLE sent by the client enters the ECMP and is 
sent to one LB. We assume that the ECMP device ensures that all packets 
from one TCP connection identified by the classical five tuple are sent 
to the same LB

b. The LB processes the SYN and decides to forward it to one particular 
server. It may add information to the packet (see previous email) or 
perform a specific computation for the token (see Christoph's draft)

c. The server replies with a SYN+ACK and one bit in the MP_CAPABLE 
option that indicates that this IP address of the server (i.e. the one 
of the load balancer) cannot be used to establish subflows

d. The client confirms with a third ACK

e. The server advertises its public address inside an ADD_ADDR option. 
This address advertisement includes one bit that indicates that it can 
be used to establish subflows

When a client sends a SYN with MP_JOIN to the address advertised by the 
server, the routing ensures that it reaches the correct server. ECMP 
works as usual and the LB does not need to be involved in the processing 
of this segment.

What happens if a client sends a SYN+MP_JOIN towards the address of the 
load balancer ? The load balancer simply replies with a RST and rejects 
the subflow. The LB might contact the server that is responsible for 
this connection and suggest the retransmission of the ADD_ADDR option 
that might have been lost.

Let me know if you think that this idea could work. If so, I can try to 
document it in more details. Comments and counter examples from load 
balancing experts are more than welcome

Olivier

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 09 Sep 2015 19:38:13 -0700

Hello,

On 09/09/15 - 23:18:46, Olivier Bonaventure wrote:
> different viewpoint than in the previous email. In some deployments several
> load balancers are used and ECMP is deployed to balance the load between
> them. A simplified architecture would be
> 
> This architecture poses a challenge for Multipath TCP since the different
> subflows, typically coming from different source addresses, that belong to
> the same Multipath TCP connection will not be delivered to the same LB and
> thus could reach different servers.
> 
> attach a public address to each server. By "address", I mean either an IP
> address (in IPv6, this would be easy) or a combination IP address + port (in
> an IPv4 environment having limited number of available public addresses). If
> each server has its own public address, then one possibility would be to
> operate as follows.
> 
> configuration
> 
> a. The SYN with MP_CAPABLE sent by the client enters the ECMP and is sent to
> one LB. We assume that the ECMP device ensures that all packets from one TCP
> connection identified by the classical five tuple are sent to the same LB
> 
> server. It may add information to the packet (see previous email) or perform
> a specific computation for the token (see Christoph's draft)
> 
> c. The server replies with a SYN+ACK and one bit in the MP_CAPABLE option
> that indicates that this IP address of the server (i.e. the one of the load
> balancer) cannot be used to establish subflows

This is similar to the proxy-bit proposed by
draft-wei-mptcp-proxy-mechanism.

> e. The server advertises its public address inside an ADD_ADDR option. This
> address advertisement includes one bit that indicates that it can be used to
> establish subflows
> 
> server, the routing ensures that it reaches the correct server. ECMP works
> as usual and the LB does not need to be involved in the processing of this
> segment.
> 
> What happens if a client sends a SYN+MP_JOIN towards the address of the load
> balancer ? The load balancer simply replies with a RST and rejects the
> subflow. The LB might contact the server that is responsible for this
> connection and suggest the retransmission of the ADD_ADDR option that might
> have been lost.
> 

I believe it does work (and actually would be beneficial as it reduces the
traffic on the loadbalancer as new subflows are directly routed to the
server).

One difficulty might be to have enough public IP-addresses, as changing the
destination port might have other problems (e.g., ISP's rate-limiting or blocking
traffic that looks like bittorrent).

Also, a server now becomes directly addressable and might thus be more
vulnerable to DDoS attacks.

But overall, I think this proposal is also a good tradeoff to
solve this loadbalancer-issue.

Cheers,
Christoph

> 
> 
> Olivier
> 
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_multipathtcp&d=BQICAg&c=eEvniauFctOgLOKGJOplqw&r=fSuJ9mpJ1IB1Lz-sIp_MCe23kZIFT6UKt4EhsewNEmA&m=7oKUbtxwqMtjY3x9J2hbtuTWFbvWo58BIQoQv7olTFM&s=84qAjYvDXiFPdApf6VriNL8UpACuDalDvI24CxwFnyo&e=

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 10 Sep 2015 11:45:10 +0200

Christoph,

> traffic that looks like bittorrent).

This is a difficult with IPv4, but given the growth of IPv6 traffic, we 
should design for IPv6 and accept to have less features in IPv4.

The key application for load balancing with MPTCP will likely be HTTP2 
on top of TLS on top of MPTCP over IPv6. This should be our main 
reference for the design.

> vulnerable to DDoS attacks.

Yes, but you can think of techniques to mitigate such attacks by 
changing the IPv6 addresses of the servers when an attack occurs on one 
of the servers. We have enough addresses in IPv6 to even think about 
IPv6 addresses that are only used for one connection.

Olivier