Subject: [multipathtcp] Fwd: New Version Notification for draft-paasch-mptcp-loadbalancer-00.txt

From: Christoph Paasch <[email-masked]>
Date: Mon, 07 Sep 2015 20:55:57 -0700

Hello,

we have submitted a draft discussing the issues when using MPTCP behind
layer-4 loadbalancers.

It will be presented during the interim meeting on Thursday.

Regards,
Christoph

----- Forwarded message from [email-masked] -----

From: [email-masked]
Date: Mon, 07 Sep 2015 20:45:11 -0700
Subject: New Version Notification for draft-paasch-mptcp-loadbalancer-00.txt

A new version of I-D, draft-paasch-mptcp-loadbalancer-00.txt
has been successfully submitted by Christoph Paasch and posted to the
IETF repository.

Name:		draft-paasch-mptcp-loadbalancer
Revision:	00
Title:		Multipath TCP behind Layer-4 loadbalancers
Document date:	2015-09-07
Group:		Individual Submission
Pages:		8
URL:            https://www.ietf.org/id/draft-paasch-mptcp-loadbalancer-00.txt
Status:         http://datatracker.ietf.org/doc/draft-paasch-mptcp-loadbalancer/
Htmlized:       https://tools.ietf.org/html/draft-paasch-mptcp-loadbalancer-00

Abstract:
   Large webserver farms consist of thousands of frontend proxies that
   serve as endpoints for the TCP and TLS connection and relay traffic
   to the (sometimes distant) backend servers.  Load-balancing across
   those server is done by layer-4 loadbalancers that ensure that a TCP
   flow will always reach the same server.

   Multipath TCP's use of multiple TCP subflows for the transmission of
   the data stream requires those loadbalancers to be aware of MPTCP to
   ensure that all subflows belonging to the same MPTCP connection reach
   the same frontend proxy.  In this document we analyze the challenges
   related to this and suggest a simple modification to the generation
   of the MPTCP-token to overcome those challenges.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

----- End forwarded message -----

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 9 Sep 2015 22:38:51 +0200

Christoph,
> we have submitted a draft discussing the issues when using MPTCP behind

Thanks for bringing this to the list. This is indeed an important 
problem. As I might be late during tomorrow's telco, let me try to 
answer by email.

A first point that we might want to clarify in the document is how ECMP 
works with Multipath TCP and regular TCP.

With regular TCP, some load balancers behave in a purely stateless 
manner and compute a hash over the five-tuple in each packet to 
distribute the load among a set of servers.

ECMP --- server1
   +----- server2

With single homed and single stack clients, this ECMP solution ensures 
that all the connections sent by one host (i.e. one IP address) will be 
received by the same server if the hashing is independant of the source 
port. Otherwise, connections from the same host might hit different servers.

If the clients are either multihomed (e.g. smartphones) or dual-stack, 
then using a hash-based load balancing that does not include the source 
port in the computation does not ensure that all the connections from 
one client will hit the same server.

With Multipath TCP, a single connection may use different subflows and 
each subflow has its own five-tuple. When ECMP is used, then subflows 
from the same Multipath TCP connection might hit different servers.

Let us now consider stateful load balancers (LB). For simplicity, we 
assume that the load balancer maintains a table that maps each TCP 
connection, identified by a five-tuple on a given server. The table 
might also be used to perform NAT. The LB can use any technique to build 
this table (random, RR, load-based solutions, ...). As an example, let 
us consider a load balancer that resieds in front of N servers as shown 
below :

LB ----- server1
  +------- server2

When the LB receives a SYN with the MP_CAPABLE option from a client, it 
must insert an entry for the corresponding five-tuple inside its table. 
For regular TCP, this would be sufficient. For Multipath TCP, this is 
not enough since it needs to ensure that all the subflows that belong to 
this connection will be forwarded to the same server. In your draft, you 
propose to change the way tokens are computed and discuss different ways 
to perform the handshake. Let me suggest a different approach that might 
be worth to compare.

To support Multipath TCP efficiently in this setup, we need to ensure 
the following properties for the tokens that are used (on the server side) :
  - tokens are unique (at time t, server1 and server2 cannot use the 
same token for different connections)
  - tokens are known by both the LB and the server that is handling for 
the connection

Instead of computing the keys (and thus indirectly the token) on the 
servers, why not selecting the keys for the MP_CAPABLE option in the 
load balancer ?

Let us consider the following scenario :

      client                       LB              server

  SYN + MP_CAPABLE (K_A)
----------------------->
(client sends normal key)

                        SYN + MP_CAPABLE (K_A,K_B)
                        ----------------------->
                      (LB computes key for server)

                                    SYN/ACK + MP_CAPABLE (K_B)
                                  <---------------------------
                                   (normal MPTCP segment)

ACK + MP_CAPABLE_ACK (K_A, K_B)
-------------------------------->
(no change to this ack)

Since the LB computes the key that will be used by the server, it can 
ensure that the (token extracted from the key) is unique and also 
remember it for future subflows to pin them to the good server. There 
are probably different techniques that the LB could use to generate 
these keys on behalf of the server, but I don't think that they need to 
be specified in an IETF document. It's up to each implementor to find an 
efficient way to do that.

One issue that needs to be discussed is how the LB can convey the key to 
the server in the SYN packet. There are different techniques that are 
possible and we should probably select one to ensure the 
interoperability between a LB and servers :
- add K_B to the MP_CAPABLE option, but this is likely to be beyond the 
maximum length of the TCP options
- add K_B in the payload of the SYN segment (but we'll need to check the 
impact on TFO)
- encapsulate the SYN segment inside something, like a UDP segment with 
a TLV format that has enough space for a SYN segment and the additional key
- invent a hack to encode the 64 bits key inside some "unused" fields of 
the IP/TCP headers (say IP id, TCP ack number, IPv6 flowlabel, TCP 
timestamp echo, ...)
- ...

At this stage, I'd opt for an encapsulation which seems to be cleaner to 
me. It also forces the servers to be aware of the presence of the LB, 
which seems a useful feature to me.

Comments are, of course, welcome

Olivier

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 09 Sep 2015 19:21:51 -0700

Hello Olivier,

On 09/09/15 - 22:38:51, Olivier Bonaventure wrote:
> Thanks for bringing this to the list. This is indeed an important problem.
> As I might be late during tomorrow's telco, let me try to answer by email.
> 
> With regular TCP, some load balancers behave in a purely stateless manner
> and compute a hash over the five-tuple in each packet to distribute the load
> among a set of servers.
> 
> ECMP --- server1
>   +----- server2
> 
> 
> With single homed and single stack clients, this ECMP solution ensures that
> all the connections sent by one host (i.e. one IP address) will be received
> by the same server if the hashing is independant of the source port.
> Otherwise, connections from the same host might hit different servers.
> 
> If the clients are either multihomed (e.g. smartphones) or dual-stack, then
> using a hash-based load balancing that does not include the source port in
> the computation does not ensure that all the connections from one client
> will hit the same server.
> 
> With Multipath TCP, a single connection may use different subflows and each
> subflow has its own five-tuple. When ECMP is used, then subflows from the
> same Multipath TCP connection might hit different servers.
> 
> Let us now consider stateful load balancers (LB). For simplicity, we assume
> that the load balancer maintains a table that maps each TCP connection,
> identified by a five-tuple on a given server. The table might also be used
> to perform NAT. The LB can use any technique to build this table (random,
> RR, load-based solutions, ...). As an example, let us consider a load
> balancer that resieds in front of N servers as shown below :
> 
> 
> LB ----- server1
>  +------- server2
> 
> 
> When the LB receives a SYN with the MP_CAPABLE option from a client, it must
> insert an entry for the corresponding five-tuple inside its table. For
> regular TCP, this would be sufficient. For Multipath TCP, this is not enough
> since it needs to ensure that all the subflows that belong to this
> connection will be forwarded to the same server. In your draft, you propose
> to change the way tokens are computed and discuss different ways to perform
> the handshake. Let me suggest a different approach that might be worth to
> compare.
> 
> To support Multipath TCP efficiently in this setup, we need to ensure the
> following properties for the tokens that are used (on the server side) :
>  - tokens are unique (at time t, server1 and server2 cannot use the same
> token for different connections)
>  - tokens are known by both the LB and the server that is handling for the
> connection
> 
> servers, why not selecting the keys for the MP_CAPABLE option in the load
> balancer ?
> 
> Since the LB computes the key that will be used by the server, it can ensure
> that the (token extracted from the key) is unique and also remember it for
> future subflows to pin them to the good server. There are probably different
> techniques that the LB could use to generate these keys on behalf of the
> server, but I don't think that they need to be specified in an IETF
> document. It's up to each implementor to find an efficient way to do that.
> 
> One issue that needs to be discussed is how the LB can convey the key to the
> server in the SYN packet. There are different techniques that are possible
> and we should probably select one to ensure the interoperability between a
> LB and servers :
> - encapsulate the SYN segment inside something, like a UDP segment with a
> TLV format that has enough space for a SYN segment and the additional key
> - invent a hack to encode the 64 bits key inside some "unused" fields of the
> IP/TCP headers (say IP id, TCP ack number, IPv6 flowlabel, TCP timestamp
> echo, ...)
> - ...
> 
> At this stage, I'd opt for an encapsulation which seems to be cleaner to me.
> It also forces the servers to be aware of the presence of the LB, which
> seems a useful feature to me.
> 

I really like this idea.

Very often, IP-in-IP encapsulation is done from the loadbalancer to the
server. Thus, the key could even be placed in an IP-option. As everything is
local to the data-center there shouldn't be an issue with adding a new
IP-option.

But as you point out in the other mail, when multiple loadbalancers are
being used we have an issue again. Maybe, there is a way to generate the key
in such a way that we can guarantee uniqueness across the loadbalancers?

Cheers,
Christoph

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 10 Sep 2015 12:03:50 +0200

Christoph,
> in such a way that we can guarantee uniqueness across the loadbalancers?

This is one possibility, but I'm not sure that we need to specify that 
unless we want load balancers from different vendors to cooperate.

The load balancing problem could be an opportunity to rethink the 
three-way handshake used by Multipath TCP so that it can work well with 
tomorrow's target protocol stack, likely HTTP/2 / TLS / MPTCP / IPv6

To deal with security issues, we have included keys in clear inside the 
SYN and SYN+ACK segments. Then the limited TCP option space has forced 
us to use hash functions to compute the IDSN and the tokens. Using a 
hash to compute the tokens creates difficulties when one wants to 
coordinate token allocations between a set of load balancers.

I wonder whether we should not explore again the work documented in
https://tools.ietf.org/html/draft-paasch-mptcp-lowoverhead-00
and
https://tools.ietf.org/html/draft-paasch-mptcp-ssl-00

Let us first ignore the security issues and make the toke explicit in 
the handshake.

                Host A                               Host B
               ----------                           ----------
               Address A1                           Address B1
               ----------                           ----------
                   |                                    |
                   |  SYN+MP_CAPABLE(Token-A)           |
                   |----------------------------------->|
                   |                                    |
                   |SYN/ACK+MP_CAPABLE(Token-B)         |
                   |<-----------------------------------|
                   |                                    |
                   |  ACK+MP_CAPABLE(Token-A, Token-B)  |
                   |----------------------------------->|

It might also be possible to remove Token-A from the initial SYN since 
there is little benefit from the server to know this token at this 
stage. The IDSN could be derived as TokenA|TokenB in one direction and 
TokenB|TokenA in the other or better a hash of these values.

The benefit of this approach is that the Token chosen by the server is 
not explicitelt sent in the SYN/ACK. Load balancers can coordinate to 
have one pool of tokens for each LB and pass them to the servers.

The next step is to authenticate the establishment of subflows. The 
best approach with HTTP/2 / TLS  is clearly to derive MPTCP keys from 
the master TLS key and use them to authenticate the subflows with the 
HMAC mechanism already defined. With this, we leverage the security 
properties of TLS and do not anymore depend on "keys" that are sent in 
clear in the SYN segments and consume precious option space. Of course, 
a drawback of this approach is that you cannot create a subflow until 
the end of the TLS handshake, but TLS is moving towards faster handshakes.

Olivier

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 10 Sep 2015 07:43:18 -0700

> unless we want load balancers from different vendors to cooperate.

I was more thinking of it in case we do not modify anything in the
MPTCP-handshake. In that case, the different loadbalancers will be
generating the keys independently. The loadbalancers can easily
make sure that they generate different keys, however because the token is a
hash of the key, collisions are possible.

Thus, we would need a way to guarantee uniqueness of the token across the
loadbalancers.

> The load balancing problem could be an opportunity to rethink the three-way
> handshake used by Multipath TCP so that it can work well with tomorrow's
> target protocol stack, likely HTTP/2 / TLS / MPTCP / IPv6

+1 on TLS

For IPv6 I'm a bit concerned because we are yet far away from full IPv6
deployment and IPv4 is proably never going away.

> To deal with security issues, we have included keys in clear inside the SYN
> and SYN+ACK segments. Then the limited TCP option space has forced us to use
> hash functions to compute the IDSN and the tokens. Using a hash to compute
> the tokens creates difficulties when one wants to coordinate token
> allocations between a set of load balancers.
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dpaasch-2Dmptcp-2Dlowoverhead-2D00&d=BQIC-g&c=eEvniauFctOgLOKGJOplqw&r=fSuJ9mpJ1IB1Lz-sIp_MCe23kZIFT6UKt4EhsewNEmA&m=d1DAUSnnbQyUTC9DUvPR9YgBw1illvfDq55MXrewq0M&s=2Xd614qho-KOJFy_LZtqx000U8div7FG31UhLM9V6WY&e=
> and
> https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dpaasch-2Dmptcp-2Dssl-2D00&d=BQIC-g&c=eEvniauFctOgLOKGJOplqw&r=fSuJ9mpJ1IB1Lz-sIp_MCe23kZIFT6UKt4EhsewNEmA&m=d1DAUSnnbQyUTC9DUvPR9YgBw1illvfDq55MXrewq0M&s=SiCZMS9Xpc2g-z1tPdqDgzwTK2evfjTpi-2OBM4Z9R4&e=
> 
> 
> Let us first ignore the security issues and make the toke explicit in the
> handshake.
> 
> 
>                Host A                               Host B
>               ----------                           ----------
>               Address A1                           Address B1
>               ----------                           ----------
>                   |                                    |
> It might also be possible to remove Token-A from the initial SYN since there
> is little benefit from the server to know this token at this stage.

Yes, if the MP_CAPABLE becomes reliable in the third ACK, we can probably
remove the token from the SYN (although we have to be creative for TFO in
that case).

> The IDSN
> could be derived as TokenA|TokenB in one direction and TokenB|TokenA in the
> other or better a hash of these values.
> 
> The benefit of this approach is that the Token chosen by the server is not
> explicitelt sent in the SYN/ACK. Load balancers can coordinate to have one
> pool of tokens for each LB and pass them to the servers.
> 
> The next step is to authenticate the establishment of subflows. The best
> approach with HTTP/2 / TLS  is clearly to derive MPTCP keys from the master
> TLS key and use them to authenticate the subflows with the HMAC mechanism
> already defined. With this, we leverage the security properties of TLS and
> do not anymore depend on "keys" that are sent in clear in the SYN segments
> and consume precious option space. Of course, a drawback of this approach is
> that you cannot create a subflow until the end of the TLS handshake, but TLS
> is moving towards faster handshakes.

I think this is the best way to go (if the working-group is fine with
introducing a rather big change to RFC6824bis).

It frees up space in the handshake, enables MPTCP behind loadbalancers
(independent of IPv4 or IPv6) and also gives incentives to enable TLS/SSL.
Additionally to that, it enables low-overhead deployments of MPTCP for
data-center environments.

Christoph