Subject: [multipathtcp] What are the existing challenges to MultipathTCP Implementation

From: "Appanasamy, Palanivelan" <[email-masked]>
Date: Mon, 25 May 2015 15:25:54 +0530

All,

I appreciate the idea of MultipathTCP, the popularity it got and  the issues it tries solving.
But, what are the challenges that  make implementers hesitant to adopt this? I believe the pace has slowed down immensely.

Perhaps, we should get more data and see where we can improve. Any thoughts or pointers?
Really appreciate some inputs/response. Thanks.

Regards,
A.Palanivelan
DMTS-Engg, Verizon Labs

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 25 May 2015 06:10:52 -0700

Hi Palanivelan,

There're already several mptcp implementations.
I think you might want to take a look at the following draft to get some
rough ideas about implementation status.

https://tools.ietf.org/html/draft-eardley-mptcp-implementations-survey-02

Thanks,
--
Yoshi

On Mon, May 25, 2015 at 2:55 AM, Appanasamy, Palanivelan <
[email-masked]> wrote:

> But, what are the challenges that  make implementers hesitant to adopt
> this? I believe the pace has slowed down immensely.
>
> *DMTS-Engg, Verizon Labs*
>
>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 25 May 2015 21:24:05 +0200

Hello,

> Perhaps, we should get more data and see where we can improve. Any

We see different types of deployments of Multipath TCP.

The first type of deployment is the end-to-end deployment with Apple as 
the example. Apple has implemented Multipath TCP in iOS and uses it for 
the SIRI application. This end-to-end scenario is possible because Apple 
controls the client and the server.

We've seen similar deployments in private networks. For example, a user 
of the Multipath TCP implementation in the Linux kernel has reported 
that he uses Multipath TCP for support the CEPH storage service. 
Apparently, he does this by assigning a loopback address to each storage 
server and one IP address per interface. The loopback addresses are 
advertised by the routing protocol and Multipath TCP does the load 
balancing between the different IP addresses of the physical interfaces. 
An advantage of this setup is that a link failure does not necessarily 
result in a reorganisation of the storage blocks.

On smartphones, we've performed lots of successful test with Android 
smartphones using the Multipath TCP implementation in the Linux kernel. 
This requires the utilisation of a proxy to reach servers running 
Multipath TCP, but this setup works with all the Android applications 
that we have tested. It enables Android smartphones to get higher 
capacity by bundling cellular and WiFi and more importantly allows to 
seamlessly switch form cellular to WiFi. This is important in areas 
where the network operators have deployed large WiFi networks, e.g. by 
providing a community WiFi service on their DSL routers.

Another growing use case is the utilisation of Multipath TCP in 
middleboxes to bond different access links (ADSL+cable, ADSL+cellular).
In this case, a proxy in the CPE converts TCP into Multipath TCP to 
enable the efficient utilisation of the bonded links and another proxy 
inside the network converts Multipath TCP back to regular TCP to reach 
the normal server. There is a growing interest for this solution to 
support the hybrid access network scenario withint the Broadband Forum. 
Several implementations for this solution are being developed, but they 
are not all public

Several companies are also working on bonding with solutions like 
OpenVPN on top of Multipath TCP.

Feel free to contact me offline if you'd like additional details

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: "Ivancic, William D. (GRC-LCA0)" <[email-masked]>
Date: Tue, 26 May 2015 15:03:26 +0000

I have recently been funded to look at some NASA specific problems with
channel bonding Iridium phones - work should begin in July.  I will not
make the Prague IETF, but hope to attend a few others.  Here is how we are
trying to use MPTCP.  If MPTCP works well we "may" consider looking at
developing some type of MPUDP, but that is To Be Determined.  There may be
better / simpler solutions.

 - Will Ivancic

The fundamental problem of
multipath aggregation is the difficulty of fulfilling a range of
conflicting
criteria. For example, it is difficult to schedule IP packets over
heterogeneous paths without causing reordering and performance penalties
at the
transport layer. It is also hard to implement a generic multipath solution
without modifying standardized protocols or changing third-party network
equipment.

A robust multipath aggregation
solution should be able to cope with any kind of path heterogeneity,
throughput
fluctuations, or jitter. Solutions should also be able to deal with
persistent
reordering of data packets. Finally, a complete multipath aggregation
solution
should support and be compatible with all existing transport protocols and
applications.

The current solution (use on NASA's Atmospherice Reseach aircraft) to the
channel multiplexing problem (bonding 4 Iridium modes to increase
bandwidht)  is to use the PPP
Multilink Protocol (PPP-MP). This has been serviceable, but has numerous
flaws that result in less-than optimal operations, costing precious time 
particularly considering the limited time of an aeronautical science
research
flight and the associated costs.  Thus, a
robust, reliable solution is highly desirable.

In order to baseline current
system operations and obtain data necessary to model the existing
implementation (and our proposed solutions), we propose to measure the
current
operations of the iridium channels and modems and model this behavior.
The models will be needed to ensure that we
have a full understanding of the current failure mechanisms and validate
our
hypothesis the PPP-MP is susceptible to high loss and unreliable links.

Next, we propose to study the
use of TCP Extensions for Multipath Operation with Multiple Addresses 
also
know as multipath-TCP (MPTCP)  to perform the channel bonding.  Multipath
TCP (MPTCP) is a set of extensions
to regular TCP to provide a Multipath TCP service, which enables a
transport
connection to operate across multiple paths simultaneously.

Unlike PPP-MP, MPTCP does not
fragment packets across flows and treats each subflow independently. Also,
the
data sequence mapping specifies a mapping from subflow sequence space to
data
sequence space. A mapping is fixed, in that the subflow sequence number is
bound to the data sequence number after the mapping has been processed.
MPTCP
allows the same data sequence number to mapped to by different subflows for
retransmission purposes.  Thus, if one
channel (modem) fails, the failed subflow can be retransmitted over another
path (channel). We believe such operation will result in a robust
communication
system as when on modem looses lock or rests, it should not affect the
other
subflows and MPTCP should automatically adapt.

This protocol specification does
not mandate any mechanisms for handling retransmissions. Rather, this will
be
dependent upon local policy. Aggressive connection-level retransmissions
policies could be implemented where every packet lost at subflow level is
retransmitted on a different subflow (hence, wasting bandwidth but
possibly reducing
application-to-application delays), or conservative retransmission policies
where connection-level retransmits are only used after a few subflow-level
retransmission timeouts occur. This is an area that we will experiment
with.  Having good channel characteristics is also critical for
understanding the trade space(hence the need to first characterize the
Iridium modem performance). Also, Multiple retransmissions are triggers
that  can be used to indicate that a subflow is performing badly and could
lead to a host resetting the subflow. Additional research is
required to understand the heuristics of how and when to reset
underperforming
subflows. For example, a highly asymmetric path may be misdiagnosed as
underperforming, unnecessarily triggering a subflow reset.
 
Once we fully understand MPTCP
applicability to channel bonding, we will utilize that knowledge to
address UPD
communications. 

Aircraft systems utilizing satellite links often utilize UDP packets to
avoid TCP¹s issues with
latency.  Since UPD does not contain
state or explicit feedback, developing a multipath UDP implementation has
some
unique challenges relative to MPTCP.  UDP
is a single packet protocol with no state. There's no relationship between
two
UDP packets - regardless of same/different port or addresses. However, at
the
application level, two UDP packets that take different paths could produce
injury to the application. Thus, reordering and losses will need to be
studied
and addressed. We will also consider the use of the multipath Real-Time
Protocol (MPRTP) as RTP has state and can adopt many of the concepts of
MPTCP.   Lessons learned from our study of MPTCP and
MPRTP will be applied to a new multipath UPD protocol (whereby the subflows
would maintain some minimal state and receive some minimal feedback in
relation
to each subflow path). Finally, an alternative to Multipath UDP (MPUDP)
may be capable
of applying channel coding prior to splitting a UDP flow over multiple
paths 
analogous to a UDP performance enhancing proxy.
It will also be evaluated as well.

Since UDP was originally
designed to be an unreliable transport and we are basically attempting to
make it
reliable (at least at least over the channel-bonded link), we will need to
evaluate our final results to ensure that we have not created something
unwieldy.  Understanding whether we need
to develop a MPUDP implementation or perform a modification and update of
PPP-MP will all be part of the research effort.

---

From: "Appanasamy, Palanivelan" <[email-masked]>
Date: Tue, 26 May 2015 13:43:15 +0530

Thanks Oliver and Nishida...Will talk to you further. Thanks.

Regards,
A.Palanivelan
DMTS-Engg, VerizonLabs
-----Original Message-----
From: Olivier Bonaventure [mailto:[email-masked]]
Sent: Tuesday, May 26, 2015 12:54 AM
To: Appanasamy, Palanivelan; [email-masked]
Subject: Re: [multipathtcp] What are the existing challenges to MultipathTCP Implementation

Hello,

The first type of deployment is the end-to-end deployment with Apple as the example. Apple has implemented Multipath TCP in iOS and uses it for the SIRI application. This end-to-end scenario is possible because Apple controls the client and the server.

We've seen similar deployments in private networks. For example, a user of the Multipath TCP implementation in the Linux kernel has reported that he uses Multipath TCP for support the CEPH storage service.
Apparently, he does this by assigning a loopback address to each storage server and one IP address per interface. The loopback addresses are advertised by the routing protocol and Multipath TCP does the load balancing between the different IP addresses of the physical interfaces.
An advantage of this setup is that a link failure does not necessarily result in a reorganisation of the storage blocks.

On smartphones, we've performed lots of successful test with Android smartphones using the Multipath TCP implementation in the Linux kernel.
This requires the utilisation of a proxy to reach servers running Multipath TCP, but this setup works with all the Android applications that we have tested. It enables Android smartphones to get higher capacity by bundling cellular and WiFi and more importantly allows to seamlessly switch form cellular to WiFi. This is important in areas where the network operators have deployed large WiFi networks, e.g. by providing a community WiFi service on their DSL routers.

Another growing use case is the utilisation of Multipath TCP in middleboxes to bond different access links (ADSL+cable, ADSL+cellular).
In this case, a proxy in the CPE converts TCP into Multipath TCP to enable the efficient utilisation of the bonded links and another proxy inside the network converts Multipath TCP back to regular TCP to reach the normal server. There is a growing interest for this solution to support the hybrid access network scenario withint the Broadband Forum.
Several implementations for this solution are being developed, but they are not all public

Several companies are also working on bonding with solutions like OpenVPN on top of Multipath TCP.

Olivier

---

From: "Appanasamy, Palanivelan" <[email-masked]>
Date: Wed, 27 May 2015 12:20:40 +0530

Hi Will,

Thanks for all the details.  This is exactly what I was looking for and this is very useful.
Should we not document this and add similar inputs from elsewhere... I think that will be really productive for this WG. Thoughts?

Regards,
A.Palanivelan
DMTS-Engg, Verizon Labs

-----Original Message-----
From: Ivancic, William D. (GRC-LCA0) [mailto:[email-masked]]
Sent: Tuesday, May 26, 2015 8:33 PM
To: [email-masked]; Appanasamy, Palanivelan; [email-masked]
Subject: Re: [multipathtcp] What are the existing challenges to MultipathTCP Implementation

I have recently been funded to look at some NASA specific problems with channel bonding Iridium phones - work should begin in July.  I will not make the Prague IETF, but hope to attend a few others.  Here is how we are trying to use MPTCP.  If MPTCP works well we "may" consider looking at developing some type of MPUDP, but that is To Be Determined.  There may be better / simpler solutions.

 - Will Ivancic

The fundamental problem of
multipath aggregation is the difficulty of fulfilling a range of conflicting criteria. For example, it is difficult to schedule IP packets over heterogeneous paths without causing reordering and performance penalties at the transport layer. It is also hard to implement a generic multipath solution without modifying standardized protocols or changing third-party network equipment.

A robust multipath aggregation
solution should be able to cope with any kind of path heterogeneity, throughput fluctuations, or jitter. Solutions should also be able to deal with persistent reordering of data packets. Finally, a complete multipath aggregation solution should support and be compatible with all existing transport protocols and applications.

The current solution (use on NASA's Atmospherice Reseach aircraft) to the channel multiplexing problem (bonding 4 Iridium modes to increase
bandwidht)  is to use the PPP
Multilink Protocol (PPP-MP). This has been serviceable, but has numerous flaws that result in less-than optimal operations, costing precious time  particularly considering the limited time of an aeronautical science research flight and the associated costs.  Thus, a robust, reliable solution is highly desirable.

In order to baseline current
system operations and obtain data necessary to model the existing implementation (and our proposed solutions), we propose to measure the current operations of the iridium channels and modems and model this behavior.
The models will be needed to ensure that we have a full understanding of the current failure mechanisms and validate our hypothesis the PPP-MP is susceptible to high loss and unreliable links.

Next, we propose to study the
use of TCP Extensions for Multipath Operation with Multiple Addresses  also know as multipath-TCP (MPTCP)  to perform the channel bonding.  Multipath TCP (MPTCP) is a set of extensions to regular TCP to provide a Multipath TCP service, which enables a transport connection to operate across multiple paths simultaneously.

Unlike PPP-MP, MPTCP does not
fragment packets across flows and treats each subflow independently. Also, the data sequence mapping specifies a mapping from subflow sequence space to data sequence space. A mapping is fixed, in that the subflow sequence number is bound to the data sequence number after the mapping has been processed.
MPTCP
allows the same data sequence number to mapped to by different subflows for retransmission purposes.  Thus, if one channel (modem) fails, the failed subflow can be retransmitted over another path (channel). We believe such operation will result in a robust communication system as when on modem looses lock or rests, it should not affect the other subflows and MPTCP should automatically adapt.

This protocol specification does
not mandate any mechanisms for handling retransmissions. Rather, this will be dependent upon local policy. Aggressive connection-level retransmissions policies could be implemented where every packet lost at subflow level is retransmitted on a different subflow (hence, wasting bandwidth but possibly reducing application-to-application delays), or conservative retransmission policies where connection-level retransmits are only used after a few subflow-level retransmission timeouts occur. This is an area that we will experiment with.  Having good channel characteristics is also critical for understanding the trade space(hence the need to first characterize the Iridium modem performance). Also, Multiple retransmissions are triggers that  can be used to indicate that a subflow is performing badly and could lead to a host resetting the subflow. Additional research is required to understand the heuristics of how and when to reset underperforming subflows. For example, a highly asymmetric path may be misdiagnosed as underperforming, unnecessarily triggering a subflow reset.

Once we fully understand MPTCP
applicability to channel bonding, we will utilize that knowledge to address UPD communications.

Aircraft systems utilizing satellite links often utilize UDP packets to avoid TCP¹s issues with latency.  Since UPD does not contain state or explicit feedback, developing a multipath UDP implementation has some unique challenges relative to MPTCP.  UDP is a single packet protocol with no state. There's no relationship between two UDP packets - regardless of same/different port or addresses. However, at the application level, two UDP packets that take different paths could produce injury to the application. Thus, reordering and losses will need to be studied and addressed. We will also consider the use of the multipath Real-Time Protocol (MPRTP) as RTP has state and can adopt many of the concepts of
MPTCP.   Lessons learned from our study of MPTCP and
MPRTP will be applied to a new multipath UPD protocol (whereby the subflows would maintain some minimal state and receive some minimal feedback in relation to each subflow path). Finally, an alternative to Multipath UDP (MPUDP) may be capable of applying channel coding prior to splitting a UDP flow over multiple paths  analogous to a UDP performance enhancing proxy.
It will also be evaluated as well.

Since UDP was originally
designed to be an unreliable transport and we are basically attempting to make it reliable (at least at least over the channel-bonded link), we will need to evaluate our final results to ensure that we have not created something unwieldy.  Understanding whether we need to develop a MPUDP implementation or perform a modification and update of PPP-MP will all be part of the research effort.

---

From: "Ivancic, William D. (GRC-LCA0)" <[email-masked]>
Date: Wed, 27 May 2015 13:01:07 +0000

Documenting the various use cases is useful and should help when enhancing the protocol.

Let me know how I can help.  I would certainly be willing to put together a small section on channel-bonding.

Will

From: <Appanasamy>, Palanivelan <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, May 27, 2015 2:50 AM
To: William Ivancic <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [multipathtcp] What are the existing challenges to MultipathTCP Implementation

Hi Will,

Regards,
A.Palanivelan
DMTS-Engg, Verizon Labs

-----Original Message-----
From: Ivancic, William D. (GRC-LCA0) [mailto:[email-masked]]
Sent: Tuesday, May 26, 2015 8:33 PM
To: [email-masked]<mailto:[email-masked]>; Appanasamy, Palanivelan; [email-masked]<mailto:[email-masked]>
Subject: Re: [multipathtcp] What are the existing challenges to MultipathTCP Implementation

- Will Ivancic

Since UDP was originally

---

From: "Appanasamy, Palanivelan" <[email-masked]>
Date: Thu, 28 May 2015 14:59:24 +0530

Wonderful. That will be a good start. I am getting few guys together on this and like to get to meaningful contributions from all those involved. Thanks.

Regards,
A.Palanivelan
DMTS-Engg, Verizon Labs

From: Ivancic, William D. (GRC-LCA0) [mailto:[email-masked]]
Sent: Wednesday, May 27, 2015 6:31 PM
To: Appanasamy, Palanivelan; [email-masked]; [email-masked]
Subject: Re: [multipathtcp] What are the existing challenges to MultipathTCP Implementation
Importance: High

Will

Hi Will,

Regards,
A.Palanivelan
DMTS-Engg, Verizon Labs

-----Original Message-----
From: Ivancic, William D. (GRC-LCA0) [mailto:[email-masked]]
Sent: Tuesday, May 26, 2015 8:33 PM
To: [email-masked]<mailto:[email-masked]>; Appanasamy, Palanivelan; [email-masked]<mailto:[email-masked]>
Subject: Re: [multipathtcp] What are the existing challenges to MultipathTCP Implementation

- Will Ivancic

Since UDP was originally