Subject: Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

From: Rao Shoaib <[email-masked]>
Date: Wed, 1 Jun 2016 09:15:50 -0700

On 05/30/2016 02:22 AM, [email-masked] wrote:
> Hi Shoaib,
>
> Please see inline.
>
> Cheers,
> Med
>
>> -----Message d'origine-----
>> De : multipathtcp [mailto:[email-masked]] De la part de Rao
>> Shoaib
>> Envoyé : samedi 28 mai 2016 10:56
>> À : [email-masked]; [email-masked]
>> Objet : Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07
>>
>>
>>
>> On 05/25/2016 12:07 PM, Olivier Bonaventure wrote:
>>> Rao,
>>>> On 05/25/2016 12:50 AM, Olivier Bonaventure wrote:
>>>>> The assumption is that this solution will be deployed in controlled
>>>>> environments and only used between the CPE and the Concentrator. Since
>>>>> the operator controls the two networks that are bound together, he can
>>>>> verify that there are no middleboxes that drop the PM option.
>>>>>
>>>>>
>>>>> Olivier
>>>> I am extremely interested in encapsulating UDP in MPTCP but this
>>>> solution is very specific and hence why should it even be standardized
>> ?
>>> There is a clear demand from operators for a solution to this problem
>>> in hybrid access networks.
>> OK but this solution is very specific to the mobile network.
> [Med] The solution is not specific to the mobile network, but targets all network providers who want to allow their customers to make use of all available network attachments (e.g., fixed, WLAN, cellular, etc.)
The way it is designed it is very specific to a particular use case and 
is not a generic encapsulation solution.
>
>   We would
>> prefer a generic encapsulation solution which has no address mapping.
> [Med] It seems that you want to cover the case of an MPTCP-enabled host communicating an MPTCP-enabled server exchanging any protocol data inside an MPTCP connection? Can you please confirm/infirm?
Yes. We want a generic MPTCP encapsulation solution with no address 
mapping. Address mapping and other features (like single connection) 
should be separate options.
>> Maybe the address mapping part should be made optional based on a flag ?
> [Med] The technical details can be worked out once we understand your deployment case.
>
> [snip]
OK.
>
>
> After re-reading the draft I think the proposal for encapsulation is
> pretty good, only minor details need to be cleared up.
>
> [Med] Thank you. We are more than open to discuss those details.
>
>
Great. I think first we need to decide if we want a generic 
encapsulation solution (which is why I want) or do we entangle it with 
the other requirements of the cellular deployment.

Shoaib

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 2 Jun 2016 20:02:22 -0700

Hi Med,

My question was if we can do the same or similar thing at the application
level, like create a mapping between a TCP/UDP socket and a MPTCP socket.
It doesn't have to be socks. Also, as socks5 supports UDP, the description
might not be very precise.

Thanks,
--
Yoshi

On Mon, May 30, 2016 at 1:58 AM, <[email-masked]> wrote:

> Re-,
>
>
>
> Please see inline.
>
>
>
> Cheers,
>
> Med
>
>
>
> *De :* Yoshifumi Nishida [mailto:[email-masked]]
> *Envoyé :* samedi 28 mai 2016 02:09
> *À :* Olivier Bonaventure
> *Cc :* Yoshifumi Nishida; Olivier Bonaventure; BOUCADAIR Mohamed IMT/OLN;
> MultiPath WG; [email-masked]
> *Objet :* Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07
>
>
>
> Hi Olivier,
>
>
>
> On Fri, May 27, 2016 at 12:38 AM, Olivier Bonaventure <
> [email-masked]> wrote:
>
> Yoshifumi,
>
>
>
>     Since the mapping is between a TCP connection (UDP flow) and an
>     MPTCP connection, it applies for all the subflows of the MPTCP
>     connection automatically. I'm not in favor of changing the mapping
>     during the lifetime of an MPTCP connection, it leads to
>     implementation complexity that are not worth the effort given that
>     we can establish another MPTCP connection bound to another flow.
>
>
> I see. Sorry. I might be confused by the name of Concentrator..
>
> Then, my another question is if we can do the same thing with
> application level.
> I am wondering if we can write an application gateway for CPE and
> Concentrator to send mapping information just like PM option, (and do
> some segmentation tasks as well) so that we can mitigate middlebox
> interventions.
>
>
> The closest solution in the application level is SOCKS. This is what KT is
> doing in Korea and what OVH is deploying for their OverTheBox solution.
> Both do this on top of MPTCP.
>
>
>
> I might want to understand pros and cons for the proposal
>
>
>
> [Med] Below is provided an excerpt from the draft:
>
>
>
>    o  No encapsulation required (no tunnels, whatsoever).
>
>    o  No out-of-band signaling for each MPTCP subflow required.
>
>    o  Carries any protocol (incl.  UDP) for the benefit of massive MPTCP
>
>       adoption (Section 5 <https://tools.ietf.org/html/draft-boucadair-mptcp-plain-mode-07#section-5>).
>
>    o  Accommodates various deployment contexts (Section 6 <https://tools.ietf.org/html/draft-boucadair-mptcp-plain-mode-07#section-6>).
>
>
>
> Which can be interpreted as follows:
>
> ·         No chatty: the solution does not overload the concentrator with
> out-of-banding signaling messages.
>
> ·         No extra delay to establish MPTCP connections.
>
> ·         Supports UDP
>
> ·         Allows to share the same external IP address among various
> subscribers (deployments suffering from IPv4 address depletion)
>
> ·         Allows to assign a single IP address for connections that
> belong to the same subscriber (deployments with no pressure on IPv4 address
> usage)
>
> ·         Allows to preserve the source IPv6 address
>
> ·         …
>
>
>
> The draft acknowledges the overhead for UDP.
>
>
>
> and others.
>
> It would be good to have this sort of discussions in the draft.
>
>
>
> [Med] We already have some text in the draft about the cons of using,
> e.g., SOCKS:
>
>
>
>    Current operational MPTCP deployments by network operators are
>
>    focused on the forwarding of TCP traffic.  In addition, the design of
>
>    such deployments sometimes assumes the use of extra signalling
>
>    provided by SOCKS [RFC1928 <https://tools.ietf.org/html/rfc1928>], at
> the cost of additional management
>
>    complexity and possible service degradation (e.g., up to 8 SOCKS
>
>    messages may need to be exchanged between two MPTCP proxies before an
>
>    MPTCP connection is established, thereby yielding several tens of
>
>    milliseconds of extra delay before the connection is established).
>
> Further, a SOCKS-based solution does not allow to support natively UDP
> bonding.
>
>
>
> Thanks,
>
> --
>
> Yoshi
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 2 Jun 2016 20:03:46 -0700

Hi Med,

On Mon, May 30, 2016 at 1:48 AM, <[email-masked]> wrote:

> Hi Yoshi,
> *Envoyé :* jeudi 26 mai 2016 08:31
> *Cc :* Yoshifumi Nishida; BOUCADAIR Mohamed IMT/OLN; MultiPath WG;
> [email-masked]
> On Wed, May 25, 2016 at 12:50 AM, Olivier Bonaventure <
> [email-masked]> wrote:
>
> Yoshifumi,
>
>
> I might want to see more feedback before the adoption call.
> I've read the draft, but I think I still have some unclear points in the
> draft.
>
> I particularly would like to understand the following points as starting
> point. Please let me know if I miss something.
>
> 1: This method requires CPEs to convert UDP or TCP into MPTCP and
> requires Concentrators to convert back MPTCP into UDP or TCP. This could
> be rather complex process than other approach such as GRE. So, what's
> the benefits to utilize MPTCP here? I think the draft should describe
> some useful scenarios to use MPTCP.
>
>
> There is already operational experience with this type of solution and it
> turns out that using MPTCP to support TCP traffic in hybrid access networks
> is a much better approach than using a tunneling technique like GRE. In an
> hybrid access network such as DSL+LTE, there are two types of problems that
> need to be tackled :
>
> 1. How to hide the different addresses used on the DSL and LTE networks
> 2. How to achieve efficient load balancing over networks having different
> characteristics (delay, bandwidth, packet losses) that change dynamically
>
> GRE solves the first problem in a clean manner. However, it causes two
> issues when you consider the second problem :
>
> a. Since paths have different delays, doing per packet load balancing
> creates massive reording that results in bad TCP performance
> b. Since the two underlying paths are hidden by encapsulation, TCP's
> congestion control scheme cannot probe the different paths and adjust ths
> congestion window correctl. This results in bad TCP performance as well.
>
> An MPTCP-based solution solves these problems in a clean manner since
> MPTCP sees the different paths and can use them efficiently.
>
> You can find some additional information in the Tessares whitepaper
> published last year
>
> http://www.tessares.net/white-papers/
>
> There is already operational experience with this type of MPTCP-based
> solutions and if you think that this would be useful, we could organise a
> presentation at the next IETF meeting on this experience
>
>
>
> I think I understand general idea.
>
> But, as far as I read the draft, I got an impression that a UDP or a TCP
> flow will be mapped to a single subflow of a mptcp session.
>
>
>
> [Med] The draft says the following:
>
>
>
> Section 5.1:
>
>
>
> The data carried in UDP datagrams belonging to
>
>    a given UDP flow are therefore transported in an MPTCP connection.
>
>
>
>    The management of MPTCP connections that are triggered by UDP
>
>    datagrams follows the guidelines documented in [RFC6824 <https://tools.ietf.org/html/rfc6824>].
>
>
>
> Section 5.2:
>
>
>
>    A flow example is shown in Figure 5 to illustrate how TCP packets are
>
>    generated to relay UDP datagrams using several subflows.  Non-SYN
>
>    messages that belong to a given subflow do not include any PM option.
>
>    Also, this example shows how subsequent UDP datagrams of this flow
>
>    are transported over the existing subflow or how a new subflow is
>
>    created.  In this example, the SYN segment issued to add a new
>
>    subflow also includes data received in a UDP datagram.
>
>
>
> and
>
>
>
>    Figure 6 shows an example of UDP datagrams that are transported over
>
>    MPTCP subflows.  Unlike the previous example, additional subflows to
>
>    transport UDP datagrams of the same flow are established in advance
>
>    between the CPE and the concentrator.
>
>
>
> Do you think additional text is needed to make it clear? Thank you.
>

Well, It is still not very clear to me how each UDP packet is mapped to a
mptcp session.
How do you distinguish each UDP packet ? Is it based on 4-tuple or others?
The notion of Internal transport session identifier is a bit ambiguous.
If 4-tuple is not used, how do you construct packets when you send back
them at CPE?

>
>
> There is no description about using multiple subflows for a single TCP/UDP
> flow, or updating subflow mappings in the middle of a session.
>
>
>
> [Med] Please refer to Figures 5 & 6.
>
>
>
> I think the draft should mention these cases.
>
>
>
>
> 2: Including data in SYN is allowed in RFC793, but I'm not very sure how
> many implementations actually accept it.
>
>
> Well, the deployment of TFO shows that this is possible.
>
> I think the draft should
> discuss this point. Also, what will happen if PM option is included in
> the payload, but it is removed by the middleboxes?
>
>
> environments and only used between the CPE and the Concentrator. Since the
> operator controls the two networks that are bound together, he can verify
> that there are no middleboxes that drop the PM option.
>
>
>
> Hmm. then, I think these presumptions should be listed in the draft.
>
>
>
> [Med] These assumptions are already called out in the draft, e.g.,
>
>
>
> Section 1:
>
>
>
> Also, it assumes the various
>
>    network attachments provided to an MPTCP-enabled CPE are managed by
>
>    the same administrative entity.
>
>
>
> Section 4.2:
>
>
>
> Given
>
>       that this option is designed to be used in a controlled
>
>       environment, this specification recommends to always place the PM
>
>       options inside the payload of a SYN segment.
>
>
>
> Do you think additional text is still required?
>

hmm, even though it is designed to be used in a certain environment, what
if it is used in other situations?
I personally think the draft is a bit ambiguous on this point.
--
Yoshi

---

From: <[email-masked]>
Date: Fri, 3 Jun 2016 05:43:33 +0000

Hi Yoshi,

Yes, SOCKS5 supports UDP … but for signgle paths. Extensions are required so that packets from a given host/CPE sent over multiple paths (i.e., with distinct source IP addresses) are “glued” so that the same external IP address/port is used by the SOCKS server. Also, the SOCKS out-of-banding signaling will induce an extra delay for handling UDP flows. This is suboptimal.

Thank you.

Cheers,
Med

De : Yoshifumi Nishida [mailto:[email-masked]]
Envoyé : vendredi 3 juin 2016 05:02
À : BOUCADAIR Mohamed IMT/OLN
Cc : Yoshifumi Nishida; Olivier Bonaventure; Olivier Bonaventure; MultiPath WG; [email-masked]
Objet : Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

Hi Med,

My question was if we can do the same or similar thing at the application level, like create a mapping between a TCP/UDP socket and a MPTCP socket. It doesn't have to be socks. Also, as socks5 supports UDP, the description might not be very precise.

Thanks,
--
Yoshi

On Mon, May 30, 2016 at 1:58 AM, <[email-masked]<mailto:[email-masked]>> wrote:
Re-,

Please see inline.

Cheers,
Med

De : Yoshifumi Nishida [mailto:[email-masked]<mailto:[email-masked]>]
Envoyé : samedi 28 mai 2016 02:09
À : Olivier Bonaventure
Cc : Yoshifumi Nishida; Olivier Bonaventure; BOUCADAIR Mohamed IMT/OLN; MultiPath WG; [email-masked]<mailto:[email-masked]>
Objet : Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

Hi Olivier,

On Fri, May 27, 2016 at 12:38 AM, Olivier Bonaventure <[email-masked]<mailto:[email-masked]>> wrote:
Yoshifumi,

The closest solution in the application level is SOCKS. This is what KT is doing in Korea and what OVH is deploying for their OverTheBox solution. Both do this on top of MPTCP.

      adoption (Section 5<https://tools.ietf.org/html/draft-boucadair-mptcp-plain-mode-07#section-5>).

   o  Accommodates various deployment contexts (Section 6<https://tools.ietf.org/html/draft-boucadair-mptcp-plain-mode-07#section-6>).

•         No chatty: the solution does not overload the concentrator with out-of-banding signaling messages.

•         No extra delay to establish MPTCP connections.

•         Supports UDP

•         Allows to share the same external IP address among various subscribers (deployments suffering from IPv4 address depletion)

•         Allows to assign a single IP address for connections that belong to the same subscriber (deployments with no pressure on IPv4 address usage)

•         Allows to preserve the source IPv6 address

•         …

and others.

[Med] We already have some text in the draft about the cons of using, e.g., SOCKS:

   Current operational MPTCP deployments by network operators are
   focused on the forwarding of TCP traffic.  In addition, the design of
   such deployments sometimes assumes the use of extra signalling
   provided by SOCKS [RFC1928<https://tools.ietf.org/html/rfc1928>], at the cost of additional management
   complexity and possible service degradation (e.g., up to 8 SOCKS
   messages may need to be exchanged between two MPTCP proxies before an
   MPTCP connection is established, thereby yielding several tens of
   milliseconds of extra delay before the connection is established).
Further, a SOCKS-based solution does not allow to support natively UDP bonding.

Thanks,
--
Yoshi

---

From: <[email-masked]>
Date: Fri, 3 Jun 2016 06:22:24 +0000

Re-,

Please see inline.

Cheers,
Med

De : Yoshifumi Nishida [mailto:[email-masked]]
Envoyé : vendredi 3 juin 2016 05:04
À : BOUCADAIR Mohamed IMT/OLN
Cc : Yoshifumi Nishida; Olivier Bonaventure; MultiPath WG; [email-masked]
Objet : Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

Hi Med,

On Mon, May 30, 2016 at 1:48 AM, <[email-masked]<mailto:[email-masked]>> wrote:
Hi Yoshi,
[SNIP]

Well, It is still not very clear to me how each UDP packet is mapped to a mptcp session.
How do you distinguish each UDP packet ? Is it based on 4-tuple or others?
[Med] It is based typically on the source IP address, source port, destination IP address, and destination port.

The notion of Internal transport session identifier is a bit ambiguous.
[Med] This is how a given flow is identified in the internal side of the CPE (i.e., LAN). This is typically, the 4-tuple of the packet as received from a host in the LAN. The 4-tuple is augmented with the “transport protocol” information because the CPE can assign the same external port for TCP and UDP.

---

From: <[email-masked]>
Date: Fri, 3 Jun 2016 07:41:51 +0000

Hi Shoaib,

I'm sure you have good reasons why TCP is not used between the client and the server in the first place. Calling out those reasons is helpful. 

As a side note, the plain mode specification does not enforce (yet) a validation check to assess whether the address conveyed in the option is distinct from an address that belong to the MPTCP node that receives the option. I.e., if that node receives an option that includes ones of its IP addresses, it is an indication that the data is to be consumed locally, not forwarded outside. Wouldn't that solve the first part of your "address mapping" concern? 
 
BTW, I added a note to the draft to indicate the following:

* An implementation must ignore PM options that include multicast, broadcast, and host loopback addresses.

Cheers,
Med

> De : Rao Shoaib [mailto:[email-masked]]
> Envoyé : mercredi 1 juin 2016 18:16
> À : BOUCADAIR Mohamed IMT/OLN; [email-masked];
> [email-masked]
> > [Med] The solution is not specific to the mobile network, but targets
> all network providers who want to allow their customers to make use of all
> available network attachments (e.g., fixed, WLAN, cellular, etc.)
> > [Med] It seems that you want to cover the case of an MPTCP-enabled host
> communicating an MPTCP-enabled server exchanging any protocol data inside
> an MPTCP connection? Can you please confirm/infirm?

---

From: Rao Shoaib <[email-masked]>
Date: Fri, 3 Jun 2016 00:53:22 -0700

> I'm sure you have good reasons why TCP is not used between the client and the server in the first place. Calling out those reasons is helpful.
MPTCP allows us to break a large UDP message into several segments and 
use multiple paths, Plus the added reliability. This reminds me of 
another issue, what happens if a fragment other than the 1st one that 
carries the header shows up first ? We need a header in each packet that 
indicates the order and association just like IP fragmentation.
> As a side note, the plain mode specification does not enforce (yet) a validation check to assess whether the address conveyed in the option is distinct from an address that belong to the MPTCP node…
Correct but why would I even want to send an IP address ? Why can't 
there be a flag to indicate that this packet carries no address 
translation information. Ideally encapsulation should be an option on 
it's own as it has nothing to do with address translation.

Shoaib

---

From: Rao Shoaib <[email-masked]>
Date: Fri, 3 Jun 2016 01:00:13 -0700

BTW another use case we have is to use multiple TCP flows for different 
priority messages. In such a deployment we would not divide a single UDP 
packet and send it over multiple TCP streams but the complete message 
will be sent on a single stream based on the priority. This would avoid 
head of line blocking due to large messages for small high priority 
messages.

Shoaib

On 06/03/2016 12:53 AM, Rao Shoaib wrote:
>> I'm sure you have good reasons why TCP is not used between the client 
>> and the server in the first place. Calling out those reasons is helpful.
> carries the header shows up first ? We need a header in each packet 
> that indicates the order and association just like IP fragmentation.
>>
>> As a side note, the plain mode specification does not enforce (yet) a 
>> validation check to assess whether the address conveyed in the option 
>> is distinct from an address that belong to the MPTCP node that 
>> receives the option. I.e., if that node receives an option that 
>> includes ones of its IP addresses, it is an indication that the data 
>> is to be consumed locally, not forwarded outside. Wouldn't that solve 
>> the first part of your "address mapping" concern?
>>>>> De : multipathtcp [mailto:[email-masked]] De la 
>>>>> part de
>>> Rao
>>>>> Shoaib
>>>>> Maybe the address mapping part should be made optional based on a 
>>>>> flag
>>> ?

---

From: <[email-masked]>
Date: Fri, 3 Jun 2016 08:39:09 +0000

Re-,

Please see inline.

Cheers,
Med

> Envoyé : vendredi 3 juin 2016 09:53
> use multiple paths, Plus the added reliability. 

[Med] Ok, thanks.

This reminds me of
> carries the header shows up first ?

[Med] This is an issue for IPv4. In order to accommodate our of order fragments, the implementation caches those till the first fragment that contains the transport information is received. We hinted this in the draft:

   In order to protect the CPE and the concentrator and minimize the
   risk of degrading the overall bonding service performance, dedicated
   resources SHOULD be reserved for handling fragments (e.g., by               
   limiting the amount of resources to process out-of-order packets).
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 We need a header in each packet that
> indicates the order and association just like IP fragmentation.

[Med] There are two cases for handling IPv4 fragments: 

(1) IPv4 fragments that are received from the Internet or IPv4 received from the LAN: for those, I proposed the following behavior (not yet discussed with my co-authors): 

5.3.1.  Receiving IPv4 Fragments on the Internet-Facing Interface of the
        Concentrator

   The forwarding of an IPv4 packet received on the Internet-facing
   interface of the concentrator requires the IPv4 destination address
   and the transport-protocol destination port for binding lookup
   purposes.  If the first packet received contains the transport-
   protocol information, the concentrator uses a cache and forwards the
   fragments unchanged (i.e., without reassembly).  A description of
   such a caching algorithm is outside the scope of this document.  If
   subsequent fragments arrive before the first fragment, the
   concentrator SHOULD queue these fragments till the first fragment is
   received.

   The processing of the first fragment MUST follow the same procedure
   as in Section 5.1.1.  The rewriting of the IP addresses of subsequent
   fragments MUST follow the instructions maintained in the binding
   table and the fragmentation cache.  The MF (More Fragments) bit and
   'Fragment offset' field MUST NOT be modified by the concentrator.

5.3.2.  Receiving IPv4 Fragments from the LAN

   If the first packet received contains the transport-protocol
   information, the CPE uses a cache and forwards the fragments
   unchanged (i.e., without reassembly).  If subsequent fragments arrive
   before the first fragment, the concentrator SHOULD queue these
   fragments till the first fragment is received.

   The processing of the first fragment MUST follow the same procedure
   as in Section 5.1.2.  The rewriting of the IP addresses of subsequent
   fragments MUST follow the instructions maintained in the binding
   table and the fragmentation cache.  The MF bit and 'Fragment offset'
   field MUST NOT be modified by the CPE.

(2) Large packets that need to be fragmented when relayed into the MPTCP connection: The proposal is to fragment those only after the UDP/TCP conversion is achieved, and reassembly at the remote side before TCP/UDP conversion. Please refer to Section 5.3.

> validation check to assess whether the address conveyed in the option is
> distinct from an address that belong to the MPTCP node that receives the
> option. I.e., if that node receives an option that includes ones of its IP
> addresses, it is an indication that the data is to be consumed locally,
> not forwarded outside. Wouldn't that solve the first part of your "address
> mapping" concern?
> Correct but why would I even want to send an IP address ?

[Med] Fair point. I was referring to -07. (An optimization would be indeed that the client does not include an IP address if the ultimate DST address is equal to the destination IP address of the MPTCP peer).   

 Why can't
> translation information. 

[Med] You don't need a flag for this. You can rely on the length of the option as an explicit indication that no address is included. 

Ideally encapsulation should be an option on
> >>>> De : multipathtcp [mailto:[email-masked]] De la part
> de
> >> Rao
> >>>> Shoaib