Subject: [multipathtcp] WG Last call for "Experience with Multipath TCP"

From: <[email-masked]>
Date: Mon, 11 May 2015 16:25:54 +0000

Hi,

we'd like to start a 2-week WG last call for this document

https://tools.ietf.org/html/draft-ietf-mptcp-experience-01?

Please send your comments to the list.

As well as suggested improvements, it's also useful to have comments such as "i've read the doc and think it's ready to progress"

This is aimed at our Milestone

Jan 2015 Use-cases and operational experiences (Informational) to IESG

here's the relevant part of our Charter

<<

The working group will also explore and document results with several
of the proposed use cases for MPTCP in more detail, to ensure that
MPTCP works well in practice and that operational experiences and
issues are understood and captured. Likely use cases are to offload
traffic from 3G to WiFi, and to manage traffic within a data centre.
Another scenario is to enable, without changing the MPTCP protocol,
operation of a single-homed, MPTCP end host on a campus network that has
multiple providers.

Prior to publishing a Standards Track specification, the working group
will document experimental results and operational experiences to-date.
This should consider not just experience with well-connected fat-pipe
networks and long-lived flows, but also consider a broader links and
types of applications; particularly looking for cases where MPTCP
could be detrimental in some way.

thanks,

Phi & Yoshi

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 18 May 2015 02:04:10 -0700

Hi,

I have reviewed this draft and I have the following comments. (sorry for
long lists)

* Title
   Experience with Multipath TCP
     --> Although the title seems to say it will describe experiences for
mptcp, I think it also contains some other stuff.
          For example, some use cases described here look like
possibilities, but haven't fully experienced yet.
          This may be fine as it's still useful info, but I think we had
better clarify what we should include in the doc.

* Section 1
   Section {{usecases}}. Seciton{{congestion}
     --> editorial miss?

* Section 2
   [HotMiddlebox13] documents a corner case with segment splitting that may
lead to desynchronisation between the two hosts.
      --> this sounds a bit problematic. Why you don't elaborate this?

* Section 2
   The results for [HotMiddlebox13] listed in the draft looks good.
However, it seems to me that it has some presumptions for the middleboxes.
   For example, I guess it might presume that middleboxes don't discard
options when it coalesces segments or don't drop options intermittently.
   If so, we might want to describe them.

* Section 3
   Description for backup mode doesn't look experiences to me.  Some
clarifications may be required.

* Section 3
   Description for single-path mode doesn't look experiences to me. Some
clarifications may be required.

* Section 3
   "From a protocol viewpoint, [Cellnet12] discusses the problem posed by
   the unreliability of the ADD_ADDR option and proposes a small
   protocol extension to allow hosts to reliably exchange this option.
   It would be useful to analyze packet traces to understand whether the
   unreliability of the REMOVE_ADDR option poses an operational problem
   in real deployments. "

     --> this seems to be useful info. I might want to see more detailed
info.

* Section 3
  "MPTCP provides a robust data transport and reduces variations in
download latencies".
      --> This looks good results, I might want to see more precise info.

* Section 3
    The use case of coupling mptcp and SDN doesn't look experiences to me.
Some clarifications may be required.
    I'm not very sure "Using an international network" is a part of SDN use
cases although this is described in the same paragraph.

* Section 4
    This seems to be very useful and important info and I personally would
like to see more detailed info.
    For example, which situation we can see performance benefits of OLIA?
How much benefit we can get.
    Is there any concern or risks to use OLIA or default CC? If OLIA is
better, shall we consider adopting it? Or, is it too early?
    It might be good if the authors can provide something on these points.

* Section 5.1
    "mptcp-dev mailing list"
      --> please provide the reference as readers might not know what it is.

* Section 5.1
   "The second subflow manager that is currently supported by the
   Multipath TCP implementation in the Linux kernel is the ndiffport
   subflow manager."
       --> we might need a reference for ndiffport?

* Section 5.1
  "This manager was initially created to exploit the
   path diversity that exists between single-homed hosts due to the
   utilization of flow-based load balancing techniques.  This subflow
   manager creates N subflows between the same pair of IP addresses.
   The N subflows are created by the client and differ only in the
   source port selected by the client."

          --> Does this mean the logic of the manager has been changed? If
so, what's the reason for this?
          --> How do we determine N? why it uses the same pair of IP
addresses?

* Section 5.2
   "However, from an implementation point-of-view supporting different
   destination ports for the same Multipath TCP connection introduces a
   new performance issue."

         --> It seems that the rest of the paragraph doesn't provide
specific explanation for performance issue.
              Is it coming from implementation proxy or other reasons?

* Section 5.2
   "This imposes an additional cost on servers, previously not existent on
legacy TCP implementations. "

     -->I understand there'll be additional cost. However, is it a very
complex mechanism to affect performance?
         I'm wondering some kinds of hash table might do some jobs.

* Section 5.3
     I think we should use "M_" state for the status of mptcp otherwise
it'll be confused with the status of TCP.

* Section 5.3
     "Figure 4 shows a very particular issue within Multipath TCP."
       --> I might miss something, but I'm not very sure the cases
described here.
            In my understanding, client should perform close() so that it
will go timewait and server can release the resource quickly.
            However, in this example, server perform close() and goes into
m_timewait. Please clarify this point.

* Section 6
     "SCTP implementations also include schedulers, but these are used to
schedule the different streams. "
        --> Does this mean all schedulers for SCTP are used to schedule the
different stream?

* Section 6
      "Unfortunately, none of the proposed techniques have been implemented
and used in real deployment."
         --> Is there any reason for this? If not, I think we had better
add "at this moment"

* Section 6
      "The simulation scenarios discussed in [PAMS2014] confirm the impact
of the packet scheduler on the performance of Multipath TCP."
         --> Can you elaborate this? Otherwise, it looks a bit ambiguous
info.

* Section 7
       "Taking the smallest MSS does not solve the issue as there might be
a case where the sublow with the smallest MSS will only participate
         marginally to the overall performance therefore reducing the
potential throughput of the other subflows."
        --> It seems to me that this approach might work in some
situations. If so, "does not solve the issue" sounds too much.

* Section 7
        Does the current linux approach perform better than others? No
drawback or potential risk in this approach?
        I would like to see more detailed analysis on this. If this is good
approach, we might want to document the detailed logic?

Thanks,
--
Yoshi

On Mon, May 11, 2015 at 9:25 AM, <[email-masked]> wrote:

> https://tools.ietf.org/html/draft-ietf-mptcp-experience-01​
>
>
> As well as suggested improvements, it's also useful to have comments such
> as "i've read the doc and think it's ready to progress"
>
>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 18 May 2015 20:28:15 +0200

Yoshifumi,
> I have reviewed this draft and I have the following comments. (sorry for

Thanks for the detailed comments. I'll try to answer them below

>       --> Although the title seems to say it will describe experiences
> for mptcp, I think it also contains some other stuff.
>            This may be fine as it's still useful info, but I think we
> had better clarify what we should include in the doc.

We could remove the discussion of the use cases from this document.

>     Section {{usecases}}. Seciton{{congestion}

Wrong conversion from markdown to text :-(

> * Section 2
>     [HotMiddlebox13] documents a corner case with segment splitting that
> may lead to desynchronisation between the two hosts.

I can expand it based on the text in [HotMiddlebox13]

>     If so, we might want to describe them.

I can add other scenarios, but this will make the text more difficult to 
read. Some of the scenarios explored in [HotMiddlebox13] are corner 
cases that are unlikely to happen in practice.

We recently analysed all the MPTCP connections received on 
multipath-tcp.org and observed the following :

Among the 184 thousands Multipath TCP connections in the trace, we 
observed only 125 of them falling back to regular TCP, which happened 
with 28 distinctive client IP addresses. These include 91 HTTP 
connections and 34 FTP connections. The FTP interference is expected and 
due to Application Level Gateways running on NAT boxes. The HTTP 
interference appeared only on the direction from server to client and 
could have been caused by transparent proxies deployed in cellular and 
enterprise networks.

Note that we do not observe interference where a middlebox removes the 
MP_CAPABLE option from the SYN in this dataset.

> clarifications may be required.

There are two issues with the backup mode that could be discussed for 
smartphones. Let us assume for simplicity that the WiFi is used as a 
primary interface and cellular as a backup interface.

The first is that by starting the backup flow (i.e. with a three-way 
handshake) over the cellular interface, the smartphone consummes energy 
and radio channel capacity (typically for a few seconds). If the backup 
subflow is not started immediately, but only when the primary fails, 
then energy and radio channel consumption are reduced.

The second issue that we observe on smartphones is that the status of a 
subflow cannot be described as a binary value (active/inactive). There 
are many situations where a subflow is considered active while it cannot 
efficiently transmit data. A typical example is a smartphone that is far 
away from a WiFi access point, but close enough to perform the 
three-way-handshake. If this subflow is a primary subflow and the 
cellular a backup subflow, then MPTCP will try to retransmit data 
several times over the WiFi interface before stopping the WiFi subflow 
and starting to use the cellular one. This can cause delays of tens of 
seconds before switching to the cellular interface.

>
> * Section 3
>     Description for single-path mode doesn't look experiences to me.
>       --> this seems to be useful info. I might want to see more
> detailed info.

This should be compared with the decision of Apple to not implement 
ADD_ADDR. We can add some text.
>        --> This looks good results, I might want to see more precise info.

This sentence comes from the conclusion of the paper IMC13b. What kind 
of more precise information do you expect ?

> * Section 3
>      The use case of coupling mptcp and SDN doesn't look experiences to
> me. Some clarifications may be required.
>      I'm not very sure "Using an international network" is a part of SDN
> use cases although this is described in the same paragraph.

This part could be removed and we could point to the MPTCP bibliography 
for other use cases https://github.com/obonaventure/mptcp-bib

I think that the use case described in a recent NSDI paper
https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/croitoru
should be added to this section because it is important for mobile devices

> * Section 4
>      This seems to be very useful and important info and I personally
> would like to see more detailed info.
>      For example, which situation we can see performance benefits of
> OLIA? How much benefit we can get.

It is possible to find simulation scenarios/topologies where one 
congestion control algorithm works better than others. However, we 
should not decide on the basis of a small number of examples.

The best approach to compare different congestion control schemes is to 
explore a wide range of network conditions. Christoph Paasch has used an 
experimental design approach to perform this analysis for a bonding 
scenario (two links without congestion). The results are described in
http://inl.info.ucl.ac.be/publications/benefits-applying-experimental-design-improve-multipath-tcp
and have been updated in
http://inl.info.ucl.ac.be/publications/improving-multipath-tcp

Based on these results, there does not seem to be a strong benefit with 
OLIA over LIA. If the WG wants to reconsider the selection of the 
default congestion control scheme, it should use a similar experimental 
approach and use different network topologies.

>        --> please provide the reference as readers might not know what

Will add.

>         --> we might need a reference for ndiffport?

There is no explicit document on this path manager afaik.

>            --> Does this mean the logic of the manager has been changed?
> If so, what's the reason for this?
>            --> How do we determine N?

N is a static configuration. Some experiments have been reported in the 
SIGCOMM'11 paper

> why it uses the same pair of IP
> addresses?

Because it is used on single-homed hosts

>                Is it coming from implementation proxy or other reasons?

This comes from an implementation issue because the stack needs to 
listen to more ports.

> * Section 5.2
>     "This imposes an additional cost on servers, previously not existent
> on legacy TCP implementations. "

I don't think that such a solution has been explored

> it'll be confused with the status of TCP.

Will add

>              In my understanding, client should perform close() so that
> it will go timewait and server can release the resource quickly.
>              However, in this example, server perform close() and goes
> into m_timewait. Please clarify this point.
>
> * Section 6
>       "SCTP implementations also include schedulers, but these are used
> to schedule the different streams. "
>          --> Does this mean all schedulers for SCTP are used to schedule
> the different stream?

That was our understanding, but maybe there are schedulers on CMT-SCTP 
implementations.

> * Section 6
>        "Unfortunately, none of the proposed techniques have been
> implemented and used in real deployment."
>           --> Is there any reason for this? If not, I think we had
> better add "at this moment"

Exact, everything in the document is to be considered "as of this writing".

> * Section 6
>        "The simulation scenarios discussed in [PAMS2014] confirm the
> impact of the packet scheduler on the performance of Multipath TCP."
>           --> Can you elaborate this? Otherwise, it looks a bit
> ambiguous info.

Will expand.

> * Section 7
>         "Taking the smallest MSS does not solve the issue as there might
> be a case where the sublow with the smallest MSS will only participate
>          I would like to see more detailed analysis on this. If this is
> good approach, we might want to document the detailed logic?

It would be interesting to see if other implementors have faced the 
issue and found other solutions to cope with it.

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: "Weixinpeng (Jackie)" <[email-masked]>
Date: Tue, 19 May 2015 03:56:21 +0000

This document provides a really comprehensive survey  about the MPTCP experiences, and that would
be valuable for the improvement of MPTCP. So I support the progress of this document.

Xinpeng

From: multipathtcp [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, May 12, 2015 12:26 AM
To: [email-masked]
Subject: [multipathtcp] WG Last call for "Experience with Multipath TCP"

Hi,

<<

thanks,

Phi & Yoshi

---

From: Jaehyun Hwang <[email-masked]>
Date: Tue, 19 May 2015 22:37:38 +0900

Olivier,

Regarding the congestion control, can you also include the Balia congestion
control scheme in Section 4?

We basically agree we have to explore a wide range of network conditions to
compare the algorithms, but Balia has shown it balances the tradeoff
between friendliness and responsiveness well, and we found out some
scenarios where Balia works better than LIA and OLIA. In addition, it has
been included in the MPTCP kernel v0.9 (and also in the NorNet Core
testbed). So we believe it would be worth mentioning.

Below links are the paper and the I-D.

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7000573

http://tools.ietf.org/html/draft-walid-mptcp-congestion-control-02

Thanks,
Jaehyun

On Tue, May 19, 2015 at 3:28 AM, Olivier Bonaventure <
[email-masked]> wrote:

> read. Some of the scenarios explored in [HotMiddlebox13] are corner cases
> that are unlikely to happen in practice.
>
> observed only 125 of them falling back to regular TCP, which happened with
> 28 distinctive client IP addresses. These include 91 HTTP connections and
> 34 FTP connections. The FTP interference is expected and due to Application
> Level Gateways running on NAT boxes. The HTTP interference appeared only on
> the direction from server to client and could have been caused by
> transparent proxies deployed in cellular and enterprise networks.
>
> handshake) over the cellular interface, the smartphone consummes energy and
> radio channel capacity (typically for a few seconds). If the backup subflow
> is not started immediately, but only when the primary fails, then energy
> and radio channel consumption are reduced.
>
> subflow cannot be described as a binary value (active/inactive). There are
> many situations where a subflow is considered active while it cannot
> three-way-handshake. If this subflow is a primary subflow and the cellular
> a backup subflow, then MPTCP will try to retransmit data several times over
> the WiFi interface before stopping the WiFi subflow and starting to use the
> cellular one. This can cause delays of tens of seconds before switching to
> the cellular interface.
>
>
>> * Section 3
> This sentence comes from the conclusion of the paper IMC13b. What kind of
> more precise information do you expect ?
>
>  * Section 3
> congestion control algorithm works better than others. However, we should
> not decide on the basis of a small number of examples.
>
> OLIA over LIA. If the WG wants to reconsider the selection of the default
> congestion control scheme, it should use a similar experimental approach
> and use different network topologies.
>
>  * Section 5.1
>>      "mptcp-dev mailing list"
> This comes from an implementation issue because the stack needs to listen
> to more ports.
>
>  * Section 5.2
> It would be interesting to see if other implementors have faced the issue
> and found other solutions to cope with it.
>
>
> Olivier
>
>
> --

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 19 May 2015 16:24:24 +0200

Hello,
>
> Regarding the congestion control, can you also include the Balia
> congestion control scheme in Section 4?
>
> We basically agree we have to explore a wide range of network conditions
> to compare the algorithms, but Balia has shown it balances the tradeoff
> scenarios where Balia works better than LIA and OLIA. In addition, it
> has been included in the MPTCP kernel v0.9 (and also in the NorNet Core

I can add links to BALIA in addition to the other congestion control 
algorithms that have been proposed for Multipath TCP. Given that the 
litterature is quickly growing in this field, I would propose to only 
list the congestion control algorithms that have been implemented in one 
Multipath TCP implementation.

Yesterday, I read again the implementation report coordinated by Phil 
and was worried by the following answer about the implemented congestion 
control schemes :

| Question 8 asks about congestion control and related issues: how     |
| traffic is shared across multiple subflows; support for 'handover';  |
| and support of RFC6356 (or other) coupled congestion control.        |
|            | UCLouvain | Swinburne |   Anon    |  Citrix   |         |
|sharing     |shared, RTT|shared     |active/back|active/back|         |
|handover    |Yes        |           |Yes        |Yes        |         |
|coupled cc  |Yes        |No         |No         |No         |         |
|other ccc   |Yes, OLIA  |No         |No         |No         |         |
|MP-PRIO & B |Yes        |No         |Yes        |Yes        |         |

I was wondering whether the other implementations had any plan to 
support one of the coupled congestion control schemes ?

Olivier

---

From: <[email-masked]>
Date: Tue, 19 May 2015 23:01:41 +0000

________________________________________
From: multipathtcp <[email-masked]> on behalf of Olivier Bonaventure <[email-masked]>
Sent: 18 May 2015 19:28

----

[phil]
i'd prefer to keep the use cases. perhaps you could say which have practical experience and which are proposed.

according to the charter, the doc covers "Use-cases and operational experiences"
and the relevant paragraphs say:-

<<The working group will also explore and document results with several

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 21 May 2015 08:17:32 -0700

Hi Olivier,

Thanks for the response.

> We could remove the discussion of the use cases from this document.

I am personally fine with it. I think the draft can describe the current
experience and provide some guidelines for future experiences and use cases.
But, I think you should articulate what's in the doc in abstract and
introduction section.

> that are unlikely to happen in practice.

I don't think you have to describe minor corner cases. But, I might want to
see some texts that navigate users to the doc if they're interested.

> MP_CAPABLE option from the SYN in this dataset.

this looks very useful experiences to me. it may be good to put this in the
doc.

> cellular one. This can cause delays of tens of seconds before switching to

If we'll put some future/potential use cases in the doc, I think this will
be useful guideline.

> more precise information do you expect ?

 Well, I am wondering if you can show some small dataset here as I feel
this sentence doesn't have much info.

> The SDN part looks a bit thin to me. It might be good to be replaced If
you have other good use cases.

> and use different network topologies.

I believe this is a very nice explanation for the current status of CC
studies. I think this type of views will be useful as a guidance for future
experiences.

> There is no explicit document on this path manager afaik.

OK. then, I might prefer something like "TCP implementation in the Linux
kernel is named ndiffport subflow manager"

> Because it is used on single-homed hosts

OK. But, what if the node is multi-homed?

> I don't think that such a solution has been explored

OK,  Then, It seems to me that "introduces a new performance issue." might
be too much..

> That was our understanding, but maybe there are schedulers on CMT-SCTP

Hmm. when we say SCTP implementations, I tend to think about both SCTP
(rfc4960) and CMT-SCTP implementations.
So, I am personally a bit skeptical about this sentence.

> and found other solutions to cope with it.

I am hoping some implementors will speak up on this...

--
Yoshi

---

From: Amani Alheid <[email-masked]>
Date: Thu, 21 May 2015 17:09:54 +0100

Hi Multipath group members
I am wondering if your document is related to MPTCP using Linux
implementation only or not. I have published papers on the behaviour of
MPTCP using three different congestion control algorithms used in
conjunction with four TCP packet reordering solutions using NS-3. The
papers are published in AINA2014 and SIMS2014.
"An Analysis of the Impact of Out-Of-Order Recovery Algorithms on MPTCP
Throughput" .
and "Performance Evaluation of MPTCP in Indoor Heterogeneous Networks."
which can be found in the following links
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6838660&tag=1
http://ijssst.info/Vol-15/No-5/data/5198a173.pdf
The study investigates how sensitive the performance is against the delay
difference between the paths used. I am thinking weather this study can add
things to your experiences or not. T
if any one is interesting I am glad to answer any question.
Best regards,
​​
*===================================*
*​Mrs. ​Amani Alheid*

*​PhD student.​Communication Systems & Networks GroupUniversity of
BristolUnited Kingdom===================================*

On 21 May 2015 at 16:17, Yoshifumi Nishida <[email-masked]> wrote:

> I don't think you have to describe minor corner cases. But, I might want
> to see some texts that navigate users to the doc if they're interested.
>
>>
> this looks very useful experiences to me. it may be good to put this in
> the doc.
>
>>
>>
>>> * Section 3