Subject: [multipathtcp] Fwd: I-D Action:draft-van-beijnum-1e-mp-tcp-00.txt]]

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 06 May 2009 20:05:21 +0200

with a link to draft, which comes handy sometimes... (especially when 
people actually want to read it :-)

-------- Mensaje original --------
Asunto: 	I-D Action:draft-van-beijnum-1e-mp-tcp-00.txt
Fecha: 	Wed, 6 May 2009 10:45:02 -0700 (PDT)
De: 	[email-masked]
Responder a: 	[email-masked]
Para: 	[email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title           : One-ended multipath TCP
	Author(s)       : I. van Beijnum
	Filename        : draft-van-beijnum-1e-mp-tcp-00.txt
	Pages           : 21
	Date            : 2009-05-06

Normal TCP/IP operation is for the routing system to select a best
path that remains stable for some time, and for TCP to adjust to the
properties of this path to optimize throughput.  A multipath TCP
would be able to either use capacity on multiple paths, or
dynamically find the best performing path, and therefore reach higher
throughput.  By adapting to the properties of several paths through
the usual congestion control algorithms, a multipath TCP shifts its
traffic to less congested paths, leaving more capacity available for
traffic that can't move to another path on more congested paths.  And
when a path fails, this can be detected and worked around by TCP much
more quickly than by waiting for the routing system to repair the
failure.

This memo specifies a multipath TCP that is implemented on the
sending host only, without requiring modifications on the receiving
host.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-van-beijnum-1e-mp-tcp-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 07 May 2009 10:31:19 -0400

Marcelo,

Thanks for the draft, I've seen presentations and discussions about it, and I have wanted to see this written up for a while now.

With regard to the statement: "In addition, SCTP as
   defined today [RFC2960] does not accommodate the concurrent use of
   multiple paths.  Additional paths are purely used for backup
   purposes."
let me encourage you to look up Concurrent Multipath Tranfer (CMT) with SCTP. While this work used SCTP for CMT, many of the issues are not SCTP-specific; they apply to multipath-TCP as well. A list follows:

    + Our Transactions on Networking paper from 2006 (http://www.fandm.edu/jiyengar/papers/cmt-ton2006.pdf) discusses several of the concerns that you bring up here, only in the context of SCTP. We discuss managing reordering and other concerns and present algorithms for managing those concerns.  Our work on CMT applies here quite directly.

    + You list 4 challenges and questions, and say that you address the first two (scheduling, reordering). We have looked extensively at those two and also the third one (receiver buffer constraints), and our results are documented.  For receive buffer (rbuf) considerations and retransmission policies, I would encourage you to see our Computer Communications paper from 2007 (http://www.fandm.edu/jiyengar/papers/rbuf-compcomm2007.pdf).

    + We also looked into using CMT for fault tolerance, extended CMT mechanisms for fault tolerance, and found that CMT can do much better than simple-old-SCTP in detecting and responding to failures. We have documented our design and our experimental results in our  IFIP Networking paper from 2008 (http://www.fandm.edu/jiyengar/papers/cmtpf-networking2008.pdf).

    + For other design considerations that are not covered in these papers, a list of unresolved issues, and for other related work, please see my dissertation on "End-to-End Concurrent Multipath Transfer Using Transport Layer Multihoming" from 2006: http://www.fandm.edu/jiyengar/papers/iyengar-dissertation.pdf

I like to think that we did quite a bit of work in this area already, and while there are unresolved issues still around, I also like to think that we got some important things down :-)  It would be very disheartening to see work that is pretty much exactly the same not using our previous work.

I am happy to discuss this further!  (I will go read your draft more carefully later today ...)

regards,
- jana

marcelo bagnulo braun wrote:
> ------------------------------------------------------------------------
> 
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

-- 
Janardhan Iyengar
Assistant Professor, Computer Science
Franklin & Marshall College
http://www.fandm.edu/jiyengar

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 07 May 2009 16:53:33 +0200

Hi Jana,

the current idea is to submit a BOF proposal for Stockholm to propose 
the formation of a WG to define the multipath extensions for TCP. It 
would be great to have people that have been doing work on this 
involved. I will circulate the current version of the BOF proposal 
shortly, so please coment if you think that the relevant aspects are 
being included.

At this stage in this effort, we are looking forward to integrate 
interested parties, so if you have any suggestions or related work that 
you would like to be included as part of this effort, please say so, so 
we can work it out.

Regards, marcelo

Janardhan Iyengar escribió:
> Marcelo,
>
> Thanks for the draft, I've seen presentations and discussions about 
> it, and I have wanted to see this written up for a while now.
>
> let me encourage you to look up Concurrent Multipath Tranfer (CMT) 
> with SCTP. While this work used SCTP for CMT, many of the issues are 
> not SCTP-specific; they apply to multipath-TCP as well. A list follows:
>
>    + Our Transactions on Networking paper from 2006 
> (http://www.fandm.edu/jiyengar/papers/cmt-ton2006.pdf) discusses 
> several of the concerns that you bring up here, only in the context of 
> SCTP. We discuss managing reordering and other concerns and present 
> algorithms for managing those concerns.  Our work on CMT applies here 
> quite directly.
>
>    + You list 4 challenges and questions, and say that you address the 
> first two (scheduling, reordering). We have looked extensively at 
> those two and also the third one (receiver buffer constraints), and 
> our results are documented.  For receive buffer (rbuf) considerations 
> and retransmission policies, I would encourage you to see our Computer 
> Communications paper from 2007 
> (http://www.fandm.edu/jiyengar/papers/rbuf-compcomm2007.pdf).
>
>    + We also looked into using CMT for fault tolerance, extended CMT 
> mechanisms for fault tolerance, and found that CMT can do much better 
> than simple-old-SCTP in detecting and responding to failures. We have 
> documented our design and our experimental results in our  IFIP 
> Networking paper from 2008 
> (http://www.fandm.edu/jiyengar/papers/cmtpf-networking2008.pdf).
>
>    + For other design considerations that are not covered in these 
> papers, a list of unresolved issues, and for other related work, 
> please see my dissertation on "End-to-End Concurrent Multipath 
> Transfer Using Transport Layer Multihoming" from 2006: 
> http://www.fandm.edu/jiyengar/papers/iyengar-dissertation.pdf
>
>
> I like to think that we did quite a bit of work in this area already, 
> and while there are unresolved issues still around, I also like to 
> think that we got some important things down :-)  It would be very 
> disheartening to see work that is pretty much exactly the same not 
> using our previous work.
>

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 07 May 2009 10:59:30 -0400

Hi Marcelo,

A BoF sounds great! I will be glad to look at the BoF proposal when it comes out.

> [...] so if you have any suggestions or related work that 
> you would like to be included as part of this effort, please say so, [...]

A good bunch of our previous work on CMT is as related as anything can ever be. There. I said so. :-)

I look forward to seeing the BoF proposal,
- jana

---

From: Michio HONDA <[email-masked]>
Date: Fri, 08 May 2009 18:18:14 +0900

Hi,

I have a bit of comments on this draft.

Sec. 3.5 Fairness and TCP friendliness
 >Until more  analysis has been performed and/or there is more 
experience with multipath TCP, a multipath TCP implementation SHOULD 
limit itself to  using no more than 3 subflows concurrently.

How the decision to allow 3 subflows is made?
For example, HTTP spec. (RFC 2116 Sec.8.1.4) describes the simultaneous 
connections should not use more than 2 connections.
The reason contains not only receiver workload, but also congestion 
control.
Hence I think you should mention how multipath TCP (using 3 subflows) 
deals with such criterions.

In addition, it would be good to allow more subflows based on shared 
bottleneck detection like 
mTCP(https://www.usenix.org/events/usenix04/tech/general/full_papers/zhang/zhang_html/)

Best regards,
Michio

-- 
Michio Honda
E-mail: [email-masked], [email-masked]
Website: http://www.micchie.net/

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 8 May 2009 16:34:49 +0200

Jana,

Thanks for all the pointers. I'll go through them to see if there are  
any that I haven't seen before.

> not SCTP-specific;

Right. However, note that RFC 2960 says that only one path is used to  
send traffic, the other path is only used for retransmissions. (I'll  
have to check with RFC 4960.)

>   + Our Transactions on Networking paper from 2006 (http://www.fandm.edu/jiyengar/papers/cmt-ton2006.pdf 
> ) discusses several of the concerns that you bring up here, only in  
> the context of SCTP.

Right, I'm familiar with this paper.

>   + You list 4 challenges and questions, and say that you address  
> the first two (scheduling, reordering). We have looked extensively  
> at those two and also the third one (receiver buffer constraints),  
> and our results are documented.

Note that for SCTP the send and receive buffer issues are different  
when there are multiple streams.

> I like to think that we did quite a bit of work in this area  
> already, and while there are unresolved issues still around, I also  
> like to think that we got some important things down :-)  It would  
> be very disheartening to see work that is pretty much exactly the  
> same not using our previous work.

I agree. But SCTP results don't always translate into TCP results. And  
there's still plenty of work to do to apply the research to  
implementations.

---

From: Janardhan Iyengar <[email-masked]>
Date: Fri, 08 May 2009 23:59:07 -0400

Hi Illjitsch,

I will also pull out other related work and add it into the wiki.

> have to check with RFC 4960.)

That is correct.

>> our results are documented.
> 
> Note that for SCTP the send and receive buffer issues are different when 
> there are multiple streams.

Can you say some more about which issues you think may be different?  Any solution for SCTP send/receive buffer issues must handle the single-stream case, which is essentially collapses, as far as buffer issues are concerned, down to the issues of an ordered TCP stream.

> there's still plenty of work to do to apply the research to 

As far as implementation is concerned, CMT has been part of FreeBSD since FreeBSD 7.0 (sysctls net.inet.sctp.cmt*).  And if you count simulators, CMT is part of the ns2 simulator (ns/sctp/).

As far as TCP/SCTP differences are concerned:
Pulling words from the BoF announcement, the first proposed work item of multipath-TCP is:

"A set of architectural guidelines for congestion dependent
multipath transport protocols. Since different transport protocols
can potentially benefit from this approach, this document describes
the motivations and the general approach that should be followed
to enable congestion dependent multipath transport."

Surely SCTP falls into the category described here. Architecturally, I expect that there will be a significant amount of overlap; it doesn't matter if it is TCP or SCTP.  Our design is one point in the design space of multipath transport.

I fully agree that the devil in the details will have to be dealt with, and there will be many differences there between TCP and SCTP; we don't have to go very far, SACK differences are significant enough!  Having said that, I expect that *at this stage*, when we are probably still having architectural discussions, there will be more similarities than differences.

- jana

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 9 May 2009 17:00:13 +0200

Hi Michio,

Thanks for your comments.

On 8 mei 2009, at 11:18, Michio HONDA wrote:

>> experience with multipath TCP, a multipath TCP implementation  
>> SHOULD limit itself to  using no more than 3 subflows concurrently.

> How the decision to allow 3 subflows is made?

Well, it needs to be low enough that in the case where a multipath TCP  
session is sharing a bottleneck with a regular TCP session where all  
the multipath subflows flow through that same bottleneck, the impact  
to the regular TCP session is limited to something reasonable.

On the other hand, it doesn't make much sense to have a multipath TCP  
with only one subflow. And two subflows could easily create a number  
of special cases, so for testing purposes three would seem like a  
reasonable number.

But like I said, this needs more analysis and experimentation.

> For example, HTTP spec. (RFC 2116 Sec.8.1.4) describes the  
> simultaneous connections should not use more than 2 connections.
> Hence I think you should mention how multipath TCP (using 3  
> subflows) deals with such criterions.

We are working on a paper about this, others are working on this as  
well.

> bottleneck detection like mTCP(https://www.usenix.org/events/usenix04/tech/general/full_papers/zhang/zhang_html/ 
> )

 From that paper:

"Intuitively, a match means the two subflows enter fast retransmit  
around the same time. This also means packets from the two flows are  
dropped at about the same time, so it is likely they share the same  
congested link."

However, this is way too simple. Congestion can present itself in very  
many different ways. For instance, there may not even any packet loss  
at all, just an increase in RTT. If the congestion is because of a  
fixed high utilization or if a queuing strategy like RED is in use, or  
if the bandwidth of the congested link is much higher than the send  
rate, losses will very likely not be correlated. Only when there is a  
burst that is enough to fill up the buffer and long enough to span the  
time between the arrivals of two packets from two subflows will there  
be a loss on two subflows at the same time.

However, if shared bottlenecks could be detected that would make the  
fairness issues much less problematic, so maybe this warrants more  
investigation.

Iljitsch

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 11 May 2009 21:07:48 +0200

Janardhan Iyengar escribió:
> Can you say some more about which issues you think may be different?  
> Any solution for SCTP send/receive buffer issues must handle the 
> single-stream case, which is essentially collapses, as far as buffer 
> issues are concerned, down to the issues of an ordered TCP stream.
>
>> I agree. But SCTP results don't always translate into TCP results. 
>> And there's still plenty of work to do to apply the research to 
>> implementations.
>
> As far as implementation is concerned, CMT has been part of FreeBSD 
> since FreeBSD 7.0 (sysctls net.inet.sctp.cmt*).  And if you count 
> simulators, CMT is part of the ns2 simulator (ns/sctp/).
>
> Surely SCTP falls into the category described here. Architecturally, I 
> expect that there will be a significant amount of overlap; it doesn't 
> matter if it is TCP or SCTP.  Our design is one point in the design 
> space of multipath transport.
>
> I fully agree that the devil in the details will have to be dealt 
> with, and there will be many differences there between TCP and SCTP; 
> we don't have to go very far, SACK differences are significant 
> enough!  Having said that, I expect that *at this stage*, when we are 
> probably still having architectural discussions, there will be more 
> similarities than differences.
>
i agree that it would be possible to have at least one part of the arch 
document to be common
there may also be other points on common, such as the interaction with 
the multipaht routing.

Regards, marcelo