Subject: [multipathtcp] Draft Multipath TCP (MPTCP) BOF description

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 07 May 2009 17:21:06 +0200

Hi,

We are planning to submit a BOF request for the next IETF.
This is a first draft of the bof proposal.

Comments are welcome, of course.

Multipath TCP (MPTCP) BOF

Recent research has suggested that by making better use of the
multiple paths between endpoints we can better utilise the
resources within the network and thus improve resource pooling.
In particular,  recent results include models of how
multipath-capable end-systems might respond to congestion signals,
and what the potential benefits might be (see [1] for a more details
and references).

If end systems could spread their load across multiple paths in the
right way, with the right reaction to the right congestion signals
from the network, then traffic would quickly move away from
congested or failed links in favor of available uncongested links.

It is fairly intuitive to see that when multipath transport is used
in conjunction with congestion-dependent load-distribution, it is
possible to push more traffic through the network and to achieve
enhanced fault tolerance. Traffic will flow throught any available
path that has spare capacity, filling unused resources, while moving
away from failures and congested resources.
The result is a more flexible network, capable of functioning well,
even when the actual traffic matrix differs substantially from the
traffic matrix envisaged when the network was designed for.

Currently used transport protocols use only one path at a time
for sending data. In particular TCP and DCCP use only one path and
they respond to congestion reducing their sending rate. SCTP
on the other hand, is aware of multiple paths, but it uses only
one at the time, reserving the alternative paths as backups
in case of failure.

The proposed work is to define the mechanisms to enable
transport protocols to use multiple paths simultaneously
and to distribute the load among the subflows of each path based
on congestion.

In particular, the proposed work is to extend TCP to support
simultaneous data exchange through multiple paths.
Two complementary approaches are proposed: On one hand,
to define a two-ended multipath TCP that uses multiple addresses a-la SCTP
to obtain multiple paths and that requires support from both
endpoints to use them.
On the other hand, single ended multipath TCP, where only
one of the endpoints involved in the communication
is multipath aware and relies on multipath routing
capabilities provided by the IP routing to deliver data towards
a legacy TCP endpoint.
These two approaches provide a smooth deployment model, starting
with the one-ended multipath TCP, that only requires one end to deploy it
and it results in immediate benefits, and then when multipath
TCP support becomes pervasive, using the two-ended multipath TCP
to obtain all the potential benefits.

The proposed work items are:
- A set of architectural guidelines for congestion dependent
multipath transport protocols. Since different transport protocols
can potentially benefit from this approach, this document describes
the motivations and the general approach that should be followed
to enable congestion dependent multipath transport.
- Extensions to current TCP to support the one-ended mutipath TCP.  
- Extensions to current TCP to support the two-ended mutipath TCP.  
- A threat analysis for multipath TCP
- An extended API describing how applications can
access to multipath information and use it.

The initial goal is that the documents produced by the WG will be
either experimental or informational. However, if when the documents
are ready the technology seems mature enough, the WG will consider
if it is appropriate to ask the IESG to advance the document as a
Proposed Standard.

With respect to the work on the support for multipath in TCP, the
only acceptable modifications to the TCP packet format are in the form
of new TCP options.

[1] D. Wischik, M. Handley, M. Bagnulo, The resource pooling principle,
ACM SIGCOMM Computer Communication Review archive, October 2008

---

From: Scott Brim <[email-masked]>
Date: Thu, 07 May 2009 13:31:59 -0400

Hi Marcelo.

(1) Up in the introductory material: As you've heard me argue, in
addition to the obvious benefits of multipath TCP, I believe it also
indirectly provides the best approach to path failure handling (better
than probing, in particular).

(2) Under work items, I don't see the work on the routers, or anyway
intermediate agents, to provide information to a single-ended multipath TCP.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 07 May 2009 19:45:02 +0200

> than probing, in particular).
agree, i will include some text on that, thanks

> intermediate agents, to provide information to a single-ended multipath TCP.
mmm, i am not sure i understnad that you mean.
the single ended approach requires the mptpc capable end to be able to 
select different paths. this can be done either b giving some hint to 
multipath capable routers or there are some other means, such as 
selecting different next hops, or tunneling to different egress routers. 
Is that what you mean?

Cause, you seem to imply that the routers need to give information to 
the mptcp capable endpoint, which i am not sure i fully understand

Regards, marcelo

---

From: Pasi Sarolahti <[email-masked]>
Date: Thu, 7 May 2009 23:57:23 +0300

Hi,

A wiki page has been established at http://trac.tools.ietf.org/area/ 
tsv/trac/wiki/MultipathTcp, including the BOF description below, and  
links to some additional material. Feel free to add more pointers to  
related material/discussion there (such as the links from Jana, that  
I didn't yet add).

- Pasi

On May 7, 2009, at 18:21, ext marcelo bagnulo braun wrote:

> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Scott Brim <[email-masked]>
Date: Thu, 7 May 2009 17:16:19 -0400

Excerpts from marcelo bagnulo braun on Thu, May 07, 2009 07:45:02PM
+0200:
>> addition to the obvious benefits of multipath TCP, I believe it
>> also indirectly provides the best approach to path failure handling
>> (better than probing, in particular).
>>   
>> (2) Under work items, I don't see the work on the routers, or
>> anyway intermediate agents, to provide information to a
>> single-ended multipath TCP.
>>   
> mmm, i am not sure i understnad that you mean.  the single ended
> approach requires the mptpc capable end to be able to  select
> different paths. this can be done either b giving some hint to
> selecting different next hops, or tunneling to different egress
> routers.  Is that what you mean?
>
> Cause, you seem to imply that the routers need to give information
> to  the mptcp capable endpoint, which i am not sure i fully
> understand

In all cases except the simplest, the endpoint would need to receive
hints from some agent -- an egress router or some other intermediary
that has knowledge of network state.  It's not enough for the endpoint
to give hints to the router.  Even if the endpoint tunnels to an
egress router, it needs an indication that the egress router is up and
can deliver the packets to the destination address.  So it's not
enough to work on the endpoint -- don't you need that agent?

Thanks ... Scott

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 7 May 2009 23:44:49 +0200

> enough to work on the endpoint -- don't you need that agent?

No. The proof of the pudding is in the eating. In principle, the host  
sends packets over a certain path, if ACKs come back the path works,  
if they don't it doesn't.

In practice things are going to be a bit more complex if we want to  
have multipath routers that work together with multipath TCP, because  
the multipath TCP needs to write a value in the packet somewhere that  
the router is going to look at. If everyone agrees on where that field  
is and what values are legal, it ends there, but if there are  
different options in different networks the host would need to know  
which options to use in a particular network.

But the situation where a good deal of routing information is exposed  
to hosts wouldn't be good, in my opinion. The great thing about  
multipath TCP is that it doesn't need that info, because it can  
determine which paths are working and how good they are for itself by  
virtue of simply sending data packets.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 08 May 2009 00:40:31 +0200

> which options to use in a particular network.
right.
I understand that one of the points Scott is raising is whehter the 
charter should cover the mptcp-router interaction.
I am not sure what the answer would be.

Regards, marcelo

---

From: Scott Brim <[email-masked]>
Date: Thu, 7 May 2009 19:04:52 -0400

Excerpts from Iljitsch van Beijnum on Thu, May 07, 2009 11:44:49PM +0200:
> In practice things are going to be a bit more complex if we want to have 
> multipath routers that work together with multipath TCP, because the 
> multipath TCP needs to write a value in the packet somewhere that the 
> router is going to look at. If everyone agrees on where that field is and 
> what values are legal, it ends there, but if there are different options 
> in different networks the host would need to know which options to use in 
> a particular network.

Good point -- the endpoint doesn't need to know which paths actually
work, just which ones might work.  However, how does the endpoint know
even that?  How does it know how to cause packets to take different
egresses?  I may be going too far too fast, but if the interesting
cases involve some agent giving the endpoint that information,
requirements for that agent should be documented, and thus in the
charter.  No?.

> I am not sure what the answer would be.

It makes sense to me to have those interactions in the charter.  I'm
asking if there is a component missing that should be included, to
support them.

Thanks much ... Scott

---

From: Joe Touch <[email-masked]>
Date: Thu, 07 May 2009 16:09:29 -0700

> requirements for that agent should be documented, and thus in the

IMO, this goes too far. I would expect it would be useful enough to talk
about "multi-address TCP" and avoid talking about "paths" when they're
not part of the architecture, nor are they knowable without doing things
like pathchar or traceroute.

It's useful enough to talk about TCP supporting multiple concurrent
endpoint addresses, and to be *agnostic* with respect to varying path
characteristics (not so much to tune to them, but to not be de-tuned by
them).

Joe

---

From: Fred Baker <[email-masked]>
Date: Thu, 7 May 2009 17:56:58 -0700

On May 7, 2009, at 2:44 PM, Iljitsch van Beijnum wrote:

> No. The proof of the pudding is in the eating. In principle, the  
> host sends packets over a certain path, if ACKs come back the path  
> works, if they don't it doesn't.

I buy that. That tells us something about source address pairs, which  
determine routing. How about finer-grain routing issues?

---

From: Michael Scharf <[email-masked]>
Date: Fri, 8 May 2009 10:20:31 +0200

> access to multipath information and use it.

IMHO this statement suggests that that API mainly allows applications
to query information about the multiple paths. Is this the expected
purpose?

Isn't there a need for an information flow in the reverse direction as
well, i. e., a (optional) API that enables applications to control and
configure the selection and usage of multiple paths?

If so, one could describe the scope of the API more explicitly.

Michael

---

From: Bryan Ford <[email-masked]>
Date: Fri, 8 May 2009 11:38:15 +0200

On May 8, 2009, at 1:04 AM, Scott Brim wrote:

>> have

But this brings up another issue - why should such a router/transport  
signaling mechanism be specific to multipath TCP?  For example, draft- 
van-beijnum-1e-mp-tcp-00 proposes a new TCP option with some fields  
for router interaction.  But what if multipath SCTP also wants to use  
the same router facilities to discover paths or indicate which paths  
to use?  Or multipath DCCP, SST, XYZ...?  Why should such network  
layer extensions be made specific to a particular transport, when they  
clearly could be useful to any multipath transport?

Of course, looking at the practicalities, this line of reasoning might  
suggest new IP options instead of new TCP options, and we know the  
difficulties of adding new IP options (i.e., routers and NATs choke on  
them).  On the other hand, we also know the difficulties of adding new  
TCP options (i.e., there isn't any option space left).  Figuring out  
how to extend the network to support such types of signaling seems to  
me to fit most clearly in the domain of, e.g., NSIS, not a BOF or WG  
specific to any particular transport.

Bryan

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 8 May 2009 14:26:23 +0200

On 8 mei 2009, at 2:56, Fred Baker wrote:

>> In principle, the host sends packets over a certain path, if ACKs  
>> come back the path works, if they don't it doesn't.

> I buy that. That tells us something about source address pairs,  
> which determine routing. How about finer-grain routing issues?

Like what?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 8 May 2009 14:37:15 +0200

On 8 mei 2009, at 11:38, Bryan Ford wrote:

> But this brings up another issue - why should such a router/ 
> transport signaling mechanism be specific to multipath TCP?  For  
> example, draft-van-beijnum-1e-mp-tcp-00 proposes a new TCP option  
> with some fields for router interaction.

It does say this:

    It is not expected that routers will make routing decisions directly
    based on the path indication option, as this option occurs deep
    inside the packet and not in a fixed place.

(However, the text before that basically assumes that routers do look  
at this field so your conclusion isn't unreasonable.)

> But what if multipath SCTP also wants to use the same router  
> facilities to discover paths or indicate which paths to use?  Or  
> multipath DCCP, SST, XYZ...?  Why should such network layer  
> extensions be made specific to a particular transport, when they  

It shouldn't, of course. If we want routers to make a path selection  
decision based on information from a multipath transport protocol, we  
need a field in packets to convey this information. And it should be  
standardized so all layers and boxes involved know how to interpret  
the field.

> Of course, looking at the practicalities, this line of reasoning  
> might suggest new IP options instead of new TCP options, and we know  
> the difficulties of adding new IP options (i.e., routers and NATs  
> choke on them).

Indeed.

> On the other hand, we also know the difficulties of adding new TCP  
> options (i.e., there isn't any option space left).  Figuring out how  
> to extend the network to support such types of signaling seems to me  
> to fit most clearly in the domain of, e.g., NSIS, not a BOF or WG  

I only had a quick look at NSIS, but isn't this mostly about signaling  
and signaling protocols? AFAIK RSVP recognizes flows, presumably using  
port numbers or maybe through the IPv6 flow label. In the former case,  
we have a problem because port numbers can't arbitrarily changed for  
an existing session and/or NATs change them in unpredictable ways. The  
flow label and DSCP bits also have existing uses and limitations but  
perhaps these could be reused for this if done carefully.

Alternatively, the path selection field could be included in a layer  
below IP, which makes it easier to come up with something that's not  
backward compatible because the encapsulation containing the field can  
be stripped away when the packet leaves the multipath-aware part of  
the network.

As the signaling part of this shares a lot with QoS I think it makes  
sense to go to NSIS for that, but there's also interactions with the  
internet layer/area and routing, as well as transport.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 08 May 2009 15:03:53 +0200

> requirements for that agent should be documented, and thus in the

mmm, i would say that if an endpoint is multipaht capable and have some 
way to express different paths, it simply does it. If the underlying 
routing infrastrucutre doesn't support that, the endpoint is not worth 
than sending all through a single path (assuming we can actually manage 
to make MPTCP fair to other single path tcps)

In other words, MPTCP marks the packets and hopes the IP (locally in the 
host or a router somehwehre else) will use that mark to send the packet 
through one path or another. If there is no such capability, the 
endpoint is not worse than the normal single path.
> asking if there is a component missing that should be included, to
My take is that there will be many different ways to provide IP support 
to MPTCP.
I don't know if it would make sense to write them down in an 
informational draft.
But i fail to see the need for a new component, if i understnad 
correctly what you say.
OTOH, it seems clear that more description of how MPTCP interacts with 
multiapth IP is needed.

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 08 May 2009 15:10:00 +0200

Joe Touch escribió:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Scott Brim wrote:
>   
But there are many situations and probably very interesting ones, where 
the node has multipaht paths and does not have multiple addresses.
For instnace, think about a server located in a big multihomed site 
using PI addresses. This server is likely to benefit from multipath TCP 
but it only has a single address.

In addition, even if you do have multiple addresses, you still need some 
support from the routing infrastrcuture.
consider the following case. One endpoint A has multiple addresses and 
the other endpoint B has a single address.

When a node A with multiple addresses sends packets, it will include 
different source addresses, but all packets will carry the same 
destiantion address. How can the nade A influence the egress path? 
clearly, today changing the source address has only one potential 
effect, that is that the packets are dropped due to ingress filters, but 
it does no have the ffect of changing the exit path

So even in the multiaddress case, we need support from the IP layer. The 
support in all the cases (multiple addres MPTCP or single address MPTCP 
is the same. That the IP layer is capable of looking into a path 
identifier included in the packet 8either the course address or other 
magic bits) and selects a different path based on those bits.

regards, marcelo

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkoDaikACgkQE5f5cImnZrvXRQCgyZ9IIjK6gEq2veMuTtl05npb
> SDgAni7GWZ6MKwrG1gbfGlAFP18m7mkZ
> =68uN
> -----END PGP SIGNATURE-----

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 08 May 2009 15:11:58 +0200

> If so, one could describe the scope of the API more explicitly.

makes sense to me
do you care to provide new text? It would be very mch appreciated

thanks, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 08 May 2009 15:15:35 +0200

Bryan Ford escribió:
> signaling mechanism be specific to multipath TCP?  For example, 
> draft-van-beijnum-1e-mp-tcp-00 proposes a new TCP option with some 
> fields for router interaction.  But what if multipath SCTP also wants 
> to use the same router facilities to discover paths or indicate which 
> paths to use?  Or multipath DCCP, SST, XYZ...?  Why should such 
> network layer extensions be made specific to a particular transport, 
> when they clearly could be useful to any multipath transport?

agree, the question is which path selector bits we use?
> specific to any particular transport.
any ideas of which bits could these be?

Regards, marcelo

---

From: Arnaud Ongenae <[email-masked]>
Date: Fri, 08 May 2009 15:31:37 +0200

> > the difficulties of adding new IP options (i.e., routers and NATs  

As I understand, you want to add an option in TCP so that it has great
chance to pass through NAT/Firewall/etc. (look as much as possible at
the original TCP). If the new option might be discarded, why do you want
so much to look like TCP and do not support SCTP (and Jana's work) to
achieve multipath transfer ? (maybe this point was already discussed
elsewhere do not hesitate to redirect me if needed)

Arnaud Ongenae
UCL (Belgium) student

---

From: Michael Scharf <[email-masked]>
Date: Fri, 8 May 2009 16:14:10 +0200

> do you care to provide new text? It would be very mch appreciated

What about this text:

"- An extended API describing how applications can use multipath
   transport. The API should also provide access to multipath
   information and enable the control of the usage."

One simple example for such control would be an extension of the
"NO_DELAY" socket option, i. e., applications could explicitly state
that they prefer a path with mimimum latency. But there are probably
some other useful control primitives as well.

Michael

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 8 May 2009 16:23:49 +0200

On 8 mei 2009, at 15:31, Arnaud Ongenae wrote:

> the original TCP).

For the one-ended multipath TCP draft it must look like normal TCP  
because otherwise the receiver (which we assume isn't multipath  
capable) won't be able to process the packet.

In this draft, the TCP option is there in case both ends are multipath- 
capable, so that when doing RTT measurements it's possible to see  
which path the other side used so it's possible to have different RTTs  
and therefore RTOs for the different paths.

> If the new option might be discarded, why do you want

I think a multipath-aware IP layer, which the one-ended multipath TCP  
needs, should also be available to other multipath transports, like  
SCTP with CMT (concurrent multipath).

---

From: "Preethi Natarajan" <[email-masked]>
Date: Fri, 8 May 2009 10:09:25 -0700

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Iljitsch 
> van Beijnum
> > If the new option might be discarded, why do you want so 
> much to look 
> > like TCP and do not support SCTP (and Jana's work) to achieve 
> > multipath transfer ? (maybe this point was already 
> discussed elsewhere 
> > do not hesitate to redirect me if needed)
> 
> I think a multipath-aware IP layer, which the one-ended 
> multipath TCP needs, should also be available to other 
> multipath transports, like SCTP with CMT (concurrent multipath).

I suppose what Arnaud is asking is the motivation to modify TCP such that
the modified TCP requires new middlebox considerations when other
multihome-capable and multipath-aware transports exist. 

I think this is different from whether or not a multipath transport requires
multipath-aware IP.

Preethi

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 8 May 2009 19:23:58 +0200

> multihome-capable and multipath-aware transports exist.

The motivations are better performance and redundancy without  
requiring application changes or any changes at the receiver.

If people put middleboxes in their networks that break MPTCP then  
obviously they won't enjoy the benefits. That is a choice that  
everyone needs to make for themselves.

---

From: "Preethi Natarajan" <[email-masked]>
Date: Fri, 8 May 2009 10:47:24 -0700

> From: Iljitsch van Beijnum [mailto:[email-masked]] 
> Sent: Friday, May 08, 2009 10:24 AM
> To: Preethi Natarajan
> Cc: 'Arnaud Ongenae'; [email-masked]
> 

Do you mean the application mods to open an SCTP (or whatever) socket
instead of TCP socket vs. mods to TCP stacks and middleboxes?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 8 May 2009 22:08:56 +0200

> instead of TCP socket vs. mods to TCP stacks and middleboxes?

We know for sure that if applications are going to switch from TCP to  
SCTP to gain multipath benefits, this creates deployment issues.  
Suppose that a browser wants to load pages from a website. How does  
the browser know if the server also supports SCTP? For IPv4 vs IPv6 we  
could solve this using the DNS, but for TCP vs SCTP this is harder.

Also, I believe current SCTP needs the application to tell it the  
addresses for the other side. Then there is the issue that SCTP must  
of course be implemented on both sides, while the big feature in the  
one-ended MPTCP is that it can work if it's only implemented in the  
sender.

Then we trade this off against the possible gain of not having to  
change middleboxes. However, you assume that incompatible middleboxes  
exist. While I'm sure this will be the case a good percentage of the  
time, it's not a universal truism.

And you assume that this same middlebox would be compatible with SCTP.  
That is a signficant leap of faith, work for SCTP NAT is still going  
on in the behave wg.

So I think there is a place for multipath TCP in general and the one- 
ended variety in particular.

---

From: Arnaud Ongenae <[email-masked]>
Date: Fri, 08 May 2009 23:40:43 +0200

A new thread for a question out of the scope of 
"Draft Multipath TCP (MPTCP) BOF description"

TCP modification vs switching to SCTP (if it has already been discussed
elsewhere do not hesitate to redirect me)

I believe that by introducing this modification to TCP (one-end MTCP)
you will give argument to those who do not want to switch to SCTP to
stay on TCP. I think that we should create an hybrid transition protocol
to produce the need to switch from TCP. This hybrid protocol exist and
was presented in :
Transparent TCP-to-SCTP translation shim layer 
from RW Bickhart thesis [1]

Why not concentrate multipath effort on SCTP which has the benefit to
natively support multihomed host and as Jana said research as been done
on the field. There are more than CMT, while doing my master thesis in
the field I discover LS-SCTP[2] or cmpSCTP[3] proposing different
interesting solutions to multipath issues.

When you take a look at this website (that you certainly know):
http://www.isoc.org/briefings/017/
you realize that using SCTP is more than simple multihomed capabilities.
I, personally, believe that letting SCTP to be alone to handle multipath
could boost his integration in the current internet.

By defending TCP modification, don't you think that those new protocols
could play the same role as NAT for IPv4 address starvation ? Maybe it's
an hard comparison but I miss IPv6. 

> could solve this using the DNS, but for TCP vs SCTP this is harder.

absolutely, this is not a trivial issue
> And you assume that this same middlebox would be compatible with SCTP.

should be compatible :(

> That is a signficant leap of faith, work for SCTP NAT is still going  

NAT is a poison on the internet (to me)

> ended variety in particular.

If you only consider multipath, yes, there is a place, especially if you
want the benefit as soon as possible, because this new protocol work
without modifying one end, you may implement it for yourself and already
use it.

regards,

Arnaud Ongenae
UCL (Belgium) student

[1] http://www.cis.udel.edu/~amer/PEL/poc/pdf/BickhartMSthesis.pdf

[2] Ahmed Abd El Al, Tarek Saadawi, and Myung Lee. Ls-sctp: a bandwidth
aggregation technique for stream control transmission protocol. Computer
Communications,
27(10):1012–1024, 2004. Protocol Engineering for Wired and Wireless
Networks.

[3] J. Liao, J. Wang, and X. Zhu. A multi-path mechanism for reliable
voip transmission
over wireless networks. Comput. Netw., 52(13):2450–2460, 2008.

---

From: "Preethi Natarajan" <[email-masked]>
Date: Fri, 8 May 2009 15:02:29 -0700

> Sent: Friday, May 08, 2009 1:09 PM
> To: Preethi Natarajan
> Subject: Re: [multipathtcp] Draft Multipath TCP (MPTCP) BOF 
> description
> 
> Suppose that a browser wants to load pages from a website. 
> How does the browser know if the server also supports SCTP? 
> For IPv4 vs IPv6 we could solve this using the DNS, but for 
> TCP vs SCTP this is harder.
> 

DNS has been proposed to identify transport protocols in the past, for ex.
see RFC 3263. From what I hear, SRV records have worked quite well for SIP.

> 
> Then we trade this off against the possible gain of not 
> having to change middleboxes. However, you assume that 
> incompatible middleboxes exist. While I'm sure this will be 
> the case a good percentage of the time, it's not a universal truism.

When NATs on different paths have to talk to each other to do the same
translation, I would assume that this requires changes to NAT
implementations...

> 
> So I think there is a place for multipath TCP in general and 
> the one- ended variety in particular.

I am sure it is. I am just wondering why the draft should be tied to TCP,
when (i) the one-ended multipath transmission could be adapted to any
transport, and (ii) some of the issues listed in
draft-van-beijnum-1e-mp-tcp-00 has been researched in-depth and solved for
another transport. The motivation to consider just TCP cannot be NAT
traversal, since, MPTCP itself might not work with the existing NAT
implementations?

Preethi

---

From: Janardhan Iyengar <[email-masked]>
Date: Sat, 09 May 2009 00:02:55 -0400

Scott,

> than probing, in particular).

Fully agree.  We have results to demonstrate this!

- jana

-- 
Janardhan Iyengar
Assistant Professor, Computer Science
Franklin & Marshall College
http://www.fandm.edu/jiyengar

---

From: Janardhan Iyengar <[email-masked]>
Date: Sat, 09 May 2009 01:56:53 -0400

Hi Iljitsch,

> SCTP to gain multipath benefits, this creates deployment issues. Suppose 
> that a browser wants to load pages from a website. How does the browser 
> know if the server also supports SCTP? For IPv4 vs IPv6 we could solve 
> this using the DNS, but for TCP vs SCTP this is harder.

Simultaneous TCP and SCTP connects. Yes, it is a hack, but it is a solution.

> addresses for the other side. 

SCTP peers exchange information about all addresses in the handshake; the application *does not* tell the initiator all the addresses for the other side.  The API is exactly the same as with TCP/UDP; in fact, Ryan Bickhart's thesis (referenced in an earlier email by Arnaud) exploited exactly this idea to give multihoming and fault tolerance benefits to any application by transparently mapping application-level TCP socket calls to SCTP function calls in-kernel.

> Then there is the issue that SCTP must of 
> course be implemented on both sides, while the big feature in the 
> one-ended MPTCP is that it can work if it's only implemented in the sender.

But to be useful, one-ended MPTCP will need to be understood by routers/middleboxes.

> Then we trade this off against the possible gain of not having to change 
> middleboxes. However, you assume that incompatible middleboxes exist. 
> While I'm sure this will be the case a good percentage of the time, it's 
> not a universal truism.
> 
> That is a signficant leap of faith, work for SCTP NAT is still going on 
> in the behave wg.

The ongoing work, if I am right, is for supporting multihoming, not basic support (which is trivial). And as I see it, that same work will become the template for multipathTCP as well, when you move to the "two-ended" mulipath case.

- jana

---

From: Janardhan Iyengar <[email-masked]>
Date: Sat, 09 May 2009 02:10:15 -0400

A couple of thoughts on the draft/BoF:

1/ "Two-ended" multipath, where both endpoints are aware:  There are several architectural and design considerations that will be common to TCP, SCTP, DCCP, SST etc., and as I see it, there is no reason why this discussion should not include all transports.

My proposal: scope the BoF more broadly.  Interested parties can then speak up about multipath for different transports and different points in the design space.  There will be shared goals, and lessons to be learned from one another.  Perhaps there can be a shared design considerations document, and then separate documents per transport. Perhaps two per transport.

2/ "One-ended" multipath seems very unclear to me... there is benefit in employing this "path" option in *any* transport. This should be a transport-neutral option, below the traditional transport.  (Perhaps in the shim6 shim layer? I am not sure yet...)  I am not convinced that there is a point in having this as a *TCP* option.  

I also do not understand how this is on the deployment path to two-ended TCP.  They seem like two different beasts: one of them needs router support to get going, the other one needs endpoint/NAT support to get going.  Don't get me wrong: I love the idea of having signaling between endpoints and routers to enable more control over multipath scheduling at the transport,  but that seems like a different trajectory than the "two-ended" multipath.  Others have said this, and I agree that "one-sided" multipath with router support does not seem to be scoped right... it seems to belong elsewhere, under the transport and independent of it.  Or am I missing something?

- jana

---

From: Joe Touch <[email-masked]>
Date: Fri, 08 May 2009 09:48:02 -0700

> Joe Touch escribió:
..
> But there are many situations and probably very interesting ones,
> where the node has multipaht paths and does not have multiple addresses.

Then you have different problems, and you need to separate them:

               |  single endpoint  |  multi endpoint
  -------------+-------------------+----------------
  single path  |       sp/se       |      sp/me
  -------------+-------------------+----------------
   multi path  |       mp/se       |      mp/me
  -------------+-------------------+----------------

Existing TCP assumes sp/se.

If you solve mp/me, you solve sp/me as a degenerate case.

So basically there are two distinct problems:

	1) making TCP work when there are multiple paths

		which has *nothing* to do with whether
		there are are multiple endpoints or not

		e.g., a single TCP connection can experience
		two different paths given multipath routing

	2) making TCP work with multiple endpoints

		which has *nothing* to do with whether
		there are multiple paths, since they
		different endpoints can easily experience
		the same path properties - including
		the same bottleneck. You can't assume
		that two different IP addresses (or other
		TCP endpoint differentiators) result in
		paths that do not share properties or
		bottlenecks.

		this *does* include striping, i.e.,
		concurrent use of alternate endpoints
		to increase BW; that is NOT the case for
		single-address.

		note that 'endpoint' could mean IP address
		or port, or anything else that could cause
		path properties to vary - keep in mind that
		policy routing peeks at the entire IP and often
		TCP headers.

Note that when I say "multiple paths" I mean paths with different
characteristics, e.g., MTU, RTT, BW, loss.

IMO, multiendpoint is a potentially important problem that may be ready
to solve via IETF work at this time. However, you need to be careful to
never assume that having multiple endpoints means distinct paths - if
they're not distinct in bottleneck, then you'll end up making TCP fight
with itself.

If you're going to solve multipath, then you need to solve it in the
single endpoint case first, IMO. If you can't solve it there, there's no
point in conflating the issue with multiendpoint.

> In  addition, even if you do have multiple addresses, you still need
> some support from the routing infrastrcuture.

IMO, you do not need support from the routing infrastructure at all. TCP
probes to detect path properties without interacting with routing right now.

You need to separate that out when you get distinct sets of values. All
you need to know is that you are getting a correlation between segments
being sent and varying path properties.

> destiantion address.

That's ONE way to do this. There are others.

> How can the nade A influence the egress path?

By varying the source address, source port, dest address, dest port, or
just about anything else in the IP or TCP header.

Joe

---

From: Salvatore Loreto <[email-masked]>
Date: Sat, 09 May 2009 19:38:36 +0300

> see RFC 3263. From what I hear, SRV records have worked quite well for SIP.
actually there is already an interesting proposal, take a look at those 
drafts:
http://www.ietf.org/internet-drafts/draft-wood-tae-specifying-uri-transports-05.txt
http://www.ietf.org/internet-drafts/draft-jennings-http-srv-05.txt

/Sal

---

From: Bryan Ford <[email-masked]>
Date: Sat, 9 May 2009 19:04:25 +0200

On May 9, 2009, at 6:38 PM, Salvatore Loreto wrote:
>>> From: Iljitsch van Beijnum [mailto:[email-masked]] Sent:  
>>> Friday, May 08, 2009 1:09 PM
>>> To: Preethi Natarajan
>>> Suppose that a browser wants to load pages from a website. How  
>>> does the browser know if the server also supports SCTP? For IPv4  
>>> vs IPv6 we could solve this using the DNS, but for TCP vs SCTP  
>>> this is harder.
>>>
>>>
>>
> actually there is already an interesting proposal, take a look at  
> those drafts:

Proposals like these are trying to take URIs in exactly the wrong  
direction, IMO, by trying to encode _more_ fragile information about  
"how exactly to access some object", rather than less as they should  
(i.e., what URNs try to do).  When someone gives you a URI or posts it  
on their web page, what you generally want is for your web (or SIP or  
bittorrent...) client to use that as the minimal information necessary  
to find and contact the desired service or object, and then negotiate  
online what specific suite of communication protocols and optional  
protocol features should be used for continuing communication.  That  
way the URI works when either the client or the server don't support  
some alternative transport or other protocol variation (i.e., they can  
fall back on a more basic protocol), in addition to when they do.   
When you encode a requirement to use a particular transport or  
protocol suite into the URI, the URI won't work at all unless both  
parties fully understand the indicated protocol suite.

There may be legitimate corner case uses of such mechanisms to encode  
protocol suites and options and such into URIs, but I haven't yet seen  
any convincing ones.  As far as I can tell the right and only solution  
to the general problem is to come up with a better way to negotiate  
protocol suites and features dynamically.

Bryan

---

From: Bryan Ford <[email-masked]>
Date: Sat, 9 May 2009 19:39:44 +0200

> elsewhere do not hesitate to redirect me)

I think this "multipath TCP versus multipath SCTP" question is  
important, but still misses a larger issue: SCTP and TCP are different  
transports providing substantially different application-visible  
semantics, but we would really like to have multipath support for  
_both_ of those transports (and perhaps others).  It's a pain for TCP  
applications to switch to SCTP because it involves programmers  
learning a new API with different semantics: e.g., the application has  
to break byte streams up into (reasonable-size) messages and use  
different system calls to tag them with a stream number and send them  
on an SCTP connection.  On the other hand, SCTP provides features that  
some applications really want that TCP doesn't, such as a partially- 
ordered connection where one stream's data can pass another stream's.   
Furthermore, these semantics are pretty much totally orthogonal to  
multipath issues: the transport functionality that needs to be  
modified for multipath is fairly independent of and readily separable  
from the transport functionality that provides application-visible  
semantics like TCP's byte streams or SCTP's message-oriented multi- 
streams.  The modifications required for multipath support in general  
mainly affect the transport's congestion control functionality, and  
many research projects and deployed protocols have demonstrated how  
congestion control can be cleanly separated from the application- 
semantic part of the transport: e.g., the Congestion Manager  
architecture, several prior variants of TCP and SCTP (e.g., Split-TCP,  
pTCP, mTCP, LS-SCTP), and my own Structured Stream Transport (SST).

So if we're going to start a BOF or WG about multipath something-or- 
other, why should it be restricted to adding multipath to one  
particular transport (and then get into debates about which transport  
that should be), when what we really need is a clean way to add  
multipath to _any_ transport?  Whether to do that by literally  
splitting the transport into multiple layers, as the HotNets-08 paper  
Jana and I wrote proposes (http://brynosaurus.com/pub/net/logjam-abs.html 
), or simply by developing a principled approach to adding multipath  
support incrementally to existing "monolithic" transports while  
considering the needs of multiple transports, amounts to technical  
details that need to be discussed by whatever BOF or WG gets formed.   
Expanding the BOF charter's scope from "multipath TCP" to "multipath  
transport" wouldn't even need to mean that all transports will  
necessarily get equal attention or that solutions for each transport  
would be developed in lock-step; it just means recognizing that there  
is more than one transport that applications and users care about and  
that would like multipath support.

Bryan

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 9 May 2009 20:08:29 +0200

On 9 mei 2009, at 19:39, Bryan Ford wrote:

> SCTP and TCP are different transports providing substantially  
> different application-visible semantics,

Another difference is that TCP is extremely simple, while SCTP has a  
more complex structure with its chunks. This and the CRC32 checksum  
make SCTP processing more CPU intensive than TCP processing. I don't  
know how big the difference is, but I'd say that SCTP isn't  
appropriate in very constrained environments (although the constrained  
environments from a few years ago, such as cell phones, are much more  
capable today, but there are devices that are even _more_ constrained)  
or very high-performance environments. Even on my 2 GHz dual core Mac  
I need to use jumboframes to saturate gigabit ethernet over TCP.

> particular transport (and then get into debates about which  
> transport that should be), when what we really need is a clean way  
> to add multipath to _any_ transport?

There are three parts to all of this:

1. Congestion control. We haven't talked about this much yet, but  
there is a lot of work to do here, which of course applies equally to  
TCP, SCTP and other transports using TCP-like congestion control

2. Multipath mechanisms. TCP doesn't have these, and we now have two  
drafts proposing them. There are papers and even implementations  
skinning this cat in different ways. We currently have a single  
address and a multiaddress proposal. Would of course be nice to be  
able to do both, as each has limitations that the other doesn't have.  
SCTP does multiaddress multipath and it could be extended with the  
single address multipath similar to TCP but the details will be  
different (option space etc) (2a: TCP, 2b: SCTP)

3. Interaction with routing. In the current draft, the assumption is  
that the host acting as a TCP sender sends packets belonging to  
different subflows in different directions, without help from the  
routing system. However, the routing system has many more  
opportunities to make path choices than hosts so if we can take  
advantage of that we get many more paths and thus better multipath.

So the question is, what should the BoF be about? 1, 2a, 2b, 3? Just  
2a? More than just 2a but less than everything?

My take:

 From a practical perspective, I can tell you that putting congestion  
control and routing protocol people in the same room isn't always very  
effective. I think we have a pretty good handle on the 2a issues, so  
we can certainly do some good engineering in this area in the IETF. If  
there is significant interest in 2b, then perhaps 2a and 2b could be  
done at the same time.

I don't think we can unleash modified transport protocols that may be  
used for 85% of the internet's traffic without addressing congestion  
control, so I guess 1 needs to be in scope. However, we should curb  
are ambitions in this area, or we won't have any usable results for a  
long time. If we can get the "first do no harm" part of the multipath  
congestion control changes done, then the rest is probably best left  
to the research community rather than the IETF.

As for 3: someone made the argument that changing IP and routing would  
be a hard sell to the IESG and it would need to happen elsewhere  
anyway. I agree. It would be more effective to wait with this work  
until parts 1 and 2 are in good shape, and then figure out if, how and  
when 3 should be done.

---

From: John Leslie <[email-masked]>
Date: Sat, 9 May 2009 14:42:41 -0400

> TCP, SCTP and other transports using TCP-like congestion control

   Absolutely! This is work (IMHO) best done under an Experimental RFC,
and I am hoping we can produce such a beast, describing the experiment.
(That could, of course, be IRTF work, but an IETF WG could make much
faster progress.)

> different (option space etc) (2a: TCP, 2b: SCTP)

   We shouldn't gloss over these differences. I believe a TCP-only
implementation can be done more quickly, and be deployed much more
quickly for a large-scale experiment (which I believe necessary to
establish _practical_ congestion-control principles).

> advantage of that we get many more paths and thus better multipath.

   I disagree that more paths necessarily means better multipath. In
particular, congestion control becomes exponentially harder when the
endpoints don't know from moment to moment which paths are being used.

   Also, deploying _any_ routing change to the actual big-I Internet
will be a slow process. I most strongly advise against putting it in
any critical path of work we hope to accomplish.

> effective.

   An understatment I'd be proud of!

> I think we have a pretty good handle on the 2a issues, so  we can
> certainly do some good engineering in this area in the IETF.

   I agree, and I'd hesitate to crowd too much more into early milestones.

> If there is significant interest in 2b, then perhaps 2a and 2b could be  
> done at the same time.

   This is a good question. I see a deployment benefit in having a single
code base for both, but I can't imagine how to get there. Absent a single
code base, it would seem we could get faster deployment by taking them
serially instead of in parallel.

> our ambitions in this area, or we won't have any usable results for a  

   I wouldn't go quite that far: we will need field experiments, which
will progress faster in IETF than IRTF. Also "no harm" is in the eye
of the beholder, and we need to better quantify such a promise.

> until parts 1 and 2 are in good shape, and then figure out if, how and  

   I agree. Changing _anything_ about routing would have to coordinate
with Routing Area, which would greatly complicate the process. I believe
we can implement (2a) _entirely_ without changes to routing or even
interaction with middleboxes. If I am right in that belief, scoping our
work (initially) to (1) and (2a) will get us into congestion-control
experiments much more quickly.

   (If I haven't been clear, let me state explicitly: we can't possibly
satisfy everyone's congestion-control desires by arguing positions: we
will need experimental results.)

--
John Leslie <[email-masked]>

---

From: Pasi Sarolahti <[email-masked]>
Date: Mon, 11 May 2009 11:11:57 +0300

> interesting solutions to multipath issues.

There is one common practical reason: I think multihomed SCTP (or any  
protocol that explicitly communicates multihomed IP addresses inside  
transport header) does not work over NATs, even if encapsulated  
inside UDP. I think it is possible to design multipath TCP so that it  
can be made to work over existing NATs.

- Pasi

---

From: Pasi Sarolahti <[email-masked]>
Date: Mon, 11 May 2009 11:24:22 +0300

On May 9, 2009, at 9:10, ext Janardhan Iyengar wrote:

> 1/ "Two-ended" multipath, where both endpoints are aware:  There  
> are several architectural and design considerations that will be  
> common to TCP, SCTP, DCCP, SST etc., and as I see it, there is no  
> reason why this discussion should not include all transports.

This might be a sensible idea: there is an expired draft on multipath/ 
mobile DCCP (draft-kohler-dccp-mobility-02) that is a bit similar to  
the multipath TCP variant with implicit address management.

- Pasi

---

From: "Rolf Winter" <[email-masked]>
Date: Mon, 11 May 2009 10:33:00 +0200

Regarding the scope of the BoF, I agree with Iljitsch here. Multi-path TCP, or any transport for that matter, that uses multiple address pairs from more than a single operator will use different paths which are as diverse as BGP will allow them to be. So, not having any interaction with routers/routing will already make multi-path transport an interesting thing to look at. Once we know how to do the multi-path TCP bit right, we can come back and see whether we can make it even better assuming that routing is involved. But unless we understand congestion control across multiple subflows and the gory engineering details of the protocol, broadening the scope beyond the purely transport related issues will unnecessarily complicate things at this point.

NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road, London W3 6BL | Registered in England 2832014 

> From: [email-masked] [mailto:multipathtcp-
> [email-masked]] On Behalf Of Iljitsch van Beijnum
> Sent: Samstag, 9. Mai 2009 20:08
> To: Bryan Ford
> Cc: [email-masked]
> Subject: [multipathtcp] TCP/SCTP and BoF description,was: Re: TCP
> modification vs switching to SCTP
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 11 May 2009 12:31:48 +0200

> see RFC 3263. From what I hear, SRV records have worked quite well  

That's because you can't do SIP without it. For HTTP it hasn't been  
successful.

Also note that SRV records only supply a port number and a weight/ 
priority, not a protocol number. So without modifications, it wouldn't  
be possible to use SRV records to differentiate between TCP and SCTP.

> I am sure it is. I am just wondering why the draft should be tied to  

So if this BoF results in a wg, and that wg would take on this work  
for both TCP and SCTP, how would that work? Would we have different  
documents for TCP and for SCTP, or one document? Is anyone  
volunteering to spend time on this for SCTP?

---

From: "Preethi Natarajan" <[email-masked]>
Date: Mon, 11 May 2009 08:53:43 -0700

> Sent: Saturday, May 09, 2009 11:08 AM
> To: Bryan Ford
> 
> Another difference is that TCP is extremely simple, while 
> SCTP has a more complex structure with its chunks. This and 
> the CRC32 checksum make SCTP processing more CPU intensive 
> than TCP processing. I don't know how big the difference is, 
> but I'd say that SCTP isn't appropriate in very constrained 
> environments 

1. CRC32c might be an intensive operation. Nonetheless CRC32c was chosen for
its superior error detection mechanisms. 
2. As you mention, the notion of "very constrained envrionments" has been a
moving target -- what we think is "very constrained" today might not be in
the future.

My larger point -- why are _we_ trying to decide which transport would be
best for today's/tomorrow's requirements? Instead, I would prefer that the
consumers (app developers) make that decision. If the app developer doesn't
want the compute-intensive SCTP and wants to use DCCP or XYZ, then so be it.
However, this can happen only if the BoF or WG is cordial to transports
other than TCP...

> > DNS has been proposed to identify transport protocols in 
> the past, for 
> > ex.
> > see RFC 3263. From what I hear, SRV records have worked 
> quite well for 
> > SIP.
> 
> That's because you can't do SIP without it. For HTTP it 
> hasn't been successful.

I was under the impression that nobody has even tried to do this with
SCTP... there are a few upcoming proposals but thats about it? I would say
its premature to say that it hasn't been successful.

> So if this BoF results in a wg, and that wg would take on 
> this work for both TCP and SCTP, how would that work? Would 
> we have different documents for TCP and for SCTP, or one 
> document? Is anyone volunteering to spend time on this for SCTP?

I know that there are volunteers for SCTP, and am sure there will be
volunteers for other transports as well. 

Preethi

---

From: Lloyd Wood <[email-masked]>
Date: Mon, 11 May 2009 17:20:46 +0100

CRC32c isn't intensive compared to e.g. message digests.

The choice of a better checksum mechanism for SCTP was predicated on  
the fact that the ones-complement checksum used in TCP/UDP is very  
weak, and doesn't catch swapped words - which is how echo servers  
worked way back when. Swap the addresses, the ones-complement checksum  
doesn't even notice.

See RFC3309 for some reasons for choosing CRC32c over the previous  
Adler32 (which is still better than the UDP/TCP checksums).

The size and complexity of the SCTP state machine and whether its  
features are desirable or necessary are better embedded arguments  
against SCTP than its choice of checksum to protect itself.

> moving target -- what we think is "very constrained" today might not  

DTN work: http://info.ee.surrey.ac.uk/Personal/L.Wood/saratoga/

<http://info.surrey.ac.uk/Personal/L.Wood/><[email-masked]>

---

From: "Preethi Natarajan" <[email-masked]>
Date: Mon, 11 May 2009 10:43:42 -0700

> From: Lloyd Wood [mailto:[email-masked]] 
> Sent: Monday, May 11, 2009 9:21 AM
> To: Preethi Natarajan
> Cc: Lloyd Wood; 'Iljitsch van Beijnum'; 'Bryan Ford'; 
> [email-masked]
> Subject: Re: [multipathtcp] TCP/SCTP and BoF description, 
> was: Re: TCP modification vs switching to SCTP
> 
> The size and complexity of the SCTP state machine and whether 
> its features are desirable or necessary are better embedded 
> arguments against SCTP than its choice of checksum to protect itself.
> 

SCTP is more complex since SCTP offers more services/features than TCP. But,
there are apps/environments for which SCTP's features improve performance in
spite of the associated complexity... 

> DTN work: http://info.ee.surrey.ac.uk/Personal/L.Wood/saratoga/

As I mentioned before we shouldn't be trying to decide which transport would
be most suitable for tomorrow's app requirements. For example, Saratoga was
invented because TCP wasn't good enough for certain environments, and I
doubt if these environments were even envisioned when TCP was designed...

Personally, I think that we should open up the playing field so that app
developers have a variety of transports to choose from...

Preethi

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 11 May 2009 20:51:02 +0200

>                |  single endpoint  |  multi endpoint

I am confused now, what do you mean by multiple endpoint?

So far, i understood we were talking about to endpoints communicating 
through TCP and where there were multiple paths between them. the 
question was whether we can use different address pairs (assigned to the 
two endpoints involved) as path selectors, but i am afriad we went out 
of synch and you have something else in mind

> 		two different paths given multipath routing
well, i think there is andditional case which is to make TCP use the 
multiple paths simultaneously, which TCP doesn't do currently

> 		paths that do not share properties or
no, i can't be certain that they will not share the smae bottle neck, 
but what may be done is that if they do have the smae bottleneck, they 
behave as single TCP flow, while if they do not have the same bottle 
neck, they actually use BW availbale.

> 		to increase BW; that is NOT the case for
why not?

It is possible to use a different path selector, set by TCP, that allows 
different packets belonging to the same TCP connectio to be routed 
through different paths.

> 		policy routing peeks at the entire IP and often
i am having difficuties to match this with the previous sentence... I 
mean, we agree that we can have a single address and use other fileds 
than the address as path selector?

> characteristics, e.g., MTU, RTT, BW, loss.
right

> they're not distinct in bottleneck, then you'll end up making TCP fight
exactly, this is critical in this effort.

> point in conflating the issue with multiendpoint.
i am still having problems to understand your description of the 
problem, i.e. the multiendpoint vs, the multipath

how would you scope the proble only to deal with multipaht and not with 
multiendpoint?
> being sent and varying path properties.

exactly, that is what i call a path selector
a path selector is something in the packets that TCP sets and that other 
part of the network looks at and sends packets with different paths 
selectors through different paths.
For TCP, all it is needed is that the packets with the same path 
selector are routed through the same path (or at least as stabel as it 
is today in the single path case).  also, it is useful, that packets 
with different path selectors are routed through different paths but if 
this doesn't happen, the systme should work as a single path TCP
> That's ONE way to do this. There are others.

such as?
(this is sincere question. We have been trying to understand how to do 
this in shim6 for many years and we couldn't find very nice answers)
> just about anything else in the IP or TCP header.
right, but in the case the destination node has a single address, all 
the rest of the fields are not currently cosndiered by routing, so we do 
require some modification in routing, afaict (but i would love to be 
convinced otherwise)

Regards, marcelo

> iEYEARECAAYFAkoEYkIACgkQE5f5cImnZrtEzwCgqTmskXCmmAKi/SJHpw5A5ZRj
> eukAnRkO0vPBnEwDf1wzJ1VG1wQs46dz
> =1KaA

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 11 May 2009 21:05:37 +0200

Preethi Natarajan escribió:
> another transport. The motivation to consider just TCP
the motivation for considering TCP is that the vast majority of traffc 
uses TCP and not SCTP. that seems to be a fact and it seems that 
defining multipaht capabilities to TCP would provide additional benefits 
for those users.

>  cannot be NAT

As described in the proposed BOF, there are two possible approaches to 
MPTCP, the one ended one and the two ended one
the two ended one works fine with nats
The one ended one, woudl require modificatiosn to the NATs of the ends 
that supports it. this means that if a site decides to deploy the one 
ended version of MPTCP in their servers, they will need tp update their 
NATs. There  is no need to update the NAT on the side that is unchanged.
So, changes are only required by the party deploying the solution, which 
doesn't seem to be the case in SCTP, since afaiu, if the peer has a nat, 
it no longer works.

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 11 May 2009 21:09:35 +0200

Janardhan Iyengar escribió:
>> the browser know if the server also supports SCTP? For IPv4 vs IPv6 
>> we could solve this using the DNS, but for TCP vs SCTP this is harder.
>
> SCTP peers exchange information about all addresses in the handshake; 
> the application *does not* tell the initiator all the addresses for 
> the other side.  The API is exactly the same as with TCP/UDP; in fact, 
> Ryan Bickhart's thesis (referenced in an earlier email by Arnaud) 
> exploited exactly this idea to give multihoming and fault tolerance 
> benefits to any application by transparently mapping application-level 
> TCP socket calls to SCTP function calls in-kernel.
>
>> Then there is the issue that SCTP must of course be implemented on 
>> both sides, while the big feature in the one-ended MPTCP is that it 
>> can work if it's only implemented in the sender.
>
>
right, but only by the middleboxes in control of the side supporting the 
OmTCP. I don't think there is need for middlebox support on the side 
that is not MTCP capable.
So, the whole point is that if one end updates their servers and 
potentially their middleboxes, they can support this.

Regards, marcelo

>> And you assume that this same middlebox would be compatible with 
>> SCTP. That is a signficant leap of faith, work for SCTP NAT is still 
>> going on in the behave wg.
>
> The ongoing work, if I am right, is for supporting multihoming, not 
> basic support (which is trivial). And as I see it, that same work will 
> become the template for multipathTCP as well, when you move to the 
> "two-ended" mulipath case.
>
> - jana
>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 11 May 2009 21:18:50 +0200

Janardhan Iyengar escribió:
> 1/ "Two-ended" multipath, where both endpoints are aware:  There are 
> several architectural and design considerations that will be common to 
> TCP, SCTP, DCCP, SST etc., and as I see it, there is no reason why 
> this discussion should not include all transports.
>
right, i agree with that

> My proposal: scope the BoF more broadly.  Interested parties can then 
> speak up about multipath for different transports and different points 
> in the design space.  There will be shared goals, and lessons to be 
> learned from one another.  Perhaps there can be a shared design 
> considerations document,

right, that seems in the scope for the proposed charter imho
> and then separate documents per transport. Perhaps two per transport.
>

i am not sure about this one. I am not sure if a single group would be 
the ideal place to discuss all transports
>
> 2/ "One-ended" multipath seems very unclear to me... there is benefit 
> in employing this "path" option in *any* transport. This should be a 
> transport-neutral option, below the traditional transport.  (Perhaps 
> in the shim6 shim layer? I am not sure yet...)  I am not convinced 
> that there is a point in having this as a *TCP* option. 
mmm, the comment about the TCP options was general in the charter. I 
mean, the sentence that says that we can only do tcp options applies to 
both two ended and single ended.
In other words, it is scoping the kind of changes we can do to TCP, in 
general.

> I also do not understand how this is on the deployment path to 
> two-ended TCP.  They seem like two different beasts: one of them needs 
> router support to get going, the other one needs endpoint/NAT support 
> to get going.
which one is which?
any transport that handles multiple paths need some way to send 
different packets throug different path, this is true for MPTCP (one 
ended or two ended), tru for SCTP and true for shim6

If multiple addresses are used, somehtings work, but still we need to 
deal with the case of changing the source address that needs to have 
some impact on routing.

I mean, the two ended case, need updating both ends and probably some 
routing of the involved sites,
the one ended flavour, needs updating one side, including the server, 
the routing and the middle boxes.

They have different deployment vectors.
In the one ended case, the site that deploys it, gets inmediate benefit 
(while more limited than the two ended version)
In the two ended version, more benefits can be achived, but needs both 
ends supporting it.

>   Don't get me wrong: I love the idea of having signaling between 
> endpoints and routers to enable more control over multipath scheduling 
> at the transport,  but that seems like a different trajectory than the 
> "two-ended" multipath.
not really, what is needed is that the routing take into account 
something defined by the MPTCP layer in order to define the egress router.

>   Others have said this, and I agree that "one-sided" multipath with 
> router support does not seem to be scoped right... it seems to belong 
> elsewhere, under the transport and independent of it.  Or am I missing 
> something?
>
mmm, have you read the draft?

regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 11 May 2009 21:25:51 +0200

Bryan Ford escribió:
> semantics,
exactly!

> but we would really like to have multipath support for _both_ of those 
> transports (and perhaps others).
sure

> It's a pain for TCP applications to switch to SCTP because it involves 
> programmers learning a new API with different semantics: e.g., the 
> application has to break byte streams up into (reasonable-size) 
> messages and use different system calls to tag them with a stream 
> number and send them on an SCTP connection.  On the other hand, SCTP 
> provides features that some applications really want that TCP doesn't, 
> such as a partially-ordered connection where one stream's data can 
> pass another stream's.  Furthermore, these semantics are pretty much 
> totally orthogonal to multipath issues: the transport functionality 
> that needs to be modified for multipath is fairly independent of and 
> readily separable from the transport functionality that provides 
> application-visible semantics like TCP's byte streams or SCTP's 
> message-oriented multi-streams.  The modifications required for 
> multipath support in general mainly affect the transport's congestion 
> control functionality, and many research projects and deployed 
> protocols have demonstrated how congestion control can be cleanly 
> separated from the application-semantic part of the transport: e.g., 
> the Congestion Manager architecture, several prior variants of TCP and 
> SCTP (e.g., Split-TCP, pTCP, mTCP, LS-SCTP), and my own Structured 
> Stream Transport (SST).
>
> So if we're going to start a BOF or WG about multipath 
> something-or-other, why should it be restricted to adding multipath to 
> one particular transport (and then get into debates about which 
> transport that should be), 
i don't think we need that debate
As you say, it is a good idea to have them in all the transport protocols.

> when what we really need is a clean way to add multipath to _any_ 
> transport?  Whether to do that by literally splitting the transport 
> into multiple layers, as the HotNets-08 paper Jana and I wrote 
> proposes (http://brynosaurus.com/pub/net/logjam-abs.html), or simply 
> by developing a principled approach to adding multipath support 
> incrementally to existing "monolithic" transports while considering 
> the needs of multiple transports, amounts to technical details that 
> need to be discussed by whatever BOF or WG gets formed.  Expanding the 
> BOF charter's scope from "multipath TCP" to "multipath transport" 
> wouldn't even need to mean that all transports will necessarily get 
> equal attention or that solutions for each transport would be 
> developed in lock-step; it just means recognizing that there is more 
> than one transport that applications and users care about and that 
> would like multipath support.
there is a tradeoff here between having enough arhcitectural perspective 
and having a charter so broad that is impossible to have concrete output.
I perosnally think that a reasonable middleground is what Jana has proposed.
We can have an architectural document about multipath transports in 
general, and focus this partiuclar effort in multiapth tcp.
If after, based on the architectural document, other efforts for other 
transport protoco spring, it woudl be a great otucome.

I am very concerned that if we loose the focus of the work, the effort 
will get nowhere

Regards, marcelo

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 11 May 2009 21:32:59 +0200

> its superior error detection mechanisms.

I understand. However, there are times when spending CPU cycles (I'm  
assuming the SCTP checksum calculation can't be offloaded to hardware)  
needed to do this are not worth the benefit it provides.

> consumers (app developers) make that decision.

Certainly, I agree.

> SCTP... there are a few upcoming proposals but thats about it?

Like I said, SRV records only have a port number, not a protocol number.

> volunteers for other transports as well.

Ok, let's think about this.

With regard to the single ended case: in addition to the multipath  
congestion control that is common to all types of multipath with  
"normal" congestion control, the tricky parts here are how to handle  
fast retransmit, how to handle stalls because of receive buffer  
depletion and correlating incoming ACKs with the path the ACKed  
packets were sent over.

Without having studied SCTP in detail, it seems to me that the one- 
ended multipath TCP draft could easily be applied to SCTP with minor  
changes. This could be done during the TCP work or immediately after  
so a single document can apply to both if there are no  
incompatibilities. If the SCTP stuff requires a lot of extra text, it  
probably makes more sense to make a separate document.

AS for the two-ended multiaddress multipath: SCTP already does all the  
protocol work, the only thing that's needed is apply the multipath  
congestion control.

Iljitsch

---

From: John Leslie <[email-masked]>
Date: Mon, 11 May 2009 15:51:22 -0400

marcelo bagnulo braun <[email-masked]> wrote:
> MPTCP, the one ended one and the two ended one:

   I quite agree, though I'm not sure we have any consensus on that.

   As I see it, so long as each "path" is distinguished by a unique
source/destination pair, the routing structure can do all the work,
and we can even arrange for SYNs to travel "first" outbound through
any NAT(s).

> The one ended one, would require modifications to the NATs of the ends 
> that supports it.

   I'm not quite that pessimistic. The one-ended version can't, of course,
distinguish paths by source/destination pairs, but the chief implementation
I see would be content providers with two (or more) upstream paths and no
NATs in them.

   The content-receiving end could have an unmodified NAT, which would be
blissfully ignorant of which path a packet takes.

   It's not immediately clear to me that there's any point in designing
special-purpose NATs for the one-ended case...

> this means that if a site decides to deploy the one ended version of
> MPTCP in their servers, they will need to update their NATs.

   Again, I'm not quite so pessimistic; but indeed, if a content-provider
has a NAT on one or more upstream paths, it would need to find what
address/port that NAT translates to. This may be challenging for an
unmodified NAT; but it's not impossible by definition...

> There  is no need to update the NAT on the side that is unchanged.

   Agreed.

---

From: Joe Touch <[email-masked]>
Date: Mon, 11 May 2009 13:39:13 -0700

> I am confused now, what do you mean by multiple endpoint?

More than one socket pair - more than one port and/or address.

> of synch and you have something else in mind

You can have different paths when you have different:
	- endpoint addresses
	- port numbers
	- or neither

Paths do not have anything to do with addresses per se.

> multiple paths simultaneously, which TCP doesn't do currently

That's case (1) above. A single TCP connection can already see - and use
- - multiple paths simultaneously, and trips badly when it happens. E.g.,
if long packets go one way and short ones go another, this will mess up
TCP's RTT calculation.

> neck, they actually use BW availbale.

When they share a bottleneck, they can easily end up with WORSE than a
single TCP of BW, because the two TCP 'sides' can fight with each other.
You'd be probing the bottleneck from both sides, and a probe on one side
can affect the other, etc.

> It is possible to use a different path selector, set by TCP, that
> allows different packets belonging to the same TCP connectio to be
> routed through different paths.

Yes - that can increase path BW, but not endpoint BW, AFAICT.

> than the address as path selector?

Yes, we're agreeing.

> problem, i.e. the multiendpoint vs, the multipath

Multi-endpoint uses different socket pairs at one or more endpoint.

Multipath experiences two different paths at the same time. Those two
paths may or may not be correlated with the use of different endpoints.

> how would you scope the proble only to deal with multipaht and not with multiendpoint?

You make TCP figure out 'path' by 'path characteristics', not by
assuming it from differences in endpoint IDs. Which is important,
because you can use different endpoint IDs and end up with the same
path. You need to find out path agnostically.

> a path selector is something in the packets that TCP sets and that
> other part of the network looks at and sends packets with different
> paths selectors through different paths. For TCP, all it is needed is
> that the packets with the same path selector are routed through the
> same path (or at least as stabel as it is today in the single path
> case).  

That's reasonable to assume, but isn't always true.

> also, it is useful, that packets with different path
> selectors are routed through different paths but if this doesn't
> happen, the systme should work as a single path TCP

That's not reasonable to assume.

> (this is sincere question. We have been trying to understand how to
> do this in shim6 for many years and we couldn't find very nice answers)

Use different source ports. Use different destination addresses. Use
different destination ports.

> than the address as path selector?

Yes. What I'm having problems with is assuming that different selectors
- - of any type - can assume different paths.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 11 May 2009 23:00:08 +0200

> Paths do not have anything to do with addresses per se.

[...]

> single TCP of BW, because the two TCP 'sides' can fight with each  

No, that shouldn't reduce throughput unless there is synchronization.  
In fact, in the classic sawtooth case it should even improve  
throughput somewhat.

But are you saying that when MPTCP invokes a number of paths, we MUST  
be able to determine which ones share bottlenecks?

Detecting identical paths shouldn't be a huge problem: here packets  
from different subflows should (almost) never be reordered relative to  
each other.

However, if the paths only partially overlap the part that's different  
can still introduce reordering so looking at that won't help.

And despite the paper that (I think) Jana posted a link to a few days  
ago, I don't think correlating the losses is robust enough to detect  
the shared bottlenecks.

I guess it would be possible to back off on one path and then see what  
happens to others and repeat this until it's clear whether or not  
there is a strong correlation, but this would take significant time,  
be fairly complex, and not work very well on links with very many  
sessions where the impact of a few subflows isn't visible above the  
noise.

So I think we should proceed under the assumption that shared  
bottlenecks will happen and multipath congestion control should behave  
in a way that isn't unduly harmful when it does.

If only we had an internet with different service classes, we could  
relegate some sessions to a lower than best effort class so we don't  
get in the way of normal traffic but still have the extra bandwidth  
when it's available.

BTW, I would prefer to stay away from the "endpoint" terminology  
because it's not entirely clear what it means. In shim6 a path is  
defined as a (source, dest) address pair. This is probably not a good  
use of the word in the context here, but "address pair" is very clear.  
Rather than just IP addresses we can also talk about transport  
addresses, which is address + port. A logical path is whatever we put  
a labal on, whether it maps to something different than another  
logical path or not. I've also used "physical path" to talk about the  
actual route that packets follow through the network although this is  
of course on layer 3 not layer 1.

---

From: Joe Touch <[email-masked]>
Date: Mon, 11 May 2009 14:10:21 -0700

Iljitsch van Beijnum wrote:
> No, that shouldn't reduce throughput unless there is synchronization. In
> fact, in the classic sawtooth case it should even improve throughput
> somewhat.
> 
> But are you saying that when MPTCP invokes a number of paths, we MUST be
> able to determine which ones share bottlenecks?

Yes.

> Detecting identical paths shouldn't be a huge problem: here packets from
> different subflows should (almost) never be reordered relative to each
> other.
>

Right.

> ago, I don't think correlating the losses is robust enough to detect the
> shared bottlenecks.
> 
> happens to others and repeat this until it's clear whether or not there
> is a strong correlation, but this would take significant time, be fairly
> complex, and not work very well on links with very many sessions where
> the impact of a few subflows isn't visible above the noise.
> 

Yes. Why does that not mean that you're back to single-path TCP behavior?

> relegate some sessions to a lower than best effort class so we don't get
> in the way of normal traffic but still have the extra bandwidth when
> it's available.
> 
> BTW, I would prefer to stay away from the "endpoint" terminology because
> it's not entirely clear what it means.

It has a known meaning in TCP - I'm talking about a socket (an
address/port pair).

Joe

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 12 May 2009 00:22:26 +0200

> More than one socket pair - more than one port and/or address.
from the application perspective, i would expect that it all looks like 
a single normal TCP socket. I mean, i would expect that any modification 
we make is transparent to the apps, so apps can take advantage of it 
without requiring any modification.

that doesn't mean we need to limit ourselves to a single address pair, 
but that the different address pairs are managed by the MPTCP layer, in 
a transparent manner

would that make sense?

> Paths do not have anything to do with addresses per se.

Right
But, in some cases, having different address pairs, does imply to have 
different paths.
But in any case, in the general situation, i think we need to 
accommodate multiple path selectors, which can be any kind of bits, from 
address pairs, ports, options, or other bits in the header
> if long packets go one way and short ones go another, this will mess up
but they don't use it simultaneously afaict
and if they do, TCP as you say is not designed to do that.
The proposal is to allow TCP to be aware that packets are being routed 
through alternative paths, characterize each of these paths based on 
congestion and decide which packets need to be forwarded through each path.

> single TCP of BW, because the two TCP 'sides' can fight with each other.
right, the idea here is to try to use coupling of the congestion control 
windows to avoid this negative effect, as per Kelly's work.

> Yes - that can increase path BW, but not endpoint BW, AFAICT.
how come?
mmm, i think i am probably doing some unstated assumptions here...
I am assuming multihoming somewhere, probably in one of the sites 
involved in the the communication.

I mean, if congestion control is limiting the amount of data in flight, 
then if we have an additional path, we will be able to push more 
traffic, right?

> Multi-endpoint uses different socket pairs at one or more endpoint.

right, this is not what i have in mind when i wrote the proposed charter
the idea is that from the socket perspective, this is exactly like a TCP 
connection
> paths may or may not be correlated with the use of different endpoints.
if you mean that different paths may or may not be related with having 
different address pairs, i agree that it is not universally true, but it 
does seems to be true in many cases (large enough to construct on top of 
that imho)

> assuming it from differences in endpoint IDs.
exactly
TCP will characterize each path available based on the characteristics 
it can measure, in particular congestion along each path
Now, then we need an interface that translates the paths as seen by TCP 
and the packet on the wire. I call this the path selector. These are 
some bits that reflect the notion of which path TCP thinks it is being 
used. So, at some point the path selector must be used by the IP or 
routing so that path with different path selectors, actually flow 
through a different path, makes sense?

>  Which is important,
agree

> That's not reasonable to assume.
what is not reasonable?
that path with different selector will be routed through different paths 
or that if this doesn't happen the multiple flows should behave as a 
single TCP flow?

right

So, we have multiple options here, but there is no universal solution 
that applies to all scenarios.

In many cases, using different address pairs will result in different 
paths. Not always, but i think that it will happen in enough amount of 
cases to make it useful
The other option is to have different next hops. Suppose a multihomed 
site, and that we can either use different egress router as next hops. 
In this case, the path selector is something local in the host, that TCP 
tells IP that a given packet needs to use one next hop and another 
packet should use a different next hop. We can even use tunnels to the 
egress routers and take it from there.
So, there is myriad of ways we can do this. Probably none of them will 
be universal.
So, i am not sure if we need to describe them or just leave it open for 
people to use whatever suits them.

 From the discussion so far, it seems that at least we need to be more 
explicit about the relation between MPTCP and IP/routing.

Regards, marcelo

> iEYEARECAAYFAkoIjPAACgkQE5f5cImnZrvc7gCgjfAKgarnOnbNHrEfNjmV7s7F
> u24AnAz3LgMqhvxfD6J4RvJvQTEixtF0
> =UoHK

---

From: Fred Baker <[email-masked]>
Date: Mon, 11 May 2009 16:06:00 -0700

>> which determine routing. How about finer-grain routing issues?

Like what if I have multiple paths through the same domain? The same  
source/destination address pair has multiple routes that it can use?  
For the sake of argument, we could postulate that they have different  
AS paths between the local ISPs or at least discrete routing paths.

What I would like to understand is what signals MPTCP would like to  
give the network and what it would like the network to do with them.  
Using different address pairs is a pretty heavy club, but may be  
workable.

---

From: Joe Touch <[email-masked]>
Date: Mon, 11 May 2009 18:32:15 -0700

...
> from the application perspective, i would expect that it all looks
> like a single normal TCP socket. I mean, i would expect that any
> modification we make is transparent to the apps, so apps can take
> advantage of it without requiring any modification.

The goal is to make this all transparent to apps, regardless of what's done.
The question is "how many problems are you trying to solve at once", or,
more specifically, which problems are you trying to solve?

> But, in some cases, having different address pairs, does imply to have different paths.

Different addresses MAY result in different paths, but doesn't have to.
IMO, you need to *test* for that condition, not assume it.

> address pairs, ports, options, or other bits in the header

Sure.

> but they don't use it simultaneously afaict

They can.

> and if they do, TCP as you say is not designed to do that. 

TCP isn't. "MULTIPATH", IMO, is.

 Or if that's not what this is focusing on, you're using the wrong word
- - you mean "multiendpoint".

> The
> proposal is to allow TCP to be aware that packets are being routed

You need to determine whether there are multiple paths. I don't accept
the assertion that "different endpoints == different paths", because
neither => nor <= is true ('implies', in either direction).
...
> then if we have an additional path, we will be able to push more

If there are two ****PATHS****. You don't have two PATHS when you have
two ENDPOINTS. They're just not the same thing.
...
> the idea is that from the socket perspective, this is exactly like a TCP connection

There are two different viewpoints:

	1) the endpoints

	2) the network

The network can see two different socket pairs, even if the endpoints
see one socket pair.

> does seems to be true in many cases (large enough to construct on top of

TCP was symmetric in most cases, until the web came along.

Designing for the currently common case is a disaster waiting to happen.
If you can test for distinct paths, do so. If not, then don't design for
it based on an assumption.

Or maybe we should call this "ethernet TCP", since -- in most cases --
it'll run over ethernet, right?

> exactly TCP will characterize each path available based on the
> characteristics
> Now, then we need an interface that translates the paths as seen by
> TCP and the packet on the wire. I call this the path selector. These are

Sounds like you're trying to bury routing inside of TCP to me.

...

Yes.

> or that if this doesn't happen the multiple flows should 
> behave as a single TCP flow?

No - I agree with that.

...
> egress routers and take it from there.

Nope. Different hops tells you that the path diverges, but not that the
path properties differ - or that the defining path properties aren't
dominated by a shared path downstream.

> So, there is myriad of ways we can do this. Probably none of them
> will be universal.
> So, i am not sure if we need to describe them or just leave it open
> for people to use whatever suits them.
> 

Yes, but that's exactly what concerns me.

Joe

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 12 May 2009 08:46:57 +0200

> give the network and what it would like the network to do with them.
right, this is exactly the point

the idea as i understand it is:
MPTCP will characterize different paths based on congestion. It will 
then associate these paths to a different path selector (those signals 
you say above) and what is expected from the IP layer is that different 
path selectors are routed through different paths, if available.

> Using different address pairs is a pretty heavy club, but may be workable.
Agree. However, in my shim6 experience, people were very reluctant to 
use source address dependent routing.
AFICT, we have the following options:
- source dst address
- different local interfaces (from the MPTCP host)
- different next hops (from the MPTCP host)
- different tunnels to different egress routers (from the MPTCP host)
- the flow label (only in IPv6)
- DSCP bits
- magical use of fragmentation fields
- some new IP option (sigh)
- some new TCP option (even more sigh)

and all of them are very far from being a nice solution i am afraid.

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 12 May 2009 09:24:31 +0200

> more specifically, which problems are you trying to solve?

We agree that using multiple paht if available improve the performance 
of the communication (i.e. BW, fault tolerance), right?
The idea is to allow TCP to get these benefits, without involving the apps.
> IMO, you need to *test* for that condition, not assume it.
that is one option.
the other option is that if you have multiple addresses, you try to use 
them. In the best case you will get different paths with different 
bottlenecks and better performance. In the worse case, you are like a 
single TCP. Of course, in order to take this approach, we must make sure 
that two flows of a single TCP don't fight against each other making 
things worse.

> - - you mean "multiendpoint".
mmm, i am not sure.
multiendpoint seem to assume different address paris or at least 
different port pairs, but it is possible that the path selector is a 
different next hop, in which case, it hardly fits in the multiendpoint 
definition (as i understand it)

the goal is multipath. We are using multiendpoint as one of the tools to 
get multipath, without guarantees that we will get it, if we do so. 
Would that be a good description?

> neither => nor <= is true ('implies', in either direction).
Let me see if i get what you mean,

Let's suppose we have a MPTCP host. This host is configured to use a 
given path selector. Let' consider two cases: option a) different 
address pairs option b) different next hops.

Now, in the case of different address pairs, there are two different 
cases. When the source address changes and when the dst address changes. 
If the intra domain routing of the site that is hosting the MPTCP node 
has been modified to support source address dependent routing, the mptcp 
host knows that using different soruce address will result in different 
paths. Using differnet dest address, if available, it is not clear what 
effect that will have. there may result in more path diversity or not.
So, in this case, i think it is reaosnable to assume that if multiple 
addresses are available , at least idfferent soruce addresses will 
result in different paths.

In the option b with a different next hop, it is reasonable to assume 
that if mptcp is deployed in this scenario, the different next hops will 
result in different egress paths. In this case, there is no guarantee 
that there is path diversity in the reverse path.

does this makes sense to you?

What i am trying to say is that mptcp is likely to be deployed using a 
path selector that would result in path diversity. If not, it would not 
be exposed to mptcp i.e. nobody will take the burden of configuring the 
paht selectors in the mptcp if no path diversity is available.

> The network can see two different socket pairs, even if the endpoints
ok,

> it based on an assumption.
what would be the disaster?
I mean, i think it is a sin equanum condition to require that if 
multiple subflows run over the same path, they do not fight against each 
other.
I mean, if we design in such a way that in the case there is not path 
diversity, or in the case there is path diversity but both paths share 
the same bottleneck, things are not worse than in the single path tcp, 
we should be ok, don't you think?

> it'll run over ethernet, right?
:-)

> Sounds like you're trying to bury routing inside of TCP to me.
but routing doens't characterize routes based on congestion. the only 
one that knows about congestion is the transport and that is what we 
need to make a wise split of the load. So, clearly we are trying to get 
a closer relationship between routing and congestion control, but each 
of them still performs its role i think. What needs to be defined is the 
interface between routing and MPTCP, and that interface is the path 
selector we have been disucssing so far.

agree

> Yes, but that's exactly what concerns me.

i understand. We should try to present the assumptions more explciitly, 
so we can have a discussion.

Regards, marcelo

> iEYEARECAAYFAkoI0Z8ACgkQE5f5cImnZruSHACfTQ9YpxJBbaZgZzeTnoZsfT2n
> 1V8An3XSQVZkKWm9Shls4zKEna1gRtvV
> =Kx9h

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 12 May 2009 13:07:07 +0200

> Yes.

Ok. I think it can't be done reliably.

(And note that in the one-ended case the multiple paths must always  
converge towards the interface holding the remote address.)

Also, why would we care?

If we simply limit the number of active subflows to something modest  
we're not doing anything that HTTP isn't doing, or that users aren't  
doing.

> Yes. Why does that not mean that you're back to single-path TCP  

Because it's acceptable to be more aggressive than a single NewReno  
flow by a factor 2 or 3 or so.

>> because

> It has a known meaning in TCP - I'm talking about a socket (an

That terminology may be useful in normal TCP but in multipath TCP we  
need to be more precise.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 12 May 2009 13:16:52 +0200

On 12 mei 2009, at 1:06, Fred Baker wrote:

> Like what if I have multiple paths through the same domain?

Why would the domain be noteworthy?

I can imagine the situation where both ends are multihomed to  
different ISPs and two paths that they use are completely disjoint at  
all levels.

I can also imagine the situation where the ends are actually single  
homed and all hops are identicial except one, which consists of set of  
bundled links between two routers. Because of the limitations on how  
ECMP works, it could easily be that one link in the bundle is  
saturated while the other one as a lot of unused capacity.

There are also situations in between. For instance, the packets  
between two neighboring ASes could flow through different interconnect  
locations. Or there could be path diversity within the AS.

All that multipath TCP sees is different RTTs and loss rates for the  
different subflows.

> Using different address pairs is a pretty heavy club, but may be  

Well, ideally we would have some bits X somewhere in the IP header  
(less ideal: a non-IP header) that tell a router "if you have multiple  
choices in the FIB for this destination, pick #X".

However, I think it's premature to start usurping header bits before  
we have a good handle on how issues like fast retransmit, buffer  
depletion and congestion control coupling between the subflows would  
work.

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 May 2009 05:19:16 -0700

Iljitsch van Beijnum wrote:
> we're not doing anything that HTTP isn't doing, or that users aren't doing.
> 
> Because it's acceptable to be more aggressive than a single NewReno flow
> by a factor 2 or 3 or so.

No, that exceeds the recommendations in standards-track documents for TCP.

> That terminology may be useful in normal TCP but in multipath TCP we

No, you need to be precise about what you're talking about.  A socket is
very precise for TCP, just insufficient for MPTCP.

Joe

---

From: John Leslie <[email-masked]>
Date: Tue, 12 May 2009 12:56:31 -0400

Iljitsch van Beijnum <[email-masked]> wrote:
>> [Iljitsch van Beijnum <[email-masked]> wrote:]
>> 
>>> But are you saying that when MPTCP invokes a number of paths, we  
>>> MUST be able to determine which ones share bottlenecks?
> 
>> Yes.
> 

   Which makes it a suboptimal goal. I would advise against accepting it.
(This does not mean we shouldn't have goal(s) of mitigating problems
_caused_by_ shared bottlenecks; merely that we shouldn't state a goal
as "avoid any shared bottlenecks".)

> converge towards the interface holding the remote address.)

   That's not quite true; but it will usually be the case.

> Also, why would we care?

   Exactly!

   Let's list a few reasons we might wish to avoid shared bottlenecks.
I'll start:

1) oscillation (it's all to easy to design congestion control that will
   oscillate between two paths, confusing congestion signals for all
   and giving less overall bandwidth than a single path);

2) over-saturation of the "innocent" end's link (the textbook example
   is one user at an enterprise with marginal bandwidth can ruin the
   experience of other users at that enterprise);

3) too much retransmission of lost packets (leading to both loss of
   bandwidth and choppy response).

   (Others, please feel free to add...)

> we're not doing anything that HTTP isn't doing, or that users aren't  

   "Something modest" may be in the eyes of the beholder. Iljitsch,
I believe is thinking three; HTTP used to think four but now is two;
in the extreme case we could get arbitrarily close to one...

   Is it worth extensive discussion _before_ the BoF whether our
target should be one or three?

   Also, does anyone want to claim that the target should be _less_
than one?

>>> bottlenecks will happen and multipath congestion control should  
>>> behave in a way that isn't unduly harmful when it does.
> 

   Even if it isn't, there's a benefit to content-providers of fault
tolerance and load balancing which chooses the best path for each
flow.

>>> because it's not entirely clear what it means.
> 

   In particular, the application has no need to deal with separate
"sockets" for a single "flow" over the multiple paths.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 13 May 2009 00:00:53 +0200

>   and giving less overall bandwidth than a single path);

Right, oscillations are never good although I don't think less total  
bandwidth will happen easily.

>   experience of other users at that enterprise);

The trouble with Reno congestion control is that the amount of loss on  
a link goes up with the square of the number of flows. So if everyone  
starts using 3 subflows average packet loss will go up by a factor 9.

Unfortunately it's hard to replace Reno because if you're less  
aggressive you get less bandwidth so it doesn't pay.

>   (Others, please feel free to add...)

Unfairness. If we both share the bottleneck with a regular flow we  
would get about a 50% share of the bandwidth each (small print  
applies) but if I use 3 subflows through that same bottleneck I'll get  
75% and you 25%.

Although not a huge deal, managing paths that provide no benefits  
creates unnecessary extra overhead.

> in the extreme case we could get arbitrarily close to one...

Yes, it is in the eye of the beholder but don't forget that the  
internet is built on the principle of "one cuts from thick wood" which  
is a Dutch expression meaning that we don't waste our time with  
subleties but only care about getting things approximately right.

> target should be one or three?

First of all, the 3 in draft-van-beijnum-1e-mp-tcp-00.txt is just a  
placeholder.

Do we agree that the magic number is the number of MSSes per RTT that  
the sum of the cwnds of the subflows belonging to a session is allowed  
to grow?

(We then want to limit the per-subflow cwnd growth to >= 0 and <= 1,  
the former to avoid one subflow eating up another's cwnd, which would  
create oscillations, and the latter to avoid that we just grow a  
single subflow cwnd by a factor X of the normal increase, because that  
would give us the best performance but no multipath.)

In order to be no slower than a regular TCP session using the fastest  
path, we MUST grow the subflow over the fastest path by 1 MSS per RTT.  
So if we want to use additional subflows, the aggressiveness factor  
needs to be > 1.

Opinions?

I'm saying 3 for now, which is on the high side, but within reason, IMO.

Don't forget that if I implement 9000-byte jumboframes I get no less  
than 36 times more bandwidth than users with 1500-byte packets.  
Compared to that, a factor 3 is quite modest.

>   Also, does anyone want to claim that the target should be _less_

So a multipath TCP would be required to be slower than a regular TCP  
session? I'm not seeing the big selling point for multipath here...

>> it's acceptable to be more aggressive than a single NewReno

> tolerance and load balancing which chooses the best path for each

That's true.

---

From: Bryan Ford <[email-masked]>
Date: Wed, 13 May 2009 13:46:16 +0200

>   and giving less overall bandwidth than a single path);

I've never quite understood exactly what the problem is here.  Any  
reasonable congestion control algorithm for the Internet must be  
stable in the presence of multiple competing flows sharing the same  
bottleneck: Reno does this by ensuring via its AIMD response that  
competing flows converge toward giving each flow a more-or-less even  
share (at least among flows of similar RTT; different RTTs as we know  
affect fairness but AFAIK do not affect stability).  What does the  
congestion control algorithm care if two competing flows happen to be  
fed from the same data source?  That only affects the contents of the  
packets that get sent, not congestion control behavior.  If single- 
path congestion control is stable in the presence of competing  
independent flows, then a multi-path transport that directs data from  
one source over multiple otherwise-independent congestion control  
flows should be equally stable.

Maybe you're thinking of a situation where the data source doesn't  
provide enough data to keep all active flows network-bottlenecked, but  
even in such situations it seems it should be easy to ensure stability  
(assuming the underlying congestion control algorithm is stable)  
merely by not doing something really stupid like trying to shift _all_  
data from one path to another just because the first path experiences  
some congestion.  There are at least two "reasonable approaches" I can  
think of offhand to distributing data among only partially- 
bottlenecked flows that should trivially ensure stability (again  
assuming the underlying CC algorithm is stable for independent data  
sources), and I'm sure there are plenty more.

(1) impose a consistent total order on the flows, and always max out  
the first before sending any data on the second, max out the second  
before sending data on the third, etc.  Then for given source load and  
network conditions, the same first set of flows will always be network- 
bottlenecked and behave like independent TCP flows, and the remaining  
flows will not be network-bottlenecked and thus CC will be  
irrelevant.  This approach might be what you want in asymmetrical  
"primary/backup link" or "cheap/expensive link" situations.

(2) always distribute traffic evenly across all flows up to the point  
there's no more data to distribute or a given flow becomes network- 
bottlenecked.  Then the network-bottlenecked flows will remain  
consistently network-bottlenecked and will behave like independent  
congestion controlled flows even if they compete with each other at  
some bottlenecks, while for non-network-bottlenecked flows congestion  
control is irrelevant because it just won't kick in.

\Presumably somebody must have already analyzed these types of  
scenarios formally and/or experimentally, or am I missing something  
fundamental here?

>   bandwidth and choppy response).

These are just instances of the same "aggressiveness amplification"  
issue you get whenever you use multiple possibly-competing TCP flows  
to support one logical activity.  The same thing happens at  
application level with HTTP's two-concurrent-connections convention,  
or with Bittorrent's many-concurrent-connections behavior, especially  
when most or all of those connections happen to be bottlenecked at the  
same home or enterprise broadband link.  It's not in any way specific  
to multipath TCP or multipath transports; it's just an artifact of  
TCP's model of competition and fairness.  We routinely live with this  
problem in the case of HTTP and Bittorrent; why is it suddenly so  
critical that multipath TCP (or multipath SCTP or whatever) has to  
solve it before multipath transports can be designed or deployed?  And  
any solution this BOF/WG might come up with for multipath transport  
probably wouldn't solve the more general problem of fairness and  
aggressiveness amplification, e.g., in the bittorrent case where the  
problem results from many different flows to different and  
administratively unrelated endpoints.

I'm not saying that we shouldn't work on solving TCP's aggressiveness  
amplification or fairness problem for multi-flow usages (either by  
multi-flow transports or multi-flow applications), just that it seems  
quite independent and orthogonal to the details of any particular  
multipath transport, and proposed solutions should ideally consider  
not just the multipath transport case but also the more general point- 
to-multipoint case that bittorrent exhibits.  Unless there's a really  
good reason not to - e.g., if the former is possible and the latter is  
simply not - but I don't believe that and haven't seen any argument to  
that effect.

Cheers,
Bryan

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 14 May 2009 16:56:16 -0400

> potentially their middleboxes, they can support this.

That assumes that the entity that updates the servers also controls _all_ middleboxes that will see incomplete connection information.  What happens if I want to employ MPTCP and my ISP uses NATs?

- jana

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 14 May 2009 17:16:12 -0400

> right, that seems in the scope for the proposed charter imho

I am not able to read it in the charter.  "In particular, the proposed work is to extend TCP to support simultaneous data exchange through multiple paths" pretty clearly seems to restrict the charter to MPTCP only.

> the ideal place to discuss all transports

It is not ideal if you scope the group to be a "TCP" group, and not a "transport" group. There are ideas that are beyond specific transports---Bryan mentioned our HotNets08 paper that I am hoping to present at this IETF that would fit this bill perfectly---that need to be discussed.  

Sure we want to deal with engineering issues at the IETF, but it is important for the engineering effort to be focussed in a mutually agreed upon direction.  Opening up the floor to discussion about multipath transport is necessary, IMHO.

And if we agree on how multipath should be done, interested parties could write up drafts on how different transports would map to it.  I understand the concern about scoping too broadly and not getting anything done, but  the concern on the other side is that scoping too narrowly may yield suboptimal designs for the long term.

> mmm, have you read the draft?

Yes, I did.  And I am still not clear on why one-sided "options" are TCP options; from where I'm seeing it, the option should be transport independent.  What am I missing?

- jana

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 14 May 2009 17:35:05 -0400

John,

> establish _practical_ congestion-control principles).

That assumes that there will be a single mechanism that will apply equally to all transports, and we are not there yet.  My design for CMT is different from the one proposed in the draft for MPTCP, and it will be worthwhile discussing the differences (CMT uses a single sequence number space, MPTCP and SST use two sequence spaces; there are arguments for both sides.)  TCP considerations will be different than ones for SCTP (option space limitations, for instance), and I expect they will be *very* different for DCCP (different ack mechanisms).

Again, I think that this BoF should be broadly scoped to allow for multiple parallel/competing architectures.  We should be cautious not to scope the group too narrowly yet.  I sincerely think that there are shared questions here that cut across transports, and there will be significant differences.  Architecturally, this conversation is exactly where we should discuss where multipath belongs on the stack.  Why not have a common flow layer for *all* transports that does multipath?  We don't have to agree to split up the transport (as Bryan and I have proposed), but it will help us see the questions here from a perspective that is beyond TCP.  And we can then map the answers back to TCP.  Incidentally, this is how I remember the "Resource-Pooling" paper as being scoped.

- jana

---

From: John Leslie <[email-masked]>
Date: Thu, 14 May 2009 19:17:45 -0400

Janardhan Iyengar <[email-masked]> wrote:
> [John Leslie <[email-masked]> wrote:]
> That assumes that there will be a single mechanism that will apply
> equally to all transports, and we are not there yet.

   I think we're in violent agreement there.

> My design for CMT is different from the one proposed in the draft for
> MPTCP, and it will be worthwhile discussing the differences (CMT uses
> a single sequence number space, MPTCP and SST use two sequence spaces;
> there are arguments for both sides.)

   Exactly the sort of thing we need to discuss under the first Charter
item...
" 
" - A set of architectural guidelines for congestion dependent
"   multipath transport protocols. Since different transport protocols 
"   can potentially benefit from this approach, this document describes
"   the motivations and the general approach that should be followed
"   to enable congestion dependent multipath transport.

> TCP considerations will be different than ones for SCTP (option 
> space limitations, for instance), and I expect they will be *very* 
> different for DCCP (different ack mechanisms).

   I tend to agree.

> Again, I think that this BoF should be broadly scoped to allow for
> multiple parallel/competing architectures. We should be cautious
> not to scope the group too narrowly yet.

   Is that first Charter item too narrow?

> I sincerely think that there are shared questions here that cut across
> transports, and there will be significant differences. Architecturally,
> this conversation is exactly where we should discuss where multipath
> belongs on the stack. Why not have a common flow layer for *all* 
> transports that does multipath?

   I don't follow how we'd get there. Implementations already pretty
thoroughly obscure "layer" boundaries.

   While I am very sympathetic to the idea of Transport layers that know
how to do multipath, I am not convinced that "one true way" is even a
desirable goal.

> We don't have to agree to split up the transport (as Bryan and I have
> proposed), but it will help us see the questions here from a
> perspective that is beyond TCP.

   That would be good, during the architecture discussion.

> And we can then map the answers back to TCP.

   _Iff_ we (ever) agree on the "one true way"...

   (Seriously, I don't think we're any more likely to converge on "one
true way" that other groups are to converge on a single "congestion
control" algorithm.)

> Incidentally, this is how I remember the "Resource-Pooling" paper
> as being scoped.

   We're talking "engineering" here, not academic papers. We tend to
avoid areas where we need to debate what would be "best."

   I'm very happy to see interest by academic types in this subject.
But unless there's a place for engineers to dive in and code something,
I fear the project would drag on forever. :^(

---

From: "Rolf Winter" <[email-masked]>
Date: Fri, 15 May 2009 13:25:55 +0200

When I read parallel and competing architectures I read IRTF not IETF. Thinking about it, when I read competing architectures I am not sure this still qualifies for being a BoF. 

> multiple parallel/competing architectures.  We should be cautious not
> to scope the group too narrowly yet.  I sincerely think that there are
> shared questions here that cut across transports, and there will be
> significant differences.  Architecturally, this conversation is exactly
> where we should discuss where multipath belongs on the stack.  Why not
> have a common flow layer for *all* transports that does multipath?  We
> don't have to agree to split up the transport (as Bryan and I have
> perspective that is beyond TCP.  And we can then map the answers back
> to TCP.  Incidentally, this is how I remember the "Resource-Pooling"
> paper as being scoped.
> 
> - jana
> 
> --
> Janardhan Iyengar

---

From: Michael Tüxen <[email-masked]>
Date: Sun, 17 May 2009 12:04:42 +0200

> CRC32c isn't intensive compared to e.g. message digests.
Hi Lloyd,

CRC32C offloading is now available in 3rd generation Intel NICs
and supported by the next release of FreeBSD. I assume that
other NICs will also support it in the future and therefore
the impact of CRC32C becomes less...

Best regards
Michael

> worked way back when. Swap the addresses, the ones-complement  
> checksum doesn't even notice.
>

---

From: Michael Tüxen <[email-masked]>
Date: Sun, 17 May 2009 12:06:27 +0200

On May 11, 2009, at 9:32 PM, Iljitsch van Beijnum wrote:

> assuming the SCTP checksum calculation can't be offloaded to  
> hardware) needed to do this are not worth the benefit it provides.
It can be using 3rd gen. Intel NICs. Support for it has recently been
added to FreeBSD.
> incompatibilities. If the SCTP stuff requires a lot of extra text,  
> it probably makes more sense to make a separate document.
>
> AS for the two-ended multiaddress multipath: SCTP already does all  
> the protocol work, the only thing that's needed is apply the  
> multipath congestion control.
>
> Iljitsch

---

From: Janardhan Iyengar <[email-masked]>
Date: Sun, 17 May 2009 14:39:30 -0400

Hi John,

>    Is that first Charter item too narrow?

No, but the second one is: "In particular, the proposed work is to extend TCP to support simultaneous data exchange through multiple paths."

I think the charter at this time should be about multipath transport, and while while TCP can be one of the transports, it should be just that.  TCP should not be the end-all for this group.

> how to do multipath, I am not convinced that "one true way" is even a

I am talking here about transport layers shipping multipath out of the transport, and leaving it to a different entity; let's call this entity the "flow" layer. But as I've said, we don't have to agree yet to doing this; I think it is a powerful idea, and I think the group should allow space for discussion of it.

> true way" that other groups are to converge on a single "congestion

There may be properties that an application desires from its transport such as low jitter or high throughput, and these properties are important when determining a congestion control algorithm.  Similarly, we will need to figure out how different multipath techniques can work given the different demands that applications are likely to have. Why shouldn't we discuss these needs, and work on multipath techniques for protocols such as DCCP that has non-Reno congestion control, if there is interest in it?  We don't have to arrive at the one "true" way of doing multipath for all applications---I agree with you that there is probably no one way---but we should surely be discussing these different application needs, and ways of doing (or not doing) multipath for them.

My larger point is that we need to discuss other transports and not only TCP.

> avoid areas where we need to debate what would be "best."

This paper is a reference from the BoF description: http://trac.tools.ietf.org/area/tsv/trac/wiki/MpTcpBofDescription

I could be wrong, but my understanding is that that paper has a strong influence on this BoF and the two drafts that were circulated.

Engineering doesn't mean that we stop debating and simply code the first draft that is circulated.  If this discussion means that this project belongs in the IRTF and not in the IETF yet, so be it.  Better that than to have a solution where coding can happen immediately but without sufficient discussion.

> I fear the project would drag on forever. :^(

I won't blame academic-types for that.  Academics aren't the ones causing heartburn on all dragged-out projects in the IETF.

- jana

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 17 May 2009 22:54:15 +0200

On 17 mei 2009, at 20:39, Janardhan Iyengar wrote:

> I think the charter at this time should be about multipath  
> transport, and while while TCP can be one of the transports, it  
> should be just that.  TCP should not be the end-all for this group.

So how would that work? Do we, for each TCP mechanism, specify the  
corresponding SCTP mechanism? And then the same for DCCP and RTP?

Or do we stop just shy of specifying specific implementation details?

> I think the group should allow space for discussion of it.

I don't think anyone is trying to stop any discussions here. However,  
there is a difference between having a discussion and a work item in  
the charter.

> My larger point is that we need to discuss other transports and not  

What is it that you want to discuss? Nobody is stopping you.  :-)

---

From: Janardhan Iyengar <[email-masked]>
Date: Sun, 17 May 2009 17:28:59 -0400

> I don't think anyone is trying to stop any discussions here. However, 

I wasn't trying to say that... apologies if I came across that way.

> there is a difference between having a discussion and a work item in the 
> charter.

Good point.  I was not making a strong point; only that if there are interested parties that want to work on SCTP and/or DCCP, then that should happen here.  I heard someone (was it Pasi?) say that there may be a draft for DCCP.  I will try to  put together CMT experiences in a draft too.

- jana