Subject: [multipathtcp] Draft MPTCP Charter

From: "Ford, Alan" <[email-masked]>
Date: Tue, 1 Sep 2009 15:27:46 +0100

Hi all,

Following on from the BOF, we do of course need a MPTCP charter to take
this WG further. I have drafted the following text based on the initial
discussion points of the BOF and the feedback/trends that were gleaned
from the session. Feedback is greatly appreciated!

Multipath TCP (MPTCP) Working Group Draft Charter
=================================================

The Multipath TCP (MPTCP) working group develops mechanisms that add the
capability of using multiple paths to a regular TCP session. This covers
both protocol extensions needed to deploy MPTCP, as well as adaptations
to congestion control to safely support multi-path resource sharing.

This working group will provide a modular architecture that is designed
to be easily extensible and adaptable. In particular, the congestion
control mechanism is separated from the mechanisms for identifying and
using multiple paths.

Initially the working group will focus on a solution requiring
modification of both endpoints. One or both endpoints are expected to
have multiple addresses which can be used to provide multiple paths
between the endpoints. Whilst this will require modification of the TCP
stack at the endpoints, all existing applications should be able to use
the new MPTCP stack without modification. In due course, an extended API
may be defined to allow applications to express particular preferences
over MPTCP operation, and gather information about MPTCP-specific
behaviour. A key measure of the success of this working group is that
any protocol modifications are deployable in an unmodified Internet and
work with existing unmodified applications. It is also essential that
all new threats introduced by MPTCP are carefully analysed and addressed
in the appropriate protocol documents.

The following components will be defined:

a. A set of architectural guidelines for congestion dependent multipath
transport protocols. This is a working document to track the technical
documents, describing how the modular components fit together, and will
describe the motivations and the general approach that should be
followed to enable congestion dependent multipath transport. In
addition, this document will provide a threat analysis of multipath
transport.

b. A basic coupled congestion control algorithm
This will specify a multipath equivalent of SACK/NewReno congestion
control.
As experience is gathered from implementations of other congestion
control algorithms, it is expected that the working group will charter
additional work in these areas.

c. Extensions to current TCP to support mutli-addressed mutipath TCP. 
This covers all on-the-wire changes required to create a two-ended MPTCP
solution using multiple addresses at one or both ends, and will include
a basic security model. It will be designed in a modular fashion to
allow alternative address management schemes to be explored in future.
This document must also include an analysis of threats introduced by
these specific MPTCP extensions.

In addition, the following two items are excluded from this charter but
could form part of a re-charter once the initial work described above
has been completed.

d. An extended API describing how applications can exploit additional
features of multipath transport. 
While MPTCP must be usable without any application changes, this API
should be an optional extension that provides access to multipath
information and enable the control of the usage. The necessity of this
work will be determined during the development of the MPTCP protocol
itself.

e. Guidance for implementers of single-ended multipath TCP
Single-ended multipath TCP does not require any on-the-wire protocol
changes to be standardised, and as such this document will focus on
practical implementation experience.

The initial goal is that the documents produced by the WG will be either
experimental or informational. However, should the experiments prove
successful the WG reserves the right to subsequently ask the IESG to
advance one or more documents as Proposed Standard.

Goals and Milestones:

Aug 2010 - Submit Architectural Guidelines and Threat Analysis as an
informational RFC
Aug 2010 - Submit basic coupled congestion control as an experimental
RFC
Aug 2010 - Submit protocol specification for MPTCP extensions as an
experimental RFC

Regards,
Alan

---

From: Marc Blanchet <[email-masked]>
Date: Tue, 01 Sep 2009 10:37:04 -0400

- looks good.
- suggestion to add: "Coordination and non-overlap of work with the MIF
working group is expected".
- should we have a specific document on threat analysis/security
considerations?

Marc.

> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

-- 
=========
IPv6 book: Migrating to IPv6, Wiley. http://www.ipv6book.ca
Stun/Turn server for VoIP NAT-FW traversal: http://numb.viagenie.ca
DTN news service: http://reeves.viagenie.ca

---

From: <[email-masked]>
Date: Tue, 1 Sep 2009 15:52:03 +0100

Inline.

Toby

> -----Original Message-----
> From: [email-masked] [mailto:multipathtcp-
> [email-masked]] On Behalf Of Marc Blanchet
> Sent: 01 September 2009 15:37
> To: Ford, Alan
> Cc: Multipath TCP Mailing List
> Subject: Re: [multipathtcp] Draft MPTCP Charter
> 
> - looks good.

Easiest thing is to include a sentence or two explaining why there isn't an overlap with their charter... By IETF rules the two groups have to be working on different things don't they?

> - should we have a specific document on threat analysis/security

I think it is better to include this within the specific protocol documents as in Alan's suggested charter below. Main reasons are a) the important thing is new threats that are introduced and these are likely to be specific to certain address management approaches and b) If you have a separate document it means implementers have to look in more than one place to get the information they need. That's not to say that in future there may turn out to be some more general security lessons to be learnt, and if that looks like being the case then perhaps those should go in to the architectural guidelines document.

---

From: Mark Handley <[email-masked]>
Date: Tue, 1 Sep 2009 15:55:58 +0100

> working group is expected".

Agreed.

> - should we have a specific document on threat analysis/security

We considered a separate threat analysis document, but it seemed
better to include it in the architectural guidelines document.  This
allows threats to be discussed in the context of specific
architectural changes and mechanisms, rather than having to duplicate
discussion of these.  The protocol-specific issues get dealt with in
the protocol spec itself.  It is very important we do a good job of
threat analysis, but splitting it out seems like it makes it more of a
tick-box that will be hard to get people to work on.  Including it
makes it the job of the architects to ensure this is covered properly.

 - Mark

---

From: <[email-masked]>
Date: Tue, 1 Sep 2009 20:20:10 +0100

Looks good to me.

Timescales - I think it's good to set a quite short timescale. Since
there was a high level of active interest at the BoF (both from people
wanting the results & people who seemed willing to do work) - and also
as there's an excellent starting position for all 3 work items. 

In item b, I'd delete "basic" as I think this sets the wrong impression.
I also wonder whether it's worth setting the objective here a bit more,
eg more throughput than ordinary tcp over the best of the paths, eg
something like: better than a simple scheme where you divide the
congestion response equally over the n paths [ie regardless of what path
the drop was on]. You could add some text near the end of the long para
in the intro, "Another key measure of the success of this working group
is that the coupled congestion algorithm achieves..."

I like the idea of the architecture etc doc being a living doc, so that
it appears along with the other docs. 

It might be nice if the architecture doc also included "a summary of the
results of the expts". (in general the ietf seems to struggle to capture
the results of its 'experiments'). Here it could be seen as part of the
reasoning for the final form of the protocols and perhaps the
recommendation for what's done next (eg in terms of moving to Stds track
or working on an extended api).

( A key measure of the success of this working group is that
{ any protocol modifications are deployable in an unmodified Internet
and
{ work with existing unmodified applications.
"any" should be deleted, or changed to "the TCP"
It's not just deployability in the unmodified internet  - (1) it needs
to work! (2) if one end is not mptcp it needs to fall back to tcp (I
think it's worth adding this requirement). I'm not sure whether it's
worth adding something explicit about middleboxes (I think probably what
you've written is enough).
Perhaps you could something like add "(although modified apps may gain
further benefits)"

Is it the IESG who would be asked about changing to stds track? Maybe
"the community"?

Thanks
phil
{ -----Original Message-----
{ From: [email-masked]
[mailto:[email-masked]]
{ On Behalf Of Ford, Alan
{ Sent: 01 September 2009 15:28
{ To: Multipath TCP Mailing List
{ Subject: [multipathtcp] Draft MPTCP Charter
{ 
{ Hi all,
{ 
{ Following on from the BOF, we do of course need a MPTCP charter to
take
{ this WG further. I have drafted the following text based on the
initial
{ discussion points of the BOF and the feedback/trends that were gleaned
{ from the session. Feedback is greatly appreciated!
{ 
{ 
{ Multipath TCP (MPTCP) Working Group Draft Charter
{ =================================================
{ 
{ The Multipath TCP (MPTCP) working group develops mechanisms that add
the
{ capability of using multiple paths to a regular TCP session. This
covers
{ both protocol extensions needed to deploy MPTCP, as well as
adaptations
{ to congestion control to safely support multi-path resource sharing.
{ 
{ This working group will provide a modular architecture that is
designed
{ to be easily extensible and adaptable. In particular, the congestion
{ control mechanism is separated from the mechanisms for identifying and
{ using multiple paths.
{ 
{ Initially the working group will focus on a solution requiring
{ modification of both endpoints. One or both endpoints are expected to
{ have multiple addresses which can be used to provide multiple paths
{ between the endpoints. Whilst this will require modification of the
TCP
{ stack at the endpoints, all existing applications should be able to
use
{ the new MPTCP stack without modification. In due course, an extended
API
{ may be defined to allow applications to express particular preferences
{ over MPTCP operation, and gather information about MPTCP-specific
{ behaviour. A key measure of the success of this working group is that
{ any protocol modifications are deployable in an unmodified Internet
and
{ work with existing unmodified applications. It is also essential that
{ all new threats introduced by MPTCP are carefully analysed and
addressed
{ in the appropriate protocol documents.
{ 
{ The following components will be defined:
{ 
{ a. A set of architectural guidelines for congestion dependent
multipath
{ transport protocols. This is a working document to track the technical
{ documents, describing how the modular components fit together, and
will
{ describe the motivations and the general approach that should be
{ followed to enable congestion dependent multipath transport. In
{ addition, this document will provide a threat analysis of multipath
{ transport.
{ 
{ b. A basic coupled congestion control algorithm
{ This will specify a multipath equivalent of SACK/NewReno congestion
{ control.
{ As experience is gathered from implementations of other congestion
{ control algorithms, it is expected that the working group will charter
{ additional work in these areas.
{ 
{ c. Extensions to current TCP to support mutli-addressed mutipath TCP.
{ This covers all on-the-wire changes required to create a two-ended
MPTCP
{ solution using multiple addresses at one or both ends, and will
include
{ a basic security model. It will be designed in a modular fashion to
{ allow alternative address management schemes to be explored in future.
{ This document must also include an analysis of threats introduced by
{ these specific MPTCP extensions.
{ 
{ In addition, the following two items are excluded from this charter
but
{ could form part of a re-charter once the initial work described above
{ has been completed.
{ 
{ d. An extended API describing how applications can exploit additional
{ features of multipath transport.
{ While MPTCP must be usable without any application changes, this API
{ should be an optional extension that provides access to multipath
{ information and enable the control of the usage. The necessity of this
{ work will be determined during the development of the MPTCP protocol
{ itself.
{ 
{ e. Guidance for implementers of single-ended multipath TCP
{ Single-ended multipath TCP does not require any on-the-wire protocol
{ changes to be standardised, and as such this document will focus on
{ practical implementation experience.
{ 
{ The initial goal is that the documents produced by the WG will be
either
{ experimental or informational. However, should the experiments prove
{ successful the WG reserves the right to subsequently ask the IESG to
{ advance one or more documents as Proposed Standard.
{ 
{ Goals and Milestones:
{ 
{ Aug 2010 - Submit Architectural Guidelines and Threat Analysis as an
{ informational RFC
{ Aug 2010 - Submit basic coupled congestion control as an experimental
{ RFC
{ Aug 2010 - Submit protocol specification for MPTCP extensions as an
{ experimental RFC
{ 
{ 
{ 
{ Regards,
{ Alan
{ 
{ 
{ _______________________________________________
{ multipathtcp mailing list
{ [email-masked]
{ https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Lars Eggert <[email-masked]>
Date: Tue, 1 Sep 2009 23:04:27 +0300

> Is it the IESG who would be asked about changing to stds track? Maybe

We shouldn't get too hung up on whether the initial milestones say  
Experimental or PS. Having them Experimental is fine, we can always  
change the status if it becomes clear that the work is more solid than  
expected and/or if sufficient experimentation will have happened  
before we last-call them.

Lars

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 2 Sep 2009 13:06:16 +0200

> modification of both endpoints.

In multi6/shim6 it was with great reluctance that we finally conceded  
that it was impossible to make multihoming work by only changing one  
end. Deployment when both ends need to support the modifications is  
many orders of magnitude harder, so why embrace a solution that  
requires changes on both ends as a starting point??

Note that changing both ends is not even necessarily a requirement for  
a multiaddressed multipath TCP: an existing multiaddressing system  
such as MIP or Shim6, or possibly even LISP could be reused.

Also, basing work on the current two-ended/multiaddressed MPTCP draft  
goes agains the modular / easily extensible and adaptable goals a  
paragraph earlier. I hate to say it, but in my opinion that solution  
isn't very elegant, and so far I haven't been able to convince anyone  
involved into making it more modular. Rather than cleanly separating  
the building blocks, it's a single monolithic design, where most of  
the work and complexity focus on something that I would consider an  
architectural afterthought: NAT traversal.

> practical implementation experience.

There are many RFCs created by the IETF that aren't protocols that  
require interoperation between different implementations. Example: RFC  
2581 (TCP congestion control), which is even standards track.

What I'm missing in all of this are the receive buffer exhaustion /  
retransmission issues.

---

From: Lars Eggert <[email-masked]>
Date: Wed, 2 Sep 2009 14:23:58 +0300

Hi,

On 2009-9-2, at 14:06, Iljitsch van Beijnum wrote:
> Deployment when both ends need to support the modifications is

because - in my personal opinion - the one-ended solution isn't  
applicable to the vast majority of nodes on the Internet, because you  
need PI space or be otherwise in control of your routing. If you're a  
datacenter, you can use it, but a lowly mobile device can't. And for  
datacenters, there are other solutions in the same space.

> 2581 (TCP congestion control), which is even standards track.

Not all standards are standards for interoperability reasons. TCP  
congestion control is a standard, because everybody needs to implement  
congestion control correctly and uniformly, so that the Internet as a  
shared resource remains useful for all. (It's a distributed algorithm.)

Lars

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 02 Sep 2009 13:54:12 +0200

Lars Eggert skrev:
> expected and/or if sufficient experimentation will have happened before
> we last-call them.
> 

Based on my experience going for PS directly may actually hinder the
compleetion and publication of specifications significantly. It might be
the produced spec and the experience with them is such that it could go
PS directly. But even in this case it might be both simpler and quicker
to first publish experimental and then later submit updated specs
combined with clear information that MPTCP works fine over the general
Internet.

Also if this is takes of and gets implemented and adopted and used then
it is more important to have a specification than what status is has.
IETF should of course strive to have the right status on its documents,
correctly reflecting maturity.

So as sponsoring AD I do recommend that the goals are initially set for
 Experimental status.

Cheers

Magnus Westerlund

IETF Transport Area Director
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 2 Sep 2009 14:45:47 +0200

> applicable to the vast majority of nodes on the Internet, because  
> you need PI space or be otherwise in control of your routing. If  
> you're a datacenter, you can use it, but a lowly mobile device can't.

Apart from the question of whether we really want a mobile device to  
run two different radios at the same time, where one is probably  
cheaper, faster and less power hungry than the other one, this is just  
an argument against the one-ended solution as it's currently proposed.  
With a mechanism to influence path selection some hops down, the one- 
ended solution gains additional capabilities (which the current two- 
ended solution can't use). But you could also take a one-ended  
solution and layer it on top of an existing mechanism that allows for  
multiaddressing, such as (like I mentioned before) MIP, Shim6 or LISP.

I'm not saying that any of the above is necessarily the best answer,  
but implicitly deciding that a monolithic two-ended multiaddress  
solution is the only possible solution in the charter seems strange to  
me.

---

From: Lars Eggert <[email-masked]>
Date: Wed, 2 Sep 2009 16:09:01 +0300

Hi,

On 2009-9-2, at 15:45, Iljitsch van Beijnum wrote:
> cheaper, faster and less power hungry than the other one

I never said that. 99% of the time, the benefit to mobile devices is  
in the ability to shift established connections between interfaces to  
conserve power more aggressively. 1% of the time it will be nice to  
pool capacity for better performance. The one-ended variant doesn't  
offer that flexibility benefit to a mobile receiver.

> , this is just
> ended solution can't use).

No mechanism to "influence path selection some hops down" currently  
exists, so this isn't a strong argument at present. Even if we had  
this mechanism now, it still wouldn't lead to benefits for mobile  
devices.

> But you could also take a one-ended

If I need to change L3 at both ends to run the one-ended L4 variant, I  
might as well instead change only L4 at both ends, no?

> solution is the only possible solution in the charter seems strange to

Well, the one-ended variant is your baby, and everyone loves their  
babies :-)

And I do believe that it's interesting to think through what could be  
done with sender-side changes only. It's just that I don't believe  
that the benefits that we have identified until now matter a whole lot  
for the majority of cases.

Lars

---

From: John Leslie <[email-masked]>
Date: Wed, 2 Sep 2009 09:49:12 -0400

>> modification of both endpoints.

   Red flag alert!

   Deployment which requires modifying both ends is orders of magnitude
more difficult than deployment which modifies only one end.

> ... basing work on the current two-ended/multiaddressed MPTCP draft  
> involved into making it more modular.

   For my money, Iljitsch is a prime driver behind this effort, and if
he's not satisfied that a solution is sufficiently modular/extensible,
we need to discuss it.

> Rather than cleanly separating the building blocks, it's a single
> monolithic design, where most of the work and complexity focus on
> something that I would consider an architectural afterthought:
> NAT traversal.

   I was given to understand that we were discussing an approach which
covered about 95% of NAT traversal cases "for free". Frankly, I think
we're ill-advised to worry about the remaining cases for an Experimental
protocol.

   NAT-traversal (beyond the simple cases of consumer NATs on one end)
should not be allowed to slow us down, IMHO.

>> practical implementation experience.

   I have heard no convincing arguments that our experimenting shouldn't
collect data on trying both two-ended and single-ended.

   I would much prefer to have the Charter call for an integrated effort.

--
John Leslie <[email-masked]>

---

From: John Leslie <[email-masked]>
Date: Wed, 2 Sep 2009 09:59:40 -0400

Lars Eggert <[email-masked]> wrote:
> need PI space or be otherwise in control of your routing.

   Is there some reason to think there aren't enough cases available
where this is already true _for_one_end_?

> If you're a datacenter, you can use it, but a lowly mobile device
> can't.

   More exactly, _today's_ lowly mobile devices can't. Betting on what
next year's "lowly mobile devices" may be able to do is unwise, IMHO.

> And for datacenters, there are other solutions in the same space.

   I'm not aware of any "other solutions in the _same_ space". (I am
quite aware of Akamai and the like; but these solutions don't survive
an actual outage of one path. Likewise, solutions depending on BGP to
"fix" routing have slow-convergence and occasional oscillation issues.)

   Iljitsch's single-ended multipath-TCP deserve experimental use; and
IMHO the best experiments would include both two-ended and single-ended
running on the same hosts in a coordinated fashion.

---

From: Lars Eggert <[email-masked]>
Date: Wed, 2 Sep 2009 17:09:16 +0300

Hi,

On 2009-9-2, at 16:59, John Leslie wrote:
> next year's "lowly mobile devices" may be able to do is unwise, IMHO.

next year's mobile devices won't use PI space and control their  
routing either.

> an actual outage of one path.

Sure they do - failover and load-balancing are the main functions of  
datacenter border routers.

Lars

---

From: "Rolf Winter" <[email-masked]>
Date: Wed, 2 Sep 2009 16:46:18 +0200

Hi,

First of all I like the charter. It's nice and focused. Straight to the point.

> such as MIP or Shim6, or possibly even LISP could be reused.

Calling MIP an existing multiaddressing system is maybe a little premature, same applies to LISP. I wouldn't call shim6 something that is something we should simply assume to be available. What I try to say is that what needs to be done is something that works without LISP, MIP or shim6. 
 
> architectural afterthought: NAT traversal.

Elegancy is in the eye of the beholder. I personally feel having both ends support it is the more elegant approach. I mean, whenever you design something you'd rather have it universally available. If that is the ultimate goal, we might as well focus on it from day one, otherwise you might end up in the situation where you have deployed, pardon the wording, a hack and have to deal with how to get from there to the ultimate goal. Basically you push the same problem out but you don't solve it. Especially, with OSes these days, if Windows and Linux maybe Symbian have it, I consider it universally deployed. That is only a patch day away, which is usually < 7 days.

If I understand the one-ended version correctly, which I might not, you need to spoof source addresses in order to not confuse the non-multipath end point, right? I mean, this might run into uRPF and all sorts of nastiness. We should not forget about that. Changing stuff, changes stuff, that applies to everything and we can discuss this until the cows come home, it just doesn't get us anywhere.

Just my take on this.

Best,

	Rolf

NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road, London W3 6BL | Registered in England 2832014

---

From: John Leslie <[email-masked]>
Date: Wed, 2 Sep 2009 10:18:15 -0400

Lars Eggert <[email-masked]> wrote:
> 99% of the time, the benefit to mobile devices is in the ability to
> shift established connections between interfaces to conserve power
> more aggressively.

   'Tis true that most "mobile" devices are battery-operated and
battery life between charges is pretty critical.

> 1% of the time it will be nice to pool capacity for better performance.

   I think you'll find that users would like this more than 1% of the
time...

> The one-ended variant doesn't offer that flexibility benefit to a
> mobile receiver.

   Think ahead, please. There's no reason a mobile device can't be
reprogrammed to run Iljitsch's single-ended multipath-TCP and pick up
the benefits talking to a legacy web-server. (There are quite a few
of those, BTW.)
   
> Well, the one-ended variant is your baby, and everyone loves their  

   I like Iljitsch's baby too!

> for the majority of cases.

   It sounds like Lars has his heart set on a Proposed Standard, and
wants to charter the quickest path to there.

   I care a lot more about designing good experiments: thus I want an
experimental platform that's easy to deploy.

   And I expect deployment of anything at all to the "average" consumer
system to take years. (Deployment to "servers" will vary from "very
quick" to "over ten years".)

   To me, Experimental status deploying both single-ended and two-ended
multipath-TCP is our best way forward.

---

From: Lars Eggert <[email-masked]>
Date: Wed, 2 Sep 2009 18:20:57 +0300

Hi,

On 2009-9-2, at 17:18, John Leslie wrote:
>   I think you'll find that users would like this more than 1% of the

right - these numbers are completely made up. Let's just say that  
during standby time, you always want to shift traffic around to  
conserve energy, and during active periods you *may* want to pool  
capacity (when you have bulk data to send/receive). Hence my argument  
that for mobile devices, we actually get the biggest benefits during  
standby.

> the benefits talking to a legacy web-server. (There are quite a few

It's not that easy. The one-ended scheme simply doesn't work if  
there's any sort of ingress filtering between the device and the DFZ,  
which is almost always the case on access networks that serve consumers.

> wants to charter the quickest path to there.

That is untrue. See the email I sent yesterday, in which I  
specifically said that we should go for Experimental. There are open  
questions that will need to be answered before this is ready for PS.

I do care a lot about getting the deployment incentives right. The one- 
ended variant is only of benefit to large sites, which have other  
mechanisms available to them to do similar things. I don't think  
that'll be enough.

(The one-ended scheme also currently exists only in idea form, whereas  
we at least have simulation results and prototype code for the two- 
ended version.)

> experimental platform that's easy to deploy.

We may have different goals. I specifically would not want this to  
result in an "experimental platform" - I want this to result in  
tangible benefits for the end systems, because otherwise it'll never  
see the light of day. If a side effect is that we can use it for  
experiments, sure, that's nice. (If all we want to do is an  
experimental platform, we don't need a working group for it.)

> quick" to "over ten years".)

There are exceptions. F-RTO, for example, was published as RFC4138 in  
2005 and I think it has been running on all major stacks very soon  
afterwards. Yes, Multipath TCP is a larger change, but it is similarly  
contained, which makes me moderately hopeful.

Lars

---

From: "SCHARF, Michael" <[email-masked]>
Date: Wed, 2 Sep 2009 18:29:00 +0200

> In addition, the following two items are excluded from this 
> charter but could form part of a re-charter once the initial 
> work described above has been completed.
> 
> d. An extended API describing how applications can exploit 
> additional features of multipath transport. 
> While MPTCP must be usable without any application changes, 
> this API should be an optional extension that provides access 
> to multipath information and enable the control of the usage. 
> The necessity of this work will be determined during the 
> development of the MPTCP protocol itself.

I have not followed all discussions so far, but I have the impression
that the interaction of MPTCP and applications is a crucial issue even
if the API ends up to be completely backward compatible. Unless there
are indeed no differences to "legacy" TCP at all, the differences should
be addressed somehow. For instance, there could be a document
(informational?) that explains the implications of multipath TCP to
application developers. This could probably also be part of activity on
architectural guidelines, but I don't find a corresponding statement in
the description so far.

IMHO it would be better if the charter addresses the implications of
multipath TCP on applications right from the beginning, even if a
detailed specification of an extended API is left open until it is
better understood.

Michael

---

From: <[email-masked]>
Date: Thu, 3 Sep 2009 09:38:34 +0100

That actually sounds to me like a good idea. Something that gives app
developers a starting point on what they can expect in terms of changed
performance, etc. Perhaps the sensible thing would be to aim to have
such a document as a later deliverable (say November 2010 even March
2011) as I imagine much of the content will be driven by experience of
using the protocol for real.

Toby

> [email-masked]] On Behalf Of SCHARF, Michael
> Sent: 02 September 2009 17:29
> To: Ford, Alan; Multipath TCP Mailing List

---

From: "Ford, Alan" <[email-masked]>
Date: Thu, 3 Sep 2009 13:30:42 +0100

Michael,

> detailed specification of an extended API is left open until it is

Thanks for the feedback. The intention of the backwards compatibility is
for applications to be able to use this with no modifications at all. An
additional API would be purely optional and allow preferences (e.g.
throughput or resilience?) to be expressed, and details (e.g. all IP
address pairs in use) to be extracted.

In the draft charter at the moment it says:

"Whilst this will require modification of the TCP stack at the
endpoints, all existing applications should be able to use
the new MPTCP stack without modification. In due course, an extended API
may be defined to allow applications to express particular preferences
over MPTCP operation, and gather information about MPTCP-specific
behaviour."

Maybe this could be made clearer with something like:

"Whilst this will require modification of the TCP stack at the
endpoints, all existing applications should be able to use the new MPTCP
stack without modification, and any impact on applications must be
considered and documented."

As for an API document in this charter cycle, I wouldn't see any
problems with doing it although at this stage we don't really know what
it would contain. We could start one speculatively, like the proposed
architecture document, and keep it updated as a living document as the
rest of the work progresses, with a target submission date of around Dec
2010. More opinions welcome :)

Regards,
Alan

---

From: John Leslie <[email-masked]>
Date: Thu, 3 Sep 2009 08:32:18 -0400

Rolf Winter <[email-masked]> wrote:
> ["Iljitsch van Beijnum" <[email-masked]> wrote:]
> 
> Calling MIP an existing multiaddressing system is maybe a little
> premature, same applies to LISP.

   Agreed. I believe Iljitsch merely offered those as an existence proof.

> I wouldn't call shim6 something that is something we should simply
> assume to be available.

   Absolutely. (Among other things, our work should have applicability
in IPv4.)

> What I try to say is that what needs to be done is something that works
> without LISP, MIP or shim6. 

   Agreed!

> Elegancy is in the eye of the beholder.

   Agreed. :^)

> I personally feel having both ends support it is the more elegant
> approach. I mean, whenever you design something you'd rather have it
> universally available. If that is the ultimate goal, we might as well
> focus on it from day one, otherwise you might end up in the situation
> where you have deployed, pardon the wording, a hack and have to deal
> with how to get from there to the ultimate goal.

   I, too, would prefer a world without hacks. But I don't see how to
avoid them, and still get deployment.

> Basically you push the same problem out but you don't solve it.
> Especially, with OSes these days, if Windows and Linux maybe Symbian
> have it, I consider it universally deployed.

   Your prejudice is showing. ;^) You should pay more attention to the
BSD Unices (including MacOSX).

> That is only a patch day away, which is usually < 7 days.

   Your optimism is refreshing... ;^)

   Actual deployment is closer to 7 years. :^(

> If I understand the one-ended version correctly, which I might not,
> you need to spoof source addresses in order to not confuse the
> non-multipath end point, right?

   You need to send IP packets with a source IP owned by the host but
not the interface on which you send them. I don't believe that's
considered "spoofing". (I routinely do it without admin privileges
on my FreeBSD systems for debugging purposes.)

> I mean, this might run into uRPF and all sorts of nastiness.

   Cisco's Unicast Reverse Path Forwarding _can_ be configured to
drop packets where a packet to the source IP would go out a different
interface that the packet came from, but IMHO is more often configured
in loose mode, where the IP merely must appear in the routing table.

   In any case, if you run afoul of RPF the packet is (consistently)
dropped, and that particular path will be observed not to work. (It
really doesn't make sense to call that "all kinds of nastiness".)

> We should not forget about that. Changing stuff, changes stuff,
> that applies to everything and we can discuss this until the cows
> come home, it just doesn't get us anywhere.

   We're talking about an Experimental protocol here (assuming the
two ADs can be believed). Experiments are _about_ changing things!

---

From: "Rolf Winter" <[email-masked]>
Date: Thu, 3 Sep 2009 15:39:53 +0200

John,

What I tried to say with uRPF, RFC 2827 etc. is that the one-ended version has been presented as the worry-free cousin of the two-ended version. That is not necessarily the case. As an example, if you are connected to two operators and use source addresses that you got from the other one, RFC 2827 suggests to not propagate that packet for good very reasons. Now, with the one-ended approach in order to get it deployed in such a scenario you need to make sure that the operator is allowing you to do that. How do you differentiate then between an MPTCP flow and a DDoS packet? Bottom line, there are issues with the one-ended version that are not negligible and I felt these need to be brought up as well. Trying to quantifying which version will have more deployment hassle with the installed base might not be fruitful as there will be a lot of guesswork involved. I feel we cannot simply assume that the one-ended version is a snap in terms of deployment.

FWIW, all I can contribute is one more opinion in this discussion and I would vote for focusing on the two-ended approach.

Best,

	Rolf

> [email-masked]] On Behalf Of John Leslie
> Sent: Donnerstag, 3. September 2009 14:32
> To: Rolf Winter

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 3 Sep 2009 16:13:25 +0200

> premature, same applies to LISP. I wouldn't call shim6 something  
> that is something we should simply assume to be available. What I  
> try to say is that what needs to be done is something that works  

Well, the currently proposed two-ended solution assumes that at least  
one end has two different addresses with reasonably different routing.  
The one-ended approach needs multihoming with provider independent  
addresses. Although of course there are systems that qualify for each,  
_most_ systems connected to the internet don't qualify for either. So  
while both current proposals have some use cases, I think it's  
premature to consider either of them the last word in multipath TCP.

As such, if we're successful, it's almost certain that we'll need to  
come up with mechanisms that allow some form of multipath TCP to exist  
on more typical hosts. At that point, a design that has been modular  
from the start will pay off in spades. Consider the situation where we  
create MPTCP 2.0 to allow hosts that don't fill the requirements for  
MPTCP 1.0 to do multipath. At that point, it would be really good if  
MPTCP 2.0 hosts can get multipath benefits when they talk to MPTCP 1.0  
hosts rather than having to first upgrade all hosts to MPTCP 1.0 and  
then redo the whole thing for MPTCP 2.0.

>> isn't very elegant

> Elegancy is in the eye of the beholder. I personally feel having  
> both ends support it is the more elegant approach.

In fact I agree with that. But the elegance of the approach is lost in  
the severely inelegant NAT traversal stuff such as the double sequence  
numbers.

> I mean, whenever you design something you'd rather have it  
> universally available. If that is the ultimate goal, we might as  
> well focus on it from day one, otherwise you might end up in the  
> situation where you have deployed, pardon the wording, a hack and  
> have to deal with how to get from there to the ultimate goal.

Until recently I had a TV that was completely digital and a DVD player  
that was completely digital. The cable between them was analog.

Upgrading code on an isolated host doesn't worry me, if the upgrade is  
worth the trouble, it will happen. However, once a protocol  
interaction between two systems is defined, it pretty much never goes  
away.

Back to the problem at hand, though.

There are some fundamental changes that need to be made to TCP to  
allow for any type of multipath: we need to solve the buffer issues  
and make RTT and loss measurements as well as (fast) retransmissions  
happen per path. Then, we need a multipath congestion control system  
that takes the ACKs and other information and returns window sizes for  
each path. Then, we need a way to make the packets actually flow over  
different paths, and a way to attribute ACKs to the correct path.

Both current draft pretty much do all of this as one monolithic  
system, which is an efficient implementation strategy, but not an  
efficient or effective design strategy. (I guess I just talked myself  
into making my draft more modular.)

> have it, I consider it universally deployed. That is only a patch  
> day away, which is usually < 7 days.

I guess no multipath for me when my Mac talks to my FreeBSD server...

> you need to spoof source addresses in order to not confuse the non- 
> multipath end point, right?

Then you assume two links with different addresses, I assume two links  
with the same address. I concede that this may be considered unusual  
by some.  :-)

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 03 Sep 2009 16:39:48 +0200

Rolf,

Rolf Winter escribió:
> John,
>
> What I tried to say with uRPF, RFC 2827 etc. is that the one-ended version has been presented as the worry-free cousin of the two-ended version. That is not necessarily the case. As an example, if you are connected to two operators and use source addresses that you got from the other one, RFC 2827 suggests to not propagate that packet for good very reasons. Now, with the one-ended approach in order to get it deployed in such a scenario you need to make sure that the operator is allowing you to do that.

Not really.
actually this is what you need for the two ended version.
the one ended version assumes PI addressing that can be used in both ISPs.
The two ended version assumes that if a node has two PA addresses from 
two different ISPs, the a packet with the source address of ISPA will be 
routed to ISPA or that if this is not the case ISPB will propagate it 
(i.e. what you describe above)

>  How do you differentiate then between an MPTCP flow and a DDoS packet? Bottom line, there are issues with the one-ended version that are not negligible and I felt these need to be brought up as well. 
this particular issue applies to the two ended version and NOT to the 
one ended version

Regards, marcelo

> Trying to quantifying which version will have more deployment hassle with the installed base might not be fruitful as there will be a lot of guesswork involved. I feel we cannot simply assume that the one-ended version is a snap in terms of deployment.
>
>>>> Note that changing both ends is not even necessarily a requirement
>>>>         
>> for
>>     
>>> What I try to say is that what needs to be done is something that
>>>       
>> works
>>     
>>>> Also, basing work on the current two-ended/multiaddressed MPTCP
>>>>         
>> draft
>>     
>>>> isn't very elegant, and so far I haven't been able to convince
>>>>         
>> anyone
>>

---

From: "Ford, Alan" <[email-masked]>
Date: Thu, 3 Sep 2009 16:35:25 +0100

Marcelo, all,

Out of these charter discussions, I'm keen to see where the community's preferences lie in terms of how and where people see these solutions being deployed and thus where effort should best be focused.

However, this needs clarification:

> Rolf Winter escribió:
> > John,
> >
> > What I tried to say with uRPF, RFC 2827 etc. is that the one-ended version
> has been presented as the worry-free cousin of the two-ended version. That is
> not necessarily the case. As an example, if you are connected to two
> operators and use source addresses that you got from the other one, RFC 2827
> suggests to not propagate that packet for good very reasons. Now, with the
> one-ended approach in order to get it deployed in such a scenario you need to
> make sure that the operator is allowing you to do that.
> 
> Not really.

That is absolutely not the case.

In the draft, we describe the multiple TCP subflows as having separate addressing. The source address on the packet is always valid for the link in use. If a packet is to be retransmitted on a different subflow it uses that subflow's addressing instead.

Regards,
Alan

---

From: "Rolf Winter" <[email-masked]>
Date: Thu, 3 Sep 2009 17:41:09 +0200

Marcelo,

> > What I tried to say with uRPF, RFC 2827 etc. is that the one-ended
> version has been presented as the worry-free cousin of the two-ended
> version. That is not necessarily the case. As an example, if you are
> connected to two operators and use source addresses that you got from
> the other one, RFC 2827 suggests to not propagate that packet for good
> very reasons. Now, with the one-ended approach in order to get it
> deployed in such a scenario you need to make sure that the operator is
> allowing you to do that.
> 
> Not really.

Assuming that doesn't make it likely for the majority of nodes. For the two ended version you don't need to make that assumption. You use what the operator gives you. What the operator gives you should better work or he's not doing a good job.

> (i.e. what you describe above)

I think we are comparing two different scenarios here. You say you are in a situation where you can connect to different ISPs using PI addresses. I am coming from, say residential customers, mobile terminals etc. that are connected with two interface to two different ISPs. These are very different in terms of what you'll be able to do. 

	Rolf

---

From: Lars Eggert <[email-masked]>
Date: Thu, 3 Sep 2009 18:54:05 +0300

On 2009-9-3, at 18:41, Rolf Winter wrote:
> I think we are comparing two different scenarios here. You say you  
> are in a situation where you can connect to different ISPs using PI  
> addresses. I am coming from, say residential customers, mobile  
> terminals etc. that are connected with two interface to two  
> different ISPs. These are very different in terms of what you'll be  
> able to do.

And the two-ended solution will - as far as I have thought it through  
- work in all cases where you can connect to two operators with (or  
without) PI space, too.

Lars

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 3 Sep 2009 17:56:24 +0200

On 3 Sep 2009, at 17:35 , Ford, Alan wrote:

> In the draft, we describe the multiple TCP subflows as having  
> separate addressing. The source address on the packet is always  
> valid for the link in use. If a packet is to be retransmitted on a  
> different subflow it uses that subflow's addressing instead.

So how do you avoid sending packets with a source address from ISP A  
to ISP B? In Shim6 we never solved this. Marcelo and Christian Huitema  
had some ideas in this area and "source address dependent routing" was  
discussed many a time, but in the end there's still nothing that makes  
the packets flow in accordance with their source addresses. And if the  
ISP then filters according to BCP 38 the packets are dropped. (This is  
Rolf's problem with the one-ended solution on a multiaddressed system.)

A good solution for this would be to extend the two-ended approach to  
allow for different subflows that use the same source and destination  
addresses, but different routing, for instance, through a different  
next hop address. So:

1: src B dst X nh A
2: src B dst X nh B

Presumably, subflow 1 won't work but subflow 2 will.

There are probably easier ways to solve this in the case where the  
exit ISP can be selected through the next hop, but this approach can  
easily be extended to interact in more advanced ways with routing,  
which I think we want to do for unrelated ways in the future anyway.

The disadvantage is that you lose the easy way to know which subflow  
an ACK is for by just looking at the addresses.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 03 Sep 2009 18:30:01 +0200

Rolf Winter escribió:
> Assuming that doesn't make it likely for the majority of nodes.

sure, i was just pointing out that the problem you stated about the 
compatibility problems with ingress filtering do not occur with the 
single ended version, but it does occur with the two ended version, agree?
>  For the two ended version you don't need to make that assumption. You use what the operator gives you. What the operator gives you should better work or he's not doing a good job.
>   

Not really
because when you connect to 2 isps, you cannot use the addresses that 
ISPA gave you to send packets through ISPB
> I think we are comparing two different scenarios here. 
agree
but in any case, the one ended version doesn't have compatibility 
problems with ingress filtering the two ended version does have those 
(in some scenarios, in particular the ones involving a host in a 
multihomed site as oposed to a multihomed node)

> You say you are in a situation where you can connect to different ISPs using PI addresses. I am coming from, say residential customers, mobile terminals etc. that are connected with two interface to two different ISPs. These are very different in terms of what you'll be able to do. 
>   

what about the residential user that is connecting its home network to 
two residential ISPs?
If it uses 2 PA blocks from each ISP and you sue the two ended version, 
then you have the issues with ingress filtering.

Regards, marcelo

---

From: "Ford, Alan" <[email-masked]>
Date: Thu, 3 Sep 2009 17:44:39 +0100

Iljitsch,

> to ISP B? 

By, err, not sending packets with a source address from ISP A to ISP B.
Send packets with a particular source address out the interface that has
that address associated with it.

> In Shim6 we never solved this. Marcelo and Christian Huitema

Remember we're just talking about doing this on the host itself.
Admittedly things get more complicated if there's a single interface but
multiple addresses. But in this case you still know what the default
router for a given address should be based on the host's subnet
configuration.

> Presumably, subflow 1 won't work but subflow 2 will.

Indeed, as it stands that wouldn't work. But if we made the "Address ID"
a subflow identifier, we could use different ports but the same
addresses as a separate subflow (and manage the next hop routing with
e.g. local policy routing) and then it would Just Work.

> an ACK is for by just looking at the addresses.

We'd still have multiple subflows, and so long as some bit of the
5-tuple is different (i.e. one of the ports) we will still know which
subflow is which.

Regards,
Alan

---

From: "Ford, Alan" <[email-masked]>
Date: Thu, 3 Sep 2009 17:51:01 +0100

Marcelo,

> really?

Yes.

> > In the draft, we describe the multiple TCP subflows as having
separate
> addressing. The source address on the packet is always valid for the
link in
> use. If a packet is to be retransmitted on a different subflow it uses
that
> subflow's addressing instead.
> 
> >
> >
> so, you are assuming that the two ended version of MPTCP only applies
to
> hosts with multiple interfaces and that each interface has a different
> IP address. this is overly restrcitive imho.

Well we assume the hosts are multi-addressed, not necessarily with
multiple interfaces. The draft title is "TCP Extensions for Multipath
Operation with Multiple Addresses" which I thought made it reasonably
clear!

> You should be able to apply it to a host located in a multihomed site
> and that the multihomed site is obtaining multiple PA blocks one from
> each provider it connects to, or is this out of the scope?

If the hosts were still multi-addressed that would work fine.

> If you intend to support this scenario, then the problem that i
describe
> above certainly occurs.

Well I guess that depends on the behaviour of the egress router. If
there were separate egress routers for each ISP, this wouldn't be a
problem. If it was the same box, so long as it manages routing
appropriate (and it probably would so as not to hit the upstream ISP's
RPF), it would still work since the router would send the right packets
to the right ISP.

Regards,
Alan

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 03 Sep 2009 18:25:53 +0200

really?
> In the draft, we describe the multiple TCP subflows as having separate addressing. The source address on the packet is always valid for the link in use. If a packet is to be retransmitted on a diff…
so, you are assuming that the two ended version of MPTCP only applies to 
hosts with multiple interfaces and that each interface has a different 
IP address. this is overly restrcitive imho.
You should be able to apply it to a host located in a multihomed site 
and that the multihomed site is obtaining multiple PA blocks one from 
each provider it connects to, or is this out of the scope?
If you intend to support this scenario, then the problem that i describe 
above certainly occurs.

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 03 Sep 2009 18:30:53 +0200

Lars Eggert escribió:
>> terminals etc. that are connected with two interface to two different 
>> ISPs. These are very different in terms of what you'll be able to do.
>
Not really, there is a piece missing i.e. how you deal with the ingress 
filtering compatibility when the different ISPs gave you a differnet PA 
block

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 03 Sep 2009 20:38:26 +0200

Ford, Alan escribió:
>>>> the one ended version assumes PI addressing that can be used in
>>>>         
> both ISPs.
>   
>>>> The two ended version assumes that if a node has two PA addresses
>>>>         
> from
>   
>>>> two different ISPs, the a packet with the source address of ISPA
>>>>         
> will be
>   
>>>> routed to ISPA or that if this is not the case ISPB will propagate
>>>>         
> it
>   

well, no
> multiple interfaces. 
good, that is what i was assuming

> The draft title is "TCP Extensions for Multipath

not really

please see http://www.shim6.org/draft-ietf-shim6-ingress-filtering-00.txt
> RPF), it would still work since the router would send the right packets

the problem is that this doesn't happen,
What you are describing is described in the draft that i pointed above 
and relies on using source address dependent routing in the end site, 
which is by far a not widely implemented technique.

But the bottom line, that i was pointing out that started all this 
discussion, is that the one ended approach does _not_ have this problem 
cause the host has a single PI address. It does happen in some 
configurations in the two ended approch. Do we agree?

Regards, marcelo

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 4 Sep 2009 09:21:38 +0100

Hi,

There's been quite a few emails going around with pros and cons for  
the two solutions. For reference, I'll try to summarize the points  
here and add my own thoughts.

Routing Issues:
- one ended:
	- targetted at servers with a single PI addresse in multihomed sites
	- changes the routing system inside the multihomed site to somehow  
route packets on different routes
	- does not work for multi-homed mobiles - If you have an unmodified  
web server and a multihomed mobile client, there is no way to inform  
the server to send data on one path or the other (remember, the  
mobile can send data over its available paths, but there is a SINGLE  
return path). Since most of the data flows one from the server to the  
client, resource pooling does not work.
- two ended
	- works for multi-homed mobiles or hosts
	- has issues with ingress filtering if PA addresses are used (this  
is the point Dino Farinacci raised during the bof). If we are in a  
multi-homed site, presumably we can change the routing system inside  
the data center to implement loose source routing (or whatever you  
call it) - which means the exit router should be chosen on the source  
address of the packet. In my book, this is equal to the changes one- 
ended needs in the routing system of data-centers, so what;s the  
argument about?

Fault tolerance (this is one of the main benefits of multipath TCP):
- one ended - has a single return path, and therefore if this path  
fails the connection fails
- two-ended - uses multiple return paths, and works as long as there  
is a pair of (a->b b->a) paths that work. Fault tolerance is superior  
to one-ended.

Sequence Numbers:
- one ended is forced to use a single one; two-ended uses multiple  
sequence numbers.

Two-ended's use of multiple sequence numbers is NOT a hack to get  
through NATs and firewalls. There are many more reasons to use a  
sequence space per subflow (please see design decisions doc!); in  
short, managing per subflow sequence numbers with gaps is much more  
difficult (see DCCP design for a working version :) ).
In fact, I am sure that using a single sequence space is not enough  
to be able to tell which path lost a packet and which not (I've been  
repeating this). When one resends a packet over a different path and  
gets an ack back, it can't know which path delivered the data. Unless  
there are convincing arguments this can be fixed or just ignored  
(fixing seems to me very difficult), I can't see a way forward for  
one-ended.

Finally, if we are only targeting major online providers (as in  
Google, etc.) the faster deployment model is to use multi-server http  
(see internet draft by Alan and Mark). All that's needed there is an  
update to http server / web browser, which can be easily rolled out  
by Google (chrome). Web browsers are updated very frequently anyway,  
so this could happen really quickly. We may not need one-ended after  
all.

Cheers,
Costin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 4 Sep 2009 10:54:41 +0200

> that address associated with it.

Note that this is a capability that hosts don't universally poses.

Normal operation is to inspect the routing table. If the routing table  
says the packet needs to go over interface A, and the source address  
is unspecified, the source address will be set to that of interface A.  
However, with TCP you can't change your source address in mid flight,  
so if after that there is a route change and now the default route, or  
a more specific route that matches the destination, goes over  
interface B, the source address for that existing session will still  
be the address belonging to interface A.

There is a "strong" host model and a "weak" host model. The strong  
host model requires packets to flow to/from addresses that match the  
interface's address, the weak host model doesn't care. (Note that  
we're talking about hosts, routers must always use the weak model.)

BSD-derived systems that I'm used to use the weak model, Windows the  
strong model as long as you don't turn on forwarding. Don't know about  
Linux. I don't know, however, if this means Windows will automatically  
send packets out the "correct" interface. I think it mostly just drops  
packets that don't conform to the strong model's idea of where packets  
should go to and come from.

Changing a stack to make packets flow through the interface that  
matches the source address is far from trivial, because now you  
basically have to maintain a routing table per interface. The only  
simple way to do this is to just use a default for every interface and  
not have a routing table with more specific routes, which might be  
appropriate on a smartphone but not on a general purpose host.

> router for a given address should be based on the host's subnet

Well, there is no automatic way to do this for IPv4 with a single  
router, so you'd have to accept multiple addresses from multiple DHCP  
servers. With IPv6 a single router can advertise multiple prefixes, so  
you would send all the packets to that router, and then the router  
would have to sort it out, which is a capability most routers don't  
have today, or it requires significant manual configuration.

> 5-tuple is different (i.e. one of the ports) we will still know which

Right, identifying subflows by the 4-tuple (I'm going to assume the  
protocol will always be 6) would allow for future refinements without  
having to change the other end, so please do the subflow  
identification that way.  :-)

Iljitsch

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 4 Sep 2009 10:32:29 +0100

Inline

> Costin Raiciu escribió:
>> Hi,
>>
>> There's been quite a few emails going around with pros and cons  
>> for the two solutions. For reference, I'll try to summarize the  
>> points here and add my own thoughts.
>>
>> Routing Issues:
>> - one ended:
>>     - changes the routing system inside the multihomed site to  
>> somehow route packets on different routes
>>     - does not work for multi-homed mobiles
> it works fine if the client is a mobile node, it will not provide  
> the benefits of the multihoming that these nodes have, but it will  
> certainly not break and will be able to communicate with them
>
> I think one characteristic you are not mentioning is that the one  
> ended version only requires the modification of one end and the two  
> ended one requires the modification of both ends, which is not a  
> minnor issue imho

Of one end + the routing inside the multi-homed site. Two-ended  
doesn't require changes to routing to get most benefits.

>
>> - If you have an unmodified web server and a multihomed mobile  
>> client, there is no way to inform the server to send data on one  
>> path or the other (remember, the mobile can send data over its  
>> available paths, but there is a SINGLE return path). Since most of  
>> the data flows one from the server to the client, resource pooling  
>> does not work.
>> - two ended
>>     - has issues with ingress filtering if PA addresses are used  
>> (this is the point Dino Farinacci raised during the bof). If we  
>> are in a multi-homed site, presumably we can change the routing  
>> system inside the data center to implement loose source routing  
>> (or whatever you call it) - which means the exit router should be  
>> chosen on the source address of the packet. In my book, this is  
>> equal to the changes one-ended needs in the routing system of data- 
>> centers, so what;s the argument about?
>
> the difference is that if you have a single IP address you will  
> never have problem with ingress filtering (so the one ended version  
> don't have any problems with ingress filtering) and the the two  
> ended version will have the problems if the source address routing  
> is not there.
If you change routing for one-ended, you can change it for two ended  
too to avoid this problem.

>> - two-ended - uses multiple return paths, and works as long as  
>> there is a pair of (a->b b->a) paths that work. Fault tolerance is  
>> superior to one-ended.
>>
>> Sequence Numbers:
>> short, managing per subflow sequence numbers with gaps is much  
>> more difficult (see DCCP design for a working version :) ).
>> In fact, I am sure that using a single sequence space is not  
>> enough to be able to tell which path lost a packet and which not  
>> (I've been repeating this). When one resends a packet over a  
>> different path and gets an ack back, it can't know which path  
>> delivered the data. Unless there are convincing arguments this can  
>> be fixed or just ignored (fixing seems to me very difficult), I  
>> can't see a way forward for one-ended.
>>
> i guess iljitsch can answer this one
>
>> Google, etc.) the faster deployment model is to use multi-server  
>> http (see internet draft by Alan and Mark). All that's needed  
>> there is an update to http server / web browser, which can be  
>> easily rolled out by Google (chrome). Web browsers are updated  
>> very frequently anyway, so this could happen really quickly. We  
>> may not need one-ended after all.
>>
> of course, there are many ways to skin a cat.
> One could argue that bit torrent is already doing what MPTCP is  
> trying to do for a significant amount of traffic, so you may don't  
> need any of this after all.
> But i guess this is not the point, the point is that both  
> approaches provide a set of benefits that are worthy.
>
> regards, marcelo
>
>> Cheers,
>> Costin
>>
>>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 04 Sep 2009 11:44:05 +0200

Costin Raiciu escribió:
>> it works fine if the client is a mobile node, it will not provide the 
>> benefits of the multihoming that these nodes have, but it will 
depend of the scenario you have in mind
I do agree that there are scenarios (like a multihomed ndoe with 
multiple interfaces) where the two ended don't need any change in the 
routing
But there are other scenarios, like the node with multiple PA addresses 
where the two ended version also needs to change the routing system (of 
both ends if they both have multiple addresses)

>>> (this is the point Dino Farinacci raised during the bof). If we are 
>>> in a multi-homed site, presumably we can change the routing system 
>>> inside the data center to implement loose source routing (or 
>>> whatever you call it) - which means the exit router should be chosen 
>>> on the source address of the packet. In my book, this is equal to 
>>> the changes one-ended needs in the routing system of data-centers, 
>>> so what;s the argument about?
>>
>> the difference is that if you have a single IP address you will never 
>> have problem with ingress filtering (so the one ended version don't 
>> have any problems with ingress filtering) and the the two ended 
>> version will have the problems if the source address routing is not 
>> there.
sure

regards, marcelo

>>> is a pair of (a->b b->a) paths that work. Fault tolerance is 
>>> superior to one-ended.
>>>
>>> Sequence Numbers:
>>> gets an ack back, it can't know which path delivered the data. 
>>> Unless there are convincing arguments this can be fixed or just 
>>> ignored (fixing seems to me very difficult), I can't see a way 
>>> forward for one-ended.
>>>
>>> http (see internet draft by Alan and Mark). All that's needed there 
>>> is an update to http server / web browser, which can be easily 
>>> rolled out by Google (chrome). Web browsers are updated very 
>>> frequently anyway, so this could happen really quickly. We may not 
>>> need one-ended after all.
>>>
>> But i guess this is not the point, the point is that both approaches 
>> provide a set of benefits that are worthy.
>>
>> regards, marcelo
>>
>>> Cheers,
>>> Costin
>>>
>>>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 04 Sep 2009 11:08:56 +0200

it works fine if the client is a mobile node, it will not provide the 
benefits of the multihoming that these nodes have, but it will certainly 
not break and will be able to communicate with them

I think one characteristic you are not mentioning is that the one ended 
version only requires the modification of one end and the two ended one 
requires the modification of both ends, which is not a minnor issue imho

> - If you have an unmodified web server and a multihomed mobile client, 
> there is no way to inform the server to send data on one path or the 
> other (remember, the mobile can send data over its available paths, 
> but there is a SINGLE return path). Since most of the data flows one 
> from the server to the client, resource pooling does not work.
> - two ended
> address of the packet. In my book, this is equal to the changes 
> one-ended needs in the routing system of data-centers, so what;s the 
> argument about?

the difference is that if you have a single IP address you will never 
have problem with ingress filtering (so the one ended version don't have 
any problems with ingress filtering) and the the two ended version will 
have the problems if the source address routing is not there.
> In fact, I am sure that using a single sequence space is not enough to 
> be able to tell which path lost a packet and which not (I've been 

> update to http server / web browser, which can be easily rolled out by 
> Google (chrome). Web browsers are updated very frequently anyway, so 
> this could happen really quickly. We may not need one-ended after all.
>
of course, there are many ways to skin a cat.
One could argue that bit torrent is already doing what MPTCP is trying 
to do for a significant amount of traffic, so you may don't need any of 
this after all.
But i guess this is not the point, the point is that both approaches 
provide a set of benefits that are worthy.

regards, marcelo

---

From: Mark Handley <[email-masked]>
Date: Fri, 4 Sep 2009 11:12:34 +0100

Folks,

It's clear from the ongoing discussion that we haven't yet documented
the options and issues for routing of outgoing packets with multi-path
TCP sufficiently.  I propose that I write a short document summarizing
the issues on this topic.  I expect this will become part of the
architecture document of the working group, but for now the important
thing is to gather the discussion together in one place so we can see
what do understand, what we don't understand,  and how host
requirements might change (if at all).

Can I request that some of the energy currently going into this email
exchange be put into reviewing the document when I've written it, so
you can tell me what I've got wrong!

 - Mark

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 04 Sep 2009 12:15:49 +0200

Hi Mark,

I think such a document exists, with proposed solutions as well.

please see

http://tools.ietf.org/html/draft-ietf-shim6-ingress-filtering-00

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 04 Sep 2009 12:35:26 +0200

Mark Handley escribió:
> Is there a more recent version?  That one expired a while back.  Is it dead now?
>
>   
the draft is dead

> I agree there's a lot of overlap, but it isn't precisely the same
> problem.
how come is not the same problem?
I mean, what is the difference that you see?

>   If the draft is dead, do you have any problem with me
> re-using text from it, where appropriate?
>
>   
sure, no problem

Regards, marcelo

>  - Mark
>
> 2009/9/4 marcelo bagnulo braun <[email-masked]>:
>   
>> Hi Mark,
>>

---

From: Mark Handley <[email-masked]>
Date: Fri, 4 Sep 2009 11:31:11 +0100

I agree there's a lot of overlap, but it isn't precisely the same
problem.  If the draft is dead, do you have any problem with me
re-using text from it, where appropriate?

 - Mark

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 4 Sep 2009 12:08:31 +0200

As it looks like we're going to be experimental for now anyway,  
wouldn't it make sense to do the following:

1. invite everyone to come up with MPTCP designs and publish them as  
experimental
2. evaluate the different MPTCPs, select one or select desirable  
components from multiple and recharter to work on that as standards  
track.

 From the discussion it looks like there are still many blank spots in  
our understanding of the issues and tradeoffs. In multi6 we had dozens  
of drafts before we finally settled on a direction for a standards  
track solution...

On 4 Sep 2009, at 11:08 , marcelo bagnulo braun wrote:

>> to be able to tell which path lost a packet and which not (I've  
>> been repeating this). When one resends a packet over a different  
>> path and gets an ack back, it can't know which path delivered the  
>> data. Unless there are convincing arguments this can be fixed or  
>> just ignored (fixing seems to me very difficult), I can't see a way  
>> forward for one-ended.

> i guess iljitsch can answer this one

I'm far from convinded this is enough of an issue to even care about,  
and certainly not enough of an issue to increase overhead in all  
packets. And:

http://www.ietf.org/mail-archive/web/multipathtcp/current/msg00129.html

---

From: Lars Eggert <[email-masked]>
Date: Fri, 4 Sep 2009 15:25:22 +0300

Hi,

> be the address belonging to interface A.

that's true in unipath TCP, but why would it be true in MPTCP? The  
socket is now associated with a set of IP addresses.

> appropriate on a smartphone but not on a general purpose host.

We're currently doing this in the testbed with one simple firewall  
(BSD) or policy-routing (Linux) rule per interface.

Also, don't forget that we have already been pointed at the MIF WG,  
who is busy working on layer-3 functions that we can leverage for  
additional benefit here.

Lars

---

From: Mark Handley <[email-masked]>
Date: Fri, 4 Sep 2009 13:39:49 +0100

> I mean, what is the difference that you see?

The focus of your draft was on providing network support for legacy
hosts at multihomed sites.

MP-TCP (both single-ended and two-ended) involves hosts at the
multihomed site that have been upgraded to support multipath.  Thus
your draft couldn't specify what the host could do, but a draft
concerning MP-TCP can (and probably should) talk about what the host
can do.

 - Mark

---

From: Scott Brim <[email-masked]>
Date: Fri, 4 Sep 2009 10:00:52 -0400

First, sorry for being so late.  I had a lot of internal work.
Catching up ...

Excerpts from Ford, Alan on Tue, Sep 01, 2009 03:27:46PM +0100:
> The Multipath TCP (MPTCP) working group develops mechanisms that add
> the capability of using multiple paths to a regular TCP session.
> This covers both protocol extensions needed to deploy MPTCP, as well
> as adaptations to congestion control to safely support multi-path
> resource sharing.
> 
> This working group will provide a modular architecture that is
> designed to be easily extensible and adaptable. In particular, the
> congestion control mechanism is separated from the mechanisms for
> identifying and using multiple paths.

The primary output should be the first paragraph (protocol and
congestion control work.  The second should be in support of the
first.  So how about: "In support of these goals, the working group
will also provide a modular architecture ..."?

> modification of both endpoints. One or both endpoints are expected
> to have multiple addresses which can be used to provide multiple

which can be used to potentially provide

> paths between the endpoints. Whilst this will require modification
> of the TCP stack at the endpoints, all existing applications should

to the TCP stack

> be able to use the new MPTCP stack without modification. In due
> course, an extended API may be defined to allow applications to
> express particular preferences over MPTCP operation, and gather
> information about MPTCP-specific behaviour. A key measure of the

Since it can't be tested very well without API enhancements, why not
make this one of the main deliverables?  

> success of this working group is that any protocol modifications are
> deployable in an unmodified Internet and work with existing
> unmodified applications. It is also essential that all new threats
> introduced by MPTCP are carefully analysed and addressed in the
> appropriate protocol documents.
> 
> a. A set of architectural guidelines for congestion dependent
> multipath transport protocols. This is a working document to track
> the technical documents, describing how the modular components fit
> together, and will describe the motivations and the general approach
> that should be followed to enable congestion dependent multipath
> transport. In addition, this document will provide a threat analysis
> of multipath transport.

I would call this the Framework.

> b. A basic coupled congestion control algorithm This will specify a

delete "basic", unnecessary (thanks Phil)
algorithm -> algorithm. 

> multipath equivalent of SACK/NewReno congestion control.  As
> experience is gathered from implementations of other congestion
> control algorithms, it is expected that the working group will
> charter additional work in these areas.

be rechartered to pursue additional work in these areas.

> c. Extensions to current TCP to support mutli-addressed mutipath
> TCP.  This covers all on-the-wire changes required to create a
> two-ended MPTCP solution using multiple addresses at one or both
> ends, and will include a basic security model. It will be designed
> in a modular fashion to allow alternative address management schemes
> to be explored in future.  This document must also include an
> analysis of threats introduced by these specific MPTCP extensions.
> 
> In addition, the following two items are excluded from this charter
> but could form part of a re-charter once the initial work described
> above has been completed.
> 
> additional features of multipath transport.  While MPTCP must be
> usable without any application changes, this API should be an
> optional extension that provides access to multipath information and
> enable the control of the usage. The necessity of this work will be
> determined during the development of the MPTCP protocol itself.

I would like to promote this.  How are you testing your implementation
right now?  You must have some kind of API.  So if you need an API
anyway, why not make it a work item?  It shouldn't block any of the
other drafts, and it is important to explore.

Thanks ... Scott

Excerpts from [email-masked] on Tue, Sep 01, 2009 08:20:10PM
+0100:
> It might be nice if the architecture doc also included "a summary of
> the results of the expts". (in general the ietf seems to struggle to
> capture the results of its 'experiments'). Here it could be seen as
> part of the reasoning for the final form of the protocols and
> perhaps the recommendation for what's done next (eg in terms of
> moving to Stds track or working on an extended api).

Yes but that doesn't have to be in this version of the WG.  It can be
added when the decision is made to move the drafts to standards track.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 04 Sep 2009 18:44:12 +0200

> hosts at multihomed sites.
right, the structure of the draft describes the problem for legacy 
hosts, indeed

> concerning MP-TCP can (and probably should) talk about what the host
the appendix A of the draft describes what changes in the host could 
help here.
I am not sure if you have in mind other changes that woudl do a better 
job...

Regards, marcelo

---

From: Scott Brim <[email-masked]>
Date: Fri, 4 Sep 2009 16:17:30 -0400

Mark: you can write text based on shim6-ingress-filtering for MPTCP
but I don't see this as a top priority issue.  We know that Multipath
TCP could be made to work in situations where multiple PA addresses
are propagated down to the endpoints, but that situation simply
doesn't exist today because of the problems already discussed with
routing, lack of communication between network and endpoint, etc.  I
suggest we nail down the protocol and algorithms, apply them where
they work today, and consider how to change the environment they run
in (MIF in the endpoint, routing, "exit discovery", tagging, etc.) as
a later goal.  The algorithms are independent of all that so they can
be worked on now.  Even with no changes in the network, MPTCP is
useful.

Scott

---

From: marcelo bagnulo braun <[email-masked]>
Date: Sat, 05 Sep 2009 00:02:19 +0200

while this is true, Mark does have a point that if mptcp implies 
modifications in the endpoint, and the ingress filtering/exit router 
selection can be solved by also modifying the host, it makes sense to do 
it all at once and install a single pack that solve both problems, and 
makes mptcp work in all situations, i guess.

Regards, marcelo

---

From: John Leslie <[email-masked]>
Date: Sat, 5 Sep 2009 17:19:27 -0400

> other drafts, and it is important to explore.

   It is possible to test on what amounts to a per-host opt-in; and
I have no problem with (Experimental status) protocol drafts that
assume only that.

   OTOH, I see major benefits from per-application (or even per-socket)
opt-in/opt-out. The details of such an API need not be stable between
Experimental and Proposed Standard.

   (I see a likely scenario in which we have separate drafts making
their way to Experimental, being merged when it comes time for
Proposed Standard. I am optimistic that we can have a "common" API
by the time they reach Experimental, but I see no serious problems
if we fail at that.)

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 10 Sep 2009 19:20:30 +0200

> "Whilst this will require modification of the TCP stack at 
> the endpoints, all existing applications should be able to 
> use the new MPTCP stack without modification, and any impact 
> on applications must be considered and documented."
> 
> As for an API document in this charter cycle, I wouldn't see 
> any problems with doing it although at this stage we don't 
> really know what it would contain. We could start one 
> speculatively, like the proposed architecture document, and 
> keep it updated as a living document as the rest of the work 
> progresses, with a target submission date of around Dec 2010. 
> More opinions welcome :)

Of course, the protocol spec has to mature before an API can be
specified, but I think that a starting point for this discussion is
needed.

I could try to write down some initial thoughts on the application API.
Is anyone else interested in this?

Michael

---

From: "Ford, Alan" <[email-masked]>
Date: Fri, 11 Sep 2009 08:20:22 +0100

Hi Michael, all,

> Is anyone else interested in this?

Unsurprisingly, I'm interested :)

When it became clear there was demand for such a document, my initial
ideas included:

- Impact on applications (if anything, even as simple as "higher
throughput" and "better resilience")
- Extended APIs
  - "Get" options - find out address pair lists, information about path
usage
  - "Set" options - set a preference on how paths are used ("e.g. I'm
low bandwidth, so only need set one up for resilience")
  - Unclear yet how these may interact with system-wide user settings
(possibly a discussion point itself)
- New advice, e.g. use connection identifiers in a tuple instead of the
traditional 5-tuple (for example - don't know yet if that makes sense)
- Issues TBD e.g. semantics of changing or non-changing addresses

How does this align with your thoughts? Happy to discuss and collaborate
off-list...

Regards,
Alan