Subject: Re: [multipathtcp] On Thursday's Multipath TCP BOF

From: "PAPADIMITRIOU Dimitri" <[email-masked]>
Date: Sun, 2 Aug 2009 10:19:29 +0200

Hi, 

> > I see value in working on a "linked" congestion control scheme (work
> > consisting in architecting and determining how efficient such scheme
> > would be compared to current practice/congestion control schemes). 
> 
>    Linked Congestion Control is fairly central to the Multipath TCP
> idea -- but is not intended as a research topic here. Rather, it is a
> way to avoid worrying about overlap of paths (some of which 
> are "obvious"
> overlaps which implementors will probably find anyway).
> 
>    By linking the congestion control, we avoid the need to hunt down
> sets of IP addresses which are "the same" in terms of forwarding path.

Concerning the second part of the last statement:

On one hand, the proposed work items include "An extended API describing
how applications can use multipath transport. The API should also
provide access to multipath information and enable the control of the
usage." 

On the other, the presentation of A.Ford states "It should appear as
regular TCP to applications" (and indeed the stack depicted during the
plenary show APP over single MPTCP running on top of multiple TCP)
assuming thus the relationship between APP and Transport layer does not
change

Clarification would be appreciated on how APP could have better control
over TCP with this proposal compared to the current situation.

This said, at minimum resolution mechanism would be needed to determine
the set of destination end-point/addresses "attached" to a given logical
endpoint (MPTCP) as the proposed mechanism adds a level of indirection. 

> > I have a couple of comments/concerns though:
> > 
> > - The BoF presentation considers that the TCP end-points controls
> > routing of the traffic along a certain path in the network 
> but routing
> > information does not reach the end-points.
> 
>    I'm not sure I understand. Clearly, the destination IP address
> "controls" the path the packet will take, and clearly the actual path
> taken isn't reported (except for "record route" option, which usually
> doesn't work anyway) to the destination host. But you seem to 
> be saying something beyond that...

To simplify, the forwarding table entry derived from the routing table
determines the outgoing interface the packet will take towards the
next-hop. The information that populates these routing tables does not
reach the TCP end-points (hosts). 
 
> > So for what it concerns routing of the traffic, per TCP connection,
> > there is no difference compared to the current situation 
> (the network
> > is not aware that two connections entering the network belong to the
> > same "set").
> 
>    Agreed. "The network" is expected simply to forward packets.
> 
> > This assumption is to be revisited because the "possible" 
> paths towards
> > (set of destinations) will be as provided by the routing system and
> > means for triggering and enforcing diversity inside network is not
> > achievable without additional mechanism (at the "networking layer"
> > level).
> 
>    We specifically avoid any expectation that the network 
> should enforce
> diversity. We expect the network to make "best effort" forwarding
> decisions: no more, no less.
> 
> > Note: The term "path" is from this perspective a bit confusing i.e.
> > inverse multiplexed TCP or multi component TCP would probably better
> > suite.
> 
>    I think we're stuck with the name. May I suggest we think of it as
> "TCP that can run over multiple simultaneous paths" rather than "TCP
> that guarantees the redundancy of multiple paths".

The difference is between the network location the packet is destined to
(destination) and how it reaches that destination (i.e., the actual path
as determined by a sequence of hops). If all nodes along the network
path forward the packet to the same next-hop (because the two prefixes
are spatially contiguous) the actual network paths won't be redundant.  
 
> > - The BoF presentation considers that controlling onto which
> > (sub-)connection the source puts traffic leads to offload 
> the network
> > from performing traffic engineering.
> 
>    I disagree. Network operators will still need to do their own
> traffic engineering. Multipath TCP is intended to give end-users some
> traffic-engineering ability separate from that of network operators.
> We claim that this can reduce the amount of network 
> traffic-engineering
> needed, but if that claim proves false, nothing really changes.

I just ask to make things clearer compared the way this topic was
presented. If the way I read your answer is correct, I think we agree
that ISP/operators still need to do their own TE (and there is no
question of offload with this proposed technique).

> > Thus, I do not see how this would lead to a situation where 
> the network
> > would be free from any traffic engineering ?
> 
>    I don't believe we've claimed any such thing.

The first presentation of the BoF refers to offloading network traffic
engineering operations. Quoting that presentation "We now understand
that multipath TCP, if done
appropriately, can go a long way towards solving network-wide traffic
engineering problems."

> > (I would even say the contrary, this would lead us back to the
> > hyper-aggregation problem).
> 
>    I don't believe we've said much about that. There's a 
> perhaps unstated
> assumption that network operators may impose artificial delay 
> or packet
> loss in order to influence Multipath-TCP allocation between paths; and
> we clearly believe that such "traffic-engineering" would be 
> less harmful than some things they're doing today.
> 
> > - On the proposed scheme itself, if the end-points assumes that
> > sub-connections (say between IP Address 1 and 2) can not be
> > re-established after detecting their failures, the degraded 
> state would
> > remain permanent since the connection between Address 1 and 
> 2 would not
> > be re-established.
> 
>    Point well taken. However, it is not clear to me that we 
> ever "detect
> failure" rather than treating what the routing system considers to be
> "failure" as, instead, "especially heavy congestion".

Is your concern whether the routing system can differentiate between
"topological" failures and heavy congestion events or not ?
 
>    The difference is, when we detect "especially heavy congestion" we
> can leave the sub-connection up until its routing problem is resolved
> (even if that takes several days).

The relationship to implicit and explicit congestion feedback from the
network is a point that may require further refinement. It is my reading
that only primal congestion control has been considered since so far in
the MPTCP context.

> > So, it may be that some of the decisions taken by the 
> end-points become
> > detrimental. At this stage, it is not clear how to prevent such
> > situation would happen.
> 
>    This does imply that we might need to consider different 
> timeouts for
> tearing down sub-connections than for non-multipath TCP connections.
> And that has not been discussed yet; but IMHO it could be left as an
> implementation detail...

I think more generally that reversion mechanisms may be considered when
dealing with failovers techniques.

Thanks,
-d.
 
> --
> John Leslie <[email-masked]>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Aug 2009 14:46:12 +0200

On 31 jul 2009, at 11:16, Joe Touch wrote:

> IMO, if an app decides to use multipath explicitly, it might as well
> have used SCTP.

SCTP in its current form does multiaddressing but not multiple paths  
at the same time or even smart path selection.

> My concern is that there are (and could/should be) applications that
> expect a TCP connection to go down when the local IP address is down.

We need a list of such applications.

> 	- as long as that socketpair is available*, the connection
> 	stays up; when not, the connection is torn down

> * = where "available" could mean just that both sides still think that
> the IP address is bound to one of their interfaces.

This is a narrower case than what we discussed after the session last  
week. Let's list some options:

1. The host has an address and the other host can reach it
2. The host has an address that is reachable for other hosts, but not  
the one involved in the MPTCP session
3. The host has an address, but it's unreachable
4. The host had and address, and can reasonably expect nobody else to  
have it now
5. The host had an address, and it may now be held by another host

I think we can all agree that we can do MPTCP in case 1. In case 5, we  
should obviously explicitly and completely remove the address from any  
MPTCP sessions. Possibly to the degree that if that's not possible, we  
kill the session.

But what about the cases in the middle?

In my opionion, 2 is also still completely ok, and 3 may or may not be  
depending on the address configuration details.

As for 4, if I have an interface identifier with the u/l bit set to  
universal, and probably also in the case of a CGA/HBA iid, I can  
reasonably expect that nobody else is going to use an address built  
from that iid on a subnet I was attached to, especially if this is a  
subnet I expect to be attached to again in the near future.

> Opt-in is the only viable solution if the semantics are changed.

I say: if we make this experimental for at least 2 years during which  
we tell application makers to use an opt out API after those 2 years  
we can reasonably turn on MPTCP by default.

---

From: Scott Brim <[email-masked]>
Date: Tue, 04 Aug 2009 09:59:08 -0400

Lars: The point of a flow label is to allow better flow identification 
for packet handling, both at intermediate points and endpoints. 
Labeling is up to the endpoints, and there can be multiple "flows" (as 
defined by flow label) based on a single "connection" (as defined by 
state shared by the endpoints).  So a "flow" could include all packets 
supported by a connection, or a subset of them.

IMHO endpoints MAY use the same flow label on different flows (aka 
subflows) taking different paths.  If those paths merged at some point 
the flows might be treated as one.  This is true with or without 
multipath transport.

Scott

Lars Eggert allegedly wrote on 07/31/2009 02:38 EDT:
> Hi,
>
> On 2009-7-31, at 0:46, Brian E Carpenter wrote:
>> Actually the flow label rules specified in RFC3697 cause a bit of
>> an issue with any form of multipath (including SCTP and SHIM6 as
>> well as multipath TCP). Do we want all threads of the same connection
>> to carry the same flow label even if the source or destination
>> addresses vary? I don't have an easy answer to that, and we ducked
>> it in 3697.
>
> RFC3697 actually says in the introduction:
>
> A flow is a sequence of packets sent from a particular source to a
> particular unicast, anycast, or multicast destination that the source
> desires to label as a flow. A flow could consist of all packets in a
> specific transport connection or a media stream. However, a flow is
> not necessarily 1:1 mapped to a transport connection.
>
> The first sentence seems to imply that the packets of different subflows
> may carry different flow labels.
>
> The second sentence is the issue. It is true for unipath/uniflow
> transport protocols, but not multipath transport protocols.
>
> The third sentence I don't understand :-)
>
> But at least to me, the decision to call this "flow label" instead of
> "connection label" implies that the intent was for this label to
> identify a sequence of packet that has topology affinity, rather than
> application connection affinity.
>
> Lars
>
>
> ------------------------------------------------------------------------
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 04 Aug 2009 07:02:08 -0700

Iljitsch van Beijnum wrote:
> SCTP in its current form does multiaddressing but not multiple paths at
> the same time or even smart path selection.
> 

Sure. Provide me a list of all applications that use TCP, and I can help
try to figure out which ones are affected ;-)

Start with:

	- apps that bind to a specific local address
	- apps that look at the address they bind to after they connect
	- apps that use addresses for validation (e.g., that refer to
	X.509 certificates, that use the DNS lookup of the address, or
	that use addresses for access control directly)

The overall problem is that this proposal is creating a new concept -
that a group of addresses are bound together - and assuming we can make
this meaningful to applications that use TCP.

My key concern here is that changing the semantics of the API is much
more difficult to diagnose and easy to blame on other things (e.g.,
insufficient permissions). See further below...

> MPTCP sessions. Possibly to the degree that if that's not possible, we

Well, AFAICT, that kills most of the use cases I've seen described.

> But what about the cases in the middle?

We'd need to understand whether those cases are:

a) variants of the last case - i.e., if you have an address that's not
reachable, that may be what YOU see, but it needs to be what the OTHER
END sees as well.

b) testable (or even knowable)

> reasonably expect that nobody else is going to use an address built from
> that iid on a subnet I was attached to, especially if this is a subnet I
> expect to be attached to again in the near future.
> 
> I say: if we make this experimental for at least 2 years during which we
> tell application makers to use an opt out API after those 2 years we can
> reasonably turn on MPTCP by default.

I'm not trying to prevent going to experimental with this, FWIW. I'm
trying to understand whether there's an objective criteria for knowing
whether we can go to standards track. There are other cases of
experimental protocols being deployed that nobody knew whether they were
OK or just that nobody knew what to look for or who to complain to, but
that has not been seen as sufficient evidence to make them standards
track (e.g., in Linux congestion control).

Joe

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 4 Aug 2009 19:25:50 +0100

Hi,

> -----Original Message-----
> From: [email-masked]
[mailto:[email-masked]] On
> Behalf Of Joe Touch
> Sent: 31 July 2009 10:16
> To: Iljitsch van Beijnum
> Cc: Multipath TCP Mailing List
> Subject: Re: [multipathtcp] On Thursday's Multipath TCP BOF

[snip]

> TCP is a service for a bytestream between an IP address/port pair, it
is
> not defined as binding to something else that happens to map to an
> IP/address port pair right now.
> 
> The implications if this would be:
> 
> 	- assume multipath TCP opens connections using a primary
> 	(initial) socket pair
> 
> I.e., you REALLY do not want the following case:
> 
> 
> 	A: adr 11, adr 12
> 	B: adr 23, adr 24
> 
> A adr 11 port 52394 connects to B adr 23 port 80
> A and B open a second TCP connection to use in the multipath set,
e.g.,
> from A 12 -> B 24
> A ifconfig's adr 11 down
> C ifconfig's adr 11 up
> Now what?

The problem here is if an application on A thinks A/11 is its address,
then signals this to B, then B tries to connect to C/11 to send it
something else. I can't think of any other cases where this would cause
a problem offhand.

For this to be a problem, this assumes that something is listening on
the given port on C/11 and the application has no handshake built in.
Sounds quite unlikely, but not infeasible. Do you know of any
applications that behave like that? (If nothing was listening, the
application would get a TCP RST or ICMP port unreachable, depending on
protocol, which would be the correct behaviour).

An alternative may be for B to maintain a mapping of adr11 to
alternative addresses on A until any sessions that had this as their
initial subflow have closed. This would effectively be moving knowledge
of paths outside of individual connections, however - something I've
been so far keen to avoid, since I fear that may be an jumbo-sized can
of worms!

> IMO, if A-B would have lost its connection if a single TCP, then it
> should have done so over multipath TCP. Note that this does allow A/11
> to have a heartbeat to B/23, OR to just ensure that both sides shut
the
> connection when they receive a RST on the original connection.

Ah, but what are "A" and "B" in this case? You use them as nodes in the
text above, and as such "A-B" would not lose its connection in single
TCP if A/12 -> B/24 was being used, only if A/11->B/23 was being used.
In MPTCP, both pairs are being used, and it seems semantically
consistent to say the connection would not be lost unless there are no
paths in use between A and B.

As Lars says in a later reply, this is intended to be Experimental. We
do not yet know if there are applications that will be harmed by this -
during the development of this work we will find this out!

Regards,
Alan

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 4 Aug 2009 19:25:48 +0100

Hi,

> Behalf Of Lars Eggert
> Sent: 31 July 2009 07:38
> To: Brian E Carpenter
> The first sentence seems to imply that the packets of different
> subflows may carry different flow labels.
> 

Yes, it's an odd one. I imagine the intention (Brian?) was to permit
multiple flows per transport connection. However, as written it would
also permit a single flow across multiple transport connections. (Of
course, we then get onto the definition of a "transport connection" - is
it a MPTCP connection or a TCP subflow?).

> application connection affinity.

I can agree with that reading, however now would probably be a good time
to sort out clarifying this :)

My initial feeling on this was that we should exploit the flow label as
an static-per-connection identifier for a MPTCP connection. However,
it's not clear if that would bring any benefits. For example, as the
source sets the flow label, this means that a flow can be identified by
a tuple of (source_addr, flow_label) - but the source address can be
different in MPTCP, so middleboxes that may have gained benefit from
such an identifier (e.g. load balancers) will still not be able to
determine anything from the flow label. Furthermore, such a rule just
goes against what I wrote above regarding the non-1:1 mapping, since it
would no longer permit multiple flows per transport connection.

Yes, definitely one for the clarifications list :)

Regards,
Alan

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 4 Aug 2009 19:25:46 +0100

Hi,

> Behalf Of PAPADIMITRIOU Dimitri
> Sent: 02 August 2009 09:19
> To: John Leslie

Typically, an application (that is either a legacy, MPTCP-unaware app,
or an app that has chosen to be MPTCP-agnostic) will be provided with a
multipath TCP connection automatically, but is

(If it has elected to exercise some control itself, such as binding to
an interface, it makes sense for this to take precedence, however).

An extended MPTCP API would allow an application to find out more about,
for example, how many paths are in use, and which interfaces they are
between. The kind of control it may wish to exercise could be to express
a preference such as "this is going to be a long-lived but very
low-bandwidth connection, so please only use MPTCP for backup purposes".
There could also (potentially) be prioritisation information, which
could be mapped to local policy - this is all TBD however!

This is only how we see it at the moment, however - further development
will indicate how best to approach this.

[snip]

> are spatially contiguous) the actual network paths won't be redundant.

True, however I don't think that goes against John's description, "TCP
that can run over multiple simultaneous paths" (note "can" :) ).

Regards,
Alan

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 05 Aug 2009 09:55:26 +1200

On 2009-08-05 06:25, Ford, Alan wrote:

...
>>     not necessarily 1:1 mapped to a transport connection.

...
> it a MPTCP connection or a TCP subflow?).

I would have to trawl the archive to be sure, but I suspect I wrote that
sentence and that the intention was to allow use cases where multiple
microflows (for example, all the TCP sessions between a particular
pair of application instances, or all the RTSP streams between them)
were given the same label. Or even, all the flows between two hosts
that happen to use the same diffserv DSCP are given the same label.

To put it another way: a flow between A and B is whatever set of packets
A chooses to label as a single flow, with no enforced mapping to
the transport layer.

However, we just didn't think about the case where A is sending to B
using more than one address pair.

>> application connection affinity.

Actually the intent of 3697 was, I think, to be neutral on that point.
It could be interpreted either way, or both ways simultaneously.
What was clear was the the flow label was not in any shape or form
an explicit routing tag like the MPLS label.

> would no longer permit multiple flows per transport connection.

Exactly. And remember that the flow label has no useful security
properties, and can be forged trivially.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 05 Aug 2009 10:02:22 +1200

On 2009-08-05 01:59, Scott Brim wrote:
> for packet handling, both at intermediate points and endpoints. Labeling
> is up to the endpoints, and there can be multiple "flows" (as defined by
> flow label) based on a single "connection" (as defined by state shared
> by the endpoints).  So a "flow" could include all packets supported by a
> connection, or a subset of them.
> 

The current rules would allow that if the sending host is using
different address pairs. But the merge point would need to know
that the various source addresses or destination addresses are
equivalent, and that raises a big security issue, because the
flow label is forgeable and would make a really lousy nonce.

   Brian

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 5 Aug 2009 10:41:13 +0200

> using more than one address pair.

It gets _really_ interesting when A and B are talking over different  
paths but using the same addresses. If routers do ECMP over the flow  
label then the flow label could be manipulated to make packets  
beloning to the same session flow over different paths.

---

From: Scott Brim <[email-masked]>
Date: Wed, 5 Aug 2009 20:25:49 -0400

PAPADIMITRIOU Dimitri allegedly wrote on 07/31/2009 03:45 EDT:
> routing of the traffic along a certain path in the network but routing

We need to sort out priorities and milestones.  Short term goals should 
just be about the endpoints acting on their own.  In that case you are 
right.  However, there is lots of useful speculation about future work 
in which endpoints and routers (or other infrastructure services) could 
work together so that endpoints could take advantage of more path diversity.

> inverse multiplexed TCP or multi component TCP would probably better

A path is a way of getting from one endpoint to another.  Subflows 
follow potentially different paths.  The collection of paths taken by 
all subflows might need a name.

> (sub-)connection the source puts traffic leads to offload the network
> from performing traffic engineering. Thus, I do not see how this would
> lead to a situation where the network would be free from any traffic
> engineering ? (I would even say the contrary, this would lead us back to
> the hyper-aggregation problem).

WG short term: if an endpoint has multiple interfaces, it can direct 
traffic to those on which it sees less congestion (taking policy, cost, 
etc. into account).  That will certainly avoid congestion as experienced 
by the endpoint, which will certainly do at least as well as current TCP 
at lowering congestion in the network.  In the future we can explore 
interactions between network TE and endpoint MPTCP.

Are you concerned about synchronization and oscillations?

> re-established after detecting their failures, the degraded state would
> remain permanent since the connection between Address 1 and 2 would not
> be re-established. So, it may be that some of the decisions taken by the
> end-points become detrimental. At this stage, it is not clear how to
> prevent such situation would happen.
>
> Note: also that some of the RTT numbers provided in the presentations
> are within recovery times achievable using fast re-routing schemes.

The endpoint (currently) doesn't know what is going on in routing. 
MPTCP will use potentially all paths that work.  If routing breaks MPTCP 
will stop using broken paths, and when FRR heals those paths MPTCP will 
use them again.  If they are on the same time scale, then on the 
broken/healed path you may get packet behavior similar to current TCP, 
except that retransmits can take place on other paths during the outage.

Scott

---

From: "PAPADIMITRIOU Dimitri" <[email-masked]>
Date: Fri, 7 Aug 2009 00:38:58 +0200

> all subflows might need a name.

A path may be defined as the "trajectory" packet follows to reach the
destination (i.e., as determined by a sequence of hops traversed by the
packets). As there is no mechanism for TCP to enforce the path followed
by the packets in the network I do not see at all where the "longer"
term objective could actually lead. Techniques that consist in driving
the path followed by packets from the source to the destination
(source-routing) in connectionless datagrams networks requires either
signaling or encoding the "route" into the packet header. 

This said, as far as I read you answer we seem to be in agreement ditto
"Short term goals should just be about the endpoints acting on their
own." we seem to disagree on the long term goals.

> traffic to those on which it sees less congestion (taking 
> policy, cost, etc. into account).  

How is policy and cost information fed back to the end-points ? 

> That will certainly avoid congestion as experienced 
> by the endpoint, which will certainly do at least as well as 
> current TCP at lowering congestion in the network.  
> In the future we can explore interactions between network TE
> and endpoint MPTCP.
>

I am concerned because there is no possibility after this BoF to
determine the actual problem statement: is there a need for a linked
congestion control scheme because "MP-TCP" is introduced that is
perceived as a suitable evolution of TCP or do we want to solve a
broader objective for which MP-TCP could be an element of solution ?

Indeed, the first presentation of the BoF refers to offloading network
traffic engineering operations. Quoting that presentation "We now
understand that multipath TCP, if done appropriately, can go a long way
towards solving network-wide traffic engineering problems." That
statement raises the following issues:

o) traffic engineering (= put traffic where unused capacity is) enforces
"path diversity" by adapting traffic routing to network conditions with
joint traffic and resource-oriented performance objectives, i.e., if TE
is not used anymore what would be the incentive to keep diversity in
traffic routing (except local topological diversity often activated upon
failure occurrence); what would be the reduction factor of diversity ? 

o) self-fishdness: network will keep at least certain performance
objectives and individual end-point/sources each their own, I fail to
see how this proposal could reconcile both sides i.e. how does it ensure
that the network actions are not antagonistic to the transport one.

o) stability conditions (quoting one of the seminal paper in this space
"Stability of Multi-Path Dual Congestion Control Algorithms": "...it
would be desirable to find decentralized, scalable conditions for the
global stability of a multi-path congestion control algorithm in the
presence of propagation delays, ..." [...] "... we found decentralized,
scalable conditions for local stability in the presence of propagation
delays for a particular choice of scheme") are partially verified; this
leads me to think that part of this work would be better addressed in
the research space (but this is another discussion).

So in brief, by considering one aspect of the problem (linked congestion
control), one can not a priori state that the global objective (of
offloading network-wide traffic engineering) can be reached because the
underlying assumptions are not (yet) verified and are sometimes opposed
to the conditions at the inception of this scheme. On the other hand,
there is no apparent consensus to scope the work (from the replies I
received on this list) to a TCP congestion control scheme.

> The endpoint (currently) doesn't know what is going on in routing. 

... and even if it received routing information, it can not enforce a
specific trajectory/ path to its packets (since the path followed by the
packet is under control of the network).

> will stop using broken paths, and when FRR heals those paths 
> MPTCP will  use them again.  
>
> If they are on the same time scale, then on the 

As stated already, when dealing with failovers techniques reversion
mechanisms shall be considered otherwise facing overall deterioriation
over time.

-d.

---

From: Fred Baker <[email-masked]>
Date: Thu, 6 Aug 2009 16:06:56 -0700

> this meaningful to applications that use TCP.

Disagree. We are saying that the application shouldn't be thinking  
about the network layer address. It should be thinking, perhaps, about  
the name of the remote system (if given an address, it should be able  
to do a reverse DNS lookup to verify it, and it should be able to  
connect to a remote application by name).

---

From: Joe Touch <[email-masked]>
Date: Thu, 06 Aug 2009 19:41:13 -0700

Fred Baker wrote:
> Disagree. We are saying that the application shouldn't be thinking about
> the network layer address. It should be thinking, perhaps, about the
> name of the remote system (if given an address, it should be able to do
> a reverse DNS lookup to verify it, and it should be able to connect to a
> remote application by name).

While I agree that apps should open connections to DNS names, the common
mechanism involves the app querying the DNS, selecting from among a set
of responses, and opening a connection to an IP address. This sequence
is necessary because TCP doesn't connect DNS names (and ports), it
connects IP addresses (and ports).

Ultimately, since the app opens a connection to an IP address, it is
that address to which it thinks it is talking.

Changing the TCP API from "socket pair" to "DNS name/port number" pair
(or better, DNSname/portname pair) would be interesting, but not
backward compatible with apps that do resolution as an explicit step.

Joe

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 7 Aug 2009 04:47:39 +0200

On Aug 7, 2009, at 4:41 AM, Joe Touch wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Fred Baker wrote:
>>
>> about
I think connecting to "DNS name/port number" is difficult. Assume that
4 addresses are registered for a name in the DNS, two of them  
belonging to
one host, two of them belonging to a different host. I think you can
not terminate one MPTCP connection on two different nodes at the same  
time...

Best regards
Michael
>
> Joe
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkp7lEkACgkQE5f5cImnZrt4CwCgunkKxYJt8Bquk6Cy3sOoxCPh
> xE4An3tO+kIqYGS9I+Ib34o19ewFqdZJ
> =nF88
> -----END PGP SIGNATURE-----

---

From: Joe Touch <[email-masked]>
Date: Thu, 06 Aug 2009 19:50:16 -0700

Michael Tüxen wrote:
>> 4 addresses are registered for a name in the DNS, two of them belonging to

Oh, I agree. But that's what I meant when I said that the Internet
doesn't have any notion of a "group of IP addresses" being a single machine.

Without that notion, you need the captain to go down with the ship,
because if the captain's gone you don't know you're on the same ship
anymore.

Joe

---

From: Aleksi Suhonen <[email-masked]>
Date: Fri, 07 Aug 2009 06:54:47 +0300

> not terminate one MPTCP connection on two different nodes at the same 

I've actually been thinking about this problem (wrt SCTP) earlier and 
one solution I came up with was DNS hints. The short presentation I gave 
is online:

http://research.nokia.com/files/host-app-based-multi-homing.pdf

Incidentally, I mention iSCSI as well on those slides. It is an existing 
multipathing protocol that runs on top of TCP. Since there has been lots 
of discussion about implementing MPTCP by using several independent TCP 
connections and an overlay (such as SSL/TLS) I suggest people look into 
how iSCSI does multipathing over TCP. I THINK the relevant RFCs are:

http://tools.ietf.org/html/3347
http://tools.ietf.org/html/3720
http://tools.ietf.org/html/5048

-- 
	Aleksi Suhonen
	Department of Communications Engineering
	Tampere University of Technology

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 7 Aug 2009 11:15:53 +0200

On 30 jul 2009, at 11:11, Fred Baker wrote:

> c) rather than even thinking about variations on the fast retransmit  
> heuristic or SACK as currently defined, I think we need some form of  
> explicit request for retransmission.

How would that be better than existing SACK and with normal fast  
retransmit executed per path?

Since the receiver doesn't know over which path a particular piece of  
data was sent, or when it was sent, it doesn't really know when to  
request a retransmission. For instance, if the piece of data was sent  
50 ms ago on a 10 ms path then obviously a retransmission would make  
sense but not so much 50 ms ago on a 250 ms path.

We do need to solve the problem where one path fails so the last few  
packets sent over it will never make it and now that path is waiting  
for an RTO or even user timeout, while all the packets on other paths  
proceed to fill up the receive buffer. If the receive buffer can't  
accommodate for the data sent on alternative paths for an RTO duration  
on the failed path this means the working paths will stall. We try to  
address that in the one-ended draft with "slow retransmit" but the  
problem exists for all multipath stream transports.

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 7 Aug 2009 13:06:29 +0200 (CEST)

On Fri, 7 Aug 2009, Michael Tüxen wrote:

> not terminate one MPTCP connection on two different nodes at the same time...

IMHO Knowing whether the addresses are on the same host should be rather 
simple - the clients sends a challenge during the 3WHS, and the 
server applies the server-unique transform to this challenge and sends the result 
to the client. Say, something like SHA1(server_secret_value + 
client_challenge). The client can compare the results from the connection 
attempts - if they are equal, we're talking to the same server or 
something that asserts that it can act as a single server.

Or am I missing something ?

thanks,
andrew

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 7 Aug 2009 14:19:57 +0200 (CEST)

> backward compatible with apps that do resolution as an explicit step.

If the MPTCP is backwards-compatible on the wire if one of the sides does 
not support it (from what I understood that's the case that for best use 
*both* sides need to be upgraded). This means that there is a possibility 
to provide such an API. And then the behaviour could be dependant on which 
API the applications used:

1) if it used conventional TCP API, then the captain sinks with the ship.

2) if it used the new API, then the captain from (1) is not in charge 
anyway - it'd be the hostname+some kind of identity token (e.g. what I 
described in my mail to Michael, being the challenge+response parts) - 
then the problem does not occur.

Taking a step back - "why do the apps do the resolution as an explicit step ?"

My personal answer to that would be "because noone gave any better 
alternative". Also because the default resolver is blocking, which creates 
the problems for the apps written in nonblocking I/O paradigm.

So I think the application developers would benefit of a new API that takes 
the burden off them of coding the resolver/address selection glue.

thanks,
andrew

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 7 Aug 2009 14:27:29 +0200

On Aug 7, 2009, at 1:06 PM, Andrew Yourtchenko wrote:

> IMHO Knowing whether the addresses are on the same host should be  
> rather simple - the clients sends a challenge during the 3WHS, and  
> the server applies the server-unique transform to this challenge and  
> sends the result to the client. Say, something like  
> SHA1(server_secret_value + client_challenge). The client can compare  
> the results from the connection attempts - if they are equal, we're  
> talking to the same server or something that asserts that it can act  
> as a single server.
>
One server responds with a SYN+ACK, one with a RST. What do you do?  
Fail the
MTCP connection?

Best regards
Michael

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 7 Aug 2009 14:54:21 +0200 (CEST)

On Fri, 7 Aug 2009, Michael Tüxen wrote:

>> simple - the clients sends a challenge during the 3WHS, and the server 
>> applies the server-unique transform to this challenge and sends the result 
>> attempts - if they are equal, we're talking to the same server or something 
>> that asserts that it can act as a single server.
>> 
> One server responds with a SYN+ACK, one with a RST. What do you do? Fail the
> MTCP connection?

No, continue working with the one that responded with SYN+ACK and not use 
the leg that went to the server responding with RST (maybe retry that 
sometime later).

I'd expect this to be transient condition rather than normal 
modus operandi - if the host foobar.example.org has two A records, in the 
conventional TCP I'd expect to be able to get identical service from 
either of the two - 'cos otherwise the application will fail 50% of the 
time.

So to that extent the FQDN _is_ the identity at the application layer and 
IP addresses are locators. Using the FQDN alone as an identity at 
transport layer is a bit unsafe, precisely because as Joe mentioned, 
there is no such concept (yet?) at transport layer - so it'd need to be 
combined with something else used as a verifier.

thanks,
andrew

---

From: "Ford, Alan" <[email-masked]>
Date: Fri, 7 Aug 2009 15:35:23 +0100

Hi,

> problem exists for all multipath stream transports.

This really depends on the implementation of the sending scheduler, but
I don't see it being a particular problem. Consider: two subflows (1 and
2) between nodes A and B. Subflow 2 dies leaving several packets
undelivered. When the RTO fires, it may well retransmit on subflow 1
instead. 

Nevertheless, the receive buffer may well be filling up fast due to
subflow 1. Optimisations could occur whereby, as the receiver will be
aware of a gap in the data from an early stage (later sequence space is
being acked on subflow 1), it may choose to preemptively retransmit on
subflow 1 anyway.

Regards,
Alan

---

From: "Ford, Alan" <[email-masked]>
Date: Fri, 7 Aug 2009 15:40:29 +0100

Hi,

> because if the captain's gone you don't know you're on the same ship

At the risk of going down the analogy rat-hole, what's going on in MPTCP
is you, the user, delegate to the captain to set up a connection from a
source to a destination. The captain picks his crew (additional
subflows), but may fall overboard later. That doesn't stop the ship
carrying on to its destination, but you don't care - the connection is
up - you trusted the captain to set up whatever was necessary to provide
this connection.

:-)

Regards,
Alan

---

From: Randall Stewart <[email-masked]>
Date: Fri, 7 Aug 2009 10:55:55 -0400

>> the same time or even smart path selection.

Thats not necessarily true. Janardhan Iyengar published quite a bit
of work on Conncurrent Multipath Transport (CMT)... which uses all paths
i.e. your multi-path TCP. For some reason I have not seen any of your  
work
mention or look at what he did in that body of work... It is currently  
implemented in FreeBSD and the only
thing lacking is a internet draft/RFC... If you are curious you can
take a look at Jana's work. And if I remember right Jana mentioned he  
might want
to document that work in an RFC form ;-)

R
------------------------------
Randall Stewart
803-317-4952 (cell)
803-345-0391(direct)

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Aug 2009 08:06:20 -0700

> up - you trusted the captain to set up whatever was necessary to provide

Yeah, but if I (the app) signed up for a trip with the captain (which is
how the TCP API works, binding to addresses), then I do care that the
captain is gone.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Aug 2009 08:16:17 -0700

Andrew Yourtchenko wrote:
>> applies the server-unique transform to this challenge and sends the
>> result to the client. Say, something like SHA1(server_secret_value +
>> client_challenge). The client can compare the results from the
>> connection attempts - if they are equal, we're talking to the same
>> server or something that asserts that it can act as a single server.
> 

First, you need to do SHA1 on every packet, not just the handshakes.

Second, what happens when an address moves after a connection happens?
E.g. let's say I'm an app that opens a connection to address A, which
then goes multipath. Then I open another connection to address A.

Does the app know that A isn't A anymore? Or that A from one connection
isn't A from another? e.g., now all of a sudden HTTP may communicate
with different parties depending on whether it uses persistent
connections or separate connections - and since those connections are
shared, opened, and closed dynamically, all of a sudden you've
introduced nondeterminism in WHO I'm talking to.

I'm saying that apps that use TCP ask to talk to an IP address, and
keeping a connection up when an address disappears will break apps that
don't expect that to happen. This is worse than just impact on things
like BGP, e.g.

That's why I think the MP goes down with the ship. It doesn't matter if
the ship goes down with a RST, a FIN, or disconnection.

Joe

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 7 Aug 2009 17:32:25 +0200 (CEST)

> First, you need to do SHA1 on every packet, not just the handshakes.

No. when we're talking about subflows (which, each by themselves, look and 
smell like normal TCP flows - with 3whs), once Bob and Alice establish the 
subflow, they can assume (as much as the TCP allows) they are talking 
between themselves simply because of the seq#/ack#. If Carol gets Alice's 
address later, and Bob tries to send a packet, then the subflow will 
reset. Note, that here I used hashing just to make the spoofing a bit more 
difficult. They could as well send an UUID or any other sufficiently 
distinct ID within the SYN-ACK segments.

> introduced nondeterminism in WHO I'm talking to.

Yes. That's why an API based on names would be needed to avoid sinking the 
captain :-)

> the ship goes down with a RST, a FIN, or disconnection.

In the address-based API, that is a fair proposal - because we 
don't know at which time the APP decides to do getpeername() (in which 
we'd need to return the captain name). That's for the conservative 
approach.

And those who want to promise they don't want to use addresses, would use 
the name-based API instead.

thanks,
andrew

> iEYEARECAAYFAkp8RUEACgkQE5f5cImnZrsKKgCgmfW0dDQM7ni+omoN0hEVR66/
> o1YAoIUaU0A76gVz9d4NQMHBhVOdh5QE
> =+EBF

---

From: Lars Eggert <[email-masked]>
Date: Sat, 8 Aug 2009 10:57:03 +0300

Hi,

On 2009-8-7, at 17:55, Randall Stewart wrote:
> i.e. your multi-path TCP. For some reason I have not seen any of  
> your work
> implemented in FreeBSD and the only thing lacking is a internet  
> draft/RFC...

CMT was brought up several times in the discussion on- and off-list.  
And I at least was under the impression that the reason it wasn't in  
an ID yet is that the coupled congestion control bit wasn't worked out  
yet for CMT?

Lars

---

From: Lars Eggert <[email-masked]>
Date: Sat, 8 Aug 2009 11:09:21 +0300

> how the TCP API works, binding to addresses), then I do care that the

It's clear that some apps signed up for a trip in general (A), while  
others signed up for a trip with a specific captain (B).

We have two ways forward:

(1) MPTCP off by default, enabled only for apps in class A
(2) MPTCP on by default, disabled only for apps in class B

I believe Joe is saying that because we cannot enumerate or otherwise  
identify all the apps in class B, approach (1) is the one we need to  
go with. And we actually know that a non-trivial number of apps are in  
class A (in my opinion, everything that can run on top of TLS, because  
those apps bind to a TLS association endpoint and not a port). And  
once MPTCP is out there, I expect app designer to quickly move to it  
for protocols that don't work over it from the beginning.

I agree that (2) would lead to more impact/larger benefits, because  
more traffic would be multipath-enabled. But at the moment, that's not  
a recommendation we can make. Once people have run experiments with  
(2) in non-trivial deployments, we may well find that the set of apps  
in class B is very small. At that point, we can still move to approach  
(2).

Lars

---

From: Michael Tüxen <[email-masked]>
Date: Sat, 8 Aug 2009 10:25:56 +0200

On Aug 8, 2009, at 9:57 AM, Lars Eggert wrote:

> an ID yet is that the coupled congestion control bit wasn't worked  
> out yet for CMT?
That is right. But one can use the CC suggested by Mark. What CMT
also provides are the procedures describing how to live with one
sequence number space for all paths.
Unlike MPTCP, which has a separate sequence number
space for all paths...

Best regards
Michael
>
> Lars_______________________________________________

---

From: Lars Eggert <[email-masked]>
Date: Sat, 8 Aug 2009 12:41:46 +0300

Hi,

On 2009-8-8, at 11:25, Michael Tüxen wrote:
> That is right. But one can use the CC suggested by Mark.

oh, absolutely. I'm halfway expecting Jana to be hacking it into CMT  
already :-)

Lars

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 8 Aug 2009 11:45:35 +0200

> an ID yet is that the coupled congestion control bit wasn't worked  

There's a bunch of papers but no overarching specification, not sure  
why this work hasn't been brought to the IETF, but immaturity would be  
one explanation.

---

From: Michael Tüxen <[email-masked]>
Date: Sat, 8 Aug 2009 11:48:51 +0200

On Aug 8, 2009, at 11:45 AM, Iljitsch van Beijnum wrote:

>> CMT was brought up several times in the discussion on- and off- 
>> list. And I at least was under the impression that the reason it  
>> wasn't in an ID yet is that the coupled congestion control bit  
>> wasn't worked out yet for CMT?
>
> why this work hasn't been brought to the IETF, but immaturity would  
> be one explanation.
.. but it isn't. It is because the missing point was shared bottleneck  
detection
where we had no solution for. Without it, it would not have made it  
through
the standardisation process.

But using Marks CC algorithm this problem is gone...

Best regards
Michael

---

From: Michael Tüxen <[email-masked]>
Date: Sat, 8 Aug 2009 12:32:35 +0200

> oh, absolutely. I'm halfway expecting Jana to be hacking it into CMT  
I don't know. If he doesn't do it I do it after getting a couple of
things clarified with Marks proposal...

---

From: John Leslie <[email-masked]>
Date: Sat, 8 Aug 2009 10:18:14 -0400

> (2) MPTCP on by default, disabled only for apps in class B

   For my BGP machines, I want MPTCP off, period.

   For either of Lars' approaches we need to define some mechanism
to say which applications opt in or opt out. Given the differing
Operating Systems in use on the Internet, this may not be trivial.

> go with.

   Joe is taking a somewhat extreme "conservative" position, which IMHO
makes sense for something we unleash on the 'Net without warning. I'm
not sure it makes sense for something clearly labeled "Experimental,"
but we are hoping to unleash this eventually, so it would be good to
see a way forward to that.

> And we actually know that a non-trivial number of apps are in class
> A (in my opinion, everything that can run on top of TLS, because  
> those apps bind to a TLS association endpoint and not a port).

   I'm not clear that we have any handle on what apps might be making
assumptions that MPTCP would break, beyond the BGP case.

   BGP is, frankly, a hack put together in ancient history (1989) to
propagate routing information along the same "fabric" that does the
forwarding. To "simplify" sharing the routing information, a "reliable"
transmission protocol was specified: meaning that we want retransmission
of garbled or dropped packets. We depend upon this "connection" being
able to send and receive "keep-alive" messages to indicate that the
link is useful for the layer-2 forwarding we will do.

   We could just as well have done everything at layer-2 with our own
retransmissions, but TCP seemed easier at the time. In retrospect,
it was probably the wrong choice, because it constrains BGP timeouts
according to TCP parameters which will inevitably be optimized for
very different needs.

   (MPTCP for BGP sessions would break the assumption that layer-2
forwarding from source-IP to destination-IP and vice-versa is proven
by the occasional keep-alive being received.)

   I have thought of another case where MPTCP might confuse an app:
"POP-before-SMTP" is a commonly-used hack to determine that an email
sending client should be trusted. The remote IP address of a POP
client retrieving email is recorded, and the SMTP server is told to
trust that IP address for a few minutes.

   In this case, a mobile user using MPTCP could initially connect
to the POP server from one IP address and complete the retrieval
after the initial IP address was no longer useful: the SMTP server
would be instructed to trust the no-longer-usable IP address instead
of the new IP address that would in fact be used for the SMTP session
that followed. (The same thing could happen with a multihomed client
whose preferred link went down due to a fibre cut.)

   But it's not clear to me that taking down the MPTCP connection
would do anything to help here: indeed, I think users would be happier
to receive their email and have to "check mail" twice in order to send.

> And once MPTCP is out there, I expect app designer to quickly move
> to it for protocols that don't work over it from the beginning.

   There may be legacy apps that nobody wants to re-compile: thus it
could make sense for a host-level option to determine whether MPTCP
would be used for "legacy" API Open calls. Always in the past, the
"dinosaur" apps have persisted on a reasonably limited number of
hosts: often "dinosaurs" themselves. ;^)

> more traffic would be multipath-enabled.

   The multi-homed and mobility benefits _depend_upon_ the majority
of apps making use of MPTCP.

> But at the moment, that's not a recommendation we can make.

   I believe we _could_ run useful experiments with Lars' option-2,
with a host-level opt-in. But for the eventual Standards-track MPTCP,
we should IMHO design some kind of application-level opt-in/opt-out.
(This need not necessarily be part of the Open() call.)

> Once people have run experiments with (2) in non-trivial deployments,
> we may well find that the set of apps in class B is very small. At
> that point, we can still move to approach (2).

   I think it would be best for the choice of (1) or (2) to be a
host-level configuration. That way, the experiments can collect actual
data on cases of observed problems.

---

From: Fred Baker <[email-masked]>
Date: Fri, 7 Aug 2009 15:08:26 -0700

On Aug 6, 2009, at 7:41 PM, Joe Touch wrote:

> While I agree that apps should open connections to DNS names, the  
> common mechanism involves the app querying the DNS, selecting from  
> among a set of responses, and opening a connection to an IP address.  
> This sequence is necessary because TCP doesn't connect DNS names  
> (and ports), it connects IP addresses (and ports).

Except that if we had proper APIs, we would give the API the name  
(which is what the application wants to connect to) and the API would  
give TCP/etc the address to use - and in this case, the set of  
addresses. There is no reason for the application to ever know the  
address, because the only thing that has any concept of the address is  
two layers below it.

---

From: Fred Baker <[email-masked]>
Date: Fri, 7 Aug 2009 15:11:56 -0700

> because if the captain's gone you don't know you're on the same ship  

I think you're getting the cart somewhat before the horse. "Multipath  
TCP" (which likely isn't TCP at all, but is some form of multipath  
transport) absolutely has to understand that a stated set of addresses  
instantiates a remote machine, and by the way this machine has a set  
of addresses, and by the way addresses that are not in the relevant  
set are probably a security consideration. If the desire is that we  
build such a thing, the requirement is that the internet-as-we-use-it  
develop a concept of such an association of addresses.

Don't lop off the idea because it doesn't exist. Look at what it takes  
to create it.

---

From: Fred Baker <[email-masked]>
Date: Fri, 7 Aug 2009 15:18:12 -0700

On Aug 7, 2009, at 2:15 AM, Iljitsch van Beijnum wrote:

>> c) rather than even thinking about variations on the fast  
>> retransmit heuristic or SACK as currently defined, I think we need  
>> some form of explicit request for retransmission.
>
> Since the receiver doesn't know over which path a particular piece  
> of data was sent, or when it was sent, it doesn't really know when  
> to request a retransmission.

If you use the example approach I suggested, it absolutely does. If I  
know that my peer is sending me cwnd bytes or messages per RTT and I  
have not received piece SSN-k*cwnd, the larger k is the smaller the  
probability that I am ever going to receive it. With that algorithm,  
the question is what value of k is the right crossover point.

> For instance, if the piece of data was sent 50 ms ago on a 10 ms  
> path then obviously a retransmission would make sense but not so  
> much 50 ms ago on a 250 ms path.

well, yes. But more to the point, if I have several paths with 25, 50,  
and 500 ms RTTs, we're observing that we may only want to use the 25  
and 50 ms paths (pruning the long delay path) and might specifically  
want to retransmit a segment that got no ack in 100 ms.

Why not use SACK and fast retransmit? think about the reordered- 
arrival behavior of a a set of paths with 25 and 50 ms RTT (never mind  
the 500 one), and how SACK and fast retransmit respond to them. A NACK  
is more stable.

> for an RTO or even user timeout, while all the packets on other  
> paths proceed to fill up the receive buffer. If the receive buffer  
> can't accommodate for the data sent on alternative paths for an RTO  
> duration on the failed path this means the working paths will stall.  
> We try to address that in the one-ended draft with "slow retransmit"  
> but the problem exists for all multipath stream transports.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 8 Aug 2009 17:50:58 +0200

On Aug 8, 2009, at 00:08, Fred Baker wrote:

> if we had proper APIs

Sorry for pointing out the obvious, but it may be worth pointing back  
to Stuart Cheshire's plenary talk:

	http://www.ietf.org/proceedings/72/slides/plenaryw-6.pdf

page 34 onwards.

(Stuart's point here being that the IPv6 migration situation has  
already made the 1980's programming interfaces obsolete, exactly for  
the reason that the socket programming interface exposes IP addresses  
to the program.  It could not really have been designed in a different  
way in 1982/1983 -- DNS was only just starting to exist!)

Gruesse, Carsten

---

From: Joe Touch <[email-masked]>
Date: Sat, 08 Aug 2009 15:18:12 -0700

Fred Baker wrote:
...
> instantiates a remote machine, and by the way this machine has a set of
> addresses, and by the way addresses that are not in the relevant set are
> probably a security consideration. If the desire is that we build such a
> thing, the requirement is that the internet-as-we-use-it develop a
> concept of such an association of addresses.
> 

If we're creating it, then please let's use a different protocol number.

The only reason my concerns are relevant is how it would change the TCP
API, and whether it can be backward compatible. IMO, it simply can't. We
can *assume* it is, and find places where that assumption breaks, but
that's all we can try to do.

If this is about creating a true fix, then let's do that (or didn't we
already in HIP, e.g.?)

Joe

---

From: [email-masked]
Date: 10 Aug 2009 13:09:06 +0100

Hi,

The main reason for not using CMT is that it works with SCTP and SCTP 
doesn't work with NATs.

The single sequence space is actually more of a "legacy" thing in CMT, 
borrowed from SCTP - according to Jana's comments in a discussion we had in 
Stockholm.

It turns out one gets many benefits from multiple sequence spaces. In 
addition, the single sequence space does NOT always ensure the sender can 
tell which path delivered a packet. For details why we chose two sequence 
spaces instead of one see the deisgn decisions document: 
http://www.cs.ucl.ac.uk/staff/C.Raiciu/files/mtcp-design.pdf

Cheers,
Costin

---

From: Michael Tüxen <[email-masked]>
Date: Mon, 10 Aug 2009 14:36:12 +0200

On Aug 10, 2009, at 2:09 PM, [email-masked] wrote:

> Hi,
>
> The main reason for not using CMT is that it works with SCTP and  
> SCTP doesn't work with NATs.
... you could use SCTP/UDP/IP, which is supported by legacy NATs. The  
problem
not solved by CMT is the shared bottleneck detection. But is not not a  
problem
when using Marks CC...
>
> The single sequence space is actually more of a "legacy" thing in  
> CMT, borrowed from SCTP - according to Jana's comments in a  
> discussion we had in Stockholm.
>
> It turns out one gets many benefits from multiple sequence spaces.  
> In addition, the single sequence space does NOT always ensure the  
> sender can tell which path delivered a packet. For details why we  
> chose two sequence spaces instead of one see the deisgn decisions  
> document: http://www.cs.ucl.ac.uk/staff/C.Raiciu/files/mtcp-design
Maybe I was not clear. I think having separate sequence number spaces  
helps
in multipath transport. I just wanted to point out that CMT in SCTP has
to deal with the problem of having only one sequence number space, a  
problem
MPTCP does not have.
> .pdf

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 11 Aug 2009 13:19:11 +0200

> It turns out one gets many benefits from multiple sequence spaces.

I disagree. It's a terrible hack.

> sender can tell which path delivered a packet.

When a packet is retransmitted over another path then an ACK doesn't  
tell you which instance made it. Not a big deal IMO but if it is, a  
two-ended MPTCP can easily indicate that a packet is a  
retransmission / ACK of a retransmission in an option.

> Iljitsch

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 13 Aug 2009 00:24:00 +0200

Hi all,

I haven't read all the email on the list, but I wanted to quickly respond to a couple of points that I read in my skim of a couple of threads (I am traveling).  Apologies if I'm repeating what's already been said:

1/ Yes, we would like to incorporate the CC work from MPTCP into CMT, and turn it into a draft.  Which is why that should be a separate doc in the MPTCP suite of docs.

2/ CMT has been implemented and it is a part of the SCTP implementation for FreeBSD.

3/ Single sequence space was an active choice for CMT:  you can get CMT without a single on-the-wire change to SCTP, which was a goal.  And, no receiver mods are required either.  CMT will work on a multihomed SCTP association, with zero SCTP receiver modifications.

4/ One of the benefits of having HeartBeats in SCTP is that you can use HBs to distinguish between different paths when trying to determine which one is working.  When you don't have HBs, as with TCP, you have to use data for probing, and you *need* to use a separate sequence space as Costin mentions for determining which path delivered a packet.

5/ If you can get SCTP through a NAT, CMT will get through as well;  CMT makes no on-the-wire changes to SCTP.

Thanks,
- jana

Michael Tüxen wrote:
>> borrowed from SCTP - according to Jana's comments in a discussion we 
>> had in Stockholm.
>>
>> addition, the single sequence space does NOT always ensure the sender 
>> can tell which path delivered a packet. For details why we chose two 
>> sequence spaces instead of one see the deisgn decisions document: 
>> http://www.cs.ucl.ac.uk/staff/C.Raiciu/files/mtcp-design
> Maybe I was not clear. I think having separate sequence number spaces helps
>>>> CMT was brought up several times in the discussion on- and 
>>>> off-list.  And I at least was under the impression that the reason 
>>>> it wasn't in  an ID yet is that the coupled congestion control bit 

-- 
Janardhan Iyengar
Assistant Professor, Computer Science
Franklin & Marshall College
http://www.fandm.edu/jiyengar