Subject: [multipathtcp] On Thursday's Multipath TCP BOF

From: Christian Vogt <[email-masked]>
Date: Wed, 29 Jul 2009 22:58:54 +0200

Dear all -

Since I won't make it to tomorrow's Multipath TCP BOF unfortunately, I
would like to express my support for this effort by email.

Multipath TCP promises to enable an attractive set of new features --
ranging from automatic fail-over, to load-balancing, to mobility
support.  Even though there are, of course, important challenges to be
surmounted, early analyses indicate the feasibility of the concept as
such.  And the strong academic background and support from key IETF
folks ensures that people with clue and sufficient time will work on it.

Therefore, personally, I believe this BOF should be given a thumbs-up
for working group establishment.

- Christian

---

From: "Rolf Winter" <[email-masked]>
Date: Thu, 30 Jul 2009 10:52:52 +0200

+1

NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road, London W3 6BL | Registered in England 2832014 

> -----Original Message-----
> From: [email-masked] [mailto:multipathtcp-
> [email-masked]] On Behalf Of Christian Vogt
> Sent: Mittwoch, 29. Juli 2009 22:59
> To: IETF Discussion Mailing List
> Cc: Multipath TCP Mailing List
> Subject: [multipathtcp] On Thursday's Multipath TCP BOF
> 
> Dear all -
> 
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Fred Baker <[email-masked]>
Date: Thu, 30 Jul 2009 11:11:24 +0200

+1.

I likewise will be unable to attend. But I think it is valuable and  
important. I do question whether perhaps it should be an IRTF effort  
rather than an IETF effort, but I think there is value in it and would  
support it being looked into.

It seems to me that there are several aspects that need consideration.  
One is the concept of multiple disparate routes being selected by the  
end station; in the Internet, the end station doesn't know anything  
about routing. In this respect, I think there are three important  
observations that could be made.

a) if there were some way to cleanly do so, such as a use of the flow  
label in some appropriate way TBD as a "route me different" flag, it  
would be nice to have the end station be able to request that the  
network deliberately spread the traffic in a stream across multiple  
routes. In the end system, if it has several appropriate addresses and/ 
or its peer does, that would include the end station sending data from/ 
to the cross product of those addresses. In the network proper,  
routers could randomly distribute traffic across equal-cost-load- 
shared paths fairly easily, and could look into less salubrious paths  
in at least some cases. In that last there be dragons, but it deserves  
looking into.

b) The set of paths so chosen will have different RTT and potentially  
other parameters. The end station will want to be aware of the impacts  
of these, and somehow prune routes (cross-products, uses of the "route  
me different" flag, etc) that aren't being helpful.

c) rather than even thinking about variations on the fast retransmit  
heuristic or SACK as currently defined, I think we need some form of  
explicit request for retransmission. One example of such an algorithm  
would be for the sender to include not only the send sequence number  
but cwnd (sender's window estimate, not receiver's buffer size) or SSN- 
cwnd in its transport header. The implication for the receiver is that  
if it receives segment SSN and is still waiting for a segment earlier  
than SSN-cwnd, it should NACK the older segment to trigger its  
retransmission. Clearly, the sender can probe for such by sending an  
empty segment that specifies cwnd=0, such as when it is terminating a  
transmission burst or has been waiting more than an RTT for an ACK and  
hasn't received it. I'm thinking along DCCP lines, therefore, in the  
use of the ACK.

On Jul 29, 2009, at 10:58 PM, Christian Vogt wrote:

> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 31 Jul 2009 10:46:53 +1200

On 2009-07-30 21:11, Fred Baker wrote:
...
> label in some appropriate way TBD as a "route me different" flag,

Actually the flow label rules specified in RFC3697 cause a bit of
an issue with any form of multipath (including SCTP and SHIM6 as
well as multipath TCP). Do we want all threads of the same connection
to carry the same flow label even if the source or destination
addresses vary? I don't have an easy answer to that, and we ducked
it in 3697.

Shim6 does say "the in-path devices need to know about the use of
the new locators even though the Flow Label stays the same." So it's
kind-of assumed that the flow label isn't changed, but a strict
interpretation of 3697 would define each {source locator, flow label}
2-uple as a separate flow, even if the flow label is the same.

    Brian

---

From: Fred Baker <[email-masked]>
Date: Fri, 31 Jul 2009 06:30:14 +0200

Well, the flow label being something that we have never actually used,  
maybe we should change the rules to make it something useful...

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 31 Jul 2009 16:48:05 +1200

I suspect this one can be fixed by clarifying rather than
changing the rules, but I have a 5 year old to-do which
is writing something about flow label use cases for the
community to discuss.

Regards
   Brian

---

From: Lars Eggert <[email-masked]>
Date: Fri, 31 Jul 2009 08:38:08 +0200

Hi,

> addresses vary? I don't have an easy answer to that, and we ducked

RFC3697 actually says in the introduction:

    A flow is a sequence of packets sent from a particular source to a
    particular unicast, anycast, or multicast destination that the  
source
    desires to label as a flow.  A flow could consist of all packets  
in a
    specific transport connection or a media stream.  However, a flow is
    not necessarily 1:1 mapped to a transport connection.

The first sentence seems to imply that the packets of different  
subflows may carry different flow labels.

The second sentence is the issue. It is true for unipath/uniflow  
transport protocols, but not multipath transport protocols.

The third sentence I don't understand :-)

But at least to me, the decision to call this "flow label" instead of  
"connection label" implies that the intent was for this label to  
identify a sequence of packet that has topology affinity, rather than  
application connection affinity.

Lars

---

From: "PAPADIMITRIOU Dimitri" <[email-masked]>
Date: Fri, 31 Jul 2009 09:45:15 +0200

I see value in working on a "linked" congestion control scheme (work
consisting in architecting and determining how efficient such scheme
would be compared to current practice/congestion control schemes). 

I have a couple of comments/concerns though:

- The BoF presentation considers that the TCP end-points controls
routing of the traffic along a certain path in the network but routing
information does not reach the end-points. So for what it concerns
routing of the traffic, per TCP connection, there is no difference
compared to the current situation (the network is not aware that two
connections entering the network belong to the same "set"). This
assumption is to be revisited because the "possible" paths towards (set
of destinations) will be as provided by the routing system and means for
triggering and enforcing diversity inside network is not achievable
without additional mechanism (at the "networking layer" level).

Note: The term "path" is from this perspective a bit confusing i.e.
inverse multiplexed TCP or multi component TCP would probably better
suite.

- The BoF presentation considers that controlling onto which
(sub-)connection the source puts traffic leads to offload the network
from performing traffic engineering. Thus, I do not see how this would
lead to a situation where the network would be free from any traffic
engineering ? (I would even say the contrary, this would lead us back to
the hyper-aggregation problem).

- On the proposed scheme itself, if the end-points assumes that
sub-connections (say between IP Address 1 and 2) can not be
re-established after detecting their failures, the degraded state would
remain permanent since the connection between Address 1 and 2 would not
be re-established. So, it may be that some of the decisions taken by the
end-points become detrimental. At this stage, it is not clear how to
prevent such situation would happen.

Note: also that some of the RTT numbers provided in the presentations
are within recovery times achievable using fast re-routing schemes. 

Thanks,
-dimitri.

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Christian Vogt
> Sent: Wednesday, July 29, 2009 10:59 PM
> Subject: On Thursday's Multipath TCP BOF
> 
> Dear all -
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 31 Jul 2009 09:50:04 +0200

> Do we want all threads of the same connection
> addresses vary?

I think Fred is talking about the case where the source and  
destination addresses actually stay the same.

This is an important intended future development of the one-ended  
multipath TCP solution. For instance, imagine the situation where  
routers do equal cost multipath (RFC 2992) load sharing across a  
bundle of links (or paths). In IPv4, the best way to do this is using  
a bunch of hash buckets with a hash over source/dest addresses and  
port numbers and the protocol number. With IPv6, this could be source/ 
dest/flow label. Unlike the IPv4 case, then it would be possible to  
influence the hash and therefore path selector without making changes  
that are imcompatible with existing receivers by using a different  
flow label for certain packets.

We've been looking for other bits that could be used for this in the  
IP header but they're hard to find...

On a mostly unrelated note, in Bob Briscoe's re-ECN session yesterday  
I suggested that he take the third ECN bit that he needs from the flow  
label if re-ECN takes off, as it would be a shame to require a 8-byte  
extension header for this while the flow label remains unused in  
practice (and could easily work with 16 bits or so).

---

From: John Leslie <[email-masked]>
Date: Fri, 31 Jul 2009 04:49:15 -0400

> would be compared to current practice/congestion control schemes). 

   Linked Congestion Control is fairly central to the Multipath TCP
idea -- but is not intended as a research topic here. Rather, it is a
way to avoid worrying about overlap of paths (some of which are "obvious"
overlaps which implementors will probably find anyway).

   By linking the congestion control, we avoid the need to hunt down
sets of IP addresses which are "the same" in terms of forwarding path.

> information does not reach the end-points.

   I'm not sure I understand. Clearly, the destination IP address
"controls" the path the packet will take, and clearly the actual path
taken isn't reported (except for "record route" option, which usually
doesn't work anyway) to the destination host. But you seem to be saying
something beyond that...

> So for what it concerns routing of the traffic, per TCP connection,
> there is no difference compared to the current situation (the network
> is not aware that two connections entering the network belong to the
> same "set").

   Agreed. "The network" is expected simply to forward packets.

> This assumption is to be revisited because the "possible" paths towards
> (set of destinations) will be as provided by the routing system and
> means for triggering and enforcing diversity inside network is not
> achievable without additional mechanism (at the "networking layer"
> level).

   We specifically avoid any expectation that the network should enforce
diversity. We expect the network to make "best effort" forwarding
decisions: no more, no less.

> inverse multiplexed TCP or multi component TCP would probably better

   I think we're stuck with the name. May I suggest we think of it as
"TCP that can run over multiple simultaneous paths" rather than "TCP
that guarantees the redundancy of multiple paths".

> from performing traffic engineering.

   I disagree. Network operators will still need to do their own
traffic engineering. Multipath TCP is intended to give end-users some
traffic-engineering ability separate from that of network operators.
We claim that this can reduce the amount of network traffic-engineering
needed, but if that claim proves false, nothing really changes.

> Thus, I do not see how this would lead to a situation where the network
> would be free from any traffic engineering ?

   I don't believe we've claimed any such thing.

> (I would even say the contrary, this would lead us back to the
> hyper-aggregation problem).

   I don't believe we've said much about that. There's a perhaps unstated
assumption that network operators may impose artificial delay or packet
loss in order to influence Multipath-TCP allocation between paths; and
we clearly believe that such "traffic-engineering" would be less harmful
than some things they're doing today.

> be re-established.

   Point well taken. However, it is not clear to me that we ever "detect
failure" rather than treating what the routing system considers to be
"failure" as, instead, "especially heavy congestion".

   The difference is, when we detect "especially heavy congestion" we
can leave the sub-connection up until its routing problem is resolved
(even if that takes several days).

> So, it may be that some of the decisions taken by the end-points become
> detrimental. At this stage, it is not clear how to prevent such
> situation would happen.

   This does imply that we might need to consider different timeouts for
tearing down sub-connections than for non-multipath TCP connections.
And that has not been discussed yet; but IMHO it could be left as an
implementation detail...

--
John Leslie <[email-masked]>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 31 Jul 2009 10:58:14 +0200

On 31 jul 2009, at 10:49, John Leslie wrote:

> it is not clear to me that we ever "detect

> (even if that takes several days).

After the BoF, Joe Touch made the point that some (unspecified)  
applications use TCP to evaluate whether some remote endpoint is  
reachable, and this is a feature we must retain. So if the initially  
chosen address pair stops working, the entire session must be torn  
down, even if there are still other working paths.

(BGP does this for eBGP but I can't see how anyone could reasonably  
want to run eBGP over multipath TCP.)

I'm not sure though how those application could have reasonably made  
this assumption. And if they couldn't, we shouldn't need to perpetuate  
this limitation. (Of course an API to opt out of MPTCP would be fine.)

---

From: Lars Eggert <[email-masked]>
Date: Fri, 31 Jul 2009 11:05:06 +0200

Hi,

> down, even if there are still other working paths.

this would severely impact the usefulness of the protocol though,  
because it destroys much of the support for nomadic connectivity that  
we'd get with MPTCP.

You know more about SHIM6 than I do, but I dimly remember that SHIM6  
keeps exposing the initial address pair to higher layers even when  
they change. If this is so, we could make the argument that the cat is  
out of the bag, i.e., that if someone ran such an app over TCP over  
SHIM6, it would similarly break, and we have IETF consensus that  
that's apparently OK.

> want to run eBGP over multipath TCP.)

Agreed. And I don't know of any other app.

> this limitation. (Of course an API to opt out of MPTCP would be fine.)

Also agreed.

Lars

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Jul 2009 02:16:03 -0700

Iljitsch van Beijnum wrote:
> chosen address pair stops working, the entire session must be torn down,
> even if there are still other working paths.
> 
> (BGP does this for eBGP but I can't see how anyone could reasonably want
> to run eBGP over multipath TCP.)

There are separate issues:

	- whether an app wants to use one of the (at least 4) variants
	of what mulltipath TCP means

	- whether an app even knows this is happening (e.g., if an OS
	sets a flag to "try to use multipath TCP for legacy apps")

IMO, if an app decides to use multipath explicitly, it might as well
have used SCTP. If it needs to go through a NAT, do something like SCTP
over UDP ;-) I.e., NAT traversal for legacy apps is not a great reason
to muck around with TCP semantics.

> this assumption. 

We discussed this. IMO, RFC 793 establishes that TCP is a service:

	A. *of* a reliable, in-order byte-stream++
		(where ++ includes congestion response, etc.,
		as added later)

	B. *between* the endpoints of the connection's socket pair
	(again, where socket pair is addr/port pair from 793,
	not a unix-ism)
		this is part of the defined API of TCP

AFAICT, (B) is as much a part of TCP as is (A).

Now, whether an app should assume that having a stable TCP connection
should be a prerequisite to anything is a good question; I don't think
it ought to be a prerequisite to dynamic interdomain routing (eBGP), but
that ship has sailed.

My concern is that there are (and could/should be) applications that
expect a TCP connection to go down when the local IP address is down.
TCP is a service for a bytestream between an IP address/port pair, it is
not defined as binding to something else that happens to map to an
IP/address port pair right now.

The implications if this would be:

	- assume multipath TCP opens connections using a primary
	(initial) socket pair

	- as long as that socketpair is available*, the connection
	stays up; when not, the connection is torn down

* = where "available" could mean just that both sides still think that
the IP address is bound to one of their interfaces.

I.e., you REALLY do not want the following case:

	A: adr 11, adr 12
	B: adr 23, adr 24

A adr 11 port 52394 connects to B adr 23 port 80
A and B open a second TCP connection to use in the multipath set, e.g.,
from A 12 -> B 24

A ifconfig's adr 11 down

C ifconfig's adr 11 up

Now what?

IMO, if A-B would have lost its connection if a single TCP, then it
should have done so over multipath TCP. Note that this does allow A/11
to have a heartbeat to B/23, OR to just ensure that both sides shut the
connection when they receive a RST on the original connection.

> And if they couldn't, we shouldn't need to perpetuate

Opt-in is the only viable solution if the semantics are changed.

Joe

---

From: Lars Eggert <[email-masked]>
Date: Fri, 31 Jul 2009 11:40:45 +0200

Hi,

> to muck around with TCP semantics.

it's not quite as simple as that (you also need the CMT extension,  
plus you need to add something like the coupled congestion control to  
CMT). But I agree that some of the appeal comes from being transparent  
to existing apps.

> AFAICT, (B) is as much a part of TCP as is (A).

Agreed. And one way to view what MPTCP does is that it extends B to  
where a connection exists between a set of "subflow endpoints" that  
can change over time, and the connection stays alive while at least  
one such subflow endpoint remains associated with the connection at  
each end system.

In other words, it ties the existence of an MPTCP connection to the  
availability of IP connectivity between the end systems, rather than  
to the availability of IP connectivity *along a single path*.

...
> Opt-in is the only viable solution if the semantics are changed.

I wouldn't go quite as far. The intent is to publish an Experimental  
specification, and one question to look at should be if there are  
applications that will break if MPTCP is shoved in underneath them.

BGP/LDP probably would, but off the top of my head I can't think of  
others. And we know that some applications should be safe. For  
example, everything that runs inside TLS should be fine, because those  
apps are already binding to a TLS context rather than an TCP endpoint.

Lars