Subject: [multipathtcp] [Fwd: Re: multipath tcp disucssion]

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 01 Apr 2009 20:48:26 +0200

forwarding on behalf of Joe Touch

-------- Mensaje original --------
Asunto: 	Re: multipath tcp disucssion
Fecha: 	Wed, 01 Apr 2009 11:42:58 -0700
De: 	Joe Touch <[email-masked]>
Para: 	marcelo bagnulo braun <[email-masked]>
Referencias: 	<[email-masked]> <[email-masked]> 
<[email-masked]>

Here's one to forward:

I am concerned about introducing a different model of a TCP connection
than the "socket pair" (srcIP, dstIP, srcport, dstport), on which many
other aspects of the Internet depend.

This includes anything that depends on a connection as being atomic:
	NATs
	IPsec
	TCP MD5 / TCP-AO
	load balancers
	user applications that listen on given socket pairs
	ICMP handling
	TCP 'signal' packet handling

The question is whether multipath TCP is consistent with those other
systems, and the impact of using multiple TCP connections to emulate a
single socket pair.

E.g., when a single connection changes state, is protected/blocked,
etc., what happens to the other connections in the set, and are they
interrelated. The problem appears to be that the desired behavior might
be different, depending on the system, e.g.:

	IPsec, TCP MD5, TCP-AO could be:

		one blocked -> all blocked

		one protected -> all protected
			where do the extra keys come from?
			who establishes them, esp. if manual?

		how do you ensure that all socket pairs have
		the same protection?

	load balancers:
		how do you ensure that different socket pairs
		end up at the same server?

	ICMP, TCP 'signal' (RST, SYN, FIN):
		does a hard error / shutdown / reset on
		one socket pair affect all socket pairs,
		or only the one signalled?

		when does it make sense to have fate sharing,
		and when does it make sense to decouple,
		and how do both ends know when to apply which?

Overall, the question is:

TCP currently presents a model of "one socket pair = one byte stream" to
both the network and to the application.

What model does multipath TCP present to the network and to the
application, and how is it enforced? Is that model compatible with the
current model (can the use of multiple socket pairs be hidden from
either the network or the application)?

- ----

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 2 Apr 2009 19:28:08 +0200

Hi Joe,

> systems, and the impact of using multiple TCP connections to emulate a

First: there are different ways to do multipath TCP. The one you're  
addressing here is where TCP manages multiple addresses, just like  
SCTP does. Two other models are the one where TCP conforms to the  
current protocol specifications (the "one ended" multipath TCP with  
only changes to the sender) or a multipath TCP that runs on top of an  
existing multi-address system like shim6.

> 	IPsec, TCP MD5, TCP-AO could be:

I think you have a good point with regard to IPsec. We spent quite a  
bit of time on that with shim6, but those results aren't immediately  
applicable to an MAmTCP because shim6 can have IPsec both below or  
above it, with TCP it's only below. My initial thinking is that if the  
initial/main address pair is IPsec protected then it should be  
enforced that additional ones are also IPsec protected.

By the way, how is this solved in SCTP?

As for TCP MD5: first of all, I don't think this is an issue in  
practice because people won't run their BGP sessions over multipath  
TCP. Second, the MD5 protection is agnostic towards the addresses. I  
don't know if the addresses are part of the MD5 calculation, but if  
they are, the other side will see the same addresses and calculate the  
same MD5. If they're not, it's even simpler.

I don't know what AO stands for.

> 		end up at the same server?

Probably with some kind of connection identifier cookie in the initial  
exchange.

> 		or only the one signalled?

This is an interesting issue, which sould be solved when the MAmTCP is  
designed. There are different ways to do this with different strengths  
and weaknesses.

> either the network or the application)?

Shim6 manages to hide the extra addresses from the application, so I'd  
say "yes" on that part. Not sure what you mean by hiding addresses  
from the network.

There are of course applications that attach a lot of significance to  
the addresses they run over. It can be argued that this is not the  
best possible design, but something needs to be in place to make sure  
nothing really bad happens in these cases. Maybe just telling people  
to not configure multiple addresses is enough here, or maybe we need  
an API so applications can signal their needs. Or something else.

Iljitsch

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Apr 2009 10:45:27 -0700

Hi, Iljitsch,

Iljitsch van Beijnum wrote:
> addressing here is where TCP manages multiple addresses, just like SCTP
> does. Two other models are the one where TCP conforms to the current
> protocol specifications (the "one ended" multipath TCP with only changes
> to the sender) or a multipath TCP that runs on top of an existing
> multi-address system like shim6.

I'm not really focusing on the way multipath is done; I'm saying that
all solutions that claim to be "TCP" need to address compatibility with
the application-layer semantics of a single TCP connection. If multiple
TCP connections are presented to the application as separate channels,
then you're just talking about coordinating separate TCP connections,
which was addressed in RFC2140, e.g.

> I think you have a good point with regard to IPsec. We spent quite a bit
> of time on that with shim6, but those results aren't immediately
> applicable to an MAmTCP because shim6 can have IPsec both below or above
> it, with TCP it's only below. My initial thinking is that if the
> initial/main address pair is IPsec protected then it should be enforced
> that additional ones are also IPsec protected.

Yes, but the questions are:
	- how is that enforcement provided
	- is it provided only for IKE'd sessions, or for manual
	keying as well?
	- how does TCP know that IPsec is doing the right thing?

That last one is the killer, IMO.

> By the way, how is this solved in SCTP?

I don't know whether it is. SCTP, however, doesn't intend to present a
single socket pair as a connection to the application, AFAICT, so
there's no expectation of the kind of security fate-sharing I'm
concerned about.

> As for TCP MD5: first of all, I don't think this is an issue in practice
> because people won't run their BGP sessions over multipath TCP.

TCP MD5 can be used for more than BGP, and TCP-AO (its replacement) can
as well. I would think that running BGP over multipath would have uses,
though - e.g., to allow BGP to remain stable when parallel links are
provided for fault tolerance.

> Second,
> the MD5 protection is agnostic towards the addresses. I don't know if
> the addresses are part of the MD5 calculation, but if they are, the
> other side will see the same addresses and calculate the same MD5. If
> they're not, it's even simpler.

TCP MD5 and TCP-AO both include the IP pseudoheader in the MAC
calculation. The problem is the same as for IPsec - fate sharing,
knowing that all TCPs are similarly protected.

> I don't know what AO stands for.

Authentication Option:

http://www.rfc-editor.org/cgi-bin/iddoctype.pl?letsgo=draft-ietf-tcpm-tcp-auth-opt-04

> Probably with some kind of connection identifier cookie in the initial

That seems to force application participation, at which point the app
can/should be doing the multiplexing onto the different TCP streams.
That works, but needs zero support from TCP (except, as noted above,
RFC2140-style coordination).

> designed. There are different ways to do this with different strengths

IMO this is a protocol specification issue, not an implementation issue.

> say "yes" on that part. Not sure what you mean by hiding addresses from
> the network.

Same kind of thing w.r.t. ICMP errors, ECN, or flow-based drops. Does
the network know these connections are related?

> the addresses they run over. It can be argued that this is not the best
> possible design, but something needs to be in place to make sure nothing
> really bad happens in these cases. Maybe just telling people to not
> configure multiple addresses is enough here, or maybe we need an API so
> applications can signal their needs. Or something else.\

My basic point is that applications think that a single byte-stream is
associated with a single socket pair because that's the model TCP
presents to them. It's not a design issue; it's architectural - the
definition of the endpoints of a bytestream is the socketpair. Changing
that changes the application semantics - i.e., requires a major change
to the API in RFC793...

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 2 Apr 2009 22:08:29 +0200

On 2 apr 2009, at 19:45, Joe Touch wrote:

> I'm saying that

Right. This is certainly NOT what we're talking about here. Today, TCP  
is a single funnel into the network. Anything that you pour into it at  
one end comes out in the same order at funnel at the other end. This  
isn't going to change in a multipath enabled TCP. What the multipath  
does is connect more than a single tube to the ends of the funnels.

An important aspect of that is that if one of the paths breaks or  
clogs up, the packets are subsequently sent over (one of) the  
remaining working path(s). And the lost packets must also be  
retransmitted over a working path, probably before a normal RTO would  
expire in order to avoid the receive buffer from filling up because of  
lost packets, making the working path stall.

>> My initial thinking is that if the
>> enforced

> 	- how does TCP know that IPsec is doing the right thing?

If IPsec implementations running on multipath TCP hosts already  
support the right API the multipath TCP could instruct the IPsec to do  
this. If not, the IPsec would have to be modified.

>> practice

> as well.

That doesn't make sense, both TLS and IPsec are superior.

> I would think that running BGP over multipath would have uses,

Well that kind of happens today when you use a single TCP session on  
multiple links between two routers. You do this by setting up the  
session between addresses tied to stable virtual addresses, which are  
statically routed over all of the links. So the packets are load  
balanced over the full set of links (although the load balancing could  
end up being that all the BGP packets only use one link, but in this  
case that doesn't matter).

>> other side will see the same addresses and calculate the same MD5.

> knowing that all TCPs are similarly protected.

There is no "all TCPs". It's one TCP session, it's just split into  
multiple subflows. So from above (the application layer) it's still  
one session so everything that you attach from above, like MD5  
protection, applies to the entire session. Only from below you _may_  
see the different subflows.

> That seems to force application participation,

I don't see that. TCP could simply negotiate additional subflows  
transparently to the application. The negotiation then fails if some  
subflows end up at another instance of a clustered service.

> at which point the app
> That works, but needs zero support from TCP

The trouble here is that the application must now split the data into  
multiple byte streams. For some stuff that's not a huge problem (file  
transfer like BitTorrent) but even in those cases it's not very  
flexible: once a piece of data is commited to a path, only an  
application level timeout or a failure allows for it to be rerouted  
over a different path. The transport layer can do this at RTT  
timescales without much extra overhead.

> IMO this is a protocol specification issue, not an implementation  

Absolutely, this can only work well if both ends agree on how to do it.

> Does the network know these connections are related?

Does the network have a notion of connections in the first place?

> that changes the application semantics - i.e., requires a major change

Disagree. Per RFC 793, it's completely normal behavior for the network  
to send some packets over one path and some over another path. Having  
the TCP sender tell the network which packets to send over which path  
is an important optimization because now it's possible to run  
congestion control instances for each path, so TCP can adapt to the  
path properties. This is all firmly within the bounds of layers 3 and  
4 and outside the scope of higher layers.

Once additional addresses enter the picture, this is arguably no  
longer the case, but NATs, different types of tunnels, MIP, shim6 etc  
set precedents here long ago: applications can't expect to know all  
the details of the addresses used in their communication.

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Apr 2009 13:28:32 -0700

> On 2 apr 2009, at 19:45, Joe Touch wrote:
..
> If IPsec implementations running on multipath TCP hosts already support
> the right API the multipath TCP could instruct the IPsec to do this. If
> not, the IPsec would have to be modified.

Agreed. I just wonder whether it's even possible to modify IPsec to do
this. I.e., IPsec (and IKE) needs to know about 'groups of transport
connections that need to be treated the same'. That can be achieved in
the configuration (i.e., using wildcards in port fields, or using
identical configurations for different connection specifications), but
there's nothing in IPsec that flags that inside IPsec.

> That doesn't make sense, both TLS and IPsec are superior.

Please check our doc. TLS does not protect a TCP connection from
TCP-level attacks, e.g., RSTs. IPsec cannot be forced to use new keys on
new connections (i.e., upon receipt of a RST, destroy the SA). TCP-AO
does both of these (TCP MD5 only the former), and is much simpler than
IPsec (which is why it is preferred for BGP).

> case that doesn't matter).

Agreed. The current solution requires that the single IP address is
routed over multiple link-layer paths; I'm suggesting utility to a
solution that does not have that constraint. I don't know if there's a
particular reason to encourage this using multiTCP, vs., e.g., single
TCP over a virtual link tunnel (e.g., GRE).

However, that begs the question of whether that technique would be
useful elsewhere - i.e., a multipath TCP solution that deploys its own
'virtual diversification layer' underneath...

> multiple subflows. So from above (the application layer) it's still one
> session so everything that you attach from above, like MD5 protection,
> applies to the entire session. Only from below you _may_ see the
> different subflows.

If they subflows use different ports, then TCP MD5 would think they are
different connections. So the answer depends on what the lower layer sees.

> subflows end up at another instance of a clustered service.

Oh - yes, I see that now. It requires modification of the endpoints, but
not necessarily at the application layer.

> application level timeout or a failure allows for it to be rerouted over
> a different path. The transport layer can do this at RTT timescales
> without much extra overhead.

It can be done inbetween the app and transport, i.e., inside the socket
buffers too. I.e., it can be done in a layer above TCP but below the
app, if isolation from the app is needed. That's partly done by things
like HTTP persistent connections with chunked responses.

> Does the network have a notion of connections in the first place?

Yes. In places I've noted, at least:

	network layer in the end host or BITW: IPsec
	NATs
	multipath routing based on flow preservation
	firewall filtering
	when ICMPs are issued (in the network or at the dest host)

> the TCP sender tell the network which packets to send over which path is
> an important optimization because now it's possible to run congestion
> control instances for each path, so TCP can adapt to the path
> properties. This is all firmly within the bounds of layers 3 and 4 and
> outside the scope of higher layers.

I said socket pair, not path. If you keep the socket pair the same
(srcIP, dstIP, srcPort, dstPort), it all works consistent with RFC793.
If you have more than one socket pair in the network supporting a single
socket pair at the endpoints, then you have changed the RFC793 model.

> Once additional addresses enter the picture, this is arguably no longer
> the case, but NATs, different types of tunnels, MIP, shim6 etc set
> precedents here long ago: applications can't expect to know all the
> details of the addresses used in their communication.

They do expect that a single socket pair corresponds to a single
connection -  no more, no less. NATs, etc. preserve this. Anything that
violates this breaks TCP.

Joe

---

From: Michael Tüxen <[email-masked]>
Date: Thu, 2 Apr 2009 22:55:09 +0200

Hi Iljitsch,

a comment in-line.

Best regards
Michael

On Apr 2, 2009, at 7:28 PM, Iljitsch van Beijnum wrote:

> only changes to the sender) or a multipath TCP that runs on top of  
> an existing multi-address system like shim6.
>
> above it, with TCP it's only below. My initial thinking is that if  
> the initial/main address pair is IPsec protected then it should be  
Have a look at
http://www.ietf.org/rfc/rfc3554.txt
but it is (up to my knowledge) not implemented yet.
> they are, the other side will see the same addresses and calculate  
> the same MD5. If they're not, it's even simpler.
>
> Probably with some kind of connection identifier cookie in the  
> initial exchange.
>
> This is an interesting issue, which sould be solved when the MAmTCP  
> is designed. There are different ways to do this with different  
> strengths and weaknesses.
>
> Shim6 manages to hide the extra addresses from the application, so  
> I'd say "yes" on that part. Not sure what you mean by hiding  
> addresses from the network.
>
> There are of course applications that attach a lot of significance  
> to the addresses they run over. It can be argued that this is not  
> the best possible design, but something needs to be in place to make  
> sure nothing really bad happens in these cases. Maybe just telling  
> people to not configure multiple addresses is enough here, or maybe  
> we need an API so applications can signal their needs. Or something  
> else.
>
> Iljitsch
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>

---

From: Michael Tüxen <[email-masked]>
Date: Thu, 2 Apr 2009 22:58:40 +0200

On Apr 2, 2009, at 7:45 PM, Joe Touch wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi, Iljitsch,
>
>> SCTP
>> above
SCTP supports two types of sockets:
One-to-One:  This is the same as TCP. A socket is either a listener
              or represents one end of an SCTP association.
One-to-Many: This is simular to non-connected UDP sockets. A socket
              can handle multiple SCTP associations. The API provides
              an association ID, which can be used to identify the
              association.
>> initial
>> is
>> I'd
>> from
>> best
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAknU+bcACgkQE5f5cImnZrtcpgCgw02cZrNdhCk6gowYRg8UyTLk
> jvwAn3DqH0SK09M3K0QmBgGPGg/Xa3UZ
> =xiYc
> -----END PGP SIGNATURE-----

---

From: Randall Stewart <[email-masked]>
Date: Thu, 2 Apr 2009 17:06:45 -0400

One other little comment for Joe...

>             an association ID, which can be used to identify the

The above is all true.. but there is one other fundamental assumption in
SCTP. An SCTP association (no matter which socket model) always assumes
the possibility of the endpoints having multiple addresses. This means
applications developed for SCTP should be aware of the fact that the
fate sharing implied by an SCTP association is a combination of
all of the addresses involved in the association. As long as a
workable communication pair exists between two endpoints the association
may continue to exist.

R
------------------------------
Randall Stewart
803-317-4952 (cell)
803-345-0391(direct)

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Apr 2009 15:50:23 -0700

> One other little comment for Joe...
...
> workable communication pair exists between two endpoints the association

Right - and this assumption is NOT true for TCP applications. This means
that SCTP applications would assume they would need to manage IPsec /
TCP MD5 / TCP-AO etc., and NAT boxes may need to do special things to
track groups of endpoint addresses associated with a single SCTP
association, whereas that's not true for TCP.

Joe

---

From: Randall Stewart <[email-masked]>
Date: Thu, 2 Apr 2009 21:04:39 -0400

I would agree with that ;-)

R
On Apr 2, 2009, at 6:50 PM, Joe Touch wrote:

> iEYEARECAAYFAknVQS8ACgkQE5f5cImnZrv7pQCggDLawADdlt39ltbGv1njeK8a
> lgAAoOcJZjB89g5A8ve+ISvgOa1nkU9y
> =EPSM

---

From: João Taveira Araújo <[email-masked]>
Date: Fri, 03 Apr 2009 12:01:54 +0100

Hi,

Just picking up on two of Iljitsch's comments:

> existing multi-address system like shim6.

(...)

> lost packets, making the working path stall.

It would be convenient to start defining what the limitations of each 
model is, and not make overarching statements that may not apply to all 
variants. I briefly asked this during the IETF presentation via jabber, 
but here goes: in the "one ended" multipath TCP, how do you retransmit 
lost segments over different paths without confusing the legacy 
receiver, who has one of his connections stalled? Mark's proposal solves 
this by adding subflow sequence numbers alongside data sequence numbers, 
at the expense of requiring a compatible receiver. In the legacy case, 
it doesn't seem quite as simple.

Cheers,
Joao.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 3 Apr 2009 13:20:11 +0200

On 3 apr 2009, at 13:01, João Taveira Araújo wrote:

> It would be convenient to start defining what the limitations of  
> each model is, and not make overarching statements that may not  
> apply to all variants. I briefly asked this during the IETF  
> presentation via jabber, but here goes: in the "one ended" multipath  
> TCP, how do you retransmit lost segments over different paths  
> without confusing the legacy receiver, who has one of his  
> connections stalled?

The legacy receiver is blissfully unaware of all of this.

Assuming some mechanism for the sender to make packets flow over  
different paths (yes, some handwaving here, for now assume the sender  
has multiple NICs that connect to the network in multiple places, but  
the multiple NICs all use the same address), the sender can simply  
send normal TCP packets over different paths that will be processed  
normally by an existing receiver. The receiver will of course see some  
reordering, but otherwise it just sees regular TCP.

As long as the sender is aware of which path is progressing the sender  
can do what's necessary to make this work. The normal cumulative ACK  
isn't all that helpful here because with the expected reordering,  
feeback about packets received out of order will be delayed until the  
holes in the seqnum space are filled in, slowing everything down.  
Fortunately, SACK is widely deployed these days so SACK can be used to  
determine which packets were received, so the sender can progress the  
appropriate path.

The single ended model does have the limitation that it can't  
conclusively correlate an ACK with the path taken by the ACKed packet  
if there was a retransmission.

---

From: João Taveira Araújo <[email-masked]>
Date: Fri, 03 Apr 2009 12:42:22 +0100

Iljitsch van Beijnum wrote:
>> model is, and not make overarching statements that may not apply to 
>> all variants. I briefly asked this during the IETF presentation via 
>> jabber, but here goes: in the "one ended" multipath TCP, how do you 
>> retransmit lost segments over different paths without confusing the 
>> legacy receiver, who has one of his connections stalled?
>

Ok, so the assumption here is that the socket pair must remain 
consistent. You could use multiple interfaces / some form of tagging / 
tunneling. Putting quibbles from a CC perspective aside for now, how 
does this conform with the semantics of TCP as Joe was describing them? 
Example off the top of my head: you get a NAT on any of the concurrent 
paths, all of the sudden your consistent socket pair appears as 
something entirely different on the other side.

> if there was a retransmission.

TBH, I don't think the single ended model has much merit, in considering 
that endpoints benefit the most from multipath, so the incentive to 
upgrade would be there, and fallback to legacy TCP would always be in 
place. The question is whether it makes architectural sense to design 
multipath as an extension of TCP or whether we're doing that because it 
seems to be the only deployment model available, and we may end up with 
an ugly hack which breaks stuff.

I'm all for it btw, just playing devil's advocate.

Joao

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 03 Apr 2009 14:19:22 +0200

João Taveira Araújo escribió:
>> normally by an existing receiver. The receiver will of course see 
>> some reordering, but otherwise it just sees regular TCP.
>

in the case of the one ended multipaht tcp, what happens is that 
differnent packets are routed over different paths by the routing 
system. This is not very different with what happens today in the case 
that there is a failure and that the routing system decides to use an 
alternative path for the following packets.

> something entirely different on the other side.

that would depend on where the nat is located, right?

> TBH, I don't think the single ended model has much merit, in 
> considering that endpoints benefit the most from multipath, so the 
> incentive to upgrade would be there,
interesting... the same comment would apply to widely deployed protocols 
such as HIP, shim6, sctp.... :-)

In other words, there seems to be a significant deployment vector 
advantage if the end that deploys the new mechanism obtain benefits, 
irrespectively whether the other end supports it or not imho
Clearly, both flavours (one ended and two ended) should play well 
toghether in the sense that if both ends support it, then most of the 
advantages should be realized, but it would improve significantly the 
deployment model if when one ends supports it, then it actually obtains 
benefits.

the use case for one ended multipath tcp is the case of a multihomed 
content provider, which by using multipaht tcp, if can improve the 
performnce perceived by the clients. I think this is a significant 
benefit and i think it can be realized wihtou need for the client to be 
updated.

> and fallback to legacy TCP would always be in place. The question is 
> whether it makes architectural sense to design multipath as an 
> extension of TCP or whether we're doing that because it seems to be 
> the only deployment model available, and we may end up with an ugly 
> hack which breaks stuff.
>
it is interesting that you qualify it as an ugly hack and that break 
stuff wihtout actually seeing any form of proposal of how this could be 
done. On your perspective, what actually breaks?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 3 Apr 2009 14:33:08 +0200

On 2 apr 2009, at 22:28, Joe Touch wrote:

>> support

> there's nothing in IPsec that flags that inside IPsec.

I'm not sure what kind of APIs exist to make applications control  
IPsec behavior. If those exist, it would of course be doable for  
multipath TCP to simply make sure IPsec behavior conforms to what the  
application specified for each subflow. If the desired IPsec behavior  
is specified in a racoon config file or the like this would be  
significantly more complex, and probably not possible the way things  
are today.

If the IPsec is implemented in the same stack as the multipath TCP the  
two can coordinate the IPsec behavior on the subflows, but in the case  
of an external system applying IPsec tunneling that isn't really  
possible. I guess this is a limitation that must be carefully  
documented.

> IPsec (which is why it is preferred for BGP).

I would have preferred to see the effort in this area spent on making  
IPsec more usable in this scenario. IPsec has an anti-replay counter  
that is sorely missed in the TCP MD5 scenario.

[BGP over multiple paths]

> The current solution requires that the single IP address is

The problem with this way of running BGP is that it only works between  
two hosts connected to the same set of links. In the case of eBGP this  
is already a requirement for other reasons so it's not a limitation,  
but it does make this type of use less useful for other purposes. I  
also don't think that using multipath TCP for BGP adds much, so I  
doubt it will be used for that in practice.

> different connections. So the answer depends on what the lower layer  

I would assume that TCP MD5 is enabled in the TCP control block, not  
on an address/port number basis. So any segment that is part of the  
session will have the MD5 applied, regardless of the addresses that  
are in the packet.

>> over

> like HTTP persistent connections with chunked responses.

You would need to add sequence numbers so the data can be put back in  
the right order at the receiving end.

> 	when ICMPs are issued (in the network or at the dest host)

Today a single session is a single flow. If a session has multiple  
flows, does this mean that middleboxes etc recognize those (sub)flows  
as belonging to the same session, or can they simply treat them as  
different flows? The latter is easier to support and would be  
sufficient in most cases, IMO. The IPsec case is an example of where  
we'd want to be able to recognize sessions rather than subflows, though.

> I said socket pair, not path.

Ok, I missed that. Do you have a pointer to something that explains  
what those are and how they're used?

As far as I know, hosts work with sockets and control blocks. Those  
obviously relate to the addresses and port numbers, but within a host  
it's the socket or TCP control block that identifies a session.

>> longer

> violates this breaks TCP.

A multiaddress multipath TCP would certainly extend the existing TCP  
model, yes.

---

From: João Taveira Araújo <[email-masked]>
Date: Fri, 03 Apr 2009 14:47:03 +0100

marcelo bagnulo braun wrote:
>> Example off the top of my head: you get a NAT on any of the 
>> concurrent paths, all of the sudden your consistent socket pair 
>> appears as something entirely different on the other side.
>

Right.

>>> conclusively correlate an ACK with the path taken by the ACKed 
>>> packet if there was a retransmission.
>>
> interesting... the same comment would apply to widely deployed 
> protocols such as HIP, shim6, sctp.... :-)
>
> deployment model if when one ends supports it, then it actually 
> obtains benefits.
>
> benefit and i think it can be realized wihtou need for the client to 
> be updated.
>

The problem here is you seem to imply that both "flavours" can coexist, 
so the two ended multipath can be seen as an extension of the one ended 
model and would bring even more benefits. My concern would be that by 
supporting one ended multipath you might concede to design limitations 
which then unnecessarily hinder the two ended model. If we come up with 
a technically superior two ended multipath proposal which can't readily 
be translated to a sender-only configuration, should we abandon it in 
favour of a proposal which manages to do both?

My hunch is that in this case, the effort required in upgrading is small 
compared to the profits you reap. The protocols you used as examples (I 
wouldn't call them widely deployed just yet) have slightly different 
obstacles to deployment. Multipath TCP plays nice with the current 
architecture and just changes how you deal with some semantics, so I'd 
liken it more to an extension of TCP's CC than a full blown protocol 
(not quite correct I know). What Joe brought up was that doing this 
could prove disruptive in some respects due to the underlying 
assumptions the TCP model has.

> stuff wihtout actually seeing any form of proposal of how this could 
> be done. On your perspective, what actually breaks?
>

I didn't classify it as an ugly hack, I said we MAY end up with an ugly 
hack, and I have seen a proposal on how this could be done. For a list 
of what could break Joe's list is a good start.

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 Apr 2009 11:07:02 -0700

...
> that is sorely missed in the TCP MD5 scenario.

It would be useful to look at TCP-AO, which deprecates TCP MD5. TCP-AO
includes a sequence number extension to protect against replay attacks.

> I would assume that TCP MD5 is enabled in the TCP control block, not on
> an address/port number basis. So any segment that is part of the session
> will have the MD5 applied, regardless of the addresses that are in the
> packet.

TCP MD5 does not specify how it is enabled. TCP-AO addresses this by
specifying protection on a socket pair basis - exactly because that is
what uniquely defines a TCP connection.

The TCB is an implementation suggestion in RFC793, not a protocol
requirement.

> the right order at the receiving end.

Agreed. Seems like that's required anyway, though - once you spread a
single stream's data over multiple connections.

> flows, does this mean that middleboxes etc recognize those (sub)flows as
> belonging to the same session, or can they simply treat them as
> different flows? 

That depends on the fate sharing you require between the flows. Do you
expect them to go to the same place? Have a set of ports/addresses that
both sides know are related? All get through or none? Any fate sharing
is impacted by these boxes.

> The latter is easier to support and would be sufficient
> in most cases, IMO. The IPsec case is an example of where we'd want to
> be able to recognize sessions rather than subflows, though.

Load balancers that send flows to different servers too. Firewalls that
permit some and deny others maybe as well. This all needs to be worked
out - and I'm not sure there *is* a solution to most of it...

> Ok, I missed that. Do you have a pointer to something that explains what
> those are and how they're used?

RFC793, page 5:
    To allow for many processes within a single Host to use TCP
    communication facilities simultaneously, the TCP provides a set of
    addresses or ports within each host.  Concatenated with the network
    and host addresses from the internet communication layer, this forms
    a socket.  A pair of sockets uniquely identifies each connection.
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    That is, a socket may be simultaneously used in multiple
    connections.

> it's the socket or TCP control block that identifies a session.

A socket is defined in RFC793 as an address/port pair. That's not
necessarily how people use the term in implementations. The TCB is
introduced in RFC793 as an implementation term. You need to work with
the semantics of a *connection*, which is what TCP is all about.

> A multiaddress multipath TCP would certainly extend the existing TCP

It is that which, AFAICT, needs development, and explanation as to
whether it is, or even can be, backwards compatible.

IMO, if you end up with something that isn't, you probably need to show
benefit over SCTP (which already exists).

Joe

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 3 Apr 2009 11:24:49 -0700

Un-mentioned in this thread is that TCP-AO does not work across
a NAT.  Reviewing Iljitsch's slides they don't show a NAT or
NAT64 so maybe this isn't a concern to anyone but me.

http://www.ietf.org/mail-archive/web/tcpm/current/msg04001.html
http://tools.ietf.org/html/draft-ietf-tcpm-tcp-auth-opt-04#section-11.2

We could create TCP-AObis, I suppose, which works over NAT 
and NAPT.

-d

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Joe Touch
> Sent: Friday, April 03, 2009 11:07 AM
> To: Iljitsch van Beijnum
> Cc: [email-masked]
> Subject: Re: [multipathtcp] [Fwd: Re: multipath tcp disucssion]
> 
> >> different connections. So the answer depends on what the 
> lower layer
> >> sees.
> > 
> iEYEARECAAYFAknWRqcACgkQE5f5cImnZrs6WACg7ZLMoO0f2iTC4kWq/K4QQiRz
> +BwAnj9a3cCMvhkiNy8dDJCOlb1mRc6a
> =3UAA

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 Apr 2009 11:58:39 -0700

> We could create TCP-AObis, I suppose, which works over NAT 

NAT traversal is addressed in the document. Direct support was rejected
because the functions of a NAT are the kind of attack TCP-AO, like
IPsec, is designed to detect.

I want to stress that once TCP-AO is complete, there are NO plans for a
- -bis. This is intended to be an upgrade to TCP MD5, used largely where
TCP MD5 is used. Protection across NATs is already supportable using UDP
tunnels, as is done with IPsec.

However, note that UDP tunnels through a NAT still force a single IPsec
association to be treated as a unit. Multiple TCP connections treated
differently wouldn't work with existing UDP tunneling solutions, e.g.,
for IPsec.

Joe

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 3 Apr 2009 12:22:59 -0700

> From: Joe Touch [mailto:[email-masked]] 
> Sent: Friday, April 03, 2009 11:59 AM
> To: Dan Wing
> Subject: Re: [multipathtcp] TCP-AO with NAT [was RE: [Fwd: 
> Re: multipath tcp disucssion]]
> 
> TCP MD5 is used. Protection across NATs is already 
> supportable using UDP tunnels, as is done with IPsec.

Yes, that is well explained in Section 11.2 of the document which I cited in
my email.  I didn't intend to ruffle feathers but rather intended to make sure
everyone is well aware that TCP-AO cannot natively survive across a NAT.

> However, note that UDP tunnels through a NAT still force a 
> single IPsec association to be treated as a unit.  Multiple TCP 
> connections treated differently wouldn't work with existing UDP 
> tunneling solutions, e.g., for IPsec.

I don't follow your last paragraph.

-d

> iEYEARECAAYFAknWXF8ACgkQE5f5cImnZrt5NACgjZt1NzSANTyEUl1M/xzwLaPW
> OKQAoN2kqUelqEIGdE1+tigyn4htXuvz
> =wAXf

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 Apr 2009 15:41:14 -0700

Dan Wing wrote:
>> I didn't intend to ruffle feathers but rather intended to make sure

TCPM discussed whether we wanted to support NAT traversal inside TCP-AO,
or whether we would assume a UDP tunnel would be needed as is done with
IPsec. We chose the latter exactly because we wanted to retain the
correlation between the socket pair understood by the endpoints and the
TCP connection - i.e., one of the points of concern I've raised with
multi-TCP.

In TCP-AO, the session protection information (i.e., keying material,
protection parameters) is determined by the socket pair, where one of
multiple keys/algorithms is selected by the keyID field. To support NAT
traversal, we realized we would need to have the endpoints establish
separate unique connection identifiers, and that would exceed TCP
semantics. TCP-AO is intended to protect TCP connections, not to
redefine them. That's why we decided as we did regarding NAT traversal.

>> I don't follow your last paragraph.

Running a single TCP connection through a single UDP tunnel results in a
1:1 correspondence between the connection and the tunnel.

If multiple TCP connections are intended to behave as a unit, they would
need to sent through a single UDP tunnel to have a similar effect -
otherwise, a NAT could end up causing the different TCP connections to
be handled differently at the receiving side - e.g., at a load balancer,
they might be directed different places.

Joe

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 3 Apr 2009 15:48:59 -0700

> redefine them. That's why we decided as we did regarding NAT 

Yes.  I was involved with those threads.  I remember.

I am not attempting to revisit that decision.

> Running a single TCP connection through a single UDP tunnel 
> results in a 1:1 correspondence between the connection and the 
> tunnel.
> 

Thanks for explaining.

If I undestand correctly, you're saying that if TCP multipath wanted to
traverse a NAT and TCP multipath used TCP-AO, then TCP multipath would need to
use the same UDP tunnel for both of the TCP paths.  If my understanding is
correct, then that means the same NAT would have to be traversed for both TCP
paths.

-d

> iEYEARECAAYFAknWkIoACgkQE5f5cImnZrsYNACgtF/DVChiVxgVynrpAamgz8SM
> wB0An1zdYk7pB0MDxs78LPQHNiEWs5aO
> =D9zx

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 Apr 2009 15:55:02 -0700

Dan Wing wrote:
> use the same UDP tunnel for both of the TCP paths. 

If TCP multipath wanted to traverse a NAT and used TCP-AO, then each
TCP-AO connection would be UDP tunneled. Whether that implies a single
TCP-AO or not depends on how TCP multipath works.

Note that I raised numerous issues: NAT, security, firewalls, etc. It'd
be useful to address how TCP multipath interacts with each. Some
combinations are also useful to address, but not necessarily all.

> If my understanding is

That's a question for TCP multipath to answer - is that what you expect,
or not?

Joe

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 3 Apr 2009 16:09:51 -0700

> Sent: Friday, April 03, 2009 3:55 PM
> To: Dan Wing
> That's a question for TCP multipath to answer - is that what 
> you expect, or not?

I expect each TCP multipath could go over a different NAT,
as that would allow a host to connect to a WiFi access point 
(e.g., at a Starbucks) and an IPv6-only "4G"/"LTE" cellular
connection that may traverse a NAT64 to connect to an IPv4 
server.

-d