Subject: [multipathtcp] High-level design decisions /architecture

From: <[email-masked]>
Date: Mon, 2 Nov 2009 17:47:54 -0000

Hi

In order to help the process of reaching consensus on the high-level
design decisions /architecture, we have gone through the current set of
I-Ds to identify items that seem like they may fall into the category of
high-level design decisions - see below. Note: this list is not to meant
to imply our support or non-support of these items (except where our
charter requires our support!); the purpose is to identify what needs to
be discussed.

Please shout if there's something we misunderstood or missed.

Please let the WG know your opinions on these items - especially if your
disagree with any, have a potential alternative, or think something is
part of 'detailed design' and not 'high-level design'.

Thanks

Phil & Yoshifumi

Protocol-related

============

*	IPv4 & IPv6 will have the same high-level design 

	*	[Comment: the Charter implies both v4 & v6 are in scope]

*	The MPTCP connection is identified, from an apps perspective, by
the addresses associated with the original path (even if that subflow
/path closes) 
*	A SYN/ACK exchange (on a single path) checks that both ends
support MPTCP, with automatic fall-back to TCP if not 
*	each MPTCP subflow looks exactly like an ordinary, semantically
independent TCP flow. MPTCP effectively runs on top. So re-transmission,
FINs etc are independent on each subflow 
*	control of the MPTCP connection (as opposed to the subflows) is
carried in TCP options, eg DATA FIN for closing an MPTCP connection 
*	the protocol involves an MPTCP stack at both ends of the MPTCP
connection 

	*	[Comment: this is in our charter] 

*	Either end of the MPTCP connection can add or remove paths
to/from the MPTCP connection 

	*	[Comment / Question: this seemed a bit unclear in the
protocol i-d, but is presumably required?] 

*	Subflow end point definition

	*	[Question: Should we allow subflows in a single MPTCP
connection to have different ports?] 

Congestion algorithm

===============

*	the goals are: [1] MPTCP is at least as good as TCP would be on
the best subflow path; [2] on any path, an MPTCP connection uses less
(or the same) capacity as a TCP connection would do; [3] MPTCP moves
traffic off more congested paths and onto less congested ones 
*	increases in the congestion windows of the subflows are coupled;
for decreases in the congestion windows, each subflow does new reno
independently (ie they are not coupled) 
*	slow start, fast re-transmit and fast recovery are the same as
RFC5681 

API

===

*	no changes to existing apps are needed to use MPTCP; MPTCP uses
the unaltered TCP socket API 
*	There is an optional, extended API 

	*	[Question: would both ends need the extended API, with a
fall-back if not?] 
	*	[Comment: presumably the actual features of the extended
API are outside the initial high-level design decisions] 

*	congestion control state is shared among application-visible
transport instances (eg multiple HTTPs between the same pair of hosts) 

*	3 application profiles are mentioned (Bulk data transport,
Latency-sensitive transport with overflow, Latency-sensitive transport
with hot stand by) 

	*	[Question: how are these supported? Is a negotiation
mechanism needed?]            

Security

======

*	exchange a token in the initial SYN for the MPTCP connection,
and include the token when subflows /paths are added to the connection 

	*	[Question: Is the same token used whether the sender or
receiver (of the MPTCP connection) adds the subflow? Is the same
technique used for removing subflows? And closing connections, ie DATA
FIN?] 

*	security mechanisms will not interfere with end-to-end
authentication etc and will be compatible with legacy middleboxes 

Modularity

========

*         the architecture has 2 components, multipath scheduler & path
manager - meaning that improved /different modules of each can be
slotted in 

o        [Comment: it is unclear from the Architecture i-d how the
congestion control & protocol i-ds map to these components; some of the
text (strangely?) appears in a section about implementation]

o        [Question: what are the mechanisms for evolvability? How is it
negotiated which new module (for congestion control or path manager) to
use?]

o        [Question: this would seem to imply we need to define an
interface between the modules?]

*         there is a default path manager module, meaning it is the MUST
fall-back option                                                 

o        [Comment: the charter restricts us - the path manager we work
on will distinguish paths by multiple addresses]

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 02 Nov 2009 20:27:55 +0100

[email-masked] escribió:
>           o [Question: Is the same token used whether the sender or
>             receiver (of the MPTCP connection) adds the subflow? Is
>             the same technique used for removing subflows? And closing
>             connections, ie DATA FIN?]
>

I have been thinking about alternatives mechanisms that without 
requiring heavy crypto would remove the threats from passive on path 
attackers that can laucnhh attacks to the token based security (and 
require the attacker to actively change the traffic in order to launch 
the attack). I think this could be worth considering since the work 
required by the
I haven't been able to write an ID about that just yet, but i was hoping 
to do something in the few next weeks.

Regards, marcelo

>          o
> · the architecture has 2 components, multipath scheduler & path 
> manager – meaning that improved /different modules of each can be 
> slotted in
>
> o [Comment: it is unclear from the Architecture i-d how the congestion 
> control & protocol i-ds map to these components; some of the text 
> (strangely?) appears in a section about implementation]
>
> negotiated which new module (for congestion control or path manager) 
> to use?]
>
> o [Question: this would seem to imply we need to define an interface 
> between the modules?]
>
> · there is a default path manager module, meaning it is the MUST 
> fall-back option
>
> o [Comment: the charter restricts us - the path manager we work on 
> will distinguish paths by multiple addresses]
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp
>

---

From: William Herrin <[email-masked]>
Date: Mon, 2 Nov 2009 17:28:52 -0400

On Mon, Nov 2, 2009 at 1:47 PM,  <[email-masked]> wrote:

Hi Philip,

Apologies if any of this rehashes discussions you've already had.

> The MPTCP connection is identified, from an apps perspective, by the
> addresses associated with the original path (even if that subflow /path
> closes)

I'm concerned about this idea. One of the corner cases where shim6
falls apart is: once the connection migrates away from the original
addresses, how do the endpoints decide whether a new connection
request for the original addresses means the migrated addresses or the
current holder of the addresses? Think: FTP.

> A SYN/ACK exchange (on a single path) checks that both ends support MPTCP,
> with automatic fall-back to TCP if not

We have an awful lot of TCP options that are only valid in the
SYN-SYN/ACK exchange and only 40 bytes to put them all. What's wrong
with starting all TCP connections in non-MP mode and allowing the
connection's initiator to request MPTCP starting with any packet that
expects an ACK?

> each MPTCP subflow looks exactly like an ordinary, semantically independent
> TCP flow. MPTCP effectively runs on top. So re-transmission, FINs etc are
> independent on each subflow

I'm almost sold on the idea of subflows. Can you give me a run down on
the pros and cons, particularly the cons?

Thanks,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 02 Nov 2009 23:45:40 +0100

> current holder of the addresses? Think: FTP.

could you expand on this?
I wasn't aware shim6 falls apart in any case neither this particular 
approach...

Regards, marcelo

---

From: William Herrin <[email-masked]>
Date: Mon, 2 Nov 2009 19:38:46 -0400

> I wasn't aware shim6 falls apart in any case neither this particular

Hi Marcelo,

Consider the following scenario:

Host A comes online with 1.2.3.4.
Host A requests a new TCP connection to 5.6.7.8.
Established with host B at 5.6.7.8.
Host B adds 8.7.6.5
Host B removes 5.6.7.8
Host C comes online with 5.6.7.8.
Host A requests a new TCP connection to 5.6.7.8.

Which host does A connect to?

If the answer is C, protocols like FTP which initiate multiple TCP
connections to the same IP address fail once an endpoint gives up its
original address. That'll also impact web browsers which employ dns
pinning.

If the answer is B, I can hijack hotmail's IPs with respect to your
mail server for the few fractions of a second it takes to establish
and move a TCP connection over to an IP address I legitimately
control. You'll then indefinitely send subsequent hotmail-addressed
email to me. If I diligently pass it on to hotmail, you may never even
know.

If we identify the connection by the original IP address even after
that address is gone, effectively lying to the application, we can
induce it to incorrect behavior. That can have security implications
that may be hard to get a handle on.

Regards,
Bill Herrin

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Nov 2009 15:44:32 -0800

William Herrin wrote:
...
> that may be hard to get a handle on.

This is the reason I raised the issue of "as goes the original IP
address, so goes the connection" during the Stockholm meeting.

I can see how to make things work when this is true, basically by
authenticating based on this address pair rather than the address pair
in the packet. If not, as noted above, there's an opportunity for
hijacking (if malicious), or at least unintended behavior (if not).

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Nov 2009 16:23:58 -0800

I'll jump in here ;-)

[email-masked] wrote:
...
>           o [Comment: the Charter implies both v4 & v6 are in scope]

Definitely.

>       the addresses associated with the original path

Yes, IMO.

> (even if that
>       subflow /path closes)

Yes, although, as noted, if the IP address goes away altogether, then
IMO the connection needs to be terminated.

>       support MPTCP, with automatic fall-back to TCP if not

Yes, with as few bytes as possible.

>       independent TCP flow. MPTCP effectively runs on top. So
>       re-transmission, FINs etc are independent on each subflow

Don't care. Seems to make sense for middleboxes that track such things,
but IMO they shouldn't care either, esp. since the packets could be
encrypted.

>       carried in TCP options, eg DATA FIN for closing an MPTCP connection

I disagree. The option stream is neither reliable or ordered, and so
it's a bad idea to put state management there. If you want a control
channel, open a separate connection IMO.

>           o [Comment: this is in our charter]

Agreed (as per the charter), but seriously this begs the question of
what is "TCP" about the result. I don't like the idea of doing this all
inside TCP solely as NAT camouflage.

>     * Either end of the MPTCP connection can add or remove paths to/from
>       the MPTCP connection
>           o [Comment / Question: this seemed a bit unclear in the

Depends on when the paths are created.

If they are created at SYN time, then it's OK IMO to develop support
asymmetrically.

If they can be created once open, AFAICT TCP is memoryless w.r.t. who
opened the connection, so I can't see why either side would have
privileged rights (or could even tell).

>           o [Question: Should we allow subflows in a single MPTCP

Seems like you'd have to - or you have to coordinate port use on
different addresses at a single endpoint constantly.

>       the best subflow path; [2] on any path, an MPTCP connection uses
>       less (or the same) capacity as a TCP connection would do; [3]
>       MPTCP moves traffic off more congested paths and onto less
>       congested ones

These sound OK, though it might be useful to consider that [1] and [2]
are requirements (IMO), where [3] is an optimization goal, and thus OK
to break.

>       for decreases in the congestion windows, each subflow does new
>       reno independently (ie they are not coupled)

I would assume that both increases and decreases are not coupled. If
they are, it seems like a subflow could use bandwidth on one path to
justify being more aggressive on another, which seems to violate [2] above.

>     * slow start, fast re-transmit and fast recovery are the same as

Yes.

>       the unaltered TCP socket API

I would say this in a way that flows into the next idea below better,
e.g., "no changes are required to use MPTCP in its simplest mode".

>           o [Question: would both ends need the extended API, with a
>             fall-back if not?]

Seems like one end could want the extended API for explicit control of
flows, but the other end doesn't need to care.

>           o [Comment: presumably the actual features of the extended API
>             are outside the initial high-level design decisions]

IMO (RFC2140, e.g.), yes ;-)

>       Latency-sensitive transport with overflow, Latency-sensitive
>       transport with hot stand by)
>           o [Question: how are these supported? Is a negotiation
>             mechanism needed?]      

Seems at odds with the claims above for [1],[2],[3]. I.e., bulk
transport seems compatible with all three, but latency-sensitive seems
like it might not be compatible with any of these.

> Security
> 
> ======
> 
>     * exchange a token in the initial SYN for the MPTCP connection, and
>       include the token when subflows /paths are added to the connection
>             receiver (of the MPTCP connection) adds the subflow? Is the
>             same technique used for removing subflows? And closing

Let's please not call this "security". Call it a flow identifier, but
it's trivial for on-path attackers to see, and anything small enough to
be appropriate for a SYN is likely to be too small to prevent exhaustion
attacks.

>       authentication etc 

Yes. Specifically IMO there ought to be a TCP-AO mode supported - e.g.,
using socketpair of the original connection for the pseudoheader used
for MAC calculation.

(i.e., where this mentions "security mechanisms" I think of TCP-AO, not
a nonce)

> and will be compatible with legacy middleboxes

Well, legacy middleboxes do all sorts of nasty things - many of which
are more correctly considered attacks - so this can't be a requirement
as worded.

E.g., some middleboxes remove unknown options. That's clearly going to
break things here. Others change packet boundaries (splitting, joining,
etc.). Others rewrite all sorts of header fields.

IMO, be more specific about what you would like to support (e.g., simple
NATs), and make this a goal, not a requirement.

> manager – meaning that improved /different modules of each can be

This seems to confuse architecture with implementation. Although they
are specified separately, an implementation might conflate the two to
the point where it cannot be modularly modified. I don't know what
you're going for here, but IMO just state that they're independently
*specified*.

> text (strangely?) appears in a section about implementation]

That might be a place where the implementation cannot reasonably
separate the two specification issues.

> negotiated which new module (for congestion control or path manager) to

This seems like an independent question, i.e., how are versions of the
protocol identified and established for each connection?

> interface between the modules?]

Sounds suspiciously like an API - which I think is needed, and is a good
idea, but you ought to get buy-in on this, since it's intraprotocol, and
could be easily confused with an implementation issue.

It'd be important to make sure this is an architecture interface, not an
implementation one.

> on will distinguish paths by multiple addresses]

See above - version issue.

Joe

---

From: Mark Handley <[email-masked]>
Date: Tue, 3 Nov 2009 00:40:34 +0000

> Which host does A connect to?

I'm not sure I see the problem.

When host B removes 5.6.7.8, that address is removed from that
connection only.  It has no relation with any other connections,
ongoing or future.  If A has any other connections with 5.6.7.8, they
are unaffected (after all, 5.6.7.8 may be a NAT with several hosts
behind it) unless they also explicitly remove the address.

When A requests a new TCP connection to 5.6.7.8, that's what A's app
specified at that moment in the connect call, and that's the address
the SYN will be sent to.  No previous behaviour for any other
connection affects that.

 - Mark

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Nov 2009 16:56:19 -0800

> the SYN will be sent to.  No previous behaviour for any other

How do you know whether 5.6.7.8 is the same place as the first
connection, or a new place? Don't you need to figure that out to know
whether you end up with new capacity, or need to share capacity?

Joe

---

From: Lars Eggert <[email-masked]>
Date: Mon, 2 Nov 2009 20:07:28 -0800

Hi,

> whether you end up with new capacity, or need to share capacity?

I don't think you care whether it's the same location or not - the  
linked congestion control proposal doesn't infer topology information  
from IP addresses.

Lars

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 03 Nov 2009 16:54:24 +0900 (JST)

From: William Herrin <[email-masked]>
Subject: Re: [multipathtcp] High-level design decisions /architecture
Date: Mon, 2 Nov 2009 19:38:46 -0400
Message-ID: <[email-masked]>

 > original address. That'll also impact web browsers which employ dns

In case of ftp, the choice would be something like this?

 1: we don't support this case. ftp will fail.
 2: ftp client/server performs getsockname() before port/pasv to check its own active IP address.
    (But, we need to define getsockname() should return active IP address instead 
     of original IP)
 3: Develop new ftp for MPTCP (using new API etc..)

I personally prefer 1.. then might do 3 later..
In case of dns pinning, I don't see any issues.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 03 Nov 2009 10:42:36 +0100

I am splitting the answer in the shim6 part and the mptcp part, so that 
people not interested in shim6 (which i guess are plenty since this is 
the mptcp ml, can simply drop this email)

> Which host does A connect to?

first of all Shim6, as one may guess from the acronym :-) only works for 
IPv6 identifiers.

So, non of them apply to shim6

Second, due to large space of IPv6 iid, the situation you describe there 
is very unlikely to happen by a coincidence.
It may be due to an attack, where the attacker is host C, agree?
 
assuming that above were IPv6 addresses, the host you actually connect 
to depends whether you are using forking or not.
But in general, you will be connecting to host B, as long as the shim6 
state is on host A

> email to me. If I diligently pass it on to hotmail, you may never even
No, In order to do that you would have had to generate an HBA for that 
address, which in computationally impossible in a reasonable timer frame.
So, no hijacking attacks in shim6

If you do think there is the possibility of a hijacking attack, please 
describe it.

So, i don't see shim6 breaking in this case...

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 03 Nov 2009 10:45:07 +0100

> Host A requests a new TCP connection to 5.6.7.8.
i don't see how this applies to MPTCP, since MPTCP as oposed to any L3 
solution works per connection.

So in the MPTCP case, the original connection would still be with host B 
and the new connection would be with host C, which is the same baheviour 
that you would have with normal TCP (only that with normal TCP the first 
connection would have break while with MPTCP is still preserved)

So i don't see this as a problem, but rather as a feature... what am i 
missing?

Regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 03 Nov 2009 10:48:19 +0100

> using socketpair of the original connection for the pseudoheader used
It would make sense to contrast the protection provided by TCP-AO witht 
he threats identified in the threat analysis and see how many of these 
it solves.

Regards, marcelo

---

From: Scott Brim <[email-masked]>
Date: Tue, 03 Nov 2009 10:00:21 -0500

> current holder of the addresses? Think: FTP.

An e2e security mechanism / token that identifies the flow.  See the
security bullet.

But there are corner cases where it is extremely difficult to establish
a stable src/dst address pair.  In particular if BGP and NAT are set up
such that you contact me on one address, I reply on my other address ...
and you do the same on your end.  Since NAT is now architecture, it is
important that initial setup be as independent from addresses as
possible.  I believe we're essentially there but we have to prove it.

> connection's initiator to request MPTCP starting with any packet that

See previous paragraph.

> the pros and cons, particularly the cons?

My biggest con is if your destination is multiply connected behind two
non-synchronized intrusion prevention devices.

Scott

---

From: Scott Brim <[email-masked]>
Date: Tue, 03 Nov 2009 10:18:00 -0500

[email-masked] allegedly wrote on 11/02/2009 12:47 PM:
>       the addresses associated with the original path (even if that
>       subflow /path closes)

... for a legacy API.  An up-to-date API, with extensions, MUST have a
way of identifying the connection which is independent of addresses.

>     * A SYN/ ACK exchange (on a single path) checks that both ends

retransmission must be sub-flow specific for the moment but we know that
can slow things down so we should actively investigate ways of
retransmitting on any sub-flow.

>             protocol i-d, but is presumably required?]

Yes, the protocol must support this.  However, as a meta-issue ... I
suggest that "path" is good for marketing but not technically exact.  An
endpoint can even have multiple addresses associated with a single
interface.  Both of them can.  We should figure out how to restrict the
use of "path" to contexts where its use is accurate.

>             connection to have different ports?]

Don't know.

>       the best subflow path; 

This is a hard one.  One use for multipath transport is in environments
where the network situation is (ahem) "dynamic", so it is difficult to
maintain a connection using a single src/dst address pair.  In this
case, if there are problems then multipath transport is certainly
better, but if one of the paths is clean, then just using that path for
a simple TCP gets better goodput then MPTCP would, trying to use that
path plus some poor ones in addition.

Let's try to find some better wording at the meeting.

>       [2] on any path, an MPTCP connection uses
>           o [Question: would both ends need the extended API , with a
>             fall-back if not?]

Agree with all.

>       transport instances (eg multiple HTTPs between the same pair of

How do you arrange for this?  Currently, if a single app has multiple
HTTPs running, each one has independent congestion control state.  HTTP
in this case is the "client layer" of transport, so each HTTP would be
an independent client, each getting its own independent MPTCP state.  No?

>       authentication etc and will be compatible with legacy middleboxes

The last bullet is difficult because some legacy middleboxes will not
allow unknown TCP options.

I don't know anything about modularity :-)

Thanks ... Scott

> ·          there is a default path manager module, meaning it is the
> MUST fall-back option                                                
>

---

From: Scott Brim <[email-masked]>
Date: Tue, 03 Nov 2009 10:28:00 -0500

> IMO the connection needs to be terminated.

Joe, that's just not necessary except for legacy endpoint stacks.  See
the iPhone and "connectByName".  If the API revolves around a connection
ID, regardless of underlying addresses, then IP addresses can come and
go as they with as long as the transport layer has some mechanism of
maintaining session authenticity and continuity.  The scenario where a
new node gets an IP address that an old node was just using is only
relevant if the connection is maintained using IP addresses ... which it
isn't.

> inside TCP solely as NAT camouflage.

Why not?  NAT is now architecture, and we have to deal with it.  You
could just as easily say we should be able to run lots of protocols over
the Internet but we have to put things in IP as "router camouflage".

> flows, but the other end doesn't need to care.

Agree.  The API is to access controls for the endpoint's implementation
of the protocol.  How an endpoint implements its APIs is independent of
the protocol.

Scott

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 07:55:28 -0800

> relevant if the connection is maintained using IP addresses ... which it

This is fine if you're defining a new transport protocol. TCP *defines*
a connection by the socket pair. If that socket pair disappears and you
keep the connection up, I would argue you need to use a different
protocol number because you've seriously changed the semantics.

> the Internet but we have to put things in IP as "router camouflage".

Inside TCP, vs. inside SCTP or a new transport - one that would interact
poorly with NATs. IP is more than a way to interact with routers; it's a
uniform end-to-end multiplexing unit, it allows e2e transport security
while allowing router interaction, etc.

NATs may be ubiquitous, but they are NOT part of the 'architecture'.
They violate what little of the architecture is specified (RFCs 1122,
1123, 1812).

Architecture is deliberate, not de-facto; the latter applies to
implementations.

> of the protocol.  How an endpoint implements its APIs is independent of

Whoa on two counts.

First, *IF*, as you say, "NAT is now architecture", then the API affects
nodes inside the network, not just at the endpoints.

Second, the API is two things - an abstract specification and an
implementation. Consider TCP, which has OPEN, CLOSE, STATUS, READ,
WRITE. Those are part of the *specification*. There's also the Un*x
"open()" call, which implements part of what OPEN specifies (arguably,
in Un*x, this also includes listen(), connect(), and bind()).

My claim here is that the API can be asymmetric, i.e., extended on one
side and not the other. That's just claiming that all extensions are
optional on both sides, and not coordinated.

Joe

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 08:16:24 -0800

> linked congestion control proposal doesn't infer topology information

That's news to me. Seems like this whole thing assumes topology
information from IP addresses.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 08:16:43 -0800

> he threats identified in the threat analysis and see how many of these

Agreed. Same for the nonce.

Joe

---

From: Costin Raiciu <[email-masked]>
Date: Tue, 3 Nov 2009 16:24:59 +0000

Hi Joe,

The proposed congestion controller gives the aggregate multipath  
connection the throughput a TCP would get on the best (fastest) of  
the  paths available (to the multipath connection). It does so only  
by looking at loss rates and rtts on the different paths. It does not  
need to know whether there is a shared bottleneck or not.

Cheers
Costin

On 3 Nov 2009, at 16:16, Joe Touch wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Lars Eggert wrote:
>> Hi,
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
>
> iEYEARECAAYFAkrwV1gACgkQE5f5cImnZrsdhwCgpGpXMJGcK3WgHIP5XC6JD6GE
> IggAn2xfi6jqoZ6RUTaUll4TDjGMS4js
> =pDxD
> -----END PGP SIGNATURE-----

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 08:27:59 -0800

Costin Raiciu wrote:
> connection the throughput a TCP would get on the best (fastest) of the 
> paths available (to the multipath connection). It does so only by
> looking at loss rates and rtts on the different paths. It does not need
> to know whether there is a shared bottleneck or not.

If it doesn't, then you will create the potential for a single multipath
group to fight with itself, which seems less than productive.

Joe

---

From: Mark Handley <[email-masked]>
Date: Tue, 3 Nov 2009 16:43:43 +0000

> group to fight with itself, which seems less than productive.

Well, so long as the total window increase per RTT of all the subflows
in a connection is no more than a single vanilla TCP would get, then
you don't fight with yourself any more than one TCP would ("fighting"
in this context is pretty much equivalent to how much unacked traffic
per RTT a flow can inject - the ack-clocked traffic already had its
space last RTT, so isn't "fighting" for extra space).

This is exactly how the proposed congestion control algorithm works
through a common bottleneck.

 - Mark

---

From: Costin Raiciu <[email-masked]>
Date: Tue, 3 Nov 2009 16:44:23 +0000

I don't understand what you mean by subflows fighting each other.

look at the examples in the picture attached .

Say for simplicity that the RTT of the multipath subflows is the same  
in both cases, and say link 1, link 2 and link 3 have the same loss  
rate p (i.e. disregard flows B,C and E).

The controller will give the **same** throughput to the multipath  
connection in the two cases, i.e. what a TCP would get if it saw loss  
rate p.

In this case, there is no throughput benefit for using multipath;  
multipath increases throughput when it can affect the loss rates. In  
the same picture, if the flows B,C and E have the same throughput as  
the multipath subflows, the connection will get:
1. (link1 + link2)/3 in the first case
2. link3 / 2 in the second case.

Hope this makes things clearer.

Cheers,
Costin

>> the

---

From: Costin Raiciu <[email-masked]>
Date: Tue, 3 Nov 2009 17:14:30 +0000

Just adding a few explanations to my previous email (which is a bit  
compressed).

> Say for simplicity that the RTT of the multipath subflows is the  
> same in both cases, and say link 1, link 2 and link 3 have the same  
> loss rate p (i.e. disregard flows B,C and E).
>
> connection in the two cases, i.e. what a TCP would get if it saw  
> loss rate p.
>
I mean, in the case when multipath cannot affect the loss rates at all.

> multipath increases throughput when it can affect the loss rates.  
> In the same picture, if the flows B,C and E have the same  
> throughput as the multipath subflows, the connection will get:
This assumes link1 and link2 have the same capacity.

> <Picture 1.png>

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 3 Nov 2009 18:35:40 -0000

Hi Phil, all,

Some comments inline.

Thing is, there are quite a large scope of topics ranging from
"architecture" to "high-level design decisions". It is quite feasible to
have fixed architectural components while high-level design can be
flexible. This not necessarily the same as "detailed design"!

From: [email-masked]
[mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: 02 November 2009 17:48
To: [email-masked]
Subject: [multipathtcp] High-level design decisions /architecture

Hi

Thanks

Phil & Yoshifumi

Protocol-related

============

 

[Alan: Agreed]

[Alan: This is still a contentious issue. If the new MPTCP API is in
use, this issue - as a concept - does not really exist. There is a
connection identifier, there are sets of addresses, etc. This is talking
about the legacy case, and what happens if an application later queries
a socket (getsockname and getpeername). I can see Bill's and Joe's
arguments here of where problems could occur. I think, however, that the
problems that may occur with returning the original addresses are less
than any other way.]

[Alan: Yes, I think this is an important high-level decision]

[Alan: Yes, absolutely. Adding paths and signalling new addresses are
also potentially separate to permit flexibility with
middleboxes/firewalls/etc.]

[Alan: Yes. By default I would expect implementations to go for the same
ports so as to maximise the chances of getting through
firewall/middleboxes, but there is no technical reason why it could not
be used between any ports. The draft specifies the mapping that an
endpoint must have.]

Congestion algorithm

===============

 

API

===

[Alan: No, the extended API is about signalling preferences to, and
getting information from, the local implementation. If it did involve
anything happening on the wire, this would simply be triggering the
operation of existing features.]

[Alan: Yes, although the use cases that drive the API decisions may also
help at a high level.]

[Alan: Huh?]

[Alan: TBD ;-)  The sender can control much of this; expressing policy
to/from the receiver is one of the key open issues.]

Security

======

[Alan: in the current design (very much open to input of course), the
sender and the receiver both have independent, locally unique tokens.
Aside from the initial handshake, where the local token is used, all
future token uses (currently limited to opening new subflows) are the
remote token. Removing subflows is just a FIN on the subflow, or
signalled via options if the address is lost. Closing connections also
does not use tokens.

We are clear that this is a very minimal handshake mechanism, simply
seeking to be no worse than regular TCP. More secure mechanisms are
FFS.]

 

Modularity

========

[Alan: This seems to be causing some concern with people as to whether
this belongs in an "architecture" document. Personally, I see it as part
of the "framework" through which the relevant components of any
Multipath TCP could be implemented. It is a way of thinking about the
functional separation required to allow modularity.]

[Alan: Will be ironed out in due course - although the congestion
control clearly lives in the scheduler, there are also some protocol
components there.]

[Alan: I would have thought that kind of decision is entirely out of
scope for a high-level framework, and even for detailed API decisions.
This is intended entirely as implementation guidance and was not written
with the intent for people to plug-and-play different modules.]

[Alan: Sure, but that doesn't prevent us designing an evolvable
framework so that different path managers could be used with much of the
rest of the protocol having minimal changes.]

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 3 Nov 2009 18:37:56 -0000

Hi Joe, all,

> channel, open a separate connection IMO.

The current proposal contains quite a lot to meet the limitations of options. Specifically: Data Sequence Mapping is done so that a data-level sequence number does not have to appear exactly at the correct point; address management has no requirement to be reliable or in-order; and DATA FIN is applied to a FIN only.

> privileged rights (or could even tell).

Yes, indeed (from the TCP level). Plus, the idea of who opens the subflow (i.e. sends the SYN) may not be the same endpoint who has requested this to happen (see address management stuff). 

> different addresses at a single endpoint constantly.

Yes, and we have a demuxing value (the token) anyway. However, for optimality it is proposed that the same ports are used on the same (and maybe different) addresses per connection.

> are requirements (IMO), where [3] is an optimization goal, and thus OK

Interesting point - a narrow distinction but a valid one. There is certain linkage between [2] and [3], however, in that while it may be okay to break [3] you can't do it to the detriment of other users ([2]).

> e.g., "no changes are required to use MPTCP in its simplest mode".

Agreed.

> flows, but the other end doesn't need to care.

Yep.

> like it might not be compatible with any of these.

Latency-sensitive would still be compatible with [1] and [2] but is not so concerned with congestion sharing. Having said that, it would be likely that the least congested path is optimal for most cases of latency-sensitive transport!

Regards,
Alan

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 3 Nov 2009 18:37:55 -0000

Hi Bill, all,

> current holder of the addresses? Think: FTP.

This is indeed an issue. However, what is likely to happen here is that
the next connection will just be rejected, the application will fail and
have to try again from scratch, this time starting with a valid
connection.

The bigger issue here is what security impacts this may have. I find it
hard to see how an attacker could engineer a compromised situation out
of this. There may be some end users receiving odd connection attempts
due to this, but that case can occur today with DNS caching.

> connection's initiator to request MPTCP starting with any packet that

We had began with assuming that a connection was either MPTCP or regular
TCP from the start, so endpoints would know whether they could make use
of Multipath immediately and start more subflows immediately.

I'm intrigued with your suggestion, though. I see your point that there
may not be any critical reason why this information should be known at
the beginning of a session. I can't immediately think of any technical
reason why doing it later in a session wouldn't work. But it would need
further thought.

Regards,
Alan

---

From: Scott Brim <[email-masked]>
Date: Tue, 03 Nov 2009 13:41:56 -0500

> protocol number because you've seriously changed the semantics.

MPTCP "looks like TCP on the wire" for each src/dst pair.  That means it
uses the same protocol number, but it doesn't mean it must behave
exactly like TCP.  We already change the response to packet
loss.  I suspect that you could do the whole TCP state machine for each
src/dst pair, but in the new API

> They violate what little of the architecture is specified (RFCs 1122,

Too late, it's the new millennium.  Dave Clark said "architect the
inevitable" (one of my favorite sayings).

> nodes inside the network, not just at the endpoints.

I don't understand this.  Please help.  I see no linkage between a
transport API in an endpoint and a NAT box in the path.

> optional on both sides, and not coordinated.

Yes, I was trying to agree :-).

Scott

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 03 Nov 2009 19:44:15 +0100

>> he threats identified in the threat analysis and see how many of these
mmm, can you think of someone that could be willing to provide the text 
of such analysis?

>  Same for the nonce.
That is already covered in the current threat analysis (see 
http://tools.ietf.org/html/draft-bagnulo-mptcp-threat-00)
i.e. the current threat analysis threats are analyzed for both a MPTCP 
that does not use any form of security and one based on tokens.
Since AO seems standard TCP protection, i think it make sense to extend 
the analysis to cover that

Regards, marcelo

> iEYEARECAAYFAkrwV2sACgkQE5f5cImnZrt0iwCeKyrt4rYoszR1MwS60Y5IkS0y
> FXIAmwUg6K05KwRc26wBPWLxTVM40l9p
> =Dgpo

---

From: William Herrin <[email-masked]>
Date: Tue, 3 Nov 2009 15:28:26 -0400

> non-synchronized intrusion prevention devices.

Hi Scott,

I thought that was one of the pros of subflows? Since each subflow is
its own 2-way TCP connection between two specific addresses they can
interact with middleboxes independently without requiring the
middleboxes to talk to each other?

Or am I missing something?

On Tue, Nov 3, 2009 at 11:55 AM, Joe Touch <[email-masked]> wrote:
>> maintaining session authenticity and continuity.
>

Hi Joe,

I'll stake out a middle ground on this one.

On the one hand, this entire exercise is academic if we don't keep the
connection up when the original address pair fails. The high-level
purpose of something like MPTCP is to break TCP's dependence on IP
address stability. So, you're right: we've seriously changed the
semantics of the TCP connection.

On the other hand, the vast majority of TCP applications won't care
about the change in semantics. If it's possible to improve these apps'
capability without requiring the developers to touch them again,
that's a plus that's hard to argue against.

If you accept the preceding two points, it leaves an open question:
what do we do about the apps which *do* depend on the non-MP
semantics? My druthers here are: first and foremost be honest, and
second: do what I say; don't try to guess what I mean.

Where the old API can't accurately present what's going on, it should
offer a "no information" response, not an approximate response.

On Tue, Nov 3, 2009 at 3:54 AM, Yoshifumi Nishida
<[email-masked]> wrote:
>  > I'm concerned about this idea.
>  2: ftp client/server performs getsockname() before port/pasv
>     to check its own active IP address. (But, we need to
>     define getsockname() should return active IP address instead
>     of original IP)

Hi Yoshifumi,

I tend to think that has to be the result as well. But if so then we
should avoid offering faulty information via the APIs. That is, don't
identify the connection by it's original IP address pair. If an
old-api application cares what the address pair is but has been
incompatibly instructed to use MPTCP, help it fail quickly rather than
giving it a stochastic success rate.

This means that explicitly binding the local address should implicitly
turn MPTCP off. It also means calls like getpeername and getsockname
should return an error or zeroed values when MPTCP is enabled so that
we explicitly notify the application that we can't provide accurate
information.

> In case of dns pinning, I don't see any issues.

DNS pinning means that the browser will indefinitely refuse to try a
different IP address for new TCP connections to a server that it's
running javascript for. This means that although the individual
connections will succeed on the changed addresses, the web session
overall will still fail.

Not TCP's problem strictly speaking, but if we want MPTCP to be useful
at a practical level we should consider how the dozen most common
application categories (like browsers using HTTP and mail servers
using SMTP) will interact with it.

On Tue, Nov 3, 2009 at 2:37 PM, Ford, Alan <[email-masked]> wrote:
>> What's wrong

Hi Alan,

My druthers would be for an API and/or system tunable delay before
MPTCP starts up so that very short lived TCP connections don't suffer
any new overhead.

The only downside that jumps out at me is that if a connection is
eligible for MPTCP we can't report to the application that the
connection definitely isn't MPTCP.

> [Alan: Yes, I think this is an important high-level decision]

Whoops, missed this one on the first read through.

Retransmit is a problem. If retransmit is strictly independent then
the unexpected collapse of any subflow kills the entire connection.
That reduces overall TCP reliability to the product of each path's
reliability.

Even if we don't withdraw a segment from a subflow once it has been
assigned, we need to be able to send another copy in a different
subflow should the first subflow stall. Whole can of worms waiting for
us there.

Regards,
Bill Herrin

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 11:34:28 -0800

Ford, Alan wrote:
...
> Latency-sensitive would still be compatible with [1] and [2] but is
> not so concerned with congestion sharing. Having said that, it would
> be likely that the least congested path is optimal for most cases of
> latency-sensitive transport!

That depends on whether you're dominated by propagation, queuing, or
capacity. If you have long message, the last two are more important. If
you have short messages, the first two are. Congestion affects only
queuing, and can be swamped by capacity or propagation.

I.e., it's not a simple decision.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 11:47:47 -0800

> src/dst pair, but in the new API

A protocol is all three of:

	packets on the wire
	state at the endpoints
	rules for transitions between the state and packets

Some changes to these argue for keeping the same protocol number, others
do not. Merely keeping the wire packets the same isn't sufficient to
argue that it's the same protocol, IMO.

> inevitable" (one of my favorite sayings).

Then we ought to all go home. My point is that we can try to play with
some cleanly defined subset of NATs, but expecting everything to be
NAT-compatible is nonsensical.

> transport API in an endpoint and a NAT box in the path.

Well, you said we had to live with NATs. NATs think that:
	a) all connections are initiated from behind a NAT,
	never into a NAT from the public side (excepting DMZ)

	b) they can modify options at will

	c) they can see the transport header at all
	(so if NATs are part of the architecture, IPsec isn't)

If the transport API uses a nonce to identify connections, then the NAT
needs to know about the nonce, e.g., to know whether the packets are
from one connection or another (since others claim we can reuse IP
addresses and the original connection doesn't need to go down).
Otherwise, a FIN from one connection could tear down an association to
another.

But you said that NATs are part of the architecture, and they don't
understand nonces, so how can you even proceed?

(I'm suggesting that LIMITED NAT support is OK, but declaring full NAT
support on all connections is good way to kill this and every other
transport)

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 04 Nov 2009 05:39:53 +0900 (JST)

From: "Ford, Alan" <[email-masked]>
Subject: Re: [multipathtcp] High-level design decisions /architecture
Date: Tue, 3 Nov 2009 18:37:56 -0000
Message-ID: <[email-masked]>

 > Yes, and we have a demuxing value (the token) anyway. However, for optimality it is proposed that the same ports are used on the same (and maybe different) addresses per connection.

Then, we might need to update Add Address option in the draft to propagate port number.
and the term Address ID might be a bit confusing.

If we think about the possibility that application can get peer's secondary IP address
outside of MPTCP (e.g. from DNS) and wants to try it, it might be convenient to 
use one single port for all subflows? 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 13:20:37 -0800

Ford, Alan wrote:
...
> reason why doing it later in a session wouldn't work. But it would need

TCP negotiates all options during the SYN - that's the only 'handshake'
there is. TCP does not define option negotiation after connection
establishment.

I.e., you can decide to start the separate connections later, but you
need to negotiate support for the option during the SYN.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 13:22:54 -0800

> I don't understand what you mean by subflows fighting each other.

I was thinking of D1 fighting D2. If their probes are coordinated and
desych'd, I understand that they won't fight, though.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Nov 2009 13:26:46 -0800

> space last RTT, so isn't "fighting" for extra space).

Fighting could occur if the subflows send packets at the same time, even
if the aggregate is the same as a single TCP would have sent. A single
TCP never sends two packets out at once (it serializes them), but
separate TCPs could easily do this (even if the OS serializes them, they
could go out simultaneously from different NICs).

Keeping in mind how TCPs like to sync, this sounds like an area to keep
an eye on.

Joe

---

From: Mark Handley <[email-masked]>
Date: Tue, 3 Nov 2009 21:45:28 +0000

> Keeping in mind how TCPs like to sync, this sounds like an area to keep

Good point.  I'd hope that buffers at the queue feeding the common
bottleneck would cope with this, but it's possible it makes the
behaviour of the aggregate at the bottleneck more bursty.  We've not
seen any evidence for this, but it's not something we've explicily
looked for.  Regular TCP is pretty busty though, sending a lot of
back-to-back packets unless explicit spacing is done, so I doubt
MP-TCP would be any worse in most topologies.  Still, definitely
another thing to look out for for in our experiments.

 - Mark

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 3 Nov 2009 22:03:56 -0000

Hi Bill, all,

> Or am I missing something?

I think Scott's concern is with an IDS that may look at traffic which
does not behave maliciously until it is spread across multiple subflows,
thus requiring the IDS to do some correlation between (what it sees as)
multiple traffic flows. But I'm sure Scott can be more specific...

> that's a plus that's hard to argue against.

I'd definitely agree with those two points.

> offer a "no information" response, not an approximate response.
[snip]
> we explicitly notify the application that we can't provide accurate

I agree with the principle behind all of that. However, I do have a
concern that recommending APIs return a null value in such a case may
itself be overloading an error signal to an application. I would hope
that applications should be able to cope with such a return value, but
it is unlikely that all will.

It may be the lesser of all evils, however.

> MPTCP starts up so that very short lived TCP connections don't suffer

That's a very good point, actually. One of my long-standing concerns has
been how to best cope with applications that only need short-lived
sessions, and don't have the extended API to express it.

Another alternative could be a system-wide threshold for traffic before
kicking in multipath.

> connection definitely isn't MPTCP.

Yes. And it also takes the initial decision to use MPTCP away from the
initiating endpoint (not necessarily a bad idea, just something that
changes what one may expect a transport protocol's behaviour to be).

> That reduces overall TCP reliability to the product of each path's

Ah, I may have read that point wrong.

What we have in the design as it stands is that, once a segment of data
is transmitted on a subflow, that segment of subflow sequence space must
only be used for that data. That does not, however, preclude also
sending the same data on another subflow. The data-level sequence
numbering will permit correct reassembly at the receiver, and this will
ensure reliability and survivability over failure.

> subflow should the first subflow stall. Whole can of worms waiting for

Ah, well, that's kinda what I was describing above. What does your can
of worms contain?

Regards,
Alan

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 3 Nov 2009 22:12:22 -0000

Hi Yoshifumi, all, 

>  > Yes, and we have a demuxing value (the token) anyway. However, for
> optimality it is proposed that the same ports are used on the same
(and maybe
> different) addresses per connection.
> 
> Then, we might need to update Add Address option in the draft to
propagate
> port number.

I don't see that we do. The general principle is that an endpoint should
use the ports that are currently in use at the other endpoint on its new
connections, to maximise getting through firewalls and middleboxes, and
also to support any port-based QoS traffic engineering, etc. That
doesn't preclude an endpoint from trying other port numbers with the
same token, but the chances of that working would seem slim. I'm not
entirely sure whether we want to explicitly permit or explicitly forbid
that, yet.

Unless, there is some other use for signalling alternative port numbers
to use. The only case I can think of may be for tricking load balancers
with the same address to route on alternative paths, but that seems
tenuous at best.

> If we think about the possibility that application can get peer's
secondary
> IP address
> outside of MPTCP (e.g. from DNS) and wants to try it, it might be
convenient
> to

The idea of using additional addresses from DNS etc is an interesting
one. I'm reluctant to suggest this in the protocol - it would seem to
open a number of potential security issues.

One thing that adds a very elementary level of security in the current
design is that signalling of new subflows occurs both over the existing
connection and separately. This is not ideal yet and needs refinement.
Such a solution would not be possible at all with the out-of-band
knowledge exchange.

Besides, an endpoint that knows it has multiple addresses should signal
that immediately if it wants them to be used. (And if it doesn't want
its other endpoint to be used, this gives it the flexibility not to
signal it).

Regards,
Alan

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 04 Nov 2009 19:57:34 +0900 (JST)

Hello Alan, 
Thanks for your comments. 

From: "Ford, Alan" <[email-masked]>
Subject: RE: [multipathtcp] High-level design decisions /architecture
Date: Tue, 3 Nov 2009 22:12:22 -0000
Message-ID: <[email-masked]>

 > entirely sure whether we want to explicitly permit or explicitly forbid

OK. Thanks for the clarification. If using different port is very limited use
I guess it's not a major thing.

From: "Ford, Alan" <[email-masked]>
Subject: Re: [multipathtcp] High-level design decisions /architecture
Date: Tue, 3 Nov 2009 18:35:40 -0000
Message-ID: <[email-masked]>

 > to/from the receiver is one of the key open issues.]

Hmm.. Suppose if you have a PC with two NICs and want to use one NIC only for back-up. 
When you want to download something from a server, how do you specify the above requirment? 
All I can think is signaling mechanism.. Can we leave this as an open issues?

 > with the intent for people to plug-and-play different modules.]

I somehow would like to know whether we're going to have one default CC 
algorithm or support multiple algorithms. 
For example, DCCP has several CC algorithms based on some profiles. 
In case of MPTCP, one CC is enough? How about non-reno type CC like CUBIC?

Maybe the answer is we could have multiple algorithms, but how to specify is
an implementation matter? 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 4 Nov 2009 11:15:49 +0000

Hi,

> I guess it's not a major thing.
I agree with Alan. There is no point in sending port numbers for  
advertised addresses, since on subsequent subflow
handshakes - which must contain the JOIN option - local demuxing is  
not done based on port, but based on the token in the JOIN option.

> All I can think is signaling mechanism.. Can we leave this as an  
This is an open issue indeed. If the endpoint is sending, it can  
control how much data to put on each subflow. When it's receiving,
our current thinking is to use ECN (or worse, packet drops) to reduce  
the bandwidth on a given subflow.

Costin

---

From: <[email-masked]>
Date: Wed, 4 Nov 2009 17:00:54 -0000

>From the Security discussion it seems:

- the WG needs to describe /analyse threats as soon as possible. Do people think that http://www.ietf.org/id/draft-bagnulo-mptcp-threat-00.txt is a reasonable starting point for this?
- until the threat analysis doc has progressed a bit, it is too early to agree on the security mechanism. We will have to wait & see whether we can decide this in Feb/March.

phil

> -----Original Message-----
> From: marcelo bagnulo braun [mailto:[email-masked]]
> Sent: 02 November 2009 19:28
> To: Eardley,PL,Philip,DEE1 R
> Cc: [email-masked]

---

From: Lars Eggert <[email-masked]>
Date: Wed, 4 Nov 2009 11:03:10 -0800

> the bandwidth on a given subflow.

With per-subflow receive windows, you can use receive window tuning,  
too.

Lars

---

From: William Herrin <[email-masked]>
Date: Wed, 4 Nov 2009 15:19:11 -0500

Hi Joe,

I assume that's a convention rather than a hard protocol requirement.
Something previous folks working in the space have deemed a "wise
idea" rather than a "all these things will break if we don't" kind of
thing, yes?

Is there a good discussion on this somewhere so I can read up on the
rationale that brought it into being?

> multiple traffic flows. But I'm sure Scott can be more specific...

Oh, I see. Like splitting the HTTP request into multiple subflows so
that the IDS can't see the request, yes?

Smells like a system configuration issue to me: if you want the IDS to
do deep packet inspection, you have to arrange for all possible flows
to and from the protected system to go past the same IDS and if your
protected systems are configured to support MP then then the IDS must
be able to recognize MP as well.

I'll entertain suggestions why it's actually worse than that, but at
the moment I don't see how that's a con versus MPTCP that breaks up
traffic some other way than subflows.

> I do have a

I figure that if we accept the proposal that we should support
applications using the legacy API at all then the best we can do for
the ones who need to know the local and/or remote address and port
information is to provide values which:

A. Are unlikely to induce a security problem, and
B. Will tend to pull up information about multipath TCP and legacy
apps if the user plugs the unexpected values into a google search.

My thought on this is that we should identify some IP address which is
routable as a destination in as few deployed IP stacks as possible. We
can register it with IANA as meaning "multipath TCP in operation but
legacy API in use."

> changes what one may expect a transport protocol's behaviour to be).

Allowing either side to start MPTCP creates some complexity since:

1. The sides' attempts to start MPTCP could overlap requiring one of
them to yield to the other.
2. Regardless of who starts MPTCP, only the connection's initiator can
start up new subflows without running afoul of the firewall.

> Ah, well, that's kinda what I was describing above. What does your can

Closed windows. When the one or more subflows stall, the buffers in
the other flows will fill while the receiver waits for one of them to
contain the next packet in the sequence. How do you make sure there's
always enough buffer left in the other subflows to retransmit packets
from the stalled flow?

Regards,
Bill Herrin

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 05 Nov 2009 05:19:58 +0900 (JST)

From: Lars Eggert <[email-masked]>
Subject: Re: [multipathtcp] High-level design decisions /architecture
Date: Wed, 4 Nov 2009 11:03:10 -0800
Message-ID: <[email-masked]>

 > With per-subflow receive windows, you can use receive window tuning,

Well, that's an idea. Yes, I agree.
I tend to agree that this can be a open issue now. But, I still have a naive question...

Compared to having a tweaky algorithm at receiver side, doing simple 
negotiation during SYN exchange would be that difficult? (and probably we will
also need to use 1 bit in Add Address option to specify primary path or etc, 
but it's still simple.) 
If we think about a small gudget like iphone, I guess simple receiving 
algorithm might be preferable.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Scott Brim <[email-masked]>
Date: Wed, 04 Nov 2009 20:05:47 -0800

> algorithm might be preferable.

The problem with putting this in signaling is that you lose richness of
what you can do.  You need to have a few standard meanings for the bits
(or bit values) available to you, and so on.  Also, the particular
effect sought here is throttling traffic, which is exactly what receive
windows are good for.  If there is a need for anything else, maybe
signaling would be more appropriate.

Scott

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 5 Nov 2009 13:20:21 +0000

Hm. The problem with per subflow receive window is ensuring you don't  
create deadlocks. Sebastien Barre already found a case where a  
deadlock can appear, and I imagine there are many more.

I do agree it's a neat way to control policy, but I am afraid it may  
introduce more problems than it solves.
Costin

On 4 Nov 2009, at 19:03, Lars Eggert wrote:

> With per-subflow receive windows, you can use receive window  
> tuning, too.
>
> Lars

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 06 Nov 2009 09:59:27 +0900 (JST)

Hello,

From: Scott Brim <[email-masked]>
Subject: Re: [multipathtcp] High-level design decisions /architecture
Date: Wed, 04 Nov 2009 20:05:47 -0800
Message-ID: <[email-masked]>

 > signaling would be more appropriate.

Hmm. I understand your point.. but I have two opinions about this.
1: If a receiver has very sophisticated control algorithms (ACK streching or
   window size tweaking, etc), it can just refuse negotiation and control data 
   transfer by itself. 
   But, this does not prevent from having negotiation mechanism. 
   It still will be useful for simple receivers.

2: To do better control, I guess receiver will need to know sender's transfer 
   algorithm. Otherwise, things might get worse. 
   This will mean we need to have a signaling mechanism anyhow.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Mon, 09 Nov 2009 16:42:54 -0800

Hi, Bill,

Sorry; this appears to have found a dusty corner of my mailbox. Here's a
late reply...

> idea" rather than a "all these things will break if we don't" kind of

It's an artifact of how TCP works.

> rationale that brought it into being?

Since it's an artifact, I don't think there is. The issue is that
unknown options are ignored (as per RFC1122). The only time when you can
find out if an option is supported in a way that you know will preceded
all other segments is to put it in the SYN.

If you put it in a later segment, you don't know when the segment will
be received. Here's the problem, when A sends to B (assuming the
connection is already open):

	A sends segment #223 with option X
		presuming this is the first segment A sends
		with option X

Does A stop sending segments beyond 223 until option X is ACK'd? This
violates window processing. If A sends segment 224 with option X, and B
receives segment 224 before 223, it MUST ignore option X.

So you can't put in a new option that has a required effect or is
confirmed; there's no way to 'stop the stream' until the endpoints sync.
The only times the endpoints sync is during the SYN.

Joe

---

From: William Herrin <[email-masked]>
Date: Mon, 9 Nov 2009 22:18:33 -0500

Hi Joe,

I'm missing something here. I would have expected TCP to hold segment
224 and not process it (except perhaps for selective ack or to
retransmit 222's ack) until after 223 had been received and processed.
In other words, unless the option was defined to be unordered, I'd
expect the receiver to process it in the same order as the packet
sequence numbers.

That's not how TCP functions? The options are required to be processed
in order received rather than ordered by sequence? Or is it just that
all options to date have been designed to be processed in the order
received? I can see the point of doing that; I'm just trying to get a
scope on the issue.

> The only times the endpoints sync is during the SYN.

My first thought would be a three-part handshake. Request option X,
option X accepted, option X in effect. Requires that the receiver
understand that order is important after sending X accepted though.

Regards,
Bill Herrin

---

From: Joe Touch <[email-masked]>
Date: Mon, 09 Nov 2009 19:27:35 -0800

> expect the receiver to process it in the same order as the packet

That requires that options are processed on the data after ordered, as
you note. However, TCP is an unordered protocol. So here's the catch.
Let's say you introduce option X in segment 224.

- - can you send it in 225, or does the protocol come to a grinding halt?

If this is the first option, you need to ACK it. But you can't ACK 225.

So first, until everything up to 225 is ACKd, you can't send 226.

And second, while that's happening the reciever can't ACK 225, because
224 might have an option that's incompatible with the one in 225. So the
transmitter keeps sending 225, and the receiver keeps ignoring it.

Ultimately, what you need is a separate sequence number space and
retransmission mechanism for these options, and rules for how they tie
to TCP's data path.

That's a HUGE, non-backward compatible change to TCP.

> in order received rather than ordered by sequence? 

Yes. Options are part of the control plane, and that's processed when
the segment is received - including ACKs. Data is passed up to the app
after reordering - it's only data that is queued.

> Or is it just that

See above. Doing otherwise requires a sequence space that doesn't exist.

> understand that order is important after sending X accepted though.

Right. How do you bury that handshake in an unordered, unreliable stream
without halting it, though?

Joe

---

From: William Herrin <[email-masked]>
Date: Tue, 10 Nov 2009 15:45:42 -0500

On Mon, Nov 9, 2009 at 10:27 PM, Joe Touch <[email-masked]> wrote:
> you note. However, TCP is an unordered protocol.
>

Hi Joe,

How about a 4-step handshake like this:

Request option X, option X accepted, option X in effect (1), option X
in effect (2).

When you receive X accepted, attach X1 to *every* outgoing packet
until you get an ack for everything up to and including the first X1
packet. Then start sending X2 in every outgoing packet until you get
an ack for any X2 packet. Then stop sending the X option on any packet
that isn't a retransmission of one where X1 or X2 was sent.

At the receiver end: until you receive a packet with X2, only packets
with X1 or X2 have option X in effect. After you see any packet with
X2, all subsequent packets have option X in effect regardless of the
presence of X1 or X2.

Nobody pauses. Options are processed in the order received. And you're
dealing with pretty trivial state changes on each end.

One concern that jumps to mind is if the receiver gets a late copy of
an already received and acked segment where option X is not in effect.
What will the receiver do with the late packet?  Let me narrow that a
bit: assuming option X does not affect the use of the TCP sequence
number in any way, what will the receiver do with a late copy of an
already received packet where it thinks X is in effect but in reality
X was not requested?

On the other hand, if option X has some other intrinsic value (such
has holding the multipath TCP connection ID for middleboxes to decode)
then you can just keep sending it in every packet which uses it for
the remainder of the connection instead of the fancy handshake.

Regards,
Bill Herrin

---

From: Joe Touch <[email-masked]>
Date: Tue, 10 Nov 2009 13:12:39 -0800

William Herrin wrote:
> packet.

What happens to retransmitted packets that were first sent before you
decided to set X?

> Then start sending X2 in every outgoing packet until you get
> with X1 or X2 have option X in effect.

So now you have a problem. Some packets are handled as if option X is in
effect, and some are not. These could arrive in any order. What if the
option has something to do with the order in which the packets are ACKd?
Or the order in which they're sent to the app? You now have essentially
two separate data streams, so you need to reimplement buffering to
handle the old stream as it was, and the new stream treats the option
exchange as a SYN-SYN/ACK-ACK exchange.

The issue is that the receiver cannot act on X until the handshake
completes (it gets X2). None of the data with X1 in it can be processed
until X2 is received. But this "processing" could mean ACKing too - you
can't even ACK the data because the option could mean "ACK every four"
or something like that.

> After you see any packet with

Not packets sent before X was set at the source. Again, you have two
separate streams now - the one before X, and the one after X2. The first
is your current connection, which is essentially draining out. The
second is your new connection - with X set - that is basically starting
anew.

At that point, it's going to be a LOT simpler to just treat this as a
CLOSE and subsequent OPEN than to do this all inside a single connection.

> Nobody pauses.

Disagree - see above.

> Options are processed in the order received.

Disagree - see above. They can't be.

> And you're

Disagree - you now need another 11-state machine to talk about all the
possible ways in which the "set X" operation can fail, and to avoid
trying to negotiate another option while the first negotiation is going on.

> already received packet where it thinks X is in effect but in reality

I think at best that there would need to be a LOT of constraints on what
the option could do. But I'm not sure there's a useful way even if
that's true to make this work.

> the remainder of the connection instead of the fancy handshake.

Options can add asynchronous information - which is how they're used
now. They simply cannot add synchronized info - which is why they can't
be used to negotiate new options - AFAICT.

Joe

---

From: William Herrin <[email-masked]>
Date: Tue, 10 Nov 2009 17:12:43 -0500

Hi Joe,

Retransmit as originally sent. Otherwise you could have two very
different packets show up for the same segment. Set X1 only on packets
whose first transmission comes after option X accepted.

> option has something to do with the order in which the packets are ACKd?

As I mentioned: constraint on the option in this exchange in that it
can't impact the handling of the sequence number. ACK is handling the
sequence number.

> Or the order in which they're sent to the app?

You mean like URG data? I'm surprised URG hasn't been deprecated given
the inconsistent implementations.

So, I gather we'd have to add a constraint that the use of URG or
reordering options is incompatible with the use of a
post-syn-negotiated stateful option.

Otherwise, there's no ordering problem, right? The stack delivers data
to the app in order by sequence number and will have made all the data
in segments before the first X1 available to the app before the sender
sends the first X2.

> Not packets sent before X was set at the source.

Excepting any late arriving duplicates (where the data will be
discarded), there's no way for the receiver to get a packet that was
sent before X was set at the source. The source confirmed that by
switching to setting X2 only after everything up to and including the
first X1 packet was acked.

> be used to negotiate new options - AFAICT.

By the way, I appreciate you bouncing this back and forth with me. I
don't doubt your conclusions but I find that trying to poke holes in a
conclusion is one of the more effective ways to learn the underlying
issues that brought it about.

Regards,
Bill Herrin

---

From: Joe Touch <[email-masked]>
Date: Tue, 10 Nov 2009 18:56:44 -0800

Hi, Bill,

> whose first transmission comes after option X accepted.

The issue is that there is no synchronization between options and
segments or segment offsets. You're now requiring that - which is a huge
complication.

> can't impact the handling of the sequence number. ACK is handling the

I think that the option has more constraints than that - it can't affect
anything in the data stream - ACK or anything else.

> the inconsistent implementations.

URG is about how quickly you tell the user about data you've ACKd. This
is about whether you need to hold data you've ACKd until the new option
is ACKd - an "anti"-URG.

> post-syn-negotiated stateful option.

More than that - you also need to hold the queue - an "anti"-URG - until
the option is negotiated. This is a new behavior not currently supported.

> in segments before the first X1 available to the app before the sender

The option could affect how the data is delivered - e.g., whether there
are markers, or the chunksize sent to the app. If so, you can't send the
data up until you negotiate the option that tags that data.

> first X1 packet was acked.

Late arriving dups have both data and header info. TCP decides how to
handle the data LAST - it operates on the header first. You can't decide
to ignore a header field based on whether the data will be dropped
because that happens last.

> issues that brought it about.

No problem... I hope it's continuing to be useful.

Joe