Subject: [multipathtcp] Interesting paper

From: Bryan Ford <[email-masked]>
Date: Tue, 27 Oct 2009 15:56:53 -0400

A while back some of us were debating the possibility of having  
multipath TCP try to "optimize" its use of its underlying legacy TCP  
flows by relaxing the underlying TCP's normal requirement that a given  
data segment always be retransmitted _on that same subflow_ until it  
is acknowledged _on that same subflow_.  It would clearly be appealing  
to relax that constraint so that, for example, if MPTCP first  
transmits a segment on one subflow, notices the first subflow is  
stalled and retransmits the segment on a second subflow, and gets an  
ack for the segment on the second subflow, MPTCP could avoid having to  
retransmit that same segment redundantly on the first subflow once it  
gets started again, and instead just transmit new data on the first  
subflow using the old (unacknowledged) sequence number space.

But no dice.  I and a few others cited concerns that there "might" be  
middleboxes out there that such behavior wouldn't play well with, but  
no one was sure how prevalent they might be.  But I think the  
following paper (and the background security literature surrounding  
it) effectively hammers the last nail into the coffin of this  
potential optimization:

	Efficient and Robust TCP Stream Normalization
	Mythili Vutukuru, Hari Balakrishnan, Vern Paxson
	http://www.mit.edu/~mythili/research/normalization.pdf

In effect, retransmitting TCP data segments with different data can be  
used as an attack method with which to evade intrusion detection  
systems, and hence the security community is hard at work defining  
efficient algorithms to detect and prevent such attacks by  
"normalizing" TCP streams and ensuring all retransmissions of a given  
segment contain the same data.  In this sense a "retransmit different  
data" optimization is similar to the classic TCP performance technique  
of ACK splitting: it can work in some scenarios, but only until the  
bad guy figures out how to use it to his advantage, at which point the  
security folks close that security hole and the optimization no longer  
works either.  So whenever MPTCP is running atop legacy TCP flows, it  
really does have to use them as if they were semantically independent  
TCP flows, unnecessary data retransmissions and all.

Bryan

---

From: Tatiana Polishchuk <[email-masked]>
Date: Tue, 27 Oct 2009 22:29:34 +0200

> Just a quick note:
>
> In our work on multipath HIP we observed it is clearly efficiently to
> always retransmit data by the "best" available path. And in practice this
> path is usually not the one which is causing the congestion and where the
> corresponding packet was lost from. This solution corresponds to the
> specific use-case then the application requires maximization of the total
> throughput. And we did not consider the middleboxes issue because HIP
> provides their successful triversal.
>
> Best,
> Tatiana
>
> On Tue, Oct 27, 2009 at 9:56 PM, Bryan Ford <[email-masked]> wrote:
>
>> A while back some of us were debating the possibility of having multipath
>> TCP try to "optimize" its use of its underlying legacy TCP flows by relaxing
>> the underlying TCP's normal requirement that a given data segment always be
>> retransmitted _on that same subflow_ until it is acknowledged _on that same
>> subflow_.  It would clearly be appealing to relax that constraint so that,
>> for example, if MPTCP first transmits a segment on one subflow, notices the
>> first subflow is stalled and retransmits the segment on a second subflow,
>> and gets an ack for the segment on the second subflow, MPTCP could avoid
>> having to retransmit that same segment redundantly on the first subflow once
>> it gets started again, and instead just transmit new data on the first
>> middleboxes out there that such behavior wouldn't play well with, but no one
>> was sure how prevalent they might be.  But I think the following paper (and
>> the background security literature surrounding it) effectively hammers the
>> last nail into the coffin of this potential optimization:
>>
>>  http://www.mit.edu/~mythili/research/normalization.pdf<http://www.mit.edu/%7Emythili/research/normalization.pdf>
>>
>> used as an attack method with which to evade intrusion detection systems,
>> and hence the security community is hard at work defining efficient
>> algorithms to detect and prevent such attacks by "normalizing" TCP streams
>> and ensuring all retransmissions of a given segment contain the same data.
>>  In this sense a "retransmit different data" optimization is similar to the
>> classic TCP performance technique of ACK splitting: it can work in some
>> scenarios, but only until the bad guy figures out how to use it to his
>> advantage, at which point the security folks close that security hole and
>> the optimization no longer works either.  So whenever MPTCP is running atop
>> legacy TCP flows, it really does have to use them as if they were
>> semantically independent TCP flows, unnecessary data retransmissions and
>> all.
>>
>> Bryan
>>
>>
>>
>> _______________________________________________
>> multipathtcp mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/multipathtcp
>>
>>
>

---

From: "Costin Raiciu" <[email-masked]>
Date: Tue, 27 Oct 2009 23:19:56 -0000

Hi Bryan,

We were aware of "traffic normalizers",hence our design decision. Mark wrote the first paper in that line of work, and the one you mention is the second :).

Cheers
Costin.
  ----- Original Message ----- 
  From: Bryan Ford 
  To: [email-masked] 
  Sent: Tuesday, October 27, 2009 7:56 PM
  Subject: [multipathtcp] Interesting paper

  A while back some of us were debating the possibility of having multipath TCP try to "optimize" its use of its underlying legacy TCP flows by relaxing the underlying TCP's normal requirement that a given data segment always be retransmitted _on that same subflow_ until it is acknowledged _on that same subflow_.  It would clearly be appealing to relax that constraint so that, for example, if MPTCP first transmits a segment on one subflow, notices the first subflow is stalled and retransmits the segment on a second subflow, and gets an ack for the segment on the second subflow, MPTCP could avoid having to retransmit that same segment redundantly on the first subflow once it gets started again, and instead just transmit new data on the first subflow using the old (unacknowledged) sequence number space.

  But no dice.  I and a few others cited concerns that there "might" be middleboxes out there that such behavior wouldn't play well with, but no one was sure how prevalent they might be.  But I think the following paper (and the background security literature surrounding it) effectively hammers the last nail into the coffin of this potential optimization:

  In effect, retransmitting TCP data segments with different data can be used as an attack method with which to evade intrusion detection systems, and hence the security community is hard at work defining efficient algorithms to detect and prevent such attacks by "normalizing" TCP streams and ensuring all retransmissions of a given segment contain the same data.  In this sense a "retransmit different data" optimization is similar to the classic TCP performance technique of ACK splitting: it can work in some scenarios, but only until the bad guy figures out how to use it to his advantage, at which point the security folks close that security hole and the optimization no longer works either.  So whenever MPTCP is running atop legacy TCP flows, it really does have to use them as if they were semantically independent TCP flows, unnecessary data retransmissions and all.

  Bryan

------------------------------------------------------------------------------

---

From: Fred Baker <[email-masked]>
Date: Wed, 28 Oct 2009 07:32:36 +0800

It seems to me that there are two fundamental problems in multipath  
TCP: identifying the peer, and convincing the network to do something  
specific in routing. If the identity of the peer is divorced from the  
traditional pseudoheader contents to something else that they are true  
of, one can readily send segments on any viable pseudoheader for that  
identity. For example, RFC 5061 allows SCTP to change its address, and  
RFC 4096 contains the ability to both list one's addresses and  
exchange identity-related cookies. Routing is potentially another  
issue - it might relate to asking the underlying protocol to "route me  
differently", or it may simply relate to the addresses one uses.

In any event, if I can change the pseudoheader and yet remain within  
the same session, problem solved. I can retransmit on a different  
pseudoeheader or I can spray traffic across many pseudoheaders.

Is that what you're after?

On Oct 28, 2009, at 3:56 AM, Bryan Ford wrote:

> flows by relaxing the underlying TCP's normal requirement that a  
> given data segment always be retransmitted _on that same subflow_  
> until it is acknowledged _on that same subflow_.  It would clearly  
> be appealing to relax that constraint so that, for example, if MPTCP  
> first transmits a segment on one subflow, notices the first subflow  
> is stalled and retransmits the segment on a second subflow, and gets  
> an ack for the segment on the second subflow, MPTCP could avoid  
> having to retransmit that same segment redundantly on the first  
> subflow once it gets started again, and instead just transmit new  
> data on the first subflow using the old (unacknowledged) sequence  
> number space.
>
> But no dice.  I and a few others cited concerns that there "might"  
> be middleboxes out there that such behavior wouldn't play well with,  
> but no one was sure how prevalent they might be.  But I think the  
> In effect, retransmitting TCP data segments with different data can  
> be used as an attack method with which to evade intrusion detection  
> "normalizing" TCP streams and ensuring all retransmissions of a  
> given segment contain the same data.  In this sense a "retransmit  
> different data" optimization is similar to the classic TCP  
> performance technique of ACK splitting: it can work in some  
> scenarios, but only until the bad guy figures out how to use it to  
> his advantage, at which point the security folks close that security  
> hole and the optimization no longer works either.  So whenever MPTCP  
> is running atop legacy TCP flows, it really does have to use them as  
> if they were semantically independent TCP flows, unnecessary data  
> retransmissions and all.
>
> Bryan
>
>

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 27 Oct 2009 16:47:42 -0700

Definitely agree with Fred.

Since this effort is going to change host stacks can we force the TCP  
and UDP pseudo header checksum to not include IP addresses? Make the  
change at one time?

That would greatly simplify things.

Dino

On Oct 27, 2009, at 4:32 PM, Fred Baker wrote:

> TCP: identifying the peer, and convincing the network to do  
> something specific in routing. If the identity of the peer is  
> divorced from the traditional pseudoheader contents to something  
> else that they are true of, one can readily send segments on any  
> viable pseudoheader for that identity. For example, RFC 5061 allows  
> SCTP to change its address, and RFC 4096 contains the ability to  
> both list one's addresses and exchange identity-related cookies.  
> Routing is potentially another issue - it might relate to asking the  
> underlying protocol to "route me differently", or it may simply  
> relate to the addresses one uses.
>
>> multipath TCP try to "optimize" its use of its underlying legacy  
>> TCP flows by relaxing the underlying TCP's normal requirement that  
>> a given data segment always be retransmitted _on that same subflow_  
>> be appealing to relax that constraint so that, for example, if  
>> MPTCP first transmits a segment on one subflow, notices the first  
>> subflow is stalled and retransmits the segment on a second subflow,  
>> and gets an ack for the segment on the second subflow, MPTCP could  
>> avoid having to retransmit that same segment redundantly on the  
>> first subflow once it gets started again, and instead just transmit  
>> new data on the first subflow using the old (unacknowledged)  
>> sequence number space.
>>
>> be middleboxes out there that such behavior wouldn't play well  
>> with, but no one was sure how prevalent they might be.  But I think  
>> the following paper (and the background security literature  
>> surrounding it) effectively hammers the last nail into the coffin  
>> of this potential optimization:
>>
>> his advantage, at which point the security folks close that  
>> security hole and the optimization no longer works either.  So  
>> whenever MPTCP is running atop legacy TCP flows, it really does  
>> have to use them as if they were semantically independent TCP  
>> flows, unnecessary data retransmissions and all.
>>
>> Bryan
>>
>>

---

From: Mark Handley <[email-masked]>
Date: Wed, 28 Oct 2009 00:07:43 +0000

2009/10/27 Dino Farinacci <[email-masked]>:
> Since this effort is going to change host stacks can we force the TCP and
> UDP pseudo header checksum to not include IP addresses? Make the change at
> one time?
>

Wouldn't this break all legacy NATs which fix the TCP checksum?

 - Mark

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 27 Oct 2009 17:39:02 -0700

> Wouldn't this break all legacy NATs which fix the TCP checksum?

Well I'd fix them too. NATs could be easier to upgrade than hosts.

Dino

---

From: Mark Handley <[email-masked]>
Date: Wed, 28 Oct 2009 00:56:18 +0000

> Well I'd fix them too. NATs could be easier to upgrade than hosts.

Several points:

1. You'd have to fix all the NATs before you could deploy your new TCP
variant, as any old NATs would break the new TCP.  But as no-one would
have deployed the TCP, I don't see why NAT vendors would change.

2.  Many hosts are also NATs, so NATs are at least as hard as hosts to upgrade.

3. NATs are embedded, so they often only get upgraded when you throw
out that old DSL modem, WiFi AP, router or firewall.

In comparison, TCP is relatively easy to upgrade.  It's got a built-in
negotiation protocol that allows new features to be enabled only when
both ends support a feature, and it has a built-in retransmission
function that allows retrying without the new features when it
encounters troublesome broken middlebox hardware.

 - Mark

---

From: Scott Brim <[email-masked]>
Date: Tue, 27 Oct 2009 20:59:04 -0400

> http://www.mit.edu/~mythili/research/normalization.pdf

The problem is that this is not a research group, and each subflow needs
to look like TCP.  Intrusion prevention systems make it need to behave
like TCP to some extent as well.  You can't just "fix" the IPSs to look
for MPTCP because multiply-connected sites will have multiple IPSs that
don't talk to each other.

In this particular case, is it possible that if MPTCP successfully sends
a segment data on a different path, it can still keep trying to
retransmit that segment on the "bad" path until it succeeds, but with a
flag for the corresponding MPTCP essentially saying "you've already seen
this but I have to resend it anyway, just ignore it"?

Scott

---

From: Scott Brim <[email-masked]>
Date: Tue, 27 Oct 2009 21:06:37 -0400

Fred Baker allegedly wrote on 10/27/2009 7:32 PM:
> It seems to me that there are two fundamental problems in multipath TCP:
> identifying the peer, and convincing the network to do something
> RFC 4096 contains the ability to both list one's addresses and exchange
> identity-related cookies. Routing is potentially another issue - it
> might relate to asking the underlying protocol to "route me

Right about pseudoheaders but that doesn't deal with the problem that
once you have sent a segment and not had it acked on a particular
src/dst pair, you can't use the same sequence number space for any other
data.

Let's ignore potential "route me differently" mechanisms for now,
because they don't exist.

Dino Farinacci allegedly wrote on 10/27/2009 8:39 PM:
>>> Since this effort is going to change host stacks can we force the
>>> TCP and UDP pseudo header checksum to not include IP addresses?
>>> Make the change at one time?
>>>

Not a research group, eh?

Scott

---

From: Fred Baker <[email-masked]>
Date: Wed, 28 Oct 2009 09:40:38 +0800

On Oct 28, 2009, at 9:06 AM, Scott Brim wrote:

>> TCP:
>> exchange

Not sure I follow that. If the pseudoheaders (by which I mean the  
tuples of addresses and ports) are interchangeable within the same  
identity, the sequence space is the same.

I must be missing something, so let me spell it out and you can tell  
me what I'm missing.

Consider two peers, Alice and Bob. Alice has two addresses, A' and A",  
and Bob and two addresses, B' and B". Alice opens the conversation  
(SYN/INIT/whatever) carrying with it the address set {A', A"}, her  
port number, the address B', and his port number, plus some form of  
stream identification nonce a la ILNP - eg, a cookie.

         http://tools.ietf.org/html/draft-rja-ilnp-dns
         http://tools.ietf.org/html/draft-rja-ilnp-icmp
         http://tools.ietf.org/html/draft-rja-ilnp-intro
         http://tools.ietf.org/html/draft-rja-ilnp-nonce

Bob replies adding the address set {B', B"} plus his nonce.

from this point on, conversations are not local to pseudoheaders; they  
are between Alice and Bob. Alice could send four messages
     {A', A-port}->{B', B-port}, sequence 1
     {A', A-port}->{B", B-port}, sequence 2
     {A", A-port}->{B', B-port}, sequence 3
     {A", A-port}->{B", B-port}, sequence 4

and Bob would receive them understanding them each to be

     {Alice, A-port}->{Bob, B-port}

and process in the sequence (1, 2, 3, 4).

OK, I'm clueless. What did I miss?

> because they don't exist.

They don't exist in IP. They do exist in IEC 14908-[23], which is  
where I got to thinking about them. Basically, if in a multipath route  
one path is noisy, they have a bit that says "try a different next  
hop". Next question is how efficacious it is; I can't say I know. But  
for multipath TCP it might be useful to be able to ask for some  
variation in routing.

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 28 Oct 2009 09:34:47 -0700

Mark, isn't this effort going to apply to IPv6 only?

If so, aren't there suppose to be no NAT deployment for IPv6? The  
operational people I talk to say they prefer using globally/publicly  
routable addresses *everywhere*.

Dino

On Oct 27, 2009, at 5:56 PM, Mark Handley wrote:

>>>> TCP and
>>>> UDP pseudo header checksum to not include IP addresses? Make the  
>>>> change
>>>> at
>>>> one time?
>>>>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 28 Oct 2009 17:37:46 +0100

> Mark, isn't this effort going to apply to IPv6 only?

AFAICT MPTCP is both IPv4 and IPv6

morevoer, MPTCP has very interesting features in nated environments, 
such as allowing to survive ongoing communications in nat based multihoming

Regards, marcelo

---

From: Scott Brim <[email-masked]>
Date: Wed, 28 Oct 2009 12:38:17 -0400

> Mark, isn't this effort going to apply to IPv6 only?

MPTCP is agnostic about IP version.

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 28 Oct 2009 16:39:23 +0000

I can answer that :). This should apply primarily to v4 for now. v6  
is future work. The wg is chartered for v4.

Costin

On 28 Oct 2009, at 16:34, Dino Farinacci wrote:

> operational people I talk to say they prefer using globally/ 
> publicly routable addresses *everywhere*.
>
> Dino
>
>>>>> UDP pseudo header checksum to not include IP addresses? Make  
>>>>> the change
>>>>> at
>>>>> one time?
>>>>>

---

From: Marc Blanchet <[email-masked]>
Date: Wed, 28 Oct 2009 17:46:13 +0100

Costin Raiciu a écrit :
> I can answer that :). This should apply primarily to v4 for now. v6 is 
> future work. The wg is chartered for v4.

v4 only does not make sense to me. v6 shall be as current as v4.

Marc.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 28 Oct 2009 17:46:26 +0100

Interesting... the charter doesn't have IPv4 nor IPv6 in it.
I assumed that we were chartered to work on both, don't we?

---

From: <[email-masked]>
Date: Wed, 28 Oct 2009 17:02:23 -0000

The Charter does not mention IP version, except that it says we will
look at:
- handling NAT interactions [to the extent you think that implies v4]
- how MPTCP interacts with ... address agility mechanisms, such as ...
SHIM6

I remember one idea was that mptcp might help transition [one path is
v4, another is v6], so hopefully both are in scope. 

If someone has thoughts about how the high-level solution might have to
be different for IPv4 & IPv6, it would be great to raise it on the list
and we can discuss in Hiroshima. My working assumption is that there's
no difference (other than detailed things like the 'Add address' option
needs to be bigger, see Section 4.3.1 of
http://www.ietf.org/id/draft-ford-mptcp-multiaddressed-02.txt ) - please
disillusion me ASAP!

best wishes,
phil

> -----Original Message-----
> From: [email-masked]
[mailto:[email-masked]]
> On Behalf Of Costin Raiciu
> Sent: 28 October 2009 16:39
> To: Dino Farinacci
> Cc: [email-masked]; Bryan Ford
> Subject: Re: [multipathtcp] Interesting paper
>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 28 Oct 2009 18:09:46 +0100

i think some things can be done differently if you assume IPv4 only or 
IPv6 only (for example, you could use CGAs for security if we go for 
IPv6 only)
However, if we decide now that we want to support both (which i think we 
should) i think it is perfeclty possible to design MPTCP so that it does 
support boht of them with the same mechanisms (modulo de address length)

Regards, marcelo

[email-masked] escribió:
>>>> 3. NATs are embedded, so they often only get upgraded when you
>>>>         
> throw
>   
>>>> negotiation protocol that allows new features to be enabled only
>>>>         
> when
>

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 28 Oct 2009 17:15:38 +0000

Hi,

My bad :).

I guess what I mean is that by only focusing  on v6 we lose in the  
short term at least; I do not want to tie multipath to v6 deployment.
I definitely agree that the design should cover both.

It seems if we design for v4, it's easy to support v6, but not the  
other way around, (because there are no NATs in v6 the protocol would  
be much simpler but not deployable on v4)

Cheers
Costin

On 28 Oct 2009, at 17:09, marcelo bagnulo braun wrote:

> i think some things can be done differently if you assume IPv4 only  
> or IPv6 only (for example, you could use CGAs for security if we go  
> for IPv6 only)
> However, if we decide now that we want to support both (which i  
> think we should) i think it is perfeclty possible to design MPTCP  
> so that it does support boht of them with the same mechanisms  
> (modulo de address length)
>
> Regards, marcelo
>
>
>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 28 Oct 2009 18:17:49 +0100

makes sense to me
I think i agree with you assumption, that if we design for IPv4 it would 
work for IPv6,

Costin Raiciu escribió:
>> However, if we decide now that we want to support both (which i think 
>> we should) i think it is perfeclty possible to design MPTCP so that 
>> it does support boht of them with the same mechanisms (modulo de 
>> address length)
>>
>> Regards, marcelo
>>
>>
>>

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 28 Oct 2009 10:20:15 -0700

> think we should) i think it is perfeclty possible to design MPTCP so  
> that it does support boht of them with the same mechanisms (modulo  
> de address length)
>
> Regards, marcelo

That was the conclusion we made with LISP. That is why the EID is a  
power-of-2 identifier for both IPv4 and IPv6. We wanted to consider  
flat-structured IDs for IPv6 either using MAC addresses, EUI-64, or  
CGA type addresses but we wanted to create EID-prefixes for  
aggregation opportunities.

Dino

---

From: Sébastien Barré <[email-masked]>
Date: Fri, 30 Oct 2009 10:13:58 +0000

Hi,

> IPv6 only (for example, you could use CGAs for security if we go for 
Well, AFAICT, the nice thing with CGA is that we can do something in 
between. I mean, as far as the first address you use for the flow is 
CGA, and thus IPv6, you can add IPv4 addresses later, since it is just a 
matter of including them in the signature. Now if we are IPv4 only, then 
we need something else.

regards,

Sébastien.
> we should) i think it is perfeclty possible to design MPTCP so that it 
> does support boht of them with the same mechanisms (modulo de address 
> length)
>
> Regards, marcelo
>
>
>

-- 
Sébastien Barré
Researcher,
CSE department, UCLouvain, Belgium
http://inl.info.ucl.ac.be/sbarre

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 30 Oct 2009 11:20:04 +0100

agree, but as you say if we want to support ipv4 only, then we need 
something else
I don't think the WG should work on IPv6 specific optimizations at this 
point however. Maybe worth pursuing those later on though

Regards, marcelo

Sébastien Barré escribió:
> CGA, and thus IPv6, you can add IPv4 addresses later, since it is just 
> a matter of including them in the signature. Now if we are IPv4 only, 
> then we need something else.
>
> regards,
>
> Sébastien.