Subject: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

From: Behcet Sarikaya <[email-masked]>
Date: Wed, 1 Jun 2016 13:37:46 -0500

Hi all,

Recent mails in this thread came to my attention because I work on the
hybrid access network issues and wish to convey my views on some of
the issues discussed.

First of all, the claims that there is strong operator interest on
this is over blown. Hybrid access is based on offloading some CPE
traffic to 3G/LTE network, I don't think mobile operators are fond of
such a thing.

Also I strongly concur with Yoshi's point of CPE to convert UDP or TCP
into MPTCP and requires
Concentrators to convert back MPTCP into UDP or TCP is a complex
process. The crux of this is the problems arising from the use of
MPTCP for tunneling, which is what this draft is advocating.

Here I recommend Joe Touch's Intarea draft
https://tools.ietf.org/id/draft-ietf-intarea-tunnels-02.txt
on IP Tunnels in the Internet Architecture. IETF developed many
tunneling protocols which are surveyed in this draft but none of them
are TCP/MPTCP based.
I also question suitability of this work to the charter. There is this
charter text:
Finally, the working group will explore whether an MPTCP-aware
  middlebox would be useful, where at least one end host is MPTCP-enabled.
  For example, potentially helping MPTCP's incremental deployment by
  allowing only one end host to be MPTCP-enabled and the middlebox acts as
  an MPTCP proxy for the other end host, which runs TCP; and potentially
  helping some mobility scenarios, where the middlebox acts as an anchor
  between two MPTCP-enabled hosts. The working group will detail what real
  problems an MPTCP-enabled middlebox might solve, how it would impact the
  Multipath TCP architecture (RFC6182), what proxy approach might be
  justified as compared against alternative solutions to the problems, and
  the likely feasibility of solving the technical and security issues.

This draft has a middlebox called Concentrator but it is not trying to
enable MPTCP hosts to communicate with TCP hosts, it is just tunneling
the traffic. So there is no charter item that fits to this draft.
I strongly suggest staying away from making MPTCP a tunneling protocol.

Regards,

Behcet

---

From: <[email-masked]>
Date: Thu, 2 Jun 2016 14:02:30 +0000

Hi Behcet,

As one of the operators very interested in this, I don’t think it is over blown and the use case you describe might be the first one for a number of operators, but others are following. Fundamentally it is about resource pooling of access networking resources (of whatever technology transporting IP).

We are well-aware of the problems of a MPTCP-based solution (and draft-boucadair tries to solve some of them). However, we are not aware of any better standardized solution for multiple paths with heterogeneous characteristics of available and varying bandwidth, varying delays, and different packet handling characteristics. In case there is better ways of solving that problem let us know. 

The need for the unfortunate proxying/concentrators is a short term solution until MPTCP is more widespread implemented and deployed. 

Kind regards/Viele Grüsse

Marcus

——-
Marcus Brunner
Swisscom

Am 01.06.16, 20:37 schrieb "multipathtcp im Auftrag von Behcet Sarikaya" <[email-masked] im Auftrag von [email-masked]>:

>_______________________________________________
>multipathtcp mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Jun 2016 10:17:43 -0700

> We are well-aware of the problems of a MPTCP-based solution (and draft-boucadair tries to solve some of them). However, we are not aware of any better standardized solution for multiple paths with…

Why wouldn't an overlay that employs dynamic routing be easier?

It wouldn't incur the delays of using reliable delivery with messages
that don't preserve IP datagram boundaries and it's already supported
with existing equipment. Further, it could use much more sophisticated
path monitoring and choice.

I.e., this is exactly what you'd get if you deploy something akin to
LISP and basically ignore the label distribution problem (which you
really don't have).

Joe

---

From: "Henderickx, Wim (Nokia - BE)" <[email-masked]>
Date: Fri, 3 Jun 2016 02:58:38 +0000

In-line

>LISP and basically ignore the label distribution problem (which you
WH> it doesn’t solve the use case where packets of a flow have to be split over multiple links and reordering + congestion control is required for these flows. If these requirements were not there you’re right that routing can be used.

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Jun 2016 21:17:46 -0700

> WH> it doesn’t solve the use case where packets of a flow have to be split over multiple links and reordering + congestion control is required for these flows. If these requirements were not there…

What you really don't want are:

     - to take a RTT hiccup to recover from a segment loss

    - to implement congestion control over your tunnels underneath
congestion control inside transport on the IP packets you carry

IMO, order preservation should be something done at the endpoints, not
in the net, but if that's all you have left you can implement that with
a fixed amount of delay at the egress collection point.

None of this is worth the complexity and side-effects of MPTCP, IMO.

Joe

---

From: <[email-masked]>
Date: Fri, 3 Jun 2016 07:19:38 +0000

Hi Joe,

If you mean with dynamic routing a packet by packet routing decision, that could work however, how do you incorporate the path characteristic feedback into that decision. Here the MPTCP mechanisms just come in handy and do the job quite well, but as you say we pay with some complexity. That complexity we feel,will have anyway with whatever solution we are going.

> Am 02.06.2016 um 19:18 schrieb Joe Touch <[email-masked]>:
> 
> 
>

---

From: "Henderickx, Wim (Nokia - BE)" <[email-masked]>
Date: Fri, 3 Jun 2016 12:37:28 +0000

In-line

>None of this is worth the complexity and side-effects of MPTCP, IMO.

WH> If you have a flows that are bigger than a given link and you want to split them over multiple paths, the applications can implement MP-TCP for sure, but this is not always the case and there are use cases as per draft where basic routing is not sufficient and you want to use the path characteristics feedback which is build in in MP-TCP to accommodate that. The alternative is to reinvent all of this in yet another protocol.

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Jun 2016 10:20:59 -0700

On 6/3/2016 12:19 AM, [email-masked] wrote:
> Hi Joe,
>
> If you mean with dynamic routing a packet by packet routing decision, that could work however, how do you incorporate the path characteristic feedback into that decision. 
That's what multipath routing has been doing for years.

AFAIK, the mechanisms in MPTCP are an instance inspired by existing
multipath routing.

> Here the MPTCP mechanisms just come in handy and do the job quite well, but as you say we pay with some complexity. That complexity we feel,will have anyway with whatever solution we are going.

Perhaps, but MPTCP is the wrong solution for an IP tunnel for many reasons:

- it will layer congestion control on top of congestion control for all
transports in the payload IP that already support it (which they should)
- it fails to preserve payload IP messages inside a single TCP segment,
which increases loss due to a lack of fate-sharing and increases latency
and jitter
- by recovering from loss, it increases latency and jitter
- it can use only one load monitoring mechanism (the one built-in)
- it can use only one traffic distribution mechanism (the one built-in)

That's a lot of baggage vs. just using tunnels and existing dynamic
multipath routing -- and the latter doesn't require a new spec.

Joe

---

From: Behcet Sarikaya <[email-masked]>
Date: Fri, 3 Jun 2016 13:36:40 -0500

On Fri, Jun 3, 2016 at 12:20 PM, Joe Touch <[email-masked]> wrote:
> Perhaps, but
>MPTCP is the wrong solution for an IP tunnel for many reasons:
>

+1

This was the point in my mail.
Thanks Joe for getting into the discussions and clarifying things on this list.

Regards,

Behcet

---

From: David Allan I <[email-masked]>
Date: Fri, 3 Jun 2016 19:29:12 +0000

Hi Joe:

Comments below prefaced by DA>

-----Original Message-----
From: multipathtcp [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Friday, June 03, 2016 10:21 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

AFAIK, the mechanisms in MPTCP are an instance inspired by existing multipath routing.

- it will layer congestion control on top of congestion control for all transports in the payload IP that already support it (which they should)
DA> Would that not be true of any tunnel mechanism that used feedback of the tunnel leg as well as the e2e application?

- it fails to preserve payload IP messages inside a single TCP segment, which increases loss due to a lack of fate-sharing and increases latency and jitter
DA> I apologize for not getting your point here.

- by recovering from loss, it increases latency and jitter
DA> AFAIK, any loss does that irrespective of the delivery mechanism. In the hybrid access case you are correcting loss with a smaller RTT are you not?

- it can use only one load monitoring mechanism (the one built-in)
DA> How many did you want?

- it can use only one traffic distribution mechanism (the one built-in)
DA> If you are saying it is "implementation constrained", well go figure :-)  

That's a lot of baggage vs. just using tunnels and existing dynamic multipath routing -- and the latter doesn't require a new spec.

DA> Can you point me to the feedback mechanisms embodied in making dynamic multipath work?  And if you are striping flows across multiple paths (absolute requirement in my application), any indication of how successful this is in minimizing the residence time of packets in the reordering function?

DA> If all we are talking about is a slightly smarter ECMP , IMO it does not cut it.

Dave

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Jun 2016 13:05:42 -0700

On 6/3/2016 12:29 PM, David Allan I wrote:
> ...

They might use a different sort of congestion feedback, one that isn't
"TCP-compatible". That's actually desirable here - i.e., having
congestion reaction happen over different timescales than TCP.

> DA> I apologize for not getting your point here.
When you write a buffer to TCP, TCP has no obligation to preserve that
as sitting inside a single TCP segment. TCP does not preserve these
message boundaries.

If your message spans multiple TCP segments, then you depend on BOTH of
them arriving. The probability of losing one or more of two segments is
higher than the probability of losing only one segment. That's the
fate-sharing part - you end up taking a RTT to recover (just like if you
lost the IP packet) when either TCP segment fails to arrive. That loss
recovery increases delay and jitter.

> DA> AFAIK, any loss does that irrespective of the delivery mechanism. In the hybrid access case you are correcting loss with a smaller RTT are you not?
You don't need to correct losses for an IP tunnel. You want to reduce
their probability, but you definitely don't want to stop the stream and
recover. IP is allowed to lose packets. The loss recovery should happen E2E.

If you have high random loss on a path you might employ FEC, which would
increase delay but in a fixed way.

If you really do want ARQ loss recovery on a sub-path, then you can
decide whether to implement that or not. However, you don't want to
force that recovery all the time nor do you want to couple it with
strict reordering - which IP does not require.

> DA> How many did you want?
Maybe you want a variant that isn't already inside MPTCP. The point is
that the load monitoring mechanism isn't configurable or easily replaced.

> DA> If you are saying it is "implementation constrained", well go figure :-) 

No, it's design-constrained. MPTCP isn't designed for plug-replaceable
load detection or traffic distribution. Dynamic multipath routing is
already designed for that.

> DA> Can you point me to the feedback mechanisms embodied in making dynamic multipath work? 

http://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/5212-46.html

> And if you are striping flows across multiple paths (absolute requirement in my application), any indication of how successful this is in minimizing the residence time of packets in the reordering function?
http://www.sigcomm.org/node/2597

> DA> If all we are talking about is a slightly smarter ECMP , IMO it does not cut it.

So you want something dumber than ECMP?

Joe

---

From: David Allan I <[email-masked]>
Date: Fri, 3 Jun 2016 22:03:47 +0000

Hello again:

Replies in line "DA2>" and some snips to eliminate clutter....

<snipped>
> - it will layer congestion control on top of congestion control for 
> all transports in the payload IP that already support it (which they 
> should)

They might use a different sort of congestion feedback, one that isn't "TCP-compatible". That's actually desirable here - i.e., having congestion reaction happen over different timescales than TCP.

DA2> As the tunnel leg would have a different RTT, would that not be true by definition?

> - it fails to preserve payload IP messages inside a single TCP 
> segment, which increases loss due to a lack of fate-sharing and 
> increases latency and jitter
When you write a buffer to TCP, TCP has no obligation to preserve that as sitting inside a single TCP segment. TCP does not preserve these message boundaries.

If your message spans multiple TCP segments, then you depend on BOTH of them arriving. The probability of losing one or more of two segments is higher than the probability of losing only one segment. That's the fate-sharing part - you end up taking a RTT to recover (just like if you lost the IP packet) when either TCP segment fails to arrive. That loss recovery increases delay and jitter.

DA2> Understood, but true of any fragmentation mechanism. 

> DA> AFAIK, any loss does that irrespective of the delivery mechanism. In the hybrid access case you are correcting loss with a smaller RTT are you not?
You don't need to correct losses for an IP tunnel. You want to reduce their probability, but you definitely don't want to stop the stream and recover. IP is allowed to lose packets. The loss recovery should happen E2E.

DA2> unfortunately that sounds like dogma to me. You already have the underlying radio layers correcting loss for the on-air interface portion.

If you have high random loss on a path you might employ FEC, which would increase delay but in a fixed way.

DA2> It takes a little FEC to fix a bit error, it takes a lot to reconstruct a missing packet. The mode of failure for a path is packet loss. So I do not really see FEC as a starter.

If you really do want ARQ loss recovery on a sub-path, then you can decide whether to implement that or not. However, you don't want to force that recovery all the time nor do you want to couple it with strict reordering - which IP does not require.

DA2> No IP does not require ordered delivery from the network. But if it was a way of life rather than a rare event it would be a problem. Poorly done striping of traffic across heterogeneous links with very dynamic transfer characteristics would result in a lot of "out of order" receipt to correct. 

> DA> How many did you want?
Maybe you want a variant that isn't already inside MPTCP. The point is that the load monitoring mechanism isn't configurable or easily replaced.

DA2> Load monitoring or load spreading.  I do not understand why you would want a configurable monitoring function.?

> DA> If you are saying it is "implementation constrained", well go 
> DA> figure :-)

No, it's design-constrained. MPTCP isn't designed for plug-replaceable load detection or traffic distribution. Dynamic multipath routing is already designed for that.

DA2> You are looking for an a la carte menu of functions?

DA2> OK, I see where you are going..... 
<snipped>

DA2> So for mechanisms that age out assignment of load spreading buckets, that is great for the core. But at the edge with very few flows, and the arrival rate I approaching link rate it strikes me that there is only a fairly narrow range of operation in which I could successfully stripe across all uplinks safely. Is that your take as well?

DA2> Now your original assertion was that for MPTCP, something needed to be invented, whereas for dynamic multipath nothing new was required.  From what I have gleaned from this conversation is that both require specification to produce a workable solution.

Dave

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Jun 2016 16:10:41 -0700

> DA2> As the tunnel leg would have a different RTT, would that not be true by definition?

The difference needs to be on significantly different timescales. Sure,
a deployment might work if the tunnel is 1/10 or 1/100 the E2E RTT, but
you can't know that for the tunneled traffic. It's safer to run
congestion feedback on the tunnel at 10x (or some value bigger than) the
largest RTT that can ever occur.

> DA2> Understood, but true of any fragmentation mechanism. 

We're not talking about fragmentation. Even if TCP runs over IPv6 and
has 1200+B segments, there's absolutely no guarantee that a sequence of
576B IPv4 packets will each occur solely inside one TCP segment. TCP can
- and will - pack, straddle, or do anything else it wants with the
buffers it is given.
> DA2> unfortunately that sounds like dogma to me. You already have the underlying radio layers correcting loss for the on-air interface portion.
Some radios do, some do not. Packets are lost in a lot of different
places, including routers.

The fact that IP is "best effort" isn't dogma; it's the one of its
defining properties.

> DA2> It takes a little FEC to fix a bit error, it takes a lot to reconstruct a missing packet. The mode of failure for a path is packet loss. So I do not really see FEC as a starter.
Some paths do employ FEC using block codes that are tolerant to burst
losses - which is what packet loss along the tunnel would look like. But
sure, it's up to you whether you need that level of recovery or not.

> DA2> No IP does not require ordered delivery from the network. But if it was a way of life rather than a rare event it would be a problem. Poorly done striping of traffic across heterogeneous links…

First, anything that uses IP and treats misordering as an error is
itself broken. IP is best effort and doesn't guarantee in-order
delivery. However, as noted, there are mechanisms to reduce the
reordering at the egress collection point, esp. for striped paths.

> DA2> Load monitoring or load spreading.  I do not understand why you would want a configurable monitoring function.?

There are a lot of ways of measuring load and a lot of ways of
interpreting those measurements. The variant in MPTCP represents one
point in that space. It might not be the one you want - you might want
to detect congestion in groups of tunnels as sets rather than as
individual paths, e.g.
> DA2> You are looking for an a la carte menu of functions?

Not necessarily; I'm saying that MPTCP comes bundled with one solution
that fits its use for two TCP endpoints. That need not be the best
solution when used as a tunnel. And - more to the point - we already
have a wide variety of those solutions.

> DA2> So for mechanisms that age out assignment of load spreading buckets, that is great for the core. But at the edge with very few flows, and the arrival rate I approaching link rate it strikes me…

The way in which the routing protocols are tuned for core vs edge
operation may certainly vary. The routing protocol solution gives you
exactly the flexibility to make that decision and adjust as needed.

> DA2> Now your original assertion was that for MPTCP, something needed to be invented, whereas for dynamic multipath nothing new was required.  From what I have gleaned from this conversation is tha…

 MPTCP needs a spec to indicate how to tunnel. The existing solution
uses existing tunnel mechanisms and existing routing protocols, but
needs no new spec.

Both then need to be deployed, and yes - that's work.

Joe

---

From: David Allan I <[email-masked]>
Date: Fri, 3 Jun 2016 23:40:11 +0000

Hi 

<snipped>
> DA2> As the tunnel leg would have a different RTT, would that not be true by definition?

The difference needs to be on significantly different timescales. Sure, a deployment might work if the tunnel is 1/10 or 1/100 the E2E RTT, but you can't know that for the tunneled traffic. It's safer to run congestion feedback on the tunnel at 10x (or some value bigger than) the largest RTT that can ever occur.

DA3> I think we are talking past each other here. I'm not sure quite what you're arguing other than it is not TCP with the rationale being timescales.  IMO the largest RTT that can occur is irrelevant (and I believe you are discussing E2E). And the ratio of E2E to tunnel is also a don't care as near as I can tell.  My best guess is you think TCP RTT measurement timescales for the tunnel itself is inadequate. Is that correct?

> DA2> Understood, but true of any fragmentation mechanism. 

We're not talking about fragmentation. Even if TCP runs over IPv6 and has 1200+B segments, there's absolutely no guarantee that a sequence of 576B IPv4 packets will each occur solely inside one TCP segment. TCP can
- and will - pack, straddle, or do anything else it wants with the buffers it is given.
DA3> OK, will park this one for now.

> DA2> unfortunately that sounds like dogma to me. You already have the underlying radio layers correcting loss for the on-air interface portion.
Some radios do, some do not. Packets are lost in a lot of different places, including routers.

The fact that IP is "best effort" isn't dogma; it's the one of its defining properties.
DA3> Therefore in your world a link or path that provides a reliable transport service is not allowed.  I would say at best you could argue that it is redundant. I'd simply describe it as an artifact of other technical choices.

<snipped>
> DA2> No IP does not require ordered delivery from the network. But if it was a way of life rather than a rare event it would be a problem. Poorly done striping of traffic across heterogeneous links…

First, anything that uses IP and treats misordering as an error is itself broken. IP is best effort and doesn't guarantee in-order delivery. However, as noted, there are mechanisms to reduce the reordering at the egress collection point, esp. for striped paths.
DA3> It may not treat it as an error, but it does treat it as far more than a minor inconvenience.  Ask the vendor whose initial OC-192 interfaces did not provide ordered transmission. (dumb striping across concatenated OC48s)

<snipped>

> DA2> You are looking for an a la carte menu of functions?

Not necessarily; I'm saying that MPTCP comes bundled with one solution that fits its use for two TCP endpoints. That need not be the best solution when used as a tunnel. And - more to the point - we already have a wide variety of those solutions.

DA3> Ahh, then if I understand you, you are objecting to the need for functionality at both ends of the tunnel vs. just decap at one end.

<snipped>
> DA2> So for mechanisms that age out assignment of load spreading buckets, that is great for the core. But at the edge with very few flows, and the arrival rate I approaching link rate it strikes me…

The way in which the routing protocols are tuned for core vs edge operation may certainly vary. The routing protocol solution gives you exactly the flexibility to make that decision and adjust as needed.

DA3> I'm not sure why you keep calling it a routing protocol solution when it is simply forwarding and some ongoing supervision to tell you when striping is not a good idea. 

 MPTCP needs a spec to indicate how to tunnel. The existing solution uses existing tunnel mechanisms and existing routing protocols, but needs no new spec.

DA3> It sounds like it also needs "experience" to know how to tune it. 

DA3> Agreed

Have a good weekend
Dave

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Jun 2016 16:50:20 -0700

> DA3> I think we are talking past each other here. I'm not sure quite what you're arguing other than it is not TCP with the rationale being timescales.  IMO the largest RTT that can occur is irrelev…

I think that if the routing algorithm uses TCP timescales to shift and
the E2E traffic uses TCP timescales to backoff, the two will interact in
bad ways.

When I say "RTT" for the tunnel congestion, I should have been more
clear - I'm talking about the timescales on which the traffic load
balancing picks different paths. That's not RTT, but it's a similar kind
of cycle time.

...

> DA3> Therefore in your world a link or path that provides a reliable transport service is not allowed.  I would say at best you could argue that it is redundant. I'd simply describe it as an artifa…

My view is that tunnel ARQ is an expensive choice and applicable in only
a very specialized set of environments. I don't see that sort of
specific environment being the intended use case for this approach.

...
> DA3> It may not treat it as an error, but it does treat it as far more than a minor inconvenience.  Ask the vendor whose initial OC-192 interfaces did not provide ordered transmission. (dumb stripi…

OC192 provides SONET service, where reordering is not permitted (at
least not outside the jitter specified for a connection).

IP is a different service with different expectations.

> DA3> Ahh, then if I understand you, you are objecting to the need for functionality at both ends of the tunnel vs. just decap at one end.

There's functionality at both ends of the tunnel in both approaches.

I'm objecting to the use of TCP (any form of TCP) to tunnel IP traffic.

> DA3> It sounds like it also needs "experience" to know how to tune it. 

First, we already have a lot of that experience (this isn't anywhere
near a new issue).

Second, at least you can tune the dynamic routing variant. MPTCP doesn't
have that option unless you rewrite that spec too.

> Both then need to be deployed, and yes - that's work.

+1 and cheers...

Joe

---

From: David Allan I <[email-masked]>
Date: Sat, 4 Jun 2016 00:46:52 +0000

And when I thought I was done for the weekend :-) I've enjoyed the discussion, so just one more :-) 

> DA3> I think we are talking past each other here. I'm not sure quite what you're arguing other than it is not TCP with the rationale being timescales.  IMO the largest RTT that can occur is irrelev…

I think that if the routing algorithm uses TCP timescales to shift and the E2E traffic uses TCP timescales to backoff, the two will interact in bad ways.

DA4> I can understand that for dissimilar algorithms (like TCP over ATM-ABR).

When I say "RTT" for the tunnel congestion, I should have been more clear - I'm talking about the timescales on which the traffic load balancing picks different paths. That's not RTT, but it's a similar kind of cycle time.

DA4> In the context of the solution you've been discussing, there is measurement time, and the arrival rate which can be striped when compared to the delta of individual path latencies. I would have to believe you need to measure that frequently in the context that an ordering guarantee is a single ended function with no redress.

DA4> That is where I would lean to a double ended function, I can actually offer the guarantee , and can stripe traffic across a greater disparity in path throughput vs. shutting striping down past a certain threshold in disparity. Which when considering how asymmetric DSL is, becomes a rather large benefit IMO.

...

> DA3> Therefore in your world a link or path that provides a reliable transport service is not allowed.  I would say at best you could argue that it is redundant. I'd simply describe it as an artifa…

My view is that tunnel ARQ is an expensive choice and applicable in only a very specialized set of environments. I don't see that sort of specific environment being the intended use case for this approach.
DA4> Well for your approach, yes. As per my summary above

...
> DA3> It may not treat it as an error, but it does treat it as far more 
> DA3> than a minor inconvenience.  Ask the vendor whose initial OC-192 
> DA3> interfaces did not provide ordered transmission. (dumb striping 
> DA3> across concatenated OC48s)

OC192 provides SONET service, where reordering is not permitted (at least not outside the jitter specified for a connection).

DA4> You missed my point, there was a router that misordered packets and caused much unhappiness. All the deployed blades needed to be replaced IIRC. Yes end systems will tolerate it, but not as a way of life.

DA4> I gathered that!

> DA3> It sounds like it also needs "experience" to know how to tune it. 

First, we already have a lot of that experience (this isn't anywhere near a new issue).

DA4> SO I understand :-)

Second, at least you can tune the dynamic routing variant. MPTCP doesn't have that option unless you rewrite that spec too.

DA4> Any HA solution will have policy as the economics of using each interface are different. So both solutions will have overall very similar options when the dust settles.

+1 and cheers...

DA4> cheers as well
Dave

---

From: Jordan Melzer <[email-masked]>
Date: Sat, 4 Jun 2016 06:39:07 -0600

Thanks, David, Joe.  That was very interesting to read through.

I'm also from an operator interested in deploying MPTCP as an access multipath mechanism.  I am concerned about breaking user applications, and I do want to highlight from the discussion how many assumptions there are on IP traffic behaviour.

As David argued, while IP doesn't guarantee traffic is delivered in order, Internet users and applications largely do expect it to be mostly delivered in order, and if it isn't, they don't guarantee to work.  As cool as FLARE is at taking advantage of TCP burstiness to be able to multipath it, its assumptions sound very fragile: it can work for TCP as an underlying protocol, when TCP is bursty.  So... it can work for some protocols, sometimes.  And what is FLARE doing?  It's trying to reduce the number of out of order IP packets when one is multipathing.

Joe, however, raised the timing of packet delivery as an issue.  Running TCP over TCP can be problematic despite TCP being able to deliver IP packets in order.  If the tunneled layer is more aggressive about retransmission than the base layer, the tunnel breaks.

As we know, size also matters.  MTU is expected to be consistent, transparent, and generous.  I don't have a good handle on what the arguments were for each solution with respect to MTU challenges.

I'm not convinced anything proposed is a good general method for transporting UDP flows across links with different characteristics.  I don't know what is happening inside the UDP flow and if I will be violating any of the unwritten assumptions written above.  Am I overly concerned, and, if so, about what?

I would consider using either approach for some subset of traffic driven by a bunch of semi-opaque policy because I work for an ISP, but I don't know if I would want to force a user's VPN onto any aggregated link.  Please convince me I do.

Regards,
Jordan

-----Original Message-----
From: multipathtcp [mailto:[email-masked]] On Behalf Of David Allan I
Sent: June 3, 2016 08:47 PM
To: Joe Touch; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

...

...

DA4> SO I understand :-)

+1 and cheers...

DA4> cheers as well
Dave

---

From: "Henderickx, Wim (Nokia - BE)" <[email-masked]>
Date: Sun, 5 Jun 2016 12:52:21 +0000

I believe that we are overlooking one major thing in this discussion. The plain MPTCP solution proposes a full proxy at different points, meaning it reinitiates a TCP/MP-TCP session at various points to accommodate the various TCP timers, etc
The draft requires each proxy to compensate for various delays in various segments. This has is deployed in various networks, mainly mobile to improve QoE today w/o MP-TCP. The plain MP-TCP enhances this solution and allows flows which are bigger than a given path to still work, whereas today they break.
Example here

EndHost (TCP) <——> (TCP) CPE (MP-TCP proxy) <——>  (MP-TCP proxy) Concentrator) (TCP) <——> App

As such the CPE and concentrator have to compensate for the various delays/feedbacks in the protocol stack.

---

From: Joe Touch <[email-masked]>
Date: Sun, 5 Jun 2016 13:49:31 -0700

> DA4> I can understand that for dissimilar algorithms (like TCP over ATM-ABR).

Actually, similar algorithms layered over each other have already been
shown to interact in bad ways - for TCP in particular.

> DA4> In the context of the solution you've been discussing, there is measurement time, and the arrival rate which can be striped when compared to the delta of individual path latencies. I would hav…

There's the measurement time, the time to make a decision, and the
hysteresis needed to avoid route flapping. And you can tune each of
those independently when you use dynamic routing, but it's not as easily
changed for MPTCP.

> DA4> You missed my point, there was a router that misordered packets and caused much unhappiness. All the deployed blades needed to be replaced IIRC. Yes end systems will tolerate it, but not as a…

A system that runs IP but does not tolerate reordering as a "way of
life" isn't IP. If there's a cost at any level, it needs to be adapted
at the endpoint (install some playout buffers and do local reordering).
Expecting in-order delivery should never be a requirement of IP.

However, as noted, there are ways to reduce the amount of persistent
reordering in multipath routing already. I cited one of them; there are
others.

...
>> DA3> I'm not sure why you keep calling it a routing protocol solution when it is simply forwarding and some ongoing supervision to tell you when striping is not a good idea.

Sorry I overlooked this one - I call it a routing protocol solution
because the routing protocol is the distributed algorithm that detects
path properties and decides how to configure forwarding accordingly.
That's what routing is. Forwarding is what the router does with incoming
packets based on the configuration of its forwarding table based on the
routing protocol. (A routing table is the data used by the routing
protocol to coordinate the algorithm and derive the forwarding table).

> DA4> Any HA solution will have policy as the economics of using each interface are different. So both solutions will have overall very similar options when the dust settles.

Both systems can be extended to have identical capabilities - except for
the part about segment boundaries and requiring ARQ stalls to fill in
dropped packets that MPTCP incurs - but dynamic routing solutions
already have all of these features now. Extending MPTCP to provide those
capabilities is just reinventing the wheel.

Joe

---

From: <[email-masked]>
Date: Wed, 8 Jun 2016 09:53:56 +0000

I believe you're right that such work would be outside the charter.

We're hoping (if we get 2 sessions) in Berlin to have some discussion about potential re-chartering. In practice we haven't managed to get much work focussed specifically on meeting the current charter item on proxy (there was some individual drafts a while back) - but there seems to be quite a bit of industry work on proxies (beyond charter's single-ended proxy). So potential discussion topic 

>From my perspective there seems to be some operator interest in hybrid. See for instance the discussions in banana bar-bofs we've had at last couple of ietfs - including description of the DT deployment. (note, not all hybrid solutions use mptcp)

Best wishes
phil

-----Original Message-----
From: multipathtcp [mailto:[email-masked]] On Behalf Of Behcet Sarikaya
Sent: 01 June 2016 19:38
To: [email-masked]
Subject: [multipathtcp] draft-boucadair-mptcp-plain-mode-07

 Hi all,

Recent mails in this thread came to my attention because I work on the hybrid access network issues and wish to convey my views on some of the issues discussed.

First of all, the claims that there is strong operator interest on this is over blown. Hybrid access is based on offloading some CPE traffic to 3G/LTE network, I don't think mobile operators are fond of such a thing.

Also I strongly concur with Yoshi's point of CPE to convert UDP or TCP into MPTCP and requires Concentrators to convert back MPTCP into UDP or TCP is a complex process. The crux of this is the problems arising from the use of MPTCP for tunneling, which is what this draft is advocating.

Here I recommend Joe Touch's Intarea draft https://tools.ietf.org/id/draft-ietf-intarea-tunnels-02.txt
on IP Tunnels in the Internet Architecture. IETF developed many tunneling protocols which are surveyed in this draft but none of them are TCP/MPTCP based.
I also question suitability of this work to the charter. There is this charter text:
Finally, the working group will explore whether an MPTCP-aware
  middlebox would be useful, where at least one end host is MPTCP-enabled.
  For example, potentially helping MPTCP's incremental deployment by
  allowing only one end host to be MPTCP-enabled and the middlebox acts as
  an MPTCP proxy for the other end host, which runs TCP; and potentially
  helping some mobility scenarios, where the middlebox acts as an anchor
  between two MPTCP-enabled hosts. The working group will detail what real
  problems an MPTCP-enabled middlebox might solve, how it would impact the
  Multipath TCP architecture (RFC6182), what proxy approach might be
  justified as compared against alternative solutions to the problems, and
  the likely feasibility of solving the technical and security issues.

This draft has a middlebox called Concentrator but it is not trying to enable MPTCP hosts to communicate with TCP hosts, it is just tunneling the traffic. So there is no charter item that fits to this draft.
I strongly suggest staying away from making MPTCP a tunneling protocol.

Regards,

Behcet

---

From: Mingui Zhang <[email-masked]>
Date: Fri, 1 Jul 2016 02:47:47 +0000

Hi,

In the paired MPTCP proxies mechanism, there is user's TCP session and there is also multipath TCP session. It is a TCP over TCP idea, correct? 
As for TCP over TCP, the following page explains why an internal meltdown effect would take place due to the interaction between two leveled retransmission mechanisms. 
http://sites.inka.de/bigred/devel/tcp-tcp.html
Do we have solid solutions for this issue nowadays?

Thanks,
Mingui

> From: multipathtcp [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Wednesday, June 08, 2016 5:54 PM
> To: [email-masked]; [email-masked]
> We're hoping (if we get 2 sessions) in Berlin to have some discussion about
> potential re-chartering. In practice we haven't managed to get much work
> focussed specifically on meeting the current charter item on proxy (there was
> some individual drafts a while back) - but there seems to be quite a bit of
> industry work on proxies (beyond charter's single-ended proxy). So potential
> discussion topic
> 
> >From my perspective there seems to be some operator interest in hybrid. See
> for instance the discussions in banana bar-bofs we've had at last couple of ietfs -
> including description of the DT deployment. (note, not all hybrid solutions use
> mptcp)
> 
> Best wishes
> phil
> 
> Recent mails in this thread came to my attention because I work on the hybrid
> access network issues and wish to convey my views on some of the issues
> discussed.
> 
> First of all, the claims that there is strong operator interest on this is over blown.
> Hybrid access is based on offloading some CPE traffic to 3G/LTE network, I don't
> think mobile operators are fond of such a thing.
> 
> Also I strongly concur with Yoshi's point of CPE to convert UDP or TCP into MPTCP
> and requires Concentrators to convert back MPTCP into UDP or TCP is a complex
> process. The crux of this is the problems arising from the use of MPTCP for
> tunneling, which is what this draft is advocating.
> 
> on IP Tunnels in the Internet Architecture. IETF developed many tunneling
> protocols which are surveyed in this draft but none of them are TCP/MPTCP
> based.
> This draft has a middlebox called Concentrator but it is not trying to enable
> MPTCP hosts to communicate with TCP hosts, it is just tunneling the traffic. So
> there is no charter item that fits to this draft.