Subject: Re: [multipathtcp] A New Draft: draft-ayar-transparent-sca-proxy-00

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Fri, 9 Mar 2012 08:23:20 -0600

Hi Tacettin et al,

I read your draft. I have the following questions & comments:

1) What specific use case(s) do you have in mind?

2) In my opinion, SCAPs logically operate on L4 since they mess around in connection-specific flow properties based on SEQ numbers. You may implement it on a lower layer but that's a different issue.

3) Why don't you simply use MPTCP? 

Thanks.

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, February 20, 2012 9:01 AM
To: [email-masked]
Subject: [multipathtcp] A New Draft: draft-ayar-transparent-sca-proxy-00

Hi all,

We have posted a new draft with the title: "A Transparent Performance  
Enhancing Proxy Architecture To Enable TCP over Multiple Paths for  
Single-Homed Hosts"  
(http://tools.ietf.org/html/draft-ayar-transparent-sca-proxy-00).

In this draft, we define a TCP Splitter/Combiner Architecture (SCA)  
that enables non-MPTCP-capable single-homed hosts to benefit from the  
multiple paths within Internet by means of transparent performance  
enhancing proxies (PEPs). Since existence of the SCA is shielded from  
the TCP end-points, it can also be deployed on the TCP end-hosts.

We invite everybody to read the draft and look forward to your comments.

Thanks,

Tacettin

_______________________________________________
multipathtcp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sun, 11 Mar 2012 01:54:46 -0800

Hi Ayar,

Sorry for the delay. I've finally found that time to read your draft.
The following points are not very clear to me. It would be great if
you could clarify.

1: It's not very clear to me what is the advantage of this protocol.
    I think this approach presumes to utilize multiple paths between routers.
    hence, it looks to me we might be able to do the similar things by
using some kinds of multipath routing between these routers.
    Or, if we run mptcp between two proxies, we might also be able to
do similar thing.
    could you elaborate the feature of this approach compared to other approach?

2: I think we'll need to know what kinds of signaling are used between
SCA proxies.
    For example, how a SCA can find other SCAs?

3: It's not clear to me how it is useful when there's only one SCA or
there are more than 2 SCAs on the path. Could you describe some
scenarios for them?

Thanks,
--
Yoshifumi Nishida

On Mon, Feb 20, 2012 at 6:01 AM,  <[email-masked]> wrote:
> In this draft, we define a TCP Splitter/Combiner Architecture (SCA) that
> enables non-MPTCP-capable single-homed hosts to benefit from the multiple
> paths within Internet by means of transparent performance enhancing proxies
> (PEPs). Since existence of the SCA is shielded from the TCP end-points, it
> can also be deployed on the TCP end-hosts.
>
>

---

From: Tacettin Ayar <[email-masked]>
Date: Thu, 15 Mar 2012 16:03:14 +0100

Dear Georg,

please find the answers to your questions inline.

Best Regards,

Tacettin

> 1) What specific use case(s) do you have in mind?

We have the following use cases in mind:

1)Bandwidth Aggregation: As mentioned in Section 1.3, SCAPs may be
used in access networks (e.g., wireless mesh network backbone, WLAN or  
femtocells) that
are connected to the Internet via bottleneck gateway links (e.g., DSL  
lines or T1), see figure
1. In that use case, SCAPs will aggregate the capacities of the GW
links.

2)Load Balancing: As mentioned in Section 4.2, SCAPs may be used to
distribute traffic via multiple paths within an access network
(Figure 4) for load balancing.

3)Redundancy: The same TCP segment may be sent via multiple paths
to create redundancy in networks with high-error rates. For
example, Pipe-1 and Pipe-2 in Figure 4 may be used to send the same
segment if access network has high packet loss rates.

4)Avoiding bottlenecks: Since SCAPs are aware of TCP flows in the
network they may redirect TCP traffic to other available paths to  
avoid bottlenecks.

>
> 2) In my opinion, SCAPs logically operate on L4 since they mess  
> around in connection-specific flow properties based on SEQ numbers.  
> You may implement it on a lower layer but that's a different issue.

Indeed, SCAPs operate on L3.5: not touching/replacing IP and TCP,
and getting their input as TCP segments.

> 3) Why don't you simply use MPTCP?

Since MPTCP is designed to be used on multi-homed end-systems!
Scalability problems might occur when MPTCP is used on proxies. For
example, Section 5.3 of RFC 6182 mentions a case in which 12.5 MB
receive buffer space is needed. On end-hosts, it may be affordable.
But, on network devices with lots of TCP flows, processing and space
overhead MUST NOT be ignored.

---

From: Tacettin Ayar <[email-masked]>
Date: Thu, 15 Mar 2012 17:54:04 +0100

Dear Yoshifumi,

please find our answers to your comments inline.

Best Regards,

Tacettin et al

>     could you elaborate the feature of this approach compared to  

Multi-path routing can not fit all the needs of TCP over multiple  
paths since performance of TCP flows is affected by the use of  
multiple-paths (e.g., because of out-of-order packets). Thus, TCP  
flows must be identified and side-effects of multi-path routing must  
be cleared.

Use of MPTCP between the proxies is not a good idea either, as MPTCP  
is designed for TCP end-hosts. When used on proxies (with lots of TCP  
flows) it will introduce too much overhead.

>     For example, how a SCA can find other SCAs?

SCAPs may use signaling for many purposes. For example, to probe for  
other SCAPs, to inform the SCAP peer about other IP addresses of the  
SCAP device, to send a series of packets to probe for pipe  
characteristics (e.g., delay, capacity, loss rate), etc.

There are also many available options for signaling between SCAPs. For  
example, to find another SCAP a SCAP may: (1) flood SCAP peer probe  
packets, (2) select some links for sending the SCAP probe packets  
(e.g., using a multipath routing protocol), or (3) send the probe  
packet only via the default TCP path, to name a few. However, we  
haven't yet analyzed all these options in detail and they may be  
subject to an extension of our draft. We believe that it is important  
to agree on the architecture and get a broader support for it, before  
going to the details of the signaling, which we obviously are willing  
to deliver.

Addressing your question, what we mentioned in our draft (Section 3.5)  
is that signaling may be in-band, out-of-band, or hybrid. As an  
example, in-band signaling may be used to probe for other SCAPs as  
follows: a SCAP that receives a TCP connection request packet (packet  
with SYN) adds a TCP option (e.g., SCAP-PROBE) to the packet to signal  
its existence and address. Assume that the packet is sent via only  
default TCP path. The packet will then follow the default TCP path  
with SCAP-PROBE option. If there is another SCAP on the path, it will  
recognize the SCAP-PROBE TCP option and obtain the address of the peer  
SCAP that generated it in order to establish the contact.  If there is  
no SCAP on the path, then TCP receiver will ignore the unrecognized  
option.

The mentioned method is similar to the MPTCP, which uses MP_CAPABLE  
option to look whether the peer supports MPTCP or not. Of course,  
vulnerabilities exist (e.g., middleboxes that drop packets with  
unknown TCP options or remove the unknown options from the packets),  
same as for MPTCP.

> there are more than 2 SCAs on the path. Could you describe some

If there is only one SCAP between TCP sender and receiver, then we  
may: (1) not use SCAP and rely on default single-path TCP or (2) use  
SCAP to distribute TCP traffic over multiple paths. In the latter case  
we have to consider the side-effects of the distribution (combining is  
done at the receiver) . For example, Simula Proxy [1] uses path delay  
estimations to calculate packet arrival times and buffer the  
potentially out-of-order packets on the shorter paths, compensating  
the different path delays. DUPACK estimation and filtering (DEF) [2]  
uses path delays to estimate number of duplicate acknowledgements  
caused by different path delays and shields them from the TCP sender.

[1] K. Evensen, D. Kaspar, P. Engelstad, A. F. Hansen, C. Griwodz, and  
P. Halvorsen, "A Network-Layer Proxy for Bandwidth Aggregation and  
Reduction of IP Packet Reordering," IEEE 34th Conference on Local  
Computer Networks (LCN 2009), pp. 585?592, October 2009.

[2] T. Ayar, B. Rathke, L. Budzisz, and A. Wolisz, "TCP over Multiple  
Paths Revisited: Towards Transparent Proxy Solutions," Accepted by The  
IEEE International Conference on Communications 2012 (IEEE ICC'12),  
June 2012.

Use of two or more SCAPs is open to discussion. We present two  
scenarios in which use of more than two SCAPs is beneficial.

-----------
SCENARIO 1:
-----------
Consider the following figure. There are two pipes between SCAP-1 and  
SCAP-2: (Pipe1 passes through Path2 and Path4 and Pipe2 passes through  
Path3 and Path5).

In this case capacity of Pipe1 is limited by the capacity of Path4  
(i.e., 1) and capacity of Pipe2 is limited by the capacity of Path3  
(i.e., 1). The aggregate capacity that may be used by the connection  
is then 2.

+----+       +----+       +----+       +----+       +----+
|    |       |    |-------|    |-------|    |       |    |
|    |       |    |Path2=4|    |Path4=1|    |       |    |
|    |-------|    |-------|    |-------|    |-------|    |
|TCP |Path1=5|SCAP|       |ROU |       |SCAP|Path6=5| TCP|
|Host|-------|  1 |-------|TER |-------|  2 |-------|Host|
|    |       |    |Path3=1|    |Path5=4|    |       |    |
|    |       |    |-------|    |-------|    |       |    |
+----+       +----+       +----+       +----+       +----+

Now, assume that router is used as a SCAP device, as shown in the next  
figure. In that case, SCAP-1 and SCAP-3 use two pipes: Pipe1=Path2 and  
Pipe2=Path3. Since SCAP-3 combines packets from these two pipes, There  
is aggregated capacity of 5 between SCAP-1 and SCAP-3. Similarly, The  
aggregated capacity between SCAP-3 and SCAP-2 is also 5. Thus,  
connection may use a capacity of 5.

+----+       +----+       +----+       +----+       +----+
|    |       |    |-------|    |-------|    |       |    |
|    |       |    |Path2=4|    |Path4=1|    |       |    |
|    |-------|    |-------|    |-------|    |-------|    |
|TCP |Path1=5|SCAP|       |SCAP|       |SCAP|Path6=5| TCP|
|Host|-------|  1 |-------|  3 |-------|  2 |-------|Host|
|    |       |    |Path3=1|    |Path5=4|    |       |    |
|    |       |    |-------|    |-------|    |       |    |
+----+       +----+       +----+       +----+       +----+

-----------
SCENARIO 2:
-----------
Now consider the following figure. SCAP-1 and SCAP-2 both have two  
interfaces and may use the upper interface pair as a pipe (Pipe1 which  
passes through Path2<->Path4<->Path6 or Path2<->Path9<->Path6) and  
lower interface pair as a pipe (Pipe2 which passes through  
Path3<->Path5<->Path7 or Path3<->Path10<->Path7).

In this case capacity of Pipe1 is limited by the capacity of Path4 or  
Path9 (i.e., 1) and capacity of Pipe2 is limited by the capacity of  
Path5 or Path10 (i.e., 1), whichever path is used between the routers.  
The capacity that may be used by the connection is 2.

                                     +---------------------+
                                     |       Path9=1       |
                                     |   +-------------+   |
                                     |   |             |   |
+------+         +------+         +-------+         +-------+          
+------+         +------+
|      |         |      |---------|       |---------|        
|---------|      |         |      |
|      |         |      | Path2=2 |       | Path4=1 |       | Path6=2  
|      |         |      |
|      |---------|      |---------|       |---------|        
|---------|      |---------|      |
| TCP  | Path1=4 | SCAP |         |ROUTER |         |ROUTER |          
| SCAP | Path8=4 |  TCP |
| Host |---------|   1  |---------|       |---------|        
|---------|   2  |---------| Host |
|      |         |      | Path3=2 |       | Path5=1 |       | Path7=2  
|      |         |      |
|      |         |      |---------|       |---------|        
|---------|      |         |      |
+------+         +------+         +-------+         +-------+          
+------+         +------+
                                     |   |             |   |
                                     |   +-------------+   |
                                     |      Path10=1       |
                                     +---------------------+

Now, assume that routers are used as SCAP devices, as shown in the  
following figure. In that case, SCAP-1 and SCAP-3 use two pipes:  
Pipe1=Path2 and Pipe2=Path3. Since SCAP-3 combines packets from these  
two pipes, There is aggregated capacity of 4 between SCAP-1 and  
SCAP-3. Similarly, SCAP-3 and SCAP-4 use four pipes: Pipe1=Path9,  
Pipe2=Path4, Pipe3=Path5 and Pipe4=Path10. The aggregated capacity  
between SCAP-3 and SCAP-4 is 4. The same procedure applies for SCAP-4  
and SCAP-2. The capacity that may be used by the connection is now 4.

                                     +---------------------+
                                     |       Path9=1       |
                                     |   +-------------+   |
                                     |   |             |   |
+------+         +------+         +-------+         +-------+          
+------+         +------+
|      |         |      |---------|       |---------|        
|---------|      |         |      |
|      |         |      | Path2=2 |       | Path4=1 |       | Path6=2  
|      |         |      |
|      |---------|      |---------|       |---------|        
|---------|      |---------|      |
| TCP  | Path1=4 | SCAP |         | SCAP  |         | SCAP  |          
| SCAP | Path8=4 |  TCP |
| Host |---------|   1  |---------|   3   |---------|   4    
|---------|   2  |---------| Host |
|      |         |      | Path3=2 |       | Path5=1 |       | Path7=2  
|      |         |      |
|      |         |      |---------|       |---------|        
|---------|      |         |      |
+------+         +------+         +-------+         +-------+          
+------+         +------+
                                     |   |             |   |
                                     |   +-------------+   |
                                     |      Path10=1       |
                                     +---------------------+

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 16 Mar 2012 14:48:34 +0200

Hi,

> overhead MUST NOT be ignored.

The statement above not correct. 

Two important points:

1) MPTCP can also work between single-homed hosts as long as the network implements Equal Cost Multipath or something similar. Here, MPTCP will open multiple subflows between the same pair of IPs, and these subflows will be placed (probabilistically) by the network on different paths.

2) MPTCP's receive buffer usage: fundamentally MPTCP (or any other multipath transport) will have to use more receive buffer to account for the different delays on the paths it uses. That is also true for any solution you might come up with - unless you are willing to tolerate sending a lot of reordered packets to the destination, which will then kill throughput and nullify any performance benefits you have from using your solution.

In particular, if MPTCP works on paths with similar RTTs, its receive buffer is equal to the sum of the receive buffers TCP would use on those same paths .

If MPTCP works on paths with (highly) disimilar RTTs, then its buffering requirements become problematic (the exact formula is max_RTT * sum(bw)). 

We have created a bunch of algorithms that we have already put in the Linux implementation (at the sender) that guarantee that:
a) when MPTCP is underbuffered, it still does at least as well as TCP on the best path with the same amount of buffer
b) as more buffer becomes available, throughput gains due to MPTCP increase.

These algorithms are presented in our NSDI 2012 paper called "How Hard Can It Be? Designing and implementing a deployable Multipath TCP". You can find the paper here: https://nets.cs.pub.ro/~costin/files/nsdi12-final125.pdf

I also plan to present a quick summary of these algorithms during the MPTCP WG meeting in Paris.

Cheers,
Costin

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Fri, 16 Mar 2012 08:32:17 -0500

Tacettin,

I went through ref [1] you provided. It would be helpful if you could send ref [2] on the ML. I didn't find it on the Internet.

Apart from signaling capabilities, I'm trying to understand the principal differences between your SCAP proposal and MPTCP.

If I understand it correctly SCAP performs "multipath routing with sub-flow granularity" opposed to MPTCP performing "multipath transport". 

SCAP steers packets along different paths by setting the packet IPs/ports accordingly. MPTCP also applies a separate Seq No space + independent TCP signaling (SYN/ACK handshake etc) to each path.

SCAP is simpler than MPTCP but it is not middle-box compliant.

Both solutions can be implemented on L3. We currently implement MPTCP on L3 for instance.

Both solutions can be realized on end-hosts, proxies or a combination of both.

Both solutions can support "proxy" discovery, concatenated pipes, etc, provided proper signaling. 

Are these observations correct? If not what am I missing? Thanks.

Regards,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tacettin Ayar
Sent: Thursday, March 15, 2012 12:54 PM
To: Yoshifumi Nishida
Cc: [email-masked]
Subject: Re: [multipathtcp] A New Draft: draft-ayar-transparent-sca-proxy-00

Dear Yoshifumi,

Best Regards,

Tacettin et al

-----------
SCENARIO 1:
-----------

-----------
SCENARIO 2:
-----------

                                     +---------------------+
                                     |       Path9=1       |
                                     |   +-------------+   |
                                     |   |             |   |

                                     +---------------------+
                                     |       Path9=1       |
                                     |   +-------------+   |
                                     |   |             |   |

---

From: Tacettin Ayar <[email-masked]>
Date: Fri, 16 Mar 2012 15:08:43 +0100

Hi Costin,

my comments are as follows.

Best,

Tacettin

Quoting Costin Raiciu <[email-masked]>:

> 1) MPTCP can also work between single-homed hosts as long as the  
> network implements Equal Cost Multipath or something similar. Here,  
> MPTCP will open multiple subflows between the same pair of IPs, and  
> these subflows will be placed (probabilistically) by the network on  
> different paths.

It is just what we say! Something must be done within the NETWORK to  
supply multiple paths.

>
> 2) MPTCP's receive buffer usage: fundamentally MPTCP (or any other  
> multipath transport) will have to use more receive buffer to account  
> for the different delays on the paths it uses. That is also true for  
> any solution you might come up with - unless you are willing to  
> tolerate sending a lot of reordered packets to the destination,  
> which will then kill throughput and nullify any performance benefits  
> you have from using your solution.
>
> In particular, if MPTCP works on paths with similar RTTs, its  
> receive buffer is equal to the sum of the receive buffers TCP would  
> use on those same paths .
>
> If MPTCP works on paths with (highly) disimilar RTTs, then its  
> buffering requirements become problematic (the exact formula is  
> max_RTT * sum(bw)).
>

I read your discussion about it in RFC 6182 (the section that I gave  
to reference) and I have no objection to that.

For this reason, we mention a select_pipes() method (Section 3.3) and  
pipe selection policy (Section 3.6) in our draft.

As you said. When the paths with similar delays are used, then the  
gaps in the receive buffer will be filled quickly and it will not be a  
problem! Only, an inevitable overhead occurs when a packet is lost and  
retransmitted.

Indeed, we are not against an MPTCP proxy! But, what we say is that,  
MPTCP proxy must consider these situations. For example, when the  
available paths have very dissimilar delays, then MPTCP SCAP should  
not admit them. An admission policy (Section 3.6) should be defined  
that is suitable for MPTCP.

> We have created a bunch of algorithms that we have already put in  
> the Linux implementation (at the sender) that guarantee that:
> a) when MPTCP is underbuffered, it still does at least as well as  
> TCP on the best path with the same amount of buffer
> These algorithms are presented in our NSDI 2012 paper called "How  
> Hard Can It Be? Designing and implementing a deployable Multipath  
> TCP". You can find the paper here:  
> https://nets.cs.pub.ro/~costin/files/nsdi12-final125.pdf

I will read the paper.