Subject: [multipathtcp] Loss and TCP Throughput

From: Abhishek Verma <[email-masked]>
Date: Thu, 17 Dec 2015 18:12:09 +0530

Hi TCP experts,

I have an observation that i am unable to explain and understand. Would
appreciate if folks here can make some sense out of this:

I have a fixed size file, 100 Mb.

I use multiple TCP streams to transfer the file over my network. I have a
10Mbps broadband connection.

There are times when i am able to fill up the pipe with very few TCP
streams (like 2-4) but there are times when i need more than 20 to fill up
my connection.

My understanding is as follows:

TCP Throughput = k (MSS/RTT)*(1/sqrt(Loss)) where k is some constant

In a file transfer between two end points the MSS and RTT are constant, so
the equation reduces to:

Throughput = k'/sqrt(Loss)

This is the throughput that i get with 1 TCP stream.
Hence, when am pushing N chunks, the throughput that i will get is
N*k/sqrt(Loss)

I hence assume that when i need more TCP streams to push my file out, the
Loss in the network is higher than when i need fewer TCP streams. This is
the part that i understand very well.

What i dont understand is this:

I figure out that i need, say 7, TCP streams to fill my 10Mbps pipe when
pushing out my file.

So, i if i do this experiment back to back 10 times, then while 80% of the
times i am able to fill the pipe with 7 streams. There are always these 20%
runs, where 7 streams gives me a BW thats lower than 10, say around 7-8
Mbps. This is something that i cannot explain.

Can somebody help me with this? Just give me some clues and i will take it
on from there.

I could rephrase my question like this:
Is it possible that for certain TCP sessions (the 20%) the Loss that i get
is more than the Loss that 80% of my TCP streams experience.

Thanks, Abhishek

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 17 Dec 2015 14:58:35 +0200

Hi,

Most likely your results show congestion upstream of your access link.

One TCP Cubic should be able to the 10Mbps of your link, provided there is no other bottleneck.

If there is another bottleneck, with many flows fighting for it, by increasing the number of TCP flows you are increasing your share of that bottleneck link, until you go to 10Mbps at which point your access link is becoming the bottleneck.

The variability you are seeing in the number of TCP connections is likely due to variability of upstream bottleneck capacity (e.g. how many other users’ flows you are fighting with).

Hope this helps.
Costin

> On 17 Dec 2015, at 14:42, Abhishek Verma <[email-masked]> wrote:
> 
> Hi TCP experts,
> 
> I have an observation that i am unable to explain and understand. Would appreciate if folks here can make some sense out of this:
> 
> I use multiple TCP streams to transfer the file over my network. I have a 10Mbps broadband connection.
> 
> There are times when i am able to fill up the pipe with very few TCP streams (like 2-4) but there are times when i need more than 20 to fill up my connection.
> 
> In a file transfer between two end points the MSS and RTT are constant, so the equation reduces to:
> 
> Hence, when am pushing N chunks, the throughput that i will get is N*k/sqrt(Loss)
> 
> I hence assume that when i need more TCP streams to push my file out, the Loss in the network is higher than when i need fewer TCP streams. This is the part that i understand very well.
> 
> I figure out that i need, say 7, TCP streams to fill my 10Mbps pipe when pushing out my file.
> 
> So, i if i do this experiment back to back 10 times, then while 80% of the times i am able to fill the pipe with 7 streams. There are always these 20% runs, where 7 streams gives me a BW thats lower than 10, say around 7-8 Mbps. This is something that i cannot explain.
> 
> Can somebody help me with this? Just give me some clues and i will take it on from there.
> 
> Is it possible that for certain TCP sessions (the 20%) the Loss that i get is more than the Loss that 80% of my TCP streams experience.
> 
> Thanks, Abhishek
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Abhishek Verma <[email-masked]>
Date: Thu, 17 Dec 2015 19:39:10 +0530

Hi Costin,

Thanks a lot for the response. It was indeed very helpful.

> One TCP Cubic should be able to the 10Mbps of your link, provided there is

It usually doesnt. I would guess then its because of the losses in the
network.

>
> If there is another bottleneck, with many flows fighting for it, by
> increasing the number of TCP flows you are increasing your share of that
> bottleneck link, until you go to 10Mbps at which point your access link is
> becoming the bottleneck.
>

Yes, i agree.

I believe the throughput equation does not hold good when the losses are
induced because of us congesting the link. So, when i increase the # of TCP
streams to fill up the pipe, then that equation may not hold good.

Is this correct -- that, loss in that equation is the general network loss
and not the loss induced by congesting the link artifically?

>
> The variability you are seeing in the number of TCP connections is likely
> due to variability of upstream bottleneck capacity (e.g. how many other
> users’ flows you are fighting with).
>

Wow. Thats strange.

I assumed that in a 10Mbps broadband connection, i will always get 10Mbps
and that will be the bottle neck. Did not imagine that its possible that i
could get choked further upstream.

I have another observation that cannot get reconciled with getting choked
upstream:

I send two 100Mb files at the same time.

On one file i get a throughput of 7Mbps, while on the other i get 3Mbps.

Because of TCP fairness, shouldnt each file get 5Mbps?

Thanks, Abhishek

> > I hence assume that when i need more TCP streams to push my file out,
> the Loss in the network is higher than when i need fewer TCP streams. This
> is the part that i understand very well.
> >
> > So, i if i do this experiment back to back 10 times, then while 80% of
> the times i am able to fill the pipe with 7 streams. There are always these
> 20% runs, where 7 streams gives me a BW thats lower than 10, say around 7-8
> > Is it possible that for certain TCP sessions (the 20%) the Loss that i
> get is more than the Loss that 80% of my TCP streams experience.
> >
> > Thanks, Abhishek

---

From: <[email-masked]>
Date: Thu, 17 Dec 2015 15:26:57 +0000

From: multipathtcp [mailto:[email-masked]] On Behalf Of Abhishek Verma
Sent: 17 December 2015 14:09
To: Costin Raiciu <[email-masked]>
Cc: MultiPath WG <[email-masked]>
Subject: Re: [multipathtcp] Loss and TCP Throughput

Wow. Thats strange.

I assumed that in a 10Mbps broadband connection, i will always get 10Mbps and that will be the bottle neck. Did not imagine that its possible that i could get choked further upstream.
[phil] are you on DSL or cable? Cable is shared access. DSL networks are designed to have a bottleneck link at the BRAS /BNG – shared point between typically a few tens of users. One way of finding out whether this is the issue is to run your tests at different times of the day – off peak, esp middle of the night, you won’t be contended, in the evening peak you will be sometimes.

Best wishes
Phil

Philip Eardley
Research and Innovation
This email contains BT information, which may be privileged or confidential. It's meant only for the individual(s) or entity named above. If you're not the intended recipient, note that disclosing, copying, distributing or using this information is prohibited. If you've received this email in error, please let me know immediately on the email address above. Thank you.
We monitor our email system, and may record your emails.
British Telecommunications plc
Registered office: 81 Newgate Street London EC1A 7AJ
Registered in England no: 1800000

---

From: Abhishek Verma <[email-masked]>
Date: Thu, 17 Dec 2015 21:22:30 +0530

Thanks Philip. This was very helpful.

We have a DSL connection. I see that i need fewer TCP streams to fill up my
pipe during my day time vis-a-vis the night hours.

Cheers, Abhishek

On Thu, Dec 17, 2015 at 8:56 PM, <[email-masked]> wrote:

>
>
> *From:* multipathtcp [mailto:[email-masked]] *On Behalf Of
> *Abhishek Verma
> *Sent:* 17 December 2015 14:09
> *To:* Costin Raiciu <[email-masked]>
> *Cc:* MultiPath WG <[email-masked]>
> *Subject:* Re: [multipathtcp] Loss and TCP Throughput
>
>
>
>
> [phil] are you on DSL or cable? Cable is shared access. DSL networks are
> designed to have a bottleneck link at the BRAS /BNG – shared point between
> typically a few tens of users. One way of finding out whether this is the
> issue is to run your tests at different times of the day – off peak, esp
> middle of the night, you won’t be contended, in the evening peak you will
> be sometimes.
>
>
>
> Best wishes
>
> Phil
>
>
>
> Philip Eardley
>
> Research and Innovation
>
> This email contains BT information, which may be privileged or
> confidential. It's meant only for the individual(s) or entity named above.
> If you're not the intended recipient, note that disclosing, copying,
> distributing or using this information is prohibited. If you've received
> this email in error, please let me know immediately on the email address
> above. Thank you.

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 17 Dec 2015 23:51:35 +0200

> I believe the throughput equation does not hold good when the losses are induced because of us congesting the link. So, when i increase the # of TCP streams to fill up the pipe, then that equation may not hold good.
> 
> Is this correct -- that, loss in that equation is the general network loss and not the loss induced by congesting the link artifically?

If you are a single flow at the bottleneck, it really does not matter what the throughput equation says; as long as you have enough buffering (at least BDP at the bottleneck), and an aggressive enough congestion control (think problems of Reno long fat pipes), you will always fill the pipe. If you don’t, then either the bottleneck is not where you think it is, or you have some sender/receiver buffer sizing issues (I don’t think this is the case).

The Reno throughput equations that exist are probably slightly off for Cubic (assuming that’s what you are using) - but, as I said, it shouldn’t matter.

As for fairness - if you increase the file size, throughput should  equalize more across the two connections.
Costin

> I have another observation that cannot get reconciled with getting choked upstream:
>

---

From: Abhishek Verma <[email-masked]>
Date: Fri, 18 Dec 2015 06:27:30 +0530

>
>
> As for fairness - if you increase the file size, throughput should
> equalize more across the two connections.
> Costin
>

And this is precisely that i am unable to understand.

Why isnt TCP fair for small file sizes. And when i say smaller, i mean
100Mb files on a 25Mbps broadband connection. Sometimes when i am pushing
the two files togethar, i get a BW of 19Mbps for one, and 6 for the other.
Most of the times its equitably distributed, but there are often these
outliers where one connection completely dominates the bulk of the pipe
capacity.

Cheers, Abhishek

>
>
> >
> >
> > The variability you are seeing in the number of TCP connections is
> likely due to variability of upstream bottleneck capacity (e.g. how many
> other users’ flows you are fighting with).
> >
> > Wow. Thats strange.
> >
> > I assumed that in a 10Mbps broadband connection, i will always get
> 10Mbps and that will be the bottle neck. Did not imagine that its possible
> that i could get choked further upstream.
> >
> > > I have an observation that i am unable to explain and understand.
> Would appreciate if folks here can make some sense out of this:
> > >

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 17 Dec 2015 20:48:35 -0800

Hi Abhishek,

I think It would be good if you can test it in a local network where
there's no competitive traffic.
--
Yoshi

On Thu, Dec 17, 2015 at 7:52 AM, Abhishek Verma <[email-masked]>
wrote:

> We have a DSL connection. I see that i need fewer TCP streams to fill up
> my pipe during my day time vis-a-vis the night hours.
>
> Cheers, Abhishek
>
>> *From:* multipathtcp [mailto:[email-masked]] *On Behalf
>> Of *Abhishek Verma

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 18 Dec 2015 09:37:53 +0100

Abhishek,
> streams (like 2-4) but there are times when i need more than 20 to fill
> up my connection.
>

This is only valid if the loss ratio is independant of the number of TCP 
connections that you have. If you increase congestion by adding more 
subflows, then Loss will also depend on N (and could increase with N).

Your measurement scenario is

client --- DSL network ---- server

I guess that the server has much more capacity than the client and you 
want to optimise the utilisation of the client DSL link.

The DSL network cannot always be considered as a blackbox. When you use 
multiple TCP subflows, you use different source and destination ports. 
If your network uses equal cost multipath, you will end up in using 
different paths for different subflows :

client --- R1---R2 ---R3 ---- server
             +         +
            R4---------R5

In this example, R1 can load balance the flows on the R1-R2-R3 and 
R1-R4-R5 paths that might have different levels of congestion.

You might use paris traceroute to check the load balancing in the DSL 
network that you use. I would also encourage you to collect packet 
traces of the different transfers and analyse the low performing flows

Olivier

---

From: Abhishek Verma <[email-masked]>
Date: Fri, 18 Dec 2015 19:38:35 +0530

Hi Olivier,

> multiple TCP subflows, you use different source and destination ports. If
> your network uses equal cost multipath, you will end up in using different
> paths for different subflows :
>
> network that you use. I would also encourage you to collect packet traces
> of the different transfers and analyse the low performing flows

Very valid point and I did think about it. However, i was unsure if there
is ECMP at the access networks or at this level.

I was under the impression that ECMP or hashing because of link aggregation
(LAGs) usually happens at the edge or the core. However, i hear you and
this would definitely explain why certain TCP streams give lower throughput
while others work well.

Let me try paris traceroute -- I wasnt aware of this.

Thanks for the lead!

Abhishek