Subject: [tcpm] How to improve TCP efficiency in a large RTT context?

From: SALLANTIN Renaud <[email-masked]>
Date: Fri, 20 Nov 2015 15:40:52 +0100

Dear all,
With the emergence of new generation satellite systems, including mega LEO constellations as well as very high throughput satellite offering terabit capacity, the problem of the end-to end performances arise again. Indeed, the large RTT (larger than 500ms for geo sat) is drastically downgrading regular TCP performance, notably during the beginning of the connection. Currently, proxies such as T-PEPs are used to minimize the RTT impact. But, their consequences in terms of security and mobility are preventing their usage in many cases and can't be considered as a sustainable solution.
Without  proxies,  the most intuitive solution is therefore to increase again the IW. Our studies therefore shown that IW10 is not sufficient and that short-lived connections continue to be the most impacted. But we do believe that an adequate use of Pacing and a larger IW should enable to mitigate the Long Fat Network latency. However  this raises a lot of questions that we would like to share with you:

·         Is there room to increase even more the Initial Window? Several major players already use IW=32 or higher (with and without pacing), that would really help for satellite communications.  But the question here is how to mitigate the burst impact?

·         Can we rely on SYN-SYN/ACK RTT measurement to dynamically adapt the slow start behavior (and notably the IW size)  since it's not possible to differentiate delay due to propagation from bufferbloat at this stage ?

·         Can we configure pacing to adapt to different networks ? Basically, 1ms is generally used in SPDY and QUIC, but is there any optimal?  For example, in case of a large RTT (either due to the delay or the congestion),  a larger pacing value (such as 5 ms) seems to offer better results.
Finally, based on several discussions we had with some of you, we are considering writing an Internet Draft on the "Pacing in TCP", and we would like to know if there are some interest for this?
Regards,
Renaud and Cédric

---

From: Mark Allman <[email-masked]>
Date: Fri, 20 Nov 2015 09:50:32 -0500

> · Is there room to increase even more the Initial Window? Several
> major players already use IW=32 or higher

I'll just flog this again: http://www.icir.org/mallman/pubs/LAJW07/

:)

(I chaired TCPM when that came out.  The IETF subsequently wised up
and found better people.  Lars was the TSV AD at the time.  His
comment on the above when it was presented at the workshop was
something along the lines of: can we really have a TCPM chair who
would propose an arbitrary IW?!  I think he meant it as a joke.  I
took it as a compliment. :) )

But, in all seriousness, we don't need to specify the IW.  Or, how
it gets set.  As long as the rest of CC is in place, there are
consequences for being too aggressive that will keep things
reasonable.  IMHO.  Not to mention that the actual impact of lousy
choices isn't all that bad.  And, of course, if we never had to
discuss the IW again, that'd be cool, too. :)

allman

--
http://www.icir.org/mallman/

---

From: Theodore V Faber <[email-masked]>
Date: Fri, 20 Nov 2015 15:40:57 +0000

On 11/20/15, 06:50, "tcpm on behalf of Mark Allman" <[email-masked]
on behalf of [email-masked]> wrote:

>and found better people.

Hey! I resemble that remark!

Well, they did find better chairs, anyway.

>discuss the IW again, that'd be cool, too. :)

I’d add a +1, but here on a Friday morning, it’ll be an Amen! instead.

---

From: BAUDOIN Cedric <[email-masked]>
Date: Mon, 23 Nov 2015 11:19:49 +0100

Basically, we are not trying to reopen the IW debate :) We came to this question mostly because 
1- for long RTT links, the current IW is clearly suboptimal
2- pacing techniques brings major improvement in term of burst impact , enabling more aggressive schemes
Our need is to  enable to better fill the satellite pipe especially for short TCP flows .
Now, we see 2 way forwards. First, adapt the IW (possibly only for long RTT flows) together with pacing to improve TCP startup in this context. Second, take benefit from even higher IW that seems already used by some important actors, where only pacing has to be added so as to mitigate burst impacts 

Cédric

[@@ THALES ALENIA SPACE INTERNAL @@]

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] 
Envoyé : vendredi 20 novembre 2015 15:51
À : SALLANTIN Renaud
Cc : [email-masked]'; BAUDOIN Cedric
Objet : Re: [tcpm] How to improve TCP efficiency in a large RTT context? 

:)

(I chaired TCPM when that came out.  The IETF subsequently wised up and found better people.  Lars was the TSV AD at the time.  His comment on the above when it was presented at the workshop was something along the lines of: can we really have a TCPM chair who would propose an arbitrary IW?!  I think he meant it as a joke.  I took it as a compliment. :) )

But, in all seriousness, we don't need to specify the IW.  Or, how it gets set.  As long as the rest of CC is in place, there are consequences for being too aggressive that will keep things reasonable.  IMHO.  Not to mention that the actual impact of lousy choices isn't all that bad.  And, of course, if we never had to discuss the IW again, that'd be cool, too. :)

allman

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 23 Nov 2015 11:04:19 -0500

> Our need is to  enable to better fill the satellite pipe especially for short TCP flows .

What sorts of IW and pacing rate values are needed to achieve
acceptable performance in the satellite systems you are concerned
about?

>From my perspective, one concern is that using a much bigger IW or
default pacing rate value can create scenarios where either the server
or client is vulnerable to DoS. Even if the larger IW or pacing rate
are only enabled for high RTTs, eg > 500ms. For example, suppose a
server sees a 500ms RTT and decides to use a larger IW or pacing
rate... what if the 500ms was due not to a large propagation delay
from a satellite link, but rather a low-bandwidth, congested, or
bufferbloated bottleneck? Then the higher IW and pacing rate are
pouring gasoline on a fire.

BTW, if you are considering writing an Internet Draft on “Pacing in
TCP", make sure you include the Linux TCP pacing work done by Eric
Dumazet, which is widely deployed (e.g. on google.com and
youtube.com):

  https://lwn.net/Articles/564978/
  https://www.ietf.org/proceedings/88/slides/slides-88-tcpm-9.pdf

neal

---

From: SALLANTIN Renaud <[email-masked]>
Date: Mon, 23 Nov 2015 17:13:09 +0100

I totally agree with the fact that enlarging the IW is dangerous in the situation you described. But I don't see what could be the repercussions of increasing the pacing rate? 

If the large RTT is due to the propagation delay, and not the congestion, you will indeed add an unnecessary delay (depending on your IW size and pacing rate). But, in the other case, you will probably lower the consequences of the congestion on the segments you sent...

PS: don't worry, we know and appreciate Eric's work on the pacing ;-)

-----Message d'origine-----
De : Neal Cardwell [mailto:[email-masked]] 
Envoyé : lundi 23 novembre 2015 17:04
À : BAUDOIN Cedric
Cc : [email-masked]; SALLANTIN Renaud; [email-masked]; Yuchung Cheng; Eric Dumazet
Objet : Re: [tcpm] How to improve TCP efficiency in a large RTT context?

On Mon, Nov 23, 2015 at 5:19 AM, BAUDOIN Cedric <[email-masked]> wrote:
> 2- pacing techniques brings major improvement in term of burst impact 
> , enabling more aggressive schemes Our need is to  enable to better fill the satellite pipe especially for short TCP flows .

What sorts of IW and pacing rate values are needed to achieve acceptable performance in the satellite systems you are concerned about?

From my perspective, one concern is that using a much bigger IW or default pacing rate value can create scenarios where either the server or client is vulnerable to DoS. Even if the larger IW or pacing rate are only enabled for high RTTs, eg > 500ms. For example, suppose a server sees a 500ms RTT and decides to use a larger IW or pacing rate... what if the 500ms was due not to a large propagation delay from a satellite link, but rather a low-bandwidth, congested, or bufferbloated bottleneck? Then the higher IW and pacing rate are pouring gasoline on a fire.

BTW, if you are considering writing an Internet Draft on “Pacing in TCP", make sure you include the Linux TCP pacing work done by Eric Dumazet, which is widely deployed (e.g. on google.com and
youtube.com):

neal

---

From: Joe Touch <[email-masked]>
Date: Mon, 23 Nov 2015 14:33:55 -0800

On 11/23/2015 8:13 AM, SALLANTIN Renaud wrote:
> I totally agree with the fact that enlarging the IW is dangerous in
> the situation you described. But I don't see what could be the
> repercussions of increasing the pacing rate?
> 
> If the large RTT is due to the propagation delay, and not the
> congestion, you will indeed add an unnecessary delay (depending on your
> IW size and pacing rate). But, in the other case, you will probably
> lower the consequences of the congestion on the segments you sent...

If the large RTT is propagation delay, then increasing the IW is exactly
the right answer, though you do want to pace the data out as well.

2140 TCB parameter reuse ought to fix the RTT issue, as well as pacing
if pacing is implemented and part of the set of parameters reused across
connections.

Joe

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 23 Nov 2015 14:56:50 -0800

> the right answer, though you do want to pace the data out as well.
right. the key issue is no easy way to reasonably verify if it's prop
delay or cong or something else.

---

From: Joe Touch <[email-masked]>
Date: Mon, 23 Nov 2015 15:01:59 -0800

> delay or cong or something else.

We ought to be able to trust that congestion will result in loss. If
it's just queuing delay and it's persistent, then it can be treated
exactly the same as propagation delay.

I.e., RED (or BLUE, or ALTQ, CODEL, etc.) ought to do the right thing
here, but in the absence of other information, TCP ought to treat delay
as delay.

Joe

---

From: Mark Allman <[email-masked]>
Date: Tue, 24 Nov 2015 11:00:00 -0500

I haven't been tracking things very closely...

> Several major players already use IW=32 or higher (with and
> without pacing),

Is this well known?  Could you point me to the results?  (I am just
interested.)

allman

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 24 Nov 2015 11:05:36 -0500

> Is this well known?  Could you point me to the results?  (I am just

Here is one such study:

  http://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-providers/

neal

---

From: Mark Allman <[email-masked]>
Date: Tue, 24 Nov 2015 11:10:19 -0500

>   http://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-providers/

cool!  thanks!

---

From: SALLANTIN Renaud <[email-masked]>
Date: Tue, 24 Nov 2015 17:13:59 +0100

I also believe that Google is using IW32 (and Pacing) with SPDY...

-----Message d'origine-----
De : Neal Cardwell [mailto:[email-masked]] 
Envoyé : mardi 24 novembre 2015 17:06
À : Mark Allman
Cc : SALLANTIN Renaud; [email-masked]
Objet : Re: [tcpm] How to improve TCP efficiency in a large RTT context?

On Tue, Nov 24, 2015 at 11:00 AM, Mark Allman <[email-masked]> wrote:
>> Several major players already use IW=32 or higher (with and without 
>> pacing),
>

Here is one such study:

neal

---

From: SALLANTIN Renaud <[email-masked]>
Date: Tue, 24 Nov 2015 17:40:36 +0100

>If the large RTT is propagation delay, then increasing the IW is exactly the right answer, though you do want to pace the data out as well.

>2140 TCB parameter reuse ought to fix the RTT issue, as well as pacing if pacing is implemented and part of the set of parameters reused across connections.

Do we know the percentage of connection establishments that can benefit from the TCB information?

---

From: Joe Touch <[email-masked]>
Date: Tue, 24 Nov 2015 09:01:09 -0800

> Do we know the percentage of connection establishments that can benefit from the TCB information? 

It depends entirely on workload and what fraction of time a connection
spends "learning" about a path.

For connections that are very short, where TCP path properties aren't
really used at all, or connections that are very long with persistent
offered load, it won't matter at all.

It's more work than it's worth for those very short connections. It
doesn't hurt long connections.

It's the medium-sized connections and long connections whose offered
load or competing flows vary a lot that are the largest beneficiaries.

Joe

---

From: Jeremy Harris <[email-masked]>
Date: Wed, 25 Nov 2015 18:56:22 +0000

> ·         Is there room to increase even more the Initial Window? Several major players already use IW=32 or higher (with and without pacing), that would really help for satellite communications.…

In combination with larger IW, would there be benefit in packets
being marked for "no roundtrip yet"?  During startup of a flow,
a bottleneck router could then fire a source-quench.

This would at least help the case where the bottleneck
is close to the source.
-- 
Cheers,
  Jeremy

---

From: Joe Touch <[email-masked]>
Date: Wed, 25 Nov 2015 13:33:59 -0800

> a bottleneck router could then fire a source-quench.

See RF6633.

Those messages are deprecated.

> This would at least help the case where the bottleneck

The best we have is ECN marking, but that helps only for the return path
ACKs, i.e., that'll take a round trip to react to.

Joe

---

From: Jeremy Harris <[email-masked]>
Date: Wed, 25 Nov 2015 21:47:50 +0000

> Those messages are deprecated.

Large IW isn't permitted, either.  We're discussing changes.
-- 
Cheers,
  Jeremy

---

From: Joe Touch <[email-masked]>
Date: Wed, 25 Nov 2015 13:53:38 -0800

> Large IW isn't permitted, either.  We're discussing changes.

We don't typically "un-deprecate" things, though.

Source quench is deprecated for a number of reasons, summarized in RFC
6633. It'd be useful to address those before assuming it can simply just
start existing.

Joe