Subject: RE: large RTT variation caused by bandwidth oscillation

From: "Farid Khafizov" <[email-masked]>
Date: Fri, 4 Jan 2002 12:15:56 -0600

In the previous discussion with Andrei Gurtov, it was pointed out that
Solaris TCP is not rfc2988 compliant and therefore (under "bandwidth
oscillation" conditions) was suffering  frequent spurious re-transmissions.
Even after adjusting minRTO to 1 sec, spurious re-transmissions were still
there. Then Phil Karn and Aaron suggested that we look into the performance
of other TCP stacks. Thus in addition to Solaris (5.6 and 5.8) TCP stack, we
also looked at the following TCP implementations:
	Linux (kernel 2.4), 
	Free BSD 4.4,  
	HP-UX (9000 series) 
	WinNT (build 1381, SP6)
To the best of our knowledge, RTO management algorithm in all these OSs is
compliant with Sec 5.3 of RFC 2988. As for the min RTO, based on the source
code, Linux 2.4 has minRTO=200 ms (HZ/5 with HZ=100), and Free BSD 4.4 has
1000 ms (1*hz with hz=100).  TCP traces suggest that HP has  RTO_min =
1second as well. We do not know what the minRTO is for WinNT.

We performed two sets of measurements: in corporate and public networks. We
used the same configuration where Solaris was experiencing frequent
re-transmissions. 

As expected, in the absence of any other factors spurious retransmissions
are significantly reduced with RFC 2988 compliant TCPs. Results vary, but
there is evidence of significant improvement. This was clearly seen while
running FTP session within corporate network.

However, when we ran FTP sessions over public network, spurious
retransmissions were still there. The picture seemed to be far more complex
than originally observed. We need more time to understand what exactly is
going on, but certainly bandwidth oscillation in *combination* with some
other factors is bringing throughput down (in some cases by 30-40%). 

It is important to note that throughput values measured in the absence of
bandwidth oscillation (BO) were close to the maximum, which clearly suggests
that BO is one of the factors degrading performance.

In order to explain what we think might be going on when we say "BO in
*combination* with some other factors ", we describe what was observed in
earlier measurements. We were able to interpret TCP traces and understand
(at least in one case) how bandwidth oscillation in combination with TCP
segment losses were keeping throughput low. 
One way to ensure that RTO doesn't expire is to keep RTT sufficiently large.
RTT can be made larger by increasing CWND. If there is at least one RTO
expiration between the bursts, TCP may not take advantage of  large CWND
during the next bursts and/or FTP sessions for the following reason. When
connection brakes, slow start threshold is being set to 1/2 of current CWND.
Then TCP goes to congestion avoidance phase sooner than needed, CWND does
not reach desirable value before the end of the burst, bandwidth oscillation
is causing RTO expiration, and the vicious cycle continued until the end of
the data call. In the next FTP session SSTHRESH, which was stored in the
routing table during the previous FTP session, is causing same effects
again. As the result TCP throughput does not reach maximum value because
CWND can never reach BDP. However, if the same measurement is repeated in
the absence of BO, throughput is very close to maximum value because CWND
reaches maximum value and network capacity is fully utilized.

Summarizing our experience, we think that bandwidth oscillation is a factor
that (alone or in combination with other factors) presents a significant
challenge for achieving maximum possible TCP throughput. Perhaps, it is best
dealt by proper network design, however, there are situations when
redesigning of networks is not possible. Considering the cost of RF
spectrum, wireless operators will need recommendations on how to configure
TCP to minimize throughput degradation. Therefore, we think that this issue
needs to be addressed in 2.5g/3g as well as LINK ID.

--Farid

> -----Original Message-----
> From:	Aaron Falk [SMTP:[email-masked]]
> 
> If it turns out that this behavior is really only exhibited on a few
> deployed stacks, we should mention it with the appropriate caveats and
> move
> on.  If, instead, that this is widespread among many TCPs it may be worthy
> of futher attention.  Can you give us some details on implementations
> which
> displayed the spurious timeouts?  
> 
>