Subject: ACK decimation using WFQ

From: Segura Ramon <[email-masked]>
Date: Mon, 30 Oct 2000 20:03:56 +0200

Hello,

Just a few lines in the context of asymmetric networking and forward
throughput limitation caused by ACK congestion. I would like to share a
recent experience on using router standard mechanisms for boosting the
forward throughput beyond the limit established by the available return
channel rate. It's basically a simple "ACK decimation" technique which,
although not being optimum, has proven to increase significantly the forward
throughput for "disadvantaged" receive nodes that use very narrowband and
long-latency satellite links to support the return channel (at 2400 bps or
4800 bps). Using IP/DVB-S on the satellite forward path, the RTT when using
those return channels can be a high as 1500 ms (Inmarsat Mini-M terminal)
and 3000 ms (UHF DAMA satcom radios). The client and server platforms under
test support large windows, Fast Retransmit / Fast Recovery with Selective
Acknowledgement (SACK), extended time stamps, and the 4-segment start-up, as
provided by the OnNet Kernel 4.0 (for Win95/98) and by Windows 2000 TCP/IP
stacks. The target bulk transfer throughput on the forward channel is 2
Mbps.

To cope with the ACK congestion limitation imposed by the narrowband & long
delay return path, we take advantage of the cumulative property of the TCP
ACK's and decimate the return ACK flow by tail-dropping them at the router
serial interface. For that we use cisco's weighted-fair-queuing (WFQ), which
also ensures fairness among the multiple return ACK streams The longer the
ACK queue (i.e. the higher the programmed WFQ drop threshold), the more
noticeable is the effect of TCP forward flow self-clocking/slaving to the
ACK return rate. By setting the WFQ drop threshold to 1, we get one ACK
transmitted out of every K (K-1 packets being dropped off the queue). In our
case, the return channel (2400 bps) can only serve four to five ACK packets
per second (each being 66 bytes long). As expected, the traffic is bursty
during the throughput build-up (i.e. slow-start), but goes and remains
steady when the pipe gets full, or when the generic traffic-shaping limit
for that flow has been reached. Under this configuration, throughputs of 1
Mbps and higher can be achieved and maintained during FTP or HTTP downloads
(@2400 bps return rate, 1500 ms RTT). Without ACK decimation, the return
channel rate would limit the throughput to less than 100 kbps (assuming
delayed ACK's as implemented by Winsock).

Since WFQ creates a queue for each "conversation" or TCP connection,
multiple FTP or HTTP sessions can be established concurrently. Each object
request will go into a different queue, and its ACKs will be "tail-dropped"
independently (and 'fairly'). Despite of the ACK's being severely dropped on
an existing connection, the SYN segments for establishing new connections
will still make it thru additional queues, with little additional delay
('ping' shows an increase of some 100-200 ms compared to an idle channel).
Moreover, the return channel can also be used to upload data on to the
server from the remote client (e.g. FTP), at very low speeds, without
disturbing the downstream-related ACK flow (in that case, the ACK decimation
rate obviously increases). When introducing errors on the forward channel,
we could see SACK behaving quite well, despite the decimation of the return
ACKs (and SACK) at the upstream router interface. Under this ACK-dropping
condition, more (than 3) DUP ACK's or SACK need to be transmitted by the
client to get the threshold number into the server.

Hope this trick may be of help,

ramon

----------------------------------------------------------------------
Ramon Segura
NATO Consultation, Command and Control Agency, NC3A 
Communications Systems Division, SATCOM Branch
The Hague, Netherlands

e-mail: [email-masked]
Tel. + 31 (0)70 314 2432 / 2448 (Lab)
Fax.+ 31 (0)70 314 2176
----------------------------------------------------------------------

---

From: Mahesh Sooriyabandara <[email-masked]>
Date: Mon, 30 Oct 2000 22:56:52 +0000

Hi Ramon,

> stacks. The target bulk transfer throughput on the forward channel is 2

This sounds quite interesting. We have been doing some work on asymmetric
networks for quite a sometime. Although it seems not like the "optimum"
solution (as you have mentioned) the technique you are proposing seems to be
quite different from other available techniques. To me it looks like a straight
forward in the sense that it uses off the shelf equipment with some parameter
tuning. No need of any sort of end system modifications. I agree with the fact
that this networking technology in your case ( which is again new to apply
these technique) can be benefited by a proper ACK modification scheme.

Although you claim from your experiments that this method works well. I think
we have to raise few questions here. See below.....

> delayed ACK's as implemented by Winsock).

Although use of Fair Queuing scheme in the return path is a nice way to ensure
fair queuing ( Some earlier and we have also shown that in our experiments)
"drop from tail" ( not the best way ?) while having a small threshold can leads
to some problems (drop of data packets flow in the return path). As a solution
indeed a larger queue will be required and that can lead to reduce the forward
throughput.

> ('ping' shows an increase of some 100-200 ms compared to an idle channel).

Yes I believe because of the per-flow queuing probability of loosing SYN can be
very low.

> client to get the threshold number into the server.

Even in our experiments we saw that in the presence of losses large number of
DUP ACKs flow in the return path can lead a time out( after FR ). Dropping DUP
ACKs larger than a threshold ( to allow fast retransmission) is necessary to
avoid that ( Some have shown this earlier and we have seen it as well).

I think I agree with your your results with SACK as well. Recent experiments we
did have shown that in a lossy path back to back SACKs can lead to a time out
same as the back to back DUP acks in TCP with out SACK. So we do see the need
of dropping SACK to gain any significant benefit by using it. We are trying to
model this by using a suitable threshold. It will be interesting to see the
effects with SACK deletion.

Any body with some experience on this. I like to hear ....

Regards,

Mahesh.