Subject: TCP

From: "Jeyasekar Antony" <[email-masked]>
Date: Sat, 15 Dec 2007 15:47:39 +0530

hi
I heard that TCP is not suitable for high speed network because of its
instability, lattency.

is it true? is there any research work going on in this context?

jeyasekar
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Fred Baker <[email-masked]>
Date: Sat, 15 Dec 2007 14:44:01 -0800

you might look at tcpm.

You might find RFC 872 interesting reading:

http://www.ietf.org/rfc/rfc0872.txt
0872 TCP-on-a-LAN. M.A. Padlipsky. September 1982.

People have been saying that TCP was the wrong answer for something  
or another for a long time. We have had to tweak the algorithms from  
time to time, and people continue to hunt for the best and most  
scalable algorithm. The discussions happen across our high speed (2.5  
and 10 G in many places) backbone, and use SMTP, which runs on TCP,  
and the only real alternative for elastic traffic that has been  
proposed (SCTP) uses the same algorithms.

Draw your own conclusions.

On Dec 15, 2007, at 2:17 AM, Jeyasekar Antony wrote:

> hi
> I heard that TCP is not suitable for high speed network because of  
> its instability, lattency.
>

---

From: Tony Finch <[email-masked]>
Date: Sun, 16 Dec 2007 23:17:02 +0000

> > is it true? is there any research work going on in this context?

Also http://www.aciri.org/floyd/hstcp.html

> The discussions happen across our high speed (2.5 and 10 G in many
> places) backbone, and use SMTP, which runs on TCP

Hardly the best example of an application that needs to get maximum
bandwidth out of a connection. Practically no SMTP connections will
get near to filling the BDP of that kind of network.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FISHER: VARIABLE 3 OR 4. SLIGHT OR MODERATE. MAINLY FAIR. MODERATE OR GOOD.

---

From: "Greg Shepherd" <[email-masked]>
Date: Sun, 16 Dec 2007 19:17:05 -0800

> > > is it true? is there any research work going on in this context?

TCP can fill a big pipe as long as the RTT isn't high and the packet
loss rate is near zero.

-G

---

From: Joe Abley <[email-masked]>
Date: Mon, 17 Dec 2007 08:55:17 -0500

> TCP can fill a big pipe as long as the RTT isn't high and the packet

Some good treatment of the limitations and capabilities of TCP over  
high bandwidth-delay product paths can be found in RFC 2488. That  
document concentrates on satellite paths (i.e. high delay, modest  
bandwidth) but many of its conclusions and suggestions are equally  
applicable to high-capacity terrestrial paths (i.e. modest delay, high  
bandwidth).

Joe

---

From: Fred Baker <[email-masked]>
Date: Mon, 17 Dec 2007 11:30:05 -0800

> is it true? is there any research work going on in this context?

It is probably worth looking into the so-called "LAN Speed Records"  
and talking with those who have achieved them. An example of a news  
report is at http://www.itworld.com/Net/1746/030317dataspeed/, and I  
found numerous other references in the same Google search. In 2003,  
which is when that report indicated that the researchers had moved a  
terabyte of data from the US to Europe (~100 ms RTT) at an average  
rate of 2.38 GBPS, most of the backbones in use were 2.48 GBPS, so an  
average rate of 2.38 GBPS is ~96% of nominal capacity, and  
considering the overheads of TCP, IP, POS, and SONET/SDH framing,  
closer to 97 or 98% of addressable capacity.

The question is what was necessary to achieve this. They didn't use  
stock stacks - they carefully tuned a set of systems to use window  
scaling and SACK, to ensure that database accesses didn't slow things  
down, and so on and so forth. They also chose or arranged a time when  
there was little if any competing traffic.

Operationally, the guys who worry about this sort of thing the most  
are probably the astronomers, who routinely move sensor data from  
radio-telescopes across the research backbones for data reduction. In  
their cases, the sensors routinely generate in excess of 1 GBPS of  
data (64 MHz symbol rate, doubled by Nyquist sampling and again by  
channel count, with two data bits and two polarities, yields 1.024  
GBPS), and could generate any multiple of that that pleases you. They  
have been using TCP, and found that running TCP over random backbones  
and routes had a lot of problems, most notably that 1.024 MBPS is  
faster than gigabit Ethernet and TCP's loss behavior takes it  
downhill from there. Recent experiments, as I understand the matter,  
are using DCCP. They do so not because TCP is bad, but because  
offered load exceeds capacity they simply can't get all of their data  
through, and DCCP is a transport designed for that case.

---

From: Matthew J Zekauskas <[email-masked]>
Date: Mon, 17 Dec 2007 15:05:11 -0500

On 12/17/2007 2:30 PM, Fred Baker wrote:

> It is probably worth looking into the so-called "LAN Speed Records" and
> talking with those who have achieved them. An example of a news report

Internet2 has sponsored some, and as part of the award the contestant is
required to say exactly how they did it so the experiment can be
reproduced... the history list with pointers to contestant sites is here:

<http://www.internet2.edu/lsr/history.html>

> Operationally, the guys who worry about this sort of thing the most are probably the astronomers, who routinely move sensor data from radio-telescopes across the research backbones for data reduction. In their cases, the sensors routinely generate in excess of 1

Actually, I believe the physicists actually worry more (or at least as
much); there's lots of data to be moved around as part of the Large
Hadron Collider that is starting up at CERN.

--Matt

---

From: Marshall Eubanks <[email-masked]>
Date: Mon, 17 Dec 2007 19:29:18 -0500

On Dec 17, 2007, at 3:05 PM, Matthew J Zekauskas wrote:

>> Operationally, the guys who worry about this sort of thing the  
>> most are probably the astronomers, who routinely move sensor data  
>> from radio-telescopes across the research backbones for data  
>> reduction. In their cases, the sensors routinely generate in  
>> excess of 1
>

Note that for VLBI for sure, and particle particle physics IMO,  
fairly high packet loss rates could easily be
accommodated with no need for retransmission, and so there is no  
reason to use TCP for these applications.

This situation cries out for some sort
of "worst than best effort" scavenger service. If anyone else feels  
the same way, we should try and arrange a Bar BOF in Philadelphia.

Regards
Marshall

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 18 Dec 2007 16:09:01 -0500

Dear Rich;

On Dec 18, 2007, at 9:18 AM, Richard Carlson wrote:

> Marshall;
>
> As always, it really depends on what the scientist needs to  
> accomplish.
>
> The VLBI community is moving raw data from multiple radio  
> telescopes to a single coorelator machine to generate a new image.   
> All the data is time stamped and those multiple streams must be  
> time sync'ed to get the coorelator to function.  As with most real  
> time data streams, timeouts and retransmissions are bad because the  
> timing gets hosed.  So it's better to drop some packets instead of  
> retransmitting them.  In addition, the application itself has  
> multiple data channels, so it is in the best position to know what  
> to drop if congestion becomes a problem.  This means that plain old  
> UDP is the best protocol for this community.
>

I agree that UDP is the best for VLBI and have been saying that for  
some time. The most fundamental thing
to consider is that the VLBI correlation coefficients are typically  
very low
(that is indeed why such high data rates are needed) and the antennae  
need to be widely separated.
Because the correlation coefficients are low, the value of any data  
packet is also low, it is literally better to use any excess  
bandwidth to send a new data packet instead of retransmitting an old  
one. So, retransmissions are not helpful in this case.

In the new 2010 VLBI program, the locations of antennae will be  
limited primarily by the need for fiber optic access. (Why, oh why,  
doesn't French Polynesia have undersea fiber access ?) It will not be  
possible to provide anything like a dedicated data network for this  
effort, sites will be at remote locations with hosts without a lot of  
resources, and the need for a UDP scavenger service is strong.

People interested in more details here can look at the proceedings of  
the 6th International e-VLBI Workshop :

http://www.mpifr-bonn.mpg.de/div/vlbi/6th_evlbi/

(Disclosure - I used to be chief scientist for the US Naval  
Observatory VLBI observing program.)

> The LHC community is trying to move large amounts of stored data  
> between compute sites.  The data flows out of the detector(s) run  
> through several triggers that filter out the uninteresting stuff so  
> only a fraction of events get recorded to local disks.  This raw  
> data then gets moved from CERN Switzerland to one of a dozen sites  
> scattered around the globe.  These remote sites provide long term  
> storage of the raw data, provide some compute resources to post  
> process the data, and support a 2nd level of distributed compute  
> sites to post process the data.  This is really a classic bulk  
> transport task with tons of data being moved around the globe.  It  
> must be done in a reliable manner (raw data backup and post  
> processed data).  TCP is the protocol of choice for this community.
>

I know some particle physicists who feel that UDP would serve here  
too, for similar reasons, but the community have gone in a different  
direction. It is true that particle physics computation tends to  
occur mostly in places with good access to network resources, which I  
think may have influenced their thinking.

Regards
Marshall

> The LHC community has used DSCP scavenger service in the past, they  
> are moving to a new model where they lease their own infrastructure  
> so they can meet the science demands.
>
> Rich
>
> At 07:29 PM 12/17/2007, Marshall Eubanks wrote:
>
> ------------------------------------
>
>
>
> Richard A. Carlson                              e-mail:  
> [email-masked]
> Network Engineer                                phone:  (734) 352-7043
> Internet2                                       fax:    (734) 913-4255
> 1000 Oakbrook Dr; Suite 300
> Ann Arbor,  MI  48104

---

From: Fred Baker <[email-masked]>
Date: Tue, 18 Dec 2007 17:39:07 -0800

On Dec 18, 2007, at 1:09 PM, Marshall Eubanks wrote:

> the need for a UDP scavenger service is strong.

Speaking for myself, I would suggest (a) getting your favorite ISPs  
to run a scavenger service (IP layer), and run DCCP over it  
(transport layer). DCCP has similarities to TCP in that it has a  
duplex connection and manages throughput; it differs from TCP in that  
it does no retransmission. The purpose of the window is to control  
the rate - like TCP, DCCP sends one window per RTT, but unlike TCP it  
discards any excess that the application might give it.

Running UDP over a scavenger service, you fill a lot of pipes with  
traffic that will presumably be dropped later. Running DCCP, you drop  
that traffic in the first place.

---

From: "Trace Bond" <[email-masked]>
Date: Fri, 21 Dec 2007 03:16:18 -0800

Hi Jeyasekar,
The way to maximize flowrate through a pipe is to attain, at steady
state, a pipe filled with moving data. For a TCP connection, this is
accomplished by allowing the source node to send a data packet to the
destination node before getting an ACK packet acknowledgement from the
destination for the previous data packet. Using the vaguely-named
"window size" field in the TCP Header, the destination tells the source
the size of a Recv_Buffer that awaits an uninterrupted flow of data. (By
the way, the source does eventually get the ACK packets.) 
Analogous to a water pipe's volume equalling x-sectional area * length,
the data formula is (see W.R.Stevens' TCP/IP Illustrated, Vol.1,
Chap.20): 
	iRecv_Buffer_in_Bytes = iBandWidth_of_Pipe_in_Bytes_per_Sec *
iRound_Trip_Time_in_Sec

Once the calculation is made, I use Microsoft WinSock2's "setsockopt(
a_SOCKET, SOL_SOCKET, SO_RCVBUF, ( char*) & iRecv_Buffer_in_Bytes,
sizeof( long))" to initialize the socket.
I throw in "setsockopt( a_SOCKET, SOL_SOCKET, SO_SNDBUF, ( char*) &
iRecv_Buffer_in_Bytes, sizeof( long))" for the Send_Buffer, but I don't
know what that is or does :-(.

Although the flowrates are as expected, my TCP connections are prone to
silent breakage. Since my data transmission must be error-free, some
complex code is needed to remake connections and to resend missing data.
I'm contemplating an alternate (and simpler) design: Divide a file (ie:
the data) into chunks (1460 Bytes for Ethernet convenience?), attach a
ChunkID to each chunk to enable re-assembly at the destination, and send
them off on a **UDP** connection. Only one low-flow TCP connection would
be necessary for supervisory duties such as requesting resends of
missing chunks. Time for some R&D.
	Trace Bond
	Vancouver

PS0: TCP/IP's and UDP/IP's transmission error checking (the "16-bit
checksum" field in the TCP/UDP Header) is quite primitive and has been
badly fooled on my watch. I recommend that the source attach to the data
some dependable CRC value that the destination can use to confirm
error-free transmission.
PS1: Latency is a component in the iRound_Trip_Time_in_Sec.

________________________________

From: Jeyasekar Antony [mailto:[email-masked]] 
Sent: Saturday, 15 December, 2007 02:18
To: [email-masked]
Subject: TCP

hi

jeyasekar