Subject: SCPS-TP, TCP and Rate-Based Protocol Evaluation

From: Eric Travis <[email-masked]>
Date: Thu, 20 Jun 2002 21:29:11 -0400

Will:

Last week you provided a link for your results of your high-speed
SCPS and rate-based protocol testing which you will be presenting at 
SpaceOps 2002:

   "SCPS-TP, TCP and Rate-Based Protocol Evaluation"

I've spent the past few evenings going over the briefing 
material and there appear to be several troublesome aspects of the 
data curves which could lead to a suspicion that some of the measured 
data may be flawed. Summaries are provided in [footnote 1]. In there 
even that there is any merit to them, there should be ample time to fix 
any problems prior to SpaceOps..

In spite of these possible discrepancies, an impartial examination 
of the provided performance curves could in fact lead to a surprisingly
upbeat assessment of the performance of all the protocols. 

Conclusions can be drawn that would be in stark contrast to those 
provided in slide 20 of the presentation - which is transcribed for 
reference in [footnote 2].

I am offering the following set of alternative conclusions and invite 
interested parties on this mailing list to independently analyze the 
results and comment back.

Eric

The *complete* text including the footnotes has been placed at: 
http://www.ipnrg.org/travis/Peer_Review/GRC_SpaceOps2002.txt

=======================================================

An alternative set of conclusions drawn from the data presented at:

http://roland.grc.nasa.gov/~ivancic/papers_presentations/SpaceOps2002_TCP_SCPS.ppt

1 Depending on the mode of operation, even very small 
  transactions such as spacecraft command and control
  traffic *may* indeed see a benefit from using a 
  rate-based protocol

2 Rate-based protocols are advisable for *any* environment
  where bandwidth reservation is practical and available.

3 The performance problems experienced with the rate-based
  protocols was most likely due to accidental misconfiguration
  of the protocols, causing them to congest the network.
 
4 With proper configuration, all the rate-based protocols 
  could have performed better in the errored environments.

  -  An increase in error frequency has the same effect as 
     increasing the rtt, and hence changes the bandwidth/ 
     delay product

  -  The measured curves illustrate the effects of window 
     limiting rather than the expected error performance 
     curve decay

  -  The buffer sizes used on the rate-based TCPs and MDP 
     should have been re-tuned to reflect the effective  
     bandwidth-delay product

5 The single stream and multi-stream test results clearly
  illustrate that the SCPS enhancements to TCP provide 
  measurable performance improvements over the TCP SACK 
  implementation tested. 

  - The value of these performance increases is 
    subjective and would need to be judged on a
    mission by mission basis

6 Even with equal performance, the deployment of a 
  rate-based TCP is more desirable than the deployment 
  of MDP or other non-TCP protocols when unicast data 
  delivery is the goal.

  - The rate-based TCP is a sending-side only modification
  - All "standard" TCP applications can be deployed 
    without modification

7 While the space community should be aware of current and
  future TCP research, the existing standard protocols and 
  capabilities (drawn from a variety of communities) appear 
  to satisfy all known mission needs.

---

From: William Ivancic <[email-masked]>
Date: Fri, 21 Jun 2002 10:55:29 -0400

>results and comment back.

Thanks for the input.  I will definitely incorporate much of this.  See 
below for a few comments.

>   traffic *may* indeed see a benefit from using a

Will Include

>2 Rate-based protocols are advisable for *any* environment  where 
>bandwidth reservation is practical and available.

Will include with as:  Rate-based protocols may be applicable for *any* 
environment
where bandwidth reservation is practical and available

>   of the protocols, causing them to congest the network.

I wish this were the case,  but we configured and tuned these to the best 
of our capabilities and consulted with the developers (Mitre, NRL, 
etc...).  The problem appears to be (at least for MDP) that the receiving 
system choked.  It can not process fast enough due to the algorithm and 
implementation of the protocol.

>      should have been re-tuned to reflect the effective

Same comment as for item 4.  We tuned these for best performance at Zero 
BER.  It is impractical to expect one to constantly, manually  tune a 
protocol for a given time-varying unknown link condition.  We believe our 
tuning was valid for today's protocols.  That being said, I believe work 
needs to be done for autotuning as I believe it is impractical and 
unreasonable to expect the general users of systems to possess the 
knowledge to tune each protocol for a particular environment.   Obviously a 
research area with lots of work being done here.

>     subjective and would need to be judged on a

Will includes something along these lines.

>   - All "standard" TCP applications can be deployed

Well, I don't consider a pure rate-base, not congestion control protocol 
TCP for reasons I previously stated - even though the protocol header is 
the same as TCP.  However, an in-kernel reliable rate-based protocol may be 
more desirable for unicast data than an application level rate-based 
protocol.   There is definitely room for debate here.   This is a 
philosophical debate.  I see merit on both sides and I don't really care to 
carry on such a debate here.  IMHO, neither answer is right or wrong and 
both are right and wrong depending on the situation.

I'll consider this, but I definitely cannot accept it as worded.

>   to satisfy all known mission needs.

Nice touch, I'll add this.

=======================================
Will Ivancic
NASA Glenn Research Center
21000 Brookpark Road    MS 54-5
Cleveland, Ohio  44135
Phone +1 (216)433-3494
Fax +1 (216) 433-8705
Yahoo Instant Messenger  ID: ivancic
http://roland.grc.nasa.gov/~ivancic

---

From: Eric Travis <[email-masked]>
Date: Fri, 21 Jun 2002 17:47:00 -0400

William Ivancic wrote:

>environment where bandwidth reservation is practical and available

I know that I'm being a nudge but - within a *space environment* 
rate-based protocols really (honestly) are always advisable from a 
performance perspective:

   The biggest performance hit one takes from a well tuned
   TCP operating in a benign environment (no congestion, 
   no corruption) is the cost of the initial bandwidth estimation. 

   There may indeed be stable/safe alternatives to slow-start 
   (an excellent research area), but they can never be cheaper 
   to any flow than a priori knowledge (read: not having to probe 
   at all).
 
   Once a source of loss is introduced, the benefits of a 
   rate-based approach become even more pronounced.

Hence, when one has bandwidth reservation available to them
and timeliness of delivery is critical (pretty much a given
if one is being as wasteful as providing dedicated bandwidth), 
then rate-based mechanisms will *always* preferable.

>I wish this were the case,  but we configured and tuned these to 
>the best of our capabilities and consulted with the developers 
>(Mitre, NRL, etc...).  The problem appears to be (at least for MDP) 
>that the receiving system choked.  It can not process fast enough 
>due to the algorithm and implementation of the protocol.

OK, but the proper tuning is only as good as the knowledge of the
effective end-to-end capacity of the path. 

When examined together, the performance graphs (slides 12, 14, 15, 
18 and 19) still strongly suggest that the intermediate buffering 
in the path can only support flows of 35-40Mbps when subjected to 
a 500ms rtt. 

In fact, the measurements from the in-kernel TCP (slide 12,
in particular the 100MB file results) are what are causing my 
stubbornness in believing that the capacity was actually somewhat 
lower than the assumed 100Mbps.

If you've gone through footnote one, items 1 & 2, verified the
end-to-end path was capable of supporting flows greater than 
35-40 Mbps with an rtt of 500ms and the tcpdump traces indeed 
verify that all the packets were actually making it to the 
receiving host's side of the Cisco 7100 than that's just the way 
it is...

But - out of curiosity, if you were hook the two test hosts 
together with just a cross-over cable (bypass the routers and SX/14), 
do they still choke at 35 - 40 Mbps?  

I confess that the reason I find the data troubling is because I 
routinely run userspace protocols that can achieve about 94Mbps of 
throughput in now modest hardware (333MHz Pentium II) using the 
same algorithms (and even some of the same code) as in the SCPS-TP 
implementation you tested, and so I'm slightly reluctant to believe 
that the bottleneck was ~35Mbps on hosts capable of sinking an 
in-kernel 65Mbps TCP connection (though those hosts should have 
been able to achieve ~ 94Mbps performance over 100baseT). 

The cost of crossing the kernel/userspace boundary is really not 
that high any more. Userspace protcols can run at very high speeds,
so the fact that the MDP and SCPS implementations couldn't handle 
the load is surprising.
 
>Same comment as for item 4.  We tuned these for best performance 
>at Zero BER.  It is impractical to expect one to constantly, manually  
>tune a protocol for a given time-varying unknown link condition.  
>We believe our tuning was valid for today's protocols.  That being 
>said, I believe work needs to be done for autotuning as I believe it 
>is impractical and unreasonable to expect the general users of systems 
>to possess the knowledge to tune each protocol for a particular 
>environment.   Obviously a research area with lots of work being done 
>here.

I agree, the tuning was indeed valid for the Zero BER case...

However, if one of the goals of the effort was to measure where the 
protocols break, then it really makes sense to try and not artificially 
limit their performance - or at least note the reason(s) for the 
performance degradation...

Practically speaking, if one can expect the end-users to have *some* 
knowledge of the end-to-end path (say, tuning for a 500ms rtt) then at 
least in *space environments* it is probably reasonable for them to 
also know the expected error characteristics of the path...

The mission folks have this data readily available.

Now, I absolutely agree that it is indeed unreasonable to expect a 
general user to be able to - or be willing to optimally tune their 
protocols (and frankly, it is part of our business model);

To stay on point, as the space environment is more managed 
(and likely to stay that way for some time) than a general user 
environment, the expected error characteristics of the space-link
will be known well in advance and the hosts *can* be tuned 
appropriately.

Auto-tuning is enticing (we've experimented a bit with it and our 
'terrestrial TCP over satellite paths' customers want it so we still 
have notions of adding it to our code-base someday), but I have 
lingering doubts as to whether any auto-tuning mechanism can tune 
fast-enough to still be relevant for any particular flow or valid 
across multiple time-disjoint flows.

In the real world I always advise folks to tune their rate-based 
buffers to conform to an expected *worst case* expected error 
conditions (at least in the case where the rate-based protocols 
are using a flow control window to prevent overwhelming the receiver), 
it is perfectly safe to over provision those buffers.

>Well, I don't consider a pure rate-base, not congestion control
>protocol TCP for reasons I previously stated - even though the 
>protocol header is the same as TCP.  However, an in-kernel reliable 
>rate-based protocol may be more desirable for unicast data than an 
>application level rate-based protocol.   There is definitely room 
>for debate here.   This is a philosophical debate.  I see merit on 
>both sides and I don't really care to carry on such a debate here.  
>IMHO, neither answer is right or wrong and both are right and wrong 
>depending on the situation.
>

Not to be a persistent pain, RFC 793 is still the root definition 
of TCP and it is not a "congestion control protocol". I'm *not* 
saying that is a good thing, but just that it is. 

A rate-paced TCP interoperates perfectly with any other implementation
of TCP (paced or not). The protocol definition covers the protocol
header format, the interaction of the pairwise state machines and 
the end-to-end semantics; The behavior of the bits over the ether
in response to external stimulus is something else.

Perhaps it would be difficult to accept a purely rate-based TCP
as a "TCP friendly" or "congestion friendly" protocol, but it 
is still TCP.

Remember, a purely rate-paced system *does* have congestion control 
but it is only suitable for use over bandwidth reserved paths:

   doing slow-start just makes no sense in these environments, 
   and if you experience loss over your reserved paths, you
   probably don't want to trigger a congestion response.

However, the important point is that it is better in general
to use a protocol with TCP's end-to-end semantics rather than
having to write new applications for new protocols. Having
a single application that works over a reserved bandwidth
deployment or a shared resource environment is preferable
to having two different applications - and needing to know
which one to use depending on who you are interacting with...

The obvious advantage for a rate-paced TCP is that the receiving 
host need not even be cognizant that it's peer is rate-paced.

As for user-space versus kernel-space implementation, the real 
requirement is for multiple entities to share the same token-bucket(s).  
The cause of your multi-streams testing grief was that the SCPS 
Reference implementation really wasn't built to support that - it 
was originally for a spacecraft were we were providing the OS... 
It wasn't expected to be used that way - even for experimentation.

Generally, if you are using the SCPS Reference Implementation 
and want/need to support multiple application, the best way is 
to configure it as a transparent gateway. 

Eric

---

From: Keith Hogie <[email-masked]>
Date: Wed, 26 Jun 2002 11:36:03 -0400

All,
  
  I have been following the discussion on Satellites running IP and it
seems to have gone off heavily in the TCP direction.  I realize this is
the TCPSAT group but the question seemed to be broader than just TCP.

  What we see for missions using IP is a very heavy use of UDP and not
TCP.  TCP will always have limitations that we feel are best avoided
by using UDP.  This allows us to do things like data transfers over 
one-way links which does not work with TCP.  I have seen people 
attempting to fix TCP for all sorts of special cases for years and
still don't see much convergence.

  For an example of other UDP use see the following:

  http://www.esa.int/export/esaCP/ESALS5OED2D_index_0.html
  http://www.gcs-salzburg.at/
  http://www.gcs-salzburg.at/products/p_odg.html 
  http://www.simple.at/
  http://www.datacast.at/technology.html

  We are looking at the MDP/NORM UDP-based file transfer protocol and
plan on using it to transfer files from the Space Shuttle payload
bay on STS-107 (scheduled to launch on July 19th but may be slipping 
right now).

  The bottom line is we want to use standard IP/UDP/TCP protocols that
are supported in every operating system available.  Using a modified
TCP requires more software support that creates more problems when
you are building and deploying operational control centers and 
level-zero processing systems.  We really prefer to use standard
protocols at layer 4 and below and do any space specific tweaks
in the application layer.

----------------------------------------------------------------------
  Keith Hogie                   e-mail: [email-masked]
  Computer Sciences Corp.       office: 301-794-2999  fax: 301-794-9480
  7700 Hubble Dr.
  Lanham-Seabrook, MD 20706  USA        301-286-3203 @ NASA/Goddard
----------------------------------------------------------------------

---

From: Eric Travis <[email-masked]>
Date: Wed, 26 Jun 2002 12:44:38 -0400

Keith,

With all due apologies to Sir Arthur Conan Doyle ("The Sign of Four"):

   "It is UDP," he said, "a seven-per-cent solution.  
    Would you care to try it?"

UDP doesn't help to avoid *any* of the problems of TCP, it just moves 
them to something above UDP. You will still need to build the required 
subset of:

  {sequencing, reordering, flow control, congestion avoidance,
   reliability, etc.}

into a "standard" application - or applications. 

If, for a particular task, the required subset of the above is indeed 
the null set (and as you note, data broadcasting is one such deployment), 
UDP is an excellent choice for an IP-based implementation of that 
particular application - but this really is not the model that most 
people fall in love with when they first hear the siren's song of 
"Internet in Space"...

>   The bottom line is we want to use standard IP/UDP/TCP protocols 
> that are supported in every operating system available.

You desires are as valid as anyone else involved; It has been said 
many times before, it's the prerogative of any given mission to choose 
their desired communications model and supporting mechanisms.
 
As for "IP/UDP/TCP protocols that are supported in *every*
operating system available" - that's not demonstrably true is it?
"Supportable" absolutely, universally supported - not quite.

We can name some cases where support had to be added to an OSes, 
so the distinction is:

  "IP/UDP/TCP protocols that *can* be supported in *every*
   operating system - period." 

This is the case for *any* layer four protocol.

Whether or not a particular layer four protocol *needs* to 
be supported by a particular operating system depends on
the assumed communications architecture (see next)

> I have seen people attempting to fix TCP for all sorts of special 
> cases for years and still don't see much convergence.

In all fairness, I've also seen people providing reliable services
over UDP for many more years, and still there is no convergence;
I don't consider a lack of convergence a bad thing :o)

> Using a modified TCP requires more software support that creates 
> more problems when you are building and deploying operational control 
> centers and level-zero processing systems.

I respectfully disagree - any UDP-based solution clearly requires 
some sort of applications software; How is the deployment of this 
application software any more complex than the deployment of something 
capable of handling TCP (which can be application software or a tiny 
box with two 100BaseT ports, etc.)?

Further, if we are going to the trouble to encapsulate the spacecraft
traffic in IP datagrams - I would imagine that the desire is to 
*eliminate* the level zero processing and to route those IP datagrams 
directly to the principle investigators... 

What role must the control centers have in the end-to-end data flow 
for this operational model?

> We really prefer to use standard protocols at layer 4 and below and 
> do any space specific tweaks in the application layer.

That's certainly a position that could be advocated - to help move that
notion forward you need to get people to converge on a full solution: 

  an application layer protocol (and it's interface) so that people 
  can build the same sort of diverse applications set that they have 
  available to them over stream and datagram sockets. 

As silly as the notion is to *me* (and I DO think it to be silly), 
the people who demonstrate the greatest support for the concept of 
"IP in Space" have expressed interest in supporting things like telnet 
and http between their spaceborne instruments and their ground-based 
infrastructure. These - and many of the other services people commonly 
associate with an "Internet experience" are TCP-based.

Unless we can build and (widely) deploy UDP-based alternatives to these, 
UDP-based solutions remains a "seven (or [1..9]7 percent solution".

So, while UDP is part of an IP-based solution set, it is not sufficient.

Eric