Subject: Storage over Ethernet/IP

From: "Jon William Toigo" <[email-masked]>
Date: Thu, 25 May 2000 17:52:14 -0400

I am seeking a few points of clarification:

1.  Fibre Channel folks have attempted to explain to me why TCP/IP could NEVER be a viable interconnect for block level storage operations.  They claim:

a.  TCP is too CPU intensive and creates too much latency for storage I/O operations.

b.  The IP stack is too top heavy and processing packet headers is too slow to support storage I/O operations.

c.  The maximum throughput of a GE TCP/IP connection is 768 Mps, which is too slow to support storage I/O operations.

Is any of this true?

2.  Adaptec has posited a replacement for TCP called STP for use as a transport for storage.  Does anyone know anything about this?

3.  Current discussions of the SCSI over IP protocol seem to ignore the issue of TCP or any other transport protocol.  Does anyone know definitively what transport is being suggested by the IBM/Cisco crowd?

4.  Another storage company is looking at Reliable UDP as a substitute for TCP in storage data transfers.  Where can I learn more about this protocol, which I am told was introduced many years ago by Cisco?

Thanks in advance for your assistance.

Jon William Toigo
Independent Consultant and Author
[email-masked]

---

From: Mike Fisk <[email-masked]>
Date: Thu, 25 May 2000 22:55:49 +0000 (GMT)

On Thu, 25 May 2000, Jon William Toigo wrote:

> 1.  Fibre Channel folks have attempted to explain to me why TCP/IP
> could NEVER be a viable interconnect for block level storage
> operations.  They claim:
> 
> c.  The maximum throughput of a GE TCP/IP connection is 768 Mps, which
> is too slow to support storage I/O operations.

This is not a theoretical limitation, but is in the ballpark reported by
many general-purpose operating systems with commodity hardware.  

>Is any of this true?

I don't believe that TCP/IP implementations couldn't be optimized to
support full link rate and low latency.  If you're building a hardware
adapter that can do SCSI and RAID fast, adding TCP shouldn't be
prohibitively hard. 

> 2.  Adaptec has posited a replacement for TCP called STP for use as a
> transport for storage.  Does anyone know anything about this?

STP is the Scheduled Transfer protocol being standardized by the ANSI T11
folks.  ST was designed to run on top of GSN (a.k.a. HIPPI-6400). In my
opinion, it is as heavy-weight as TCP with respect to most of the things
stated above.  It does have the potential advantage of being designed from
scratch to support zero-copy access to user space using specialized
interface cards.

> 3.  Current discussions of the SCSI over IP protocol seem to ignore
> the issue of TCP or any other transport protocol.  Does anyone know
> definitively what transport is being suggested by the IBM/Cisco crowd?

I believe the assumption is that you will have a local network with no
packet loss or significant bit error rate.  Basically, you assume that
your ethernet is as reliable as your SCSI cable or fiber-channel network.
For a well engineered, fully-switched LAN, that may be a reasonable
assumption.

-- Mike Fisk, RADIANT Team, Network Engineering Group, Los Alamos National
Lab See http://home.lanl.gov/mfisk/ for contact information

---

From: Dave Nagle <[email-masked]>
Date: Thu, 25 May 2000 19:27:02 -0400

Jon,

Original Message
----------------
 >> 1.  Fibre Channel folks have attempted to explain to me why TCP/IP could =
 >> NEVER be a viable interconnect for block level storage operations.  They =
 >> claim:
 >> a.  TCP is too CPU intensive and creates too much latency for storage =
 >> I/O operations.
 >> b.  The IP stack is too top heavy and processing packet headers is too =
 >> slow to support storage I/O operations.

  There is a lot of work to show that this is not true.  Check out Van
Meter's 1998 ASPLOS paper "VISA - Netstations virtual internet SCSI
adaptor."

 Perhaps more importantly, there are many companies that are building
TCP in silicon ASICs.  This should make TCP's performance comparable
to Fibre Channel.  Both TCP/IP and FC provide about the same
functionality ... reliable, in-order transmission.  

The bottom line is that FC is done in hardware while TCP has
traditionally been done in software. Therefore, previous performance
numbers are not going to be fair.  Once TCP is in silicon, its
performance should be roughly equal to FC.

 >> c.  The maximum throughput of a GE TCP/IP connection is 768 Mps, which =

 I believe there are higher numbers (especially with Jumbo
Frames). Alteon's web site show's 920 Mbps.  Microsoft and Duke
University have both shown TCP performance o 1Gb+/s performance over
other networks.

  BTW, why is 768 Mbps too slow for storage.  Many apps (e.g.,
transaction workloads) are I/O's per second bound, not bandwidth
bound.  Also, even if storage over IP/ether is a bit slower than FC,
the benefits of leveraging IP's infrastructure (i.e., routers,
switches, NICs, network management, networking people) is a huge
advantage.  

 There is also the issue of SCSI over TCP/IP in the SAN vs. the
LAN/WAN.  Some companies, focusing on the SAN, are building
SCSI/lightweight transport/IP while others, focusing on the WAN,
propose SCSI/TCP/IP.  It may be the case that SAN and WAN traffic use
different transport protocols to gain a bit of extra performance in
the SAN.  

 >> Is any of this true?
 >> 
 >> 2.  Adaptec has posited a replacement for TCP called STP for use as a =

    From Paul von Stamwitz's posting to the ips mailing list ...
   
      The link to the SEP draft is
      http://www.ietf.org/internet-drafts/draft-wilson-sep-00.txt
   
      The press release is at:
	http://www.adaptec.com/adaptec/press/release000504.html
   
    The demo shows a Gb ethernet controller transporting SCSI traffic to several
    targets through an off-the-shelf 100TX switch with a Gb  uplink. The targets
    are ethernet to U160 SCSI bridges with one or more SCSI  drives attached. The
    host controller runs under NT4.0 at appears to the OS as a  SCSI host bus
    adapter.
   
    The architecture is based on Adaptec's SCSI Encapsulation Protocol
    (SEP).  SEP is mapped on top of TCP/IP or a light-weight transport
    protocol specifically designed for SANs.
    
    An SEP overview was presented at the IPS BOF in Adelaide last  month and an
    internet draft on SEP was submitted to IETF this week. I will  forward the
    link as soon as it becomes available. This draft is informational
    only and intended to aid in this group's work toward an industry
    standard SCSI transport protocol over IP networks.

 >> 3.  Current discussions of the SCSI over IP protocol seem to ignore the =
 >> issue of TCP or any other transport protocol.  Does anyone know =

   Current SCSI over IP discussions are not ignoring TCP ... they are
   definitely considering TCP as the primary transport.  See the ips
   web site at:
 
     http://www.ece.cmu.edu/~ips

 >> 
 >> 4.  Another storage company is looking at Reliable UDP as a substitute =
 >> for TCP in storage data transfers.  Where can I learn more about this =
 >> protocol, which I am told was introduced many years ago by Cisco?

  Companies to look at include:

     nishansystems.com
     interprophet.com
     san.com
     arkresearch.com

  Also, I believe that the IETF IP over FC working group is now
looking at FC over IP.

dave...........

David Nagle
Director, Parallel Data Lab
Senior Reseach Computer Scientist
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-3898 (office)
412-268-3890 (fax)
http://www.ece.cmu.edu/~bassoon

---

From: "Jon William Toigo" <[email-masked]>
Date: Thu, 25 May 2000 20:08:50 -0400

Thanks for the feedback, Mssrs. Fisk and Nagle,

I think a problem for IT folks who are hearing early statements about SANs
based on GE has to do with an issue to which you both alluded.
Specifically, what parameters -- bandwidth, throughput, latency, etc. --
must designers consider when evaluating or building a storage networking
interconnect?

Put another way, when I design an aircraft, I know about lift, drag and
other engineering parameters and can plug them into calculations that will
enable me to design a wing to lift X number of pounds.  When it comes to
storage networks, I cannot get a straight answer from any vendor regarding
the parameters that must be observed or satisfied -- whether stated as
straightforward quantities/formula or general rules of thumb -- in order to
develop a working storage networking interconnect!

What factors determine the amount of bandwidth required?
What amount of latency can be tolerated?
How quickly must erred information be re-sent?
Does this all depend upon the characteristics of the storage traffic itself?
Are the parameters application dependent?

Surely, traditional SCSI bus design delivered a solution that must be
equaled or improved upon by SAN interconnects such as FC, GE or Infiniband
for the latter to be regarded as viable storage interconnects.  This can't
be rocket science:  Is there a convenient set of storage architecture design
parameters here that I am simply overlooking?

I have no axe to grind with the FC folks, but there seems to be a holy war
shaping up around FC versus GE as a storage interconnect.  I am tracking
strategies for TCP offload or ASICS speed-up and agree that the optimization
of TCP/IP functionality to support the use of GE and 10 speed GE as a SAN as
well as a LAN interconnect.  I find such a solution to be quite practical,
but I do not say that FC is inferior.  There are many roads that lead to
Rome, as the saying goes.  What is of concern to me (and to my readers) is
to avoid deploying a technology (FC, for example) that may need to be
"forklift upgraded" within a year.

Please let me know if you are aware of any storage networking design
criteria that must be addressed by any interconnect regardless of its
underlying protocol.

Thanks,

Jon Toigo
Independent Consultant and Author
The Holy Grail of Data Storage Management
[email-masked]

----- Original Message -----
From: Dave Nagle <[email-masked]>
To: Jon William Toigo <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, May 25, 2000 7:27 PM
Subject: Re: Storage over Ethernet/IP

> Original Message
could =
>  >> c.  The maximum throughput of a GE TCP/IP connection is 768 Mps, which
=
>  >> 2.  Adaptec has posited a replacement for TCP called STP for use as a
=
>  >> 3.  Current discussions of the SCSI over IP protocol seem to ignore
the =

---

From: RJ Atkinson <[email-masked]>
Date: Thu, 25 May 2000 23:10:16 +0100

At 22:52 25-05-00 , Jon William Toigo wrote:

>c.  The maximum throughput of a GE TCP/IP connection is 768 Mps, which is 
>too slow to support storage I/O operations.

Provably false.  In fact TCP throughput above 768 Mbps over 1518-byte GE
has been demonstrated publicly in the past in several different fora.
At a recent DoE meeting there were several different examples cited,
though I don't have the details at hand this minute.

Ran
[email-masked]

---

From: [email-masked]
Date: Thu, 25 May 2000 23:38:20 -0400

> develop a working storage networking interconnect!

We have over 100 years of experience in the design of fixed-wing
aircraft - the Wright brothers were following onto several years
of glider design.

We have about 10 years experience in the design of high-speed
networks - I think a decade ago, the NSFNet had just completed
a move from 56KB links to T1, and were just starting to think
about T3 links (I may be off a year or two here).

The Red Baron's plane had 20 years of design experience behind it.
How many seconds would you give him against an F-14?

The vendors don't have rules of thumb yet.  Rules of thumb are a clear
indication that a field is mature.  And in this industry, "mature" is
equivalent to "no longer a buzzword".
				
				Valdis Kletnieks
				Operating Systems Analyst
				Virginia Tech

---

From: "Bernard Aboba" <[email-masked]>
Date: Thu, 25 May 2000 23:07:16 -0700

I too have heard these arguments. When I heard them I felt a sense of deja
vu -- anyone remember
when the conventional wisdom was that "voice will never run over IP?" In
fact, most of the
assertions below are fallacies or soon will become fallacies. The only real
argument is
about the exact form the technology will take -- NAS vs. SAN, etc.

 > a.  TCP is too CPU intensive and creates too much latency for storage I/O
operations.

There are now task specific processors and co-processors that can handle 1
Gbps line
rate today, and will run at 10 Gbps line rate in 18-24 months. So this
argument has
already fallen by the wayside.

 >b.  The IP stack is too top heavy and processing packet headers is too
slow to support storage I/O operations.

Too slow? If that were true, we wouldn't be able to handle OC-192, would we?
The real question is
how much the chips, switch fabric and specialized memory will cost, and how
competitive this will
be with existing technologies such as Fibre Channel, both for short and long
haul.

That figure was achieved with minimal hardware acceleration. Pushing it by
an order of magnitude within 24 months
is not unimaginable. If you were willing to throw more hardware at the
problem, it might be possible to handle
a 1 Gbps bit rate on 8 lambdas at the same time *today*. How does 8 Gbps of
throughput today sound, with 80 Gbps
in 18-24 months?

 > Is any of this true?

No.

---

From: Keith Moore <[email-masked]>
Date: Fri, 26 May 2000 10:23:06 -0400

None of the cited limitations of TCP performance are true.
they are also missing the point.

If you're going to run storage access over IP then you are potentially
allowing it to be run over the global Internet.  If you do that you need
good authentication and privacy, and (if you try to do them in software)
authentication and encyption will eat far more in performance than 
anything inherent to TCP or IP.

and no, it's not acceptable to assume that the storage device will be 
behind a firewall.

Keith

---

From: Karl Auerbach <[email-masked]>
Date: Fri, 26 May 2000 08:32:05 -0700 (PDT)

> slow to support storage I/O operations.

There were some papers published duing the late '80's or early '90s by
John Romkey and I belive Dave Clark and Van Jacobson about the length of
instruction sequences to handle TCP.  I'm not sure that those ever became
RFCs.

Those papers came up with figures indicating that if one structures code
"correctly" and if the net path is "clean" (i.e. not a lot of packet loss,
reordering, replication, etc) than the per-packet instruction sequences
(sans IP checksum calculation) were potantially very short.

Does anyone have the references to these papers?

		--karl--

---

From: Steve Blake <[email-masked]>
Date: Fri, 26 May 2000 13:58:33 -0400

> Does anyone have the references to these papers?

David D. Clark, Van Jacobson, John Romkey, and Howard Salwen, "An Analysis
of TCP Processing Overhead", IEEE Communications, vol. 27, no. 6, June 1989,
pp. 23 - 29.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven L. Blake                  <[email-masked]>
Ericsson IP Infrastructure                  (919)472-9913