Subject: NFSv4 over SCTP ?

From: Brent Callaghan <[email-masked]>
Date: Thu, 05 Dec 2002 14:37:04 -0800

NFS network traffic is often an interesting mix of very large
messages (file transfer) and small, more interactive messages
like GETATTRs and LOOKUPs.  NFS version 2 put a limit of 8KB
on message size.  NFS version 3 allows the client and server
to negotiate message size.  It's common for v3 implementations
to limit messages to 32KB, but with the advent of cheaper memory
and higher bandwidth networks, larger transfers are attractive,
e.g. 256KB or greater.

The availability of compound requests in NFSv4 is an incentive
to make NFS messages even bigger, particularly where latency
is large on a WAN.

NFS messages over TCP are encapsulated by the RPC record marking
protocol: a 4 byte count followed by one or more segments
contiguous on the TCP stream.  A large NFS message must be
transmitted as a contiguous RPC record.  This is fine on
our high-speed ethernet LANs, but problematic on some lower
bandwidth, long haul WAN links, typical of Internet connections.

A single NFS message can take a long time to transmit if the
bandwidth is low.  For instance, over a 1.5MB T1 link, a
32K NFS READ or WRITE will take 200ms to transmit - perhaps longer
if it waits for an ACK.  A series of large messages generated
by read-ahead or write-behind can effectively block the
link to more interactive traffic.

You'll likely see this effect if you're accessing your
home directory across a WAN and a process drops core.
While waiting for the core file to be written, you'll
get lousy performance when you try to "ls" a directory.
The effect is magnified on a multi-user client machine.
A single user's file transfer can severely affect the
interactive traffic of other users.

An easy workaround is just to increase the number of
TCP connections between the client and server.  That
will certainly provide alternative paths for the traffic
and make it less likely that a single large message will block
all messages.  However, TCP connection setup is a relatively
heavyweight operation.  Connection overhead could be significant
if a client, intentionally or unintentionally generates hundreds
or thousands of connections.

Hence the interest in SCTP.  It provides a couple of features
that might be attractive to NFS - or any RPC-based protocol:

	- Built in record marking

	- Support for multiple streams on a single connection

RPC messages could be mapped directly to SCTP datagrams. These
datagrams would flow over any one of a large number of SCTP
streams.  SCTP streams are cheap to set up - a client could
easily associate a unique SCTP stream per RPC client handle.

The benefit would be that a single, large RPC message mapped
to an SCTP datagram can no longer block access to the link
since SCTP datagrams are segmented into multiple packets
that do not have to be contiguous on the wire.

I don't think there's significant work in changing existing
RPC stacks to support SCTP.  But there would need to be a more
significant effort to have RPC/SCTP adopted by application and
system providers who currently seem quite happy with RPC
running over UDP and TCP.

I'm interested in opinions:

	o Are large NFS messages problematic on WAN links ?

	o Is it sufficient just to increase TCP connections ?

	o Would SCTP provide any significant relief ?

	o Should the NFSv4 WG do anything about it ?

	o Is there a better solution ?

    Thanks
		Brent

---

From: Mike Eisler <[email-masked]>
Date: Thu, 05 Dec 2002 15:40:05 -0800

>     o Are large NFS messages problematic on WAN links ? 

Over my < 1 mbit links, I used NFS primarily for file transfer, rather
than access, since the latency is too high otherwise. But that's NFSv[23].
Maybe NFSv4 with delegations makes this a non-problem. I don't
have experience with much faster high distance links.

>     o Is it sufficient just to increase TCP connections ? 

Sure but now NFS/RPC is doing something that SCTP does.

>     o Would SCTP provide any significant relief ? 

Unknown to me if relief is necessary.

>     o Should the NFSv4 WG do anything about it ? 

Yes, but for different reasons.
Right now, there is no specification
for SCTP in ONC RPC:

    - what's the netid? NFSv4 formalized
      the defacto standard netids of tcp, tcp6,
      udp, udp6 for ONC RPC but there are no netids
      for sctp.

    - what flavor of transport is SCTP relative to ONC RPC.
        ONC RPC understands:

        - unreliable datagram transports
          with built in record marking

        - reliable stream transports with
          no built in record marking

It seems to me SCTP is a different
thing. Like TCP it is reliable (right?)
yet datagram like in its bulilt
in record marking, and independent
multiple data flows. Right now,
the Linux NFS team is treating ONC
RPC over SCTP in exactly the same
way it treats ONC RPC over
UDP (this is possible because there
is apparently a SOCK_DGRAM
interface to Linux SCTP in
addition to SOCK_STREAM.
So that means the potential for lots of
retransmits over perfectly
good connections, though I
understand a higher timeout is
being used for Linux RPC/SCTP
than for Linux RPC/UDP. The soon
to be published NFSv4
strongly discourages clients from
retransmitting requests while
the connection is live.

I do think that adding SCTP to ONC
RPC is something we need
to do (and this wg owns onc rpc)
for completeness more than
anything else, and also to
prevent a defacto standard for
operation of NFSv4 over SCTP
from being established (as it
was with NFSv[23] over TCP, with
undesirable consequences).

SCTP seems to have lots of synergy
with RDMA. If this wg pursues NFS/RDMA,
it should confine itself
to SCTP rather than pursing the
futile task of adding record
marks to TCP. The ips WG tried it
and failed, the rddp wg tried it
and failed, and so we will fail too.
Therefore, support for
RDMA implies support for SCTP.

 >   o Is there a better solution ?

Is there another "IETF-approved congestion
management transport?" "-)

---

From: Guy Harris <[email-masked]>
Date: Thu, 5 Dec 2002 16:25:26 -0800

>           no built in record marking

And, at least at one point, "experimentation [was] underway" for
reliable packet transports ("packet" implying "built-in record
marking"); to quote RFC 1057:

   There are other possibilities for transports besides datagram- or
   connection-oriented protocols.  For example, a request-reply protocol
   such as VMTP [2] is perhaps a natural transport for RPC.  The Sun RPC
   package currently uses both TCP and UDP transport protocols, with
   experimentation underway on others such as ISO TP4 and TP0.

and I think there may even have been Connectathons where RPC-over-TP4
was tested, ages ago.

> thing. Like TCP it is reliable (right?)

To quote RFC 2960:

   SCTP is a reliable transport protocol operating on top of a
   connectionless packet network such as IP.  It offers the following
   services to its users:

      -- acknowledged error-free non-duplicated transfer of user data,

	...

      -- sequenced delivery of user messages within multiple streams,
         with an option for order-of-arrival delivery of individual user
         messages,

	...

> interface to Linux SCTP in

SOCK_DGRAM, or SOCK_SEQPACKET?  The "Sockets API Extensions for
Stream Control Transmission Protocol" I-D:

	http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-sctpsocket-05.txt

suggests SOCK_SEQPACKET, as that's a better fit (and SOCK_SEQPACKET goes
all the way back to 4.2BSD, I think, so it's not as if it's a Shiny New
Invention).

That I-D also specifies a SOCK_STREAM interface.

I suppose one could implement SOCK_DGRAM, although any code that assumes
SOCK_DGRAM implies unreliable might end up implementing stuff atop SCTP
that SCTP does for it.

> good connections,

If they're treating it as SOCK_DGRAM rather than SOCK_SEQPACKET or if
Linux RPC/SCTP treats SOCK_SEQPACKET as equivalent to SOCK_DGRAM.

> undesirable consequences).

Would that either be an update to RFC 1831 or an addendum? In either
case, that'd probably be an update to section 4 "TRANSPORTS AND
SEMANTICS", although one could argue that section already speaks of
reliable and unreliable transports and of stream-oriented transports,
and therefore, as SCTP is a reliable transport and not a stream-oriented
transport (in the way that TCP is), implies how ONC RPC-over-SCTP should
be done.

---

From: Guy Harris <[email-masked]>
Date: Thu, 5 Dec 2002 16:50:05 -0800

> and failed, and so we will fail too.

I think I remember the RDDP session at the Atlanta IETF deciding to make
SCTP a required transport but not make TCP a required transport, or
something such as that favoring SCTP.  I don't think they decided to
abandon the record-marking effort for TCP, although I guess they might
still be thrashing over how to do it.  (I don't know whether Tom
Talpey's on the nfsv4 list; if he is, and I misremember, I'm sure he'll
correct me.)

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 05 Dec 2002 17:03:04 -0800

> Talpey's on the nfsv4 list; if he is, and I misremember, I'm sure he'll

Yes, SCTP is the required transport for RDDP, though the
WG discussion is still dominated by TCP framing.

To quote Caitlin Bestler, one of the RDDP participants:

     "SCTP is indeed a superior protocol to MPA/TCP for DDP, as it
     is for many applications.

     Realistically, however, if DDP relied solely upon SCTP it
     would *not* meet the objective to work with IP networks *as
     they are*. There are too many environments where middleboxes
     have no SCTP capability, or at least no SCTP ability that
     has been enabled."

	Brent

---

From: RJ Atkinson <[email-masked]>
Date: Fri, 6 Dec 2002 10:22:28 -0500

On Thursday, Dec 5, 2002, at 20:03 America/Montreal, Brent Callaghan 
>     have no SCTP capability, or at least no SCTP ability that

A pox on middleboxes.  Just makes operational life more complicated.
Folks silly enough to deploy them don't have room to gripe.  Rather
like someone complaining about their foot hurting right after they
shot themselves in the foot deliberately...

Mumble.

Ran

---

From: Mike Eisler <[email-masked]>
Date: Fri, 06 Dec 2002 10:32:38 -0800

>     have no SCTP capability, or at least no SCTP ability that

Sure the RDDP wg is discussing it, even days after the RDDP WG chair, who
is also the IPS wg chair, told the wg to stop discussing it, because he
remembers (and we were all lurking on the IPS wg alias when it happened,
right?) that the Transport Area Advisor put and end to similar discussions
within the IPS wg.

Summarizing what the Transport Area Directors have
been saying to IPS, nfsv4, and probably rddp for the last 6 years
(and there have been several ADs, and they all say the same thing),
I get this message:
    - tcp is perfect for what it does, leave it be
    - sctp is there for you to do the things you need
    - don't use UDP, not unless you are willing
        to add an IETF approved congestion management layer
        above it, i.e. effectively implement sctp over UDP.
        Wasn't UDP invent partially for the reason to allow
        prototyping of new transports?

So, if the nfsv4 wg thinks sctp is not viable for the reason
Caitlin Bestler lists, then that leaves adding a layer over
UDP. People should read Joe Touch's recent submissions. I love
the one where he suggests that if transitting firewalls is
a primary requirement, then all new protocols should layer over
http. :-

    -mre

---

From: "Talpey, Thomas" <[email-masked]>
Date: Fri, 06 Dec 2002 15:20:25 -0500

Both RDDP-over-TCP and RDDP-over-SCTP are chartered, and have
been since RDDP became a WG. In fact RDDP/TCP is the first
deliverable which is why it dominates the conversation there. See
<http://www.ietf.org/html.charters/rddp-charter.html>.

Yes, "record marking" for TCP (actually, markers and alignment both)
is the hottest topic at the moment. It has historically sparked it, no
reason to change that! :-)

Tom.

---

From: Eric Werme USG <[email-masked]>
Date: Fri, 06 Dec 2002 16:46:59 -0500

like GETATTRs and LOOKUPs.

A bigger problem is that multiple connections do bad things to congestion
control, e.g.
http://rfc-2760.rfcindex.com/rfc-2760-22.htm
  3.5.2 Research 

     Research on the use of multiple parallel TCP connections shows 
     improved performance [IL92,Hah94,AOK95,AKO96].  In addition, research 
     has shown that multiple TCP connections can outperform a single 
     modern TCP connection (with large windows and SACK) [AHKO97]. 
     However, these studies did not consider the impact of using multiple 
     TCP connections on competing traffic.  [FF99] argues that using 
     multiple simultaneous connections to transfer a given file may lead 
     to congestive collapse in shared networks. 

http://www.pdl.cmu.edu/mailinglists/ips/mail/msg00728.html
 >Basically (from a quick read), the paper, on "An Application-Level
 >Solution to TCP's Satellite Inefficiencies", recommends breaking a
 >single TCP connection into multiple connections at the
 >application level, to increase throughput on satellite circuits.
 >It is not too surprising that this increases the TCP throughput, but it
 >is still not a good idea.  For a single TCP connection, a single packet
 >drop results in the throughput for that connection being cut by half,
 >and then increased by roughly one packet per RTT.  For a TCP connection
 >that has instead been separated into N different TCP connections, a
 >single packet drop results in one of the N connections, receiving
 >1/N-th of the total bandwidth, having its throughput cut in half.  Thus
 >the bandwidth of the aggregate connection has its bandwidth reduced to
 >(1 - 1/(2N))-th of its former bandwidth - that is, the larger the value
 >for N, the smaller the aggregate bandwidth is cut.  And then, because
 >each TCP connection continues to increase its congestion window by one
 >packet per RTT, for those TCP connections that have not yet reached the
 >receiver's advertised window, the aggregate TCP connections together
 >increase their bandwidth by up to N packets per RTT.

   

Most of what I know about SCTP comes from a Cthon talk last year, so
let me know gently if I put my feet in my mouth below.  To efficiently
use a SCTP link, you want to get all the packets you can out, up to the
window size.  So you can still get a big bunch of writes ahead of your
getattr, right?  If you have enough writes to exceed the window size,
then the application winds up blocking on the data connection or TCP's
or SCTP's buffering code.

Some of my slides from http://www.connectathon.org/talks02/werme.pdf
show an inefficent state in BSD sockets where multiple writers are
waking up and going back to sleep because there's no space in the
socket.  One result is that message stream gets severely disordered. 
Perversely, threads wanting to send small requests often _can_ get
their request in and send it well before the big writes make it out!
The overall behaviour is broken, and my simple workaround of making
the TCP window bug enough to hold everything exacerbates the long
delay issue by preserving ordering.

I've thought of adding a NFS-controlled queue that releases messages
when there is space in the socket, and that could fix both the
thundering herd wakeups and the disordering that sockets allow.  It
could be setup to allow requests out FIFO style, on a whatever fits
basis, or other scheme.  Half of SCTP in a few pages of code?

I implemented in the server a reservation scheme where NFS requests on
a TCP connection don't wake a server thread until there is space for
the reply.  Therefore I'm preserving ordering.  Writes want to be
pulled off quickly to open the receive window.  Reads might deserve
special treatment since they eat so much reply space.

I guess this winds up being most helpful to writes that are bigger
than the send window.  My windows are bigger than the write size.  At
Connectathon I ran with 500KB windows and everyone survived the window
scaling.  That seems like a tremendous amount, but it's only 4 msec of
Gigabit time!  (Add to list of things to worry about - communication
link speeds span too many orders of magnitude, and 1 Gbps isn't
fast enough anymore.)

I don't know, most of our customers have big servers behind the systems
that do talk to the WAN.  Certainly all this is a bigger issue for NFS
V4 given its design goals.

From my above quotes, I'd say no, at least beyond a few.  In my
situtation, it would be easy to use two connections, one for nfsiod
threads (read and writes) and one for synchronous calls (mostly little
messages).

One odd thing I do is open a connection _per mount_ and all requests
flow over that.  It makes some locking issues easier and bursts of
activity generally happen per file or FS anyway.

My sense is that SCTP appears to be a nice fit for NFS, but it's no
panacea.  It seems most useful to release requests and replies when
earlier packets were lost or delayed, but I wouldn't think that would
be a big win.  One potential problem is if a client is sending several
write requests (or the server several read replies) is that you want
to get the earlier I/Os out of the way to start disk I/O or let the
application have its data.  However, if SCTP is breaking the I/Os into
fragments and interleaving them, the I/Os will all complete at about
the same time.  Effectively, the earlier I/Os get delayed, the later
ones finish when they would with a TCP link that preserves I/O order. 
Don't assume that SCTP is at least neutral, it may introduce new
problems!

Simply permit it?  NFS over TCP development surprised us a bit - it
took a few years before some of the odd behaviours settled out.  And
then it took me a few more years before discovering that NFS over TCP
write requests and read replies got _more_ disordered than with UDP. 
I guess permit it and think about it would be more appropriate.

Probably not, but knowing the size of the problem may be the more
important question.

	-Ric Werme

-- 
Eric (Ric) Werme         |  [email-masked], [email-masked]
Hewlett-Packard Co.      |  http://ewerme.home.attbi.com/

---

From: Som <[email-masked]>
Date: Fri, 6 Dec 2002 14:45:24 -0800 (PST)

>application level, to increase throughput on

Interesting observation. However, the statement
"Application Level" looks incomplete to me. Few
points:

1. One can view that real application is something
what is above NFS. In that case, multiple applications
using same mount point can have different TCP
connections. Is this ok as per the paper?

2. Once can also argue that (as per the paper) that a
single NIC should generate only one TCP connection (so
a layer above TCP multi-plexing all these TCP
connections). Is this what is expected as per the
paper?

3. NFSv4 still can use multiple TCP connections: at
least two. One used as control channel other one for
read/write data. This way it will probably not violate
anything like TCP's congestion control. This way NFS
can response better with heavy traffic and still
responding to interactive commands.

4. 3 can be done even if work for SCTP is needed.

Thansk, Som.

=====
"We spend our life in three spatial dimensions, confident that what we can see with our telescopes is all there is, ignorant of the possibility of 10-dimensional hyperspace"...Michio Kaku

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

---

From: Som <[email-masked]>
Date: Sun, 8 Dec 2002 21:50:02 -0800 (PST)

From that webpage, it looks like
(Look under Goals/Milestones section)

1. RDDP/SCTP mapping will be done by Dec 02 when
RDDP/TCP mapping will be done by Feb 03. That doesn't
confirm what you are saying.

2. Or, are you referring to those oct 03 dates where
RDDP/SCTP will be submitted for consideration as
proposed standard? 

Its true the date is same for RDDP/TCP, but there is
an important difference. RDDP/TCP is for consideration
as informational RFC.

So from the information on the IETF webpage, it looks
like that RDDP-SCTP is more interesting to IETF
(though there are more discussions on RDDP/TCP).

Thanks, Som.

=====