Subject: RE: NFSv4 over SCTP ?

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 6 Dec 2002 08:22:20 -0800

> NFS network traffic is often an interesting mix of very large
> messages (file transfer) and small, more interactive messages
> like GETATTRs and LOOKUPs.  NFS version 2 put a limit of 8KB
> on message size.  NFS version 3 allows the client and server
> to negotiate message size.  It's common for v3 implementations
> to limit messages to 32KB, but with the advent of cheaper memory
> and higher bandwidth networks, larger transfers are attractive,
> e.g. 256KB or greater.

Higher bandwidth allows you to do larger transfers without
making each one so large (in time) that it hogs the link, 
unacceptably delaying interactive traffic.  If your bandwidth
is smaller, on the other hand, very large transfers become
unattractive, for reasons you discuss below.

> The availability of compound requests in NFSv4 is an incentive
> to make NFS messages even bigger, particularly where latency
> is large on a WAN.

I don't think there is a major effect here.  Let's consider this
in terms of your two classes mentioned above.  COMPOUND-ing of
operations in the small interactive class is going to occur 
but while this move the average up a small amount, it isn't
going to affect the size distribution significantly.  Multiple
small requests COMPOUND-ed stay small.

There may be cases in which you COMPOUND large and small but
again the change is not substantial.  32K plus a small 
request is essentially 32K still.

So now consider multiple large operations.  I can see that 
you might COMPOUND multiple 8K writes to different files
(or to disjoint byte ranges in a single file) in a single 
request.  This could substantially raise the average size
but not the maximum.  However, I don't see the big incentive
to do this.  This is going to be asynchronous traffic (at 
least I don't know of any application/API combinations where
synchronous reads or writes to multiple files woud be done),
so latency may be hidden easily.  So rather than investing in 
COMPOUND-ing logic, the simpler choice is to run more biod's.

I'm assuming that if you do COMPOUND large requests you are
going to pretty much stay within the size you negotiated.
If you thought 32K was right, you aren't going to COMPOUND
multiple 32K writes, making a write longer than you wanted.
It hurts when you do that, so you wouldn't do that. 

> NFS messages over TCP are encapsulated by the RPC record marking
> protocol: a 4 byte count followed by one or more segments
> contiguous on the TCP stream.  A large NFS message must be
> transmitted as a contiguous RPC record.  This is fine on
> our high-speed ethernet LANs, but problematic on some lower
> bandwidth, long haul WAN links, typical of Internet connections.

Yes.  The bandwidth of the link will affect the maximum
size of the message that it makes sense to transmit. 

> A single NFS message can take a long time to transmit if the
> bandwidth is low.  For instance, over a 1.5MB T1 link, a
> 32K NFS READ or WRITE will take 200ms to transmit - perhaps longer
> if it waits for an ACK.  

And a 256K IO, perfectly fine on gig-e, would take over a second,
on the T1, making 256K a poor choice in that environment.  Similarly,
if 32K is too large, then smaller might be better, especially since
the large requests are typically handled asynchronously, making
the latency issue less pressing.
  
> A series of large messages generated
> by read-ahead or write-behind can effectively block the
> link to more interactive traffic.

Yes, but this is a different effect than than that caused
by the record marking issue.  If you are waiting more than
the time to finish the first message, it is not because TCP
forces you to.  From TCP's point of view, the next message
to be sent could be the more pressing interactive one.
That's no consolation to the user waiting for a response
but it is important to keep in mind as we consider solutions.

> You'll likely see this effect if you're accessing your
> home directory across a WAN and a process drops core.
> While waiting for the core file to be written, you'll
> get lousy performance when you try to "ls" a directory.
> The effect is magnified on a multi-user client machine.
> A single user's file transfer can severely affect the
> interactive traffic of other users.

This is more likely to be a result of (lack of) message
prioritization, than the record marking issue per se.  If 
you have an important resource being shared in a fashion 
that guarantees that the resource hogs requests are
effectively favored over more interactive traffic, then 
the interactive traffic is not going to be handled
properly.

The history is that cpu time was expensive in the past and
we have adequate prioritization machanisms for that (even 
though CPU is now pretty cheap), but that network bandwidth
is now the most critical resource and we are far less likely
to have adequate priority mechanisms to use that in an
appropriate fashion.

> An easy workaround is just to increase the number of
> TCP connections between the client and server.  That
> will certainly provide alternative paths for the traffic
> and make it less likely that a single large message will block
> all messages.  However, TCP connection setup is a relatively
> heavyweight operation.  Connection overhead could be significant
> if a client, intentionally or unintentionally generates hundreds
> thousands of connections.

Another case of it hurting when you do that.  Your categorization
of requests suggests a solution.  Use more connections but not
many more.  I suggest two.  One for interactive traffic and one
for large requests.  The next step would be to give the interactive
connection priority for the link.  This could be done independently
by client and server simply on the basis of message size. 

> Hence the interest in SCTP.  It provides a couple of features
> that might be attractive to NFS - or any RPC-based protocol:
> 
> 	- Built in record marking
>
>	- Support for multiple streams on a single connection
>
> RPC messages could be mapped directly to SCTP datagrams. These
> datagrams would flow over any one of a large number of SCTP
> streams.  SCTP streams are cheap to set up - a client could
> easily associate a unique SCTP stream per RPC client handle.
>
> The benefit would be that a single, large RPC message mapped
> to an SCTP datagram can no longer block access to the link
> since SCTP datagrams are segmented into multiple packets
> that do not have to be contiguous on the wire.

Sure.  If you have an SCTP implementation and it is stable,
performannt, etc., then using it this way makes a lot of 
sense.  Of course, the server or client you are interacting
with has to also have a good level of SCTP support.  That's
where the chicken-and-egg problem arises.

> I don't think there's significant work in changing existing
> RPC stacks to support SCTP.  But there would need to be a more
> significant effort to have RPC/SCTP adopted by application and
> system providers who currently seem quite happy with RPC
> running over UDP and TCP.

I haven't read the spec, but people tell me SCTP is better.  OK.
But it hasn't had enough compelling advantages to be adopted since
people are satisfied with UDP (I don't know why) and TCP.

We could encourage SCTP adoption by making SCTP support a must
implement for NFS-v4 (or even make TCP support a must-not-implement
for NFS-v4 :-), but I don't think we want to do that.  It would either
be ignored, or discourage adoption of NFS-v4.

I think we should eliminate any impediments to NFS-v4 over SCTP but
I think that's as far as we can go.

> I'm interested in opinions:
> 
>	o Are large NFS messages problematic on WAN links ?

Clearly they are.  

> 	o Is it sufficient just to increase TCP connections ?

That's part of the solution although they need to be managed
intelligently.  Also sensible message sizing, reasonable
COMPOUND-ing, attention to priority issues.  Some of this
would be easier with SCTP, if you had it.

> 	o Would SCTP provide any significant relief ?

Yes, but if you didn't do anything else about the issue, it 
would not be sufficient.

> 	o Should the NFSv4 WG do anything about it ?

Yes.  We should make sure that there is a clear inter-operable
definition of how NFSv4 over SCTP will work.  But if do the
definition, then there better be multiple implementations, or
else it will drop out when we go to proposed standard.

>	o Is there a better solution ?

What "better" means depends on your situation, particularly your
SCTP story.  For most people, given the way things are now, solving
the issue in the context of TCP will be better.

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 6 Dec 2002 15:11:00 -0600

> It hurts when you do that, so you wouldn't do that. 

One may choose to COMPOUND smaller requests to reduce the trips down
the stack to send multiple RPCs.  That will help reduce overall
utilization of code path and potentially interrupts.

-- 
Spencer

---

From: RJ Atkinson <[email-masked]>
Date: Sat, 7 Dec 2002 01:57:00 -0500

> people are satisfied with UDP (I don't know why) and TCP.

I believe UDP deployment is mostly a matter of inertia:
	- some vendors/implementations prefer to use NFS/UDP by default
	- there are many deployed boxes configured for NFS/UDP
		for reasons that are only historical (i.e. folks would not
		deploy UDP again, but lack the ergs to change an existing
		deployment
	- some operational folks haven't used NFS/TCP and so don't perceive
		significant advantages

If one considers the UDP situation and reaches conclusions not unlike
those above, then one ought to consider that:
	- it is not clear that NFS/SCTP has significant operational advantages
		over NFS/TCP
	- there will be a very very long transition period where an 
implementation
		will need to support UDP (for backwards compatibility), TCP (same),
		as well as SCTP.
	- supporting more transport protocols in NFS must be more expensive 
than
		supporting fewer transport protocols (for implementers & operators)
	- interoperability will be problematic, at least at first

> be ignored, or discourage adoption of NFS-v4.

Yes.  And nearly any additional complexity in NFSv4 will be problematic.
Some few of us think NFSv4 already too complex.

> I think that's as far as we can go.

I would not object to that approach.  I would object to starting up
new work on supporting NFSv4 over SCTP.

>> 	o Are large NFS messages problematic on WAN links ?

The issue is not really "WAN links", but rather the
end-to-end bandwidth or bandwidth-delay product.

> would be easier with SCTP, if you had it.

Seems like an implementation quality issue to me.   Implementers
with clues can  do intelligent things with TCP.  Implementers
without many clues are unlikely to do intelligent things even
if they were using SCTP.  SCTP isn't a panacea here.

> Yes, but if you didn't do anything else about the issue, it

I'd say (1) there isn't a substantial problem today and
(2) any prospective improvement would not be significant (from
an operations perspective).

>> 	o Should the NFSv4 WG do anything about it ?

Not right now.  If folks insist on doing stuff with it, PLEASE
put it into a separate spec from the base NFSv4 spec, so that
SCTP does not delay or complicate the base NFSv4 spec.

Ran
[email-masked]