Subject: NFS over TCP stuff

From: [email-masked]
Date: Thu, 18 Jul 2002 10:50:04 -0400 (EDT)

Hi,

(Sorry, I figured these were just random comments, so I didn't bother
attributing comments to authors, since they were in several different
messages.)

> As a user myself, I have been unaffected by these issues as far as I
> know.  As a developer, they constitute rules which govern my
> implementation, but these issues are never visible above either the RPC
> or the NFS levels.  They can not be or the implementation is broken.

As a sysadmin, I can tell you users get very irritated by
"NFS Server not responding". (I don't know how many times I've said,
"No, it's not broken, the server is just slow".:-)

However, from an industry point of view, having the NFS server collapse
horribly is a good thing. That's what encourages customers/users to fork
over big bucks for bigger servers:-)

> Not necessarily.  Currently, servers can shed load by dropping requests.
> A requirement to transmit NFSERR_DROPPED replies would be more work.

The problem is that, when a server silently drops a request, it doesn't
provide any feedback to the client. As such, the client keeps retrying
and the server keeps dropping. To me, this can result in "more work".

> You can't know the link state with any certainty until you transmit
> something.  The question is: do you transmit some kind of null 
> request, or the original request ? 

SO_KEEPALIVE was all I've ever used (in
both directions on the connection) and it did work, albiet taking a long
time sometimes (15minutes in worst case?).

> I believe current practice with most clients is to increase the
> timeout significantly when TCP is used - since we assume TCP
> will do a better job of handling dropped/lost packets.

And if the server ever does drop a request, this means a long wait and
irritated users. (In another post, 30sec wasn't long enough, so wait one
minute or two minutes or...) I never did put a retransmit in my BSD code
over TCP (except after a reconnect) and I never heard complaints. (I don't
know if the BSD descendants ever added it or not?)

Way back when, I envisioned a server that wouldn't drop requests and a
client that would only retransmit after a reconnect. I can't remember
if I discussed this (12 years ago) in "Lessons Learned Tuning 4.3Reno..."
along with the other NFS over TCP stuff.

Have fun with it, rick

---

From: Eric Werme USG <[email-masked]>
Date: Thu, 18 Jul 2002 12:02:22 -0400

SO_KEEPALIVE was all I've ever used (in
   both directions on the connection) and it did work, albeit taking a long
   time sometimes (15minutes in worst case?).

One thing an implementation could do it declare the TCP connection "unhealthy"
and stop sending anything on it until it gets a reply or RST from one of
those NULL calls.  Our Cluster FS code already does something like that.

   

I found the comp.protocols.nfs discussion I referred to, see
http://groups.google.com/groups?threadm=3vljoq%2440t%40light.twinsun.com&rnum=1
A fine old-fogey thread from 1995.  Even has comments from Vernon Schryver
and Wayne Hathaway.  Haven't heard from them in ages.  And Mike is there
advocating RPC_NULLs, that may be the true start of this discussion.

	-Ric

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 18 Jul 2002 11:16:51 -0700

[email-masked] wrote:
> (snip)

Though in practice I don't think it's serious.  It's not written
into any spec, but common practice it to utilize exponential backoff in
the timeout.  So a continued lack of response from the server will cause
the client to infer that a prompt response will not be forthcoming.
In a worst case scenario, the server will drop packets intermittently
so that the client doesn't achieve full backoff, but I've no idea
whether these retransmissions are a common source of server overload.

Having said that, it's unfortunate that the protocol doesn't allow
the server to advertise its capacity to accept load and provide some
gentle back-pressure, rather like a TCP window.  This would be 
particularly valuable for read-ahead and write-behind.  If the
client gets too aggressive with async ops, it can exceed the
server's capacity.  The only way the server has to signal this
it to drop requests, and this has a significant performance impact.

A back-pressure protocol that would allow the client to utilize
the server's full capacity *without exceeding it*, would be
valuable for links of any speed.

	Brent

---

From: Mike Eisler <[email-masked]>
Date: Thu, 18 Jul 2002 11:50:33 -0700

> it to drop requests, and this has a significant performance impact.

v3 added NFS3ERR_JUKEBOX. v4 has others in addition.
They are primitive, but the fact that they don't get
get used suggests to me that a more complex protocol is
even less likely to be used.

	-mre

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 18 Jul 2002 13:46:58 -0700

> even less likely to be used.

NFS3ERR_JUKEBOX was very specifically aimed at customers
of hierarchical storage management systems (CDROM robots)
that complained about "NFS server not responding" messages
while waiting for the robot to do its thing.  All it
really accomplished was to get rid of a distracting message,
so no big deal.

A "back pressure" protocol would allow NFS clients to exploit
unused server capacity.  For example, I think most implementations
have hard-coded limits on the amount of read-ahead and
write-behind concurrency.  A performance advantage might
be a more compelling feature.

	Brent