Subject: Anything else not on the issues list?

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 17 Jul 2002 11:11:47 -0700

Mike Kupfer's recent contribution prompts me to ask if
there is anything else that people are working on that
is not on the issues list.  We can't track what we don't
know about, and for these purposes, let's assume that if
it isn't on the issues list, we don't know about it.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 17 Jul 2002 12:52:51 -0700

Here is one issue that the WG needs to make a decision on,
which I've brought up a few times over the years.

When TCP support was added to NFSv2, there were no formal
specified rules for whether a client should issue a retry
in the event the remote procedure call timed out, even if
the TCP connection did not break. Since many NFS servers
dropped requests silently (say due to temporary resource
unavailability) even those sent over TCP, the client pretty
much had to retry the NFS request.

With NFSv3, the JUKEBOX error was added, which would
allow an NFS server to tell a client that its request has
been dropped and that it should retry. However, the NFSv3
specification did not address the issue of retries over
TCP, perhaps because experience with NFS/TCP was limited
to a few implementations, and not most of the commercial
implementations.

So as a result, we can see 32 KB WRITE requests being
retried over a slow, but responsive network link, putting
unnecessary load onto an already stressed link or server
and just making the problem worse.

Today, we have lots of experience with NFS/TCP and so have
the opportunity to do this right. If we want to take the
opportunity, I propose the following:

    When processing a request received over a reliable
    transport such as TCP, the NFSv4 server MUST NOT
    silently drop the request, except if the transport
    connection has been broken. Given such contract between
    NFSv4 clients and servers, clients MUST NOT retry a
    request unless one or both of the following are true:

    - The transport connection has been broken
    - The procedure being retried is the NULL procedure

    Since some transports, including TCP, do not always
    synchronously inform a peer when the other peer
    has broken the connection (for example, when an NFS
    server reboots), it is necessary that the NFSv4 client
    "probe" the connection to see if has been broken.
    Use of the NULL procedure is required to do so. So,
    when a client experiences a remote procedure call
    timeout (of some arbitrary implementation specific
    amount), rather than retrying the remote procedure
    call, it should instead issue a NULL procedure call
    to the server. If the server has died, the transport
    connection break will eventually be indicated to the
    NFSv4 client, and then client can then reconnect, and
    then retry the original request. If the NULL procedure
    call gets a response, the connection has not broken.
    The client can decide to wait longer for the original
    request's response, or it can break the transport
    connection and reconnect before re-sending the request.

    For callbacks from the server to the client, the same
    rules apply, but the server doing the callback becomes
    the client, and the client receiving the callback
    becomes the server.

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 17 Jul 2002 18:15:18 -0400

Yes, please do this.

It would make a huge operational improvement if this were
universally implemented.  I really really care about this
as a user.

Ran
[email-masked]

---

From: Eric Werme USG <[email-masked]>
Date: Wed, 17 Jul 2002 21:32:49 -0400

[email-masked] suggests wrt NFS over TCP retransmits:

Lessee, at 9600 baud, 32 KB will take about 30 seconds, so I can see
a problem there.  I'll leave 110 baud as an exercise for the reader.
The sys admin who would do that to his system deserves a new job.  Or
should implementations time link speed and set rsize/wsize accordingly?
(Yes, bag of worms, I'm not proposing that.)

   

       ...

This all seems like a lot of effort to determine if the connection is
still up.  For the slow transport case, instead of making a client
decide what's going on based on the individual requests, we could look
at all the traffic over the (each!) TCP connection.  All production
client code I know of closes idle connections after 4-6 minutes.  What
about leaving things to that code, but make it clear that when a TCP
connection is closed due to being idle that a new connection be created
and requests retransmitted?  As long as the connection is indeed alive,
we ought to see something happen after 4-6 minutes!  (Umm, 32 KB at 110
baud, well, does _anyone_ do that?)  Only if there are several connections
over that slow link would I expect problems.

If the server has crashed and is not yet back up, the TCP FIN will be
unacknowledged.  The attempt to create a new connection will also
stall.  When the server comes up, the FIN will get garner a RST,
almost as good as a proper close.  The new connection will succeed,
the old requests retransmitted, and all will be well except maybe for
some non-idempotent requests.

One problem with this idea is what happens when the NFS server is wedged
but TCP/IP is alive.  Like on the QA system Monday.  (Sigh.)  The client
was initiating closes, but the server wasn't going to react until the
TCP stream drained.  It didn't, so the client got left with a connection
in one of the FIN_WAIT states and went off and created a new socket which
filled up with retransmits.  At least your idea of sending RPC_NULLs will
slow the client resource consumption rate.

	-Ric Werme

---

From: Mike Eisler <[email-masked]>
Date: Wed, 17 Jul 2002 19:22:34 -0700

> still up.  For the slow transport case, instead of making a client

OK, what I wrote seemed to require a particular implementation
approach for detecting "uppness", which the
spec should not do. Instead what it should say
is that instead of using retries of 32KB WRITES to probe,
one is instead allowed to use retries of more compact NULLs.

So, from where I wrote "it is necessary", change
"it is necessary" and the rest of the paragraph to:

    so the NFSv4 client may want to actively
    "probe" the connection to see if has been broken.
    Use of the NULL procedure is one permitted way to do so.
    So, when a client experiences a remote procedure call
    timeout (of some arbitrary implementation specific
    amount), rather than retrying the remote procedure
    call, it could instead issue a NULL procedure call
    to the server. If the server has died, the transport
    connection break will eventually be indicated to the
    NFSv4 client. Then client can then reconnect, and
    then retry the original request. If the NULL procedure
    call gets a response, the connection has not broken.
    The client can decide to wait longer for the original
    request's response, or it can break the transport
    connection and reconnect before re-sending the 
    original request.

---

From: RJ Atkinson <[email-masked]>
Date: Thu, 18 Jul 2002 00:34:15 -0400

>     Use of the NULL procedure is one permitted way to do so.

??? s/permitted/recommended/ ???

>     connection and reconnect before re-sending the

The above is pleasantly neutral.  However, it offers no advice
for the well-intentioned but less wizardly implementer.
In NFS, as in the Internet in general, clue appears to be
either a constant or to have linear growth over time --
whereas the number of folks in the NFS community appears
to grow geometrically.  So having the occasional free clue
documented as such would increase the chance that the code
I have to deploy would be goodly.

Could we consider offering some (entirely optional to
follow) good advice for such well-intentioned non-wizard
NFS implementers (in this regard) please ?

Thanks,

Ran

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 18 Jul 2002 10:09:36 -0700

Written this way, the advice to substitute a null call for a
"fat" retransmission seems like good advice, particularly
on a slow link - whatever the version of NFS.  

Has anyone tried it ?

	Brent