Subject: Re: questions/comments on draft-ietf-nfsv4-rfc3010bis-02-04.txt

From: David Robinson <[email-masked]>
Date: Thu, 22 Aug 2002 11:17:13 -0500 (CDT)

> Interesting. This seems to take us down the slippery slope of
> fully sequenced RPC that goes beyond just locking operations.
> On the other hand, even with NFSv[23], clients are doing synchronous
> IO when advisory locking is on, so there is already an expectation
> of reduced IO throughput when locking is enabled.
> 
> Still, in the past, we've relied on the duplicate request
> cache to catch dangling writes, and operational experience
> for the past 15 years or so says it works fine.
> I think returning _OLD_STATEID on
> an I/O ought to be a server implemention choice.

This scenerio has been within the locking design from day one,
locking operations are synchronization points (aka membars),
no I/O operation issued before one can be processed after one.
While it may seem restrictive, reality is that virtually all
implementations will wait for all outstanding I/Os to complete
before changing their lock state, it makes little sense to
give up control of data when you have not been assured that
it has been written. This leaves only retransmitted operations
to receive the errors which a client will hopefully discard as
a reply to a request that is no longer outstanding.

If the server is given the option of rejecting or not rejecting
I/O operations with an old stateid then we have defeated the
primary purpose for including them in I/O operations in the
first place.  We would need to add new configuration attributes
so the server can tell the client that it does not protect against
corruption caused by replays so that the client can try something
to save itself.

I think the works fine for 15 years is primarily because customers
have simply avoided locking in NFS. A better perspective is to
look at the expectations in the Windows world where they
actually regularly use locking and depend on the transport to
maintain sequencing.

	-David

---

From: Eric Werme USG <[email-masked]>
Date: Fri, 23 Aug 2002 15:51:43 -0400

> actually regularly use locking and depend on the transport to

Our production system serves mail via NFS and also accesses things like
.forward files via NFS.  I haven't been dragged over to help with an Email
problem in ages, especially after changing the NFS client to use a single
source port so we don't exhaust the privileged port space.  And we still
have (shudder) rpc.lockd!

We _have_ seen some failures of the DRC in the QA stress tests.  Two
key things have changed in the last 15 years that make me fear the ice is
getting awfully thin:

1) We used to think that 50% usage of 10BaseX networks meant they were
   severely overloaded.  Today we can saturate 1000BaseT, at least with
   big I/Os and Jumbo frames.
2) Some non-idempotent ops take little time, e.g. unstable V3 writes or
   almost anything on systems with big battery backed caches.

NFS over TCP helps quite a bit thanks to guaranteed delivery, long NFS
retransmit delays, and servers that infrequently crash.  Of course, we
haven't kept up TCP window sizes with the media speed.   How many of
us use windows that buffer a second's worth of 1000BaseT data?  I was
experimenting at Connectathon with a 500 KB window, I "just couldn't" bring
myself to try 125 MB!

How long does a DRC entry need to stick around?  At the very least a
couple seconds, or whatever a minimum UDP retransmit delay is.  If
several processes are trying to delete a couple million files and one
reply gets lost, The DRC needs to be big enough to hold the full speed
delete stream from the other processes.  Call it 250 bytes per
request, that's 2,000 bits or 2 usec of clock time at 1000BaseT rates.
Two seconds of DRC time is a million requests.  Tru64's DRC ain't that
big....

	-Ric Werme