Subject: Re: filehandle volatility, stateid, and "state leakage"

From: "Kendrick M. Smith" <[email-masked]>
Date: Thu, 1 Feb 2001 13:59:04 -0500 (EST)

On Wed, 31 Jan 2001, Neil Brown wrote:

> On Tuesday January 23, [email-masked] wrote:
> > 
> > Looking back at the post-bakeoff mailing list postings, it seems to me
> > that we never reached consensus on whether stateids should be per
> > lockowner or per (lockowner,file)-pair.  The last mention I see of this is
> > in Mike Eisler's message dated 11/03, detailing changes that should be
> > made in the spec if it is decided that stateids should be per lockowner.  
> > Since those changes have not yet been made, is it safe to assume that
> > stateids will remain per (lockowner,file)-pair?
> > 
> > My reason for bringing this up is related to a problem that I have run 
> > into in our client implementation.  Consider this scenario:
> >   1.  Client A opens a file.
> >   2.  Client A acquires locks on the file.
> >   3.  Client B renames the file.  Because the filehandles of this server
> > are volatile on rename, this changes the file's filehandle.
> >   4.  Client A tries to release its locks.
> > In step 4, the client must issue a PUTFH for the file, since for a LOCKU
> > operation, the spec requires the current filehandle to match the file
> > being unlocked.  Because the filehandle changed in step 3, the PUTFH will
> > fail with error NFS4ERR_BADHANDLE.
> 
> The server does not have to give an error here.  Just because the
> filehandle-that-would-be-generated has changed doesn't mean filehandles
> that were generated earlier need no longer be recognised.  It is quite
> valid for a server to recognise multiple different filehandles as
> referring to the same file.  There is a "unique_handles" attribute
> which allows the client to discover if this might happen.
> 
> In your implementation, there should be no problem locating the file for
> an old (pre-rename) filehandle, especially if it is still in the
> dcache, which I suspect it would be if there were a current lock on
> the file.  So when Client A presents an "old" filehandle, there should
> be no problem recognising it and doing what A expects.
>
> Things might be bit harder if the server was restarted, but they
> aren't impossible.  This is where the FH4_NOEXPIRE_WITH_OPEN flag is
> particularly relevant.
> If a server provides neither FH4_PERSISTENT nor
> FH4_NOEXPIRE_WITH_OPEN, then a client cannot really trust the server
> enough to do locking reliably.
> To provide FH4_NOEXPIRE_WITH_OPEN semantics with your implementation
> you would need to do a bit of extra work:
>   Whenever a file which is open is renamed to a different directory,
>   you would need to log to stable storage enough information to
>   reliably recognise old filehandles. I suspect that the minimum that
>   you could get away with is a record which says:
>     "File X on filesystem Y is now in directory Z".
>   where "X" is probably an inum+generation or whatever is meaningful
>   to the filehandle.
>   Given such a log, then when an old filehandle is presented during
>   the state-reclaiming process, you can find out which directory (or
>   directories) the file is now in, and you and do the necessary
>   permission checks.
>

If it is a requirement that an NFSv4 server recognize old filehandles for
the duration of an OPEN (or disallow renames of open files, which we don't
want to do, since this is legal in almost all unix filesystems, and
existing software may depend on this behavior), it is certainly possible
for us to arrange for server to do so.  If nothing else, we could always
keep a hash table of filehandles associated with open files, and search
this table if we fail to find a filehandle during a PUTFH.

But I'm still arguing in favor of removing this requirement from the
protocol, and letting the stateid alone identify the open file.  (Actually
I'm in favor of doing this for all operations requiring a stateid, with
the caveat that if a "special" stateid is specified during a READ or
WRITE, the current filehandle is required.) My argument for making this
change is based on 3 assumptions:

1.  There is no compelling reason for requiring the current filehandle to
be set (and agree with the operated-on file) during stateid-requiring
operations (such as error-checking, weird "corner cases" during client
reboot, etc.)

2.  If we remove this requirement, the necessary server-side changes can
be made more or less "for free" in currently existing server 
implementations.  For example, on our Linux server, we look up the stateid
first, then execute a few lines of code which check for agreement between 
the current filehandle and the operated-on file.  Making this change would
amount to removing these lines of code, and not adding any.  Future
implementations would not have to worry about inserting these filehandle
checks.

3.  This requirement can be changed in the spec without hindering the RFC
standards process.  I think it's safe to say that we're still in a stage
where minor tuning of the spec will still be needed (note that making this
change doesn't involve changing the underlying protocol at all, just
removing a few lines of text where the requirement is stated).  I would be
willing to submit a list of spec changes to the mailing list.

If any of these assumptions are in error, I think that my proposal should
be retracted.  On the other hand, my argument in favor of the proposal is:

  Current and future server implementations which need volatile
  filehandles will have to deal with this extra issue of recognizing old
  filehandles for the duration of an OPEN, which introduces more
  complexity and "corner cases" to test.  Given the current complexity of
  the protocol, I would like to streamline away unecessary complexity if
  possible, particularly if the streamlining can be done "for free"
  (assumption 2).  (In addition, one could argue that this change will
  slightly improve the efficiency of the protocol, since it avoids the
  overhead of some PUTFH's, but I find it doubtful that the change would
  be significant.)  Personally, I am worried that, because NFSv4 is so
  complex, there will be a long delay between release of the initial
  implementations and production use, as corner cases are hunted down
  and stamped out.

- Kendrick