Subject: filehandle volatility, stateid, and "state leakage"

From: "Kendrick M. Smith" <[email-masked]>
Date: Tue, 23 Jan 2001 16:10:25 -0500 (EST)

Looking back at the post-bakeoff mailing list postings, it seems to me
that we never reached consensus on whether stateids should be per
lockowner or per (lockowner,file)-pair.  The last mention I see of this is
in Mike Eisler's message dated 11/03, detailing changes that should be
made in the spec if it is decided that stateids should be per lockowner.  
Since those changes have not yet been made, is it safe to assume that
stateids will remain per (lockowner,file)-pair?

My reason for bringing this up is related to a problem that I have run 
into in our client implementation.  Consider this scenario:
  1.  Client A opens a file.
  2.  Client A acquires locks on the file.
  3.  Client B renames the file.  Because the filehandles of this server
are volatile on rename, this changes the file's filehandle.
  4.  Client A tries to release its locks.
In step 4, the client must issue a PUTFH for the file, since for a LOCKU
operation, the spec requires the current filehandle to match the file
being unlocked.  Because the filehandle changed in step 3, the PUTFH will
fail with error NFS4ERR_BADHANDLE.  Futhermore, the client will not be
able to recover the new filehandle, since it has no way of knowing the
new filename specified by client B in step 3.  Therefore it will never be
able to release its locks on the file (or close it, since the CLOSE
operation has the same filehandle restriction as the LOCKU operation).
This "state leakage" will persist for the lifetime of the client (assuming
that the client periodically emits RENEW operations)!

One possible solution to this problem is to add the restriction that,
while a file has been opened on the server, the server cannot change its
filehandle.  For our Linux server, this would be quite problematic; our
filehandles have to be volatile on rename (in case anyone is curious
why, I've summarized the reasons in a note at the end of this message).

Another solution, which I think is the "right" way to handle this issue,
is:
  1.  Agree that stateids are per (lockowner,file)-pair, rather than per
lockowner.  This means that, in a LOCKU or CLOSE operation, the stateid
provided by the client is sufficient to specify the file; the current
filehandle would be redundant.
  2.  Remove the restriction for the spec that, in a CLOSE or LOCKU
operation, the current filehandle has to be set to the opened file.  The
stateid specified with the operation now identifies the file.

What does everyone think of this?  Is there a reason I'm unaware of for
requiring the current filehandle to be set in a CLOSE or LOCKU operation?  
This proposal would give the NFSv4 stateid similar semantics to a UNIX
file descriptor; in UNIX, a file descriptor can be used to refer to a file
uniquely for as long as it remains open, even if the filename changes.
This proposal would let the NFSv4 stateid refer to a file uniquely for as
long as it remains open, even if the filehandle changes (although the
analogy between stateids and file descriptors isn't perfect, since the
stateid can change, whereas a file descriptor is constant).

If the only reason that the filehandle is required to agree with the open
file during a CLOSE or LOCKU is to give the server an error check, I
personally think that the possible benefits of this additional check would
be outweighed by the problems created by this "state leakage" issue.  
After all, the server can still do error-checking on the stateid, and
determine if it is stale, bad, or old.  Feedback?

Kendrick

p.s.  In case anyone is wondering why our Linux server's filehandles have
to remain volatile, here is an explanation.  When a client sends a
filehandle, we have to verify for security purposes that the file is
contained in one of the directories approved for export on the server.  
If the file is a directory, this is easy; we just traverse parent
directories until we find either an export, or the root of the filesystem.
If the file is a non-directory, however, we have a problem; we can't
uniquely determine the directory containing it, since it may be hardlinked
into several directories on the server.  At best, we could traverse a list
of directories containing the file, to try to find one contained in an
export (which would present a performance problem), but even this is
impossible since the Linux low-level fs interfaces don't provide a way to
obtain a list of all directories containing a given (non-directory) file.
Therefore, our filehandles have to include, in addition to the inode
number of the file, the inode number of its containing directory, so that
we can locate the parent directory for export checking.  When the file is
renamed, its parent directory may also change, which means the filehandle
can change.  Our volatile filehandles are therefore a by-product of a
server environment in which (1) we don't want to export the entire
filesystem, only certain directories, and (2) because of hardlinks, it is
not possible to determine the directory containing a given file.

---

From: Neil Brown <[email-masked]>
Date: Wed, 31 Jan 2001 10:33:59 +1100 (EST)

On Tuesday January 23, [email-masked] wrote:
> fail with error NFS4ERR_BADHANDLE.

The server does not have to give an error here.  Just because the
filehandle-that-would-be-generated has changed doesn't mean filehandles
that were generated earlier need no longer be recognised.  It is quite
valid for a server to recognise multiple different filehandles as
referring to the same file.  There is a "unique_handles" attribute
which allows the client to discover if this might happen.

In your implementation, there should be no problem locating the file for
an old (pre-rename) filehandle, especially if it is still in the
dcache, which I suspect it would be if there were a current lock on
the file.  So when Client A presents an "old" filehandle, there should
be no problem recognising it and doing what A expects.

Things might be bit harder if the server was restarted, but they
aren't impossible.  This is where the FH4_NOEXPIRE_WITH_OPEN flag is
particularly relevant.
If a server provides neither FH4_PERSISTENT nor
FH4_NOEXPIRE_WITH_OPEN, then a client cannot really trust the server
enough to do locking reliably.
To provide FH4_NOEXPIRE_WITH_OPEN semantics with your implementation
you would need to do a bit of extra work:
  Whenever a file which is open is renamed to a different directory,
  you would need to log to stable storage enough information to
  reliably recognise old filehandles. I suspect that the minimum that
  you could get away with is a record which says:
    "File X on filesystem Y is now in directory Z".
  where "X" is probably an inum+generation or whatever is meaningful
  to the filehandle.
  Given such a log, then when an old filehandle is presented during
  the state-reclaiming process, you can find out which directory (or
  directories) the file is now in, and you and do the necessary
  permission checks.

I suspect that in real life, renaming of files which are open does not
happen very often, so this log would not impose an excessive
performance hit.

>  Futhermore, the client will not be
> What does everyone think of this?

I'm actually in favour of not ever requiring a filehandle if you have
a stateid, but that has be discussed before - search for "statehandle"
in the list archives and you should find the right thread.

NeilBrown