Subject: RE: filehandle volatility, stateid, and "state leakage"

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 23 Jan 2001 14:05:09 -0800

Kendrick Smith wrote:
> Looking back at the post-bakeoff mailing list postings, it seems to me
> that we never reached consensus on whether stateids should be per
> lockowner or per (lockowner,file)-pair.

I don't think we have reached consensus on any of the major outstanding
issues: stateid scope, are 64-bit stateid's big enough, and the NT server
vs. open-downgrade imbroglio.  There was a flurry of discussion and then
nothing on the mailing list.  I suspect (or hope) that it's basically a
case of everybody being too busy coding nfs-v4 implementations to follow
up.   

> The last mention I see of this is
> in Mike Eisler's message dated 11/03, detailing changes that should be
> made in the spec if it is decided that stateids should be per lockowner.  
> Since those changes have not yet been made, is it safe to assume that
> stateids will remain per (lockowner,file)-pair?

Since the current RFC embodies both interpretations (in different places), 
I don't think any assumption is safe right now, if you are talking about 
the ultimate resolution.

OTOH, my server has (lockowner,file)-pair stateid's and it won't change for
Connectathon unless a decision to go to per-lockowner stateid's happens
within the next few weeks.  I'm assuming that is true for a lot of
implementations, so staying with with per (lockowner,file)-pair stateid's
is probably safe for Connectathon, at least.
  
> 
> My reason for bringing this up is related to a problem that I have run 
> into in our client implementation.  Consider this scenario:
>   1.  Client A opens a file.
>   2.  Client A acquires locks on the file.
>   3.  Client B renames the file.  Because the filehandles of this server
> are volatile on rename, this changes the file's filehandle.
>   4.  Client A tries to release its locks.
> In step 4, the client must issue a PUTFH for the file, since for a LOCKU
> operation, the spec requires the current filehandle to match the file
> being unlocked.  Because the filehandle changed in step 3, the PUTFH will
> fail with error NFS4ERR_BADHANDLE.  Futhermore, the client will not be
> able to recover the new filehandle, since it has no way of knowing the
> new filename specified by client B in step 3.  Therefore it will never be
> able to release its locks on the file (or close it, since the CLOSE
> operation has the same filehandle restriction as the LOCKU operation).
> This "state leakage" will persist for the lifetime of the client (assuming
> that the client periodically emits RENEW operations)!

I feel a volatile filehandle tirade coming on, but I suppose I can choose 
not to commit it to writing. :-)

> One possible solution to this problem is to add the restriction that,
> while a file has been opened on the server, the server cannot change its
> filehandle.  For our Linux server, this would be quite problematic; our
> filehandles have to be volatile on rename (in case anyone is curious
> why, I've summarized the reasons in a note at the end of this message).
> 
> Another solution, which I think is the "right" way to handle this issue,
> is:
>   1.  Agree that stateids are per (lockowner,file)-pair, rather than per
> lockowner.  This means that, in a LOCKU or CLOSE operation, the stateid
> provided by the client is sufficient to specify the file; the current
> filehandle would be redundant.

Only for LOCKU and CLOSE, I suppose.  For READ and WRITE, this would 
not work, since you are allowed to specify the special stateid's
which don't identify the file.

>   2.  Remove the restriction for the spec that, in a CLOSE or LOCKU
> operation, the current filehandle has to be set to the opened file.  The
> stateid specified with the operation now identifies the file.

That wouldn't be unprecedented.  RENEW takes a stateid but no filehandle
is mentioned in the spec (even the current filehandle).  My server
at least uses the stateid and doesn't look at the current filehandle 
for RENEW. 
  
> 
> What does everyone think of this?  Is there a reason I'm unaware of for
> requiring the current filehandle to be set in a CLOSE or LOCKU operation?

My guess is that it just seemed natural to always pass the filehandle.
In cases in which special stateid's may not be used, the server has
to be able to determine what file you are talking about, so I don't
see that requiring it explicitly is adding anything.
  
> This proposal would give the NFSv4 stateid similar semantics to a UNIX
> file descriptor; in UNIX, a file descriptor can be used to refer to a file
> uniquely for as long as it remains open, even if the filename changes.
> This proposal would let the NFSv4 stateid refer to a file uniquely for as
> long as it remains open, even if the filehandle changes (although the
> analogy between stateids and file descriptors isn't perfect, since the
> stateid can change, whereas a file descriptor is constant).

Also, you can't read and write with stateid's alone, unlike file 
descriptors.
 
> 
> If the only reason that the filehandle is required to agree with the open
> file during a CLOSE or LOCKU is to give the server an error check, I
> personally think that the possible benefits of this additional check would
> be outweighed by the problems created by this "state leakage" issue.  
> After all, the server can still do error-checking on the stateid, and
> determine if it is stale, bad, or old.  Feedback?
> 
> Kendrick
> 
> p.s.  In case anyone is wondering why our Linux server's filehandles have
> to remain volatile, here is an explanation.  When a client sends a
> filehandle, we have to verify for security purposes that the file is
> contained in one of the directories approved for export on the server.  
> If the file is a directory, this is easy; we just traverse parent
> directories until we find either an export, or the root of the filesystem.
> If the file is a non-directory, however, we have a problem; we can't
> uniquely determine the directory containing it, since it may be hardlinked
> into several directories on the server.  At best, we could traverse a list
> of directories containing the file, to try to find one contained in an
> export (which would present a performance problem), but even this is
> impossible since the Linux low-level fs interfaces don't provide a way to
> obtain a list of all directories containing a given (non-directory) file.
> Therefore, our filehandles have to include, in addition to the inode
> number of the file, the inode number of its containing directory, so that
> we can locate the parent directory for export checking.  When the file is
> renamed, its parent directory may also change, which means the filehandle
> can change.  Our volatile filehandles are therefore a by-product of a
> server environment in which (1) we don't want to export the entire
> filesystem, only certain directories, and (2) because of hardlinks, it is
> not possible to determine the directory containing a given file.
> 
> 
>

---

From: Salit Gazit <[email-masked]>
Date: Tue, 23 Jan 2001 15:03:03 -0800 (PST)

Kendrick,

>lockowner or per (lockowner,file)-pair.  The last mention I see of this is
>fail with error NFS4ERR_BADHANDLE.

The error should be NFS4ERR_FHEXPIRED to indicate that the filehandle
expired so the client can try to look it up (but I assume that the
server probably does not know that it expired when it gets the old
filehandle?).
Obviously if the path to it was changed by a different client,
the lookup will still fail.

>Futhermore, the client will not be

Would it work to have the server set the attribute
fh4_expire_type to (FH4_NOEXPIRE_WITH_OPEN | FH4_VOL_RENAME), and reject
rename requests while the file is open, unless the directory is the
same (perhaps with an exception when the renaming client is the
one holding the lock)?

Salit

---

From: Eric Kustarz <[email-masked]>
Date: Tue, 23 Jan 2001 15:41:06 -0800 (PST)

>From: "Noveck, Dave" <[email-masked]>
>To: "'Kendrick M. Smith'" <[email-masked]>, [email-masked]
>Subject: RE: filehandle volatility, stateid, and "state leakage"
>Date: Tue, 23 Jan 2001 14:05:09 -0800
>MIME-Version: 1.0
>

>is probably safe for Connectathon, at least.
We (SUN) too will be going with stateid as a (lockowner, file) pair.

eric