Subject: Re: [nfsv4] LOCK and locker

From: Rick Macklem <[email-masked]>
Date: Fri, 1 Jul 2011 16:40:28 -0400 (EDT)

Thomas Haynes wrote:
> Rick and I have been going over an issue with the following sequence:
> 
> 
> 
> 
> - I'll use labels like O1, L1 for the stateids, to save typing
> Packet#
> 119 open lockfile1279
> 120 ok fh f7ca5653 stateid O1
> 135 open lockfile1279
> 136 ok (should be same fh) stateid O2
> 137 Lock stateid O1 new_lock_owner lock_seqid 0
> 138 ok stateid L1
> 143 LockU fh f7ca5653 stateid L1 lock_seqid 1
> 144 ok
> 147 Close fh f7ca5653 stateid O2
> 148 ok
> 149 Close fh f7ca5653 stateid O1
> 150 ok
> 
> - now on to the next test which creates the file again, etc
>  but does pretty much the same thing
> 159 open lockfile1279
> 160 ok fh f6ca5753 stateid O3
> 175 open lockfile1279
> 176 ok (should be same fh) stateid O4
> 177 Lock f6ca5356 stateid O3 new_lock_owner lock_seqid 0
> 178 ok stateid L2
> 183 LockU fh f6ca5356 stateid L2 lock_seqid 1
> 184 NFS4ERR_BAD_SEQID
> 
> It's the NFS4ERR_BAD_SEQID that gets turned into EACCES.
> 
> For some reason, the Netapp server thinks a lock_seqid == 1
> is incorrect the second time this is done. (I surmise it is
> because the lock_owner string is the same for #137 and #177
> and the Netapp server didn't reset lock_seqid back to 0 when
> it did #177, even though the client specified a new initial
> value of 0 in the #177 request.)
> 
> As far as I know, reuse of lock_owner strings is allowed. If
> not, a client would have to remember them forever to avoid reuse.
> 
> 
> 
> 
> In summary, a lock-owner is created for file #1, that file is closed
> (and
> removed), and then the same lock-owner is presented for the next file.
> 
> 
> 3530 has this to say about it:
> 
> 
> 
> 
> 
>    The locker argument specifies the lock_owner that is associated
> with
>    the LOCK request.  The locker4 structure is a switched union that
>    indicates whether the lock_owner is known to the server or if the
>    lock_owner is new to the server.  In the case that the lock_owner
> is
>    known to the server and has an established lock_seqid, the argument
>    is just the lock_owner and lock_seqid.  In the case that the
>    lock_owner is not known to the server, the argument contains not
> only
>    the lock_owner and lock_seqid but also the open_stateid and
>    open_seqid.  The new lock_owner case covers the very first lock
> done
>    by the lock_owner and offers a method to use the established state
> of
>    the open_stateid to transition to the use of the lock_owner.
> 
> 
> 
> 
> 
> The crux of this text is whether the lock_owner is known to the server
> or is new to the server. The NetApp server already knows about the
> lock-owner via the first file open. It is easy enough for the server
> to
> let the client be smart and reuse the lock_owner string. If the client
> did it incorrectly, i.e., the lock-owner was based on the PID and in
> use
> elsewhere, then there will be problems found later on.
> 
> 
> How many client implementors reuse the lock_owner string?
> 
> 
> Or considering this is the first time we've hit this issue in our
> server,
> how many client implementors think it is reasonable to reuse the
> lock_owner string?
> 
Here's a summary of what I can see from some testing I've been doing
and a look at the old [email-masked] archives, etc:

On whether there is a single lock_seqid for all uses of a given
lock_owner4 string:
Recent discussion and the following seem to indicate the spec meant
there to be a single lock_seqid.
  http://www.nfsv4.org/nfsv4-wg-archive-dec-96-jan-03/3154.html
  - look at the end of the message, which seems to suggest multiple
    lock stateids, but one lock_seqid

however...
Most servers (Solaris10, Linux, FreeBSD and maybe others, but not all)
appear to create a separate lock_seqid for each lock stateid issued in
reply to a Lock Op with open_to_lock_owner4 using the same lock_owner4 string.

--> The Solaris10 and Linux clients avoid the issue by not reusing the
    same lock_owner4 string.
--> A patch that has now been tested against the server where it did
    not appear to have separate lock_seqids for this case avoids the
    problem by appending the FH into the lock_owner4 string, so that the
    lock_owner4 string is different for different files that a <pid> is
    locking concurrently.
*** I think clients must avoid the case where they are using the same
    lock_owner4 string for multiple open_to_lock_owner4 Lock Ops
    (for different files with different open stateids). If the client
    does this, it will get multiple lock stateids, but won't be able
    to provide a lock_seqid sequence that works for all extant servers.

On whether or not a lock_owner (lock_owner4 string plus lock_seqid) will
outlive the Close of the open stateid(s) used in the open_to_lock_owner4
case(s) of Lock Ops.
--> RFC3530 seems to say that it "may" outlive the Close.
    http://www.nfsv4.org/nfsv4-wg-archive-dec-96-jan-03/3954.html
    - discusses the server replying an error if the open_to_lock_owner4
      specifies a new lock_seqid for the case where the lock_owner4 string
      is already in use "while open". As such, it hints that "after close"
      this no longer applies, but isn't explicit w.r.t. it.
David Noveck states it does outlive the Close.
Trond Myklebust and I don't see why this is the case.
--> My recently tested patch does a ReleaseLockOwner on the lock_owner4
    string before the Close. This seems to fix the case where the FreeBSD
    client will reuse the same lock_owner4 string when the same process
    reopens the same file and then does more Lock Ops on it. (For the
    FreeBSD client, the string represents <pid + fh>, so that the server
    will correctly check for lock conflicts between processes within the
    client.) Reuse is hard to avoid for this case, since the FreeBSD VFS/VOP
    layer doesn't notify the client when a POSIX process that was doing
    locking terminates. It only knows when the file is closed and can be
    reopened later by the same process.
*** I hope that ReleaseLockOwner on the lock_owner4 string will be sufficient
    to allow reuse of the same lock_owner4 string after that, for a different
    open stateid from an Open Op done after the ReleaseLockOwner.

Since there is no discussion that I can see in RFC3530 w.r.t. the use of
the lock_seqid in the open_to_lock_owner4, I think it would be nice to clarify
that this sets a new initial value for the lock_seqid# used by the client
for the lock_owner4 string IF there has been a Close of the open stateid.
No confirmation is required, since the open_to_lock_owner4 Lock Op is sequenced
via the open_seqid.
(To me, this in analagous to the server seeing a different open_seqid for an
 open_owner4 when there are no opens, with the server using that as the new
 initial value of the open_seqid#, except for open, confirmation is required.)
--> Without this, I think ReleaseLockOwner should get rid of the
    <lock_owner4 string + lock_seqid#> permitting reuse of the lock_owner4 string,
    but I admit I find the description of ReleaseLockOwner vague enough that
    I can't be sure of this?

Btw, long ago I asked about this, but I can't see an answer in the archive.
   http://www.nfsv4.org/nfsv4-wg-archive-dec-96-jan-03/3544.html

If anyone thinks that what my patched client (using different lock_owner4
strings for different files + ReleaseLockOwner before reuse of the same
lock_owner4 string for a reopen/lock of the same file) will now do won't work
with their server, please let me know.

Thanks for the help with this, rick