Subject: RFC3010's state model with a unix/posix client

From: Spencer Shepler <[email-masked]>
Date: Tue, 31 Jul 2001 14:56:35 -0500

There is an outstanding issue with respect to the support of POSIX
locking requirements and the corresponding state management in
RFC3010.  

To restate the problem, with a Posix client in the event of a fork(),
the child process has a file descriptor set that matches that of the
parent process.  The child is then able to read/write and more
importantly lock byte ranges of those file descriptors.  With the
current definition/interpretation of the stateid, a client
implementation can not properly provide the Posix locking
functionality for the child process.  The reason for this is that the
lock_owner represents a process at the client and that lock_owner is
associated at OPEN time.

There have been a couple of suggestions to enable posix locking at the
client.  The first was to have the client interpret the lock_owner more
broadly and therefore arbitrate locking interaction at the client
first (between processes that share heritage) and then move requests
to the server for final arbitration.  The second suggestion was to
introduce a OPENFH (for lack of a better operation pneumonic) that
would take a filehandle and be used to associate a new lock_owner with
what was to be an active or open file at the client.

The first of these suggestions places additional implementation burden
at the client.  One can argue that this implementation burden will be
present in support of delegations.  However, if the client
implementation does not provide support for delegations then this work
will not be present.  Also, it is unclear if there are implementation
or protocol problems with this particular interpretation.

The second suggestion of using OPENFH also does not solve some
implementation issues.  For example, the Solaris API for filesystems
does not provide a reference to the "file descriptor" returned to a
process as a result of open().  The Solaris filesystem API passes the
vnode or underlying filesystem data structure. This vnode represents
the file on that client filesystem for all processes who hold an
open() reference.  Therefore, the NFSv4 client on Solaris has the
pointer to the shared filesystem data structure and the process-id of
the requestor to use as keys to find related data structures.  In the
case of a child process making a reference to a file descriptor that
was inherited from the parent, the Solaris NFSv4 client would not be
able to reliably find the data structure that contains the stateid.
Therefore, the presence of the OPENFH operation does not resolve how
the Solaris NFSv4 client would keep from having dangling references to
an OPEN file at the server.

At the bakeoff, we had some discussions about these issues and
formulated a possible solution that would allow for the model that
most Unix/Posix clients implement while offering a solution that would
satisfy a Windows client as well.  

The Ideas/Proposal:

The first major change is to move to a two-level state model in
NFSv4.  Yes, this is a major change but builds on the existing
experience in NFSv4 of using stateids and sequence numbers to handle
the various cases of retransmissions, etc.  This model is similar to
the Unix/Posix model for open files that are shared between parent and
child and file locking which is not.

With this new model, there is what we will refer to as an OPEN state
stream that is represented by an open_owner/open_stateid.  This
open_owner/open_stateid is the "parent" of any locking state that is
instantiated at the client for the open file.  The open_owner
generally represents a parent process and its children since they
share file descriptors.  The client implementation may choose to
expand that interpretation to include all processes that open() the
file and that share credentials (as an example).

From this open_owner/open_stateid, the client can then instantiate a
"traditional" lock_owner by referring to the open_stateid.  The
lock_owner then can be used for a file locking state stream (which
would include read/write requests).  Generally, this lock_owner would
be initiated when a file lock request is made and would represent the
process.

Both open_owner and lock_owner would have their own sequence_ids so
that related requests are ordered at the server.  The rules that exist
today for replay would remain in place.

The OPEN operation would look something like this (a lot of existing
detail has been left out to simplify the discussion):

OPEN

	(cfh), name, open_owner, open_seqid -> (cfh), open_stateid

LOCK
	(cfh), open_stateid, lock_owner, lock_seqid -> (cfh), lock_stateid

There would still be an OPEN_CONFIRM call to be used when the server
first encounters an open_owner/seqid.  We would also introduce a
LOCK_CONFIRM to establish the lock_owner/seqid as well

Because we are introducing an open_owner that potentially represents
multiple processes at the client and the OPEN call uses a name for the
OPEN, it is difficult or burdensome for the client to serialize OPEN
operations to the server.  This introduces the problem that the client
may generate two OPEN requests to the server that use the same
open_owner.  This would be the case where hardlinks are present or the
client chooses not to serialize OPENs.

If this occurs, the server will return two open_stateid values to the
client but the client will not know which one to use.  To address this
problem and the outstanding request that stateid values be enlarged in
the protocol, the stateid will become a little less opaque and look
something like this:

  stateid4 {
  	 uint32_t	seqid;
	 opaque		serverstuff<12>;
  };

The server will use the seqid to indicate the order in which the
stateids for the related *_owner value are created.  

In the case of simultaneous OPENs, the client will inspect
open_stateid.seqid to determine which stateid is the most recent.  The
client will then use that stateid for subsequent operations.

For the Windows client that does not require the ability to have
multiple lock_owners associated with an open_owner, there will be a
one lock_owner per open_owner.

As an optimization, the OPEN operation can also be augmented to handle
the general case of one open_owner to one lock_owner with something
like:

OPEN

	(cfh), name, open_owner, open_seqid, lock_owner, lock_seqid -> 
		     (cfh), open_stateid, lock_stateid