Subject: Re: NFSv4 Locking case study

From: Stevan Steve Allen <[email-masked]>
Date: Wed, 5 Mar 2003 08:32:16 -0800

A server condition is described in the draft without a corresponding error
or attribute to indicate the condition.

Chpt 8.1.4. (for example, some UNIX-based servers support a "mandatory lock
bit" on the mode attribute such that if set, record locks are required on
the file "before" I/O is possible).

I haven't heard anything back from the WG on my 2/27 post (copied below),
just wondering if anyone is looking at this.  Thanks.

Existing protocol attributes and errors do not describe the 8.1.4 verbiage
above.

o NFS4ERR_LOCKED indicates a conflicting lock. A UNIX-based server may fail
the I/O request because there is "no lock".
o OPEN4_RESULT_LOCKTYPE_POSIX does not indicate byte range locking is
required according to Mr. Eisler.
o NFS4ERR_OPENMODE is similar to 8.1.4 but only describes file share
failures, not byte range.

From: Mike Eisler <[email-masked]>
Date: 03/02/2003

|> restricted.  Clients should not interpret this error as re-tryable.  For
|> reference, the NFSv4 draft mentions similar byte range semantic
enforcement
|> (chpt. 8. "for example, some UNIX-based servers").   We suspect a need
for
|> a specific error to represent this condition.   NFS4ERR_LOCKED is not
|> appropriate as it signifies a re-tryable condition.  We are proposing an
|> permanent error to be returned to the application layer.
|
|What if the application then decides to retry? How has this
|solved anything?

Different platforms support different locking semantics.  For example, a
server which supports lock upgrades may support a write while holding a
shared lock.  Other servers not supporting lock upgrades normally expect an
exclusive lock around both the read and write.  In some cases, the
application (and client on the appls behalf) can not determine or support
the locking semantics expected.  In this event the "user" must be notified.
Existing example...

NFS4ERR_OPENMODE states the file has imposed access requirements (decided
by the file owner or platform).  The application I/O request will continue
to fail until the proper file share is obtained.  The error is always
returned to the application (see draft for reasons).  If the application
does not support the expected file share semantics, the application can not
be used "at all" to access this file.  The client user needs to be
notified.

Draft example. Lock upgrades may not be supported by the server.  A retry
of the I/O or attempted lock upgrade (by client or appl) may continue to
fail indefinitely.  The error is to be returned by to the application.

For "required" byte range locking, I made a case in my prior post that the
client may be unable to determine the proper host byte range locking
semantics.  The byte range requirements can only be handled by a proper
application.  As such, NFS4ERR_OPENMODE and the similarly proposed
NFS4ERR_RANGEMODE are application requirements.  The error indicates the
application needs to obtain a proper lock, or the user needs to use a
different application which supports the expected locking semantics.

We are a prominent server platform with legacy files requiring byte range
locking for access.  I am looking to remove NFS deficiencies and allowing
more NFS solutions to customers.  The only current option the customer has
to prevent access is using ACLs or NFS4ERR_OPENMODE.  NFS4ERR_OPENMODE and
the proposed NFS4ERR_RANGEMODE has the same effect (restricting access) and
performs it better than ACLs (i.e. an application bug may not be caught
when only using ACLs).   Platform or file owner locking semantics
requirements needs to be returned to the application.

Since this always represents an application requirement, there is no need
to draft the clients reaction (masking the error).  We do not want the
client to mask the proposed NFS4ERR_RANGEMODE error

|> Question.  OPENs rflags "OPEN4_RESULT_LOCKTYPE_POSIX indicates the
server's
|> file locking behavior supports the complete set of Posix locking
|> techniques."  I do not believe this indicates I/O requests require
|> proper<?> byte range locking is required.
|
|Correct. If he server returns NFS4ERR_LOCKED though, this
|tells the client that byte range locking is needed.

I disagree.  According to the draft, for some (all?) implementations
NFS4ERR_LOCKED indicates a conflicting lock.  A simple retry of the I/O may
later be successful (without a lock).  NFS4ERR_LOCKED is not appropriate
when there are no locks held on the region.  NFS4ERR_RANGEMODE indicates
the application does not have the proper lock.

Original Post

Given applications which support byte range, or file share semantics are
considered well behaved. Applications which do not support such semantics
are considered rouge applications according to file access needs of various
customer solutions and platforms.  Decided by the file owner, such rouge
applications may need to be permanently restricted from file access.

File access requirements may describe a server which does not support share
upgrade and expects an exclusive latch to be held around both the read and
write request.   A client wrapping an open, byte range lock or providing
share upgrade requests on behalf of unprotected I/O (i.e. a stateid of all
zeros) may not provide the required semantics as expected from a well
behaved application.  To do so is a challenge given mixed semantics of
various platforms.  As such, there is a need in the protocol for the file
owner (server) to reflect solution/platform file access requirements to the
requesting application layer.  The NFS protocol describes such server to
application communication as servers which may require proper open file
share semantics may return NFS4ERR_OPENMODE for improper READs or WRITEs
when mandatory locking is in effect.   For the client, Section
14.2.36.(Write) describes returning the error to the application as the
server may not support a share upgrade function or attempted the operation
and failed.  The "does not support upgrade" describes an permanent
application requirement which could not (should not) be interpreted by the
client.

Need:

To ensure proper semantics, range level sharing solutions for performance
may require file share only or stateid of all zeros implementations to be
restricted.  Clients should not interpret this error as re-tryable.  For
reference, the NFSv4 draft mentions similar byte range semantic enforcement
(chpt. 8. "for example, some UNIX-based servers").   We suspect a need for
a specific error to represent this condition.   NFS4ERR_LOCKED is not
appropriate as it signifies a re-tryable condition.  We are proposing an
permanent error to be returned to the application layer.

This approach allows NFS server implementations to enforce needed file or
platform level locking semantics and provides reasonably predictable access
to public files during run time.  (plus catches a few appl bugs).  Given,
performance solutions such as byte range sharing usually limit participants
to similar platforms or products.   We believe this opens up more files and
increases possible NFS solutions with minimal change.

A new return value NFS4ERR_RANGEMODE for I/O type requests is proposed.  An
attribute would also be beneficial.  This is also a file owner enforced
restriction.   Descriptions below.

note: Upon receiving the error, the client should not
    obtain a byte range lock or open and retry the I/O
    request on behalf of the application.  The error
    should be returned to the client application as it
    represents a permanent failure condition (e.g.
    application not well behaved).  If we are past
    cutoff for new errors, what is the proposed
    substitute for nfs4err_rangemode below which
    UNIX based servers should return?

NFS4ERR_OPENMODE existing return
    The client attempted a READ, WRITE, LOCK or
    SETATTR operation not sanctioned by the stateid
    passed (e.g. writing to a file opened only for
    read).

NFS4ERR_RANGEMODE new return
    The client attempted a READ, WRITE, LOCK or
    SETATTR operation not sanctioned by the stateid
    passed (e.g. writing to a file without an exclusive
    byte range lock).

note: the following BYTE_RANGE & FILE_SHARE attributes
    assist clients in identifying locking semantics
    being enforced by the server.  i.e. Useful for
    determining and providing proper delegations.

BYTE_RANGE new attribute
    Application POSIX byte range locking is required.
    The server may not support lock upgrades.
    The stateid of all zeros or ones may not be
    supported.  File share only platforms such as DOS
    and Windows may not be supported.

FILE_SHARE new attribute
    Application open file share semantics are required.
    The stateid of all zeros or ones may not be
    supported. The server may not support open upgrades.

UNSPECIFIED attribute
    This is a courtesy attribute to assist in quickly
    identifying unexpected failures.

    The server has requirements or restrictions not
    reflected in the NFS protocol.  Restrictions may be
    due to platform or product limitations. Servers may
    also impose file usage requirements to address
    customer solutions or file access requirements.
    Refer to the host server documentation for
    additional information.
    (this is the minimal attr proposal being requested)

---

From: David Robinson <[email-masked]>
Date: Wed, 05 Mar 2003 11:48:32 -0600

> just wondering if anyone is looking at this.  Thanks.

In following this thread I am having a hard time understanding
exactly what semantics are trying to be supported and thus
am having a hard time understanding where NFSv4 may be flawed.
Most likely my confusion is caused by you working in an
OS/Application space that I am unfamilar with.

First, it is important to note that the "lock" or "share" that
is acquired as part of an OPEN operation has no interaction
with any of the byte range "locks". Logically speaking it
is a mechanism to determine if a client can get the file handle
necessary to access the file. But in reality, because file handles
are not unique per client nor secret, the lockid/stateid is the
"token" that is passed back to the client for subsequent
use to communicate back to the server that a particular I/O
is related to the OPEN previously performed. It is expected
that a client requesting an OPEN with DENY_ALL has exclusive
access to that file. However in the presence of the magic
stateids (1's and 0's), this access control may be bypassed.
But, as part of the initial design it was expected that server's
may ignore these magic stateid's and deny access when an
appropriate OPEN had not occured.  I believe that if the
obvious mapping of an NFSv4 OPEN to a Windows OpenFile
occurs, this will be true. Based on how locking is
implemented in VMS, it is also likely that magic stateids
will be failed in many cases. (The RFC is a bit weak with
respect to the acceptable semantics of the magic stateids).
So, it is acceptable that a server implements semantics
were access to a file is not allowed except using a stateid
that is derived from a successful OPEN. This may cause some
client's to fail to access files that in similar situations
on other server's would be allowed.

In the case of byte range record locks, it is expected that
the server will enforce access control based on the locks
held and the stateid transmitted with the I/O request. As
before, we have the magic stateids to contend with, most
servers will allow them to be used to bypass the record
lock controls with a design center that you can only
corrupt yourself, not another client.  But as indicated before,
there is an expectation that some servers will fail access
to requests using the magic stateids. There are local file
systems semantics that cannot do otherwise.

But the interesting design point that I think you are hitting
is that the NFSv4 semantics are defined such that if no
record locks are held on a byte range, it is expected that
any client that has access to that file can perform I/O
on it. There is not a semantic that requires a lock to
be held before an I/O is to occur.

However, I believe that you can implement a server that forces
clients to acquire manditory locks before performing I/O
and do so in a compliant manner.  Because all clients must
be configured to deal with the case where an I/O fails
due to another client holding a conflicting lock, this
semantic can be leveraged. The client has no mechanism to
reliable determine if a lock is held for a particular range
(LOCKT not withstanding). So a server could be constructed
such that it would only allow I/O requests when the corresponding
byte range lock is held, and any other I/O request would be
failed. A participating client would simply wrap the I/O
inside a LOCK/LOCKU pair as needed. Non-participating clients
would find that virtually all of their I/O requests would
fail, but this is no different than if some other client
grabbed an exclusive lock on all bytes of all files. It is
true that the server returning NFS4ERR_LOCKED would be a lie,
but it is a lie that all clients are already required to
handle and cannot reliably detect that it is actually a lie.
On the client side, an implementation that always performs
I/O requests on only locked byte ranges will interoperate
with any compliant NFSv4 server, however it may not get as
strong a set of semantics as it desires.

So, if you built a client and server in this fashion, while
both may exibit "strange" behaviour they should be considered
standards compliant. From a market perspective I doubt many
Unix or Windows shops would buy such a server. For a client
that expects such strict behaviour, finding the "right"
server will probably be left as an administrative task.  Although
there is the possibility that the server generate a named
attribute that a client can query to see if this unique
behaviour is supported.

Overall, while I can see ways that we can enhance the protocol
to provide these sorts of semantics. I would not support
such a change as it would add a significant burden to those
servers that cannot implement them, the need appears tbe quite
small, and there looks like a reasonable workaround.

	-David