Subject: Re: [nfsv4] Newest (and maybe even final) proposed text changes re "client-persistent delegations"

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 9 Mar 2012 20:33:36 +0000

Have we reached consensus on this stuff?

I going to want to send a draft out by Monday that incorporates all of our
final changes for 3530bis.

On 2/22/12 2:47 PM, "[email-masked]" <[email-masked]> wrote:

>***(INTRO)**************************************************************
>************************************************************************
>
>This is version 4 of a set of changes to bis-16, to deal with issues
>related to client-persistent delegations (new name for clarity) and
>also to clarify stuff that I ran into making the original changes.
>
>Thanks to everybody who offered comments and corrections regarding
>previous versions.
>
>Changes from version 3 are very limited in scope.  See (DIFFS) for
>details
>
>In my view, we have reached Seems-like-Consensus and I hope this
>is the final version.  Anybody that disagrees should make his
>views known (with any required deltas specified) soon.
>
>
>***(TOC)****************************************************************
>************************************************************************
>
>   INTRO   -- Above
>   TOC     -- This list of stuff in here
>   DIFFS   -- Discussion of changes from version 3
>   FINAL?  -- current proposal for changes to deal with persistent
>              delegations plus all the stuff that I had to fix/correct/
>              clean up to address in connection with those
>   9.6.3   -- Discussion about correcting current (bis-16) organization
>              of 9.6.3.
>
>***(DIFFS)**************************************************************
>************************************************************************
>
>The paragraph beginning "A Server MAY" has been replaced.  The text is
>alternative (3M) from the discussion I had with Rick and Trond.  There
>is an additional paragraph (plus 4 bulleted items) to provide some
>clarification.
>
>That's all.
>
>***(FINAL?)************************************************************
>************************************************************************
>
>===> In section 9.1.3.4, in the third set of bulleted items, replace
>===> the fourth bulleted item therein by:
>
>  o  If the stateid represents revoked state or state lost as a result
>     of lease expiration, then return NFS4ERR_EXPIRED,
>     NFS4ERR_BAD_STATEID, or NFS4ERR_ADMIN_REVOKED, as appropriate.
>
>===> replace the last paragraph of section 9.1.10, by:
>
>   Requiring open confirmation on reclaim-type opens is avoidable
>   because of the nature of the environments in which such opens are
>   done.  For CLAIM_PREVIOUS opens, this is immediately after server
>   reboot, so there should be no time for open-owners to be created,
>   found to be unused, and recycled.  For CLAIM_DELEGATE_PREV opens, we
>   are dealing with either a client reboot situation or a network
>partition
>   resulting in deletion of lease state (and returning NFS4ERR_EXPIRED).
>   A server which supports delegation can be sure that no open-owners
>   for that client have been recycled since client initialization or
>   deletion of lease state and thus can ensure that confirmation will
>   not be required.
>
>===> replace the first paragraph of the second bulleted item in 9.5
>     with the following:
>
>    o Any operation made with a valid clientid (DELEGPURGE, RENEW,
>      OPEN, LOCK) or a valid stateid (CLOSE, DELEGRETURN, LOCK, LOCKU,
>      OPEN, OPEN_CONFIRM, OPEN_DOWNGRADE, READ, SETATTR, or WRITE).
>      In the latter case, the stateid must not be one of the
>      special stateids consisting of all bits 0 or all bits 1.
>
>===> replace the second sentence of the third paragraph of section
>     9.6.2 by:
>
>   During the grace period, clients recover locks and the associated
>   state by reclaim-type locking requests (i.e., LOCK requests with
>   reclaim set to true and OPEN operations with the claim types of
>   CLAIM_PREVIOUS and CLAIM_DELEGATE_PREV).
>
>===> replace the second sentence of section 9.6.3 by the following:
>
>   If this occurs, the server may cancel the lease and free all
>   locks held for the client.
>
>===> Replace the first two paragraphs of section 9.6.3.1 by the
>===> following:
>
>   As a courtesy to the client or as an optimization, the server may
>   continue to hold locks, including delegations, on behalf of a client
>   for which recent communication has extended beyond the lease period,
>   delaying the cancellation of the lease.  If the server receives a lock
>   or I/O request that conflicts with one of these courtesy locks or if it
>   runs out of resources, the server MAY cause lease cancellation to
>occur at
>   that time and henceforth return NFS4ERR_EXPIRED when any of the
>stateids
>   associated with the freed locks is used. If lease cancellation has not
>   occurred and the server receives a lock or I/O request that conflicts
>with
>   one of the courtesy locks, the requirements are as follows:
>
>   o in the case of a courtesy lock which is not a delegation, it MUST
>free
>     the courtesy lock and grant the new request.
>
>   o In the case of lock or IO request which conflicts with a delegation
>     which is being held as courtesy lock, the server MAY delay resolution
>     of request but MUST NOT reject the request and MUST free the
>delegation
>     and grant the new request eventually.
>
>   o In the case of a requests for a delegation which conflicts with a
>     delegation which is being held as courtesy lock, the server MAY
>     grant the new request or not as it chooses, but if it grants the
>     conflicting request, the delegation haled as courtesy lock MUST be
>     freed.
>
>   If the server does not reboot or cancel the lease before the network
>   partition is healed, when the original client tries to access a
>courtesy
>   lock which was freed, the server SHOULD send back a NFS4ERR_BAD_STATEID
>   to the client.  If the client tries to access a courtesy lock which
>   was not freed, then the server SHOULD mark all of the courtesy locks as
>   implicitly being renewed.
>
>===> Following that material, and appearing after what is now the second
>===> paragraph of 9.6.3.1, add the following new section,
>====> (ORG9.6.3)
>
>9.6.3.2 Lease cancellation
>
>   As a result of lease expiration, leases may be cancelled, either
>   immediately upon expiration or subsequently, depending on the
>occurrence
>   of a conflicting lock or extension of the period of partition beyond
>   what the server will tolerate.
>
>   When a lease is cancelled, all locking state associated with it is
>   freed and use of any the associated stateids will result in
>   NFS4ERR_EXPIRED being returned.  Similarly, use of the associated
>clientid
>   will result in NFS4ERR_EXPIRED being returned.
>
>   The client should recover from this situation by using SETCLIENTID
>   followed by SETCLIENTID_CONFIRM, in order to establish a new clientid.
>   Once a lock is obtained using this clientid, a lease will be
>established.
>
>===> Update the following items in section 9.6.3.2 as follows and move the
>===> section to follow the new 9.6.3.2, making this section 9.6.3.3
>
>   NFS4ERR_EXPIRED:
>
>      All locks have been freed as a result of a lease cancellation
>      which occurred during the partition.  The client should use a
>      SETCLIENTID to recover.
>
>   NFS4ERR_ADMIN_REVOKED:
>
>      The current lock has been revoked before, during, or after
>      the partition.  The client SHOULD handle this error
>      as it normally would.
>
>   NFS4ERR_BAD_STATEID:
>
>      The current lock has been revoked/released during the partition and
>the
>      server did not reboot.  Other locks MAY still be renewed.  The
>      client need not do a SETCLIENTID and instead SHOULD probe
>      via a RENEW call.
>
>====> Following the transported 9.6.3.2 (now 9.6.3.3), start the following
>====> new section, adjusting section numbers as indicated below in
>====> (ORG 9.6.3)
>
>9.6.3.4 Edge conditions
>
>===> After what had been the fourth non-bulleted paragraph of section
>9.6.3.1
>===> and would now be the second non-bulleted paragraph of 9.6.3.4,
>===> add the following paragraph.
>
>    The above are directed to CLAIM_PREVIOUS reclaims and not to
>    CLAIM_DELEGATE_PREV reclaims, which generally do not involve
>    a server reboot.  However, when a server persistently stores
>    delegation information to support CLAIM_DELEGATE_PREV across
>    a period in which both client and server are down at the same
>    time, similar strictures apply.
>
>===> Replace section 10.2.1 by the following
>
> There are three situations that delegation recovery must deal with:
>
>   o  Client reboot or restart
>
>   o  Server reboot or restart
>
>   o  Network partition (full or callback-only)
>
>   In the event the client reboots or restarts, the confirmation of
>   a SETCLIENTID done with an nfs_client_id4 with a new verifier4 value
>   will result in the release of byte-range locks and share reservations.
>   Delegations, however, may be treated a bit differently.
>
>   There will be situations in which delegations will need to be
>   reestablished after a client reboots or restarts.  The reason for
>   this is the client may have file data stored locally and this data
>   was associated with the previously held delegations.  The client will
>   need to reestablish the appropriate file state on the server.
>
>   To allow for this type of client recovery, the server MAY allow
>   delegations to be retained after other sort of locks are released.
>   This implies that requests from other clients that conflict with these
>   delegations will need to wait.  Because the normal recall process may
>   require significant time for the client to flush changed state to the
>   server, other clients need to be prepared for delays that occur because
>   of a conflicting delegation.  In order to give clients a chance to get
>   through the reboot process during which leases will not be renewed,
>   the server MAY extend the period for delegation recovery beyond the
>   typical lease expiration period.  For open delegations, such
>delegations
>   that are not released are reclaimed using OPEN with a claim type of
>   CLAIM_DELEGATE_PREV.  (See Section 10.5 and Section 15.18 for
>discussion
>   of open delegation and the details of OPEN respectively).
>
>   A server MAY support a claim type of CLAIM_DELEGATE_PREV, but if it
>   does, it MUST NOT remove delegations upon SETCLIENTID_CONFIRM and
>   instead MUST make them available for client reclaim using
>   CLAIM_DELEGATE_PREV.  The server MAY NOT remove the delegations until
>   either the client does a DELEGPURGE, or one lease period has
>   elapsed from the time the later of the SETCLIENTID_CONFIRM or the
>   last successful CLAIM_DELEGATE_PREV reclaim.
>
>   Note that the requirement stated above is not meant to imply that
>   when the client is no longer obliged, as required above, to retain
>   delegation information, that it should necessarily dispose of it.
>   Let's look at some specific cases:
>
>   o when the period is terminated by the occurrence of DELEGPURGE,
>     deletion of unreclaimed delegations is appropriate and desirable.
>
>   o when the period is terminated by a lease period elapsing without
>     a successful CLAIM_DELEGATE_PREV reclaim, and that situation
>     appears to be the result of a network partition (i.e. lease
>     expiration has occurred), a server's lease expiration approach,
>     possibly including the use of courtesy locks would normally
>     provide for the retention of unreclaimed delegations.  Even in
>     the event that lease cancellation occurs, such delegation should
>     be reclaimed using CLAIM_DELEGATE_PREV as part of network partition
>     recovery.
>
>   o When the period of non-communicating is followed by a client reboot,
>     unreclaimed delegations, should also be reclaimable by use of
>     CLAIM_DELEGATE_PREV as part of client reboot recovery.
>
>   o When the period is terminated by a lease period elapsing without
>     a successful CLAIM_DELEGATE_PREV reclaim, and lease renewal
>     is occurring, the server may well conclude that unreclaimed
>     delegations have been abandoned, and consider the situation as
>     one in which an implied DELEGPURGE should be assumed.
>
>   A server that supports a claim type of CLAIM_DELEGATE_PREV MUST support
>   the DELEGPURGE operation, and similarly a server that supports
>   DELEGPURGE MUST support CLAIM_DELEGATE_PREV.  A server which does
>   not support CLAIM_DELEGATE_PREV MUST return NFS4ERR_NOTSUPP if the
>   client attempts to use that feature or performs a DELEGPURGE operation.
>
>   Support for a claim type of CLAIM_DELEGATE_PREV, is often referred to
>   as providing for "client-persistent delegations" in that they allow
>   use of client persistent storage on the client to store data written
>   by the client, even across a client restart.  It should be noted that,
>   with the optional exception noted below, this feature requires
>persistent
>   storage to be used on the client and does not add to persistent storage
>   requirements on the server.
>
>   One good way to think about client-persistent delegations is that for
>   the most part, they function like "courtesy locks", with a special
>   semantic adjustments to allow them to be retained across a client
>restart,
>   which cause all other sorts of locks to be freed.  Such locks are
>generally
>   not retained across a server restart.  The one exception is the
>   case of simultaneous failure of the client and server and is discussed
>   below.
>
>   When the server indicates support of CLAIM_DELEGATE_PREV (implicitly)
>   by returning NFS_OK to DELEGPURGE, a client with a write delegation,
>   can use write-back caching for data to be written to the server,
>deferring
>   the write-back, until such time as the delegation is recalled, possibly
>   after intervening client restarts.  Similarly, when the server
>indicates
>   support of CLAIM_DELEGATE_PREV, a client with a read delegation and
>   an open-for-write subordinate to that delegation, may be sure of the
>   integrity of its persistently cached copy of the file after a client
>   restart without specific verification of the change attribute.
>
>   When the server reboots or restarts, delegations are reclaimed (using
>   the OPEN operation with CLAIM_PREVIOUS) in a similar fashion to byte-
>   range locks and share reservations.  However, there is a slight
>   semantic difference.  In the normal case, if the server decides that a
>   delegation should not be granted, it performs the requested action
>   (e.g., OPEN) without granting any delegation.  For reclaim, the
>   server grants the delegation but a special designation is applied so
>   that the client treats the delegation as having been granted but
>   recalled by the server.  Because of this, the client has the duty to
>   write all modified state to the server and then return the
>   delegation.  This process of handling delegation reclaim reconciles
>   three principles of the NFSv4 protocol:
>
>   o  Upon reclaim, a client reporting resources assigned to it by an
>      earlier server instance must be granted those resources.
>
>   o  The server has unquestionable authority to determine whether
>      delegations are to be granted and, once granted, whether they are
>      to be continued.
>
>   o  The use of callbacks is not to be depended upon until the client
>      has proven its ability to receive them.
>
>   When a client has more than a single open associated with a
>   delegation, state for those additional opens can be established using
>   OPEN operations of type CLAIM_DELEGATE_CUR.  When these are used to
>   establish opens associated with reclaimed delegations, the server
>   MUST allow them when made within the grace period.
>
>   Situations in which there us a series of client and server restarts
>where
>   there is no restart of both at the same time, are dealt with via a
>   combination of CLAIM_DELEGATE_PREV and CLAIM_PREVIOUS reclaim cycles.
>   Persistent storage is needed only on the client.  For each server
>   failure, a CLAIM_PREVIOUS reclaim cycle is done, while for each client
>   restart, a CLAIM_DELEGATE_PREV reclaim cycle is done.
>
>   To deal with the possibility of simultaneous failure of client and
>   server (e.g. situations like power outages, etc.), the server MAY
>   persistently store delegation information so that it can respond to
>   a CLAIM_DELEGATE_PREV reclaim request which it receives from a
>   restarting client.   This is the one case in which persistent
>   delegation state can be retained across a server restart.  A server
>   is not required to store this information, but if it does do so, it
>   should do so for write delegations and for read delegations, during
>   the pendency of which (across multiple client and/or server
>   instances), some open-for-write was done as part of delegation.  When
>   the space to persistently record such information is limited, the
>   server should recall delegations in this class in preference to
>   keeping them active without persistent storage recording.
>
>   When a network partition occurs, delegations are subject to freeing
>   by the server when the lease renewal period expires.  This is similar
>   to the behavior for locks and share reservations, and, as for locks
>   and share reservations it may be modified by support for "courtesy
>   locks" in which locks are not freed in the absence of a conflicting
>   lock request.  Whereas, for locks and share reservations, freeing of
>   locks will occur immediately upon the appearance of a conflicting
>   request, for delegations, the server may institute period during which
>   conflicting requests are held off.  Eventually the occurrence of a
>   conflicting request from another client will cause revocation of the
>   delegation.
>
>   A loss of the callback path (e.g., by later network configuration
>   change) will have a similar effect in that it can also result in
>   revocation of a delegation  A recall request will fail and revocation
>   of the delegation will result.
>
>   A client normally finds out about revocation of a delegation when it
>   uses a stateid associated with a delegation and receives one of the
>   errors NFS4ERR_EXPIRED, NFS4ERR_BAD_STATEID, or NFS4ERR_ADMIN_REVOKED
>   (NFS4ERR_EXPIRED indicates that all lock state associated with the
>   client has been lost).  It also may find out about delegation
>revocation
>   after a client reboot when it attempts to reclaim a delegation and
>   receives NFS4ERR_EXPIRED.  Note that in the case of a revoked
>   OPEN_DELEGATE_WRITE delegation, there are issues because data may
>   have been modified by the client whose delegation is revoked and
>   separately by other clients.  See Section 10.5.1 for a discussion of
>   such issues.  Note also that when delegations are revoked,
>   information about the revoked delegation will be written by the
>   server to stable storage (as described in Section 9.6).  This is done
>   to deal with the case in which a server reboots after revoking a
>   delegation but before the client holding the revoked delegation is
>   notified about the revocation.
>   Note that when there is a loss of a delegation, due to a network
>   partition in which all locks associated with the lease are lost,
>   the client will also receive the error NFS4ERR_EXPIRED.  This case
>   can be distinguished from other situations in which delegations
>   are revoked by seeing that the associated clientid becomes invalid
>   so that NFS4ERR_STALE_CLIENTID is returned when it is used.
>
>   When NFS4ERR_EXPIRED Is returned, the server MAY retain information
>   about the delegations held by the client, deleting those that are
>   invalidated by a conflicting request.  Retaining such information
>   will allow the client to recover all non-invalidated delegations
>   using the claim type CLAIM_DELEGATE_PREV, once the
>   SETCLIENTID_CONFIRM is done to recover.  Attempted recovery of
>   a delegation that the client has no record of, typically because they
>   were invalidated by conflicting requests, will get the error
>   NFS4ERR_BAD_RECLAIM.  Once a reclaim is attempted for all delegations
>   that the client held, it SHOULD do a DELEGPURGE to allow any remaining
>   server delegation information to be freed.
>
>===> replace section 13.1.5.1 (NFS4ERR_ADMIN_REVOKED) by:
>
>   A stateid designates locking state of any type that has been revoked
>   due to administrative interaction, possibly while the lease is valid,
>   or because a delegation was revoked because of failure to return it,
>   while the lease was valid.
>
>===> replace section 13.1.5.2 (NFS4ERR_BAD_STATEID) by:
>
>   A stateid generated by the current server instance was used which
>   either:
>
>      o does not designate any locking state (either current or
>superseded)
>        for a current (state-owner, file) pair.
>
>      o designates locking state that was freed after lease expiration but
>        without any lease cancelation, as may happen in the handling of
>        "courtesy locks".
>
>===> replace section 13.1.5.3 (NFS4ERR_EXPIRED) by:
>
>   A stateid or clientid designates locking state of any type that
>   has been revoked or released due to cancellation of the client's
>   lease, either immediately upon lease expiration, or following a later
>   request for a conflicting lock.
>
>===> Note the paragraph with the hanging "CLAIM_DELEGATE_PREV:" in
>===> section 15.18.5.
>
>===> In that paragraph, replace "; used after the client reboots:" with
>===> the following:
>
>   This claim type is for use after a SETCLIENTID_CONFIRM and before
>   the corresponding DELEGPURGE in two situations: after a client reboot
>   and after a lease expiration that resulted in loss of all lock
>   state.
>
>===> after that paragraph, add the following material with indenting that
>     matches the non-hanging portion of the noted paragraph.
>
>   The following errors apply to use of the CLAIM_DELEGATE_PREV claim
>   type:
>
>   o NFS4ERR_NOTSUPP is returned if the server does not support this
>     claim type.
>
>   o NFS4ERR_INVAL is returned if the reclaim is done at an inappropriate
>     time, e.g. after DELEGPURGE has been done.
>
>   o NFS4ERR_BAD_RECLAIM is returned if the other error conditions do not
>     apply and the server has no record of the delegation whose reclaim
>     is being attempted.
>
>===> Replace the last two paragraphs of 15.7.4 by the following:
>
>   This operation in provided to support clients that record delegation
>   information on stable storage on the client.  In this case,
>   DELEGPURGE should be issued immediately after doing delegation
>   recovery (using CLAIM_DELEGATE_PREV) on all delegations known to
>   the client.  Doing so will notify the server that no additional
>   delegations for the client will be recovered allowing it to free
>   resources, and avoid delaying other clients who make requests that
>   conflict with the unrecovered delegations.  All client SHOULD use
>   DELEGPURGE as part of recovery once it is known that no further
>   CLAIM_DELEGATE_PREV recovery will be done.  This includes clients
>   that do not record delegation information on stable storage, who would
>   then do a DELEGPURGE immediately after SETCLIENTID_CONFIRM.
>
>   The set of delegations known to the server and the
>   client may be different.  The reasons for this include:
>
>   o a client may fail after making a request which resulted in delegation
>     but before it received the results and committed them to the client's
>     stable storage.
>   o a client may fail after deleting its indication that a delegation
>exists
>     but before the delegation return is fully processed by the server.
>   o in the case in which the server and the client restart, the server
>     may have limited persistent recording of delegation to a subset of
>     those in existence.
>   o a client may have only persistently recorded information about a
>subset of
>     delegations.
>
>   The server MAY support DELEGPURGE, but its support or non-support
>should
>   match that of CLAIM_DELEGATE_PREV:
>   o A server may support both DELEGPURGE and CLAIM_DELEGATE_PREV
>   o A server may support neither DELEGPURGE nor CLAIM_DELEGATE_PREV
>
>   This fact allows a client starting up to determine if the server is
>prepared
>   to support persistent storage of delegation information and thus
>whether it
>   may use write-back caching to local persistent storage, relying on
>   CLAIM_DELEGATE_PREV recovery to allow such changed data to be flushed
>   safely to the server in the event of client restart.
>
>***(ORG9.6.3)***********************************************************
>************************************************************************
>
>===> The section organization of 9.6.3 is messed up.  See below.
>===> The big issue is that all of the edge condition stuff is part of
>===> 'courtesy locks' when  it shouldn't be
>
>===> Here's what exists now.
>
>9.6.3.  Network Partitions and Recovery
>9.6.3.1.  Courtesy Locks
>9.6.3.1.1.  First Server Edge Condition
>9.6.3.1.2.  Second Server Edge Condition
>9.6.3.1.3.  Handling Server Edge Conditions
>9.6.3.1.4.  Client Edge Condition
>9.6.3.1.5.  Client's Handling of NFS4ERR_NO_GRACE
>9.6.3.2.  Client's Reaction to a Freed Lock
>
>
>===> Here's a suggested replacement;
>
>9.6.3.  Network Partitions and Recovery
>9.6.3.1.  Courtesy Locks
>9.6.3.2   Lease Cancellation (NEW)
>9.6.3.3.  Client's Reaction to a Freed Lock
>9.6.3.4.  Edge Conditions (NEW)
>9.6.3.4.1.  First Server Edge Condition
>9.6.3.4.2.  Second Server Edge Condition
>9.6.3.4.3.  Handling Server Edge Conditions
>9.6.3.4.4.  Client Edge Condition
>9.6.3.5.  Client's Handling of NFS4ERR_NO_GRACE
>
>
>
>
>
>
>
>_______________________________________________
>nfsv4 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Fri, 9 Mar 2012 16:30:25 -0500

I thought so when I wrote it, and I've had nobody indicate any issue with it.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 9 Mar 2012 21:37:37 +0000

I didn't hear any humming. :->

I'll add it this weekend and get a new draft out.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 9 Mar 2012 22:21:02 +0000

> I thought so when I wrote it, and I've had nobody indicate any issue with it.

Assuming we do have consensus, how do we make this apply to NFSv4.1 too?
Is that going to require another errata?

It would be nice if we could add a "blanket errata" to the effect that
all text in RFC3530bis applies to RFC5661 too unless explicitly
overridden. Is that allowable?

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com