Subject: [nfsv4] RECLAIM_BAD vs RECLAIM_CONFLICT vs NO_GRACE

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 2 Mar 2012 16:05:03 -0500

The RECLAIM_BAD error was never well explained in rfc 3530, and several
serious bits of further confusion have crept into 3530bis.

RECLAIM_BAD was originally intended to cover the case where the
information that the server keeps in stable storage shows that the
client should not be able to perform a reclaim of the given lock.

Unfortunately NO_GRACE is recommended for that purpose in a few spots,
but that should be reserved for the case where a client has actually
missed the grace period entirely (or when the server keeps no such
stable-storage information and does not allow reclaims at all).

This is an interoperability bug in the case of a server that keeps
finer-grained locking information in its stable storage, in which case
RECLAIM_BAD indicates that the particular reclaim is not permitted,
while NO_GRACE indicates that none will.  (And a quick glance at the
Linux client code suggests that at least one implementation uses that
distinction: it looks to me like it does give up on all further reclaims
in the NO_GRACE case.)

Here's a proposed fix for 3530bis.

Some of the same errors are in rfc 5661.

--b.

diff --git a/nfsv4_middle_errors.xml b/nfsv4_middle_errors.xml
index c680a6d..547a14b 100644
--- a/nfsv4_middle_errors.xml
+++ b/nfsv4_middle_errors.xml
@@ -885,29 +885,29 @@
     <section title="NFS4ERR_NO_GRACE (Error Code 10033)" 
              anchor="err_NO_GRACE">
       <t>
-        A reclaim of client state was attempted in circumstances in 
-        which the server cannot guarantee that conflicting state has 
-        not been provided to another client.  As a result, the server cannot
-        guarantee that conflicting state has not been provided to
-        another client.
+	A reclaim of client state has fallen outside of the grace period
+	of the server.  As a result, the server can not guarantee that
+	conflicting state has not been provided to another client.
       </t>
     </section>
     <section title="NFS4ERR_RECLAIM_BAD (Error Code 10034)" 
              anchor="err_RECLAIM_BAD">
       <t>
-        A reclaim attempted by the client does not match the
-        server's state consistency checks and has been rejected 
-        therefore as invalid.
+        The client attempted to reclaim state that the server knows it
+	no longer holds.  Unlike NFS4ERR_RECLAIM_CONFLICT, this can
+	occur between correctly functioning clients and servers: the
+	"edge condition" scenarios described in <xref target="ss:fl:cl" />
+	leave only the server knowing whether the client's locks are
+	still valid, and NFS4ERR_RECLAIM_BAD is the server's way of
+	informing the client that they are not.
       </t>
     </section>
     <section title="NFS4ERR_RECLAIM_CONFLICT (Error Code 10035)" 
              anchor="err_RECLAIM_CONFLICT">
       <t>
-        The reclaim attempted by the client has encountered a conflict
-        and cannot be satisfied.  Potentially indicates a misbehaving
-        client, although not necessarily the one receiving the error.
-        The misbehavior might be on the part of the client that 
-        established the lock with which this client conflicted. 
+        The reclaim attempted by the client conflicts with a lock
+	already held by another client.  Unlike NFS4ERR_RECLAIM_BAD,
+	this can only occur if one of the clients misbehaved.
       </t>
     </section>
   </section>
diff --git a/nfsv4_middle_file_locking.xml b/nfsv4_middle_file_locking.xml
index 08cdab2..fd479e1 100644
--- a/nfsv4_middle_file_locking.xml
+++ b/nfsv4_middle_file_locking.xml
@@ -1735,7 +1735,7 @@
             after the server reboots, the record that client A's lease expired
             means that another client could have acquired a conflicting record
             lock, share reservation, or delegation.  Hence the server must
-            reject a reclaim from client A with the error NFS4ERR_NO_GRACE.
+            reject a reclaim from client A with the error NFS4ERR_RECLAIM_BAD.
           </t>
 
           <t>
@@ -1743,7 +1743,7 @@
             second time, the record that the client had an unexpired record
             lock, share reservation, or delegation established before the
             server's previous incarnation means that the server must reject
-            a reclaim from client A with the error NFS4ERR_NO_GRACE.
+            a reclaim from client A with the error NFS4ERR_RECLAIM_BAD.
           </t>
 
           <t>
@@ -1879,17 +1879,18 @@
           </t>
         </section>
 
-        <section title="Client's Handling of NFS4ERR_NO_GRACE">
+        <section title="Client's Handling of reclaim errors">
 
           <t>
-            A mandate for the client's handling of the NFS4ERR_NO_GRACE error
-            is outside the scope of this specification, since the strategies
-            for such handling are very dependent on the client's operating
-            environment.  However, one potential approach is described below.
+	    A mandate for the client's handling of the NFS4ERR_NO_GRACE
+	    and NFS4ERR_RECLAIM_BAD errors is outside the scope of this
+	    specification, since the strategies for such handling are
+	    very dependent on the client's operating environment.
+	    However, one potential approach is described below.
           </t>
 
           <t>
-            When the client receives NFS4ERR_NO_GRACE, it could examine
+            When the client's reclaim fails, it could examine
             the change attribute of the objects the client is trying
             to reclaim state for, and use that to determine whether
             to re-establish the state via normal OPEN or LOCK requests.
@@ -1933,7 +1934,7 @@
                 The client MAY NOT want to do a SETCLIENTID and instead
                 SHOULD probe via a RENEW call.
               </t>
-              <t hangText="NFS4ERR_NO_GRACE:"> <vspace blankLines='1' />
+              <t hangText="NFS4ERR_RECLAIM_BAD:"> <vspace blankLines='1' />
                 The current lock has been revoked during the partition and
                 the server rebooted. The server might have no information
                 on the other locks. They may still be renewable.

---

From: <[email-masked]>
Date: Fri, 2 Mar 2012 20:03:00 -0500

> stable-storage information and does not allow reclaims at all).

I disagree with you about edge condition handling.

I agree that the paradigm should be:

    NO_GRACE; this reclaim can't be granted and neither will any others
              you try.  Deal with it.
    BAD_RECLAIM:  this reclaim can't be granted but others you try 
                  might well be.
              
I think the current text needs some work and I don't think that always returning
BAD_RECLAIM on edge conditions will help things.  In many cases a coarse-grained
Edge-condition detector would validly decide that no reclaims would be granted 
for that particular client and in that case, NO_GRACE would be justified, even though
other clients might be allowed to reclaim locks.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Friday, March 02, 2012 4:05 PM
To: Thomas Haynes
Cc: [email-masked]; Noveck, David
Subject: RECLAIM_BAD vs RECLAIM_CONFLICT vs NO_GRACE

--b.

           <t>
@@ -1743,7 +1743,7 @@
 
           <t>
@@ -1879,17 +1879,18 @@
           </t>
         </section>

           <t>
 
           <t>

---

From: "J. Bruce Fields" <[email-masked]>
Date: Sat, 3 Mar 2012 10:06:59 -0500

> other clients might be allowed to reclaim locks.

OK, that's sensible.  I'll try for another revision by Monday.--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 5 Mar 2012 10:16:55 -0500

> OK, that's sensible.  I'll try for another revision by Monday.--b.

Here's take two.

--b.

diff --git a/nfsv4_middle_errors.xml b/nfsv4_middle_errors.xml
index c680a6d..d1556b2 100644
--- a/nfsv4_middle_errors.xml
+++ b/nfsv4_middle_errors.xml
@@ -885,29 +885,34 @@
     <section title="NFS4ERR_NO_GRACE (Error Code 10033)" 
              anchor="err_NO_GRACE">
       <t>
-        A reclaim of client state was attempted in circumstances in 
-        which the server cannot guarantee that conflicting state has 
-        not been provided to another client.  As a result, the server cannot
-        guarantee that conflicting state has not been provided to
-        another client.
+        The server cannot guarantee that it has not granted state to
+        another client which may conflict with this client's state.
+        No further reclaims from this client will succeed.
       </t>
     </section>
     <section title="NFS4ERR_RECLAIM_BAD (Error Code 10034)" 
              anchor="err_RECLAIM_BAD">
       <t>
-        A reclaim attempted by the client does not match the
-        server's state consistency checks and has been rejected 
-        therefore as invalid.
+        The server cannot guarantee that it has not granted state to
+        another client which may conflict with the requested state.
+        However, this applies only to the state requested in this call;
+        further reclaims may succeed.
+      </t>
+      <t>
+        Unlike NFS4ERR_RECLAIM_CONFLICT, this can occur between
+        correctly functioning clients and servers: the "edge condition"
+        scenarios described in <xref target="ss:fl:cl" /> leave only the
+        server knowing whether the client's locks are still valid, and
+        NFS4ERR_RECLAIM_BAD is the server's way of informing the client
+        that they are not.
       </t>
     </section>
     <section title="NFS4ERR_RECLAIM_CONFLICT (Error Code 10035)" 
              anchor="err_RECLAIM_CONFLICT">
       <t>
-        The reclaim attempted by the client has encountered a conflict
-        and cannot be satisfied.  Potentially indicates a misbehaving
-        client, although not necessarily the one receiving the error.
-        The misbehavior might be on the part of the client that 
-        established the lock with which this client conflicted. 
+        The reclaim attempted by the client conflicts with a lock
+        already held by another client.  Unlike NFS4ERR_RECLAIM_BAD,
+        this can only occur if one of the clients misbehaved.
       </t>
     </section>
   </section>
diff --git a/nfsv4_middle_file_locking.xml b/nfsv4_middle_file_locking.xml
index 08cdab2..f9bf036 100644
--- a/nfsv4_middle_file_locking.xml
+++ b/nfsv4_middle_file_locking.xml
@@ -1735,7 +1735,8 @@
             after the server reboots, the record that client A's lease expired
             means that another client could have acquired a conflicting record
             lock, share reservation, or delegation.  Hence the server must
-            reject a reclaim from client A with the error NFS4ERR_NO_GRACE.
+            reject a reclaim from client A with the error NFS4ERR_NO_GRACE or
+            NFS4ERR_RECLAIM_BAD.
           </t>
 
           <t>
@@ -1743,7 +1744,8 @@
+            a reclaim from client A with the error NFS4ERR_NO_GRACE or
+            NFS4ERR_RECLAIM_BAD.
           </t>
 
           <t>
@@ -1879,17 +1881,18 @@
           </t>
         </section>

           <t>
 
           <t>
-            When the client receives NFS4ERR_NO_GRACE, it could examine
+            When the client's reclaim fails, it could examine
             the change attribute of the objects the client is trying
             to reclaim state for, and use that to determine whether
             to re-establish the state via normal OPEN or LOCK requests.
@@ -1933,11 +1936,16 @@
                 The client MAY NOT want to do a SETCLIENTID and instead
                 SHOULD probe via a RENEW call.
               </t>
-              <t hangText="NFS4ERR_NO_GRACE:"> <vspace blankLines='1' />
+              <t hangText="NFS4ERR_RECLAIM_BAD:"> <vspace blankLines='1' />
                 The current lock has been revoked during the partition and
                 the server rebooted. The server might have no information
                 on the other locks. They may still be renewable.
               </t>
+              <t hangText="NFS4ERR_NO_GRACE:"> <vspace blankLines='1' />
+                The client's locks have been revoked during the partition and
+                the server rebooted. None of the client's locks will be
+                renewable.
+              </t>
               <t hangText="NFS4ERR_OLD_STATEID:"> <vspace blankLines='1' />
                 The server has not rebooted. The client SHOULD handle
                 this error as it normally would.

---

From: <[email-masked]>
Date: Mon, 5 Mar 2012 12:36:13 -0500

Looks fine to me.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Monday, March 05, 2012 10:17 AM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: Re: RECLAIM_BAD vs RECLAIM_CONFLICT vs NO_GRACE

Here's take two.

--b.

           <t>
@@ -1743,7 +1744,8 @@
 
           <t>
@@ -1879,17 +1881,18 @@
           </t>
         </section>

           <t>
 
           <t>

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 6 Mar 2012 07:11:53 +0000

Everyone please review.  If no one complains by the end of the week, please
file this as an errata, Bruce.

Spencer

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Monday, March 05, 2012 9:36 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [nfsv4] RECLAIM_BAD vs RECLAIM_CONFLICT vs NO_GRACE
> 
> Looks fine to me.
> 
> > > > Unfortunately NO_GRACE is recommended for that purpose in a few
> > > > spots, but that should be reserved for the case where a client has
> > > > actually missed the grace period entirely (or when the server
> > > > keeps no such stable-storage information and does not allow reclaims
> at all).
> > >
> > > I think the current text needs some work and I don't think that
> > > always returning BAD_RECLAIM on edge conditions will help things.
> > > In many cases a coarse-grained Edge-condition detector would validly
> > > decide that no reclaims would be granted for that particular client
> > > and in that case, NO_GRACE would be justified, even though other
> clients might be allowed to reclaim locks.
> >
> diff --git a/nfsv4_middle_errors.xml b/nfsv4_middle_errors.xml index
> c680a6d..d1556b2 100644
> +              <t hangText="NFS4ERR_RECLAIM_BAD:"> <vspace
> + blankLines='1' />
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4