Subject: [nfsv4] RECLAIM_COMPLETE mandatory for 4.1 client?

From: Benny Halevy <[email-masked]>
Date: Tue, 21 Feb 2012 07:59:01 -0800

We've hit an issue with the linux 4.1 sever post linux 3.0 where the
server will keep returning
NFS4ERR_GRACE forever on OPEN until the client sends RECLAIM_COMPLETE
based essentially on section 18.51.3:

   Whenever a client establishes a new client ID and before it does the
   first non-reclaim operation that obtains a lock, it MUST send a
   RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
   locks to reclaim.  If non-reclaim locking operations are done before
   the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned.

However, several client implementers never send RECLAIM_COMPLETE,
expecting the server
to return to normal operation mode after the grace period is over, a-la 4.0.

To support this notion (and frankly, this was my interpretation of the
spec until yesterday)
it seems that there are vague parts of the spec. that seem to indicate
that any state reclaim
issues are only strictly applicable to the grace period itself.

8.4.2.1.  State Reclaim
   The grace period may last until all clients that are known to
   possibly have had locks have done a global RECLAIM_COMPLETE
   operation, indicating that they have finished reclaiming the locks
   they held before the server restart.

   If the server can reliably determine that granting a non-reclaim
   request will not conflict with reclamation of locks by other clients,
   the NFS4ERR_GRACE error does not have to be returned even within the
   grace period, although NFS4ERR_GRACE must always be returned to
   clients attempting a non-reclaim lock request before doing their own
   global RECLAIM_COMPLETE.

so, after the grace period is done, the server can reliably determine that
granting a non-reclaim request will not conflict with reclamation of
locks by other clients
since no other client can now reclaim its locks. Therefore I see no reason for
the client to send RECLAIM_COMPLETE at this point, right?

On a related note, this condition refers to any non-reclaiming
"locking operations"
and looking at the spec it might be confusing to new readers of the
spec to determine
that OPEN falls in this category of locking operations, so this might
be worth clarifying
in the errata + 5661bis.

Benny

---

From: Rick Macklem <[email-masked]>
Date: Tue, 21 Feb 2012 18:54:53 -0500 (EST)

> the client to send RECLAIM_COMPLETE at this point, right?
Well, I believe the reason for RECLAIM_COMPLETE might be related to the
observation I had made during testing of NFSv4.0, where it was possible
for a lot of clients holding a lot of locks to not get them all reclaimed
before the "grace period of at least one lease duration" had elapsed.

RECLAIM_COMPLETE tells the server that it has done reclaims. Once the
server sees that from all clients, it can end grace safely instead of
using some arbitrary time limit.

And I hate to say it, but whether or not it's necessary, the words seem
clear on this, so "the horse has already left the barn" so to speak, imho.

Btw, I fixed the FreeBSD client for this recently (and the v4.1 client
hasn't been released yet). If the one Tom is testing this week suffers
from this, just email and I can pass a patched kernel along to him).

rick

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 21 Feb 2012 19:37:47 -0500

> using some arbitrary time limit.

There's that, and there's the obvious "end-the-grace-period-early"
optimisation, and then it's also necessary to avoid some edge
conditions.  (In 4.1 RECLAIM_COMPLETE tells the server that from now on
it can track the set of locks the client will reclaim on *next* reboot,
and is what a typical server would use as the point to record a
stable-storage record for that client.  A longer explanation is lost in
moderation somewhere.)

> clear on this, so "the horse has already left the barn" so to speak, imho.

(Not sure which way you're saying it's left.)

--b.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 22 Feb 2012 00:06:03 -0500 (EST)

> (Not sure which way you're saying it's left.)
I should have just said that the "RFC seems clear on this" as in a
RECLAIM_COMPLETE must be done before non-reclaim ops.

rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 21 Feb 2012 10:20:30 -0800

> to return to normal operation mode after the grace period is over, a-la 4.0.

That's an extremely clear violation of the above MUST.

Also, note that the RECLAIM_COMPLETE isn't just an optimization to end
the grace period early, it's also what the server is supposed to use to
determine when a client can be trusted to reclaim the right locks after
the next reboot:

	"When the RECLAIM_COMPLETE is received, the server and client
	are again in agreement regarding reclaimable locks and both
	booleans in persistent storage can be reset, to be set again
	only when there is a subsequent event that causes lock reclaim
	operations to be questionable."

So there's another interoperability problem here, that wouldn't be
noticed till after a reboot: a server following the example
reboot-recovery implementation here would therefore deny reclaims to any
client that doesn't send RECLAIM_COMPLETE.

Making this requirement fuzzier would leave us open to some odd edge
conditions.

E.g. suppose a server that instead considers the client responsible for
its locks on first locking operation (as in 4.0), and suppose a client
that follows the letter of the 4.1 spec:

Then in the following scenario:

	- client A acquires lock x and y
	- server reboots
	- client A reclaims lock x
	- network connection between A and server goes down
	- grace period ends
	- client B acquires lock conflicting with y
	- server goes down before client A's lease expires
	- network connection between A and server comes up

At this point:
	- The server has recorded that client A is allowed to reclaim
	  state, so will allow it to reclaim any locks it wishes.
	- Client A sees it is allowed to reclaim locks.  Since it
	  assumes the server would not normally allow this unless it had
	  reached RECLAIM_COMPLETE, it therefore is permitted to assume
	  that the server knows somehow that it could not have granted
	  conflicting locks.  So the client concludes that it can safely
	  reclaim both x and y.

So client A attempts to reclaim y, and the server permits it.

There are may be ways to avoid situations such as the above in the face
of 4.1 clients that ignore the spec mandate, but given the clear MUST in
the spec, the burden of showing the result is correct should really be
on anyone that wants to change that requirement....

--b.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 22 Feb 2012 10:52:01 -0800

> RECLAIM_COMPLETE must be done before non-reclaim ops.

Since quite a few folks didn't understand the spec this way I believe
it warrants a clarification.

Benny

---

From: Tom Talpey <[email-masked]>
Date: Wed, 22 Feb 2012 21:10:10 +0000

Bruce Fields wrote:
>> server will keep returning NFS4ERR_GRACE forever on OPEN until the 
>> client sends RECLAIM_COMPLETE based essentially on section 18.51.3:
>> 
>> expecting the server to return to normal operation mode after the 
>> grace period is over, a-la 4.0.
>
>Also, note that the RECLAIM_COMPLETE isn't just an optimization to end the grace period early,
>it's also what the server is supposed to use to determine when a client can be trusted to reclaim
>the right locks after the next reboot:

Agreed, but shouldn't the server stop returning NFS4ERR_GRACE when its grace timer expires? After all, not all clients might be able to reconnect, reclaim, and signal RECLAIM_COMPLETE. If the server doesn't run a grace timer, it might never recover.

In other words, there appear to be two incorrect behaviors here, right? The quoted paragraph should perhaps read:

--   Whenever a client establishes a new client ID and before it does the
--   first non-reclaim operation that obtains a lock, it MUST send a
--   RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
--   locks to reclaim.  If non-reclaim locking operations are done before
--   the RECLAIM_COMPLETE *and before the grace timer expires*,
--   an NFS4ERR_GRACE error will be returned.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 22 Feb 2012 13:52:41 -0800

> Agreed, but shouldn't the server stop returning NFS4ERR_GRACE when its grace timer expires? After all, not all clients might be able to reconnect, reclaim, and signal RECLAIM_COMPLETE. If the serve…

Sure, but such clients still need to send RECLAIM_COMPLETE when their
done recovering (whether the recovery was complete or not) to ensure
client and server agree on the set of the locks they hold at that point.

> --   an NFS4ERR_GRACE error will be returned.

A client that sends a non-reclaim operation before RECLAIM_COMPLETE is
not complying with the protocol.

Such non-compliance has subtle consequences that won't be noticed until
you test some odd reboot-recovery cases.

I tend to think it will save some pain in the long run to be harsher up
front.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 22 Feb 2012 17:02:56 -0500

Gah, the list is sending each mail from my redhat address to moderation.
Resending from my home address:

> Agreed, but shouldn't the server stop returning NFS4ERR_GRACE when its grace timer expires? After all, not all clients might be able to reconnect, reclaim, and signal RECLAIM_COMPLETE. If the serve…

Sure, but clients arriving after that timer expires still must send
RECLAIM_COMPLETE when they're done recovering (even they have nothing to
reclaim, or if the reclaims failed) to ensure client and server agree on
the set of the locks they hold at that point.

(Note: the language does *not* say ERR_GRACE is returned to all clients
until all clients have sent RECLAIM_COMPLETE--the error is returned only
to the client that's trying to send a non-reclaim without a preceding
RECLAIM_COMPLETE.)

A client that sends a non-reclaim operation before RECLAIM_COMPLETE is
not complying with the protocol.  In such cases we have the choice to
recommend an error, or recommend ignoring the problem, or leave the
server's response unspecified.

In this case the non-compliance has subtle consequences that may not be
noticed until you test some odd reboot-recovery cases.

So in this case I'm inclined to think it will save some pain in the long
run to continue recommending that we be harsher up front.

--b.

---

From: <[email-masked]>
Date: Wed, 22 Feb 2012 17:29:11 -0500

> Agreed, but shouldn't the server stop returning NFS4ERR_GRACE when 
> its grace timer expires? 

Yes and no. :-)

Let me clarify that.

I don't know what the spec says exactly but I think we should think of this in 
two parts.

1) there is a client-specific obligation to return GRACE to clients that haven't
   done a RECLAIM_COMPLETE.

2) there is a global, non-client-specific obligation to rerun grace to everyone until
   some end of grace period, which, as Tom points out, cannot be unbounded in time.

> After all, not all clients might be able to reconnect, reclaim, and signal RECLAIM_COMPLETE. 
> If the server doesn't run a grace timer, it might never recover.

So Tom and I agree on the non-client-specific obligation and the spec should follow suit,
if it doesn't already.

But what about the client-specific obligation?  If there isn't a separte obligation,
with its own termination point then we are facing a choice between the possibility of
an unbounded global grace period, and giving up on RECLAIM_COMPLETE.  

Does the spec actively deny the possibility of two separte grace obligations or is it
just confusing?

---

From: Benny Halevy <[email-masked]>
Date: Wed, 22 Feb 2012 18:40:48 -0800

> Does the spec actively deny the possibility of two separte grace obligations or is it

To me, it just seems confusing.

Benny