Subject: Re: [nfsv4] RECLAIM_COMPLETE mandatory for 4.1 client?

From: <[email-masked]>
Date: Tue, 13 Mar 2012 11:24:16 -0400

> > Does the spec actively deny the possibility of two separte grace obligations or is it
> > just confusing?
> > 
>
> To me, it just seems confusing. 

Let's see if we can make it less confusing.

Following is an outline of a proposal to address this as an errata.  If there is
general agreement that this is within a few degrees of the right direction, I can
refine it into proposed errata text.

So we have the following in 18.51.3.

E>  Whenever a client establishes a new client ID and before it does the
E>  first non-reclaim operation that obtains a lock, it MUST send a
E>  RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
E>  locks to reclaim.  If non-reclaim locking operations are done before
E>  the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned.

"will be returned" ??  Why?  Why not "MUST" or "SHOULD"?  Those would make it 
clear that the server has an obligation to do that, and the context would
strongly imply it was client-specific.  As it is, it is easy to suppose that
that this happens naturally and the only way for it to happen naturally is if 
the grace period were not over, which gets into the problem that Tom was 
(rightly) concerned about.

So to answer my question above, the spec passively denies the possibility of
two separate grace obligations.

How about replacing the above with;

P>  Whenever a client establishes a new client ID and before it does the
P>  first non-reclaim operation that obtains a lock, it MUST send a
P>  RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
P>  locks to reclaim.  If non-reclaim locking operations are done before
P>  the RECLAIM_COMPLETE, the server SHOULD return an NFS4ERR_GRACE error,
P>  even though the grace period (as seen by other clients) may have
P>  completed.
P>
P>  Note that even after a RECLAIM_COMPLETE is done, the client may also
P>  receive an NFS4ERR_GRACE error when doing non-reclaim locking operations,
P>  if the server grace period is still in effect.

Now on to 8.4.2.1 and this is an area which definitely is confusing.

E>   The grace period may last until all clients that are known to
E>   possibly have had locks have done a global RECLAIM_COMPLETE
E>   operation, indicating that they have finished reclaiming the locks
E>   they held before the server restart.  

Note "may" and not "MAY".  Also note that although a client "may" do this
stupid thing, it is still something that he should not do, if he doesn't
want people to be upset about eternal grace periods.  Finally, note that
this statement doesn't include clients that have not had locks that have
not done a  RECLAIM_COMPLETE.

E>  This means that a client that
E>  has done a RECLAIM_COMPLETE must be prepared to receive an
E>  NFS4ERR_GRACE when attempting to acquire new locks.  

E>  In order for the
E>  server to know that all clients with possible prior lock state have
E>  done a RECLAIM_COMPLETE, the server must maintain in stable storage a
E>  list clients that may have such locks.  

"must" adds to the confusion.

E>  The server may also terminate
E>  the grace period before all clients have done a global
E>  RECLAIM_COMPLETE.  

The spec says that you may do something sane also.

E>  The server SHOULD NOT terminate the grace period
E>  before a time equal to the lease period in order to give clients an
E>  opportunity to find out about the server restart, as a result of
E>  sending requests on associated sessions with a frequency governed by
E>  the lease time.  

Finally, a "SHOULD NOT".  Isn't this where we should start, with something you
should do and leave all the possible sane and insane variants for later?

E>  Note that when a client does not send such requests
E>  (or they are sent by the client but not received by the server), it
E>  is possible for the grace period to expire before the client finds
E>  out that the server restart has occurred.

Isn't this just saying that If people don't obey the protocol, by renewing 
their leases, things are going to break?

So how about the following paragraphs as replacement for the above.

P>  The server SHOULD NOT terminate the grace period before a time equal 
P>  to the lease period in order to give clients an opportunity to find 
P>  out about the server restart, as a result of clients sending requests 
P>  on associated sessions with a frequency governed by the lease time, as
P>  as they are obliged to do, in order to avoid lease expiration.  A server
P>  MAY extend the grace period beyond this minimum in order to maximize
P>  the chances that clients will have an adequate opportunity to reclaim
P>  all their locks and do a RECLAIM_COMPLETE, even in the face of difficult
P>  network conditions. 

P>  A server has the ability to terminate the grace period before
P>  all clients have done a RECLAIM_COMPLETE, and needs to have that 
P>  flexibility, since otherwise, a non-compliant client could prevent
P>  the grace period from ever ending.  Note that when the grace period ends
P>  before a client has reclaimed all its locks and done a RECLAIM_COMPLETE,
P>  it will be in a confusing situation in which:
P>    o non-reclaim locking requests get a GRACE error.
P>    o lock reclaim requests get a NO_GRACE error
P>  
P>   A client in this situation needs to do a RECLAIM_COMPLETE to acknowledge
P>   loss of its unreclaimed locks and proceed to normal processing.

P>   A server which maintains in stable storage a list of clients that may
P>   have reclaimable lock state, may attempt to make the grace period last 
P>   until all such clients have done a RECLAIM_COMPLETE operation, indicating 
P>   that they have finished reclaiming the locks they held before the server 
P>   restart.  However, given the possibility that a client might fail to do an
P>   appropriate RECLAIM_COMPLETE, such a server has to have the ability to 
P>   terminate the grace period short of that point.  

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Wednesday, February 22, 2012 9:41 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] RECLAIM_COMPLETE mandatory for 4.1 client?

On 2012-02-22 14:29, [email-masked] wrote:
>> Agreed, but shouldn't the server stop returning NFS4ERR_GRACE when 
>> its grace timer expires? 
> 
> 
> Yes and no. :-)
> 
> Let me clarify that.
> 
> I don't know what the spec says exactly but I think we should think of this in 
> two parts.
> 
> 1) there is a client-specific obligation to return GRACE to clients that haven't
>    done a RECLAIM_COMPLETE.
> 
> 2) there is a global, non-client-specific obligation to rerun grace to everyone until
>    some end of grace period, which, as Tom points out, cannot be unbounded in time.
> 
>> After all, not all clients might be able to reconnect, reclaim, and signal RECLAIM_COMPLETE. 
>> If the server doesn't run a grace timer, it might never recover.
> 
> So Tom and I agree on the non-client-specific obligation and the spec should follow suit,
> if it doesn't already.
> 
> But what about the client-specific obligation?  If there isn't a separte obligation,
> with its own termination point then we are facing a choice between the possibility of
> an unbounded global grace period, and giving up on RECLAIM_COMPLETE.  
> 
> Does the spec actively deny the possibility of two separte grace obligations or is it
> just confusing?
> 

To me, it just seems confusing.

Benny

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
> Sent: Wednesday, February 22, 2012 4:10 PM
> To: J. Bruce Fields; Benny Halevy
> Cc: Lev Solomonov; [email-masked]
> Subject: Re: [nfsv4] RECLAIM_COMPLETE mandatory for 4.1 client?
> 
> Bruce Fields wrote:
>> On Tue, Feb 21, 2012 at 07:59:01AM -0800, Benny Halevy wrote:
>>> We've hit an issue with the linux 4.1 sever post linux 3.0 where the 
>>> server will keep returning NFS4ERR_GRACE forever on OPEN until the 
>>> client sends RECLAIM_COMPLETE based essentially on section 18.51.3:
>>>
>>>    Whenever a client establishes a new client ID and before it does the
>>>    first non-reclaim operation that obtains a lock, it MUST send a
>>>    RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
>>>    locks to reclaim.  If non-reclaim locking operations are done before
>>>    the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned.
>>>
>>> However, several client implementers never send RECLAIM_COMPLETE, 
>>> expecting the server to return to normal operation mode after the 
>>> grace period is over, a-la 4.0.
>>
>> That's an extremely clear violation of the above MUST.
>>
>> Also, note that the RECLAIM_COMPLETE isn't just an optimization to end the grace period early,
>> it's also what the server is supposed to use to determine when a client can be trusted to reclaim
>> the right locks after the next reboot:
> 
> Agreed, but shouldn't the server stop returning NFS4ERR_GRACE when its grace timer expires? After all, not all clients might be able to reconnect, reclaim, and signal RECLAIM_COMPLETE. If the server doesn't run a grace timer, it might never recover.
> 
> In other words, there appear to be two incorrect behaviors here, right? The quoted paragraph should perhaps read:
> 
> --   Whenever a client establishes a new client ID and before it does the
> --   first non-reclaim operation that obtains a lock, it MUST send a
> --   RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
> --   locks to reclaim.  If non-reclaim locking operations are done before
> --   the RECLAIM_COMPLETE *and before the grace timer expires*,
> --   an NFS4ERR_GRACE error will be returned.
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 13 Mar 2012 12:24:14 -0400

> P>  the RECLAIM_COMPLETE, the server SHOULD return an NFS4ERR_GRACE error,

So that last SHOULD means in part that "there may exist valid reasons in
particular circumstances to ignore" the recommendation.

Do we have any examples of such reasons?

I'd rather go with the MUST and hopefully reduce the chances of clients
violating the client-side MUST escaping into the wild.

Looks good otherwise.

These look good to me; nit:

> P>  as they are obliged to do, in order to avoid lease expiration.

I'm running out of breath there.  Maybe something like this?:

	The server SHOULD enforce a grace period at least as long as the
	lease period, to give clients renewing their state regularly the
	opportunity to find out about the server restart.

--b.

> P>  A server

---

From: <[email-masked]>
Date: Wed, 14 Mar 2012 09:51:04 -0400

> particular circumstances to ignore" the recommendation.

To tell you the truth, I was just skittish about going all the way from "will"
to "MUST" in one step.

> Do we have any examples of such reasons?

I don't know of any but, since it says "may exist valid reasons", I have no way 
of being sure that they don't exist.

The one reason I can think of, to accommodate existing clients, would probably
not count as "valid".
 
> violating the client-side MUST escaping into the wild.

Makes sense. 

> Maybe something like this?:

>	opportunity to find out about the server restart.

Definitely an improvement.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Tuesday, March 13, 2012 12:24 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] RECLAIM_COMPLETE mandatory for 4.1 client?

Looks good otherwise.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 14 Mar 2012 11:43:23 -0400

> To tell you the truth, I was just skittish about going all the way from "will"

Hah.  Or if you read "will" as meaning that something is an
incontrovertible fact, it could be a step down to merely state the the
ietf requires it to be true....

(Where's the rfc defining "to be", anyway?)

--b.