Subject: [nfsv4] NFS4ERR_EXPIRED, the problem continues

From: <[email-masked]>
Date: Tue, 7 Feb 2012 12:56:59 -0500

The background here is that, in connection with the discussion of persistent delegation reclaim, Tom urged us to look at bis-16, since it was good to get more eyes on  what is going to be the v4..0 spec some time coming up.

The good new is that Tom was wise to make that request and I did look at bis-16 with regard to these issues.  Come to think of it, that's the bad news as well, since the fact that there were some problems to find, is exactly why Tom was wise to make the request.  Sigh!

So here' s the problem.  First we have:

13.1.5.3. NFS4ERR_EXPIRED (Error Code 10011)
A stateid designates locking state of any type that has been revoked
due to expiration of the client's lease, either immediately upon
lease expiration, or following a later request for a conflicting
Lock

Then we have two things which contradict it.

Section 9.6.3.1 has the he following paragraph:

If the server does not reboot before the network partition is healed,
when the original client tries to access a courtesy lock which was
freed, the server SHOULD send back a NFS4ERR_BAD_STATEID to the
client. If the client tries to access a courtesy lock which was not
freed, then the server SHOULD mark all of the courtesy locks as
implicitly being renewed.

Which contracts the last clause of 13.1.5.3.

Note that section 10.2.1 (which I'm trying to rewrite for the CLAIM_DELEGATE_PREV stuff) says:

A client normally finds out about revocation of a delegation when it
uses a stateid associated with a delegation and receives the error
NFS4ERR_EXPIRED. It also may find out about delegation revocation
after a client reboot when it attempts to reclaim a delegation and
receives that same error. Note that in the case of a revoked
OPEN_DELEGATE_WRITE delegation, there are issues because data may
have been modified by the client whose delegation is revoked and
separately by other clients. See Section 10.5.1 for a discussion of
such issues. Note also that when delegations are revoked,
information about the revoked delegation will be written by the
server to stable storage (as described in Section 9.6). This is done
to deal with the case in which a server reboots after revoking a
delegation but before the client holding the revoked delegation is
notified about the revocation.

Although it doesn't explicitly say so, the implication is that NFS4ERR_EXPIRED is what you
get when a delegation is revoked without lease expiration (which I'd call "normally" :-).  So if not,
NFS4ERR_EXPIRED, what is returned in this case?

Note that section 9.1.3.4 says:

 o  If the stateid represents revoked state, then return
    NFS4ERR_EXPIRED or NFS4ERR_ADMIN_REVOKED, as appropriate.

Which suggests that NFS4ERR_EXPIRED is returned in this case.

So, in addition to the editing issue of making things consistent, we have a substantive issue to
decide about.

The motivation of the courtesy lock stuff seems to have been to provide the client a signal
(NFS4ERR_EXPIRED) that indicates, "you have lost all your locks"  (rather just one or a few),
but the treatment of delegation revocation vitiates that.

I think I need such an indication in some form, because there is a need to specify conditions when you may
do persistent delegation recovery and the occasion of losing all your locks due to a network partition is
one of those situations (while the occasion of losing a few locks, or having a specific delegation revoked is
certainly not one of those situations)..

Anybody have any ideas on how to resolve this?

---

From: Rick Macklem <[email-masked]>
Date: Tue, 7 Feb 2012 20:04:29 -0500 (EST)

David Noveck wrote:
> The background here is that, in connection with the discussion of
> persistent delegation reclaim, Tom urged us to look at bis-16, since
> it was good to get more eyes on what is going to be the v4..0 spec
> some time coming up.
> 
> The good new is that Tom was wise to make that request and I did look
> at bis-16 with regard to these issues. Come to think of it, that's the
> bad news as well, since the fact that there were some problems to
> find, is exactly why Tom was wise to make the request. Sigh!
> 
I'm confused by this para? I don't see how a server would "normally"
revoke a delegation unless the lease has expired. When it does a CB_RECALL,
it is supposed to wait at least a lease duration for a DelegReturn
before revoking the delegation.

Did you mean to say "after lease expiration" and not "without lease
expiration"? (By "after lease expiration" I mean that the client has
failed to renew the lease within the lease duration. I don't meant that
the server has thrown all open/lock state for the client away because
it didn't renew the lease within the lease duration.)

> Which suggests that NFS4ERR_EXPIRED is returned in this case.
I think the case where the lease has not expired is NFS4ERR_ADMIN_REVOKED
as I noted, above. Also, see the comment below.

> but the treatment of delegation revocation vitiates that.
If the server "revokes" a delegation after lease expiration, but does
not "revoke" the rest of the open/lock state held for a client, then
I see a problem, which I think is the one you are referring to.

For that case (the FreeBSD server holds onto state after the lease
has expired, which is a variant of "courtesy locks", but throws all the
open/lock related state away when any of it is being thrown away, so it
doesn't have to handle this case), I can see the problem and it seems that
a server must return NFS4ERR_BAD_STATEID for this case, unless defining
a new error code NFS4ERR_EXPIRED_STATEID could be added to the spec?

> Anybody have any ideas on how to resolve this?
I think that, if a new error like NFS4ERR_EXPIRED_STATEID can't be added
it will be better to use NFS4ERR_BAD_STATEID for the case where a stateid
was revoked, but others are still valid vs NFS4ERR_EXPIRED for the case
where all state for a client has been revoked.

If NFS4ERR_EXPIRED is overloaded to refer to both "this stateid" and
"all state for the client" then I think a client will have difficulty
determining how to recover from NFS4ERR_EXPIRED.

rick
> 
> 
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4