Subject: [nfsv4] NFS4ERR_EXPIRED, the saga continues

From: <[email-masked]>
Date: Fri, 10 Feb 2012 12:35:38 -0500

So far, among people willing to discuss the subject, there seems to be general agreement that NFS4ERR_EXPIRED should indicate loss of all locks due to lease cancellation.  Assuming that quasi-consensus holds, there are other related matters to discuss.

Section 9.6.3.2 says:

  NFS4ERR_EXPIRED:

      All locks has been revoked during the partition.  The client
      should use a SETCLIENTID to recover.

The big issue is the second sentence.  Notice it says "should" and not "SHOULD".

I had been under the impression that when you had lease cancellation, you did have an invalid clientid and
would need SETCLIENTID to recover.

But when I discussed giving back NFS4ERR_STALE_CLIENTID, Rick convinced me that it wasn't
valid and I note that there doesn't seem to be any other CLIENTID-NG errors.  Also, when I looked
at the discussion of network partition, it didn't say anything about doing the SETCLIENTID in this
case.

So is this just a case of "you should recover but I can't tell you have to, i.e. you can just panic if you really want to"?

Specifically, if you have a network partition (and lease cancellation), and you do a SETCLIENTID,
Do you get back the same client id or a new one (if the client hasn't rebooted).

What happens if you don't and you do a RENEW.  What error, if any, do you get?

---

From: Rick Macklem <[email-masked]>
Date: Sun, 12 Feb 2012 11:20:04 -0500 (EST)

David Noveck wrote:
> So far, among people willing to discuss the subject, there seems to be
> general agreement that NFS4ERR_EXPIRED should indicate loss of all
> locks due to lease cancellation. Assuming that quasi-consensus holds,
> there are other related matters to discuss.
> 
Well, since no one else has responded, I guess I'll throw in my $0.01 worth.
(I'll mostly state what the FreeBSD server and client currently do. I can't
 say whether or not this is correct behaviour.)

> would need SETCLIENTID to recover.
That is my impression as well. (I think the "should" might just have
been because there is no good way for a client to recover from this.)
The FreeBSD server "throws away" the confirmed record for a clientid
when a lease has expired and a conflicting open/lock request arrives.
After that, it will return NFS4ERR_EXPIRED for the clientid and all
associated stateids, if they were generated in the current boot instance.

As such, the client can no longer use the clientid.

If the FreeBSD client receives NFS4ERR_EXPIRED it:
- does Setclientid/SetclientidConfirm to get a new clientid. It uses a
  new client.verifier, but that wouldn't matter for the FreeBSD server.
- does Opens with CLAIM_NULL for all files that were open, but do not
  have associated byte range locks on them. (It always does opens with
  ACCESS_DENY_NONE, so I believe that this is relatively safe.)
  Byte range locks and their associated opens are "thrown away", since
  they can't safely be recovered. (Unfortunately FreeBSD does not have
  a "lost locks" signal, so processes might not know that they've lost
  their locks. They will not be able to do further I/O or locking.)

> So is this just a case of "you should recover but I can't tell you
> have to, i.e. you can just panic if you really want to"?
> 
Unfortunately, I don't think there is a good way for a client to recover
from this. I've described the compromise that the FreeBSD client currently
uses above, but if there is a better way, I'd like to hear about it.

> Do you get back the same client id or a new one (if the client hasn't
For the FreeBSD server, it is a new one. Since the FreeBSD client uses
a different client.verifier, I believe the same should be true for other
servers?

> What happens if you don't and you do a RENEW. What error, if any, do
Prior to doing a SetClientid/SetClientidConfirm to get a new clientid,
Renew with the old clientid will reply NFS4ERR_EXPIRED for the FreeBSD
server. (That is on the list of errors for Renew.)

> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Mon, 13 Feb 2012 16:36:21 -0500

> > The big issue is the second sentence. Notice it says "should" and not
. > "SHOULD".
> been because there is no good way for a client to recover from this.)

As you have laid it out:
O If you don't do SETCLIENTID, you can't do anything
O If you do SETCLIENTID, your recovery may not be great but it's the best 
  you can do.

So maybe it should say:

    The client SHOULD use a SETCLIENTID to do recovery or something kind  
    of like it.

Maybe not :-)

> associated stateids, if they were generated in the current boot instance.

Despite throwing away the confirmed record, does it remember the clientid so it
knows to return NFS4ERR_EXPIRED?

> As such, the client can no longer use the clientid.

Do you know of anywhere in 3530 or bis-16 that specifies this?  I'm not saying
your server is wrong, but they should agree and if your server isn't wrong,
the spec should be corrected.

>  new client.verifier, but that wouldn't matter for the FreeBSD server.

Are you saying that if the verifier were the same, it would do the same
thing?

> uses above, but if there is a better way, I'd like to hear about it.

I'd like to hear about any other ways, so we create spec text that is compatible
with all existing clients and servers, if that is possible.

> server. (That is on the list of errors for Renew.)

That's good to know.  It certainly means the description of NFS4ERR_EXPIRED 
will have to include some text to address its use to indicate a problem with 
a clientid.

BTW, what does your server return if you use the old clientid in a RENEW 
after doing the SETCLIENTID to get the new one?

---

From: Rick Macklem <[email-masked]>
Date: Tue, 14 Feb 2012 19:02:39 -0500 (EST)

David Noveck wrote:
> > > I had been under the impression that when you had lease
> > > cancellation,
> > > you did have an invalid clientid and
Depends on what you mean by "remember". It doesn't not have any record
of the clientid. It returns NFS4ERR_EXPIRED when the clientid is for the
current boot instance (a boot instance is part of the clientid) and it
doesn't have any record for it. (So, it "remembers" by not having a record
for it and by generating clientids such that the same clientid won't be
generated again within the boot instance and by putting the boot instance
in the clientid.)

Stateids related to a clientid have that clientid imbedded in the stateid,
so the same applies to stateids.

> the spec should be corrected.
Not that I can see. The only "hint" is the above statement
"The client should use a SETCLIENTID to recover." Beyond that, if a server
allows the clientid to be used for "new stateids (issued after a network
partition heals, as indicated by a Renew reply or similar), then the server
would have to keep track of the "old" stateids forever (actually until a
server reboot), so that it can reply NFS4ERR_EXPIRED when those old stateids
are used by a client. To do this, I think it would require keeping those
stateid records "forever", which could constipate a server. (A server implementor
might try and do this by partitioning the stateid's "bit space" between old and
new, but it could conceivably have to do this many times for a flakey network
connection. Sooner or later I think the partitioning scheme would fail, because
there are a finite # of bits in a stateid?) Yea, kinda convoluted, but I think
it's an argument for not allowing a clientid to work after it has expired?

> Are you saying that if the verifier were the same, it would do the
In the case of the FreeBSD server, yes, since the server has no record
of the longhand nfs_client_id4. However, I don't believe that this behaviour
should be expected of a server. The FreeBSD client always uses a different
client.verifier field for the SetClientID used during NFS4ERR_EXPIRED
recovery.

> > > valid and I note that there doesn't seem to be any other
> > > CLIENTID-NG
> > > errors. Also, when I looked
> > > at the discussion of network partition, it didn't say anything
> > > about
> > >doing the SETCLIENTID in this
> > > case.
> >
> > > Do you get back the same client id or a new one (if the client
> > > hasn't
> > rebooted).
> >
> > > What happens if you don't and you do a RENEW. What error, if any,
> > > do
> > > you get?
> >
> > "SHOULD".
> > I had been under the impression that when you had lease
> > cancellation,
> > you did have an invalid clientid and
> > Do you get back the same client id or a new one (if the client
> > hasn't
> > rebooted).
> >

---

From: Rick Macklem <[email-masked]>
Date: Wed, 15 Feb 2012 11:11:53 -0500 (EST)

> the spec should be corrected.
Btw, you might want to look at this email thread in the archive. Here the
url for one of the messages in it. (Back in Oct. 2004.)

http://www.ietf.org/mail-archive/web/nfsv4/current/msg01220.html

It had NFS4ERR_ADMIN_REVOKED in the subject header, but discusses
NFS4ERR_EXPIRED as well.