Subject: RE: [nfsv4] re: NFS4ERR_EXPIRED or NFS4ERR_STALECLIENTID?

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 9 Apr 2004 15:20:19 -0700

Spencer wrote:

I would like to reiterate Dave's comment above that the server "do the
best it can". 

A client will likely treat EXPIRED differently than BAD_STATEID in that
EXPIRED 

applies to the entire clientid and BAD_STATEID is just for that
particular file (or 

likely to be that particular file). The recovery paths could apply
different 

approaches and if a simple timestamp in the stateid provides a directive

then EXPIRED would have greater precedence than BAD_STATEID in 

my opinion and experience. 

I think what you are saying/implying is very reasonable, but the spec is
not very 

clear/forthcoming about these issues.  Section 8.8 talks about
validating your

stateid's when a lease expiration may have occurred, but the section
ends its

discussion of validation with a statement that just talks about failure
generically

and does not discuss the two errors above.  I think it would help (me)
to add  

to the last paragraph something like:

When the error return is NFS4ERR_EXPIRED, the client is entitled to
assume that

all of the stateid's associated with the same clientid as the one used
have been freed

as a result of the lease expiration.  On the other hand, when the error
is NFS4ERR_BAD_STATEID,

the client is only entitled to assume that that particular stateid is
invalid, presumably due 

to consequences of the lease expiration, such as loss of a lock to a
conflicting request.

It may not assume that other stateid's associated with the same clientid
have also been lost,

until such information has been determined for the stateid's
individually.  It is helpful for a 

server to save information about expired leases so that it may give the
NFS4ERR_EXPIRED 

appropriately, but when when it cannot save such information for
whatever reason, it should

return NFS4ERR_BAD_STATEID in order to not to mislead the client into
releasing lock 

state that the server might still be hold.

Does that correspond to your understanding?

One other thing is that if you look at the error definitions, the
assumptions that I have been

making (and that I think you are making as well) are not exactly
obvious:

   NFS4ERR_BAD_STATEID   A stateid generated by the current server
                         instance, but which does not designate any
                         locking state (either current or superseded)
                         for a current lockowner-file pair, was used.

  NFS4ERR_EXPIRED       A lease has expired that is being used in the
                         current operation.

But a lease has expired in all the cases we are talking about.  Being
more

explicit here would help I think.

 

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Monday, April 05, 2004 3:33 PM
To: Noveck, Dave
Cc: Stevan Steve Allen; [email-masked]
Subject: Re: [nfsv4] re: NFS4ERR_EXPIRED or NFS4ERR_STALECLIENTID?

On Apr 5, 2004, at 12:36 PM, Noveck, Dave wrote: 

You can distinguish STALECLIENTID without saving clientid's forever. 

If you put the server instance (or reboot time) in one half of the
client id then you can determine if the server has rebooted, simply by
comparing that field. 

The spec text does not talk about EXPIRED with regard to client id's. It
does with respect to stateid's.  However, the ops that list
STALECLIENTID as an error also list EXPIRED so I don't see what else you
could return if you get a clientid you don't recognize and it doesn't
show that you have rebooted. 

One exception is DELEGPURGE.  It shows STALECLIENTID but not EXPIRED so
it isn't clear what you would do if you get a clientid you don't
recognize and it doesn't look stale.  INVAL would be the obvious answer
but that isn't on the error list either :-( 

So I think you have to be able to distinguish clientid's you gave out
from those your predecessor instance(s) would have given out essentially
forever.  However, for anything that looks like it cam from the current
server instance, if you don't recognize it, you return EXPIRED and there
is no issue of having to save them or distinguish those that expired
from others.  There is no BAD_CLIENTID error .  This conclusion does
seem to be at variance with the definition of EXPIRED, "A lease has
expired that is being used in the current operation", but there doesn't
seem to be any other error to use. 

Which brings us to stateid's since there is a BAD_STATEID error.  I
believe you can do the same thing for stateid's that you do for
clientid's to distinguish when to return STALE_STATEID.  This leaves the
question of distinguishing EXPIRED from BAD_STATEID. 

I don't see where the spec gives any useful guidance on how one is to
distinguish these two cases without storing expired stateid's forever
(have I missed it?).  My answer is to do the best you can but
practically speaking I think clients have to be prepared to get
BAD_STATEID and do what they would have done if they got EXPIRED.
Either one says the server is still up, you lost your lock, and if you
try to re-establish it, you may not succeed and even if you do, you
should be aware that someone may have had a conflicting lock in the
interim, i.e. you cannot assume continuity of your lock possession. 

I would like to reiterate Dave's comment above that the server "do the
best it can". 

A client will likely treat EXPIRED differently than BAD_STATEID in that
EXPIRED 

applies to the entire clientid and BAD_STATEID is just for that
particular file (or 

likely to be that particular file). The recovery paths could apply
different 

approaches and if a simple timestamp in the stateid provides a directive

then EXPIRED would have greater precedence than BAD_STATEID in 

my opinion and experience. 

-----Original Message----- 

From: Stevan Steve Allen [mailto:[email-masked]] 

Sent: Monday, April 05, 2004 12:21 PM 

To: [email-masked] 

Subject: Re: [nfsv4] re: NFS4ERR_EXPIRED or NFS4ERR_STALECLIENTID? 

Spencer,  I fell off the wagon here. 

> NFS4ERR_STALE_CLIENTID is only returned when the server 

> has restarted. 

Seems the server may not find a server generated clientid for 2
conditions. 

  o Server restarted 

  o Clientid was released (freed) 

Your statement indicates the server has to determine which condition
occurred.  Rick asked how long the clientid has to be kept around (so
separate errors can be returned). 

Thanks, 

Stevan C. Allen

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 12 Apr 2004 14:48:28 -0500

On Apr 9, 2004, at 5:20 PM, Noveck, Dave wrote:

>  A client will likely treat EXPIRED differently than BAD_STATEID in 
> that EXPIRED
>
>  I think what you are saying/implying is very reasonable, but the spec 
> is not very
>
> discussion of validation with a statement that just talks about 
> failure generically
>
> as a result of the lease expiration.  On the other hand, when the 
> error is NFS4ERR_BAD_STATEID,
>
> It may not assume that other stateid's associated with the same 
> clientid have also been lost,
>
>  server to save information about expired leases so that it may give 
> the NFS4ERR_EXPIRED
>

With one addition in that the stateid may just be bad; meaning that 
there has
been an implementation error at the client or server and it is just not 
recognizable.
Hard/impossible to recover from other than having the client release 
all state for itself.
A client may choose (depending on the history of the file) to "reopen" 
the file with a new
open_owner to avoid any possible implementation issues.

The "bad" nature of a stateid is mentioned briefly in 8.1.3 when 
suggesting
an implementation methodology.

> explicit here would help I think.

The case I am thinking about is that the stateid contains a timestampt
from the server instance and the remainder is some internal identifier.
The server can inspect the timestamp portion and reasonably guess
that it is from its current instantiation but the remainder may not 
match
any internal mapping it currently has in that it just may not validate 
even
without the expiration of a lease.  Therefore, the server could 
determine
that the lease has or has not expired for that stateid and then validate
the remainder, etc. to determine if BAD_STATEID should be returned.

In any case, I agree that some wording and clarification as has been
given in this thread would be very helpful.

Spencer

> The spec text does not talk about EXPIRED with regard to client id's. 
> It does with respect to stateid's.  However, the ops that list 
> STALECLIENTID as an error also list EXPIRED so I don't see what else 
> you could return if you get a clientid you don't recognize and it 
> doesn't show that you have rebooted.
>
> One exception is DELEGPURGE.  It shows STALECLIENTID but not EXPIRED 
> so it isn't clear what you would do if you get a clientid you don't 
> recognize and it doesn't look stale.  INVAL would be the obvious 
> answer but that isn't on the error list either :-(
>
> from those your predecessor instance(s) would have given out 
> essentially forever.  However, for anything that looks like it cam 
> from the current server instance, if you don't recognize it, you 
> return EXPIRED and there is no issue of having to save them or 
> distinguish those that expired from others.  There is no BAD_CLIENTID 
> error .  This conclusion does seem to be at variance with the 
> definition of EXPIRED, "A lease has expired that is being used in the 
> current operation", but there doesn't seem to be any other error to 
> use.
>
> clientid's to distinguish when to return STALE_STATEID.  This leaves 
> the question of distinguishing EXPIRED from BAD_STATEID.
>
>  A client will likely treat EXPIRED differently than BAD_STATEID in 
> that EXPIRED
>