Subject: Some issues in section 8.5.3

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 24 Apr 2001 12:18:34 -0700

I think there are some problems here that need to be addressed in
the next spec go-round.

> 8.5.3.  Network Partitions and Recovery
> 
>    If the duration of a network partition is greater than the lease
>    period provided by the server, the server will have not received a
>    lease renewal from the client.  If this occurs, the server may free
>    all locks held for the client.  As a result, all stateids held by the
>    client will become invalid or stale.  Once the client is able to
>    reach the server after such a network partition, all I/O submitted by
>    the client with the now invalid stateids will fail with the server
>    returning the error NFS4ERR_EXPIRED.  Once this error is received,
>    the client will suitably notify the application that held the lock.
>
>    As a courtesy to the client or as an optimization, the server may
>    continue to hold locks on behalf of a client for which recent
>    communication has extended beyond the lease period.  If the server
>    receives a lock or I/O request that conflicts with one of these
>    courtesy locks, the server must free the courtesy lock and grant the
>    new request.

It should probably say here that the stateid's will at this point 
become invalid and their subsequent use will result in NFS4ERR_EXPIRED.

Also, if the associated lease is renewed, then these courtesy locks
become ordinary (discourteous?) locks, once again.

>    If the server continues to hold locks beyond the expiration of a
>    client's lease, the server MUST employ a method of recording this
>    fact in its stable storage. 

What is "this fact"?  As the sentence is weitten it seems that it is
the fact that the locks have been held past the expiration of the 
client's lease.

However, if we look at the example below, it doesn't seem that this
is the relevant fact.  The relevant fact, which you would have to
record, seems to be that the client which thinks it has a lock, has
lost it and that there is a (potential or actual) conflicting lock
which may have been granted.  This would apply however, to servers 
that release expired locks immediately as well as those that try to 
be more courteous.  And if that is true, then "If the server continues
to hold locks beond the expiration of a client's lease, the server
MUST ...", implying that other servers need not, is definitely wrong.

>                              Conflicting locks requests from another
>    client may be serviced after the lease expiration.  There are various
>    scenarios involving server failure after such an event that require
>    the storage of these lease expirations or network partitions.  One
>    scenario is as follows:
>
>         A client holds a lock at the server and encounters a network
>         partition and is unable to renew the associated lease.  A
>         second client obtains a conflicting lock and then frees the
>         lock.

But this can happen on a server which frees the expired locks immediately,
just as well.  A conflicting lock can still be granted after lease
expiration in either case.

Either you have:

     Lease expiration

     Lock is freed

     Conflicting request made
  
     Conflicting request granted

Or you have:

     Lease expiration

     Lock becomes "courtesy" lock

     Conflicting request made 

     Lock is freed

     Conflicting request granted

The final state of affairs on the server and what the client sees are
identical.

>                 After the unlock request by the second client, the
>         server reboots or reinitializes.  Once the server recovers, the
>         network partition heals and the original client attempts to
>         reclaim the original lock.
>
>   In this scenario and without any state information, the server will
>   allow the reclaim and the client will be in an inconsistent state
>   because the server or the client has no knowledge of the conflicting
>   lock.
>
>   The server may choose to store this lease expiration or network
>   partitioning state in a way that will only identify the client as a
>   whole. 

A server that frees all locks on expiration might record the fact of
the expiration.  A server that kept the locks around might record the
fact that some conflicting lock has been granted that revoked a lock
held by this client.  Or either could do something more fine-grained. 

>           Note that this may potentially lead to lock reclaims being
>   denied unnecessarily because of a mix of conflicting and non-
>   conflicting locks. 

In the case of a server that freed locks on lease expiration, they
might be denied even if there were no conflicting locks.  But that's
a quality of implementation issue. 

>                       The server may also choose to store information
>   about each lock that has an expired lease with an associated
>   conflicting lock.  The choice of the amount and type of state
>   information that is stored is left to the implementor.  In any case,
>   the server must have enough state information to enable correct
>   recovery from multiple partitions and multiple server failures.

Which has to do with being able to determine when a conflicting lock
might have been granted, but not necessarily knowing that it has.

Now just when you thought when it safe to go back in the water...
there is another case where a conflicting lock may be granted that isn't
covered by the above discussion.

So here is the scenario:

   Client A gets a lock.  

   Network partition so that A and the server cannot communicate.  

   Server reboots (before client A's lease has expired so no expiration 
   or conflicting lock recorded).  

   Grace period goes by without A getting a chance to reclaim his lock, 
   because of the network partition.  

   Server grants a lock (to client B) that would have conflicted with 
   the lock that A held (and still thinks he has), but the server 
   doesn't know it is conflicting because he knows nothing about A's 
   lock or about A at all.

   Client B releases his lock.

   Server reboots again.

   Network partition heals.

   Client A tries to reclaim his lock, during the grace period, and
   succeeds, since it is during the grace period and the server had
   no occasion to record any conflicting lock.  This results in exactly
   the same problem that occurred in the example which is in 8.5.3:
   the client gets his lock, unaware that a conflicting lock has been
   granted and returned in the interim.

I think what a server would have to do prevent this from happening is
keep a table (in stable storage) of the clients (in terms of client-
opaque-sring/verifier) to which is it has granted a lock during the
current server instance.  Upon reboot, the right to reclaim during 
the grace period has to be limited to clients which actually had locks
to reclaim during the previous server instance (and not those eons
past).  This isn't terribly onerous to implement (one write to a file
the first time you grant a lock for a given client and rename over the
existing version at the end of the grace period).  I'd certainly
prefer coding it to trying to describe it in the spec.

A server which maintained essentially all lock state in stable storage
would have enough information so that it could determine when conflicting
locks where granted without depending on client reclaims wouldn't
have to do this, which just makes it even more complicated for the
spec to describe exactly what the server has to do under various 
conditions.