Subject: idea - persistent locks

From: "Wittle, Mark" <[email-masked]>
Date: Thu, 15 Mar 2001 20:12:10 -0800

Why PERSIST Locks
-----------------
PERSIST locks are provided to aid clients in performing
recovery in at least two specific situations.  The first is
where both client and server volatile state is lost.  An
example is power loss to an entire cluster.  Because the
client failed, it cannot reclaim a lock to regain its state
at the time of failure.  Because the server failed, it cannot
keep the client's lock state intact until the client's lease
expires.

The other case is for clients to recover for each other in
cases where one client takes a long-term failure and does not
restart within the lease period.  In this case, another client
needs to perform recovery and would like to have the locks held
until it finishes.

Description
-----------
The model of PERSIST locks is that rather than being released
following certain failure events they become broken.  Their
state must survive client failures, server failures, and
network partitions.  Where a normal lock would have become
subject to revocation, PERSIST locks enter a state of being
breakable.  The specific conditions for a lock to become
breakable are either a client lease expiration or a server
restart grace period expiration.  When a lock is breakable,
any conflicting lock request causes the lock to be broken.
The lock also becomes broken when a client reboot occurs.

When a PERSIST lock is broken, it behaves much like a normal
lock to read and write operations.  These requests will
receive one of the status values:

 - ERR_LOCKED:  		if a null StateID (all zeroes)
				was specified.

 - ERR_STALE_STATEID:	if a non-null StateID was
				specified.

It is only when clients either try to acquire a lock with LOCK
or inquire about a lock with LOCKT that they see it is broken 
via the status ERR_BROKEN.

To release a broken PERSIST lock, a client must issue a LOCK
request with the REPAIR option.  The client would then
presumably perform some recovery actions and then could release
the lock with LOCKU.

A lock is specified to be PERSIST by setting the PERSIST option
on the LOCK request.

Client Failures
---------------
Client failures are seen by the server in two ways: as lease
expirations and as new client verifiers indicating a client
reboot.

The effect of client failures on PERSIST locks is to put them
in a breakable state.  When the lock is in a breakable state,
the client still has an opportunity to renew its lease if it
does so before any conflicting LOCK request is serviced. 
This behavior is designed to allow the client to continue
following a network partition and recovery.  If a conflicting
LOCK request comes in before the lease is renewed, the lock
becomes broken.  The lock request causing the break as well as
any subsequent conflicting lock requests will receive the
status ERR_BROKEN.  Repair of the lock requires a client to
request the lock with the REPAIR option and to then release
it.

Server Failures
---------------
Clients can detect server failures through the error
STALE_STATEID.  After receiving this error, the client must 
establish a new Stateid and reestablish its lock state.

Immediately following a server restart, the server enters a
grace period equal in length to the lease period.  During
this time read, write, and lock requests other than Reclaim
lock requests return the error ERR_GRACE, unless the server
can determine that all valid locks for the file  have
already been reclaimed.  Locking behavior during the grace
period is the same for all locks regardless of whether they 
are normal or PERSIST locks.

During the grace period, clients can reclaim locks (LOCK with
reclaim option).  After the grace period expires, non-PERSIST
locks that were not reclaimed are made available to all
clients.  PERSIST locks that are not reclaimed during the
grace period enter the state of being breakable.  [Note that
after a server failure there is really no point in going to
the breakable and  then the broken state.  We could move it
immediately to broken.]  The lock remains in a breakable 
state until the first conflicting lock request arrives at the
server.  At that time the lock becomes broken.  The lock
remains broken until a client requests it with the REPAIR
option and subsequently releases the lock.

If a PERSIST lock becomes breakable at the end of the grace
period, the client can still reclaim it, as long as the server
is sure that no intervening conflicting lock has been granted
(that is, the lock was not repaired and then made breakable
because of a different PERSIST lock).

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 15 Mar 2001 22:34:07 -0800

Mark Wittle writes:
> network partitions.

This implies a mechanism on the server that allows the
persistent lock state to be stored in stable storage.

There are apps already that use lock files to maintain
persistent locking state.  They can do that now already
with NFS.  Lock file performance can be poor, because
a file create has to be synchronous with the disk, but
that need not be so if the server uses NVRAM.  NVRAM
is particularly effective at cancelling out creates
of very short lived files.

So, I'm wondering if a lot of the persistent lock semantics
could be handled via lock files. Rather than implement
detailed persistent lock semantics, perhaps we could just
make it easier to manipulate lock files.

	Brent