Subject: [nfsv4] Fallout from the global namespaces review: state recovery...

From: Trond Myklebust <[email-masked]>
Date: Wed, 25 Jul 2007 09:30:24 -0400

As seen from the minutes of the review, there was some concern about the
recovery semantics suggested in the paragraph starting at line 11578:

        In place of a file-system-specific version of RECLAIM_COMPLETE,
        servers may assume that an attempt to obtain a new lock, other
        than be reclaim, indicate the end of the client's attempt to
        reclaim locks for that file system.  [NOTE: The alternative
        would be to adapt RECLAIM_COMPLETE to this task].

For one thing, this requires clients to keep the filesystem entirely
quiescent while doing recovery.

More worrisome is perhaps the fact that it relies on the existence of
some process running on the client that requires a new lock in order to
complete the state recovery. There is no suggestion for what the client
should do if it only wants to recover existing state, and doesn't need
to open a new file or take a lock.

In order to address issues like this, I'd therefore like to propose an
alternative to the current state recovery model. Rather than extend
RECLAIM_COMPLETE to take an fsid argument etc..., I'd like to propose a
model that extends the sessions model. I believe this has several
advantages over the existing idiom.

---------

The proposal is to add a new csa_flag CREATE_SESSION4_FLAG_RECOVER that
indicates to the server that the session being created will be used for
state recovery. If none of the exported filesystems are in grace at the
time, then the server should return an error NFS4ERR_NO_GRACE.
If the client MUST not set CREATE_SESSION4_FLAG_CONN_BACK_CHAN in
conjunction with CREATE_SESSION4_FLAG_RECOVER (return NFS4ERR_INVAL). 

Once the recovery session has been established, then all state recovery
operations MUST be sent using that session. It is an error for the
client to issue non-recovery OPEN, LOCK, WANT_DELEGATION operations on
the recovery session, and a server should return NFS4ERR_OP_ILLEGAL in
this case. 

While state recovery is going on, servers MAY choose to return
NFS4ERR_DELAY on all requests issued on non-recovery sessions in order
to optimise performance for state recovery, or if it does not satisfy
the requirements to safely process lock, READ or WRITE requests during
the grace period.

Once the client is done recovering state, it indicates this to the
server by issuing a DESTROY_SESSION call for the recovery session. The
server may also choose to terminate the grace period early by
invalidating the session (although it SHOULD NOT do so before at least
one lease period has expired in order to give clients the opportunity to
discover the reboot).

------------

This proposal would also obsolete the RECLAIM_COMPLETE operation, since
the DESTROY_SESSION operation takes over its role.

Cheers
  Trond

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 25 Jul 2007 09:37:32 -0400

> one lease period has expired in order to give clients the opportunity to

I forgot to mention. One other of the advantages of using a special
session when it comes to migration, is that in addition to telling the
server when recovery has ended it also gives it an idea of whether or
not the client has _started_ recovery.

Unlike the traditional reboot recovery process, when migrating from one
server to another, it may turn out that the client already holds state
on the new server. If so, then looking out for an EXCHANGE_ID or
traditional SESSION_CREATE does not suffice in order to figure out if
the client wants to recover state. Looking out for the creation of a
recovery session does...

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 25 Jul 2007 10:36:18 -0400

One issue that concerns me is the need to try to clamp down on changes
to the minimum necessary to address problems found in the current spec.
In other words, we have reached the point where simply making the
protocol better, as opposed to workable or correct, is a sufficient
reason to make a change.

I think Tronds's proposal is kind of neat, but I think that by creating
kinds of sessions, devoted to particular purposes, it probably means
significant change to some of the text of the sessions chapter, or at
least another careful review, to make sure that we don't need any.  This
is a substantial conceptual change, even though I think it is a
direction that might have been worth exploring if the v4.1 clock were
not so far along. 

So Trond raises the issue of a client with no locks to get beyond those,
if any, that it has reclaimed.  I agree this is a problem, but I don't
think his change really addresses it.  It has the same sort of problem
that RECLAIM_COMPLETE does.  As he indicates, DESTROY_SESSION, for the
appropriate type of session, is the equivalent of RECLAIM_COMPLETE.
Given that it has the same issue of delineating the scope of recovery
completion with regard to transferred fs's.  A number of fs's can be
transferred at approximately the same time so it not clear that when you
create such a session, how you can ensure that the client and server
have the same view of the set of fs's for which recovery will be done by
a particular session, and thus whose recovery phase will be terminated
by the associated DESTROY_SESSION.

At the review meeting, I attempted to hand-wave away the need for an
fs-specific recovery-complete mechanism, by arguing that if source and
destination fs's were sufficiently co-ordinated for the set of clients
with locks to be known by the destination, then it isn't much harder for
the source to tell the destination all of the locks, in which case
recovery-complete is not necessary since you can mark off the locks as
they are actually recovered.

The problem with that argument is that it is valid in the case of a
planned transfer but perhaps not in the case of an unplanned transfer.
Clearly it is much easier for the source to notify the destination of
each new client that has locks than it is to update the destination so
its image of the client lock corpus is precisely in sync with that of
the source.  On the other hand, transfers of writable filesystem (and
read-only ones are trivial with regard to recovery-complete issues),
sufficient to meet the semantic requirements for those to work
effectively, would need extensive interaction between source and
destination, in any case.  At that point we have to deal with the fact
that exact semantics for transfers of writable file systems is still not
fully settled.

So I think our options here are:

     Explain why this case is unlikely be important, and argue 
     that the worst thing that the absence of any specific provision
     for it will result in a normal-sized grace period in a
     case that is unlikely to happen very much.

     Strengthen the kludge suggested in the text by making it clear
     that any attempt to get a lock via non-reclaim means, even 
     if not well-formed (e.g. share mode of none, or zero-length
     string as the name) will be recognized as ending the 
     reclaim for the client if done with the current-fh in the
     fs.  This would allow a client to do something to delimit
     the recovery phase even if he didn't want to get any additional
     locks.  

     Add a RECLAIM_COMPLETE_FS which takes an fsid and tell the
     client to use it.

I think the idea of building on the session model is interesting and
probably will be important as v4.x goes forward (someday), but it just
seems too late for 4.1.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Wednesday, July 25, 2007 9:30 AM
To: [email-masked]
Subject: [nfsv4] Fallout from the global namespaces review: state
recovery...

---------

------------

Cheers
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 25 Jul 2007 14:25:44 -0400

> by the associated DESTROY_SESSION.

Yes. If the client is recovering state on several filesystems at the
same time, then it does indeed mean that you have to keep all fses in
grace until the last DESTROY_SESSION has been called. Maybe that is a
reason to keep RECLAIM_COMPLETE_FSID?

If you do not have the session, however, how exactly do you know that
the client actually wants to recover state on one of the migrated
filesystems? There may be several replicas on several servers, and the
client may have chosen to migrate to one server for one of the
filesystems, and another server for the other filesystem.

Are you suggesting that clients must run through the entire list of
replicas and call RECLAIM_COMPLETE_FSID on all servers?

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 25 Jul 2007 17:47:29 -0400

> replicas and call RECLAIM_COMPLETE_FSID on all servers?

No.

Let's divide the cases up.  First we have the most common case in which
there is likely to be multiple replicas, where they are read-only.  In
that case it has always been assumed that locks don't really matter.
Nobody can change anything, so the utility of locks for locking out
other people changing things isn't there.  They are locked out from
changing things inherently.  Maybe we need to make this more explicit,
but I don't there is a substantive issue to resolve.

The second case where there are multiple writable and replicas
fs_locations_info indicates that the relationship among these is that
they are all in the simultaneous-use class.  In that case, getting  a
lock on one replica is the equivalent of getting it on another, whether
because the various replica descriptions are alternate paths to the same
thing or they are paths to different things which are kept in sync via
some means (cluster-fs code etc.).  In this case, RECLAIM_COMPLETE_FS
(or whatever we do to delimit the end of the recovery period) done for
one replica should be considered as done for the whole set.  There
should be sufficient co-ordination to allow this, by the nature of the
relationship which is being advertised. 

So now we get to the last category where we have multiple writable
replicas which are declared such that a lock on is not a lock on all.
This is related to the issue Rob Thurlow raised about moving the
proposed text more to a model where we tell people what guarantees whey
can and can't expect, rather than mandating one level of support here.
However, given the fact that you giving people wrote, we have to be as
clear as possible about the effects of proceeding without certain levels
of guarantee.  In the particular case transferring use from one fs to
multiple writable fs's where these fs's are not co-ordinated as to
locking, it is clear that the applications will have to be specifically
tolerant of this situation because we are allowing, for example, an
application which has opened file with DENY=ALL, to, after the transfer,
have a corresponding (and otherwise conflicting) open done by another
client.  So in that case, I can still see you want a grace period so
that the client can reclaim his OPENs, even though the guarantee he gets
is quite reduced.  However, I just don't see any way that these
individual servers can implement a shortened grace period, because it
may not see all the clients.  So I would just say that in that
situation, the servers if they set up that situation have to accept that
they are buying into a full grace period rather than one that could be
shortened due to reclaim-completion-indications.  That doesn't mean that
the client should not indicate that it is finished reclaiming locks, but
in most cases, the value will be minimal.  If the servers are capable of
co-ordnating their reclaim-completion-indications, then they can do
this.  

> reason to keep RECLAIM_COMPLETE_FSID?

I was worried not so much about the delay until the last
DESTROY_SESSION, but the ambiguity about which fs's had or had not been
reclaims.  If I set up a session and I reclaim some locks for a
particular fsid, then once I destroy that session the server can
consider that my client's reclaims for that (or those) fsids has been
completed.  The troublesome case is if I don't have any current locks to
reclaim.  In that case, I can't see how the server can be aware of my
client's completion of the (empty) recovery phase for that particular
fs. 

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Wednesday, July 25, 2007 2:26 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: RE: [nfsv4] Fallout from the global namespaces review:
staterecovery...

Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 25 Jul 2007 18:17:46 -0400

On Wed, Jul 25, 2007 at 05:47:29PM -0400, Noveck, Dave wrote:
> other people changing things isn't there.

I didn't think "read only" meant "static".  I would have thought that
read delegations could still be useful on a read-only replica, for
example.  So "locks don't really matter" seems like an overstatement.

Uh, but I may have lost the point of the discussion here.  If you're
updaing read-only replicas using an occasional rsync, I suppose you'd
still like to know how soon after a reboot it's safe to allow the rsync
job to run.  But in that particular case probably nobody cares much if
the rsync has to be delayed by a minute longer than necessary.  I dunno.

--b.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 25 Jul 2007 18:27:03 -0400

You're right, I did exaggerate.  There are cases where locks do
matter in the case of read-only replicas and read delegations
are a good example.

With regard to the basic issue, I think we are still in the same
place.  As far as delimiting the end of the recovery-phase for
that fs, there is no real way to truncate the grace period, if
you are typically going to have a subset of the source's set
of clients as your clients, unless you have some sort of 
inter-replica co-ordination.

So I think where this leaves us is:

     We need some way of indicating the end of the reclaim-phase
     for a particular fs in these cases.  We aren't exactly sure 
     which one to choose.

     It is the server's responsibility to use that to shorten the
     grace period if it can and to ignore it, if it is of no value
     because it doesn't have the complete set of servers anyway.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Wednesday, July 25, 2007 6:18 PM
To: Noveck, Dave
Cc: Trond Myklebust; [email-masked]
Subject: Re: [nfsv4] Fallout from the global namespaces
review:staterecovery...

--b.