Subject: [nfsv4] [3530bis] LEASE_MOVED

From: Thomas Haynes <[email-masked]>
Date: Mon, 28 Feb 2011 12:30:56 -0600

I'm looking at what we captured for LEASE_MOVED:

> •First server should return LEASE_MOVED on
> –RENEW, OPEN, READ, WRITE, SETATTR (stateid carrying operations)
> –Until THE client has probed all filesystems (key phrase)
> –More than one filesystem has moved (potentially)
> –Server MUST track this for each client
> –NB: 4.1 sessions provides for client identity mapping…
> •Client query/probe of each filesystem accessed to find which has moved
> •No identifiers readily available (e.g. IP address)
> •Suggestion: PUTFH+GETATTR+RENEW
> –RENEW contains the clientid and thus the server can respond appropriately
> –GETATTR is being used to capture the content of the fs_locations value for that filesystem

I'm reading this as:

If a client receives a LEASE_MOVED error from a server, then it MUST probe
all filesystems.

If a server sends LEASE_MOVED to a client, then it MUST continue to send
LEASE_MOVED until the client probes all filesystems. This implies that
if a migration event finishes right after the LEASE_MOVED error, then the
server will track the client and continue to send LEASE_MOVED until
all probes are done. Further, if another client tries to access the migrated
filesystem, it will not see LEASE_MOVED.

This sounds overly complex, what am I missing here?

(Need to account for a reboot in each case.)

The second issue was that:

> •Client may not handle LEASE_MOVED errors at all and will not probe for filesystem relocation
> •Client will then be stuck with the error returns until the server “forgets” the migration event
> •No direct solution; what can be done to mitigate?
> –Server configuration (export option) to identify clients that should not receive this error (?)
> •Capture the fact that clients may not deal with this.
> •NB: error on RENEW renews the lease for the client (in this context)
> •Client MUST(?) do the PUTFH+GETATTR+RENEW processing regardless of full support for migration

I'm reading this as:

If a client receives a LEASE_MOVED error from a server and it does not
support migration, then it MUST still probe all filesystems.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 28 Feb 2011 13:47:24 -0500

On Mon, 2011-02-28 at 12:30 -0600, Thomas Haynes wrote: 
> all probes are done.

Not all filesystems. It is limited to those filesystems on which the
client holds state.

IIRC, the algorithm proposed by Dave Noveck was that the server holds
one bit for each client that is holding state on the migrated
filesystem. As a client iterates through the filesystems, it will
eventually hit the migrated fs, at which point the server clears the bit
associated with that client. At that point, that particular client stops
receiving NFS4ERR_LEASE_MOVED.

> Further, if another client tries to access the migrated

If the second client did not hold any state on that filesystem, it will
see NFS4ERR_MOVED the next time it tries to access it.

> support migration, then it MUST still probe all filesystems.

Yes, but given that no clients today do, servers SHOULD time out after a
while (and stop sending LEASE_MOVED), in order to allow applications to
resume working on those filesystems that were not migrated.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 28 Feb 2011 13:55:08 -0600

> receiving NFS4ERR_LEASE_MOVED.

So if a client detects that the server is no longer sending NFS4ERR_LEASE_MOVED,
then it can infer that it can stop probing the remaining filesystems.

> resume working on those filesystems that were not migrated.

But as the server has no clue that the client doesn't support NFS4ERR_LEASE_MOVED,
it has to take the same algorithm for one that does support it. Which means we should
restate the behavior for both cases as:

If a client gets a NFS4ERR_LEASE_MOVED from the server, then it MUST probe all
filesystems for which it holds state. It may stop when the server stops replying with
NFS4ERR_LEASE_MOVED.

If the server sends NFS4ERR_LEASE_MOVED to a client, then it MUST continue
to send NFS4ERR_LEASE_MOVED until the client probes for all migrated filesystems
for which that client has state. As the client may not support probing, the server must
be ready to timeout for that client.

If we can state that the PUTFH+GETATTR+RENEW sequence alone in a compound is
the probe, then the server could detect non-compliant clients. However, it might be better
for the server to always timeout. I.e., if the number of suspect filesystems is large, the
timeount might be more timely than the full probe.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 28 Feb 2011 15:37:48 -0500

> then it can infer that it can stop probing the remaining filesystems.

That is my understanding of the spec. 

> be ready to timeout for that client.

That wording seems ambiguous to me. You can't both mandate that the
server must wait for the client to probe and at the same time state that
the client may not probe...

How about something along the lines of:

Upon receiving the NFS4ERR_LEASE_MOVED error, a client that supports
filesystem migration MUST probe all filesystems on which it holds open
state. Once it has successfully probed all those filesystems which are
migrated, the server MUST resume normal handling of stateful requests
from that client.

In order to support legacy clients that do not handle the
NFS4ERR_LEASE_MOVED error correctly, the server SHOULD time out after a
wait of at least two lease periods, at which time it will resume normal
handling of stateful requests from all clients. If a client attempts to
access the migrated files, the server MUST reply NFS4ERR_MOVED.

> timeount might be more timely than the full probe.

A timeout is always more robust, however I think a "SHOULD" is more
appropriate here (see above text proposal), since the existing protocol
wording will work fine with non-legacy clients.

I'm not sure that we should mandate that the client must perform the
PUTFH+GETATTR+RENEW. Doing OPEN, OPEN_CONFIRM, OPEN_DOWNGRADE, CLOSE,
LOCK, LOCKU, READ (w/stateid), WRITE (w/stateid), SETATTR (w/stateid)
will all communicate to the server that the client has probed the
filesystem, and communicate to the client that the filesystem has moved
(since the server will return NFS4ERR_MOVED).

Note that if the client gets an NFS4ERR_MOVED, it should probably
continue to probe other filesystems, otherwise it may miss the fact that
more than 1 filesystem was migrated if that is the case.

Cheers
  Trond

NetApp