Subject: [nfsv4] High-availability filesystem switching and the NFSERR_FHEXPIRED error

From: Steven Huntington <[email-masked]>
Date: Sun, 17 Jul 2011 19:33:44 -0700

Hi all,

This is a question about NFSERR_FHEXPIRED handing and while not strictly
about the NFSv4 protocol, is an effect of the protocol.

Consider a hypothetical customer that is using a high-availability (HA)
framework to switch between systems for maintenance mode work, but wants
the served data to not be interrupted.  This customer has two filesystems,
which are managed copies of each other, but separate --  including having
different filesystem id (fsid) values.  Control of which of these
filesystems is served is controlled by a symlink in the name space that is
contained within the mount path, that points to one filesystem on one side
of the HA environment and the other filesystem on the other, while the
mount path itself does not change.  Both filesystems are available to both
HA partitions but the "old" one is not to be used by the active environment
on the "new" side as the "old" filesystem will be undergoing maintenance.

When the HA switch is done, the filehandles will encounter
NFSERR_FHEXPIRED, as they should, and as a result of the FSID's changing,
the NFS client is complaining, as it should.  My question is this:

Barring a "well, you can't do that", is there a possibility of having an
industry-standard option on the client implementations, perhaps on a mount
event, to tell the client that for THIS PARTICULAR mount, tolerate an fsid
change on mount re-establishment, because the customer administrator really
really REALLY knows what he's doing?  For this situation, the customer can
use NFSv3 because a server can do its own symlink resolution on the mount
protocol RPC and would eliminate the client renavigating and verifying the
mount node, but in NFSv4 the resolution is moved to the client and the
client also then verifies the fid/fsid of the mount node per the protocol.
The (increasingly less hypothetical) customer wants to use NFSv4, using
this sophisticated HA management software to flip between systems.  What I
am looking for is some sort of weight to the idea that perhaps a client
IMPLEMENTATION would do well to provide an option (on the mount command is
my recommendation) to tolerate FSID change on the FHEXPIRED recovery when
the option is present.

Cheers,

Steve Huntington, IBM San Jose

Phone:       (408) 463-2716 (T/L 543-2716)
Fax:         (408) 463-4727
Office:      SVL 90-D215
Internet:    [email-masked]
Lotus Notes: Steven Huntington/San Jose/IBM@IBMUS

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 18 Jul 2011 05:38:27 -0400

Steve,

Can you get close to what you want with the fs_location_info attributes
in NFSv4?  They address many of the same kinds of considerations, seems
to me.

                                Craig

From: Steven Huntington [mailto:[email-masked]] 
Sent: Sunday, July 17, 2011 10:34 PM
To: [email-masked]
Subject: [nfsv4] High-availability filesystem switching and
theNFSERR_FHEXPIRED error

Hi all,

Consider a hypothetical customer that is using a high-availability (HA)
framework to switch between systems for maintenance mode work, but wants
the served data to not be interrupted. This customer has two
filesystems, which are managed copies of each other, but separate --
including having different filesystem id (fsid) values. Control of which
of these filesystems is served is controlled by a symlink in the name
space that is contained within the mount path, that points to one
filesystem on one side of the HA environment and the other filesystem on
the other, while the mount path itself does not change. Both filesystems
are available to both HA partitions but the "old" one is not to be used
by the active environment on the "new" side as the "old" filesystem will
be undergoing maintenance.

When the HA switch is done, the filehandles will encounter
NFSERR_FHEXPIRED, as they should, and as a result of the FSID's
changing, the NFS client is complaining, as it should. My question is
this:

Barring a "well, you can't do that", is there a possibility of having an
industry-standard option on the client implementations, perhaps on a
mount event, to tell the client that for THIS PARTICULAR mount, tolerate
an fsid change on mount re-establishment, because the customer
administrator really really REALLY knows what he's doing? For this
situation, the customer can use NFSv3 because a server can do its own
symlink resolution on the mount protocol RPC and would eliminate the
client renavigating and verifying the mount node, but in NFSv4 the
resolution is moved to the client and the client also then verifies the
fid/fsid of the mount node per the protocol. The (increasingly less
hypothetical) customer wants to use NFSv4, using this sophisticated HA
management software to flip between systems. What I am looking for is
some sort of weight to the idea that perhaps a client IMPLEMENTATION
would do well to provide an option (on the mount command is my
recommendation) to tolerate FSID change on the FHEXPIRED recovery when
the option is present.

Cheers,

---

From: Chuck Lever <[email-masked]>
Date: Mon, 18 Jul 2011 21:20:37 +0100

On Jul 18, 2011, at 3:33 AM, Steven Huntington wrote:

> Hi all,
> 
> This is a question about NFSERR_FHEXPIRED handing and while not strictly about the NFSv4 protocol, is an effect of the protocol.
> 
> Consider a hypothetical customer that is using a high-availability (HA) framework to switch between systems for maintenance mode work, but wants the served data to not be interrupted. This customer has two filesystems, which are managed copies of each other, but separate -- including having different filesystem id (fsid) values. Control of which of these filesystems is served is controlled by a symlink in the name space that is contained within the mount path, that points to one filesystem on one side of the HA environment and the other filesystem on the other, while the mount path itself does not change. Both filesystems are available to both HA partitions but the "old" one is not to be used by the active environment on the "new" side as the "old" filesystem will be undergoing maintenance.
> 
> When the HA switch is done, the filehandles will encounter NFSERR_FHEXPIRED, as they should, and as a result of the FSID's changing, the NFS client is complaining, as it should. My question is this:
> 
> Barring a "well, you can't do that", is there a possibility of having an industry-standard option on the client implementations, perhaps on a mount event, to tell the client that for THIS PARTICULAR mount, tolerate an fsid change on mount re-establishment, because the customer administrator really really REALLY knows what he's doing? For this situation, the customer can use NFSv3 because a server can do its own symlink resolution on the mount protocol RPC and would eliminate the client renavigating and verifying the mount node, but in NFSv4 the resolution is moved to the client and the client also then verifies the fid/fsid of the mount node per the protocol. The (increasingly less hypothetical) customer wants to use NFSv4, using this sophisticated HA management software to flip between systems. What I am looking for is some sort of weight to the idea that perhaps a client IMPLEMENTATION would do well to provide an option (on the mount command is my recommendation) to tolerate FSID change on the FHEXPIRED recovery when the option is present.

FSID changes are problematic for clients.  The "industry standard option" to do high-availabilty is described in the sections of RFC 5661 that discuss file system migration, replication, and referrals.

Oversimplifying a little, for a given dataset, the server advertises multiple dataset locations to clients via the FS_LOCATIONS or FS_LOCATIONS_INFO attribute.  The client can then choose one of these locations when the one it is using stops working.  Caveat: no-one has implemented clients that do this yet that I know of.  But ideally this happens entirely transparently to applications and administrators -- no mount option needed.

You might also want to look at RFC 5716 which introduces FedFS, a mechanism for scalably managing this dataset location info across multiple servers.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Robert Thurlow <[email-masked]>
Date: Wed, 20 Jul 2011 10:28:34 -0600

Steven Huntington wrote:
> changing, the NFS client is complaining, as it should. My question is this:
> 

Hi Steve,

A problem you have is that you, as designer/admin, "know" that
only the FSID is changing, but the filehandle is opaque to the
client by definition.  All it knows is that the filehandle has
changed.  The typical response to this is to use saved pathnames
to "remap" the filehandles it has cached.  And that means you
have to hope that nobody has done a "mv file file.old" when the
client wasn't looking, or you will unknowingly get the wrong
inode.  There are other failure modes as well.  If you could
describe how the bits in the filehandle could change without
a remap operation, you might be able to do something smarter,
but that seems a little hard to justify putting into the
protocol.

Many HA setups share storage so that the filehandles can be
kept consistent to the client.  Some have also abstracted the
storage filesystems to permit filehandles to be exactly
duplicated, with inodes and FSIDs preserved.  If you have
any control over your server source and your only problem is
your FSID, that seems pretty tractable.  If you can't match
inode numbers either, that's harder.

I agree with folks pointing to fs_locations_info in NFSv4.1,
it gives the client visibility into the storage details to
let it make better decisions here.  It doesn't solve all of
the problems, especially with pathname-based filehandle remaps.

Rob T