Subject: RE: [nfsv4] Migration questions

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 25 Oct 2004 17:02:55 -0400

Let me give you some history on the migration feature in v4.

At the time the migration feature was being thought about, 
there was active interest in developing a server-to-server
migration protocol, to allow a smooth multi-vendor transfer
of the file and directory data, and possibly state and lock
information, to allow a co-operative transfer of file service
with minimal impact on the client.

At that time, one school of opinion was that it would be best
to wait for that effort to complete and then add the migration
feature to the v4 client-server protocol, in a minor version.

But in thinking about the development and roll-out of such a
feature, the anticipated timeline looked kind of daunting, to
those thinking about such an effort as it looked like the
long pole consisted of development of the server-server protocol,
followed by development of the extensions to the v4 client-
server protocol, the implementation of clients capable of 
dealing with that, and finally the gradual and general deployment
of clients capable of using the migration feature.  In other words,
it looked like it would take forever to get something going
that way.

So the current migration approach was an attempt to get done
whatever migration support that could be done in clients and 
servers, without waiting for a server-to-server migration 
protocol, which was presumed to be on its way.  As things 
have turned out, that effort did not maintain its momementum
and ground to a halt.

So, to get back to the client-server world, there were two
anticipated possibilities.  One was a single-vendor migration
using persistent filehandles.  A server implementor could
relatively easily arrange his filehandles so they were unique
among all machines within the scope of migration.  The filesystem
could be copied and the old filehandles honored at the new server.
The idea here was that even though we'd like a muti-vendor 
solution, enabling the single-vendor solution would get us 
part of the way there, while work proceeded on the multi-vendor
approach.

Another approach is use of volatile filehandles but there are
some issues that restrict use of volatile filehandles for migration.
It doesn't seem to me that the cost of saving the file name 
information is one of them.  The first thing to note here is that
for most file handles you know about, you are caching them and so 
you do have information that would allow you to have the names for 
these file handles.  Otherwise, what is the value in caching them?
However, if you have migration and volatile  file handles, the 
simplest thing to do with cached file handles is to forget about 
them.  Caching is an optimization but when you would have to go 
to the server to re-instantiate, you might as well just wait for 
a new reference.

There are filehandles for which the client typically has name 
information and the filehandle cannot be thrown away (e.g. for
open files), but the troublesome issue is that while it has a name, 
it might not be correct.   If I open a file, I know a name 
that the file once had, but with typical Unix semantics I have
no guarantee that the name has not changed or even that the file
has any name.  So in that environment, using the name to get a new
filehandle may get you a different file.  Servers may not allow
you to change the name of an open file but this is semantics that
most clients are not expecting and might not like.  My conclusion
is that volatile file handles for migration, are useful only in
particular cicumstances.

There are certainly many applications in which they are useful, the
most common being data which read-only or read mostly.  Dealing
with archival data, the issue of name change while a file is open may
not be significant.  In general though, this is an issue, unless people
are prepared to deal with non-traditional semantics for open.

There has been some discussion of ways to address this issue.  One
way, in the context of a migration protocol, is for the servers to
form an fh translation table which the client can interrogate to ask
the migration destination server what the new file handle is that
corresponds to a particular old one.

Another way to address this is to allow the client to ask for the 
current filename of given open file.  Such a call would still be
valid after the migration event and could be used to get the name
to be looked up on the new server.  You would need the grace period
to be applied to rename and delete requests on the new server to
give the client a guarantee that the new name is still the current
one.

I think the overall situation is that the current handling of file
handles in migration is a limited one which is useful in many cases
but not complete and that some v4 extensions will be required to 
cleanly address the general case. 

-----Original Message-----
From: Chenggong Fan [mailto:[email-masked]]
Sent: Saturday, October 09, 2004 4:57 PM
To: [email-masked]
Subject: [nfsv4] Migration questions

Hi,

I have some questions related to the management of file handles after 
receiving NFS4ERR_MOVED in the migration case (not the pure referral 
case). 

If a file handle is not a volatile file handle, currently the NFSv4 
client does not remember the associated pathname with each file handle, 
correct?  If that's the case, it seems difficult to make the 
NFS4ERR_MOVED and GETATTR fs_location useful in the migration case.

Let's say the client has been accessing this filesystem for a while, and

suddenly it gets the MOVED error.  Without the associated pathname, how 
would the client be able to use fs_location information to look up the 
new file handle in the new file system?   With filehandle being an 
opaque object, the client shouldn't just replace the fsid field in the 
file handle.  It seems that for migration to work, all file handles need

to be volatile, (or in other words, client need to keep track of 
pathnames associated with all the file handles it knows)?   Is it too 
much a load on the client?

Second, what should the client do to the other file handles that have 
the same fsid?  Should it not do anything about them?  Or should it 
expire all of them?  If the first case, the access to each fh will get a

separate MOVED error, and the scenario is the same as the last 
paragraph.  If this is the case, file-level referral actually might be 
possible.  If the second case, again the client needed to remember the 
pathname to look up the new file handles in the new filesystem, or all 
file handles will be stale.

After all it looks that "pure referral" might be easier to support than 
migration redirection.  Am I missing anything?

Charles

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Chenggong Charles Fan <[email-masked]>
Date: Mon, 25 Oct 2004 17:22:19 -0700

Hi Dave,

Thanks for the detailed background introduction.  I understand from your 
explanation that using cached file names to do the lookups again after 
migration is problematic when a file name is out of date (e.g. the file 
has been renamed by another client).  It requires addional 
specifications such as the two methods you outlined to make the complete 
case work.

The motivation for standardizing the server-server migration protocol is 
very interesting.  It's a tough problem, especially if the migration is 
to be done transparently to the client, both during the migration and 
after the migration.  Rainfinity has a solution called RainStorage to do 
transparent migration between multi-vendor NFS and CIFS servers.  Let me 
give a quick overview of how it works for NFS.  (By the way, we do have 
a pending parent application for this method of migration).

Let's suppose we'd like to move a directory from filer A to filer B.  
RainStorage is a transparent proxy device that is inserted in front of A 
and B.  Clients continue to mount A or B, without knowledge of the 
existence of the RainStorage box, but RainStorage can see all the NFS 
traffic going to A or B.  RainStorage starts the migration transaction 
by copying files from A to B, and creating an internal fh translation 
table of all the files copied.  For any clients who are accessing any 
files already copied or being copied, the client request is sent to both 
A and B using the fh translation table.  The reply is only sent back to 
the client if received from both A and B.

This way the administrator does not need to shutdown user access at all 
throughout the migration, and at the end, we have a synchronous mirror 
created between A and B for the data that has been migrated, including 
all the user updates that happened during the migration.  This process 
is one way of doing the server-server migration transparently.

Now it comes the question of client cut-over.  Currently for NFS v2 and 
v3, RainStorage cuts over into redirection phase, where for clients who 
are accessing that directory on A, their NFS requests are being 
redirected to B (without their knowledge).  At the same time the global 
namespace map (nowadays, automounter map being the most popular) is 
updated to reflect the new location of the directory (assuming that 
directory is a mount point).  RainStorage can stay in redirection mode 
arbitrarily long, till all clients have auto-unmounted and 
auto-remounted to the destination B.  RainStorage typically stays in 
band for hours.  This process is how we deal with the client-server 
aspect of the migration. 

It would be nice for the client-server part to be built into the v4 
protocol, so that all RainStorage has to do is to update the NFSv4 
namespace server and file server A.  File server A can return 
appropriate error code to the clients (or simply time out), and clients 
can go back to the namespace server to look up the new location, and 
look up their file handles again with the cached file names.

In summary RainStorage is one example how a vendor solution can do the 
server-server part,  i.e. migrate data between mutli-vendor servers, 
within the boundaries of the existing protocol definition.  The 
motivation is to define client-server interactions that make the 
post-migration client cut-over quicker and more transparent.

Thanks,
Charles