Subject: RE: NFS Vendor's Conference / Interest in replication and migrati on

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 7 Dec 2001 13:45:19 -0800

My concern is that if we create a general protocol for this, 
we not forget the distinctive requirements of NFSv4 migration.  
In particular, migration in v4 is designed to be transparent to
a client application which remains running across the migration. 
A client may have open files which need to be re-established on 
the migration target with no more disruption than they would be 
in the event of a server reboot.  There are a couple of issues 
that arise from this.

The first is that the replication/sync protocol cannot be 
purely filename-oriented, even though this might work for
other applications.  There needs to be some way to allow
the open files to be re-established.  Right now, in v4,
the model is that the handles have to be exactly the same
on both sides, which is conceivable in the I-have-my-own-
protocol-in-which-I-control-the-horizontal-and-the-vertical
world but probably not otherwise.

Of course you could have handles are volatile on migration
but then you have the slight detail that you might wind up 
writing to the wrong file. 

There may be ways around this within the context of the current
v4 but I think we should consider the possibility that we 
might make some small changes to the V4 migration model 
(in a minor version, not holding up the the current v4) to 
allow a public replication protocol to function effectively.

The other issue I see is that NFS-v4 in many cases allows a
the server implementation a great deal of choice in the semantics
actually provided to the clients.  This poses special difficulties
for an application which might see significant semantic change
while it has a file system mounted and even while it has a file
open.  I think we need some way for the client to insure,
if he wants to, that things will not change in disconcerting
ways.  One way to do this is for a replication protcol to 
provide ways of representing his particular file system semantics
and the recipient system either emulating those or declaring
his inability to do so, leaving it up to the client to determine
whether he can live with semantic change while mounted.   

So the conclusion I draw from this is that if we want something
that will be useful to support NFS-v4 migration, we need to
do it within the V4 working group.  This doesn't mean that the
resulting protocol will be useful only for NFS-v4 migration, but
I think that if the focus were on a general replication protocol,
and it was done in another forum, you might wind up with something 
that would be useful for everything except NFS-v4, which certainly
wouldn't be my choice.

-----Original Message-----
From: Brent Callaghan [mailto:[email-masked]]
Sent: Tuesday, December 04, 2001 2:51 PM
To: [email-masked]
Subject: Re: NFS Vendor's Conference / Interest in replication and
migration

Robert Thurlow wrote:
> I want to find out how this response lines up with that of this
> working group, and have received some time on the agenda at the
> NFSv4 session at the coming IETF.  I think the time has come to
> hash out if we should do this work in the NFSv4 working group or
> in some other way (e.g. a separate IETF WG or outside the IETF),
> and to find out who is interested.  Please post your opinions here
> or bring them to Salt Lake City.

I'm a big fan of migration and replication. I use it every day to
sync web pages outside the Sun firewall to a web server. The
NFSv4 mailing list archive (incl this message) is propagated this
way ( http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive ).
The files comprising the Hypermail archive are sync'ed with a
lovely utility called "rsync"  (see http://www.rsync.org )

Rsync is fast and efficient.  It synchronizes just the files that
have changed, and uses a clever rolling hash algorithm to compute
the chunks of a file that have changed (good when syncing large
tar or zip archives).

Although rsync is a great tool for me, it just whets my appetite
for this level of filesystem synchronization and file movement.
It would be great if there were a tool that would allow NFS
servers (not just v4 ones) to relocate and synchronize files
amongst themselves.  Unfortunately, I can't recommend rsync
unreservedly because:

	- The protocol is not published.  It exists only within the
	  implementation of rsync.

	- Current implementations are all UNIX or LINUX based.
	  There appears to be no attempt at adapting to Windows
	  or MacOS.

	- It doesn't scale.  It's great for sync'ing small
	  collections of files, but for large filesystems
	  with a million files or so, most of the sync 
	  time would be taken up just walking the filesystem
	  to find changes.

My guess is that an industrial strength filesystem sync'er
that fixed these problems would be quite popular.  The big
advantage over block-level protocols is that you can sync
between different filesystem types, e.g. ext2fs and WAFL.
It would be great not *having* to know what kind of filesystems
they are - just as with NFS you can access files without knowing
the filesystem.

My hope is that we can come up with a protocol that will spawn
a variety of implementations: from small rsync-style apps, to
heavy-lifter, industrial file syncers that handle millions of
files at a time.

The IETF would be a great venue for this protocol work - and
currently the NFSv4 WG seems like a good WG to base it.

A thumbs up from me - FWIW.

	Brent

---

From: Brent Callaghan <[email-masked]>
Date: Fri, 07 Dec 2001 15:49:44 -0800

"Noveck, Dave" wrote:
> in the event of a server reboot.

I like that model.  It certainly makes it a lot easier to
re-establish state - rather than figure out some way to
standardize the representation of client state for transmission
from one filesystem to another.

> allow a public replication protocol to function effectively.

Yup.  I think volatile filehandles could work quite well
if there's some way to track renames during the migration
process and propagate those too.

> whether he can live with semantic change while mounted.

Yes, resolving filesystem differences will be challenging.

I'm optimistic about this though; applications expectations
of filesystems are not very high (I can run lots of UNIX
apps against my Java NFS server which tells horrible lies
about file attributes).  We focus a lot on things like
ACL models and extended attributes, but these features are
rarely used directly by apps.

What we have in our favor is a lot of commonality between
filesystems.  If you're going to write a filesystem then
you start with an assumption of support for existing
semantics.  So, even if we lose an ACL or two along the
way during a migration, I don't think it'll affect apps
that much.  I'm assuming here that sysadmins will be smart
enough not to migrate an NTFS to a FAT or a UFS to s5FS ..

> that would be useful for everything except NFS-v4, which certainly

Yep, that's my hope too.  Though it would be good if the protocol
is sufficiently generic that it can get the interest and participation
of the WebDAV and HTTP folks.

	Brent

---

From: Erez Zadok <[email-masked]>
Date: Sat, 8 Dec 2001 13:59:32 -0500 (EST)

In message <[email-masked]>, Brent Callaghan writes:
[...]
> enough not to migrate an NTFS to a FAT or a UFS to s5FS ..

Smart or not, users will _want_ to migrate b/t heterogeneous servers.  They
will complain (correctly or not) "why can't you migrate nfsv4 to
heterogeneous servers but nfsv4 works b/t heterogeneous clients and
servers."

I think if we're going to tackle heterogeneous replication, we should do
everything possible to avoid loss of information.  A half-working protocol
is worse than no protocol at all.  I'm concerned about permanently losing
attributes once you start migrating and replicating from A to B, B to C,
etc.  You'll wind up with a final replicate that has the lowest common
denominator features (8.3 all-cap :-)

Suppose server A migrates to B, and B to C.  A and C support ACLs, B
doesn't.  One idea I'd suggest is that all attributes that cannot be
migrated be written into some sort of a machine-readable format file
("extras") on the target f/s of the migration (B).  That file can be passed
on to another server (C), its contents read, and that new server can
use/apply the features that are listed in that file.  Server C can then
establish an identical replica to A.

This kind of technique can help us avoid losing data.  In the above example,
it'd be rather annoying if you lost all ACL info b/c you migrated A->B and
then B->A back (maybe A was down for a period of time while emergency
maintenance is applied to it).

What goes into this "extras" file will be an interesting question.  And, how
do you represent essentially arbitrary file attributes, how much space would
it take, etc.?  Assuming the replication protocol will be based on NFSv4,
the simplest idea I have at this time is simply to record a log of all NFS
calls and their attributes which the migration target server could not
support.  The log can even contain high-level XDR representations, making it
easier to replay this log when it's passed to a target migration server.

One issue might be the name of such a special file.  Maybe we can [ab]use
the already special sequences of .nfsXXXX ?

Erez.

---

From: Brent Callaghan <[email-masked]>
Date: Sat, 08 Dec 2001 19:01:13 -0800

> establish an identical replica to A.

We already have a similar issue with NFSv4 file access.  The protocol
provides a generic filesystem view that includes some features that
cannot be supported on some filesystems.  ACLs spring to mind.  The
protocol allows the server just to return an error if the client attempts
to access an unsupported feature (and the client can query if a feature
is supported).  The server has the option of attempting to simulate the
attribute, though my guess is that few implementations will go this far.
For instance, it's one thing to store the data in an ACL - quite another
thing to reproduce the semantics. A lot of attributes have associated
semantics, e.g. quota information could be stored, but it's hard to
enforce it.

If you have a metadata file tagging along with a data file on each
migration, then you could have a motely collection of inconsistent
attributes after several migrations.

My inclination is to keep the model simple and warn the user/admin
of potential metadata loss on migration. In practice, this won't have
much impact if the filesystems are reasonably similar - for instance,
if migrating from an ext2fs to a reiserfs or even ufs to ntfs.

Keep in mind that we already have archive formats like tar and zip
that seem to do a reasonable job at moving data between dissimilar
filesystems with good results.  There's metadata loss there too, but
it seems that our expectations of metadata preservation are adjusted
accordingly when the dissimilarity is high.

	Brent

---

From: Robert Thurlow <[email-masked]>
Date: Mon, 10 Dec 2001 10:24:07 -0700 (MST)

> a client application which remains running across the migration. 

I agree, Dave.  If we create a replication-migration protocol
general enough to work with other file sharing protocols and
applications, I think it is still very important to have enough
"knowledge" of NFSv4 to permit a client to present correct
behaviour to the client.  I'm hoping that with a fairly generic
core protocol with appropriate collections of attributes, we can
bring in the detail needed to do this without impeding other
uses of the protocol.

> do it within the V4 working group.

I would strongly prefer this, but I suppose if we could engage
the right people and make NFSv4 support a requirement, we might
be able to do the right thing.  But the farther we get from the
NFSv4 WG, the harder this would be.

Rob T

---

From: Brian Pawlowski <[email-masked]>
Date: Mon, 10 Dec 2001 09:38:47 -0800 (PST)

Our area directors suggested we check with the SLP working group - and
the meeting we attended the chairs said that the working group
SPECIFICALLY was not pursuing data movement/location - just services.

I don't think there is a drop place for migration/replication back
end.

The only remote possible overlap is that NDMP is running a BOF this
IETF meeting to see if they can form a working group.  It has been
bandied about a bit that NDMP may form basis of one form of a
migration/replication protocol in the future?

---

From: Robert Thurlow <[email-masked]>
Date: Mon, 10 Dec 2001 10:42:21 -0700 (MST)

> heterogeneous servers but nfsv4 works b/t heterogeneous clients and

People will do this, but I don't think we need to do heroics to
protect them as much as advise them of consequences.  You should
never lose data by surprise, but a lot of the time you may not
really care whether you do or not, or know your dataset doesn't
depend on something like named attributes.  You should always be
informed of issues like this when setting up a migration or
replication, and given the chance to bail or make another choice.

BTW, you postulated an A->B and a B->A migration because host A
might be down for extended repairs.  Surely in this scenario, you
would create a replica and promote it to read-write, and then
synchronize with the original data when it was back online?  That
way, you would lose the extra stuff only temporarily.

Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Mon, 10 Dec 2001 10:51:25 -0700 (MST)

> migration/replication protocol in the future?

NDMP has some nice attributes, but as currently specified, it
has a couple of fatal flaws in a replication/migration situation:

  - no real security, especially in the bulk data transfer traffic,
    but also in integrity of commands after initial session setup

  - no interoperability as we would recognize it, because the bulk
    data transfer is in whatever format the host owning the original
    copy finds convenient to use

The three-way transfer is a lot like what I think we want to do
for control or replication and migration, but you can tell how
much this is tuned to tape backup and restore.

Rob T

---

From: Brian Pawlowski <[email-masked]>
Date: Mon, 10 Dec 2001 09:57:49 -0800 (PST)

The NDMP BOF is to form a WG to look at evolution of NDMP.

That is why I proposed it as a remote possibility with some juncture in
future.

Until then, as far as I can tell, V4 WG must plod along (in conjunction
with rechartering).