Subject: Re: Replication and migration requirements

From: Robert Thurlow <[email-masked]>
Date: Thu, 16 Aug 2001 11:26:21 -0600 (MDT)

I wrote:

> They're a fairly large set, and I'd like to find out who cares
> about which parts.

Here are my thoughts on this.  Since we did NFS client failover
in Solaris 2.6, it's been an occasional issue that we didn't
provide a good tool to create replicas.  When we look at the
geographical diversity of our customers, a good way to create
replicas is important, so we have a block-based replication
tool as other vendors do.  With NFSv4, I think our customers
will want something more standard between different types of
servers, and something more useful than 'cp -r' or 'rdist'.

> o Transparent client failover
>   o For read-only replicas and migrated writable volumes

This is summarizing the holy grail.  For read-only replication,
this does not seem too hard.  For migration, things are harder.
I like the idea of trying to transfer lock and delegation state
and forcing some reboot-like recovery effort onto the client.
But I am willing to punt if need be.

> o Performance
>   o Bandwidth conservative
>   o Differences propagated
>   o Restartable
>   o Client lockout time minimized

This seems important; if the performance is not good especially
for incremental updates of replicas, we have a problem.  I've
been thinking about transaction logs and snapshots to get the
minimum set of changes, and I've also been looking at Andrew
Tridgell's rsync algorithm, which could help when a log is not
available.  This might be one of the first forks in the road
for deciding how the protocol operates.

> o Security
>   o As good as V4

This is very important; fortunately, we do have NFSv4 potentially
providing a lot of infrastructure.  I say 'potentially' because I
do not at this point assume that the wire protocol will be an RPC
based application; we might do better to stuff XDR'd data down a
TCP socket.  This choice is another fork in the road.

> o Scalable
>   o Huge file systems
>   o Small file s ystems

This is important, but also seems linked to performance to me.
I don't know how this requirement guides our design.

> o Capability negotiation between "peers"(?)
>   o I believe this referred to things like attribute differences

This does have to do with whether your filesystem will look
the same in all respects after a migration.  I think the right
way to do this is to try to inform the sys admin about potential
loss of data before the migration or replication begins, to give
them the choice of how to handle it.  If a database relies on
named attributes to operate, you must not transfer its files to
a server that does not support named attributes, but if you know
you don't care, you might proceed.

> o Efficient multi-way replication (propagation)

This is good, but if multiple connections are used for propagation
it's not obvious how this would affect the server-to-server protocol.
If we were to choose a multicast protocol instead, this would be
relevant.

> o Correctness
>   o "Atomic" propagation of file system from client view
>   o Failover to a correct replica version

As Brian's issues pointed out, we have no basis for tracking a
replica version in NFSv4.  In practice, I don't think this is
necessary.  Here at Sun, our internal network has a large-scale
replica set of binaries and support files, and uses no version
number.  If an application needs a version of a data set, the
application should create one.

> o TCP/ IP based
>   o No legacy UDP requirement

This seems obviously correct (and implies not using multicast).

> Issues
> 
> o Single vs. multi-master
>   o Multi-master entails conflict resolution
>   o Proposal: Single master copy sufficient - objections?
> o Disconnected operation irrelevant
>   o Corollary to above
>   o Proposal: Disconnected operation for clients not required - objections?

I agree that multi-master and disconnected use are out of scope.
One thing this leaves open is the possibility of having both a
read-write master and a number of read-only replicas.  AFS did
this by just having separate mount points; DCS/DFS did a better
job via the token manager.  If the namespace can inform clients
where the read-write master is, it should be possible to have
clients forget about alternate replicas while a process on the
client is modifying to a file.  This is mostly work done by the
client, and likely has no bearing on the server-to-server protocol.

> o Migration/replication only for V4
>   o Not a general (rdist) mechanism

I think this server-to-server protocol should not stand alone
from NFSv4, but clearly if you have managed to move a filesystem
coherently and correctly between servers, any other protocol
can share the data with clients.  I do believe people will use
this to move filesystems to be used by NFSv2, NFSv3 and CIFS
clients, and I think that's good.  I do not want requirements
of those protocols affecting what we design here, though.

Comments?

Rob T

---

From: Brent <[email-masked]>
Date: Thu, 16 Aug 2001 15:01:18 -0700

> But I am willing to punt if need be.

Migration of data/metadata is one thing, but transient state
like locks/delegations seems terribly fragile and complex. 
I recommend the punt.

> for deciding how the protocol operates.

Yes, rsync is good for small, informal replication efforts. But it
won't scale to truly huge filesystems with millions of files.
It may be that there is no single algorithm for determining a
list of deltas.

> TCP socket.  This choice is another fork in the road.

Yep.  I think Legato's backup protocol works this way: XDR'ed
data on a TCP stream.  As far as security goes, I think some
kind of connection-based security might be better than the
transaction-based of RPC protocols, e.g. TLS.

> I don't know how this requirement guides our design.

This might be where a split between an rsync-like algorithm 
(small file systems) and transaction-log algorithm  (large
file systems) is appropriate.

> you don't care, you might proceed.

I'd hope that this compatibility function not be overly
chatty or interactive.  It might be implemented as
a separate query command that a sysadmin can use when 
setting up a new replica.  Then the replica sync can
(perhaps via cron script) can be run with appropriate settings.

> of those protocols affecting what we design here, though.

I think we should try to keep this protocol independent of v4
and make it a general protocol that's independent of the
data access protocol.  I'd hate to see this protocol get larded
up with requirements unusual to any single file access protocol,
e.g. replication of filehandles or file open state.

	Brent

---

From: Mike Eisler <[email-masked]>
Date: Thu, 16 Aug 2001 16:26:42 -0700

> like locks/delegations seems terribly fragile and complex.
...
> e.g. replication of filehandles or file open state.

If these end up being requirements, I suspect that
it will means a replication/migration protocol for only
read-only filesets. The  client side failover
experiment is 2.6+ is read-only, and yet has been very successful.
This is because  we know from half a decade of
operational experience that locking  and preservation of metadata like
filehandles, fileids on read-only
filesets is not needed.

It is easy for me to imagine that with writeable filesets, there will
be many problems without migratable locking and metadata.

	-mre

---

From: Uresh Vahalia <[email-masked]>
Date: Thu, 16 Aug 2001 19:41:12 -0400

We should define the objectives for protocol support of replication/migration.
Do we want to:

1.  Define a standard way for a server to replicate/migrate data to another,
possibly heterogenous, server?
2.  Provide useful mechanisms for servers to send/receive file data and
attributes to each other so that server vendors can build effective
replication/migration solutions?
3.  Provide ways for clients to control what gets replicated/migrated (whole file
systems, directory trees, individual files), when, and where?
4.  Allow clients to recognize the existence of replicas or be informed that some
data has been migrated elsewhere, and to optimize their data access accordingly
(load balance among replicas, for instance)?
5.  All of the above.

Since most servers support multiple protocols (at least, multiple versions of
NFS), it seems impractical to tie this stuff too closely to replication.

Uresh

=====================

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 17 Aug 2001 09:30:53 -0600 (MDT)

> If these end up being requirements, I suspect that it will means a
> replication/migration protocol for only read-only filesets.
...
> be many problems without migratable locking and metadata.

I hear you; if we talk about NFSv4's improved stateful model
and migration support, but have to add the footnote that the
customer has to choose between these features, will migration
be considered useful, or just a source of bugs we can't fix
because of the protocol definitions?  It does seem to me that
with proper handling, migrating state should be possible.
Practical?  I don't yet know.

Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 17 Aug 2001 09:30:50 -0600 (MDT)

> (perhaps via cron script) can be run with appropriate settings.

Clearly, you only want to know this kind of information when
you initially set up the replica, not when you're propagating
changes to the same set of servers.

> e.g. replication of filehandles or file open state.

The problem with making it independant of NFSv4 is that I think
we want correct replication of ALL of the NFSv4 attributes when
possible; I expected to re-use the nfsv4.x file to be able to
express these attributes in the server-to-server protocol.  If
this is the generic effort of an separate WG, can we do this?
And if we can't do this, can we achieve correct replication and
migration?

Rob T

---

From: RJ Atkinson <[email-masked]>
Date: Fri, 17 Aug 2001 11:47:29 -0400

At 11:30 17/08/01, Robert Thurlow wrote:
>will migration be considered useful, or just a source of bugs 
>we can't fix because of the protocol definitions? 

As I've noted before, I think it is needless complexity
that's just going to cause me operational problems.
However, user input doesn't seem to carry very much weight
in this WG...

Ran
[email-masked]

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 17 Aug 2001 10:00:21 -0600 (MDT)

> We should define the objectives for protocol support of

I think we should mainly be focused on defining a protocol
which permits a server to contact another server to initiate
and complete a data transfer to create or maintain filesystem
replicas, or to move a filesystem.

> possibly heterogenous, server?

I believe this should be our priority here, with "heterogeneous" 
being the key word.  Sun has a block-level replication product
which is good, so does NetApp - but they're only capable of doing
that with the same brand of machines.  Islands of functionality
like this do not meet customer needs.

> replication/migration solutions?

I may be misunderstanding this, but I don't think we're interested
in building a "toolkit" - we want a protocol servers can speak to
transfer entire filesystems and then propagate updates to replicas.
The IETF does protocols and not APIs, so this seems necessary.

> 3.  Provide ways for clients to control what gets replicated/migrated (whole
> file systems, directory trees, individual files), when, and where?

I don't think we are standardizing how an administrator at her
desk communicates her desire to have servers achieve a migration
or replication activity, but perhaps we should be doing that.

> 4.  Allow clients to recognize the existence of replicas or be informed that
> some data has been migrated elsewhere, and to optimize their data access
> accordingly (load balance among replicas, for instance)?

Right now, we haven't been talking much about how resource info
is "published" so that clients can know about it.  It is clearly
important, and customers have BIG complaints that this is not
standardized, leaving them to manage more islands of functionality.

But replication only adds a couple of requirements to this issue
("give me multiple names" and "permit this info to be mutable"),
so I would think this is not a goal for this work.  If we wanted
to discuss doing NFS resource discovery in a standard way, I'd
think that would be a useful effort, but not tied to replication
and migration.

Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 17 Aug 2001 10:06:13 -0600 (MDT)

> that's just going to cause me operational problems.

So Ran, as a user, do you want to use replication?  Migration?
Do you favor tough restrictions (e.g. quiescence) on what can be
migrated so that that can be done safely?  Also, since we're so
far talking about a separate protocol, it's hard to see what
you're reacting to.

> However, user input doesn't seem to carry very much weight

IMO, clear and constructive commentary always carries weight.

Rob T

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 17 Aug 2001 11:15:05 -0500

> However, user input doesn't seem to carry very much weight

Ran,
  Are you saying that you don't need any replication or migration
backend protocol to support data management or that you don't want
those types of hooks/features in the NFSv4 protocol itself.

-- 

- Spencer -

---

From: Brent Welch <[email-masked]>
Date: Fri, 17 Aug 2001 09:52:37 -0700

Regarding points 1 and 2 below, one candidate for a server-to-server
protocol is NDMP, which is used to copy file systems from filers to
tape servers, or from filers to filers. 

>>>Robert Thurlow said:

--
Brent Welch
Software Architect, Panasas Inc
Pioneering Smart and Infinitely Scalable Storage Networks

www.panasas.com
[email-masked]

---

From: RJ Atkinson <[email-masked]>
Date: Fri, 17 Aug 2001 12:57:23 -0400

At 12:15 17/08/01, Spencer Shepler wrote:
>those types of hooks/features in the NFSv4 protocol itself ?

        Good question.  Myself, I don't need a replication or
migration backend protocol.  I accept that some other users
probably do want the ability to replicate/migrate stuff
between/among servers.

        So my main concern is that we don't do this inside NFSv4,
which already appears to be more complex than necessary and
already is an operational concern.  If replication (etc) were
done in a totally decoupled way so that no additional gorp 
gets added to NFSv4, that would be OK with me.

        It has not been clear to me (I couldn't be in London due
to serious illness so am a bit out of sync) that this work was 
being undertaken in such a totally decoupled manner.   Are folks
intending to undertake this as a separate protocol, decoupled
from NFSv4 ?  If yes, kindly pardon my confusion.

Regards,

Ran
[email-masked]

---

From: Brian Pawlowski <[email-masked]>
Date: Fri, 17 Aug 2001 10:12:57 -0700 (PDT)

NFS V4 as it stands includes some info to support client to server
interactions for client failover in a migration or replication
situation.

The work being proposed is a separate protocol to support
the feature defined for NFS V4.

Not adding anything new - only a server-to-server protocol to support
what is there.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 17 Aug 2001 10:26:35 -0700

> However, user input doesn't seem to carry very much weight

That AFS persists, and the users who persist it list
fileset migration as a major reason suggests to me that user
input is carrying weight.

---

From: RJ Atkinson <[email-masked]>
Date: Fri, 17 Aug 2001 13:43:04 -0400

>input is carrying weight.

Mike,

        I haven't seen that input on this mailing list or at any
meetings I've been at.  I must have been asleep that week. 
I know I'm not the only user on this list, though I do seem to
be the main user who comments on stuff (sigh; I wish other users 
on the list would comment more on the list).

        I have worked at sites that used AFS.  At the sites I am 
familiar with, the primary reasons for AFS were (in priority order):
        1) Perceived better security through Kerberos integration
        2) Global filesystem namespace so that files have the
                same absolute path no matter where one accesses 
                them from (e.g. same path from one's work desktop       
                or from one's university account)
        3) Enhanced central administrative control of things 
                because of the "AFS cell controller" design choice

Note well that those aren't my own views on AFS, just the views
of folks at the sites I've worked at that happened to use AFS
(in addition to NFS).

Cheers,

Ran
[email-masked]

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 17 Aug 2001 12:50:33 -0600 (MDT)

> meetings I've been at.  I must have been asleep that week. 

I tend to think that this is not a forum where many AFS users
would coalesce, which is too bad - I think we'd benefit from
more familiarity with it.

>                 because of the "AFS cell controller" design choice

I like AFS; I worked on a port to a new platform in a previous
position.  The security model is good, and it does things like
replication, migration and backup (at least architecturally!) very
well.  I find my thoughts very much guided by some of what AFS
does well.  I always shook my head at what they left out, though -
byte-range locking and ability to support HSM, for example.

Rob T

---

From: Brent <[email-masked]>
Date: Fri, 17 Aug 2001 12:53:45 -0700

> tape servers, or from filers to filers.

Isn't NDMP used just to coordinate an over-the-wire dump/restore
as in NDMPcopy ?

I don't see anything in the protocol that makes it a good basis
for a more general or efficient replica sync protocol.

	Brent

---

From: Brent <[email-masked]>
Date: Fri, 17 Aug 2001 13:18:42 -0700

Mike Eisler wrote:
> :
> :

I thought NFSv4 was designed so that clients wouldn't be noticeably
affected if the server crashed, lost their state, and recovered.
The clients are ultimately responsible for re-establishing state.

So, switching to another writable copy shouldn't be any worse
than a server reboot should it ?

	Brent

---

From: Brent <[email-masked]>
Date: Fri, 17 Aug 2001 13:38:55 -0700

> And if we can't do this, can we achieve correct replication and

Realistically, we won't ever have correct replication/migration
unless it's across homogeneous filesystems.  I think we'll be
happy with something that mostly works, even if it's not 100%
perfect.

As for the NFSv4 attributes.  Yes, the NFSv4 attribute set is
a good guide to the kind of attributes that a replication
protocol might support, though there are a few virtual attributes
in there that may not be appropriate, e.g. the "changed" attribute
is an opaque thingy that cannot easily be represented physically
in any filesystem.

	Brent

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 17 Aug 2001 15:49:46 -0600 (MDT)

> is an opaque thingy that cannot easily be represented physically

I think you missed my point - if the IETF work is done in a separate
working group, will that working group be permitted to use or refer
to parts of the nfsv4.x protocol definition to specify the server-
to-server protocol?  If not, don't we have an issue?

Rob T

---

From: Brent <[email-masked]>
Date: Fri, 17 Aug 2001 14:57:50 -0700

> to-server protocol?  If not, don't we have an issue?

Depends whether a repl/mig protocol is inspired by some feature
of v4, or whether it uses some piece of the protocol verbatim
and becomes dependent on NFSv4 RFC's.

I prefer inspiration.

	Brent

---

From: Mike Eisler <[email-masked]>
Date: Fri, 17 Aug 2001 15:48:12 -0700

> than a server reboot should it ?

Except that servers for mission critical applications are
rarely rebooted, whereas if fileset migration is useful, it won't
be rare. 

These is lease state and meta-data state.

lease state. The existing NFsv4 protocol allows server to
avoid grace periods after server reboots
if they records lease state in persistent
storage. If the fileset transfer protocol punts on
transfering lock state, then this cuts into the
availability of the fileset, because the receiving server will
have to return NFS4ERR_GRACE for the duration of the grace period
to re-establish leases.

Let's say a server is rated at 99.99 per cent availability,
and lets say you can market the server such that the 99.99
refers to per-fileset availablility.
This works out to around 3000 seconds of downtime per fileset
year. I've certainly seen systems up for years at time; 99.99 is not
even state of the art these days.
If you've got a grace period of 60 seconds, you can migrate a
filesystem 50 times per year, among such systems; probably
ok.  Add another 9, and you can migrate it 5 times per year. 
That's starting
to push it. 99.9999 percent availability and you can migrate one
every couple years. So for some users, migration of
writeable filesets is useless without migratable lock state.

meta-data state. State like persistent file handles, fileids, etc., 
if not handled in the server-to-server
protocol is going to cause obvious problems.

	-mre

---

From: salit gazit <[email-masked]>
Date: Fri, 17 Aug 2001 16:03:12 -0700

> is an opaque thingy that cannot easily be represented physically

Is there a need for replication of nfsv4 attributes?
The mandatory and recommended attributes are derived from filesystem
or server internal attributes that may differ between the filesystems
(e.g. fileid, fsid, expire type, etc.).  The named attributes
will be implemented as a directory in the filesystem (right, Jeff?),
so they will be replicated as part of the filesystem by default.

Salit
-- 
Standard disclaimers apply.

MailTo:[email-masked]

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 17 Aug 2001 18:13:33 -0500

> (e.g. fileid, fsid, expire type, etc.).  The named attributes

Or the attribute capabilities may be the same and being able to
replicate them is a desired feature.

> so they will be replicated as part of the filesystem by default.

That is an implementation detail that a replication/migration can not
rely upon.  Native filesystems may be implemented in varying ways that
may allow for transfer via a well-defined mechanism but not as part of
what would be considered "normal" filesystem access.

-- 

- Spencer -

---

From: Uresh Vahalia <[email-masked]>
Date: Sat, 18 Aug 2001 00:21:29 -0400

> unless it's across homogeneous filesystems.  I think we'll be

Absolutely.  But if it is merely something that kind of sort of works
most of the time, customers would doubtless seek safe haven in a
homogenous solution.

---

From: Uresh Vahalia <[email-masked]>
Date: Sat, 18 Aug 2001 00:12:23 -0400

> like this do not meet customer needs.

Most serious server vendors have some form of block-level replication products.
Some of us also have filesystem replication products, albeit homogenous.

It is hard to imagine that heterogenous file system replication is a real customer
need, although you will always find the bleeding edge guys who absolutely must
have it.  It could be a very exciting technology to develop, but commercially it
is likely to be a solution looking for a problem.

> The IETF does protocols and not APIs, so this seems necessary.

To clarify my point -- you could build a minimal protocol that provides essential
communication based on which vendors could build different kinds of solutions, or
you could do something much more comprehensive, which in effect would become a
complete recipe for replication/migration.

> or replication activity, but perhaps we should be doing that.

Migrating an entire file system is useful, but often not good enough.  The ability
to specify which parts of the file system should be migrated makes this extremely
powerful and useful.

> think that would be a useful effort, but not tied to replication

Whether part of replication this effort or not, this is a key feature that solves
some important customer problems.  As we have noted, many vendors have a
homogenous replication protocol.  However, this cannot be effectively leveraged by
clients, except for pure read-only environments (here too, I am not sure how many
other client implementations have incorporated Sun's automounter enhancement).

Let us assume a server vendor (or cooperating vendors) has a server replication
solution with one writer and one or more readers.  Assume further that this
solution has the ability to switch the writer role from one server to another,
perhaps instantly.  To make this useful, clients need a way of knowing (a) the
number of replicas and their location, (b) which one of them is writable, and (c)
any change in this configuration.  The same applies in the more complex cases of
multiple writers.

Likewise, for migration, it is very useful for a server to be able to inform a
client that a particular data set is located on a different server, so that
adminstrators can freely move data sets around without disrupting client access
and major management hassles.

Finally, why does migration require ANY protocol support beyond regular NFS?  Even
v3 should work just fine.

Uresh

---

From: Brent <[email-masked]>
Date: Mon, 20 Aug 2001 11:55:43 -0700

> even state of the art these days.

Yes, if you want to make three 9's availability a requirement for
this protocol, then you have to move the lock and lease state.

But if it becomes a requirement, then the result will be
a protocol so complex to implement, that nobody will do it.

All we have today is proprietary, block-level protocols like
Sun's SNDR, EMC's SRDF, or NetApp's SnapMirror.  I don't
think any of those move file open or lock state either, but
I'm sure they're widely used, and by mission critical apps.

A good initial stab would be a protocol that does a good
job of creating read-only replicas and the synchronization
of periodic snapshots.  If that is successful (and if anyone's
interested) then apply some read-write and high availability
requirements to a follow-on standard.

	Brent

---

From: Mike Eisler <[email-masked]>
Date: Mon, 20 Aug 2001 12:09:55 -0700

> I'm sure they're widely used, and by mission critical apps.

I'm sure that none of them are used with writeable, migratable
filesets.

> of periodic snapshots.  If that is successful (and if anyone's

Which is exactly where we end up if we constrain the
problem in the manner you listed (which prompted my response).
	-mre

---

From: Brent <[email-masked]>
Date: Mon, 20 Aug 2001 12:55:40 -0700

> most of the time, customers would doubtless seek safe haven in a

... for which block-level solutions already exist. No problem with that.

I'm guessing that most apps make fairly minimal demands on filesystem
features so that they have maximal portability.

So I hope it's reasonable to assume that a protocol that works across
heterogeneous filesystems, albeit with less than 100% support of features,
will be useful and popular.  Isn't that the premise behind NFS ?

	Brent

---

From: Robert Thurlow <[email-masked]>
Date: Mon, 20 Aug 2001 16:20:58 -0600 (MDT)

> a protocol so complex to implement, that nobody will do it.

It is far from obvious to me that trying to migrate state will
result in an unimplementable mess.  Surely it's worth some
thought experiments before we limit ourselves to tackling a
replication protocol only?

Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Mon, 20 Aug 2001 16:20:55 -0600 (MDT)

> It is hard to imagine that heterogenous file system replication is a real
> customer need, although you will always find the bleeding edge guys who
> absolutely must have it.  It could be a very exciting technology to develop,
> but commercially it is likely to be a solution looking for a problem.

Our customers regularly tell us (sometimes loudly and colourfully)
that we can't just put things into Solaris - that we need to make
sure the stuff we do is also available on other vendors' systems.
That's what makes me believe that heterogenous replication and
migration is worth doing.  Whether there are enough interested
vendors to make this viable, and enough customer demand that it
matters once built, I don't know.  Do you believe your customers
would not care about a heterogeneous transfer protocol?

> Migrating an entire file system is useful, but often not good enough.  The
> ability to specify which parts of the file system should be migrated makes
> this extremely powerful and useful.

From my past with AFS, I was used to people using volumes which
were the right size, so that policy set on the volume would be
fine.  My home directory was a volume, not a subdirectory on a
huge filesystem like it is now.  Clearly, if you are forced by
a shortage of LUNs or partition slots to share huge filesystems,
a finer-grained policy is desireable, but it seems like it could
be just a workaround.  I know we're working to permit right-sized
partitions so a per-filesystem policy is more correct.  Do you
have an example that shows this as more generally useful?

> Whether part of replication this effort or not, this is a key feature that
> solves some important customer problems.
...
> Let us assume a server vendor (or cooperating vendors) has a server
> replication solution with one writer and one or more readers.  Assume
> further that this solution has the ability to switch the writer role from
> one server to another, perhaps instantly.  To make this useful, clients need
> a way of knowing (a) the number of replicas and their location, (b) which
> one of them is writable, and (c) any change in this configuration.

Agreed here; I'd love to see this more standardized.  Do others on
the alias want to see us work to solve this?

> Finally, why does migration require ANY protocol support beyond regular NFS? 
> Even v3 should work just fine.

This is a good question; in answering, I've been hitting some
some assumptions that a separate protocol can be faster and more
able to pare down transferred data to a minumum.  While I still
think a protocol which uses XDR structures over a TCP connection
could be faster, that's not necessarily true.

If we have a module which can bypass any block boundaries and
sizes, and which has either the ability to get precise lists of
changes from the shared filesystem or the ability to calculate
this with rsync, we can do very well.  But we can't get a couple
of things I think are important.

Without a seperate protocol, state migration can't happen, so
truly seamless migration is not possible; the client/server
protocol has no op for this nor a need for such an op.  And
management of the transfer seems impossible without some kind
of out-of-band way to control the process, e.g. to reserve a
partition for a new replica, or to inform a peer that we're
done sending data.  So I think that even if we used NFS ops to
transfer most data and metadata, we need something more.

Rob T

---

From: Brent <[email-masked]>
Date: Mon, 20 Aug 2001 16:37:19 -0700

> replication protocol only?

I've no objection to thought experiments, but we need to focus
on basic file-level data replication first. If we can't come
up with a simple protocol that's scalable and efficient, then
the game's over.

We need to prioritize the requirements list.  Then we can
decide which features are near the bottom, and likely to
fall off.

	Brent

---

From: Brent <[email-masked]>
Date: Mon, 20 Aug 2001 17:44:39 -0700

> the alias want to see us work to solve this?

Well, the Service Location Protocol provides a great infrastructure
for developing this kind of capability.  Replica location is necessary,
but in this context it's a distraction. We should draw a clean boundary
between the mechanics of moving/replicating data - and naming it.

Customers already have mechanisms for replicating data that are
separate from how they name it. I think we should maintain this
flexibility.

> Even v3 should work just fine.

NFS is not designed to move large amounts of data/metadata efficiently.

For instance, you'll move a file collection much more quickly in a
tar or zip file over FTP than a "copy -r" over NFS.

Neither does NFS have any efficient way to ask a server for a list
of files or directories that need to be updated.

	Brent

---

From: Uresh Vahalia <[email-masked]>
Date: Fri, 01 Jun 2001 14:57:40 -0400

> separate from how they name it. I think we should maintain this

True.  The two are somewhat separate problems.  However, the naming problem, which
includes what view a client has of a replicated environment, will have a strong
influence on any server-to-server replication/migration solution/protocol, since the
latter will have to meet the expectations of the former.