Subject: Re: I-D ACTION:draft-thurlow-nfsv4-repl-mig-design-00.txt (Fwd)

From: Ted Anderson <[email-masked]>
Date: Tue, 10 Sep 2002 07:45:10 -0400

On 6/27/2002 Robert Thurlow wrote:
> I ...  would appreciate folks looking at the draft and
> sending me comments and issues.

Comments on draft-thurlow-nfsv4-repl-mig-design-00.txt[0] dated June
2002.

In section 1.1.1 I'd quibble a bit.  Replication is basically used for
two reasons: making a copy "here" for locality/performance and making a
copy "there" for redundancy/availability.  Updates can be "pulled" as
well as "pushed".  I suppose in the interests of full disclosure, I
should admit that I have come to view replication rather broadly,
specifically including arrangements commonly referred to as caching.
There is spectrum here, and it seems to me that caches that do
prefetching (e.g. Web proxies) or hoarding (e.g. Coda) operate very much
like replicas.

It might be helpful to define two terms for the end-points: primary and
secondary instead of, or in addition to, the more directional source and
target

In section 1.1.2, is it useful to explicitly describe migration as a
form of replication where the "primary" designation moves from one site
to another?  After establishing the new replica on the target, the old
replica, now a secondary, may be deleted.  One task of the change of
primary-ness step is exchanging mapping information for open file
handles (i.e. the fh_expire_type change mentioned below).

In section 4.1 no message types are mentioned that address consistency
management.  For example, it would be valuable to have a way to receive
notification when the primary is updated.  Maybe this functionality is
addressed by another protocol, as is administrative control?  If this
functionality is to be separate there needs to be a common means to
identify files that are the subject of both protocols.  This need
impinges on the question of preservation or mapping of file handles
between servers, that is the client-server and server-server protocols
need a common way refer to files.  Are pathnames the best means of
forming a persistent reference to files or can another,
architecturally-neutral, method be worked out?

Section 4.4 mentions the rsync protocol.  An interesting variation of
the method of sending only unique data (not necessarily modified bytes)
is the "A Low-bandwidth Network File System"[1] which identifies SHA
hashes of file segments available on both the source and target system
to avoid sending redundant data.  The Venti Archival Storage System[2]
also uses SHA hashes of blocks to index its data.  These projects
suggest that a hash-based approach to minimizing the cost of
transferring updates is worth considering.

Section 6.3 describes replication of subsets using fsids to label the
subsets.  How can subsets be defined or redefined on the fly?  It would
seem necessary to be able to change the fsid of a file while it is
in-use, but is this possible within the NFSv4 client-server
architecture?  Since some files may be in-use essentially permanently,
any repartitioning of a filesystem's replication subsets may stumble
across these files.  Must repartitioning be a strictly off-line process?

Since preserving readdir cookies across a replication or migration event
is likely to be impossible in most cases, it might be useful to specify
some well-defined procedure for clients to take when they receive the
NFS4ERR_BAD_COOKIE error.

Some other questions: What about reconstructing file handles when a
client switches from one replica to another?  What happens when replica
updates (esp. deletes & renames) affect open files?  Should the server
expire all file handles on a replica whenever an update is received (or
only under some conditions)?

Ted Anderson
(aka [email-masked])

[0] 
http://www.ietf.org/internet-drafts/draft-thurlow-nfsv4-repl-mig-design-00.txt
[1] http://www.pdos.lcs.mit.edu/papers/lbfs:sosp01/
[2] 
http://www.usenix.org/publications/library/proceedings/fast02/quinlan.html

---

From: Robert Thurlow <[email-masked]>
Date: Tue, 10 Sep 2002 13:25:51 -0600 (MDT)

Hi Ted, thanks for all the comments.  Here are my thoughts:

> copy "there" for redundancy/availability.

True - using multiple copies nearby to increase performance
could be interesting, though this might be more relevant to
services other than NFS due to cache consistency issues.

> Updates can be "pulled" as well as "pushed".

The current proposal deals only with making information flow
from the master copy out to the replicas, and this flow is
unidirectional (modulo rsync-like examination of files).  The
way a transfer is initiated is unspecified - earlier thoughts
of also defining an third-party admin protocol to control this
were abandoned since we have things like SNMP and WBEM/CIM.
That means that the impetus to perform an update can come from
a human being, from the destination server, or from a "cron"
job, and happen out-of-band to cause the source to contact
the destination server(s) via this protocol.  We do need to
also define the SNMP MIB or WBEM/CIM schema to do this.

> to another?

The working group has considered having multiple replicas with
one designated the master, to which writes would be directed.
I don't think this was ever declared to be out of scope, but
the cache consistency issues in NFS client implementations and
the current lack of a standard way to tell the client about it
cast long shadows over it.  How important does this seem to you?

> management.

I guess that so far, my direction has been to collect changes
to a filesystem into replica update batches and propagate them
in order to meet the goals set for the replicas, like the "how
far out of date may replicas be?" setting I recall from DFS.
If you want more consistent replicas, you send out updates more
often.

It sounds like you're thinking about client-to-server-like
operations to control individual files.  This would help an
intermediate cache, certainly, and you might get closer to
a guarantee that clients will see changes in a very short
time period.  But on the other side, if you want to implement
filesystem consistency, where a complete database transaction
is propagated intact, the current filesystem-wide update is
better.  I'm not sure which compels more.  I'd like to see NFS
be able to use proxy caches, but I wonder if the NFSv4 core
protocol isn't a better place to work on it?

> transferring updates is worth considering.

This appears to be related to work discussed by Juan Gomez in
http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2002/0256.html
and the following thread.  I'm skeptical - is there really that
much dpulication in segments to justify this work?  Juan was
going to do some analysis of how this would work with real-world
data.

> across these files.  Must repartitioning be a strictly off-line process?

I think repartitioning is a disruptive one-time event that is
just as painful as it is today, but the hope is that once the
filesystems are "right-sized", this doesn't need to happen
again.  This may not hold true over the life of the data, I
can see now that you mention it.  I don't really see a way
for this not to be disruptive, unfortunately.  Ideas?

> NFS4ERR_BAD_COOKIE error.

I agree.

> Some other questions:

> What about reconstructing file handles when a
> updates (esp. deletes & renames) affect open files?

The current expectation is that the client will remap via a
compound lookup operation based on a saved pathname.  An
optimization is for the client to be able to know when it
can use the same filehandle for the new server, but we're
not making an effort to optimally handle a situation where
only certain bytes of a filehandle change because of an
fsid change.  We do want to explore how to handle renames,
which will mess up the pathname lookup.

> Should the server

I'd hope not, as I expect it to be normal for updates to be
localized to a small number of files, and for the server to
be able to expire only those filehandles.

Rob T