Subject: Re: Replication/Migration issues list

From: Ted Anderson <[email-masked]>
Date: Tue, 10 Sep 2002 07:58:05 -0400

On Friday 23 August 2002 16:34:30 MDT [email-masked] wrote:
 > This is a list of the open issues with the current Internet
 > draft on replication and migration, found at:
 >
 > 
http://www.ietf.org/internet-drafts/draft-thurlow-nfsv4-repl-mig-design-00.txt
 >
 > We talked about most of these to some degree at the last IETF,
 > but the turnout was light enough that we didn't make much
 > progress.  I'd appreciate interested parties looking at this
 > and giving your opinion. For extra credit, you could comment
 > on the draft as well :-)

 > Rob T
 > -----
 >
 > RPC or just XDR?
 >   This is still outstanding, pending some further analysis and
 >   perhaps some prototyping.  The current draft leans towards
 >   the servers passing XDR-encoded messages across a regular
 >   TCP connection to bypass performance limitations and data
 >   transfer size limits, but is there any real gain from this?

I am not sure I understand this summary of the issues and the strawman
was a little sketchy about this as well.  Does this sound like a fair
restatement of the choices?  A TCP stream approach allows decoupling of
the SEND_XXX msgs from the confirm messages and thus permits pipelining.
This should be faster than a strictly request / response architecture,
at least in a single threaded environment.  If it is necessary to
support replication over UDP, then RPC might be a good way to do this,
as RPC also provides reliability and flow control which TCP already has.

 > Security - TLS or per-message GSS-API?
 >   Since we have a situation where two peers are connecting to
 >   transfer a block of data, we don't really need to put strong
 >   security support in every message, according to IETF norms.
 >   The current draft focuses on per-message security, mentioning
 >   TLS at just one point.  What should we do?
 >
 >   Advantage of TLS: single 'open' message, lower overhead
 >   Advantage of per-message: we do it already with V4

Would per-message security allow multiple principals to be used to
authenticate different parts of a filesystem transfer?  For example,
imagine that the replica was operating a bit like a prefetched cache,
and the target was fetching different parts of the filesystem using
different users' credentials.  This might be important when the source
and target don't trust each other sufficiently well to share an
all-powerful principal.  It would seem that using TLS to permanently
secure the whole channel all would make this approach more difficult.  I
suppose a different TLS authenticated channel could be used for
transfering each part of the replica.

 > Compression - do we need it?
 >   This can reduce bandwidth, but we need to negotiate it and pick
 >   an algorithm(s) to use.  This is probably worth specifying - we
 >   just need to find the right algorithms.

As this will be an optional features, providing the scaffolding for
negotiating the algorithms might be enough to start.

 > Encryption - do we need it?
 >   This is necessary if we do not use RPC and RPCSEC_GSS, because
 >   it must be possible for the data's integrity and privacy to be
 >   protected.  Again, we just need to find the right algorithms.

Wouldn't support for encrypting the channel automatically fall out of
adopting TLS, as it does when using RPCSEC_GSS?  Surely, this is a
solved problem we can delegate to a preexisting RFC.

 > Other bandwidth reduction ideas - do we bother?
 >   Some other ideas (e.g. detection of duplicate blocks) have been
 >   proposed, but is there really going to be benefit to justify it?
 >   I'm inclined to close this without action.
 >
 > rsync equivalent - do we bother and how to approach it?
 >   Some have suggested the protocol drop into an rsync-like mode
 >   where the two servers compare objects when insufficient information
 >   is available.  Probably a good idea if we can detect when to do it
 >   and come up with protocol.

The key with rsync and other bandwidth reduction techniques is that they
rely on two-way communication.  If this can be accomodated then a whole
suite of schemes like this are probably feasible.  One could imagine a
plug-in that exchanged messages for a while using a sub-protocol and
exited to the main protocol when the file's data was transferred.  So
specifying a way to negotiate the sub-protocol and defining an envelope
for the sub-protocol's msgs might do the trick.

 > What base NFSv4 protocol changes would be useful?

Two from my an earlier message are these:

Defining replication subsets using fsid may require that the fsid of a
file is not fixed.  A mechanism for changing a file's fsid might make
this method of redefining subsets more practical.

A better defined means of recovering from readdir cookie failure after a
migration or replication event would improve the transparency of readdir
sequences.

 >   Some things in RFC3010bis are not ideal for replication/migration,
 >   so we have always been open to minor revisions which would make
 >   this work better.  Ideas which have been brought forth include:
 >
 >   - Change the definitions of the fh_expire_type as defined in:
 >     http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2002/0078.html

This proposal seems like a good idea.  The proposal needs a way to list
open file handles when a filesystem is shipped to the target.  For each
handle, the file it refers to must be unambiguously identified.

 >   - Allow the client to know whether filehandles on the destination
 >     server are the same or not, based on the results of the transfer.
 >     The new info would likely have to be made available as more
 >     attributes the source server makes available.

By "the same" do you mean that persistent handles would be honored at
the new servers?  This suggests that file handles should have type ids
or something like that.  Or maybe you are suggesting here that instead
the source and target servers to determine this out of band somehow (by
comparing implementation identifiers?) and then tell the client.

The alternative to this "sameness", I gather, is volatile file handles.
What I have not quite understood about volatile file handles is how
frequently they can be avoided.  Both fsid-boundary crossings and
switching from one replica to another are events that can result in
changes in the fs_expire_type attribute.  Since both of these may occur
unexpectedly as a client traverses a pathname, it seems that the
conservative client must track pathname information needed for
refreshing volatile file handles whenever these events are a
possibility.  However, these seem like normal occurrences, at least in a
large environment.  So I am lead to assume that, except for a few very
circumscribed situations, clients always have to be prepared for file
handle expiration and so must track pathname information.

Anyway, with that background, the problem with the above idea is that
the client really needs to know when it first opens the file whether the
handle is persistent and will stay that way.  If a file is in-use when
the file system is migrated, isn't it too late to tell it that the
destination server has different file handles?

Ted Anderson

---

From: David Robinson <[email-masked]>
Date: Tue, 10 Sep 2002 10:12:07 -0500

Ted Anderson wrote:
> at least in a single threaded environment.  

There is nothing in RPC that doesn't allow pipelining, even in
a single threaded environment. Requests, which can be of arbitrary
size potentially many megabytes, do not necessarily have to
complete before the next request is issued. NFS clients have
done this for years, biods, async threads, etc. At the wire
level, and RPC protocol such as NFS is not significantly
different or more complex than a "pure" TCP protocol
like iSCSI. If you choose to use a common easy to use
library or code generator, you will not get an optimal
implementation. In almost all network protocols that want
some degree of reliability, there is a response to every
message (reply). The only real distinguishing factor
is the ratio of response to the number of bytes sent.

The important design consideration is the linking
of the result of one request to the sending of another.
"Statelessness" is what some like to call it. Linking will
kill pipelined performance.

> If it is necessary to

I hope for replication/migration we just take UDP off the table
entirely. In hindsight we left to much accomadation for
UDP in NFSv4.  But, TCP connections do drop and you do
need to know how to recover, so TCP is not "reliable"
enough data. So we can't just send a gigabyte without
some backflow responses to let us know how much got through
in case of connection failure. I'll probably agree 8KB
is probably too small a chunk size.

> for the sub-protocol's msgs might do the trick.

What you are describing here is what most people call
request/response RPC.  The only thing we need to do is
to get the overhead out of the way on data operations
so we get high bytes per message ratios. Simply allowing
large transfer sizes should be sufficient.

	-David

---

From: Robert Thurlow <[email-masked]>
Date: Tue, 10 Sep 2002 13:27:18 -0600 (MDT)

> at least in a single threaded environment.

Yes.  I don't think many of us are seeing UDP as a priority -
more to the contrary for this work.

> different users' credentials.

Per my previous message, I'd say that an intermediate cache
hasn't really been considered so far.  But yes, per-message
would be a good thing to solve this.

> negotiating the algorithms might be enough to start.

I agree.

> solved problem we can delegate to a preexisting RFC.

Yes, Mike Eisler had mentioned this in
http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2002/0597.html.
It does seem that we don't have to do new work here.

> for the sub-protocol's msgs might do the trick.

I agree, this could be a good way to handle this.

> comparing implementation identifiers?) and then tell the client.

I think that somehow, we need to be able to negotiate whether
the source server should send filehandles and whether the
destination can honor them; part of this may indeed involve
expressing how the filehandle is constructed, but it may
reduce to just an awareness that both servers are running
the same operating system and filesystem.  If we discover
that the filehandles can be honored, the client should get
to make use of this information.

> ... it seems that the

Likely true.  Fortunately, the overhead of tracking the pathnames
has not proven to be significant in practice (Solaris NFS client
failover does this).

> destination server has different file handles?

Yes :-(

Rob T