Subject: Re: Repl-mig protocol skeleton

From: Ted Anderson <[email-masked]>
Date: Tue, 10 Sep 2002 08:02:19 -0400

On Tuesday 27 August 2002 17:31:09 MDT [email-masked] wrote:
 > I've been working on a strawman protocol for replication/migration,
 > and I don't have that yet, but I do have this skeleton I can start
 > gluing straw onto.  This is what the protocol looks like from here;
 > let me know if you see problems so far.  I'll keep on adding detail
 > and work towards getting this into Internet Draft format.
...
 > Initiation/restart phase messages:
 >
 > The source server starts by sending OPEN_SESSION_NEW (for a new
 > transfer) or OPEN_SESSION_RESUME (to resume an old transfer).
 > The destination server responds with either OPEN_SESSION_CONFIRM
 > or OPEN_SESSION_DENY.  SECINFO_REQ and SECINFO_RESP and the
 > security tokens in each message will be necessary if we are
 > requiring per-message security.

What about "pull" replicaion?  Generally, how does the target get to
specify which parts of the primary it wants to receive?  For example,
only the files accessible to the principal authenticating this channel,
or the file just requested by a user (aka a cache miss).

What about filesystem version numbers, which allow source and target to
converse about specific filesystem instances?  This would allow the
target to request an explicit update, i.e. changes from version x to y.
In the absence of version numbers, perhaps dates could suffice, allowing
something similar to an incremental dump from source to target.  In
general, the target wants some say in what data it receives.

 > OPEN_SESSION_NEW - create new transfer session
 > - Security token
 > - Session ID
 > - Source FS path
 > - Destination FS path

How does the source know the destination path?  In general, there needs
to be some kind of filesystem identifier that all replication
participants would use to refer to filesystems.

A specification of the file system implementation would help peers to
know each other better.  It would provide for negotiating special (maybe
private) capabilities as well as whether the file handles are
compatable.  Maybe a product name plus major and minor version numbers
would do the trick.

 > Data transfer phase messages:
 >
 > The source server begins processing filesystem objects in some fixed
 > order which will permit checkpointing and restarting in case of some
 > problem or operator abort.  SEND_OBJECT_METADATA is sent first, then
 > SEND_FILE_DATA messages will be sent for non-directory objects.  If
 > outstanding lock state for an onject exists on the source server, it
 > will be sent via SEND_LOCK_STATE messages.  If the source server
 > cannot capture all create and remove operations on a directory
 > reliably, SEND_DIRECTORY_CONTENTS will permit the destination
 > server to validate its directory entries.  Named attributes are
 > handled with SEND_OBJECT_METADATA messages with IS_NAMED_ATTR
 > set to true, and apply to the previous non-named-attribute which
 > was handled.  CLOSE_OBJECT is used to indicate that all data and
 > named attributes of an object have been transferred.  At any time,
 > the source server may set a checkpoint with SEND_CHECKPOINT.  All
 > messages are confirmed by the destination server with CONFIRM_MESSAGE.

What about sending file handles for open files?

I don't understand how IS_NAMED_ATTR works.  In what way does the
message apply to the previous unnamed attribute?  Where is the name of
the attribute specified?

The comments about "capture all create and remove operations" applies
only to sending directory updates, right?  Renames would be good to
capture explicitly (compared to noticing a create / remove pair whose
File ID matched) as it would allow the target server to preserve open
files more easily.

 > SEND_OBJECT_METADATA - send metadata about onject
 > - Security token
 > - Message ID
 > - File ID
 > - Object name
 > - Object type
 > - Object ACL
 > - IS_NAMED_ATTR
 > - NFSv4 attrs
 > - Other attrs

s/onject/object/.

Is Object name the name of this object in the parent directory or its
whole pathname?

Is File ID the file handle this object is known by, or is it opaque
information used only by compatible servers to preserve persistent file
handles (i.e. the inode / unique)?

 > SEND_FILE_DATA - send file data
 > - Security token
 > - Message ID
 > - File ID
 > - Data offset
 > - Data length
 > - IS_HOLE
 > - Data

I would think that missing data segments would be implicitly assumed to
be holes.  What does IS_HOLE mean?

In this msg File ID is redundantly linking it to the previous
SEND_OBJECT_METADATA msg?

 > SEND_LOCK_STATE - send file lock state
 > - Security token
 > - Message ID
 > - File ID
 > - Array of:
 >   - Lock offset
 >   - Lock length
 >   - Lock owner

Need a way to send share reservation (aka open locks) state too.

 > SEND_DIRECTORY_CONTENTS - send names of objects in a directory
 > - Security token
 > - Message ID
 > - File ID
 > - Dir cookie
 > - EOF flag
 > - List of object names

What are the issues with mapping dir cookies between servers with
different directory structures?  Is this field just for use by
compatable servers, or for handling readdir operations that are in
progress during a migration event?

 > CONFIRM_MESSAGE - confirm any data transfer message
 > - Security token
 > - Message ID

Not just data messages but all messages.  It would be nice to have a way
to confirm a whole sequence of messages, or is each message to be
confirmed synchronously?  Are Message IDs consecutive?  In the
asynchronous case, how are outstanding confirm messages synchronized
with Abort/Close Session messages?  Who is is responsible for handling
missing confirm messages?  Is the sender to retransmit affected messages
on some schedule?

 > Termination phase messages:
 >
 > ABORT_SESSION may be used at any time by either server to terminate
 > a session prematurely; a checkpoint ID is recommended to permit
 > restart if possible.  CLOSE_SESSION is the normal termination
 > message, and must be issued by the source server first and then
 > issued by the destination server as a confirmation.

If ABORT_SESSION contains a Checkpoint ID, doesn't it need explicit
confirmation and hence a message id?  Or maybe it would be more sensible
to just send a checkpoint followed by an abort.

Ted Anderson

---

From: Robert Thurlow <[email-masked]>
Date: Tue, 10 Sep 2002 13:27:38 -0600 (MDT)

> specify which parts of the primary it wants to receive?

Per my other message, the target server would ask out-of-band,
and would get updates to the whole filesystem, which is the
basu unit of replication.

> general, the target wants some say in what data it receives.

This has been discussed on the alias before, and is not out of
scope, but it isn't reflected here since it seems more of an
application-level thing.  I'd expect that when versions are
really needed, there's an application which can put that info
into a file in the root of the filesystem so it knows the state
of the data.  This doesn't support "I want version 2.23".  How
would the NFS client (as opposed to a particular application)
make use of this?

> How does the source know the destination path?

This would come in through the SNMP or WBEM/CIM operations
controlling the transfer.

> compatable.  Maybe a product name plus major and minor version numbers

I agree; this should be represented in the initial negotiation.

> What about sending file handles for open files?

The assumption was that we would rely on the client to initiate
recovery of state after it learned of the migration event or
switched to a new replica.  Can you think of a scenario where
this would not work?  It is possible to add if needed.

> message apply to the previous unnamed attribute?  Where is the name of

Apologies, this was poorly explained.

Imagine a file "A" with a named attribute "B".  What I had thought
of was that you would send a SEND_OBJECT_METADATA for the object
"A" and then issue SEND_FILE_DATA until done, and then issue a
SEND_OBJECT_METADATA with the name "B" and IS_NAMED_ATTR set to
indicate that "B" is a named attribute, and then send SEND_FILE_DATA
for the named attribute until done.  This means you need to retain
that you're dealing with named attributes for "A" until you get a
new SEND_OBJECT_METADATA without IS_NAMED_ATTR set.

> File ID matched) as it would allow the target server to preserve open

Yes, renames and removes need to be expressed as separate messages,
and having this would be better.  Listing the directory's objects
was a capability for less-endowed servers which couldn't be
sure to capture all rename and remove events.

> Is Object name the name of this object in the parent directory or its

Probably the whole pathname for objects other than named attributes.

> handles (i.e. the inode / unique)?

I was thinking of the NFSv4 "fileid" attribute.

> be holes.  What does IS_HOLE mean?

If you're doing a replica update, you need a way to note that
a chunk of the file has been deallocated without sending over
zero-filled blocks.

> SEND_OBJECT_METADATA msg?

Yes.

> Need a way to send share reservation (aka open locks) state too.

I agree - this is an oversight.

> progress during a migration event?

I think it's just for compatible servers, and am not very
convinced it's useful in that regard.  I agree that we need
to think more about dir cookies.

> missing confirm messages?  Is the sender to retransmit affected messages

You have more questions here than I have answers :-)  I would
expect monotonically increasing message IDs, and would hope
to rely on the transport to retransmit until it failed, and
then to rely on the checkpointing information for restarts.
This seems to tie to the discussion about whether NFS servers
should be able to drop requests when busy, rather than just
closing the TCP window and making TCP do the work.  A lot more
thought is needed here.

> to just send a checkpoint followed by an abort.

I'd lean towards the former from here.

Rob T