Subject: Comments on draft-ietf-nfsv4-repl-mig-proto-00.txt

From: Mike Eisler <[email-masked]>
Date: Fri, 13 Dec 2002 13:55:13 -0800

Comments on draft-ietf-nfsv4-repl-mig-proto-00.txt

This goes back to the design i-d, where it stated, without
too much justification, that RPC would not be used in
transfer protocol. In looking at the strawman protocol, I
see a whole lot of stuff unspecified, that RPC would have
taken care of automatically:

	- security token formats
	- security negotiation (via RPCSEC_GSS and
		SPNEGO)
	- versioning
	- op code values for each message type

I also see that in this non-RPC protocol, many if not all
of the operations have confirmation messages.

I suggest borrowing a technique from the old NLM protocol
where some of the "procedures" with void returns, in fact
had no responses at all. If there needs to be an extension
to the XDR description language to tag certain procedures
as being messages without responses, then I'm all for it.

I realize that some of the things I'm observing are a
result of this being a strawman, but I can also see us
getting bogged down in specifying what is effectively a
layer in the protocol stack, when we already have one, ONC
RPC, to use today. Time is of the essence.

In section 2.2, the RMmessage_id type ... does the first
message in a session start at zero?

In section 2.5 (and section 7), why is the nfsace4 type
imported from nfsv4? If the other attribute types aren't
importanted, there should be no need for the ace type.

In section 4.1 and 7, there are no op code values for the
session message types.

In section 4.4, regarding the RM_UTF8NAMES capability flag,
why doesn't the nfsv4 migration protocol assume UTF8 file names?
So why does this bit need to exist?

Regarding the RM_FHPRESERVE capability, unless the source
and target know the fh format, this capability seems useless.
Since NFSv4 defines file handles as opaque, the capability is
indeed useless, unless source and target know they have compatible
file handles. My suggestion: 

	Include in the OPEN_SESSION_NEW arguments a field
	of type utf8str_cis (from the nfsv4 spec) which is
	a dns domain name that serves as an
	"implementation" identifier to uniquely and
	unambiguously identifies the file handle format. If
	a particular implementer has multiple fh formats
	(e.g. because it has both big endian and little
	endian platform support), then it can use multiple
	domain names or subdomain names. If the source and
	target have different domain names, it is still
	possible the target understands the source's file
	handle format, and it can still turn on the
	PRESERVE capability.

Does RM_FHPRESERVE also imply preservation of fileid?

On a related note, preservation of directory cookies is
another capability that would be good to support.

Also, since RM_FHPRESERVE might not be supported, it
implies, to me at least, that any server planning on
supporting the source side of this this protocol and
currently returning a value of FH4_PERSISTENT in the
fh_expire_type attribute should instead warn its NFSv4
clients up front, and return the FH4_VOL_MIGRATION bit.
Otherwise, clients will be out of luck when their data is
migrated away.

In sections 4.6 and 4.7, we have an apparently a pair of
messages that are sent synchronously in response to the
OPEN_SESSION_NEW and OPEN_SESSION_RESUME messages. Looks
like a remote procedure call to me.

In Section 5.1, Data transfer phase messages, shouldn't
there be a SEND_DELEGATION_STATE message as well?

In section 5.3 there is a RM_CIFS_ATTR type. Why is this
needed? This is the NFSv4 migration protocol, and in any
case, the set of NFSv4 attributes is rich enough to
encompass what Windows environments need.

In section 5.8 SEND_RENAME, it seems that a SEND_LINK
message is also needed.

In section 5.12, CONFIRM_MESSAGE, it says CONFIRM_MESSAGE
is used to confirm every data transfer message. First this
certainly looks like RPC. But second, I don't think it is
desirable to confirm every data transfer message, because
while I think ONC RPC should be used beneath this protocol,
I also think there is benefit to a paradigm that doesn't
ACK each and every message from the target. Instead, delete
CONFIRM_MESSAGE, and replace with with a true RPC that
allows the source to ask the target the sequence number of
the last data transfer message it has received.

SEND_CHECKPOINT though, ought to be a traditional RPC.

In section 6.3, ABORT_SESSION overloading ABORT_SESSION to
be both the confirmation from the target, or an initiated
message from the target is going to make implementation
complex.  Just have two different message types, or better
yet, RPCs.  (and make CLOSE_SESSION an RPC as well).

Section 6.2 implies that if a session is explicitly aborted
via ABORT_SESSION, it can later be resumed. It seems to me
that there ought to be an operation to allow the source to
tell the target that the session will not be resumed, and
another operation to tell the source that the target is
aborting and it will not maintain the previous state:
resumption is futile. Maybe this can be done with specific
error codes, but specific operations seems less error
prone.

In section 7 and else where, what are the definitions of the
various compression types? Do the compression types only
apply to the data field of the SEND_FILE_DATA operation.

Also, not to belabor the point, but the .x file lacks op
code values. This oversight is a direct result of not using
RPC.

Other things that were covered in the specification:

- the checkpointing model makes it obvious what the recovery
	model is when the target dies, but what happens when the
	source dies? How long must the target maintain the state
	(and disk space) of the incomplete transfer?
	Perhaps this should be negotiated when the session
	is first created.

- the protocol does not have any message types for telling
	the source to start to do the referral. Is this 
	implied by confirmation of the CLOSE_SESSION message?

- Can there be multiple sessions per TCP connection?

- Presumably this protocol requires an IETF approved
	congestion management transport, so TCP is the
	required transport.

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 19 Dec 2002 11:48:47 -0700 (MST)

Hi Mike,

Again, thanks, and comments inline.

> taken care of automatically ...

> of the operations have confirmation messages.

Every message is intended to have a confirm of some kind,
except for ABORT_SESSION, because the expectation is that
the connection would be closed after it was sent.

> RPC, to use today. Time is of the essence.

So let's open this up for discussion.  Suppose we make at
least a first attempt (in the Connectathon 2003 timeframe)
with an RPC-based protocol, based on this strawman.  Does
that make sense for others who might want to try this at
C-thon 2003?  I can certainly be convinced, because I have
less time for prototyping than I'd initially hoped.  What
about some of the other design team folks?

> message in a session start at zero?

Likely; shoulda been specified.

> importanted, there should be no need for the ace type.

I brought in fattr4's as well; I wanted to use them verbatim
when we're working with NFSv4 metadata.

> So why does this bit need to exist?

What I was thinking about here was to try to encode if
the server supports arbitrarily complex UTF8 names, or
if they will be mapped to something like simple ASCII.
The hope was that the source server could know this
early on and inform the admin that metadata might be
impacted on this destination server.

> 	handle format, and it can still turn on the

Yes, this was underspecified and you have pointed towards
a way out.  I had also thought (fantasized?) about trying
to encode the way the filehandle was put together - i.e.,
these bytes are invariant in this filesystem, these bits
represent the specific file, and these bits represent the
share point.  The biggest problem is that this assumes a
filehandle will consist of all of these parts, which is
probably not sufficiently general.

> Does RM_FHPRESERVE also imply preservation of fileid?

For domains where it is useful, it probably will, but I
don't know if it should be specified so.

> another capability that would be good to support.

Agreed - I hadn't thought about that.

> Otherwise, clients will be out of luck when their data is

Definitely agreed.

> there be a SEND_DELEGATION_STATE message as well?

Possibly helpful; I'd expected we'd recall delegations
just before pushing the final set of changes, and not
issue new delegations until completion.

> encompass what Windows environments need.

That's there so that vendors that sell multi-protocol
servers can express any CIFS attributes we can't encode
with NFSv4, and to illustrate the intent that we're not
supporting NFSv4 only, just NFSv4 first.  If this is not
useful, it can go, but I'd like to maintain support for
alternate attribute "profiles" in the future.

> In section 5.8 SEND_RENAME, it seems that a SEND_LINK

Agreed.

> SEND_CHECKPOINT though, ought to be a traditional RPC.

Good points; more fodder for just using RPC.

> yet, RPCs.  (and make CLOSE_SESSION an RPC as well).

How does the destination perform an abort?  Do we have to
define that the source implement a server side to receive it?

> error codes, but specific operations seems less error

I guess I expect the abort stuff to be in a crisis handler,
where a lot of computation is not welcome.  The implementation
could advertise if it can do restart or not at connection
time, which might accomplish enough here - would that work?

> apply to the data field of the SEND_FILE_DATA operation.

Underspecified again, and yes.

> 	Perhaps this should be negotiated when the session

Agreed on the latter point.

> 	implied by confirmation of the CLOSE_SESSION message?

That was my assumption; one further message/RPC to verify
that the destination is ready for this might be in order.

> - Can there be multiple sessions per TCP connection?

I'd prefer separate TCP connections for separate sessions.

> 	congestion management transport, so TCP is the

Yes; I thought I had specified that somewhere.

Rob T

---

From: Mike Eisler <[email-masked]>
Date: Thu, 19 Dec 2002 12:17:47 -0800

>about some of the other design team folks?
Cool. Let's cover the security thing and make
krb5, krb5i, krb5p MANDATORY too.

Speaking of Connectathon, in addition to testing
the strawman, it would be nice to do some NFSv4 client
testing to see if NFSv4 referrals (fs_locationed,
NFS4ERR_MOVED, etc.) actually work. This doesn't
necessarily require a complete or partial implementations
of the strawman repl-mig protocol.

>when we're working with NFSv4 metadata.

There are dozens of attribues other than nfsace4, and they aren't
in the strawman, It should be sufficient to refer to the types
coming from nfsv4,  rather that having duplicates in the
repl-mig protocol's .x file.

>impacted on this destination server.

The target might be storing just US-ASCII, or US-ASCII
plus  the 8 bit Euro character set, or the entire
16-bit only Unicode. And there are probably intermediate degrees
of capability. So in general I don't know how this information gets 
conveyed.
I think it is best to use the existing NFSv4 practice for letting the target
reject names it can't convey, and thus give the source the ooption of
mapping the names to simpler characters.

>probably not sufficiently general.

Right, designing a file handle description lanugage is a fantasy.

>don't know if it should be specified so.

If there is no implication, we may need RM_FILEID
since there's prose in the v4 spec that talks about leveraging
fileid to tell if file handles are unique.

>issue new delegations until completion.
That could be bad, just as we agree that it would
be bad if overall system forced clients to re-establish lock state
on the target. Recall of the delegation can result in a flurry of
byte range lock requests (per delegation) getting synced
to the source, and then the target (via the transfer protocol).
That's just one example. I think SEND_DELEGATION_STATE should
be in; sources don't have to use it, and we could even make
it a negotiable capability.

>alternate attribute "profiles" in the future.

We can encode any attrbute in NFSv4, just add a minor  version.
Vendors that with multi-protocol servers are accomodated whether they
support NFSv4 or not, CIFS or not.

Otherwise, here's what we have:

    - underspecified and/or opaque attributes that aren't in
        control of IETF, or

    - well specified attributes that are in the transfer
        protocol, but not in the NFSv4 protocol, thereby denying
        NFS users the capability to read these attributes.

>define that the source implement a server side to receive it?

We have to allow for bidirectional RPC on the same connection. This
doesn't seem like it should be hard to implement, even in kernels.

>time, which might accomplish enough here - would that work?
There are cases where the target and server will be able to restart or not,
and it may not know when the session is created (e.g. the source or target's
disk has crashed, or ENOSPC has occurred on the target).

>Underspecified again, and yes.

Also, there are likely intellectual property issues around some of those
compression types. We want to tread carefully here; optional,
negotiable compression is goodness since it is possible
(probable?) ipr issues won't let us make
any mandatory.

>I'd prefer separate TCP connections for separate sessions.

Works for me, just needs to be stated,

>Yes; I thought I had specified that somewhere.
Thisi is the only mention of TCP I found,
but I think adopting the Area Director
approved language from the nfsv4 spec is a
good thing to do now:

   As discussed in [DESIGN], this protocol draft will not use
   RPC [RFC1831] but will use XDR [RFC1832] formatted messages over TCP
   to maximize efficiency.

		-mre

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 19 Dec 2002 15:20:56 -0800

We're aiming to build a standard file replication
and migration protocol but nobody is wants to prototype it.

And there appears to be at least two groups of enthusiastic
implementors with running code - but no standard protocol.

	Rsync
		http://samba.anu.edu.au/rsync

	Unison
		http://www.cis.upenn.edu/~bcpierce/unison/contact.html

Could there be mutual benefit here ?

We can argue that the Replication/Migration protocol for NFSv4
has lots of lock state, attributes etc that are not directly
supported by these filesystem synchronization applications.

However, I'll argue that 90% of Replication/Migration is
simply the efficient movement of filesystem changes and very little
of it is v4-specific.  These user/developer communities
have a lot of hands-on experience in this area.

We could solicit these groups to read the requirements doc
and submit protocol drafts inspired with their expertise
in filesystem replication.  I've already suggested to
both group's mailing lists that they take a look at the NFSv4
repl/mig work.

Just a thought ...

	Brent

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 20 Dec 2002 14:43:43 -0700 (MST)

[RPC]
> krb5, krb5i, krb5p MANDATORY too.

Certainly, if we go with RPC this is essential.

Agreed.

> repl-mig protocol's .x file.

Okay; I don't want duplicate (and especially inconsistent)
definitions here.  I just don't know of a prior example of
pointing correctly to something whose encoding is defined
in another RFC.  Can anyone think of an analogue?

> mapping the names to simpler characters.

But by then you may have been terabytes into the transfer
and will have just messed up the client by that change.

I think we need to try to codify the capabilities somehow
to be able to provide a warning before the transfer starts.

> fileid to tell if file handles are unique.

Agreed - thanks for the reminder.

> it a negotiable capability.

Okay, I'm sold - it's in.

>         NFS users the capability to read these attributes.

In the past, at least some people (Brent, your cue) have talked
about this being able to grow up into a generalized replication
protocol that might be used with other types of data, e.g. WebDAV.
The "attribute profiles" were a way to define NFSv4 first, but
permit other things to plug in later should someone want to do
that.  If this seems architectually wrong (i.e., more is wrong
than just the way I dealt with it in the draft), then we should
kill it now.  I'm running out of energy to defend this, myself,
and want to get on with stuff that matter more.

> disk has crashed, or ENOSPC has occurred on the target).

Does this need to be a new RPC or a new argument to ABORT_SESSION?

> (probable?) ipr issues won't let us make

Agreed!

Agreed.

Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 20 Dec 2002 14:47:50 -0700 (MST)

> implementors with running code - but no standard protocol.
...
> Could there be mutual benefit here ?

...
> I've already suggested to

So, given that you've done that, who would take what action?
I just can't quite tell what you think would be needed.

Rob T

---

From: Brent Callaghan <[email-masked]>
Date: Fri, 20 Dec 2002 14:14:14 -0800

I've already dropped an Email in both group's mailing
lists pointing at the IETF protocol effort.

Our only action should be to welcome their input and
expertise in this technology.

My hope is that someone might have a go at modifying
Rsync or Unison to implement a strawman protocol or even have
a go at describing their protocols to us.

I like Dave Noveck's suggestion for a separate nfsv4-wg-mig-rep
mailing list for those who are interested just in the repl-mig
protocol.

	Brent