Subject: Comments on draft-ietf-nfsv4-repl-mig-proto-00.txt

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 19 Dec 2002 11:09:22 -0800

RPC/XDR or something else?

First thing is the RPC issue that Mike Eisler brought up.
Unless there is some strong reason to do otherwise, let's follow
the path of least resistance.  We've got enough difficult issues
at the application level, given the range of different filesystems
we are going to deal with, without dealing with transport-level
stuff.  If RPC is really broken in this case for some reason,
then we can go in a different direction, but until someone makes 
a strong case for that, I'd stick with RPC.  

The design refers to the efficiency issue.  It says you don't 
want to send lots of small packets and I agree.  I think the
messages as they are now constructed don't lend themselves
very well to simply being transposed to rpc's.  In fact I would 
argue that they have some efficiency problems even in the 
non-RPC model (security token on every piece).  So I think you 
need a send RPC that allows you send a whole buffer of the tiny
data items that you now have as RPC's.  

I also have problems with the statement, "The use of XDR is also 
subject to change".  I suppose that at this stage everything is 
subject to change, but why call this out?  I'm not a big XDR fan 
and find it kind of annoying for a little-endian server to talk 
to a little-endian client in big-endian, but let's keep things 
in proportion.  This happens in a whole lot of protocols, and 
it would be hard to argue that this one has a greater requirement 
to avoid byte-swaps than all those others.  If someone really 
wants to devote the effort to coming up with an alternative to 
XDR for this, I wouldn't reject it out of hand, but to me this 
"subject to change" language screams out "We're not serious, 
yet.  Don't try to build a prototype because nothing is decided".  
OK, I'll stop ranting now.

Server semantics

Intermittently, there has been discussion of the consequences of the
fact that the spec leaves the "semantics" of the file system unspecified
and up to the server.  One implication of this is that a client may
find one compliant server meets its needs while another does not. 
While an exhaustive list of server semantics and other characteristics
is not possible, we have discussed publishing a "taxonomy" of server
characteristics so that server would have a standard format in which 
to publish their choices, so that clients could more easily check 
suitability up front.  There has also just recently been discussion
of making such information available within the protocol (in a minor
version) for the client's reference (e.g. defining a big structure 
with all relevant characteristics and making that a new read-only 
per-fs attribute).  

It occurs to me that a server-server migration protocol has a 
much greater need for the definition of such a server-semantics 
structure.  First, if your file  system allows renames while
the file is open, for example, then migrating fs's to a server
which does not is liable to seriously discomfit the client.  I'm
not saying the administrator shouldn't be able to override that
but the protocol should make the information available for 
negotiation.  Also, the destination server may be capable of
adopting for export multiple semantic models (by configuration
or other options) so that we should allow the source to tell
the destination server whether it want to prohibit renames of 
open files, for example.  There is a ton of stuff like this.
If my server supports UTF-8 names but only for characters upto
64K, then I'd like the destination to do the same, especially
if I ever want to migrate that fs back.  It is true that there
is a skeletal provision in the strawman for "capability negotiation"
but I think that is focused on capabilities with regard to 
to the data transfer or at least the data to be transferred.
I'm thinking that we need something broader where the main 
focus (at least for me) is the functions of the server with
respect to clients and the need to preserve the semantics the
client's see.

So what I think makes sense here is for someone to gather up 
all of this server-characteristics stuff in a separate draft that
defines a big structure or set of structures.  This could then
be pulled in by the migration/replication protocol as a basis 
for negotiation and by a future minor version as a new attribute.
The obvious thing is to make that someone me as a punishment for
bringing this up :-), but maybe I can avoid that.  It occurs to me that
if there's someone out there who has been interested in getting more
involved in the work of nfsv4-wg but has found a lot of the existing
stuff kind of daunting, this would be an ideal low-barrier-to-entry
kind of thing to do first.  It doesn't have all the complicated history
that the base protocol has and that makes some of the minor version
stuff difficult as well.  Anybody interested in taking this on?
If you might be and want to talk about it, send me some mail and
I can answer any questions you have.

Section 4.4:

It says that this creates a session to send the full contents or an
incremental, but I don't see where the distinction is made.  The
source and the target had better agree which it is.

Section 5.1: 

If we are going the RPC route, as Mike and I think is desirable, then
this breakup of messages does seem to make the overhead a bit high.
Why not have one SEND RPC covering all the current SEND_* messages,
except for maybe SEND_CHECKPOINT.  Then create a union of all the
individual current SEND-type stuff and allow a SEND RPC to send 
a variable-length array of such elements.  That seems a lot better
to me.

If you did that, sec_token and msg_id could be taken out of all of
individual data items.  sec_token would be superseded by the switch to
rpc but msg_id would be in the SEND rpc.

Section 5.2:

In the first paragraph, the ordering constraints need to be clarified.
It isn't clear whether you are saying that all DATA messages have to
happen before all LOCK_STATE messages before all SHARE_STATE messages.
Also, since the filesystem may be changing as data is transferred,
people may want to go back and send changes that happened while
the rest of the transfer was going on.  Not clear how that will
be integrated with the fixed (e.g. inode) order. 

As far as the directory stuff, it isn't clear what algorithm you are
supposing will be used to reconstruct changes.  You talk about 
removes and renames but not creates or links.  I'm supposing that 
somehow, you are gathering the names of objects as the objects
themselves are transferred, and that creates are somehow dealt with
that way, but I don't see how this would work, given what is defined.

I wonder how your treatment of named attributes will work, 
particularly on incremental.  It doesn't seem to have the rename and 
remove capability, so I'm not sure how deletion of a named attribute
will be handled.  If it is handled by simply not sending that 
named attribute, then you would be forced to send all named attributes
for a file any time anything in the file changed (even one byte
or the access time if you are sending those changes).

Section 5.3:

is_named_attr is determinable from the obj_type so why is it there.
obj_acl should be in the attributes and not a separate thing.  The
same argument can be made for obj_type.  It is an attribute.  Note
that fs (rather than file) attributes should not be sent for every
object, so there should be a provision to send this once.  I'm
not clear about file_id.  Is this the same as the fileid attribute,
in which case you already have it once.  I don't understand the
RM_CIFS_ATTR stuff.  The v4 attributes were extended to include a 
lot of CIF stuff.  I think that if we want to put in places for
fs-specific metadata extensions, we should do that, but that should 
be in addition to v4 stuff, rather than as an alternative to it.

I note that the parent directory is not in this message so I don't
know how directory information could be gathered from such messages.
Maybe I just don't understand what is intended here.  In any case, I 
don't think you could add the fileid of the parent (or it would have
to be optional) because many fs's don't have this information when
going through the fs in inode order.

Section 5.4:

You have is_hole, length, and data, which includes the length.  So
you have length twice except in the case of a hole.  Why not have
SEND_DATA which doesn't have length (the data is OK) and SEND_HOLE
which has no data but the length, and is_hole goes away.  I think
that's nicer.

Section 5.5:

There is no definition of RMowner that I can see.  Clearly it contains
the opaque owner string, but what else?  Clientid won't work, will it?
I'd think you'd need the client-string.  There is no provision to send
that stuff over.

RMlock_desc has no provision for the exclusive bit.

It is not clear how it envisioned that this will work for incrementals.
Is it the assumption that all lock state is sent over for a file, if it
has changed?  For example, if I send a set of locks for a file in one
transfer, and then in the next I don't send any information for that
file, what does the destination assume that it still has the same locks?
In if they have all gone away, I send a null set of locks.

Section 5.6:

Basically some of the same issues as in 5.5.

Section 5.7:

I assume the file_id is the file-id of the directory.  It says that this
will cause removal of the designated object.  In the case of hard links,
I assume that actual removal is not meant (i.e. that you only decrement
the link count).  But there is a problem here.  Suppose you made a link
to an existing file and deleted the old link.  If the directory in which 
the old link resided came first in the inode order, then it would have
a zero link-count at one point.  Is it assumed that deletions would happen
at the end of an incremental.  I think that is the only thing that would 
work.

I think the incremental model needs some tightening up at the very least.
Of course, it could be that I don't understand it, in which case I'd argue
that the description needs to be reworked to make it more understandable.

Section 5.8:

This only works for renames within a directory.  How are renames between
directories handled?

Mike mentioned the possibility of SEND_LINK.  Does that have a role in 
resolving the above?

Section 5.9:

The description is confusing.  It should start out with the basic 
function, which I assume is to send the initial directories,
rather than incrementals.  Or am I wrong about that, you have the
names come in with the object itself and say that this is just
for renames/removes.  Are you assuming that somehow directories
will be constructed from the files that are in them?  But I
don't see how that can be done, especially if there are multiple-
linked files.

The second sentence in this section doesn't seem to make sense as
written.

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 19 Dec 2002 14:00:11 -0800

Rob submitted the "Server-to-Server Replication/Migration Protocol
Design Principles" document as an Internet Draft back in June:

   http://www.ietf.org/internet-drafts/draft-thurlow-nfsv4-repl-mig-design-00.txt

But it doesn't appear on the WG charter page and it'll expire
and disappear unless it gets an RFC number.

Any plans to make this doc more permanent or are we
still discussing the requirements ?

	Brent

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 19 Dec 2002 15:14:23 -0700 (MST)

> still discussing the requirements ?

Since I got comments from Mike Eisler on this last week and
since we're still not going strong on implementation work,
we're still discussing contents.  I have some edits which I
plan to submit to the IETF before the Christmas break, and I
would like to publish this as a working group draft, rather
than as a personal draft as I did.  I expect I will submit
this as "draft-ietf-nfsv4-repl-mig-design-00.txt"; the new
name forces the -00 numbering.

Should this be an RFC?  Probably an informational RFC some
day, but it looks a long way from that now.  A requirements
document will help when it's available.

Rob T

---

From: Brian Pawlowski <[email-masked]>
Date: Thu, 19 Dec 2002 14:44:50 -0800 (PST)

Spencer? we need a -01 revision of doc from some comments flying back 
and forth.

Ball is in my court on updating milestones.

---

From: Brian Pawlowski <[email-masked]>
Date: Thu, 19 Dec 2002 14:54:33 -0800 (PST)

> Spencer? we need a -01 revision of doc from some comments flying back 

What I meant to say was: Spencer, anything else Rob needs to do but update
and resubmit -01 doc?

beepy

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 19 Dec 2002 17:41:12 -0600

> document will help when it's available.

Informational and may be combined with the requirements document based
on content and timing.  You need WG chair approval however. :-)

-- 
Spencer

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 19 Dec 2002 17:21:02 -0700 (MST)

> the path of least resistance.

Thanks for the input.  While I recall thinking of a couple
of edge cases RPC might not handle well, the only real
motivation was to reduce overhead, since efficiency was
one of the things we wanted in a separate protocol.

Other folks, e.g. the CITI folks?  What do you think about
RPC forever?

> data items that you now have as RPC's.  

I'd considered just saying, "use IPSec" to solve security.
I chose not to do so because I knew several implementors
who will have more experience with GSS-API/RPCSEC_GSS style
security, and wondered if all vendors had IPSec available
to their purposes.  If we go to RPC and don't turn back,
this is moot, but I had wanted to have the discussion about
what style of security was going to be better.

> subject to change, but why call this out?  ...  this 

I'd had advice to consider just describing bytes-atop-TCP,
and actually tried that; it was a mess, and XDR was a way
to get something expressed reasonably clearly.  The lack
of committment was genuine.

Also, remember that this was a strawman, meant to prompt
discussion.  I expected it to get a few respins before we
got to prototyping.  Sounds like I suddenly have respin
fodder from you and Mike :-)

[Regarding semantics taxonomy]
> structure.

Very much agreed.

> It is true that there

I think we need something broader, as well.  What you see in
the spec is a few examples I knew about, but we need more, and
it should cover anything the client would care about after a
migration.

> for negotiation and by a future minor version as a new attribute.

This sounds good.  Best wishes and fetes to the lucky author :-)

> source and the target had better agree which it is.

I didn't think it was essential to distinguish - why is it?
A hint would be the fs_size vs. tr_size, but we can do better
if need be.

> a variable-length array of such elements.

Yes, this could be good.  Of course, we're reinventing COMPOUND.
I imagine some respins of the "why not just use a big honking
compound NFSv4 op" discussion.

> happen before all LOCK_STATE messages before all SHARE_STATE messages.

Agreed this is poorly worded.  What I wanted was to permit an
arbitrary mix of ops relating to one file after the initial
SEND_OBJECT_METADATA.

Regarding data changing on the fly, we had a discussion at
Sun recently about continuous replication, i.e., if you
leave the connection up, you could propagate changes as
they happened.  I don't know that this needs to be a goal
for Phase 1, but I think we shouldn't preclude this in
the protocol.  Right now, I think this would work, but
we'd be heavy on SEND_OBJECT_METADATA.

> that way, but I don't see how this would work, given what is defined.

Links are missing, and that's just a bug, as Mike noted.  I
did think that creates were an obvious result of a
SEND_OBJECT_METADATA, if the object didn't already exist.

> is_named_attr is determinable from the obj_type so why is it there.

What I expect is that you use SEND_OBJECT_METADATA with
the is_named_attr flag set to "step into" named attribute
space, and then you use SEND_REMOVE and SEND_RENAME as
you would otherwise.  I haven't thought hard about this,
though, and of course haven't specified the latter ops
to take relative pathnames.  What I wanted to avoid was
duplication of a bunch of ops.

> in which case you already have it once.

This all sounds right.

> I don't understand the

See my reply to Mike; I do want fs-specific attribute profiles.
Making V4 mandatory could limit future uses, but I could be
convinced on that.

> which has no data but the length, and is_hole goes away.  I think

Okay.

> I'd think you'd need the client-string.  There is no provision to send

Darn, I missed that, despite compiling with rpcgen.  You're
guessing right about the contents I'd assumed.

> RMlock_desc has no provision for the exclusive bit.

Agreed.

> In if they have all gone away, I send a null set of locks.

Good question - my basis was thinking you'd avoid locks on
the first "bulk data" pass and then lock out changes and
send the locks as they were then, and then migrate.  This
doesn't cover replication, and breaks worst for continuous
replication unless we can note locks being released as well.
Needs work, certainly.

> Section 5.7:
> 
> I assume the file_id is the file-id of the directory.

It's not clear that I meant this to be the directory; too
many cosmic rays since then.  This needs a rethink.  Same
with SEND_RENAME - I don't think I meant this to be limited
to the same directory.

> that the description needs to be reworked to make it more understandable.

Agreed.  Assume my thinking is incomplete on this.

> for renames/removes.

Yes - I thought this was only for renames and only when you
couldn't actually generate remove and rename events - you'd
send the list, and the destination would make directories
which were not listed go away.  Do we need this?  I know I
can't notice remove and rename events on all of our local
filesystems, and saw a need to try to solve the problem.

Rob T

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 19 Dec 2002 17:19:12 -0800

> what style of security was going to be better.

My impression of a replication/migration protocol is that it's
not a transactional protocol, like file access, but a flow
of deltas.  It doesn't make sense to divide up the flow into
arbitrary RPC messages. RPC doesn't add much value here.

I'm not arguing against XDR encoding - that's still appropriate.
A good example of a "flow" protocol that uses XDR is Legato's
Networker product.  It has characteristics that are similar
to this protocol's goals: efficient movement of file data
and metadata across a network between heterogeneous systems.

If it's a flow based protocol, then it makes sense to apply
security to the flow as a whole - not individually to the
deltas within it.  For instance, if you're pushing deltas
from one system to another, the credentials of the processes
that made the changes are long gone.  The only credential of
interest is that of the "superuser" or user authorized to
effect the changes.  I recommend the stream be secured.
There's already a wide choice of suitable methods, TLS,
IPsec, SSH etc.

	Brent

---

From: David Robinson <[email-masked]>
Date: Thu, 19 Dec 2002 22:45:49 -0600

RPC or not?

RPC the protocol is more than capable and does not
have much more overhead than most other compareable
protocols. When doing a streaming protocol with the
ability to recover and restart in the face of errors
or connection loss, you will have effectively what is
in an NFS/RPC header. The biggest cause of inefficency
are the implementations and not the protocol. As an
example, many kernel RPC implementations contain the
full complexity to handle any arbitrary RPC protocol
when in reality they only support one (NFS) maybe two
protocols. Efficient implementations are possible,
and I suspect hardware versions such as Bluearc are
examples. The protocol is not the issue, poor implementations
(especially those starting with the Sun reference source)
are.

The other part of RPC is how the upper level protocol
dictates its use. For many operations like an NFS getattr
it usually expects a response before issuing a subsequent
request. But nothing in RPC prevents a client from
issuing a 100 READ requests before looking for a response
and keeping the pipeline 100 deep. Except poor implementations.

XDR?

Personally I hate XDR, it is a bad mix between a standard
protocol header description (i.e. iSCSI) and a self describing
format that does neither well. If you toss out the esoteric
bits that few protocols use (float, pointers) it is really
just a language to describe a fixed headers in network byte
order. Dropping variable sized objects such as arrays and
unions and the complexity is no greater than that of a standard
protocol like iSCSI. (it is probably possible to describe iSCSI
in terms of XDR) The worst part of XDR is that it allows lazy
protocol design with inefficient layout or hard to automate
encode/decode. Combine that with lousy implementations that
make recursive function calls just to decode 4 bytes and
you will find the root of most complaints.

So while I don't like XDR, if the protocol designers pay
attention to how the bits actually layout on the wire
(e.g. make 8 byte quantities start on 8 byte boundaries)
using XDR as the description language is reasonable.

Given the framework and investment in RPC and already defined
security mechanisms, I would stick with RPC unless someone
can show how it won't meet the needs. Citing bad implementations
should not be a consideration.

	-David

---

From: Mike Eisler <[email-masked]>
Date: Fri, 20 Dec 2002 10:22:24 -0800

Brent Callaghan wrote:

>
> Robert Thurlow wrote:
>
>>> The design refers to the efficiency issue.  It says you don't want 
>>> to send lots of small packets and I agree.  I think the
>>> argue that they have some efficiency problems even in the non-RPC 
>>> model (security token on every piece).  So I think you need a send 
>>> RPC that allows you send a whole buffer of the tiny

For the most part the data transfer protocol is not transactional, but there
are lots of exceptions:

    - session creation
    - session resumption
    - check pointing
    - session termination/aborting

And the current strawman is specified to have a return message for
every request message, which I've already complained about; the SEND_*
messages should be flows, not transactions.

 > arbitrary RPC messages. RPC doesn't add much value here.

We are going divide to the flow into something though. What's the
qualitative difference between dividing it up into a some unspecified
something, or something that is well specified (ONC RPC), and
has no return calls? This has been done for years with the
NLM protocol.

> There's already a wide choice of suitable methods, TLS,

Another argument I've heard in this thread against RPC, or rather sending
lots of little RPC requests versus a single big one is
per RPC request overhead. A cursory examination of
TLS, IPsec, SSH, etc. will reveal that they too break the stream
into chunks (think it through ... how practical is it to perform an
integrity/encryption operation on the entire stream, especially when
you might not know in advance the length of the stream?)
and so they too have their overheads. If one is
concerned about per request overhead, then IPsec is a
lousy choice, since the overhead is per physical packet.
The advantage of IPsec is that there are off the shelf hardware
assists for it. But IPsec is essentially host-based authentication,
and we would still need user to user authentication for this protocol.
TLS doesn't really have it unless you are using user certificates,
or Kerberos authentication. Instead, http over TLS relies on
password authentication ... so user authentication is practically
speaking, outside TLS.

The issues of NFS over fast media and some analysis of
trends in processor speed versus network medium speak
have made me realize that even simple RPCSEC_GSS authentication
is going to be costly for really fast media. Obviously these
issues apply even more so to the data transfer protocol.
I've had some thoughts about what we could do about it NFSv4,
without compromising strong end-user
authentication, but the solutions are applicable to
transfer protocol. I'll send my thoughts in a separate message/thread.

    -mre

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 20 Dec 2002 11:15:11 -0800

If the issue is encoding efficiency and there aren't going to be lots of
octet streams per packet then PER is probably the best encoding.  But a
migration protocol will likely involve sending large numbers of large
data chunks with fairly little header overhead, so that the inefficiency
of XDR as an encoding will hardly matter.  As Mike Eisler points out,
the crypto overhead will be pretty hefty already, so will the encoding
overhead be significant?  (As long as XML is not used I'd say "no" :)

If the efficiency of XDR does turn out to be an issue, couldn't the
encoding be changed later?  Switching to PER would involve using ASN.1
instead of ONC RPC, of course, but the structure of the types might not
change much...  I rather suspect that noone in this WG will want to use
ASN.1/PER though :) , so that leaves ONC RPC/XDR and what else? SSHv2 or TLS
encodings?

As for RPCSEC_GSS performance, it may be possible to use a single
context to provide integrity/privacy for an entire stream, at the stream
level, with each operation referencing a GSS context handle for
authorization on the server side.  No?

Cheers,

Nico

---

From: Mike Eisler <[email-masked]>
Date: Fri, 20 Dec 2002 14:24:11 -0800

>ASN.1/PER though :) , so that leaves ONC RPC/XDR and what else? SSHv2 or TLS

The use of ASN.1 introduces huge barriers to entry.
XDR has an open source, free and *complete*
compiler. Can't say the same thing about ASN.1.
I'm suprised/skeptical to learn that PER/ASN.1 is
faster than XDR, but I don't claim to know
much about either.

>encodings?

SSH and TLS define an encoding language for themselves,
but not for their payloads. If there are SSH and TLS hardware
accelerators, then that's a point in their favor. If
not, what's the point?

>authorization on the server side.  No?

That's what I have in mind. I'll follow up today.

    -mre

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 20 Dec 2002 14:27:33 -0800

> faster than XDR, but I don't claim to know

I'm not proposing that ASN.1 actually be used for this protocol, but
merely pointing out that there aren't a lot of alternatives.  I don't
think a pure "bytes-on-the-wire" specification is good enough, though a
TLS or SSH-like approach is quite reasonable.  And ONC RPC / XDR is not
unreasonable, IMO.

> accelerators, then that's a point in their favor. If

*nod*

> That's what I have in mind. I'll follow up today.

Good, very good.

Nico
--