Subject: RE: Proposed charter addition for NFS Version 4 Working Group

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 11 Dec 2002 12:36:53 -0800

R> J. Atkinson wrote: 
> On Wednesday, Dec 11, 2002, at 12:10 America/Montreal, Noveck, Dave 
> wrote:

> > This argument, that we should avoid any additions to NFS-v4
> > because the protocol is too complex, would also apply to
> > almost any minor versioning work, and the charter already
> > includes that.
>
>
> Yes.  And please lets not make a bad situation worse.

OK.  But I still don't understand the relationship between 
your view about that the complexity of NFS-v4, which if I'm not 
mis-stating it is that some very bad initial design decisions
were made leading to excessive scope and thus complexity, and,
your view about what is required to support much higher speeds
than are current today ("clever implementations").

Put another way, if all of the V4 design decisions had gone the 
way that you prefer, it seems to me that you would still oppose
adding RDDP support.  You simply don't buy the RDDP argument, as
some people don't.  We could have that argument again, but it
isn't really possible to resolve finally since the range of possible
clever implementations is pretty large (A says X might work, B
says that won't work because of Y, repeat ad nauseum).

> > As far as the complexity of the existing protocol, let me try
> > to defend it, without getting defensive.  I believe that the
> > complexity of the protocol (in terms of pages of spec let's
> > say) flows from the requirements and that to get something
> > significantly simpler, you would need to have signficantly
> > reduced requirements.
>
> I question whether the intent of the charter was to change NFS
> from a UNIX/POSIX-centric file system model to a super-generic
> file system model (e.g. one that is designed for a widely used
> proprietary file system).  And that's a long standing doubt of mine,
> not something new.

I'm at a disadvantage here, as I've never examined the charter
in the depth necessary to argue the point.  When I first started 
working with V4, share reservations and named attributes, to cite 
two examples, were already part of the protocol and while those
did add to complexity, I considered them part of the landscape.

If we had decided otherwise, then things would have been 
different, and the protocol might have been done sooner.  Some
people would have liked that result while others would have 
thought that things were missing.  So it seemed to me at the
time that there was a consensus on this point.  Apparently,
that consensus did not include you.  However, if we had tried
to go the other way, there would definitely not have been a
consensus, so it might be that the protocol would still have
been quite delayed.

> > NFS-v4 added a great deal of complexity by amalgamating two
> > fairly different paradigms of file access: UNIX/NFS and NT/CIFS.
> 
> This was a big mistake.  All the major NFS server vendors dealt
> with that issue by offering both NFS service and NTFS service
> (separately, but using the same backend storage).  That latter
> approach is both widely successful and a better strategy than
> trying to build a Ginzu knife protocol, IMNVHO.

As someone whose company has done a lot in the development of
that very strategy, I'm not going to argue that it is bad.  But
I think it would have been very difficult for the IETF to develop
a file access protocol and adopt the approach that "windows-derived
file access features need not apply".

The fact is that this decision (mistaken in your opinion) was made.
The question is "What do we do now"?  If that decision has fatally
compromised the v4 protocol (which I certainly don't believe,
no matter how I would have made that initial decision), then
I would think that the answer would be to rip it up and start
again on an IETF UNIX/POSIX file access protocol (certified Gates-
free if you will).  Instead your answer is to keep the protocol
exactly as it is.  That would seem to be a way to prolong the 
misery.
  
> >  Each protocol would itself
> > be simpler than NFS-v4 but that would push the complexity onto
> > the users and administrators that would have to deal with
> > multiple protocols.
>
> As a user and administrator, NFSv4 has not bought me *anything*
> in the way of simplicity.  I've already got to deal with both
> NTFS and NFSv3.  NFSv4 increases my complexity with no substantive
> benefits to me as a user or administrator.  This isn't the first
> time I've said this, and I think I might be the only user or 
> administrator
> in this WG, but the WG is pretty consistent in picking complexity
> over simplicity.  Simplicity and ships-in-the-night filesystems
> would make my life a lot better.

>From reading the above, what I hear is that you are OK with dealing
with CIFS and NFS-v3 and that you feel that v4 can only hurt.  So
I'm curious about what it is that you think should be in NFS-v4.
Your list would be different than other peoples' certainly.  The
question is how to resolve that discrepancy.  Putting in only
things on everyone's lists would make v4 the same as v3.  Putting
in anything that was on anyone's list would make v4 even bigger than
it is today.  We did something in-between.  You can always argue
that it would have been better to make other choices and if we adopted
your list, other people would disagree with those choices as well. 

I can't say whether you're the only administrator, but I think
almost everybody on this list is an NFS user.  Will everybody who
doesn't use NFS, please sound off?  Also, any other administrators. 

> > The other area (and I believe this has a closer relationship to
> > the RDDP issue) where the single-protocol approach adds complexity
> > to NFS-v4 is that v4, while maintaining its function as a file
> > access protocol in high-bandwidth low-latency (essentially in-
> > machine-room) environments, was also intended to provide good
> > performance in high-latency (or even very-high-latency) environments
> > and that is challenging.
> 
> NFSv3 is doing a fine job of both for me.  Any my machine room is loaded
> with high-capacity/low-latency GigE links since we make IP/GigE
> switch/routers.  I don't see how folks can substantively argue against
> an existence proof, which is what I have.

I recognize that NFSv3 works fine at the level of gigE.  Maybe this wasn't
clear.  When I said "v4, while maintaining its function as a file
access protocol in high-bandwidth low-latency (essentially in-
machine-room) environments", I should have said "maintaining NFS's
function".  I recognize that v3 works fine here although I continue
to believe that 10 gig-E is different and will require protocol support
such as is provided by RDDP.

The point about V4 and complexity is that when you attempt to 
span such a wide range of different environments, as V4 does, there
is additional complexity, although I don't find it as troublesome as
you do, and since I've actually had to do a lot of our V4 implementation,
it's not like I'm able to ignore the complexity.

> > So as to the question of the need for this (the NFS-v4 RDDP work), it 
> > turns
> > on the question of the need for very high-speed file access over IP, 
> > equivalent
> > to fs's on traditional direct-connected disk.
> 
> A neat trick is how you re-defined the problem so that operator/user
> considerations were taken out of the equation.  I'd say that the 
> operator/user
> needs to be able to deploy/operate this thing successfully are a lot 
> more
> important than a hypothetical concern about high-performance situations.

I intended no such trick.  I think you can argue about whether high-performance
or deployability/operatibility/maintainability is more important.  But it
really doesn't make any difference.  A system needs good performance and
deployability.  If performance is inadequate, good deployabilty doesn't 
help.  If a system is not deployable, good performance doesn't help.

The point is that if V4 is so complex as to be non-deployable (and I don't
think that is the case and do think this is an area where clever implementations
can make things easier for administrators), then we have to fix that problem.
It is not a reason to avoid consideration of performance issues.

I don't consider 10 gig-E at all hypothetical.  Like death and taxes, faster
networks will arrive, whether we'd like that to happen or not.

> And I don't agree with your analysis of the issue.

I recognize that.

> I also don't think that IETF needs to standardise zero-copy 
> implementation
> methods.  It did not need to do so with TCP.  It does not need to do so 
> now.
> Clever implementers will figure out a way to minimise those costs.  Less
> clever implementers might not.  Either way, the protocol doesn't need 
> to change.

This is again the argument about whether the IETF needs to do the RDDP
work.  I don't think there is much point about arguing clever implementations
and whether they would work (and if had one that worked I'd be running to 
the patent office and not discussing it with the working group :-). 
I don't :-(

But even if a suitably clever implementation is possible, I don't see the
value in absolutely avoiding RDDP.  The I-want-to-read-the-data-and-my-buffer-
is-here-so-when-you-read-the-data-please-put-it-directly-in-my-bufffer
paradigm, seems like a good thing to me, so if the IETF has a protocol for
that, then using it for one of the major applications of networks in which
data is read into buffers at high speed just seems like good sense, as far
as I'm concerned.

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 11 Dec 2002 17:36:55 -0500

On Wednesday, Dec 11, 2002, at 15:36 America/Montreal, Noveck, Dave 
wrote:
>  So it seemed to me at the

Oh, make no mistake, the WG has always conformed to the IETF's
rough consensus process rules.  I have no process complaints
with NFS WG to date.  Mind, it also seems to be the case that
I'm the only operator/user person participating here, which
might account for my views being outside the mainstream of
the participants (as different from lurkers) in this WG.

> file access features need not apply".

Oh, I think that would have been trivially simple.

> exactly as it is.  That would seem to be a way to prolong the

Actually, my current plan is to avoid deploying NFSv4 as long as
I can -- continuing to ride NFSv3 as long as I'm able to do so.
I do *hope* that I'm wrong in my estimation of the operational
complexity -- but I'm unable to convince myself that I'm wrong
on that (so far).

> I'm curious about what it is that you think should be in NFS-v4.

Not clear to me that we needed a major revision in NFS, other than
maybe to tidy up the security loose ends.  Hyperbole aside, it
is convenient that NFS traffic has better security options in NFSv4.
Unfortunately, most security that is actually secure also inflicts
operational pain.  I've never figured out a way to avoid the pain
when deploying security.

> doesn't use NFS, please sound off?  Also, any other administrators.

More data is helpful certainly, but I fully expect the conclusion
to be the same as in the past:
	the protocol folks want to go tweak the protocol
	My opinion is an outlier, relative to the bulk of active participants 
in the WG

> It is not a reason to avoid consideration of performance issues.

I'm not ready to say "entirely non-deployable".  I am ready to say that 
it
looks more painful to deploy.  Now implementer folks can sometimes hide
some of the pain, and I encourage that.  I hope I'm wrong.

> networks will arrive, whether we'd like that to happen or not.

We've shipped 10 GigE, so I'm not claiming that it won't get deployed.
I am questioning whether it is necessary to change the NFS protocol
to make NFS work well over 10 GigE.  At least one other person today
indicated that he does not think it is necessary to change the NFS
protocol to make NFS work well over 10 GigE (he might have meant that
changes could be confined to RPC rather than NFS itself, but his
precise meaning was not crystal clear to me).

Ran
[email-masked]

---

From: Neil Brown <[email-masked]>
Date: Thu, 12 Dec 2002 12:35:06 +1100

> doesn't use NFS, please sound off?  Also, any other administrators. 

Well, seeing you asked.  I'm an administrator (and user and
developer..)

As an administrator I want my protocol to 'just work'.   I should be
able to configure all sorts of interesting things, but I shouldn't
have to in most cases.
I should be able to expect that different implementations will
inter-operate seamlessly, and the each implementation will be
bug-free.   (No great surprises here).

How does that relate to complexity of the protocol and the possible
consideration of RDDP??

Well, as an developer I know that correctness and inter-operability of
implementations is closely linked with complexity, but more with the
complexity of the conceptual model than with the complexity of the
protocol. 
A typical programmer will have a much better chance of writing correct
code if they can understand the protocol - if they can hold a
model of the protocol completely in their head.
This may allow that the protocol has lots of features and be complex
in that sense, but it requires that the features in the protocol have
a clean, elegant, uniform model.

My assessment of NFSv4 as it stands is that the complexity of the
protocol is reasonable, but that the complexity of the model is
terrible.  It is terribly in-elegant.  Particularly in the state
model and the filehandle model, but also in other areas.  As such I
don't have a great deal of confidence that implementors will get it
right.  But I don't think there is any hope of major improvements.  I
just have to trust that we have a bunch of exceptional programmers.

But how does this relate to RDDP?

Well I prefer to think of this (assuming that I understand it
correctly) as a special case of protocol-offload.  The idea is to
teach the NIC enough about the particular protocol that it can do some
of the labourious tasks for you.  These can include IP fragment
assembly,  Checksum verification, Possibly crypto seal/unseal/
sign/verify.  The particular part of protocol-offload that RDDP
relates to is identifying potions that need to be page-aligned and
DMAing them into page-aligned buffers.  Obviously this could be done
without any change to the protocol: the NIC could simply be taught
enough of the intricacies of NFSv4 (it already will need to know some
of IP and TCP (or SCTP) and RPC.  Equally obviously, a little help
would go a long way: if the NIC can ignore most of the protocol detail
and get to the bits that it needs quickly, then it will work better.
[Though my understanding is that with NFSv2/v3 it is quite possible to
guess where the data is because a) it is always at the end of the
datagram, b) it is 4byte aligned and c) if it isn't a large power-of-2 in
size (i.e. 4096, 8192, 32768) then it probably isn't worth optimising.]

The question of the moment is: would the changes to the protocol to
make protocol-offload easier (and hence cheaper and more common) make
it more complex to implement.  i.e. would it make the conceptual model
more complex and so make the protocol harder for the average
programmer to understand and work with effectively. 

I think that the answer is definately "NO".  By their very nature,
these changes must be very simple so that even a NIC can understand
them at very high speed.  So it seems very likely that they will
 a/ be quite orthogonal to the rest of the protocol, and
 b/ be very localised and direct.
with leads me to feel comfortable that they will probably be
implemented correctly.

As for other issues raised in this discussion:
 1/ I fully agree with all Dave's arguments for have just one protocol
    and getting it to meet all needs.
 2/ While, in keeping with the above, I am happy for NFSv4 to include
    support for WIN32 APIs, I doubt that it will really be very
    relevant. I don't see Microsoft including an NFSv4 client as
    standard with a Windows release anytime soon (please correct me if
    I am wrong) so it will be easier for an admin to run Samba (or
    some other CIFS server) alongside NFS than to roll out some third
    party NFS client onto all those Windows boxes.
    Which will come first: NFSv4 being a must-have for Windows, or 
    Windows becoming irrelevant. :-)

NeilBrown