Subject: V4 complexity (was re: Proposed charter addition)

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 12 Dec 2002 08:59:35 -0800

Neil Brown wrote: 
> On Wednesday December 11, [email-masked] wrote:
> > 
> > I can't say whether you're the only administrator, but I think
> > almost everybody on this list is an NFS user.  Will everybody who
> > doesn't use NFS, please sound off?  Also, any other administrators. 
> > 
> 
> Well, seeing you asked.  I'm an administrator (and user and
> developer..)
>
> As an administrator I want my protocol to 'just work'.   I should be
> able to configure all sorts of interesting things, but I shouldn't
> have to in most cases.
> I should be able to expect that different implementations will
> inter-operate seamlessly, and the each implementation will be
> bug-free.   (No great surprises here).

One protential issue is that the protocol, in order to accommodate
many possible implementations leaves the semantics unspecified, as
not part of the protocol.  For example, a NFS-v4 server may reject
rename of an open file or may allow it.  It's hard to fault this
position, since such behavior is, after all, not part of the protocol,
but it does pose interoperability issues.  There are a large number of
possible implementation choices and so substituting one fully-compliant
v4 server with another may result in totally different behavior,
causing the client application to fail.

In practice, at least initially, V4 servers will implement the NFS/UNIX
semantics we know and love/hate/are-ambivalent-about.  But when there
are differences, the protocol's interoperability story will suffer.
I'd like us to do something to forestall this.  One idea is to list
all the possiblities we can think of, so a server vendor can provide
his choices in a coherent way, and the client can see if there are
differences to worry about.  This might be something for that never-
quite-started implementation RFC.  

> How does that relate to complexity of the protocol and the possible
> consideration of RDDP??

> Well, as an developer I know that correctness and inter-operability of
> implementations is closely linked with complexity, but more with the
> complexity of the conceptual model than with the complexity of the
> protocol. 

> A typical programmer will have a much better chance of writing correct
> code if they can understand the protocol - if they can hold a
> model of the protocol completely in their head.
> This may allow that the protocol has lots of features and be complex
> in that sense, but it requires that the features in the protocol have
> a clean, elegant, uniform model.

If one of the goals was to accommodate windows semantics, than a 
clean, elegant, uniform model, would not be acceptable.  Sorry.  
Couldn't resist.

> My assessment of NFSv4 as it stands is that the complexity of the
> protocol is reasonable, but that the complexity of the model is
> terrible.  It is terribly in-elegant.  Particularly in the state
> model and the filehandle model, but also in other areas.  As such I
> don't have a great deal of confidence that implementors will get it
> right.  But I don't think there is any hope of major improvements.  I
> just have to trust that we have a bunch of exceptional programmers.

Yes, but but "exceptional" has come to have such a range of conno-
tations :-)

I'm not sure what you mean about the filehandle model.  Are you 
referring to volatile filehandles?  If so, I'm in agreement.  I'm
definitely in the "Peristent filehandles -- accept no substitutes"
camp.

As to the state stuff, I don't think it is the state *model* that
is complex.  The state model is that there are a set of clients
which identify themself, get an id.  Then, you have individual 
owners (i.e. processes) which open files, and get an id to 
identify that open file.  Multiple owners (processes) can get
locks on the same open file and the processes each get an id
that identifies the state of their particular locks on the
file.  Open-upgrade/downgrade is less than elegant but overall,
the model is pretty clean.

The complexity comes, not in the model, but in the protocol
specifics.  Most of that is in some form of replay protection.
Sequence numbers, per-object last request info, the OLD_STATEID
stuff, some object recycling and confirmation complexities all
derive from the fact that you are doing stuff that is stateful and
the old NFS-v2/v3 at-least-once model of things is inadequate.
That is the one problem that is solved (at least I hope it is 
solved) in a half-dozen (at least) different places in various
different ways.  The problem that NFS-v4 faced was that NFS
implementations had a solution for replays, that did not work
well enough (i.e. that you could specify it in the protocol and
show it worked) to adopt for the state stuff, but did not work
poorly enough to be gotten rid of (and replaced with something
else).  Not biting the bullet and taking on this issue is the
biggest mistake we made, and we can't blame it on the charter
(at least I don't think we can).

I've come to this conclusion after some experience of another
approach.  The DAFS locking model is very similar to that of V4 
(actually it is currently RFC3010-based), but it has exactly-
once semantics.  DAFS and V4 share the wafl code that handles 
the locking model, and we have tried to separate out the code
that is v4-specific.  The result is that I have a very
visible indication (via an if) where a major section of 
code can simply be avoided since replay protection is somebody
else's job (the overall protocol by a separate mechanism
rather than something for the locking code to do).  This has
given me a severe case of exactly-once-semantics-envy.   

One thing that we need to do to deal with v4 complexity is more 
sophisticated interoperability tests, including looking at forcing
pathologoies.  At the last bakeathon, we first tried the connectathon
tests while rebooting the server.  Let me propose soemthing else
for the upcoming connectathon.  Let's try the connectathon tests
while randomly dropping replies.  To avoid not testing very much,
let's make sure that our NFS reply caches do not cache the state
management stuff (we'll need to finesse open-create somehow) so
that the code within the v4 state management that is supposed to
deal with cases in which the reply cache doesn't, has something 
substantive to do.  I expect this to be a productive exercise
(in terms of bugs found), but if not, the resulting mental
serenity is nothing to sneeze at.

---

From: Neil Brown <[email-masked]>
Date: Fri, 13 Dec 2002 11:38:29 +1100

> causing the client application to fail.

As I have said before (and was disagreed with), NFSv4 is a protocol,
not filesystem.
Saying a server is fully NFSv4 complient says lots of usefully things,
but it doesn't tell me everything about the server.
It tells me that my NFSv4 client will be able to communicate effectively with
the server, and that it will be able to access all the functionality
that server provides and that my client understands.  But it doesn't
tell me what semantics that server provides.
So I might want to know that a server is:
   Fully compliant NFSv4 with WIN32 semantics
   Fully compliant NFSv4 with POSIX semantics
   Fully compliant NFSv4 with per filesystem configurable semantics
   Fully compliant NFSv4 with POSIX semantics, journalling, and hourly snapshots.
etc.
Any one of might be a must-have for some people, and un-interesting to
others.
The value of NFSv4 is that
 a/ I can get lowest-common-denomitor semantics (which meets 90% of
    needs) with any client and any server.
 b/ I can get precise semantics with carefully selected client and
    server. 

So I could have POSIX and WIN32 clients spread around, and a few
different servers, some POSIX, some WIN32, some with special
features.  And every client could access every server, but where there
are special needs, I can make sure the need, the client, and the
server line up without using a different special-purpose protocol.

> definitely in the "Peristent filehandles -- accept no substitutes"

On the other hand, I am definately of the "Persistence is completely
un-necessary, throw it out" camp..  However a consequence of that is that
you need to put more weight on stateids to identify a file that is
open... but we have been there before (statehandles).

Ask yourself, why does a client *need* to know that a filehandle will
be stable *forever*.  Then ask, exactly what are the stability
requirements on a filehandle for the client to be able to work.
I think the answer is:
  1/ The filehandle needs to be stable at least for the lifetime time
     of a single compound request.  i.e. a filehandle produced by an
     operation in a commound must be recognised by any subsequent
     operation in that compound.
  2/ It is helpful for caching efficiency (by not a strong
     requirement) for a filehandle to be stable longer than this.
     There are diminishing returns every time you double the life
     expectancy though.
  3/ A filehandle needs to be stable while a file remains open.

I think that a client has no use for any more guarantees that those
(and 2 isn't really a guarantee), and so the protocol should not
provide any more.  A server certainly can if it wants to, but who
cares. 

> the model is pretty clean.

You are immersed in the model and understand it well.  I'm not sure
that gives you the best perspective.

There are two quite distinct 'stateid's, one for open/locking, and one
for delegation.  Why?
There are stateids and sequence numbers that both have to change for a
number of operations.  Why?
There are stateids and filehandles that both uniquely identify a
particular file. Why?

And I still haven't found a justification for the concept of 'owners'
or 'processes' being included in the protocol.

> (at least I don't think we can).

I think that if the model were elegant, there would be no need for
complexity in the protocol.  It would just fall out as obvious.
This isn't just hypothetical.  I have a model that I think meets all
the needs and is (again, I think) quite elegant.  But it isn't NFSv4.
Every so often I think I should write it up and implement it and call
it NeilsFileSystemV1, but I doubt it woule be worth the effort.

As for the 'not biting the bullet', I agree with that bit.

>                                                     This has

exactly-once-semantics that is uniform across all filesystem
operations that need it is a feature of my model.  I agree that
it is very valuable.

NeilBrown