Subject: RE: V4 complexity (was re: Proposed charter addition)

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 16 Dec 2002 10:17:44 -0800

Neil Brown wrote: 
> On Friday December 13, [email-masked] wrote:
> > Neil Brown wrote: 
> ...
> > 
> > But there's also named attribute support, ACL's with a large number of 
> > possible implementation flavors ...
> > 
> > I'm not saying this is an insurmountable problem, but it is an area of 
> > worry.  It would be nice if there were generally accepted definitons of
> > these items and a semi-standard list so that implementations could concisely
> > specify what they supported in a way that made it easy for clients to
> > understand their choices.
> 
> Yes, I agree that it would be useful to have some sort of taxonomy of
> file server features that NFSv4 can work with:
>   - some that are assumed universal ('files', 'directories')
>   - others that are optional
> Arguably this should have been what the working group started with,
> but that is easy to say in hindsight.
> It might even be useful for the client to be able to query the server
> concerning elements of this taxonomy.  Currently the client can
> probably work most of it out, but it isn't obvious.

I had been mainly thinking about such a taxonomy as a way for users
to decide on compatibility between applications on the client and the
support available on the server.  So I was thinking of something that
would be written or on a web page and that you would mainly look at 
before you installed your server or when you configured it.

However, once you have such a taxonomy, making the data available to
the client via a protocol extension seems like a sensible thing to do.

> > > Ask yourself, why does a client *need* to know that a filehandle will
> > > be stable *forever*.  Then ask, exactly what are the stability
> > > requirements on a filehandle for the client to be able to work.
> > > I think the answer is:
> > >  1/ The filehandle needs to be stable at least for the lifetime time
> > >     of a single compound request.  i.e. a filehandle produced by an
> > >     operation in a commound must be recognised by any subsequent
> > >     operation in that compound.
> > 
> > I don't think the protocol provides that guarantee now, does it?
> 
> Given that NFS4ERR_FHEXPIRED is allowed other than for PUTFH, I guess
> it doesn't :-(

I'm happy about that.  The thought of having to lock filehandles as
they change and unlock that the end of the compound seems like a lot of
work to deal with a situation that the client can simply deal with.  Of
course, seeing that I don't work on the client, I would think that, 
wouldn't I?

> > I don't expect it to happen very often but I think clients have to
> > be prepared for this issue.
>
> Bizarre, don't you thing?
> Client:  PUTROOTFH, LOOKUP(usr), LOOKUP(local) LOOKUP(lib) LOOKUP(X11)
> Server: OK OK OK FSEXPIRED
> Client: What now? try again I guess...
> Client:  PUTROOTFH, LOOKUP(usr), LOOKUP(local) LOOKUP(lib) LOOKUP(X11)
> Server: OK OK FSEXPIRED
> Client: Hmm, this could take all day
> Client:  PUTROOTFH, LOOKUP(usr), LOOKUP(local) LOOKUP(lib) LOOKUP(X11)
> Server: Ok OK OK OK  FSEXPIRED
> 
> Obviously this is very unlikely.  But if the protocol allows it
> (which maybe it does) the client should cope.  Does anyone want to
> stand up and say their client will cope with this (assuming that after
> 4 or 5 tries it actually gets success) ?

I think servers have to help clients get this right.  If your server 
has volatile file handles, there should be a way to cause the above to 
happen more frequently (i.e. once per hour instead of once-per-million-
years) so that clients can use that to test.

> > 
> > >  2/ It is helpful for caching efficiency (by not a strong
> > >     requirement) for a filehandle to be stable longer than this.
> > >     There are diminishing returns every time you double the life
> > >     expectancy though.
> > 
> > Diminishing returns but they are substantial for a while, although
> > they get real small somewhat short of "forever".
>
> Certainly.  So it should be encouraged where possible (and I suspect
> any server could provide filehandles that are reasonably stable most of
> the time).  The client should have a right to expect that filehandles
> are mostly fairly stable.  But it never needs to know that they are
> always absolutely stable.

> > 
> > >  3/ A filehandle needs to be stable while a file remains open.
> > 
> > I think we may be able to find some common ground here.  The question 
> > I have is over the definition of "while a file remains open".  
> > Specifically, if I have a file open and the server reboots, is the
> > the file considered "open" for these purposes until either I reclaim 
> > the open or the grace period expires (and that only starts when the
> > server comes up again).  And of course there is a similar issue for
> > migration.
>
> Yep, 'open' stays open until explicit close or lease expiry, server
> reboots not withstanding.  This might mean that sometimes a server
> needs to store a filehandle->file mapping to stable storage, but this
> would be the exception rather than to rule and shouldn't be a
> performance burden.
>
> For migration, I would like to serve to be able to say "This file has
> been moved to that fileserver, and now has filehandle X".  I think
> this would ease the burden of state maintenance across a migration.

I agree.  There has been some discussion that the with the development
of the migration/replication protocol there might be a small set of
changes to the base protocol (via a minor version) to enable better
support for heterogeneous migration.
 
> > Speaking of such migration, a related persistence issue is the 
> > lifetime of fileid.  There is no strong persistence language about
> > fileid, but on the other hand, there is nothing abut changing it
> > either.  API's like stat pretty much allow you to assume that the
> > id stays the same but very few application depend on that (although
> > we may later find some that do subtly in ways we hadn't anticipated).
> > In any case, I think people have been assuming that in heterogeneous
> > migration fileid is going to change with the filehandle, but it
> > isn't clear exactly how that would affect applications.  Maybe this
> > is something that should be nailed down as we work on the migration/
> > replication protocol.
>
> My position is that the concept of a stable unique fileid is an
> unsustainable API.  It just cannot be implemented in general, even
> when just considering local filesystems.
> The best you can do is provide something that is mostly unique and
> mostly stable, and maybe provide some new API to allow a client to
> check if two files are the same if they really need to know.

OK, but we still have stat and programs that use it and the man
pages that tell people writing programs about it, etc.

> I think NFSv4 shouldn't be trying to do this impossible here, but
> rather should make the limits of 'possible' explicit.

Everybody knows that there are limits but getting agreement on
exactly what they are may be difficult.

> > >
> > > You are immersed in the model and understand it well.  I'm not sure
> > > that gives you the best perspective.
> > 
> > OK.  But I'm having trouble with the idea that not understanding the
> > model well somehow gives one a better perspective.  You gotta admit
> > that's wierd.  Let's go on though.
> 
> It's a "Trees and Woods" thing.  You obviously understand the trees
> very well, but let me show you what the woods look like (to me at
> least). 

OK, the woods look different to you and we can discuss the differences
in our views.  I have no problem with that.  I'm not going to say that 
my knowledge of the trees guarantees that I right, but I still don't
think it hurts.

> > 
> > > There are two quite distinct 'stateid's, one for open/locking, and one
> > > for delegation.  Why?
> > 
> > Because a delegation and an open are two distinct things.  You may have
> > any number of opens by different processes (OK, I know you have problems
> > with that concept in the protocol but bear with me a minute), all of
> > which are subsumed within a single delegation. Normally, the server
> > doesn't see these but there are delegation recall situations in which
> > you had better carefully distinguish the individual opens from the
> > delegations that cover them.
>
> Distinct... but still very similar.
> For example: consider
>  1/ a successful OPEN_DELEGATE_NONE response to an
>  OPEN4_SHARE_DENY_BOTH request
> and
>  2/ a successful OPEN_DELEGATE_WRITE response to an
>  OPEN4_SHARE_DENY_NONE request

> Both contain a promise that no other client has an active open on the
> file, and that no other client will be allowed an open until this
> client releases it's open.

Not so.  In the second case, the client may well release its open without
releasing it's delegation.  It may have multiple processes do a number
of open's under the delegation, although normally the client will not
send these to the server until it is about to return the delegation.

The point is that in V4 as it is, we have the concept of owners (processes)
and this causes it to be necessary to distinguish between an open which
applies only a single process (and there may be multiple such for a given
client) and a delegation which applies to the client as a whole.  In V4 as
you would want it, there would be no owners, making the distinction between
these two tiny, just as you are saying that it is.  But that isn't v4 as it
is now and you can't evaluate this piece of the protocol without keeping that
in mind.

If v4 had no owners, it would indeed be bizarre to have separate stateid's
for the open and the delegation.  But I don't think it would have been done
that way, if that were the case.

> The important difference is that in the second case the server
> reserves the right, and the responsibility, to request that the client
> release the server from that  as soon as possible.

But when he releases the delegation, then the open may or may not be present, 
and this will control whether the waiting other client gets the file
he is requesting or a share-conflict error (since the deny-mode of the
open may or may not conflict with the open-mode of the other client).

> I submit that this is the sum total of the important difference between
> the two, and further that this difference is not enough to warant an
> extra stateid being in the second response.

> As you note there are issues with the concept of processes.  It is not
> possible the argue this issue alone as lots of issues in the current
> protocol are tightly intertwined.  But if you forget your deep
> understanding of the current trees for the moment and look to the big
> picture, can you see that important difference between these two is
> worth about one bit (either the server has the right to reclaim, or it
> doesn't)?

If I were to forget my understanding of the protocol, then I might think 
this, but it would not be correct to do so.  I would draw the wrong 
conclusions.  The big picture is great if it is the correct big picture
and my definition of "correct" in this context is "ultimately consonant
with the details" (which are real no matter how much you want to denigrate
them as unimportant).  I think we have an unresolvable philosophical
difference here.  I think the forest *is* the set of trees, the supporting
soil, and geology, plus associated fauna and non-tree flora (and lets
not forget all the life in all the minor kingdoms, etc.) and the more you
know about the details the better off you are.

> > 
> > > There are stateids and sequence numbers that both have to change for a
> > > number of operations.  Why?
> > 
> > Point for my side, I think.  All of the stuff with stateid's changing and 
> > sequence numbers derives in one way or another (often indirectly) from 
> > replay protection.  As to that particular issue, I'd have to research
> > it in detail, but the basic problem is that once you wind up having
> > stateid's change (because of replay protection) and have operations
> > that reference owners (that pesky concept again) be sequenced, again
> > for replay protection, you are going to wind up with a lot of such
> > sequencing.  Whether in any particular case, having them both 
> > changing was overkill or necessary in some case, is hard for me to
> > say without looking at a lot of detail. 

> You claim a point too quickly.  In fact your last sentence gives me the
> point, as with the complexity in the model of the current protocol it
> is 'hard' to say without looking at a 'lot of detail'.

I don't agree here.  No surprise.  I continue to believe that the v4 
model is not as difficult as you make out.  Certainly, it has processes
which yours would not, but while that makes it larger, I don't see the
difficulty.  As to your claim to have won this point, I just don't see
it.  I agree that the replay protection model is terrible consisting of
many separate mechanisms.  In some cases, these result in overkill, and
while I don't like that, I prefer overkill to a situation in which something
is not properly covered.  So, I'm happy, given that this has not been done
right, that people erred on the side of safety.  Nevertheless, it is an
unsatisfactory situation and we agree on that.

> My perception of the development (which could well be inaccurate) is that:
> 1/ The concept of a stateid was developed first.  On the whole it
>    looks like quite a reasonble concept for the need at hand.
> 2/ it was found that it wasn't quite sufficent to handle the
>    sequencing of error cases, so seqid was added.

I wasn't around for this and haven't studied the archives but it is my
impression that seqid has been around from the beginning.  Anybody remember?

> The missing step was:
> 3/ Observe that seqid is sufficient for most of what stateid was used
>    for, so discard the changing aspect of stateid.
> 
> Certainly sequencing and replay protection is important. But that
> doesn't mean we need two concepts which both provide it.  More on this
> below... 

I'm not sure exactly what you plan to do and I don't think that such minor
tweaking with what exists is worth pursuing.  V4's problem is that there 
are at least five separate mechanisms to deal with replay/sequencing issues:

     The classical reply cache which is not mentioned in the spec but is 
     in implementations and takes time but is not reliable enough to be 
     the basis for the v4 state-related features.

     The setclientid-confirm mechanism.

     The seqid's.

     The changing stateid's.

     The open-confirm mechanism.

I may have forgotten some.  There are simply too many mechanisms.  I now
realize that while the protocol works, this is just not supportable in the
long term.  I now know that with a simple mechanism separate from owners
and stateid's, you can provide exactly-once semantics, get rid of changing
stateid's, seqids, open confirm, the classical reply cache, all of that
and give the non-state-dependent stuff better replay protection to boot.
And I know this can be done in a protocol that has a similar state
model except for all the reply/sequencing stuff.  I've seen that other 
forest and those other trees and they are simply more healthy.  If I
knew then what I know now, I would have argued for a single mechanism
until everybody was quite tired of me.   

> > 
> > > There are stateids and filehandles that both uniquely identify a
> > > particular file. Why?
> > 
> > I don't understand the problem you see.  A stateid identifies an 
> > open file.  Another stateid identifies a different open file.  They
> > both point at the same file /x, but they are for different processes
> > (can't avoid it) with different open modes let's say.  So I don't
> > see any problem (or even lack of elegance) in having different sorts 
> > of identifiers for the open file and the file /x.  I'm not saying
> > that it is impossible to come up with a different approach (and
> > yours might be better), but what we have seems a reasonable extension
> > given where we cam from with v2/v3 and nothing that worries me.
> 

> Ok, having two different things (two stateids or a stateid and a
> filehandle) that both refer to the same file might not be a bad thing
> because, as you say, they might include other information (e.g. open
> mode).  But why do you need *both* in the one request?  Why does a
> READ need to have both the filehandle and a stateid?  This is
> redundancy which is not a good thing in a protocol.

I agree in part.  There are two reasons why this is the way it is,
only one of which strikes me as horrible and the other as more excusable.

The horrible reason is the idea (and I use the term loosely) of 
"special"  stateids.  These are stateids (all-zero at least, maybe
we got rid of all-one) which don't designate a file, so in that
case you have to have the filehandle.

The other reason is that in the structure of the protocol, you almost
always have a current filehandle in the environment.  So given that 
you need the stateid, and there will (almost certainly be a filehandle),
it doesn't seem so bad to require that they be the same.  N the
case of a WRITE, it is particularly worrying if the current filehandle
and the stateid designate different files.  It doesn't seem so bad
to me, given the rest of V4, that the implicit parameter and the explicit
one have to match.

> Lets look inside this concept of a stateid for a moment. 
> A stateid combines the concept of
>   - a particular client
>   - a particular file
>   - a particular point in a sequence of requsts from that client and
>     for that file.
> and possibly some aspect of different threads of control within that
> client, which is shared with the 'lock-owner' concept.

Not possibly, actually.  I think you could take process away (see below)
but this stuff is already very hard to deal with and not focusing on
the protocol as it is, and considering on a single change, makes it
even more hard to deal with, and be understood.

> 
> The 'particular file' concept is shared with the filehandle which
> always accompanies a stateid.
> The 'particular point in sequence' concept is share with the seqid
> which often accompanies the stateid.
> The 'particular client' concept is shared with the clientid which never
> (I think) accompanies the stateid.

> Do you not see an inelegance there?

I see some inelegance, yes.

>
> There are appear to me to be two ways to be elegant:
> 1/ Upgrade that stateid so that the sequencing information in it is
> adequate for all tasks, so the seqid can be discarded and also discard
> the filehandle from those operations that take a stateid.
> This acknowledges the strength and centrality of the stateid.
>
> 2/ Downgrade the stateid so that it disappears.  Include the seqid
> everywhere that sequencing is needed. Include that clientid (or
> nfs4_lockowner) explicitly as well, so that the concepts that are
> hidden inside a stateid become explicit.
>
> I think that the second of these would work better in practice.  It
> might appear to include more bytes in a request (clientid and sequence
> number instead of just stateid), but I don't think it is significant.
> 
> [Also, I appreciate that as described here, it appears to require
> complete sequencing of all requests from a given client.  The actual
> design doesn't but I don't want to bore you with details].

I think I've been here before.  I didn't like having these two either.
I don't consider that part of the model, but rather part of V4's troubled
approach to replay protection.  I tried a lot of things and wasn't able
to do significantly better.  I'm not unwilling to believe that you 
succeeded where I failed (well maybe a little bit :-)

My approach to this matter is now somewhat different because of a number
of developments:

     V4 is a done deal and so we have to live with the inelegance,
     at least for a while.

     If this issue is to be addressed, then it must be justified by 
     something other than the desire for greater elegance.

     If this issue is to be addressed, then it must be in the context of
     a minor version, and I don't think a minor version that simply
     modified all the existing ops and replaces them with new ones is
     acceptable.  So the solution space is quite constrained.

     I've come to the conclusion that going from five replay/sequencing 
     mechanisms to four is no big deal.  I think we need to go to one.
     That's what I'd like to have happen in a minor version, if we can.
     I think that issue cannot be the impetus for a minor version but
     if we are going to do one, we should consider fixing what I believe
     is the biggest mistake we made in the protocol.

> > 
> > > And I still haven't found a justification for the concept of 'owners'
> > > or 'processes' being included in the protocol.
> > 
> > The most clear justification is in byte-range locking.  If I (some process)
> > get a lock (shared) on a file and you do too (another process) for a partially, 
> > overalpping range then there has to be some way to distinguish the you and me 
> > so that when I unlock my lock, it's clear whose lock is being unlocked.
> > Similarly with POSIX locking, if I get a lock, then I get the same lock 
> > again it is a no-op, but if you (a different process) try to get a lock,
> > you will eaither get a different lock (if shared) or wait (if exclusive).
> > All of this can happen within the same open due to the wonderfulness of
> > UNIX fork.
> > 
> > I don't see how you can specify locking (even at the level of the model)
> > without reference to the concept of a process.  It might be possible
> > in some locking model but if you have to support the ones that exist
> > out there (and we do), I don't see how you can avoid the concept and 
> > it then winds up mapping into something at the protocol level as
> > well.

> If you and I are processes on a single NFSv4 client, and you have a
> lock and I try to get a conflicting lock, why does the protocol need
> to know about that at all?  My lock should be blocked or denied at the
> client without ever getting onto the wiree.  When you release your
> lock, the client conveys that to the server, and then tries to claim
> my lock now that it sees no local conflicts.  The protocol, and the
> server, just sees one client making locking requests that never
> conflict with themselves.  It doesn't need to know about any
> client-local concept of processes or lock owners.  All that should be
> managed locally inside the client.
> 
> Still no need for the protocol to know about processes.

OK.  I see your point.  I agree that this is not necessary.  If you think
it is desirable for a protocol to be conceptually minimal, then this would
argue that it be removed.

I don't take that position.  Having this in the protocol does not bother 
me.  This is another area in which I don't know the history but I'm going 
to guess that this approach was simply borrowed from NLM.  NLM has a whole
lot of troublesome features but as far as I know the fact that processes
are recorded as the owner of locks, which I'll call that process-direct-
to-server model, has never been a real problem.  So it seemed natural to 
bring it over from NLM to v4.  I think the approach was to take the 
operations that NLM provided integrate them into NFS, changing what 
didn't fit v4 or had been troublesome in the past.  That seems sensible
to me.

If you did things your way, which I'll call the client-intermediated model,
then an enormous amount of existing client locking infrastructure would
have to change, and I don't see any clear benefit to doing that.  With
NLM and V4 today, the client VFS routines either do local locking (local
fs case) or they send an RPC with the rpc depending on which remote
fs we are talking about.  If we did things using the CI model, then v4
locking would be different.  It would use the local locking code (same
sorts of inter-process lock resolution code and in addition do something
different to get the remote locks.  That can be done, but I suppose
people decided that it would be better to avoid having to substantially
modify their locking infrastructures.  That is, if they thought of this
alternate approach at all.

> > >
> > > I think that if the model were elegant, there would be no need for
> > > complexity in the protocol.  It would just fall out as obvious.
> > > This isn't just hypothetical.  I have a model that I think meets all
> > > the needs and is (again, I think) quite elegant.  But it isn't NFSv4.
> > 
> > I'm tempermentally skeptical of the "elegance conquers all" school of
> > system design.  There's too much messy ugly stuff from the past (that
> > pesky users insist on) that one has to carry over.
> 
> Therein lies the challenge of design: to find a model that
> incorporates and generalises all that ugly stuff as best as possible.
> It may not be as elegant as one would like, but I still maintain that
> the more elegance, the better.

All other things being equal, certainly.

> > 
> > Don't get me wrong.  I don't think it was a fatal mistake.  I don't
> > think the protocol is unimplmentable or anything like that.  But I
> > do think that it has made things more complex than they should be
> > and has caused the protocol development to take much longer than it
> > should.  I would say a year.
> > 
> 
> I agree that the protocol is not unimplementable, so it wasn't fatal
> in that sense.
> But there are other benefits from simplicity:  it is easier to argue
> that the protocol is 'correct' and 'secure'.  How sure are you that
> NFSv4 is correct and secure?
> I guess the interoperability testing goes a long way to confirming
> correctness, but it doesn't help much in security.
> Infact I believe there is a security flaw in NFSv4.  It is not a great
> big open hole or anything like that.  It is just a rather
> difficult-to-exploit denial-of-service issue that probably wont be a
> problem in real life, but it is there none-the-less.

I'm all for simplicity.  Who was it that said, "Everything should be as 
simple as possible, but not more so."?  :-)

---

From: Neil Brown <[email-masked]>
Date: Thu, 19 Dec 2002 11:40:51 +1100

> course, seeing that I don't work on the client, I would think that, 

We must be thinking about this differently (but that isn't a
surprise).  In my mind the 'current filehandle' is not so much an NFS
file handle - as in a string of bytes - but is more a filesystem
handle, such as a Unix filedescriptor.
PUTFH maps to the file and returns a filedescriptor.  operations
operate on this filedescriptor.  Then GETFH generates a filehandle
based on the current filedescriptor.
It isn't a case of 'locking' the filehandle as much as keeping a
counted reference on one which doesn't seem very burdensome at all.

> years) so that clients can use that to test.
Yes!  A Byzantine server which stretched the limits of the protocol
would be a great testing tool.

> send these to the server until it is about to return the delegation.

But why would a client want to release any open without releasing the
delegation?  The whole point of a delegation is the client doesn't
have to tell the serve what it is doing.
As you say, it "normally" "will not".  So there is no need for the
protocol to be able to support this.

Let me give you a more concrete picture of what I imagine this part of
the woods would look like if we could trim away the excess foliage.
Note that I'm *not* saying NFSv4 should be changed to be like this, I
think it is too late for that.  I'm presenting this as an elegant
yet fully functional model which helps to show, by contrast, how
complicated NFSv4's model.

 For each client, and for each file, the server maintains precisely
 one 'state' object which contains share reservations (read/write),
 sharing modes (deny-read/ deny-write), recallable sharing modes
 (may-recall-read, may-recall-write), and a bunch of byte-range locks
 (and possibly some other things).

 When a client wants to open a file it presents the identity of the
 file (e.g. directory fh and component name) and some requested
 sharing reservations/modes.   If this open is allowed, a filehandle
 and the current reservations/modes/recallability is returned.
 Note that the modes given may be broader than those requested, but in
 such cases will be marked as 'recallable'.  This effects a
 delegation.

 If the client already has the file open (the client may not have been
 able to know this due to hard links, but the server is) then the
 filehandle returned is *guaranteed* to be the same as the file handle
 returned for the earlier open.  The client is required to check for
 this and subsequently treat the two opens as one as far as the server
 is concerned.

 Subsequently a local application may close a file, or the server may
 request a recall of a delegation.  If the close only releases access
 types that have been 'delegated', then no action is needed by the
 client.  Otherwise it should (possibly flush pending writes and)
 advise the server what share reservations/modes it requires.
 i.e. It gives a filehandle and a list of *_SHARE_ACCESS_* and
 *_SHARE_DENY_* bits as dictated by any current application access.
 The server will then reply, acknowledging release of some accesses,
 and possibly delegating some more.

 In fact, a client without a delegation on a heavily accessed file
 might periodically use this same request (filehandle+current access modes)
 in the hope that the server might eventually decide to delegate
 access to it.

 The final close uses this same operation, but asserts no access
 rights.  The server is not permitted to make any delegations in the
 case where no access is requested.

In this model, any complexity related to multiprocessing clients, and
wierd POSIX locking models is kept firmly in the client where it
belongs.

> that way, if that were the case.

I think somewhere in here is the nub of the problem for me.  
I have no particular beef with the development process for NFSv4.  A
lot of new ideas were being tried and integrated into a new protocol,
and a lot of very good ideas came out of that.  And because of some
aspects of earlier parts of the design (e.g. lock owners) later parts
(e.g. delegation) were less clean than they could be.  This is
perfectly understandable.

What happened then was:
 "Oh dear, the deadline is approaching.  We need to get to draft
 standard.  The protocol will just have to be good enough.  Let's
 implement this stuff and make it work."

rather than:

 "Well, we've learned a lot.  Let's go back to the beginning and see
 which bits of what we've got can be thrown away and which bits really
 are good".

I must admit that I personally lay a lot of blame on the deadline
(rfc2339) for the way NFSv4 has turned out.  A dead line that, by the
way, has been extended at least once.

> know about the details the better off you are.

Yes, I think there is a philosophical difference.

You have a commitment to NFSv4 as it is and so the forest *is* the
trees and you have to live in it - a best a little pruning is possible.

I have no such commitment and am interesting in Network File Access in
general.  I think there is a much better way to build the forest.  By
transplanting all the really nice trees that you have pruned, but
leaving behind some of the other trees (and vines and creepers) I
think we can build a forest which is a much nicer home.

Ofcourse, I cannot build it myself, and there would be no point in
living there alone.  But I still think there is value in maintaining
the dream and sharing the dream.  Who knows ... maybe someone else will
see it too.

> I may have forgotten some.  

I would add
	create mode EXCLUSIVE4 and the createverf verifier.
it is crucially different to the others, but it is still a replay/sequencing
mechanism and to some extent should be considered together.

> succeeded where I failed (well maybe a little bit :-)

This is how I would do sequencing:
   We already have a concept of a 'client'.  It is represented by a
   'clientid'.
   First, allow that we can have several 'clientid's for the one
   client. A new clientid is allocate (if the server is willing) with
   something like:

      FORKCLIENTID: clientid -> newclientid

   Each clientid has an associated sequence number.  It starts at 0
   when the clientid is created, either by SETCLIENTID or
   FORKCLIENTID.
   Each clientid  has a cached "reply" - a complete reply to
   some compound request.

   Now, and request can contain a PUTCLIENT operation:

      PUTCLIENT: clientid seqnum barrierflag -> 

   This asserts the the current request is from the given client.
   If 'seqnum' is greater than the current sequence number of the
   client id, log a protocol error and drop the request.
   If 'seqnum' is less than the current sequence number, then:
       If seqnum is precisely one less, and barrier flag is set,
          and we have a cached "reply", send the reply.
       else drop the request
   If 'seqnum' equals the current sequence number then:
       Clear the cached "reply"
       if barrierflag is clear, succeed and perform the remaining operations.
       if barrierflag is set then:
		increment the current sequence number
		perform the remaining operations
		cache the whole reply with this clientid.

   Now the client has complete control over sequencing anything.  It
   is not restricted to single threading anything if it doesn't want
   to. 

   Note that here we are using the power of compound to separate the
   sequencing from the actual operation, thus making sequencing both
   simple and uniform.

   One interesting issue that comes from combining this with the
   earlier discussion of state is that:
    a/ the client needs to sequence all operations on a particular
       file
    b/ the client may not know which file a given OPEN request is
       affecting until it gets the filehandle in the reply.  This
       makes it hard for the client to sequence the OPEN with, e.g. a
       separate close.

   To deal with this, the server must enforce sequencing of operations
   on a given file.  One approach is to track which files have
   signifcant operations in cached replies, and not to allow further
   significant operations on them.  Another approach is to require the
   operation before and after an OPEN use the same clientid as the
   OPEN.  One way or the other it is fairly easy to address.

   Give then existance of PUTCLIENT, FORKCLIENT can just use the
   'current clientid' so we have:

     FORKCLIENTID: (currentclientid) -> newclientid
     PUTCLIENTID: clientid seqnum barrierflag -> (currentclientid)
     DROPCLIENTID: clientid seqnum

   DROPCLIENT must be sequenced by some OTHER clientid if there is
   one, and must give the seqnum that the dropped clientid is upto.

Thankyou for listening and responding.... I appreciate it.

Have a merry christmas,
NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Thu, 19 Dec 2002 11:15:56 -0800

>counted reference on one which doesn't seem very burdensome at all.

Assuming that we go the route of adding delegations
to directories, we will probably add open
state as well (think of them as share reservations
with deny set always to NONE). At that point
there's little need for persistent file handles
on the interesting file objects (regular files and directories),
because there is a deterministic  way for the client to inform the
server that it no longer needs to refer to the file
handles: CLOSE, lease expire, or SETCLIENTID.
For symlinks, named pipes, sockets, and special devices,
mostly (more on this modifier later) persistent file handles
are still a reasonable implementation choice if for no
other reason in that adding OPEN operations for those
objects is overkill (to me at least).

I say mostly persistent, because persistent file handles
are dead. Unless there are implementors planning
on using the forth coming replication and migration
protocol to work only among homogenous servers, persistent
file handles are dead.

> which bits of what we've got can be thrown away and which bits really

Brent and I ran the NFSv4 BOF in 6 years and 7 days ago, and here we
only one implementation has shipped. Total re-examination would
have resulted in zero implementations shipping. Still, anyone is free to
look at our mistakes, and implement the world's finest file access protocol.

There are in fact lots of efforts doing that: CODA, OpenAFS,
DafsCollaborative, etc.

> way, has been extended at least once.

Your version of history differs from mine. RFC3010 was published as
a Proposed Standard by the extended deadline, per the agreement between
ISOC and Sun. Despite the fact that 3010 was dead on arrival as an
implementable protocol, at that point ISOC had total and exclusive
control over NFS v4.0 and its successors. So IETF and the
nfsv4-wg could have done whatever it wanted. We had rough consensus as
a working group that we didn't want to revisit everything, and instead
decided to fix RFC3010.

The faults of NFSv4 aren't in Sun (Inc.), they are in ourselves.

> trees and you have to live in it - a best a little pruning is possible.

If you took a poll of those of us committed
to NFSv4, and asked us if DCE/DFS or AFS had been opened up before
1996, would we have jumped off the NFS ship, I think you'd get a 
resounding yes.
I felt/feel no great need to reinvent the great  features that
CMU and OSF developed in the 1980s. However, the owners of the DFS and
AFS intellectual property made some very poor choices,
whereas the owners of the NFS ipr made some very savvy choices, so here we
are. It's fabulous that AFS is now open, but it is too late for me.
I'm invested in progress, not perfection.

    -mre

---

From: Neil Brown <[email-masked]>
Date: Fri, 20 Dec 2002 15:35:36 +1100

> objects is overkill (to me at least).

For symlinks there is no concept of OPEN.  For the rest, the open is
entirely implemented on the client and the server plays no part.  So I
think non-persistence is fine for them. 

> The faults of NFSv4 aren't in Sun (Inc.), they are in ourselves.

It is certainly true that laying blame is very hard (not to mention
unproductive) and the any blame is probably spread around a lot.

I remember in late 1999 first reading about the protocol in draft-1,
seeing lots of issues with the protocol and the state model in
particular, and finding it very hard to get anyone to understand the
complexities that I saw.  I felt, possibly incorrectly, that this was
in-part due to a perceived need to get to draft standard in the next 6
months, though maybe the reason is that I didn't express myself very
well.  Over the subsequent years I have seen a number (I didn't count)
of issues causing confusion and being discussed, and the protocol being
tweaked to cope with, that simple would not have arisen in a simpler
model. 

This lead to my frustration exposed in the above laying of blame.

Please accept my apologies if it offended.

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Fri, 20 Dec 2002 10:36:01 -0800

>think non-persistence is fine for them. 

Clients will want to cache the file handles
if they can (and they won't bother if ANY volatile bit is on) in a name
cache. So for performance reasons, the aforementioned objects
ought to be mostly persistent. I understand that for the Linux NFS server,
there are security reasons that dictate the use of volatile file handles.
Given that symlinks are readable by all, and that access control
to the other objects is a client side thing, such security issues shouldn't
apply to the special objects. But I may not understand the Linux issues
correctly or completely.

>Please accept my apologies if it offended.

No offence here; I don't work for Sun.

    -mre