Subject: RE: V4 complexity (was re: Proposed charter addition)

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 19 Dec 2002 08:41:21 -0800

Neil Brown wrote: 
> On Monday December 16, [email-masked] wrote:
> > Neil Brown wrote: 
> > > 
> > > Given that NFS4ERR_FHEXPIRED is allowed other than for PUTFH, I guess
> > > it doesn't :-(
> > 
> > I'm happy about that.  The thought of having to lock filehandles as
> > they change and unlock that the end of the compound seems like a lot of
> > work to deal with a situation that the client can simply deal with.  Of
> > course, seeing that I don't work on the client, I would think that, 
> > wouldn't I?
> > 
> 
> We must be thinking about this differently (but that isn't a
> surprise).  In my mind the 'current filehandle' is not so much an NFS
> file handle - as in a string of bytes - but is more a filesystem
> handle, such as a Unix filedescriptor.
> PUTFH maps to the file and returns a filedescriptor.  operations
> operate on this filedescriptor.  Then GETFH generates a filehandle
> based on the current filedescriptor.
> It isn't a case of 'locking' the filehandle as much as keeping a
> counted reference on one which doesn't seem very burdensome at all.

I was thinking that you would actually have to prevent the filehandle
from expiring, in which case you have a situation where each request
using it essentially has a shared lock while expiration required an
exclusive lock.  Is that compatible with what you were thinking when
you referred a "counted reference".  If so, this difference may just
be a consequence of me being more miserly about nanoseconds than you.

However if I look at your description I'm wondering if you are intending
something different.  You speak of something like a Unix fd and I
think that has some interesting semantic consequences.

Is it the case that in your model, I can have a long running request
with a current fd that points to a file-handle that another request
see as expired?  Another request might, conceivably, get the expired
error and then the client map the name into a new replacement fh, 
all while another request is acting on the old filehandle and not
getting any expired indication.  Maybe that's OK (and I can't see
anything it would break), but it is kind of strange in that the
handle expiration is not seen by everybody at the same time.

The other related issue is suppose you have a PUTFH - op - GETFH,
for whatever reason.  In your method, you might return a different
filehandle than the one you started with.  Also this strikes me
as strange, even though I don't know of any negative consequences.
Or maybe this isn't what you were saying.  Are you supposing that
the file handle would be copied into the fd?

> > 
> > I think servers have to help clients get this right.  If your server 
> > has volatile file handles, there should be a way to cause the above to 
> > happen more frequently (i.e. once per hour instead of once-per-million-
> > years) so that clients can use that to test.
> > 
> Yes!  A Byzantine server which stretched the limits of the protocol
> would be a great testing tool.

It's much easier to write testing tools that do weird things in the role
of client, each such tool can do one strange set of things, without needing
a whole lot of infrastructure.  For a server, you need to have a lot of 
stuff working or you can't play at all.  I think the best approach for
a Byzantine server is to take a normal server and invest in options that
allow it do weird things of various descriptions.

> > > > >
> > > > > You are immersed in the model and understand it well.  I'm not sure
> > > > > that gives you the best perspective.
> > > > 
> > > > OK.  But I'm having trouble with the idea that not understanding the
> > > > model well somehow gives one a better perspective.  You gotta admit
> > > > that's wierd.  Let's go on though.
> > > 
> > > It's a "Trees and Woods" thing.  You obviously understand the trees
> > > very well, but let me show you what the woods look like (to me at
> > > least). 
> > 
> > OK, the woods look different to you and we can discuss the differences
> > in our views.  I have no problem with that.  I'm not going to say that 
> > my knowledge of the trees guarantees that I right, but I still don't
> > think it hurts.
> > 
> > >
> > > Distinct... but still very similar.
> > > For example: consider
> > >  1/ a successful OPEN_DELEGATE_NONE response to an
> > >  OPEN4_SHARE_DENY_BOTH request
> > > and
> > >  2/ a successful OPEN_DELEGATE_WRITE response to an
> > >  OPEN4_SHARE_DENY_NONE request
> > 
> > > Both contain a promise that no other client has an active open on the
> > > file, and that no other client will be allowed an open until this
> > > client releases it's open.
> > 
> > Not so.  In the second case, the client may well release its open without
> > releasing it's delegation.  It may have multiple processes do a number
> > of open's under the delegation, although normally the client will not
> > send these to the server until it is about to return the delegation.
> 
> But why would a client want to release any open without releasing the
> delegation?  The whole point of a delegation is the client doesn't
> have to tell the serve what it is doing.
> As you say, it "normally" "will not".  So there is no need for the
> protocol to be able to support this.

Assume you have the process-direct-to-server model as v4 does.

So you do an open for process A and get back a delegation stateid and
a stateid for A's open.  I'm not saying this is the only possible way 
to model this, but given the model that V4 chose, this seems a very 
clean way for the protocol to implement this.  I think it makes the
protocol cleanly match the model.  You don't like the model but that's
another story.

So let's go on with case above:

    B does an open on the file.  The client has a delegation and so
    it does nothing except locally does the open.

    Now B does a close.  Because of the delegation nothing is done and
    we have saved an OPEN and CLOSE request.  Cool.

    Now C does open an on the file.  All local.

    Now A does a close.  You could send the close since it might make
    sense to update the server in advance, but you might wonder if A
    is going to open it again, so you do nothing.

    Now A terminates.  At this point you know that at some point you
    will have tell the server that A isn't open any more.  You could
    wait until delegation return and compound the CLOSE with it or
    you could send it now.  In that case the server has a partially
    updated view of the state covered by the delegation.

    Because of a conflict the delegation is recalled.

    The client has to tell the server that C has an open on the file
    (of whatever flavor) and then return the delegation.  So he does
    does an OPEN CLAIM_DELEG_CUR and DELEGRETURN.

It might be possible to make all that more minimal in the protocol but
I don't think it would be any easier to deal with.  At least any way to 
simply the messages without doing violence to the model.

> Let me give you a more concrete picture of what I imagine this part of
> the woods would look like if we could trim away the excess foliage.
> Note that I'm *not* saying NFSv4 should be changed to be like this, I
> think it is too late for that.  I'm presenting this as an elegant
> yet fully functional model which helps to show, by contrast, how
> complicated NFSv4's model.

>  For each client, and for each file, the server maintains precisely
>  one 'state' object which contains share reservations (read/write),
>  sharing modes (deny-read/ deny-write), recallable sharing modes
>  (may-recall-read, may-recall-write), and a bunch of byte-range locks
>  (and possibly some other things).

>  When a client wants to open a file it presents the identity of the
>  file (e.g. directory fh and component name) and some requested
>  sharing reservations/modes.   If this open is allowed, a filehandle
>  and the current reservations/modes/recallability is returned.
>  Note that the modes given may be broader than those requested, but in
>  such cases will be marked as 'recallable'.  This effects a
>  delegation.

>  If the client already has the file open (the client may not have been
>  able to know this due to hard links, but the server is) then the
>  filehandle returned is *guaranteed* to be the same as the file handle
>  returned for the earlier open.  The client is required to check for
>  this and subsequently treat the two opens as one as far as the server
>  is concerned.

>  Subsequently a local application may close a file, or the server may
>  request a recall of a delegation.  If the close only releases access
>  types that have been 'delegated', then no action is needed by the
>  client.  Otherwise it should (possibly flush pending writes and)
>  advise the server what share reservations/modes it requires.
>  i.e. It gives a filehandle and a list of *_SHARE_ACCESS_* and
>  *_SHARE_DENY_* bits as dictated by any current application access.
>  The server will then reply, acknowledging release of some accesses,
>  and possibly delegating some more.

>  In fact, a client without a delegation on a heavily accessed file
>  might periodically use this same request (filehandle+current access modes)
>  in the hope that the server might eventually decide to delegate
>  access to it.

>  The final close uses this same operation, but asserts no access
>  rights.  The server is not permitted to make any delegations in the
>  case where no access is requested.

> In this model, any complexity related to multiprocessing clients, and
> wierd POSIX locking models is kept firmly in the client where it
> belongs.

You don't talk about how this would work for byte-range locking.  I haven't
thought it abut it in detail but I worry about the case of a POSIX client
and non-POSIX (e.g. Windows) server.  It is part of semantics of the 
Windows locking interface that locks are not divisible or mergeable and
I'm wondering how you would deal with this case. 

> > > 
> > > If v4 had no owners, it would indeed be bizarre to have separate stateid's
> > > for the open and the delegation.  But I don't think it would have been done
> > > that way, if that were the case.
> 
> I think somewhere in here is the nub of the problem for me.  
> I have no particular beef with the development process for NFSv4.  A
> lot of new ideas were being tried and integrated into a new protocol,
> and a lot of very good ideas came out of that.  And because of some
> aspects of earlier parts of the design (e.g. lock owners) later parts
> (e.g. delegation) were less clean than they could be.  This is
> perfectly understandable.

They were complicated certainly but I object to "less clean".

> What happened then was:
>  "Oh dear, the deadline is approaching.  We need to get to draft
>  standard.  The protocol will just have to be good enough.  Let's
>  implement this stuff and make it work."
> 
> rather than:
> 
>  "Well, we've learned a lot.  Let's go back to the beginning and see
>  which bits of what we've got can be thrown away and which bits really
>  are good".

I agree that that sort of transaction was active.  I'm not sure that on
this particular issue, the decision would have gone to the client-
intermediated model, even if the issue had been re-opened.  Even though
the protocol is simpler, client implementation would be more complex,
and while I'm OK with that :-), I think many other people would not 
have been.  The result of opening up the issue again may have been 
delay with no change.

> I must admit that I personally lay a lot of blame on the deadline
> (rfc2339) for the way NFSv4 has turned out.  A dead line that, by the
> way, has been extended at least once.

> > If I were to forget my understanding of the protocol, then I might think 
> > this, but it would not be correct to do so.  I would draw the wrong 
> > conclusions.  The big picture is great if it is the correct big picture
> > and my definition of "correct" in this context is "ultimately consonant
> > with the details" (which are real no matter how much you want to denigrate
> > them as unimportant).  I think we have an unresolvable philosophical
> > difference here.  I think the forest *is* the set of trees, the supporting
> > soil, and geology, plus associated fauna and non-tree flora (and lets
> > not forget all the life in all the minor kingdoms, etc.) and the more you
> > know about the details the better off you are.
> > 

> Yes, I think there is a philosophical difference.

> You have a commitment to NFSv4 as it is and so the forest *is* the
> trees and you have to live in it - a best a little pruning is possible.

I have an investment in NFSv4 (as do other people) and I'd like to
see that investment realize some benefits for people, and not make
those hostage to all of our mistakes.  If the protocol is implementable
despite the mistakes and I think it is, then in some case fixing them,
may not be possible.  And if the protocol is simply more complex that 
it would otherwise be, and people know how to deal with that, then there
may be no need to change.

I know we're stretching this metaphor to or beyond the breaking point,
but I just can't seem to help my myself.  (Stop me before I metaphy(?) 
more!).

This is a forest not a garden so the pruning may be significant.  Bring
your chainsaws.  We have to set a limit somewhere short of nuclear
explosions but if the need exists, we can make changes in minor versions.

> I have no such commitment and am interesting in Network File Access in
> general.  I think there is a much better way to build the forest.  By
> transplanting all the really nice trees that you have pruned, but
> leaving behind some of the other trees (and vines and creepers) I
> think we can build a forest which is a much nicer home.

> Ofcourse, I cannot build it myself, and there would be no point in
> living there alone.  But I still think there is value in maintaining
> the dream and sharing the dream.  Who knows ... maybe someone else will
> see it too.

V4 and even V4.x (for all x) is not the last word in network file systems.
I'm also interested in Network File Access in general and want to see
work on other approaches continue.  It would be great if ideas from that
work were eventually to find their way into a V4.x (or V5 someday).

If that is desirable, then I'd recommend starting with V4 and changing the
things that you have a strong desire to change, leave the other things
alone, and see what you come up with.  That makes it easier to do 
implementations and makes your work more assimmilable later.  I'm
not saying that you have to live within the minor versioning rules.
Don't let them compromise your vision, but on the other hand, limiting
your changes to the set of things you think are really important,
avoids a diversion of effort.  Also, this makes it easier to compare
and contrast your fs with V4.

An example of what I'm thnking of is DAFS.  It was very radical in some
ways (RPC/XDR thrown out, COMPOUND on the scrapheap of history, etc.)
but anything that wasn't on its agenda was left in the V4 state (actually
it was RFC3010) including a lot of things you don't approve of but that's
part of the point.  We can see the changes that DAFS made from V4 and
their upsides and downsides more easily because it didn't change everything.  
The result is something that has a lot to offer the V4 community as we go 
forward.

> > > The missing step was:
> > > 3/ Observe that seqid is sufficient for most of what stateid was used
> > >    for, so discard the changing aspect of stateid.
> > > 
> > > Certainly sequencing and replay protection is important. But that
> > > doesn't mean we need two concepts which both provide it.  More on this
> > > below... 
> > 
> > I'm not sure exactly what you plan to do and I don't think that such minor
> > tweaking with what exists is worth pursuing.  V4's problem is that there 
> > are at least five separate mechanisms to deal with replay/sequencing issues:
> > 
> >      The classical reply cache which is not mentioned in the spec but is 
> >      in implementations and takes time but is not reliable enough to be 
> >      the basis for the v4 state-related features.
> > 
> >      The setclientid-confirm mechanism.
> > 
> >      The seqid's.
> > 
> >      The changing stateid's.
> > 
> >      The open-confirm mechanism.
> > 
> > I may have forgotten some.  

> I would add
> 	create mode EXCLUSIVE4 and the createverf verifier.
> it is crucially different to the others, but it is still a replay/sequencing
> mechanism and to some extent should be considered together.

Yes.

> > 
> > I think I've been here before.  I didn't like having these two either.
> > I don't consider that part of the model, but rather part of V4's troubled
> > approach to replay protection.  I tried a lot of things and wasn't able
> > to do significantly better.  I'm not unwilling to believe that you 
> > succeeded where I failed (well maybe a little bit :-)

> This is how I would do sequencing:
>    We already have a concept of a 'client'.  It is represented by a
>    'clientid'.
>    First, allow that we can have several 'clientid's for the one
>    client. A new clientid is allocate (if the server is willing) with
>    something like:
>
>      FORKCLIENTID: clientid -> newclientid
>
>   Each clientid has an associated sequence number.  It starts at 0
>   when the clientid is created, either by SETCLIENTID or
>   FORKCLIENTID.
>   Each clientid  has a cached "reply" - a complete reply to
>   some compound request.

This in general is very interesting to me.  I'm looking for some
solution to the general sequencing/replay problem (possible exception
of open-create-exclusive) that will work in the V4/COMPOUND environment.
DAFS, where I first saw the beauty/value of amalgamating all that,
does things in an entirely different way that wouldn't carry over to
V4.

Let me explain that approach so that you can understand where I'm
coming from.  DAFS is session-oriented and when a seesion is created,
some number of streams are negotiated (this falls out of the necessity 
to flow control rdma) and so each request has a stream number and
a sequence number within the stream.  Then everything is sequenced
as you want.  The replay cache (only used when reconnecting) allows
you to remember one operation for each stream.  When you issue a 
request for a stream, you implicitly recognize that you have recieved
the previous response for the stream so the replay cache information
is strictly bounded.  Then you don't need sequence id in the messages 
or changing state-id or open-confirm.  It's a nice package. 

>   Now, and request can contain a PUTCLIENT operation:
>
>      PUTCLIENT: clientid seqnum barrierflag -> 
>
>   This asserts the the current request is from the given client.
>   If 'seqnum' is greater than the current sequence number of the
>   client id, log a protocol error and drop the request.
>   If 'seqnum' is less than the current sequence number, then:
>       If seqnum is precisely one less, and barrier flag is set,
>          and we have a cached "reply", send the reply.
>       else drop the request
>   If 'seqnum' equals the current sequence number then:
>       Clear the cached "reply"
>       if barrierflag is clear, succeed and perform the remaining operations.
>       if barrierflag is set then:
>		increment the current sequence number
>		perform the remaining operations
>		cache the whole reply with this clientid.
>
>   Now the client has complete control over sequencing anything.  It
>   is not restricted to single threading anything if it doesn't want
>   to. 

I assume that you then don't rely on sequence id within operations.

I don't really understand the barrierflag.  Why would you issue one
of these with the barrier flag off, as opposed to just not having a 
PUTCLIENT?

>    Note that here we are using the power of compound to separate the
>    sequencing from the actual operation, thus making sequencing both
>    simple and uniform.

As a die-hard anti-COMPOUNDer, I note that if you put this information
in the RPC request, you wouldn't need it in COMPOUND.  But given that
we have COMPOUND, it is nice that you can add this feature by using
it.  And it fits very well in the minor versioning model.

>    One interesting issue that comes from combining this with the
>    earlier discussion of state is that:
>    a/ the client needs to sequence all operations on a particular
>       file
>    b/ the client may not know which file a given OPEN request is
>       affecting until it gets the filehandle in the reply.  This
>       makes it hard for the client to sequence the OPEN with, e.g. a
>       separate close.
>
>    To deal with this, the server must enforce sequencing of operations
>    on a given file.  One approach is to track which files have
>    signifcant operations in cached replies, and not to allow further
>    significant operations on them.  Another approach is to require the
>    operation before and after an OPEN use the same clientid as the
>    OPEN.  One way or the other it is fairly easy to address.

Or if you apply this to the current v4 model, then you don't have to
do that sequencing at all.

>
>    Give then existance of PUTCLIENT, FORKCLIENT can just use the
>    'current clientid' so we have:

>      FORKCLIENTID: (currentclientid) -> newclientid
>      PUTCLIENTID: clientid seqnum barrierflag -> (currentclientid)
>      DROPCLIENTID: clientid seqnum

>    DROPCLIENT must be sequenced by some OTHER clientid if there is
>    one, and must give the seqnum that the dropped clientid is upto.

I like this approach.  It fits very well with COMPOUND and minor
versioning.  Since this happen after a client-id is established,
I think that still leaves us with more than one sequencing/replay
mechanism (other than open-create-exclusive):

     setclient-id confirm.

     PUTCLIENT/replay cache.

I think I can live with that though.

> Thankyou for listening and responding.... I appreciate it.
>
> Have a merry christmas,

You too.

---

From: Neil Brown <[email-masked]>
Date: Fri, 20 Dec 2002 15:14:43 +1100

> the file handle would be copied into the fd?

No, it is what I am saying.  I think you understand exactly how I
imagine filehandles would be managed on the server.
If a server used volatile filehandles then
   PUTFH - GETFH
could well return a different filehandle than was put.  Infact the
Linux NFSv4 server would work that way.  You can convince Linux to
include the directory inode in a filehandle so that moving between
directories changes the filehandle.  It should be fairly easy to
demonstrate this with the Linux NFS server and a scriptable client.

> protocol cleanly match the model.  You don't like the model but that's

I think we might be violently in agreement here.
The current model makes a distinction between open and delegation, and
you rightly argue that if the distinction is in the model, then it
should be in the protocol.  I agree.

I'm saying that the distinction does not need to be in the model, in
which case it obviously wouldn't be in the protocol.  I think you
agree that a model in which the distinction is much less strong is
possible, and that with such a model, there would be no place in the
protocol for the distinction.

Then I start talking about the primacy of elegance and we start to
part company.

> I'm wondering how you would deal with this case. 

I think this issue doesn't change between NFSv4 and my vision.
Section 8.2 seems to say that it is up to the application to not try
to do anything that a WIN32 server wouldn't support, but allows for
POSIX clients talking to POSIX server to do what they like, but
doesn't allow the client to ask the server if it POSIX in this sense
(though I guess it could try a sub-range operation and see).

Certainly in my model the client would not merge lock requests from
different application but would keep them separate in case the server
had a WIN32 style locking protocol.

> or changing state-id or open-confirm.  It's a nice package. 

This sound's very like my model, though set in a somewhat different
context.
A 'stream number' maps to a 'clientid'.
Each stream, and each clientid, has a sequence number and a replay
cache of one entry.
As RPC doesn't recongnise the existance of sessions, each request is
effectively it's own session, so the replay cache in my model is
relevant for every request.

> I assume that you then don't rely on sequence id within operations.

No.  seqid in individual operations becomes unnecessary.

> of these with the barrier flag off, as opposed to just not having a 

Having barrierflag not set is the analog of using a stateid but not
changing it, as is done by the WRITE operation (and READ, though
sequencing READ is less significant).

When you send a lot of write requests, you want them to be sequenced
properly between any preceding lock/open/whatever, and any following
unlock/close/etc.  However you don't need any sequencing within them
(though you might if you found a need to over-write a block without
any locking changes).
So PUTCLIENID with barrierflag==FALSE would normally only be used with
WRITE operations.

If you are happy to have lots of clientids per client -  enough to cover
the maximum number of in-flight write requests - then barrierflag
would not be needed.

> it.  And it fits very well in the minor versioning model.

We do have very different perspectives, don't we!!  I think COMPOUND
is great.  It's like down-loading a tiny program into the server to be
run there.  I liked some of the ideas in Brent's original proposal that
were dropped (conditionals etc).  I bet you don't :-)

> do that sequencing at all.

No, because opens are per-lockowner and each lockowner must fully
serialise their requests (which makes sense).
I admit that the extra machinery in NFSv4 (lockowners/stateids) make
this case a little easier to handle.

> I think I can live with that though.

I could probably reduce it to one, but I would only bother if creating
a completely new protocol. :-)

On the open-create-exclusive issue... I've toyed with the idea that
the create verifier could be implicit in the request instead of having
to be explicit.  e.g. it could be a hash of the 'id' in the
nfs_client_id4 structure.  This would be fairly unique among clients -
as unique as a locally generated createverf.

If this were used as a verifier, and if clients always included a
PUTCLIENTID in requests with CREATE, then mkdir, mknod etc could be
implemented as exclusive creates (as they should be) without any
explicit support in the protocol.

NeilBrown