Subject: RE: V4 complexity (was re: Proposed charter addition)

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 13 Dec 2002 08:33:40 -0800

Neil Brown wrote: 
> On Thursday December 12, [email-masked] wrote:
> > Neil Brown wrote: 
> > > On Wednesday December 11, [email-masked] wrote:
> > > > 
> > > > I can't say whether you're the only administrator, but I think
> > > > almost everybody on this list is an NFS user.  Will everybody who
> > > > doesn't use NFS, please sound off?  Also, any other administrators. 
> > > > 
> > > 
> > > Well, seeing you asked.  I'm an administrator (and user and
> > > developer..)
> > >
> > > As an administrator I want my protocol to 'just work'.   I should be
> > > able to configure all sorts of interesting things, but I shouldn't
> > > have to in most cases.
> > > I should be able to expect that different implementations will
> > > inter-operate seamlessly, and the each implementation will be
> > > bug-free.   (No great surprises here).
> > 
> > One protential issue is that the protocol, in order to accommodate
> > many possible implementations leaves the semantics unspecified, as
> > not part of the protocol.  For example, a NFS-v4 server may reject
> > rename of an open file or may allow it.  It's hard to fault this
> > position, since such behavior is, after all, not part of the protocol,
> > but it does pose interoperability issues.  There are a large number of
> > possible implementation choices and so substituting one fully-compliant
> > v4 server with another may result in totally different behavior,
> > causing the client application to fail.

> As I have said before (and was disagreed with), NFSv4 is a protocol,
> not filesystem.
> Saying a server is fully NFSv4 complient says lots of usefully things,
> but it doesn't tell me everything about the server.
> It tells me that my NFSv4 client will be able to communicate effectively with
> the server, and that it will be able to access all the functionality
> that server provides and that my client understands.  But it doesn't
> tell me what semantics that server provides.
> So I might want to know that a server is:
>   Fully compliant NFSv4 with WIN32 semantics
>   Fully compliant NFSv4 with POSIX semantics
>   Fully compliant NFSv4 with per filesystem configurable semantics
>   Fully compliant NFSv4 with POSIX semantics, journalling, and hourly snapshots.
> etc.
> Any one of might be a must-have for some people, and un-interesting to
> others.

But there's also named attribute support, ACL's with a large number of 
possible implementation flavors ...

I'm not saying this is an insurmountable problem, but it is an area of 
worry.  It would be nice if there were generally accepted definitons of
these items and a semi-standard list so that implementations could concisely
specify what they supported in a way that made it easy for clients to
understand their choices.

> The value of NFSv4 is that
>  a/ I can get lowest-common-denomitor semantics (which meets 90% of
>     needs) with any client and any server.
>  b/ I can get precise semantics with carefully selected client and
>     server. 

> So I could have POSIX and WIN32 clients spread around, and a few
> different servers, some POSIX, some WIN32, some with special
> features.  And every client could access every server, but where there
> are special needs, I can make sure the need, the client, and the
> server line up without using a different special-purpose protocol.

So I don't expect you to have any problems here.  Am I wrong to worry
that some other people will?

> > 
> > I'm not sure what you mean about the filehandle model.  Are you 
> > referring to volatile filehandles?  If so, I'm in agreement.  I'm
> > definitely in the "Peristent filehandles -- accept no substitutes"
> > camp.
>
> On the other hand, I am definately of the "Persistence is completely
> un-necessary, throw it out" camp..  

Oh.  One of *those*.  I'll be on my guard :-)

> However a consequence of that is that
> you need to put more weight on stateids to identify a file that is
> open... but we have been there before (statehandles).

> Ask yourself, why does a client *need* to know that a filehandle will
> be stable *forever*.  Then ask, exactly what are the stability
> requirements on a filehandle for the client to be able to work.
> I think the answer is:
>  1/ The filehandle needs to be stable at least for the lifetime time
>     of a single compound request.  i.e. a filehandle produced by an
>     operation in a commound must be recognised by any subsequent
>     operation in that compound.

I don't think the protocol provides that guarantee now, does it?
I don't expect it to happen very often but I think clients have to
be prepared for this issue.

>  2/ It is helpful for caching efficiency (by not a strong
>     requirement) for a filehandle to be stable longer than this.
>     There are diminishing returns every time you double the life
>     expectancy though.

Diminishing returns but they are substantial for a while, although
they get real small somewhat short of "forever".

>  3/ A filehandle needs to be stable while a file remains open.

I think we may be able to find some common ground here.  The question 
I have is over the definition of "while a file remains open".  
Specifically, if I have a file open and the server reboots, is the
the file considered "open" for these purposes until either I reclaim 
the open or the grace period expires (and that only starts when the
server comes up again).  And of course there is a similar issue for
migration.

I see your point.  One thing to note in this connection is that the
protocol allows filehandles which do not obey 3 (under any definition).
Maybe that "feature" is something that will drop out of the final
protocol.

> I think that a client has no use for any more guarantees that those
> (and 2 isn't really a guarantee), and so the protocol should not
> provide any more.  A server certainly can if it wants to, but who
> cares

I will because it is easy and I suspect that for most servers it 
will be the same, until we get to the wonderful world of heterogenous
migration.  (Coming soon to a protocol near you!).

Speaking of such migration, a related persistence issue is the 
lifetime of fileid.  There is no strong persistence language about
fileid, but on the other hand, there is nothing abut changing it
either.  API's like stat pretty much allow you to assume that the
id stays the same but very few application depend on that (although
we may later find some that do subtly in ways we hadn't anticipated).
In any case, I think people have been assuming that in heterogeneous
migration fileid is going to change with the filehandle, but it
isn't clear exactly how that would affect applications.  Maybe this
is something that should be nailed down as we work on the migration/
replication protocol.

> > 
> > As to the state stuff, I don't think it is the state *model* that
> > is complex.  The state model is that there are a set of clients
> > which identify themself, get an id.  Then, you have individual 
> > owners (i.e. processes) which open files, and get an id to 
> > identify that open file.  Multiple owners (processes) can get
> > locks on the same open file and the processes each get an id
> > that identifies the state of their particular locks on the
> > file.  Open-upgrade/downgrade is less than elegant but overall,
> > the model is pretty clean.
>
> You are immersed in the model and understand it well.  I'm not sure
> that gives you the best perspective.

OK.  But I'm having trouble with the idea that not understanding the
model well somehow gives one a better perspective.  You gotta admit
that's wierd.  Let's go on though.

> There are two quite distinct 'stateid's, one for open/locking, and one
> for delegation.  Why?

Because a delegation and an open are two distinct things.  You may have
any number of opens by different processes (OK, I know you have problems
with that concept in the protocol but bear with me a minute), all of
which are subsumed within a single delegation. Normally, the server
doesn't see these but there are delegation recall situations in which
you had better carefully distinguish the individual opens from the
delegations that cover them.

> There are stateids and sequence numbers that both have to change for a
> number of operations.  Why?

Point for my side, I think.  All of the stuff with stateid's changing and 
sequence numbers derives in one way or another (often indirectly) from 
replay protection.  As to that particular issue, I'd have to research
it in detail, but the basic problem is that once you wind up having
stateid's change (because of replay protection) and have operations
that reference owners (that pesky concept again) be sequenced, again
for replay protection, you are going to wind up with a lot of such
sequencing.  Whether in any particular case, having them both 
changing was overkill or necessary in some case, is hard for me to
say without looking at a lot of detail. 

> There are stateids and filehandles that both uniquely identify a
> particular file. Why?

I don't understand the problem you see.  A stateid identifies an 
open file.  Another stateid identifies a different open file.  They
both point at the same file /x, but they are for different processes
(can't avoid it) with different open modes let's say.  So I don't
see any problem (or even lack of elegance) in having different sorts 
of identifiers for the open file and the file /x.  I'm not saying
that it is impossible to come up with a different approach (and
yours might be better), but what we have seems a reasonable extension
given where we cam from with v2/v3 and nothing that worries me.

> And I still haven't found a justification for the concept of 'owners'
> or 'processes' being included in the protocol.

The most clear justification is in byte-range locking.  If I (some process)
get a lock (shared) on a file and you do too (another process) for a partially, 
overalpping range then there has to be some way to distinguish the you and me 
so that when I unlock my lock, it's clear whose lock is being unlocked.
Similarly with POSIX locking, if I get a lock, then I get the same lock 
again it is a no-op, but if you (a different process) try to get a lock,
you will eaither get a different lock (if shared) or wait (if exclusive).
All of this can happen within the same open due to the wonderfulness of
UNIX fork.

I don't see how you can specify locking (even at the level of the model)
without reference to the concept of a process.  It might be possible
in some locking model but if you have to support the ones that exist
out there (and we do), I don't see how you can avoid the concept and 
it then winds up mapping into something at the protocol level as well.

> > 
> > The complexity comes, not in the model, but in the protocol
> > specifics.  Most of that is in some form of replay protection.
> > Sequence numbers, per-object last request info, the OLD_STATEID
> > stuff, some object recycling and confirmation complexities all
> > derive from the fact that you are doing stuff that is stateful and
> > the old NFS-v2/v3 at-least-once model of things is inadequate.
> > That is the one problem that is solved (at least I hope it is 
> > solved) in a half-dozen (at least) different places in various
> > different ways.  The problem that NFS-v4 faced was that NFS
> > implementations had a solution for replays, that did not work
> > well enough (i.e. that you could specify it in the protocol and
> > show it worked) to adopt for the state stuff, but did not work
> > poorly enough to be gotten rid of (and replaced with something
> > else).  Not biting the bullet and taking on this issue is the
> > biggest mistake we made, and we can't blame it on the charter
> > (at least I don't think we can).
>
> I think that if the model were elegant, there would be no need for
> complexity in the protocol.  It would just fall out as obvious.
> This isn't just hypothetical.  I have a model that I think meets all
> the needs and is (again, I think) quite elegant.  But it isn't NFSv4.

I'm tempermentally skeptical of the "elegance conquers all" school of
system design.  There's too much messy ugly stuff from the past (that
pesky users insist on) that one has to carry over.

> Every so often I think I should write it up and implement it and call
> it NeilsFileSystemV1, but I doubt it woule be worth the effort.

If you do write it up, I'd like to see it.

> As for the 'not biting the bullet', I agree with that bit.

> >                                                     This has
> > given me a severe case of exactly-once-semantics-envy.   
>
> exactly-once-semantics that is uniform across all filesystem
> operations that need it is a feature of my model.  I agree that
> it is very valuable.

It's easy in the model and local filesystems have it as a matter of
course.  The issue is providing it in the protocol.  NFS-v2/v3 
avoided the issue and got by with it because of statelessness and
the fact that the reply cache seemed to work well enough (also, the
POSIX model of locking lends itself easily to idempotent locking 
requests).  V4 couldn't take the same approach.  So instead of
solving the problem generally at the lowest level of the protocol,
it solved it a dozen times in different ways for different operations,
and as the operations changed, we solved it again.  That was a 
mistake.

Don't get me wrong.  I don't think it was a fatal mistake.  I don't
think the protocol is unimplmentable or anything like that.  But I
do think that it has made things more complex than they should be
and has caused the protocol development to take much longer than it
should.  I would say a year.

---

From: Neil Brown <[email-masked]>
Date: Mon, 16 Dec 2002 12:04:11 +1100

> Neil Brown wrote: 
...
> understand their choices.

Yes, I agree that it would be useful to have some sort of taxonomy of
file server features that NFSv4 can work with:
  - some that are assumed universal ('files', 'directories')
  - others that are optional
Arguably this should have been what the working group started with,
but that is easy to say in hindsight.
It might even be useful for the client to be able to query the server
concerning elements of this taxonomy.  Currently the client can
probably work most of it out, but it isn't obvious.

> I don't think the protocol provides that guarantee now, does it?

Given that NFS4ERR_FHEXPIRED is allowed other than for PUTFH, I guess
it doesn't :-(

> be prepared for this issue.

Bizarre, don't you thing?
 Client:  PUTROOTFH, LOOKUP(usr), LOOKUP(local) LOOKUP(lib) LOOKUP(X11)
 Server: OK OK OK FSEXPIRED
 Client: What now? try again I guess...
 Client:  PUTROOTFH, LOOKUP(usr), LOOKUP(local) LOOKUP(lib) LOOKUP(X11)
 Server: OK OK FSEXPIRED
 Client: Hmm, this could take all day
 Client:  PUTROOTFH, LOOKUP(usr), LOOKUP(local) LOOKUP(lib) LOOKUP(X11)
 Server: Ok OK OK OK  FSEXPIRED

Obviously this is very unlikely.  But if the protocol allows it
(which maybe it does) the client should cope.  Does anyone want to
stand up and say their client will cope with this (assuming that after
4 or 5 tries it actually gets success) ?

> they get real small somewhat short of "forever".

Certainly.  So it should be encouraged where possible (and I suspect
any server could provide filehandles that are reasonably stable most of
the time).  The client should have a right to expect that filehandles
are mostly fairly stable.  But it never needs to know that they are
always absolutely stable.

> server comes up again).  And of course there is a similar issue for

Yep, 'open' stays open until explicit close or lease expiry, server
reboots not withstanding.  This might mean that sometimes a server
needs to store a filehandle->file mapping to stable storage, but this
would be the exception rather than to rule and shouldn't be a
performance burden.

For migration, I would like to serve to be able to say "This file has
been moved to that fileserver, and now has filehandle X".  I think
this would ease the burden of state maintenance across a migration.

> is something that should be nailed down as we work on the migration/

My position is that the concept of a stable unique fileid is an
unsustainable API.  It just cannot be implemented in general, even
when just considering local filesystems.
The best you can do is provide something that is mostly unique and
mostly stable, and maybe provide some new API to allow a client to
check if two files are the same if they really need to know.

I think NFSv4 shouldn't be trying to do this impossible here, but
rather should make the limits of 'possible' explicit.

> that's wierd.  Let's go on though.

It's a "Trees and Woods" thing.  You obviously understand the trees
very well, but let me show you what the woods look like (to me at
least). 

> delegations that cover them.

Distinct... but still very similar.
For example: consider
 1/ a successful OPEN_DELEGATE_NONE response to an
 OPEN4_SHARE_DENY_BOTH request
and
 2/ a successful OPEN_DELEGATE_WRITE response to an
 OPEN4_SHARE_DENY_NONE request

Both contain a promise that no other client has an active open on the
file, and that no other client will be allowed an open until this
client releases it's open.
The important difference is that in the second case the server
reserves the right, and the responsibility, to request that the client
release the server from that  as soon as possible.

I submit that this is the sum total of the important difference between
the two, and further that this difference is not enough to warant an
extra stateid being in the second response.

As you note there are issues with the concept of processes.  It is not
possible the argue this issue alone as lots of issues in the current
protocol are tightly intertwined.  But if you forget your deep
understanding of the current trees for the moment and look to the big
picture, can you see that important difference between these two is
worth about one bit (either the server has the right to reclaim, or it
doesn't)?

> say without looking at a lot of detail. 

You claim a point too quickly.  In fact your last sentence gives me the
point, as with the complexity in the model of the current protocol it
is 'hard' to say without looking at a 'lot of detail'.

My perception of the development (which could well be inaccurate) is that:
1/ The concept of a stateid was developed first.  On the whole it
   looks like quite a reasonble concept for the need at hand.
2/ it was found that it wasn't quite sufficent to handle the
   sequencing of error cases, so seqid was added.

The missing step was:
3/ Observe that seqid is sufficient for most of what stateid was used
   for, so discard the changing aspect of stateid.

Certainly sequencing and replay protection is important. But that
doesn't mean we need two concepts which both provide it.  More on this
below... 

> given where we cam from with v2/v3 and nothing that worries me.

Ok, having two different things (two stateids or a stateid and a
filehandle) that both refer to the same file might not be a bad thing
because, as you say, they might include other information (e.g. open
mode).  But why do you need *both* in the one request?  Why does a
READ need to have both the filehandle and a stateid?  This is
redundancy which is not a good thing in a protocol.

Lets look inside this concept of a stateid for a moment. 
A stateid combines the concept of
  - a particular client
  - a particular file
  - a particular point in a sequence of requsts from that client and
    for that file.
and possibly some aspect of different threads of control within that
client, which is shared with the 'lock-owner' concept.

The 'particular file' concept is shared with the filehandle which
always accompanies a stateid.
The 'particular point in sequence' concept is share with the seqid
which often accompanies the stateid.
The 'particular client' concept is shared with the clientid which never
(I think) accompanies the stateid.

Do you not see an inelegance there?
There are appear to me to be two ways to be elegant:
1/ Upgrade that stateid so that the sequencing information in it is
adequate for all tasks, so the seqid can be discarded and also discard
the filehandle from those operations that take a stateid.
This acknowledges the strength and centrality of the stateid.

2/ Downgrade the stateid so that it disappears.  Include the seqid
everywhere that sequencing is needed. Include that clientid (or
nfs4_lockowner) explicitly as well, so that the concepts that are
hidden inside a stateid become explicit.

I think that the second of these would work better in practice.  It
might appear to include more bytes in a request (clientid and sequence
number instead of just stateid), but I don't think it is significant.

[Also, I appreciate that as described here, it appears to require
complete sequencing of all requests from a given client.  The actual
design doesn't but I don't want to bore you with details].

> it then winds up mapping into something at the protocol level as

If you and I are processes on a single NFSv4 client, and you have a
lock and I try to get a conflicting lock, why does the protocol need
to know about that at all?  My lock should be blocked or denied at the
client without ever getting onto the wiree.  When you release your
lock, the client conveys that to the server, and then tries to claim
my lock now that it sees no local conflicts.  The protocol, and the
server, just sees one client making locking requests that never
conflict with themselves.  It doesn't need to know about any
client-local concept of processes or lock owners.  All that should be
managed locally inside the client.

Still no need for the protocol to know about processes.

> pesky users insist on) that one has to carry over.

Therein lies the challenge of design: to find a model that
incorporates and generalises all that ugly stuff as best as possible.
It may not be as elegant as one would like, but I still maintain that
the more elegance, the better.

> should.  I would say a year.

I agree that the protocol is not unimplementable, so it wasn't fatal
in that sense.
But there are other benefits from simplicity:  it is easier to argue
that the protocol is 'correct' and 'secure'.  How sure are you that
NFSv4 is correct and secure?
I guess the interoperability testing goes a long way to confirming
correctness, but it doesn't help much in security.
Infact I believe there is a security flaw in NFSv4.  It is not a great
big open hole or anything like that.  It is just a rather
difficult-to-exploit denial-of-service issue that probably wont be a
problem in real life, but it is there none-the-less.

NeilBrown

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 16 Dec 2002 08:23:59 -0800

> On Friday December 13, [email-masked] wrote:

[...]
> rather should make the limits of 'possible' explicit.

A common case here is an application that uses stat() to determine
when a configuration file has changed - several metadata changes, such
as modification data, inode (fileid) number change, size change, etc...
can indicate to the application that the file has changed.

Another common case is wher an application holds a file open and uses
stat() to find out when the *path* by which it opened that file no longer
refers to the open file.  Syslogd being a typical example of such an app.

I don't think it's much to ask that the fileid of any one file be stable
for at least as long as the file is open.  If this is the only stability
requirement for fileid and the fileids are only as stable as that then
in the first case [above] the application may end up re-reading its
config file more often than it might if fileids were stabler.  In the
second case [above] there would be no effect.

There are, of course, other scenarios.

And note that an API for file equality testing is not enough - an API
for accessing (subscribing to) the stream of events relating to a given
file (or directory / file path) would be a more general solution , but
its use would also be far more complicated to use than fileids are today
(see the first example above).

If the only effect of fileid non-stability is suprious configuration
file re-reading then the problem is not very severe.  But I do think
that fileids have to be stable for the duration of file opens, at least
outside the context of file system migration (and perhaps open files'
fileids could survive filesystem migration... at least until they are no
longer open).

Perhaps we can figure out just how bad the impact would be on a variety
of POSIX apps of fileid instability.

Cheers,

Nico
--