Subject: RE: Stateid and seqid

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 9 Feb 2001 10:39:29 -0800

Neil Brown wrote:
> On Monday February 5, [email-masked] wrote:
> > Neil Brown wrote:
> > 
> > An excellent writeup that I think exposes the history of V4
> > adding state through the design, instead of knowing the state
> > needed and creating a design.

I'm speculating but I think one of the problems in designing
the state management was that we were hobbled by the tradition
of nfs being stateless (We don't need no stinking state).
I think we had faulted in almost all of the state that 
eventually got into V4 before we cut the statement about
statelessness from the spec introduction.

As far as statefulness goes, we were in denial for a while.

> 
> Thankyou.  As they say, hindsight gives 20/20 vision.

That's the optimial case.

> 
> > 
> > > You may have guessed much of my model by the foregoing treatment of
> > > the current model, so I'll try not to be too verbose.
> > > In this model I will completely leave out the lockowner level, because
> > > I cannot convincingly argue for it's existance.  You might find it an
> > > interesting exercise to see what it takes to add the lockowner level
> > > back in.  Hopefully you will be pleasantly surprised.
> > 
> > I think the flaw in your reasoning is the fact that there is
> > a level between the client and the openfile that allows the
> > same file to be opened but with different state and privileges.  
> > Thus the need for a lock owner.

The identity of the lockowner has to be visible to the server.

That's a separate issue from whether it has the particular role in
the state hierarchy that it does in the current protocol.

> > 
> > Consider the case where two separate entities on a client open the
> > same file with different permissions and each obtains a non-conflicting
> > lock. Having only a single "cookie" per openfile will require that
> > any state change cause all requests that are intransit to fail and
> > be recovered.  If in this example one of the entities UNLOCKs their
> > portion of the file that was read LOCKed, this changes the cookie
> > but it does not effect the ability of other entities to access the
> > file with their old cookies.  The addition of a lockowner allows an
> > intermediate state between the client and the openfile that prevents
> > every state changing operation from serializing all operations.

You must have some other level.  Otherwise too many things get
serialized and performance becomes unacceptable, especially where
latency is high.

> 
> I'm not actually arguing against the lockowner level at this point,
> just saying that I find it hard to argue for it.
> 
> However, there is no need for an extra level to provide separate
> cookies for separate entities accessing the same file.
> Hidden in "My proposed protocol" it says:
> 
>    -  a "name" for the object for which state is being created.
>       For a client, the "name" is an nfs_client_id4
>       For an openfile, the "name" might be a filehandle or a
>       filehandle plus a basename, and may include a uniquifier to
>       allow one client to hold multiple openfiles on the one file.
>       The details an implications of this are beyond the current 
>       discussion.

This could provide enough information to the server, but it doesn't
solve the problem of all the open being serialized because they
are part of the same state level.

You need at least one additional level, whether it is done exactly
at the lockowner boundaries or not.

> 
> Note the "uniquifier to allow one client to hold multiple openfiles on
> the one file".  Think of the uniquifier as an index into the client's
> open file table.
> This explicitly allows the client to hold two indpendant opens on the
> same file, even within the same "process" on the client.
> 
> Given this, we could probably get rid of the open upgrade/downgrade
> issue that seems to bother the WinNT server implementors, and could
> clearly resolve the question:
>   What happens if a client opens the one file using two different
>   names, and getting two different filehandles? 
> Currently, I think that the client needs to inspect the fileid to see
> if they are the same open.  In my proposal that isn't necessary. The
> client gives a different uniquifier and knows that it has a different
> "open" (i.e. a different, independant, cookie-sequence).

The open/upgrade downgrade issue is not just that NT servers want (or
need) multiple opens instead of upgrade/downgrade.  It is also that
UNIX clients don't want (or can't have) multiple independent opens
that they have to manage and present to the server.  NT is open-file
oriented while UNIX VFS interfaces are vnode-orineted and don't
want to know about open files.  I think the fact that this has been
discussed without vituperation may have obscured how fundamental the
issue is.  (Can you imagine Bill Gates and Scott MacNealy discussing
this?).

We could put multiple opens in the current protocol very easily.
A second open would produce a new stateid which the client would
use, without inspecting the fileid.  The problem is that this would
not work for UNIX clients, as far as I can see.

I did propose a somewhat unsatisfactory compromise for this issue
a while back.  There wasn't any response either positive or 
negative.  I can try to find it and repost it if anyone is
interested.

---

From: Neil Brown <[email-masked]>
Date: Thu, 15 Feb 2001 15:50:09 +1100 (EST)

> negative.  I can try to find it and repost it if anyone is

Now that you mention it, I do vaguely recall you mentioning this
before.  I would be interested in seeing your compromise again if it
isn't too hard to dig up.

I guess this is closely related to how the OS caches things.

Because the Unix philosophy has always been that multiple access to a
file is fine, there is a file cache that all processes can access
equally.   The processes talk to the cache, the cache talks to the
device.  In particular, it is the cache that writes to the device, not
a particular process.... so how do you choose which lockowner to
associate with a particular write request .. and does it matter?

The DOS/Windows philosophy was always "only one process can have a
given file open at a time" so the cache is probably a lot closer the
the file descriptor than it is in Unix.  File SHAREing was only added
later, and I suspect this is reflected in the internals of the FS
code.
So presumably a client implementation for WIN32 would be able to
present separate opens to the server in a way that Unix cannot (I
think actually that Linux could, but it wouldn't make for the most
efficient implementation and would probably cause problems for mmap).

So the setting is:
 - Unix clients can only present one open per file per process (or if
   I had my way, one open per file per client :-), and need to be able
   up-grade/down-grade the open mode between nothing/read/write/both.
 - WIN32 client can (presumably) present multiple opens per file,
   probably one per filedescriptor (or equivalent).
 - Unix kernel-based-servers and probably cope with anything (?)
 - WIN32 server (I don't know enough to suggest what level they might
   be at) cannot provide upgrade/downgrade of share reservations.
 - Unix user-space servers don't know what "DENY_*" means, and cannot
   convert a file descriptor from read to write or visa-versa.

Lets assume that:
  Unix clients never make "DENY_*" requests - they only assert an
  access type.  They only present one openfile per file per lockowner,
  and may want to upgrade/downgrade the access type on this file, but
  only by adding or subtracting, never both at once.

  WIN32 clients do make use of DENY_* requests, but (somehow) present
  different openfiles for the one file+lockowner.  These will only
  ever by OPEN_DOWNGRADEd to zero access, and will never be involved
  in a secondary OPEN (which might upgrade the DENY_* modes).

It should then be possible to implement the server over WIN32, or
anywhere.
When asked to open a file with no DENY_* modes, the server potentially
opens 2 file descriptors - one for READ access, one for WRITE access.
Any shared byterange locks and any read requests are made on the READ
file.  Any exclusive byterange locks and any write requests are made
on the WRITE file. An OPEN_DOWNGRADE can be handled by closing one of
these. An OPEN which does an upgrade can be handled by opening the one
that is currently closed.

Any opened file that does actually DENY something gets its own local
(to the server) open.

Is that at all similar to what you suggested?

It doesn't deal with the awkwardness that an OPEN might, without the
client being directly told, get interpreted as an openupgrade and not
actually create new state, but that is a separate issue.

NeilBrown