Subject: RE: Stateid and seqid

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 9 Feb 2001 10:00:00 -0800

Neil Brown wrote:
> What I will try to do is:
>  1/ present a model for how state-maintenance can/should/does work.
>     This model will be motivated by the current spec, but kept as
>     minimal as possible.
>  2/ describe how the current protocol maps to the model.  This should
>     both aid in the understanding of the protocol, and show the
>     problems that I see in the protocol.
>  3/ Present an alternate protocol for state management that is a
>     better fit for the model.
>  
> 
> I should say from the outset that I'm not asserting that the protocol
> is flawed in the sense that it cannot communicate the information that
> needs to be communicated.  I am fairly sure that the protocol atleast
> comes extremely close to being able to communicate successful.
> My problem with the protocol is that I find it very hard to go that
> extra step from from "extremely close" to "definately there", and I
> believe this is due to lack of a clear abstract model to use when
> reasoning about the protocol.
> 
> It could be that providing a clear model and a clear description of
> how the protocol fits that model will be enough to make the protocol
> easy to work with -  Dave Noveck suggests that possibility in a recent
> Email.   I will leave it to the reader to judge on that issue.

I hope I didn't say "easy".  "Easier" is what I'm hoping for.

> 
> 
> Enough pre-amble - on to some real detail.
> 
> -------------- The Model -----------------------------
> 
> What is state:
>  
>    The area of the protocol that I dealing with here is state
>    management.  The server maintains a certain amount of state on
>    behalf of clients, and it is critical that the servers view of the
>    state remain consistant with the clients view of the state despite
>    an imperfect communication channel (Allowing loss, deuplication,
>    and re-ordering).  The protocol needs to assure this consistancy.
> 
>    The basic elements of state are things like locks and share
>    reservations.  I am not particularly interested here with what
>    state can be recorded, or what state transitions are allowed, but
>    only with how those state transitions are communicated and
>    confirmed. 
> 
>    It is clear from the NFSv4 RFC that the basic unit of state
>    maintenance known to the protocol is the "openfile" (my term).
>    An "openfile" is a particular reference by some client to a
>    particular file.  It is certainly the case that different clients
>    may each reference the same file and maintain different state
>    (different locks) on that file.  It may also be the case that one
>    client could maintain multiple references to the one file, though
>    this possibility would need to be justified (later).

OK but I would think that it is pretty clearly justified by the fact
that cleints typically maintain multiplte independent entities (e.g.
processes) that are capable of opening files and the client needs to
be able to act as their agent when opening files and getting locks, etc.

> 
>    It is, I think, worth noting that the basic element of state could
>    be chosen to be bigger or smaller.  We could, for example, treat
>    all locks on all files held by a particular client to be a single
>    unit of state. Alternately we could consider each byte of each file,
>    which may be locked independantly, to have it's own element of
>    state.  The "openfile" is the unit using in the current spec, and I
>    believe it is a good one, so we will start there.
> 
>    There is some value in treating number of related elements of state
>    as a single large state for certain purposes.  For example, the
>    current spec considers all openfiles associated with a particular
>    client to be a single object for the purposes for lease extension,
>    and for discarding old state when it is confirmed that a client has
>    failed.  Thus we have atleast two levels of state.
> 
>    The RFC indicates two other levels of state.  One that is not
>    explicitly spelled out, but is there implicitly, is the "server"
>    level.  The total state maintained on a server sometimes needs to be
>    treated as a whole, such as to describe what is lost on server
>    restart. The other level described in the RFC is the "lockowner"
>    level. This level goes between the openfile and the client, in that
>    each openfile is owned by a lockowner, and each lockowner is owned
>    by a client.
> 
>    This is, I think, sufficient justification to treat state as
>    hierarchical.  There is a clear hierarchy of state elements:
>    server, client, lockowner, openfile where the state of an time at
>    each level is wholely contained in the state of its parent at the
>    preceeding level.
> 
> How is state managed:
> 
>    State is stored on the server.  While the client will keep it's own
>    records of the current state, the state stored on the server is
>    authoritative.

An exception is reclaim logic.  When the server re-initializes, he
accepts the clients reclaim request as presumptively correct,
if not authoritative.
    
>    In general, state transitions happen as a direct result of a
>    request by the client - e.g. "LOCK" or "OPEN".  It is possible,
>    though comparatively rare, for state transitions to be effected
>    independantly be the server.  This can happen, for example, in the
>    case of lease expiry, or server admin intervention (section 8.7)
> 
>    The particular goal for the state management protocol is to ensure
>    that the client and server both know when a state transition has
>    occurred, and that the both know that the other knows.
> 
>    Considering only the client initiated transitions for now, the
>    client is only uncertain of the server's state between when it sends
>    a request, and when it receives a reply.  To provide the needed
>    guarantees we require that the client either:
>       1/ persist with any request state-changing until it gets a reply
>         (as in section 8.6)
>    or
>       2/ accepts complete loss of state on the server - in the event
>          of losing contact with the server.
> 
>    The server's period of uncertainty about the client's view of state
>    is between sending a reply and getting some new request that in
>    some way confirms that the client received the reply.  For the
>    server to become certain about the clients view, there needs to be
>    some way for the client to confirm reply.  The model presented in
>    the RFC (which I believe is a good one) is to use a "cookie" (my
>    term) to represent a particular state.  The architypal cookie in
>    the spec is the stateid.

During the period in which the client is uncertain about the server's
view of the state, he may want to change the state yet again.  If he
cannot start to do so during that window, you may have unacceptable
performance constraints due to excessive single-threading of requests.

I suspect this is what has driven (at least in many cases) the choices
that the spec has made in establishing its state heirarchy, although
this model makes that all that clearer.

Beyond state-changing requests, the protocol must also deal with 
state-dependent request such as IO.  These requests don't want to
bear the burden of any single-threading but the but bounding the
validity window for such requests due to possible delays (e.g.
due to nasty routers) is a problem that the protocol needs to 
address.  Your model shows how terribly ad hoc the current
solution is, but it does work.
 
> 
>    The minimal detail required of this cookie is that
>     1/ from the server's response, the client can deduce a new cookie.
>     2/ in subsequent requests the client can show that it has the new
>        cookie.
> 
>    Note that this does not require the literal cookie to appear in
>    responses and requests, though that is certainly an obvious
>    implementation.  An alternate implementation would be a universal
>    sequence number. The client includes the sequence number in
>    requests - satisfying 2, and the client increments it whenever it
>    sees a new response from the server - satisfying 1.  This would
>    work for the current purpose, but has other negative implication
>    for the protocol (total serialisation).
> 
>    The "cookie" is shorthand for a particular state.  Whenever there
>    is a new state, there must be a new cookie.
> 
>    For server-initiated state changes, the client can find out that
>    something has happened by receiving a failure status for a request
>    with a cookie that is no longer valid.  The spec isn't clear on
>    exactly how the client informs the server that it has figured out
>    the new state.  I think that the neatest approach is that the
>    client should send unlock/close requests to release any state that
>    is in question, and that in these requests, old cookies should be
>    accepted.

Not sure if you consder server reboot a server-initiated state
change.  In that case, the client doesn't unlock/close.  I guess in
that case the SETCLIENTID tells the server that the client 
know about the reboot (or if didn't, they both agree that they are
starting with a clean slate).
 
> 
>    Ok, it took a little while to get here, but we now have an
>    hierarchical state and a cookie for state confirmation.
> 
> State transitions:
> 
>    There are three sorts of state transitions that we need to
>    consider:
>     1/ creating a new state
>     2/ modifying a state
>     3/ discarding an old state.

What are the simultaniety restrictions that are assumed there?

I assume that only one type-2 request may be outstanding for a
a given state.  Also that no type-3 request for a given state
may be issued while a type-2 request for that same state is
outstanding.  And clearly you can't have multiple type-3's
for the same state, nor can you issue a type-2 or type-3
before you have completed the request that created it (a
type-1).

These are related to the child state.  I'll speculate below
about the simultaneity contraints relative to the parent state.
 
> 
>    For each of these, we need to understand:
>     A/ what cookie must be presented by the client to convince the
>        server that the client and server views of state are sufficiently
>        consistant.
>     B/ what cookies get changed or created and how are these returned.
> 
>    I will assert the answers to these questions and leave the
>    justification to when I compare the model to the current protocol.
> 
>    When creating a new state, the cookie for the parent state must be
>    presented, and a new cookie is created and returned for the new
>    state.  The cookie for the parent state doesn't change.

I'm guessing that this (not changing the parent state) would allow
multiple simultaneous type-1 requests specifying the same parent.

This makes me uneasy as I consider OPEN's.  An open creates a piece
of state but I think that sequencing opens and close as the current
protoocl does is a good thing and a lot of problems would flow from
not doing that.  The issue is that while OPEN creates a new state, it
also modifies the parent state in ways that would not allow multiple
open from a given lockowner to be handled without sequencing them.

>    When modifying a state, the cookie for that state must be presented
>    and a new cookie for the new state is returned.  Note that only the
>    bottom level of state can be directly modified.  Higher levels are
>    modified by creating/deleting a lower level state.
> 
>    When deleting a state, the cookie for the state and the parent
>    state must be presented, and a new cookie for the parent state is
>    returned.

Which I assume means that only one type-3 specifying a given parent
can be outstanding at a given time.

> 
>    Each of the above assumes that the request succeeds and that the
>    state transition happens.  Indeed, at this level of abstraction, we
>    must assume that every request either:
>     1/ will always result in failure (e.g. protocol violation) or
>     2/ does cause a state transition.  It may not be the state
>        transition that was requested, but there must be a new cookie,
>        so that both the server and client can be sure that they agree
>        on the result of the request.  
>    As we shall see later, it is easy to include a light-weight
>    component to each cookie to accomodate the trivial state change of
>    "another request failed".
> 
>    Also, the above does not cope with describing the creation of a new
>    server or the destruction of a new server.  These are not really
>    relevant to the client, and hence to the protocol, except that the
>    client needs to get a cookie for the server to use when creating or
>    destroying a client.  For this, we simply need a way for the client
>    to ask the server for a current cookie.
> 
>    So that is basically the model: hiearchical state, cookies, and
>    rules for cookie handling on state transition.
> 
> 
> ---------------------- The current protocol  ----------------------------
> 
>   We will examine the current protocol by looking at the different
>   objects at the different level in the hierarchy, and look at what
>   sort of cookie they have and how it is communicated.
> 
>   The protocol defines four levels of state: server, client,
>   lockowner, openfile.
> 
>   Server:
>    Server creation and deletion is not part of the protocol.
>    The "cookie" for the server state is  effectively the
>    setclientid_confirm verifier that the SETCLIENTID operation
>    returns.  More discussion of this below.
>    
>   
>   Clients:
> 
>    A client is created by a SETCLIENTID operation followed by a
>    SETCLIENTID_CONFIRM operation.
>    This is very similar to the "get server cookie" followed by "create
>    client given a server cookie" approach that the model suggests. 
>    The setclientid_confirm value serves adequately as a server cookie.
> 
>    A client cannot be explicitly deleted in the protocol, except by
>    creating a new client with the same cookie but a different verifier. 
>    Rather the server is allowed to delete client state after it hasn't
>    heard from the client for a while
> 
>    The clients "cookie" is, as a first approximation, the clientid
>    returned by SETCLIENTID, though it should be clear that this
>    doesn't quite meet the needs of a cookie, because a cookie must
>    change when a child state is deleted, but a clientid never changes.
>    We will address this issue at a lower level of the protocol.
> 
> 
>   Lockowner:
> 
>    Object Creation:
> 
>     A lock-owner is created by the OPEN_CONFIRM operation which follows
>     an OPEN operation which tried to use a non-existant lockowner (or
>     possibly by an OPEN operation which is received before the server
>     has discarded any lockowner state).
> 
>     In our model, a creation request must be accompanied by the cookie
>     for the parent.  The OPEN request contains the clientid of the
>     parent, but we argue above that this is not enough for a cookie.
>     In OPEN_CONFIRM, an open_confirm verifier is also sent.  This is
>     an opaque datum provided by the server and effectively serves as
>     the remainder of the cookie.
> 
>     One way to look at this, is that the server maintains a client
>     cookie which is the clientid plus something else.  When the client
>     makes a request that requires the "something else", the server
>     replies by saying "you also need this datum to confirm the
>     request".  The client then confirms.

Normally the lockowner/seq-id pair is the something else.  In the 
current protocol, the client doesn't know whether that is enough
and has to be told.

> 
>     So the "cookie" for the "client" is the clientid plus the
>     open_confirm verifier.
> 
>   Object destruction:
> 
>     There is no way for the client to indicate to the server that a
>     lockowner can be released.  It can only release all lower-level
>     state (UNLOCK locks and CLOSE files) and then wait.
>     The server is free to release the lockowner object if it is
>     "empty" and has not been used for a while.
> 
>     The model requires that the cookie for the parent client must
>     change.  This will manifest in the open_confirm verifier being
>     different next time.
> 
>    Object cookie:
> 
>     The lockowner does not get a cookie created for it by the server.
>     Rather the cookie is a combination of an nfs_lockowner4 (which is a
>     clientid4 plus an opaque owner) and sequence id.  The sequence id
>     (seqid) is negoatiated at lockowner creation time.
> 
>     This combination of nfs_lockowner4 and seqid is somewhat awkward
>     as the seqid is also involved in preresenting state at a lower
>     level.

Somewhat awkward.  Definitely.

> 
> 
>  Openfiles:
> 
>   Object creation:
>     An "openfile" object is created with the "OPEN" operation.  The
>     client must present the "cookie" for the lockowner, being the
>     nfs_lockowner4 structure and the seqid (and possibly must present
>     a verifier with OPEN_CONFIRM, but that is to confirm the creation
>     of the lockowner, not to confirm the creation of the openfile.
> 
>   Object creation failure:
> 
>     It is quite possible for openfile creation to fail in a way that a
>     subsequent identical request could succeed.  One possibility is a
>     request for exclusive create: the file could be removed between
>     the initial request and the retransmit.  Another possibility is a
>     conflicting share reservation.
>     For this reason the "cookie" that is presented for the parent state -
>     the lockowner -  must change when an OPEN fails for one of these
>     reasons.

OK.  So state-creating requests that can fail may change the parent
state, without creating a new element of state.  So the client must
not only send the parent cookie, he must be prepare for it to change.
This implies to me that stat-creating and state-deleting requests 
for the same parent state must be serialized.

>     The part of the "cookie" that changes is the seqid.
>     The server doesn't explicitly return a new seqid, but implicitly
>     indicates that it must change.
>     For the protocol to work reliably, it must require that the 
>     seqid change in an agreed way (increment) whenever the server
>     fails an OPEN request for a reason which could be transient.
> 
> 
>   Object deletion:
> 
>     openfiles can be explicitly closed using the CLOSE operation.
>     The model requires that the parent cookie be presented for this
>     operation and be changed by it.
>     nfs_lockowner4 part of the cookie is deducable by the stateid, and
>     the other part, the seqid, is explicitly given.  The seqid is
>     implicit changed by the completion of the operation, thus changing
>     the whole cookie.
> 
>   Object state change:
>   
>     The openfile is the only object that can undergo a state change
>     without creating/deleting a child object.
>     The state changes happen with LOCK and UNLOCK (and maybe
>     OPEN_DOWNGRADE and possibly OPEN and maybe even others).
>     
>     Each such request passes a stateid plus seqid, and return an new
>     stateid on success and implicitly update the state id on failure
>     (and on success).
> 
>   Object cookie.
> 
>     The cookie for an openfile is the stateid returned by the OPEN
>     operation, combined with the seqid.  The seqid component is needed
>     to reflect failed LOCK attempts.  Here we see that the seqid is
>     part of the cookie for both the openfile and the lockowner.
> 
> Summary of issues:
> 
>  - While there is a clear hierarchy of state, the way that state is
>    indicated (the cookie) is very different at different levels.
>  - It is unclear what the purpose of the lockowner level is.
>    I am happy to detail my view of the lockowner level but this email
>    is too long as it is, so I have cut it out and will send it
>    separately if anyone is interested.

I'll try to collapse my view into a few sentences.

Lockowners as protoocl elements (apart from their role in the state
hierarchy), need to exist to represent the various independent 
actors by which files may be opened/locked, and for which the 
client implementation acts as the agent.  Various semantics on
the server (particlularly in their UNIX variants) depends on the
identity (or lack thereof) of a lock requestor and the lockowner 
for the lock which represents a potential conflict. 

Lockowners exist as elements in the state hierarchy because of
a) general confusion, which your presentation has helped dispel,
and b) the fact that splitting the hierarchy at this level is 
a good fit since the resulting simultaneity restrictions work for
lots of applications in that only doing one OPEN/LOCK/UNLOCK/CLOSE
for a given process at a given time is not too onerous.

>  - The one 'seqid' plays a similar, though independant, role for 
>    the lockowner level and the openfile level.
>    Also, the rules for incrementing it do not make clear it's function
>    (and do not seem to be generally agreed on).
> 
>    My rules for incrementing the seqid value in the current protocol
>    would be:
>     1/ The spec MUST specify exactly which return status for which
>         operations cause the seqid to be incremented:
>     2/ These MUST include any status which indicates a condition which
>        is transient such that a subsequent receipt of an identical
>        operation could succeed.

Yes, definitely.

>    
> 
> ------------------------- My proposed protocol -----------------------
> 
> You may have guessed much of my model by the foregoing treatment of
> the current model, so I'll try not to be too verbose.
> In this model I will completely leave out the lockowner level, because
> I cannot convincingly argue for it's existance.  You might find it an
> interesting exercise to see what it takes to add the lockowner level
> back in.  Hopefully you will be pleasantly surprised.

Unless I'm misunderstanding how this would work, it seems that the
client could only have one OPEN/CLOSE at a time.  That would be
reason enough add another level (e.g. lockowner) to the hierarchy.

Another possiblity is to retain lockowner in its actor-identification
function but take it out of the state-hierarchy.  You could allow
the creation of multiple levels in the state-hierarchy solely to
allow greater request simultaneity on the part fo the client.
The client could choose the boundaries of the state elements
and even the number of levels solely as ways of optimizing the
transmission (trading off setup-teardown time against the degree
of simultaneity desired).  With a more regular model, that would
be possible.  I fear I'm positioning myself on the Lunatic Fringe 
now so I guess I'll just stop here.

> 
> Foundation:
>   There is an hierarchy of state maintained on the server.  The levels
>   of the hierarchy are:
>       server
>       client
>       openfile
> 
>   The state of an object at each level is represented by a "stateid"
>   and a "failurecount". These together form the cookie.  
> 
>   There is precisely one server level state on each server.  It is
>   created by the server at start time and assigned a unique server
stateid.

I was having trouble with this.  It seemed that it was bizarre variant
in that all other state cookies could only change when you changed them
while this one was routinely changed when other clients made requests.

But now I see that this one never changes because the only thing that
happens si that new clients get created which does change the 
parent state and cleint creation can't file (?) so no changes for
that.

What about deletion of clients?  Can they happen?

I still think it's bizarre.  I just mean it in a derogatory way any 
more.
 
>   All other levels of state are created by a request from the client.
>   Any request to create state for an object at a particular level
>   requires that the client present
>    -  a current stateid and failure count for a parent state level
>       above the target level. 
>    -  a "name" for the object for which state is being created.
>       For a client, the "name" is an nfs_client_id4
>       For an openfile, the "name" might be a filehandle or a
>       filehandle plus a basename, and may include a uniquifier to
>       allow one client to hold multiple openfiles on the one file.
>       The details an implications of this are beyond the current 
>       discussion.
> 
>   Thus to create state for client, the server stateid and failure
>   count must be presented along with a nfs_client_id4.
>  
>   To create state for an openfile, a current client id and
>   failurecount must be presented along with a filehandle and,
>   possibly, a basename.
> 
>   In each case, other information relevant to the specific object may
>   be required.  The state/failurecount/name is the minimum in common.
> 
>   If the requested state is successfully created then a new stateid is
>   returned. 
> 
> 
>   State for objects are deleted on explicit request from the client. 
>   The request identifes the parent state with stateid and
>   failurecount, and also the target state bby stateid.
>   A successful response to state deletion includes a new stateid for
>   the parent state.
> 
>   Whenever a new stateid is given to the client by the server, the
>   associated failurecount is agreed to be zero.  Whenever the client
>   presents a stateid+failurecount in an state-creation request to the
>   server,  there are three possibilities wrt the handling of state:
> 
>     1/ The state change succeeds, a new stateid is created and
>        returned to the client.  The failurecount is agreed to be zero.
>        The server will remember the previous stateid/failurecount and
>        the operation that suceeded at least until the new stateid is
>        seen from the client.  This allows the server to reply
>        consistantly in the event of retransmission.
>     2/ The state change operation could fail for a reason that is
>        potentially transient. e.g. there are conflicting locks
>        current, or the server temporarily ran out of resourse. In this
>        case the failurecount is implicitly incremented by client and
>        server.  The server remembers the return status at least until
>        it sees a request from the client with the same stateid and a
>        new failurecount.  This allows it to reply consistantly in the
>        event of retransmission.
>     3/ The state change operation could fail for some reason that is
>        permanent, such as EXPIRED or FHEXPIRED or several others.  In
>        this case, no change of state is need as retransmission will
>        result in the same (or sufficiently similar) reponse.
> 
>   The server stateid can be requested of the server at any time.

One thing this doesn't address is IO (i.e. requests that depend on 
state but don't change it).  You want to avoid excessive serialization
without letting requests (WRITEs particularly) being acted upon
when they are no longer valid.  The current solution which uses
optimistic issue with retry on OLD_STATEID isn't pretty but it
serves the need.
 
> I apologise for the length of this email, appreciate your patience in
> reading, and look forward to your comments.

No apologies necessary.  This is a big contribution to our (at least
to my) understanding.  Thanks for taking the time to write it.

---

From: Neil Brown <[email-masked]>
Date: Thu, 15 Feb 2001 14:50:27 +1100 (EST)

> accepts the clients reclaim request as presumptively correct,

An interesting point.  From a high-level perspective, it probably is
appropriate to think of the clients information as being authoritative
immediately after a server restart.  However when considering the
practicalities, I don't think that  the server will give any more
weight to a request made during the "grace" period than it will at any
other time - except that delegation requests can be made at this time.

I think that the rule for the server comes quite close to:
 During grace period, only accept state-changing requests that are
 marked as "reclaiming",
 At other times, only accept state-changing requests that are not
 marked as "reclaiming"
In both cases the server should do the same checks about permission
and conflicting locks.

> this model makes that all that clearer.

Serialisation of requests is definately an important issue.
Infact, if serialisation were OK, we could manage with just one level
of state per client.  But obviously this is unacceptable.
I had thought that serialisation at the client level wouldn't be a
problem because:
 1/ When sub-state is created, the client cookie doesn't change so
    multiple creates can happen in parallel and
 2/ If sub-state is "emptied" (i.e. unlock all locks, downgrade all
    opens) first, then destroying the state becomes a simple
    house-keeping operation that doesn't interact with any "real"
    work, and so can be done serially by some background thread.

However on describing the model competely, I see (and you point out
below) that 1 doesn't apply, as the client cookie must change on a
failed create.
So I suspect that the need to reduce serialisation would be sufficient
justification for another level of state.

> starting with a clean slate).

Yes, I would consider server reboot as a server-initiated state change, and
no, the "neatest approach" mentioned above does not really cover that.
I guess it depends on how much information is deemed to be in each
possible error code:
  NFS4ERR_STALE_STATEID  and
  NFS4ERR_STALE_CLIENTID
              say "All state associated with the <server> level
              state that corresponds with the given stateid has been
	      revoked:  Client needs to start again from scratch.
  NFS4ERR_EXPIRED say "The state at the level of the stateid presented
              has been revoked:  Client needs to attempt to "close"
	      that stateid - it may find that state at a higher level
	      has gone too.

> before you have completed the request that created it (a

I agree with all of that.

> open from a given lockowner to be handled without sequencing them.

Is the issue that you are seeing this:
  a given lockowner on a given client issues two parallel open
  requests for two different names that happen to refer to the same
  file.  This creates creates only one openfile state, but it is not
  clear which returned cookie should be used for that state.

I think that is an problem with OPEN (don't get me started on OPEN:-),
but I think we agree that OPENs do need to be serialised to cope with
the possible failure of an OPEN (though I don't think you had got
there by this stage).

> can be outstanding at a given time.

Yes, though if it were possible to pass state cookies forward through
a COMPOUND operation (like you can for filehandles) I think it could
work to include several type-3 requests for a given parent in a single
COMPOUND.
> current protocol, the client doesn't know whether that is enough

My interpretation is that "normally" clients are fairly long lived,
while lockowners are relatively short lived, so for most clients, at
least one lockowner will have been disposed for.  Thus from the
paragraph on page 139 for RFC3010:

      A lazy server implementation might require confirmation for every
      nfs_lockowner for which it has no record.  However, this is not
      necessary until the server records the fact that it has disposed
      of one nfs_lockowner for the given clientid.

the server will mostly (normally) have to require confirmation.  The
only time it doesn't need to is between client creation and first
lockowner disposal.

> for the same parent state must be serialized.

Yes :-(
There is an alternate approach for state-creation:
We could require that state creation always succeeds in that a new
state is created, but that it may not always have the properties
desired.
(e.g. I OPEN and ask for READ/DENY_WRITE, and I get given an open
which (I am told) is only READ.  Probably though, if not all
access/deny request can be given, then none should be given).
If this state is not useful to me I can then OPEN_DOWNGRADE if
necessary and schedule a CLOSE.  As mentioned above, state-deletion
(CLOSE) must be serialised, but that isn't as much of a problem.

The big problem I see with this approach is NFS4ERR_RESOURCE.
This is an error that in the class of "a retry might work", so it must
cause a state change.
Expecting a failurecount counter to be incremented when the server is
having resource problems isn't much of an ask.  Asking it to always be
able to allocate a new cookie, even when low on resources might be too
much to expect.

> for the lock which represents a potential conflict.

I have heard this claimed more than once, and I still don't get it.
Can you help me?
I cannot see why the client shouldn't appear to the server as being a
single lockowner: arbitration between any idependant agents on the
client would be done by the client:

   Agent-1 asks for a lock
                       Client asks server for the lock
		                              Server grants the lock.
		       Client grants the lock to Agent-1
   Agent-2 asks for a conflicting lock
                       Client detects conflict and blocks Agent-2
   Agent-1 releases lock
                       Client grant lock to Agent-2
   Agent-2 uses lock
   Agent-2 releases lock
                       Client releases lock
		                              Server acknowledges
                                                release of lock

It could be suggested that the client should release and reclaim the
lock with the server before granting it to Agent-2, and that is
certainly an issue worth considering, but I am sure that locking could
be don't quite effectively without the server knowing "who" on the
client has the lock.

Certainly telling the server and letting it arbitrate works too, but I
don't see that it is necessary.

> now so I guess I'll just stop here.

Hmmm. I like it... It is possibly excessive in it's generality, but it
does help to make the issues clear.

> I still think it's bizarre.  I just mean it in a derogatory way any 

I'm beginning to have doubts about the server level as well.. I hope
they can be sorted out.
I basically extrapolated upwards to get the server-state concept, and
it seemed to work and to fit the current protocol (as well as the rest
of the model fitted the protocol).
Some issues:
  - Yes, clients can and should be deleted, though the server is
    allowed to destroy a client-state if it hasn't heard from the
    client for a while (least-time and all that).
    + this implies that the client should give
         current-server-cookie + current-client-cookie
      and get back new-server-cookie.
      However it would have to ask for the current-server-cookie,
      which seems pointless, and doesn't really need the
      new-server-cookie.

      Maybe the right thing is not to have the client destroy it's own
      state, but simple to cause a (type-2) state change to "this
      client state will never be used again".  Then the server can
      confidently clean up and update it's cookie.

  - When a client is reclaiming state after a server restart, the
    server must be able to determine if the client is reclaiming a
    sufficiently recent state.  The third paragraph of section
    8.5.3 (Network Partitions and Recovery) talks about this.   If the
    server has expired locks due to a network partition and then
    restarted, the client can not reliably reclaim locks any more -
    the state is not sufficiently recent.  I think that same holds if
    a server has restarted twice during a network partition.  A client
    coming back in contact after the partition should not be able to
    reclaim locks from 2 restarts ago.

    It would be nice if this sort of issue could be embodied in the
    server state cookie, but I don't see a clean way to do it.
    It would require to server to update it's cookie periodically
    (when it destroys client state), and then to prompt all client to
    refresh their idea of the server cookie (NFS4ERR_GET_NEW_COOKIE to
    all requests which have a stateid... not nice).
    
So, yes. That is an aspect of my model that is a bit incomplete.  I
see now that the concept of "server-state" really needs to tie in with
the issues around server restart, and I had been avoiding those issues
to keep the model simple.

> optimistic issue with retry on OLD_STATEID isn't pretty but it

I didn't address that, because I thought the current approach was
adequate and would fit equally well into my model.
Two thoughts:

 1/ a generous server could keep a short history of recent states for
   each open_file and could accept an IO with an old cookie if it was
   not contraindicated by subsequent state changes (i.e. no lock or
   unlock on that region).
   [Hmmm. a similarly generous server could in a similar way allow
   OPENs not be serialised even in the event of occasional
   failure.. The amount of parallelism would be determined by the server's
   history length (the generosity factor).]

 2/ One issue that neither model addresses explicitly is the
    possibility of two different write requests for the same region
    being re-ordered in flight by a byzantine router.  It's a pretty
    slim possibility I think, but given that we have allowed the
    "byzantine router" argument in the past...
    I essentially requires a "barrier" operation which changes the
    state cookie without actually locking or unlocking anything.  This
    would ofcourse mean that the generous server in point 1 must not
    allow write requests from before a barrier.

Thanks for your response,

NeilBrown