Subject: RE: Stateid and seqid

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 31 Jan 2001 11:19:14 -0800

Neil Brown wrote: 
> On Friday January 26, [email-masked] wrote:
> > Q1. Can anybody explain why do we need both stateid and seqid?
> > (consider that stateid is shorthand to lockowner+file, and stateid
includes
> > some
> > incrementing sequence inside itself)
> > Note, that there is no operation, that takes seqid only without stateid
> > (except OPEN, which is the
> > only operation to create stateid).

> I spent a while trying to understand these issues properly a while
> back and the simplest answer that I have to your question is:
>
>  Every request that might change state contains a stateid and a
>  seqid.

Except for OPEN.

>    If the operation is successful and state changes, then the stateid
>      changes.

According to David Robinson, the server may in some cases return the
same stateid (ie. not change it).

>    If the operation fails and the state doesn't change, then the
>      seqid changes.

The current approach is that seqid does not change if the operation
fails.  That's what people are implementing to, but there has been
disagreement about whether the spec says that (e.g. Mike Eisler believes 
it does)

> This doesn't describe the full picture, as the seqid changes if the
> operation succeeds, but I think it does draw out the central purpose of
> the seqid - that being to record failed attempts to change state.

The consensus was that there was no need to record that.  If an 
operation fails, nothing *really* happened so there is no need for
a corrective to just letting people retry and get at-least-one-attempt-
but-exactly-one-acted-upon-operation semantics.

> I actually think, and I believe that the recuring discussions support the
> idea, that the current model for state management is much too complex.
> It is a great prototype as it deal with (I believe) all the important
> issues and presents real solutions to them.  However it seems to be
> much too complex to be understood by implementors (and possibly even
> by the designers, I'm not sure).

I certainly think that the current design could be cleaned up some,
but I think a fair amount of the complexity is inherent in what we
are trying to do.  We want to support stateful operations but have
relatively simple recovery from server reboots while working over
the unreliable transports, all the while satisfying additional goals
for operation in high-latency environments (i.e the complexity 
introduced by COMPOUND).

My feeling about the recurring discussion is that they don't show
problems with the current protocol but reflect major problems in the
way it has been described.  The spec treats seqid, stateid, leases,
more or less separately, and without a lot of implementation detal
(out of a desire I think not to constrain the implementation).  When
you get to coding the implementation, you find that lots and lots
of questions aren't answered by the spec, because then you need to
decide what order the various checks, each of which hqas been 
described in a different section of the spec, have to be done.

> And like any good prototype, I think that it needs to be redesigned
> from scratch.  I think that the result could easily be very similar in
> a lot of ways to the current scheme, but could still be a lot clearer.

> I have a very clear idea in my head as to how it could be done.  I
> tried to present in on the list a while ago (over a year a think) but
> I didn't get much (or any) response.  I'm happy to try again if anyone
> is interested.

I'm certainly curious to see if we could have something simpler (but
not too simple).  Even if people don't think it enough better to
change v4 at this point (and that is going to be a high hurdle),
it would be valuable as input to the next minor version.

---

From: "Sergey Klyushin" <[email-masked]>
Date: Wed, 31 Jan 2001 15:25:19 -0500

> same stateid (ie. not change it).
Probably, it's possible, if stateid based on lockowner, but not
lockowner+file pair.
If locking(open, close,..) state of the file was changed, new stateid should
be returned.
    > disagreement about whether the spec says that (e.g. Mike
    > Eisler believes it does)
    >
Looking more close to this problem, I would increment seqid, if operation
itself failed,
(e.g. LOCK), but would not increment seqid, if some precondition checking
failed (e.g.
checking stateid or seqid). Here are some more details.

1. seqid not changed if operation failed.

Server stores seqid X, client sends LOCK with seqid X+1. stateid OK, seqid
OK
Server tries to LOCK file, but operation fails (e.g. some other lockowner
locked file)
Server doesn't change seqid X and doesn't store error_reply (it may store
error_reply,
but it would be useless)
Client decrement seqid to X to be ready for next request
Client resend (not network replay) LOCK request with seqid X+1.
Everything starts from the very beginning.
Server doesn't know if it's replay or new request.

2. seqid changed if operation failed.

Server stores seqid X, client sends LOCK with seqid X+1. stateid OK, seqid
OK
Server tries to LOCK file, but operation fails (e.g. some other lockowner
locked file)
Server does change seqid to X+1 and does store error_reply
Client doesn't decrement seqid.
Client resend (not network replay) LOCK request with seqid X+2.
Now Server does know if it's replay or new request and because error reply
stored
Server may resend it from the cache.

I would prefer second scenario, but once again I would not change seqid if
some
precondition checking failed.

    > it would be valuable as input to the next minor version.
My personal opinion is to make first version of protocol much clear, even if
it
delays RFC approval. It will make our future much easy to clean it up
right now then to discover bugs in different implementations just because
misunderstanding of the protocol.

---

From: "Sergey Klyushin" <[email-masked]>
Date: Wed, 31 Jan 2001 17:25:46 -0500

Paragraph
14.2.18.  Operation 20: OPEN_CONFIRM - Confirm Open

says:

      The server must hold unconfirmed OPEN state until one of three
      events occur.  First, the client sends an OPEN_CONFIRM request
      with the appropriate sequence id and confirmation verifier within
      the lease period.  In this case, the OPEN state on the server goes
      to confirmed, and the nfs_lockowner on the server is fully
      established.

      Second, the client sends another OPEN request with a sequence id
      that is incorrect for the nfs_lockowner (out of sequence).  In
               ^^^^^^^                         ^^^^^^^^^^^^^^^
      this case, the server assumes the second OPEN request is valid and
      the first one is a replay.  The server cancels the OPEN state of
      the first OPEN request, establishes an unconfirmed OPEN state for
      the second OPEN request, and responds to the second OPEN request
      with an indication that an OPEN_CONFIRM is needed.  The process
      then repeats itself.

What Server should do if it receives second OPEN request with correct (X+1)
seqid?
Here are two possible answers:
1. Consider that sequence is confirmed, even without OPEN_CONFIRM. Server
sets confirmed state.
Personally, I don't like it. Server could receive OPEN_CONFIRM later with
seqid from first OPEN.
2. Consider this case like bad seqid also. My feeling, that it would work
better.

Sergey

---

From: Neil Brown <[email-masked]>
Date: Thu, 1 Feb 2001 15:30:08 +1100 (EST)

> same stateid (ie. not change it).

I can see, from a operational perspective, that you could argue this
 - all operations that effect state maintenance also have a seqid to
   ensure sequencing and
 - the operations that only have a stateid (read/write) will not be
   adversely affected by certain state changes - e.g. gaining a lock.

however I think that if you try to understand a protocol like NFSv4
from a purely operation perspective, you will get bad confused very
quickly (Well, maye you won't but I suspect that many would, myself
included).   Complexity needs to managed by abstraction (that is what
computer science is all about) and so we need to understand and reason
about the protocol at an abstract level.

When you are reasoning like this (abstractly), having a "stateid" that
"uniquely defines the locking state granted by the server for a
specific lock owner for a specific file" and then not changing that
stateid when the locking state changes is a recipe for disaster.

> disagreement about whether the spec says that (e.g. Mike Eisler believes 

ButButButButButButButButButButButButButBut
Numbers in [brackets] are seqids.  As our story opens, the "last
sequence number (L) received" is [L].

 Client say [L+1] "lock this bit of that file".
 Server says "No, there is a conflicting lock" but reply is delayed in
      transit.  As this is an error return (NFS4ERR_DENIED), the
      server doesn't touch the seqid (you seem to say), it leave it at
      [L]. 
 Client is impatient and again says [L+1] "lock this bit of that file".
 Client gets first reply and says "oh well, such it life".
 SomeotherClient unlocks "this bit of that file".
 Server gets resend.  The rpc reply cache has been busy and it doesn't
   catch this one. The seqid looks fine ([L+1] just like the last time), and
   the server notices that there is no conflicting lock now, so it
   returns with success. It updates the "last sequence number
   received" to [L+1] and stores "the response that was returned.
 Client doesn't notice this reply because it doesn't really care any
   more.  Maybe it was lost in transit anyway - it happens.
 Client tries to [L+1] "lock that other bit of that file".
   Now this has the same seqid as the last successful lock request
   for this lockowner, so "the stored response is returned" -
   success.

Net result: client thinks that it locked "that other bit" and server
thinks that it has "this bit" locked.  Not a good situation.

Am I missing something?

(all quotes from rfc3010 section 8.1.5).

> it would be valuable as input to the next minor version.
I'm working on it, but it wont be today.

NeilBrown