Subject: Stateid and seqid

From: "Sergey Klyushin" <[email-masked]>
Date: Fri, 26 Jan 2001 18:02:44 -0500

Q1. Can anybody explain why do we need both stateid and seqid?
(consider that stateid is shorthand to lockowner+file, and stateid includes
some
incrementing sequence inside itself)
Note, that there is no operation, that takes seqid only without stateid
(except OPEN, which is the
only operation to create stateid).

Q2.
Some time ago here was a discussion about the order of checking stateid,
seqid and lease.
As far as I remember conclusion was :
1. check stateid. OK means that stateid_seq_number =
stored_stateid_seq_number.
OLD_STATEID means stateid_seq_number = stored_stateid_seq_number - 1
BAD_STATEID some other errors.
2. check seqid. OK means seqid = stored_seqid + 1
replay means seqid = stored_seqid and NFS Server should return stored reply
error - everything else.
3. check lease

Question: how is it possible to receive request with the previous seqid
(seqid = stored_seqid)
and not the same stateid. Is it real retransmission?

Thanks,
Sergey

---

From: David Robinson <[email-masked]>
Date: Sat, 27 Jan 2001 10:35:49 -0800

> only operation to create stateid).

The original logic was that the client controls the seqid space
while the server controls the stateid space.  Stateid's are
not required to be a monotonically increasing integer while
seqid's are. Given a stateid the client cannot tell what the
next stateid may be. In theory a server may not change the stateid
after a transaction where the seqid is required, so they were
kept seperate.

> and not the same stateid. Is it real retransmission?

In theory, a client upgrading a lock from read to write (shared to
exclusive) does not require that a stateid be changed.  All outstanding
I/O requests are still valid because the client is given more
permissions.  But the seqid must change to prevent a subsequent
request such as unlock to be ordered correctly.

	-David

---

From: "Sergey Klyushin" <[email-masked]>
Date: Sat, 27 Jan 2001 19:48:25 -0500

> -----Original Message-----
    > From: David Robinson [mailto:[email-masked]]
    > Sent: Saturday, January 27, 2001 1:36 PM
    > To: Sergey Klyushin
    > Cc: [email-masked]; [email-masked]
    > Subject: Re: Stateid and seqid
    >
    >
    > Sergey Klyushin wrote:
If implementation of NFS Server changes stateid on reply
(otherwise I don't see why to return stateid, if it's not changed),
I can't see correct processing of replay scenario.
I'll try to explain with more details.
Replay request has seqid = stored_seqid, but request's stateid !=
stored_stateid.
So, to follow above scenario (first check stateid, then check seqid) error
BAD_STATEID or
OLD_STATEID will be returned, before checking seqid.

BTW, seqid = stored_seqid doesn't mean real replay, because full request
must be the same, and
NFS Server should keep previous request to compare it with current one. In
some implementation
it's done on RPC layer.

From my point of view, including index inside stateid is enough to monitor
replay without seqid at all.
(which makes live more easy)

Sergey

---

From: "Kendrick M. Smith" <[email-masked]>
Date: Sun, 28 Jan 2001 13:26:32 -0500 (EST)

> OLD_STATEID will be returned, before checking seqid.

I suppose this means that, to guarantee correct replay protection, a 
server implementation must retain the last expired stateid (per lockowner)
in its stateid table.  This way, if a replay is received, the server can
recognize the replayed stateid, compare seqids, and replay its cached
response instead of erroneously returning NFS4ERR_OLD_STATEID.  This is a
good point which, to my knowledge, hasn't come up before.  Should
something like this be written into the spec (or at least the
implementor's guidelines)?

> (which makes live more easy)

It sounds to me like you are suggesting removing seqids from the protocol
entirely, and relying on stateids to provide replay protection for
non-idempotent operations.  Perusing the protocol, it seems that every
non-idempotent operation does change a stateid, so it should be possible
to use stateids for replay protection and still have the correct replay
semantics.  So I think that this is really a performance issue, not a
correctness issue.  It is the same issue as, what "granularity" should the
server use when storing responses in its replay cache?  Under the current
spec, the server will cache the most recent outstanding request on a
per-lockowner basis.  If replay protection were done using stateids
instead of seqids as you suggest, the server would cache the most recent
outstanding request on a per-(lockowner, file) basis, so the replay cache
would consume more server memory.  The flip side of this is that,
currently, non-idempotent operations have to be serialized on a
per-lockowner basis, i.e. a client can't have two concurrent outstanding
requests with the same lockowner (or seqids might get out of sync on the
client & server).  If replay protection were done using stateids,
non-idempotent operations would be serialized on a per-(lockowner, file)
basis.  A client could have two concurrent requests with the same
lockowner, as long as they were for different files.  So the issue of
seqids vs. stateids for replay protection is really a performance tradeoff
between increased server memory and better concurrency properties.  My
personal intuition is that it is best to continue using seqids for replay
protection, since the memory cost of caching on a per-(lockowner, file)
basis is probably quite significant, but this is something that can only
really be settled by measurement and profiling.  Does anyone know of
performance studies that might shed some light on this issue?   - k

---

From: Neil Brown <[email-masked]>
Date: Wed, 31 Jan 2001 10:44:04 +1100 (EST)

> only operation to create stateid).

I spent a while trying to understand these issues properly a while
back and the simplest answer that I have to your question is:

  Every request that might change state contains a stateid and a
  seqid.
    If the operation is successful and state changes, then the stateid
      changes.
    If the operation fails and the state doesn't change, then the
      seqid changes.

This doesn't describe the full picture, as the seqid changes if the
operation succeeds, but I think it does draw out the central purpose of
the seqid - that being to record failed attempts to change state.

I actually think, and I believe that the recuring discussions support the
idea, that the current model for state management is much too complex.
It is a great prototype as it deal with (I believe) all the important
issues and presents real solutions to them.  However it seems to be
much too complex to be understood by implementors (and possibly even
by the designers, I'm not sure).

And like any good prototype, I think that it needs to be redesigned
from scratch.  I think that the result could easily be very similar in
a lot of ways to the current scheme, but could still be a lot clearer.

I have a very clear idea in my head as to how it could be done.  I
tried to present in on the list a while ago (over a year a think) but
I didn't get much (or any) response.  I'm happy to try again if anyone
is interested.

NeilBrown

---

From: Brent <[email-masked]>
Date: Tue, 30 Jan 2001 17:13:52 -0800

> by the designers, I'm not sure).

Count me as one who is not thrilled with the level of complexity of
the state management.  It's certainly raised a lot of questions as
we've gotten our fingers dirty attempting to implement it.  I worry
too that it will cause interoperability problems, not for normal
operation, but during those edge conditions where clients are
retransmitting, servers crashing and networks dropping packets.

On the other hand, now that we're building this protocol into
running code, we're reluctant to change the protocol without
a good reason.  Any protocol change means code redesign, rewrite,
retest and another spin on the RFC. 

> I didn't get much (or any) response.  I'm happy to try again if anyone

Go ahead and run it by us again.  With some implementation experience
behind us, we may view your proposal differently.  Though I recommend you
preface your proposal with the reasons why the current state model
is flawed.

	Brent