Subject: RE: Stateid and seqid

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 28 Jan 2001 20:30:15 -0800

Kendrick Smith wrote:
> 
> On Sat, 27 Jan 2001, Sergey Klyushin wrote:
> 
> > If implementation of NFS Server changes stateid on reply
> > (otherwise I don't see why to return stateid, if it's not changed),
> > I can't see correct processing of replay scenario.
> > I'll try to explain with more details.
> > Replay request has seqid = stored_seqid, but request's stateid !=
> > stored_stateid.
> > So, to follow above scenario (first check stateid, then check seqid)
error
> > BAD_STATEID or
> > OLD_STATEID will be returned, before checking seqid.
> 
> I suppose this means that, to guarantee correct replay protection, a 
> server implementation must retain the last expired stateid (per lockowner)
> in its stateid table.  This way, if a replay is received, the server can
> recognize the replayed stateid, compare seqids, and replay its cached
> response instead of erroneously returning NFS4ERR_OLD_STATEID.  This is a
> good point which, to my knowledge, hasn't come up before.  Should
> something like this be written into the spec (or at least the
> implementor's guidelines)?

I don't think you have to separately save the last superseded (I think
that's what you mean here; expired has some other connotations) stateid.

What I do is first use the stateid to map to an entry in my table of
state structures.  At this point, I may generate STALE_STATEID and
BAD_STATEID errors, but not OLD_STATEID.  This requires that I know
the range of valid values of the stateid subfield which is incremented
on each change, but I need that anyway to properly distinguish between
BAD_STATEID and OLD_STATEID.

Once I have the address of a state structure, I can use this to find the
corresponding owner structure which has the current seqid and replay 
information.  So at this point, I can notice a replay and respond.

If it is not a replay (and we don't have a BAD_SEQID error), then I
can check the stateid against the current stateid in the state structure
and generated aN OLD_STATEID error if appropriate.

I thought this was discussed in the session Andy organized regarding
order-of-processing issues at the last bake-off.  I thought this
was on the flow-charts that we came up with.  Then I again I was
pretty tired so maybe my memmory is faulty.  In any case I think this
should be documented somewhere (implementation RFC?).
 
> 
> > BTW, seqid = stored_seqid doesn't mean real replay, because full request
> > must be the same, and
> > NFS Server should keep previous request to compare it with current one.
In
> > some implementation
> > it's done on RPC layer.
> > >From my point of view, including index inside stateid is enough to
monitor
> > replay without seqid at all.
> > (which makes live more easy)
> 
> It sounds to me like you are suggesting removing seqids from the protocol
> entirely, and relying on stateids to provide replay protection for
> non-idempotent operations.  Perusing the protocol, it seems that every
> non-idempotent operation does change a stateid, so it should be possible
> to use stateids for replay protection and still have the correct replay
> semantics. 

I don't think so.  In the case of open, I don't see how you can do without
a seqid.  The fact that it generates a stateid (updated or not) doesn't
give you any replay protection.

I send an open for read-write, deny=read-write.  It succeeds and I send
back a stateid.  The reply is lost.  A replay of the OPEN is sent and
it fails because open for read or write is denied, which is definitely
not what you want.  You might fall back on various kind of ad hoc replay 
protection mechanisms specific to OPEN but that doesn't give me a very good 
feeling.

> So I think that this is really a performance issue, not a
> correctness issue.  It is the same issue as, what "granularity" should the
> server use when storing responses in its replay cache?  Under the current
> spec, the server will cache the most recent outstanding request on a
> per-lockowner basis.  If replay protection were done using stateids
> instead of seqids as you suggest, the server would cache the most recent
> outstanding request on a per-(lockowner, file) basis, so the replay cache
> would consume more server memory.  The flip side of this is that,
> currently, non-idempotent operations have to be serialized on a
> per-lockowner basis, i.e. a client can't have two concurrent outstanding
> requests with the same lockowner (or seqids might get out of sync on the
> client & server).  If replay protection were done using stateids,
> non-idempotent operations would be serialized on a per-(lockowner, file)
> basis.  A client could have two concurrent requests with the same
> lockowner, as long as they were for different files.  So the issue of
> seqids vs. stateids for replay protection is really a performance tradeoff
> between increased server memory and better concurrency properties.  My
> personal intuition is that it is best to continue using seqids for replay
> protection, since the memory cost of caching on a per-(lockowner, file)
> basis is probably quite significant, but this is something that can only
> really be settled by measurement and profiling.  Does anyone know of
> performance studies that might shed some light on this issue?   - k

I think it would be possible to serialize all opens and  closes for a 
given lockowner on the on hand, and then use a separate mechanism
stateid-only or stateid-plus-file-specific seqid to handle sequencing
of locks and unlocks on per-(lockowner+file) basis.  My guess is that
the performance benefit of this approach would be limited to very
few applications.  Most multi-threaded apps that would want to have
multiple locks or unlocks in flight would probably be working most
of the time on a single file so would get no benefit from a scheme
that allowed multiple requests to be in-flight only when they were
to different files.

---

From: "Sergey Klyushin" <[email-masked]>
Date: Mon, 29 Jan 2001 19:56:30 -0500

> > > So, to follow above scenario (first check stateid, then check
seqid) error
    > BAD_STATEID and OLD_STATEID.
How "old" could be stateid? Probably, any operation, that takes both stateid
and seqid,
makes range of valid values of the stateid subfield equal 1?
BTW, how useful ERROR_OLD_STATEID for client? What client suppose to do
after receiving
this error?
    > information.  So at this point, I can notice a replay and respond.
So, you don't store stateid (or even full request) for reply message, but
you are based
only on seqid = stored_seqid? How you could be sure, that it's real replay,
but
not logical error?
I want to be sure at this point, that I understand replay correctly: for me
it's real
retransmission of exactly the same package. Is it true?
BTW, do you check lease time before reply?
    > can check the stateid against the current stateid in the
    > state structure and generated aN OLD_STATEID error if appropriate.
    >
    > > > NFS Server should keep previous request to compare it with current
one. In
    > > > some implementation it's done on RPC layer.
monitor
    > > > replay without seqid at all. (which makes live more easy)
    > >
I don't see any place when you need both seqid and stateid in one request.
OPEN may take seqid and generate stateid, and stateid will be used later
without seqid
But, probably, changing seqid to stateid is really too late now
(even if it's needed only for OPEN).

    > > per-lockowner basis.
Indeed, Server should keep cache per lockowner, but current agreement of
stateid
is lockowner+file. That's why I don't believe, that checking seqid without
stateid
is enough for replay indication.
    > > If replay protection were done using stateids

---

From: "Kendrick M. Smith" <[email-masked]>
Date: Tue, 30 Jan 2001 11:10:46 -0500 (EST)

On Mon, 29 Jan 2001, Sergey Klyushin wrote:

>     > I don't think you have to separately save the last superseded (I 
>     > think that's what you mean here; expired has some other
>     > connotations) stateid.
>     >
>     > Once I have the address of a state structure, I can use this to
>     > find the corresponding owner structure which has the current seqid
>     > and replay information.  So at this point, I can notice a replay
>     > and respond.
>     >

In our server implementation, we can't determine the lockowner 
corresponding to an old stateid without retaining an entry in our stateid
table.  (We found doing this to be impractical using 64-bit stateids.)  So
it seems that retaining the last superseded stateid may or may not be an
issue, depending on the server implementation.  I do think that this would
be a useful remark for the implementor's guidelines (but probably not
needed in the spec).

> after receiving this error?

I was wondering this too.. On our client, we couldn't really make use of
the distinction between NFS4ERR_OLD_STATEID and NFS4ERR_BAD_STATEID
(although NFS4ERR_STALE_STATEID is certainly useful, as an entry point to
the server-reboot recovery code).  Offhand, does anyone know the rationale
for making seperate errors NFS4ERR_OLD_STATEID, NFS4ERR_BAD_STATEID?

> So, you don't store stateid (or even full request) for reply message,
> but you are based only on seqid = stored_seqid? How you could be sure,
> that it's real replay, but not logical error? I want to be sure at
> this point, that I understand replay correctly: for me it's real
> retransmission of exactly the same package. Is it true? BTW, do you
> check lease time before reply?

We only retransmit the response if the old request and new request are
byte-by-byte identical, including rpc headers.  The client's lease is
checked and renewed only if the seqid is correct.  Is this what everyone
else is doing?

> (even if it's needed only for OPEN).

It does seem that, if per-lockowner seqids were used only for OPEN, and
per-(lockowner, file) stateids were used everywhere else, replay
protection would still be ensured.  But this would have the side effect of
requiring the server to cache replays on a per-(lockowner, file) basis; my
feeling is that this would negatively impact performance.

Kendrick