Subject: RE: [nfsv4] more re: client re-using lock_owner

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 20 May 2005 08:49:26 -0400

OK, I see.  Since the spec allows the server to forget owner state
when there are no associated opens, it follows that returning OK
and requiring open-confirm cannot be illegal, since a server which
has dropped that open state will do precisely that.

However, even though it is not illegal, it still strikes me as
a very bad thing to do.  You have enough information to figure
out that the seqid is wrong so that it just seems better to tell 
the client that, even though you might not be able to if you had
dropped that info.  With door #2 you have a server in which 
getting a bad owner seqid when you have no opens brings on an 
attack of amnesia, so you always respond as if you had dropped
the owner info.   

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]
Sent: Thursday, May 19, 2005 8:01 PM
To: [email-masked]
Subject: [nfsv4] more re: client re-using lock_owner

Well, I'm happy to continue using Door #1, since that's what my code already
does. However, I'm a little nervous that I've missed some fundamental
concept, since I can't see what the problem with Door #2 would be.

Here's an example scenario in more detail:
The client does...
1	Opens "xx" with lock_owner "P1" and seqid#0, gets Stateid0
2	Does an OpenConfirm (as required by the server) with seqid#1, so the
		seqid# is properly initialized, using Stateid0
3	Opens "yy" with lock_owner "P1" and seqid#2, gets Stateid1
4	Opens "zz" with lock_owner "P1" and seqid#3, gets Stateid2
5	Closes "xx" with seqid#4 and Stateid0
6	Closes "yy" with seqid#5 and Stateid1
7	Closes "zz" with seqid#6 and Stateid2

Now, I can see why #2 is needed and I can see why the seqid# has to
be correct for #2-#7.

But, it now seems that lock_owner "P1" is back to what I might call State0.
(No open/lock state associated with lock_owner "P1", which has an
 OpenConfirm'd seqid#.)
Continuing on...
8a	Server throws away lock_owner "P1" (allowed, but not required by
		Sec. 8.1.7 as I understand it)
8	Client opens "aa" with lock_owner "P1" and seqid#0

Now, if 8a has been done, the server would allow 8, requiring an
OpenConfirm for the new seqid# sequence starting at 0.

However, if 8a hasn't been done by the server, 8 will not be allowed and
the server will reply NFS4ERR_BADSEQID, for what I meant by Door #1.

Whereas what I meant by Door #2 was to allow 8, re-initializing the
seqid# sequence at 0 and requiring another OpenConfirm.
(Note that this case only occurs after at least the sequence
of Open, OpenConfirm and Close have occurred. I am not talking about the
case of just an unconfirmed Open.) Since only one of the Ops in sequence
could be replayed, I just can't see what the problem with doing this is?

Am I making sense? rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: David Robinson <[email-masked]>
Date: Fri, 20 May 2005 12:51:28 -0500

> attack of amnesia, so you always respond as if you had dropped

This seens unduly harsh. Section 8.1.8 says that the server upon
seeing a lock_owner for the "first time" will require an
OPEN_CONFIRM. But the spec fails to talk very clearly
about what is the "first time". It is clear that while there
is active state for that lock_owner it can't be the first
time and the server MUST return back NFS4ERR_BADSEQID, the
client has screwed up state that it is required to maintain.

However, if there is no active state, how long a time period or
what events must occur before the reuse of a lock_owner
qualifies to be the "first time"? And how is a client expected
to be able to reliably determine that fact?  The other factor is
that if the client had done something major like rebooted and
lost any history, it may not have any way to construct a valid
seqid so it must start with a new lock_owner. Seemingly harmless
to do such a thing, but I can imagine scenerios where a client
would rather not keep trying new values until it hit one that
worked. Eventually the server will forget that lock_owner and
will think it is the "first time", so why not make that a
more pro-active event?

So I would like Door #2 Monty...

Reasonable behavior should be:

1) Client MUST use the correct next seqid if there is active state
    for that lock_owner.
2) Client SHOULD use the correct next seqid if there is no
    active state for that lock_owner.
3) The server MUST return NFS4ERR_BADSEQID if the OPEN lock_owner
    has an incorrect seqid and there is active state for that
    lock_owner.
4) The server SHOULD allow the OPEN if the lock_owner has the
    correct seqid and there is no active state for that lock_owner.
5) The server SHOULD (MUST?) return OPEN4_RESULT_CONFIRM if the
    lock_owner has an incorrect seqid and there is no active state
    for that lock_owner.

So in normal processing everyone SHOULD make a reasonable effort
to maintain enough information to minimize the need to use
OPEN_CONFIRM. But in the absence of active state, if one side
forgets, it is useful to allow the lock_owner to be reused
by sending OPEN4_RESULT_CONFIRM instead of NFS4ERR_BADSEQID.

	-David

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 20 May 2005 14:01:39 -0400

On Fri, May 20, 2005 at 12:51:28PM -0500, David Robinson wrote:
> to be able to reliably determine that fact?

Why would it need to?  I can't see why a client would need to reuse a
lock_owner.

> The other factor is that if the client had done something major like
> rebooted and lost any history, it may not have any way to construct a
> valid seqid so it must start with a new lock_owner.

In this case it's going to do a setclientid and blow away all its old
state anyway, isn't it?

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 20 May 2005 14:30:55 -0400

> will think it is the "first time", so why not make that a

Errm.. If the client has rebooted, then the server is expected to clear
all state as soon as the SETCLIENTID negotiation is done.

As for the "first time" issue: our client, at least, is coded so that we
don't reuse the same open owner strings. Encoding a uniquifier is hardly
an unsolvable problem, and it also ensures that we don't end up
confusing the replay caches on the server.

>     lock_owner has an incorrect seqid and there is no active state

I assume you too mean "open_owner" and not lock_owner?

So assume 2 OPEN requests come in for the same open_owner. One turns out
to be a replay of an old OPEN from a previous generation of the same
open owner, and the other one is an actual new request: how should the
server deal with it assuming rule (5)?

AFAICS, this would appear to be basically the same problem that TCP has
on connections to the same port. Once you close the connection, then you
need a suitable moratorium period in order to allow replays to die out
before you allow a new connection. The length of that moratorium,
however, is more of a "best practices" issue rather than a protocol
issue since it depends on the nature of the transport used (10GigE,
10Mbit, multipathing, really old and stupid routers, etc.).

Cheers,
  Trond

---

From: David Robinson <[email-masked]>
Date: Fri, 20 May 2005 18:52:14 -0500

> Why would it need to?  I can't see why a client would need to reuse a

There is no need to, but I don't see any reason to force them to
when the open_owner is not related to any active state.

> In this case it's going to do a setclientid and blow away all its old

As others have pointed out, bad example, I withdraw it. But I think
the points still hold.

 > all state as soon as the SETCLIENTID negotiation is done.

yup....

:g/lock_owner/open_owner/

 > confusing the replay caches on the server.

Yes, it is not hard to create new open_owners, but is it required?
I don't see how this will affect reply caches as those should be
based on XID and IP addresses.

 > server deal with it assuming rule (5)?

If there is a replay, then it is presumed that the first OPEN
succeeded and there is active state which means rule #5
doesn't come into play.  It is rule #3 and NFS4ERR_BADSEQID as
some servers are already doing today. If the initial request
never actually made it to the server, then there is no active
state and client never saw a reply, so everything just
works as if it were the first request.

 > 10Mbit, multipathing, really old and stupid routers, etc.).

But in TCP we have only the sequence number to defend against replays,
in NFS we have the RPC level XID and the open_owner seqid space
to defend against wayward packets (also it would have to be on a new
TCP connection).

I assume this is Mike's obfuscated reply address.

[email-masked] wrote:
 > Here's what bothers me about #2. Let's say reason for
 > the re-used open_owner is not because the client has
 > forgotten, but because of a retry of an OPEN,
 > but the client still remembers the open_owner state.

If the previous OPEN suceeded the server will have state
so the replay must cause NFS4ERR_BADSEQID and never an
open confirm.  So this will act just as you expect.

 > If we return NFS4ERR_BADSEQID,
 > then this doesn't perturb the existing sequence number state
 > for the open_owner. If we request open confirmation, then
 > thr server has two choices:

 > 1. Perturb the state. So if the client in fact had not forgotten
 >    about the open_owner's previous sequence number, the next time
 >    client goes to use a sequence number from the previous use of
 >    the open_owner, he gets NFS4ERR_BADSEQID. And gets very
 >    confused.

This would only happen if we have a buggy client. If the server
requests an open_confirm he is telling the client that they
are establishing new state. If a client holds onto
its old state  after getting a OPEN4_RESULT_CONFIRM it is
just broken.

 > 2. Provisionally perturb the state to address the issue of (1)
 >    above. The server has to maintain two parallel states
 >    (similar to Schrodinger's Cat experiment), one
 >    of which collapses based on a subsequence use.

This shouldn't be needed. The server either has active state to which
it must reply NFS4ERR_BADSEQID if it is incorrect, or it
has no active state to which it requests an open_confirm if
it is not the next seqid.

When the server has no active state, the visible response to the
client is no different than if the OPEN was the first one seen
by the server and the open_owner can be safely reused. The only
difference is if the client uses the next seqid for the open_owner
the open_confirm is unneeded, but this is what we have already.

	-David

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 20 May 2005 23:22:20 -0400

> based on XID and IP addresses.

NFSv4 does not rely on RPC to provide only-once semantics. See section
8.1.5 and 8.1.6.

Note: this is one of the areas where RFC3530 also confuses lockowners
and open_owners. Those sections discuss the behaviour for CLOSE, LOCK,
LOCKU, OPEN, OPEN_CONFIRM, and OPEN_DOWNGRADE, so they are talking about
both - it is a historical accident that RFC3010 only talked about
lockowners

> works as if it were the first request.

Why? What is stopping something like the following from happening:

<connection 1>
OPEN
||||
<network partition or server thread hangs>
			<client initiates new connection>
					OPEN
					CLOSE

					OPEN
<network partition/hang clears>
||||
first OPEN is processed on server

You lose all ordering guarantees when the client is expected to
reconnect every time it replays a request.

> to defend against wayward packets (also it would have to be on a new

NFSv4 does _not_ allow you to rely on XIDs. (See the sections listed
above.) Rather, the rules for ensuring only-once semantics are given by
the paragraph on Page 73 that says:

        If a request (r) with a previous sequence number (r < L) is
        received, it is rejected with the return of error
        NFS4ERR_BAD_SEQID.  Given a properly-functioning client, the
        response to (r) must have been received before the last request
        (L) was sent. If a duplicate of last request (r == L) is
        received, the stored response is returned. If a request beyond
        the next sequence (r == L + 2) is received, it is rejected with
        the return of error NFS4ERR_BAD_SEQID.

I can see no mention anywhere in the RFC of the word "XID", nor do I see
anything that would allow the server to assume it can override the above
rule should the client resend an RPC request using a different XID or on
a different TCP connection.

Cheers,
 Trond

---

From: David Robinson <[email-masked]>
Date: Sun, 22 May 2005 18:11:39 -0500

> reconnect every time it replays a request.

Having the server request an OPEN_CONFIRM vs BADSEQID doesn't
change the replay semantics.  You didn't put numbers in your example
so lets use more details:

1) Client sends OPEN oo1 seq1
							it gets delayed a long time
2) Client replays OPEN oo1 seq1 on new connection
3) Server performs OPEN now expecting seq2
4) Client sends CLOSE seq2
5) Server performs CLOSE and now has no active state

If the client's original OPEN now gets unstuck and gets
presented to the server, what happens?

The server can legally be on one of two conditions, it either
remembers that it is expecting seq3 or may not have remembered
or had its internal variables garbaged collected.

If the server returns BADSEQID (which it legally can) the
client will see that but it also will know that it is
for an RPC that it has already discarded when it issued the
replay, the XID is what governs this. A client replaying
a request on a different connection with the same XID
is just a broken RPC client. XID reuse can only be done
for transport level retransmits. This is independent of
a duplicate request cache being used or not.

If the server returns a request for OPEN_CONFIRM the client
is in the same condition as above. It has already disposed
of the initial OPEN and will fail to OPEN_CONFIRM ending
the chain. This is not unlike an impatient client abandoning
a normal first OPEN before the server responds.

In your example, it also appears to have the condition
where after the CLOSE a new OPEN has succeeded on the
new connection before the original OPEN gets processed.
In this case the server MUST return BADSEQID because it
has active state and it is an old value. If I recall the
original discussions on this, it was proposed that the
server simply drop the request as the client has clearly
already moved past the original request. But it was decided
to still respond in case some client could use this information
in some way outside the required behavior (a clue that there
may be bad hardware in the network?).

> a different TCP connection.

The XID is a bit of a red herring, but it is still true that NFS
has more information and state than TCP so it is not a correct
analogy to compare the operation with that of TCP port reuse
moritoriums. In TCP it must maintain the old sequence number
because if it doesn't it has no way to detect if the packet
is from an old connection. With NFS level OPENs there are two
states that the server may be in, it may either have active
state on the openowner or it is maintaining the last seqid
as an optimization, in this case it can defend against replays.
The other case is if it has never seen an openowner or has
garbage collected its state, it request an OPEN_CONFIRM
effectively asking the client 'are you sure'? The lack
of confirming is what requires TCP to have the moritorium.

There is no weakening of replay prevention by having
the server request an OPEN_CONFIRM when there is no
active state instead of just issuing a BADSEQID. In fact
it is exactly the same as a replay that arrives after
a server garbage collects any cached open owner state.
The advantage is that it lessens the requirement that
the client not reuse open_owners and takes out one
round trip if it does reuse.

	-David

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 23 May 2005 13:21:06 -0400

> a duplicate request cache being used or not.

Yes, however you are assuming that ignoring a reply is always an
acceptable thing to do. In a stateful model, like we have here, the
client may be missing a state change.

> may be bad hardware in the network?).

The server is not allowed to return BADSEQID if the sequence number
happened to be correct as in the following race.

<connection 1>
OPEN (seqid=0)
OPEN_CONFIRM(seqid=1)
OPEN (seqid=2)
||||
<network partition or server thread hangs - client gives up on request>
			<client initiates new connection 2>
					OPEN(seqid=2)
					CLOSE(seqid=3)
---
					OPEN (seqid=0)
					OPEN_CONFIRM (seqid=1)

<network partition/hang clears on connection 1>
||||
OPEN (seqid=2) is processed on server

If so, and if the client discards the reply (as it normally would,
because the XID didn't correspond to any outstanding requests) then it
will suddenly find that the server is returning NFS4ERR_OLD_STATEID on
subsequent READs and WRITEs.

Furthermore, the server thinks that the client is holding a share lock
and/or a delegation on the file that the client didn't record because it
discarded the reply. That will lead to a lot of delays when some
conflicting share lock comes in and/or the server tries to recall said
delegation. The share lock conflict may even require administrator
intervention in order to be solved.

There are also other interesting variants on the above scenario of the
form

<connection 1>
OPEN (seqid=0)
OPEN_CONFIRM(seqid=1)
OPEN (seqid=2)
||||
<network partition or server thread hangs - client gives up on request>
			<client initiates new connection 2>
					OPEN(seqid=2)
					CLOSE(seqid=3)
---
					OPEN (seqid=0)
					OPEN_CONFIRM (seqid=1)

<network partition/hang clears on connection 1>
||||
OPEN (seqid=2) is processed on server
					OPEN/CLOSE/LOCK/... (seqid=2)

...where the server sees the last request as a replay of the "invisible"
OPEN. Depending on how it has constructed its DRC, it may replay a bogus
reply.

Cheers,
  Trond

---

From: David Robinson <[email-masked]>
Date: Mon, 23 May 2005 13:35:27 -0500

> client may be missing a state change.

But this is a fundamental of an RPC, replies need to be processed
in the context of the caller. The the caller has abandoned
the original request when it issues the replay it has no
way to match up the now stale original reply. In theory
a caller could maintain state for all requests ever sent but
not replied to in order to do cleanup, but I don't know
any RPC implementations that do this. So the stale reply will
just get dropped. More likely a client will drop the connection
before issuing the replay so it doesn't have to worry about
stale replies.

> happened to be correct as in the following race.

But this race exists today in existing implemenations.  Simply insert
a sufficently long pause after the CLOSE(seqid=3) so that the server
has garbaged collected any old open_owner state as described in 8.1.7
and this exact scenerio will happen. The server will not send
BADSEQID because it is now a "first time" OPEN.

> subsequent READs and WRITEs.

This could also happen if the time is long enough that the seqid
wraps before the original OPEN gets processed. The only real
defense here is that a client needs to track if it has replayed
an OPEN and on last close abandon that open_owner for as long as
possible. Again, this problem exists today if the server garbage
collects its open_owner state. Clients must be very careful about
issuing replays, I don't think it is right to ask the server to
maintain perpetual state to cover all the edge cases that are
better elimimated by the client being smarter on replays.

But this is not the case that we originally started talking about,
replays are rare. What should a server respond with when it gets
an OPEN with an out of range seqid and there is no active state
on that open_owner. The server can, and will, respond back with
either a request for an OPEN_CONFIRM or a BADSEQID. I still
contend that it is nicer for the server to request the OPEN_CONFIRM.

	-David

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 23 May 2005 15:08:29 -0400

> BADSEQID because it is now a "first time" OPEN.

No! That race does not currently exist, since existing clients are _not_
assuming that they can reuse open owners.

At best, the current scenario leads to a hanging OPEN(seqid==0) (which
does not involve any state being set up until an OPEN_CONFIRM has been
issued). Replaying the last CLOSE(seqid==XXX) should not normally cause
state to be set up again, nor will replaying Some_OPERATION(seqid==YYY <
XXX) => BADSEQID.

> better elimimated by the client being smarter on replays.

Right, and this is why adding a new rule to RFC3530 that states that
open owners can be reused by clients would be bad. To do so increases
the number of possible race scenarios for no gain in functionality.

> contend that it is nicer for the server to request the OPEN_CONFIRM.

By returning BADSEQID, the server is helping client writers to debug
their implementations instead of just giving them more rope. I would
contend that is far more helpful in the long run.

Cheers,
  Trond