Subject: [nfsv4] Out of sync stateid scenario that looks difficult to resolve

From: Carl Burnett <[email-masked]>
Date: Thu, 11 Sep 2003 17:30:54 -0500

Consider the following:

1. Client opens file for read.

2. Another process opens the (same) file for write (using same open owner 
as #1).

3. Server gets and processes the request (#2), but client never gets reply 
(RPC timeout, N/W condition, impatient user hits ^c with an interruptible 
mount).

4. Process from step 1 issues a read to the server. This will be done with 
the stateid from step 1, while the server thinks the correct stateid comes 
from step 2. The server will return an error (STALE_STATEID).

There seems to be no graceful way to recover from this condition where the 
server and client stateids are hopelessly out of sync. How does a client 
recover without producing errors to the application?

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: 11 Sep 2003 18:55:37 -0400

>>>>> " " == Carl Burnett <[email-masked]> writes:

     > (RPC timeout, N/W condition, impatient user hits ^c with an
     > interruptible mount).

     > the stateid from step 1, while the server thinks the correct
     > stateid comes from step 2. The server will return an error
     > (STALE_STATEID).

Assuming that the server has received both OPEN calls, then the
stateid on the server in this case should be READ | WRITE. See
paragraph 8.11 in RFC3530...

IOW the read should succeed.

Cheers,
  Trond

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Sep 2003 18:28:56 -0500

> paragraph 8.11 in RFC3530...

Yes. The file has been upgraded to READ|WRITE.

> IOW the read should succeed.

No. The stateid returned after the first OPEN is no longer valid
because the second has been processed by the server.

It is the responsibility of the client to resend the OPEN request to
ensure that the client and server do not become out-of-sync.

In the case of an interrupted request (assuming the application is
interrupted and will exit), the client should have an alternative
agent that resends (not RPC retransmit since that is not "allowed")
the OPEN request.  If the server had already processed it, it will
return the previous results per the matching seqid.  If it had not
been processed, it will be the second time.  In either case, the
client and server are now at a known state.  In this example, if the
application has exited and the file closed, the client would
presumably send a OPEN_DOWNGRADE().

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: 11 Sep 2003 19:51:46 -0400

>>>>> " " == Spencer Shepler <[email-masked]> writes:

     > No. The stateid returned after the first OPEN is no longer
     > valid because the second has been processed by the server.

     > It is the responsibility of the client to resend the OPEN
     > request to ensure that the client and server do not become
     > out-of-sync.

Blech...

     > In the case of an interrupted request (assuming the application
     > is interrupted and will exit), the client should have an
     > alternative agent that resends (not RPC retransmit since that
     > is not "allowed") the OPEN request.  If the server had already
     > processed it, it will return the previous results per the
     > matching seqid.  If it had not been processed, it will be the
     > second time.  In either case, the client and server are now at
     > a known state.  In this example, if the application has exited
     > and the file closed, the client would presumably send a
     > OPEN_DOWNGRADE().

So now I'm confused. Does this also mean that you cannot
upgrade/downgrade the open stateid if there is already a lock stateid
hanging off it?

Seems to me that we're going to have to go back to the old idea of the
client keeping 1 open owner for Read, a different one for Write, and a
third one for Read/Write. Leads to extra state on the server, but is
a much less fragile scheme...

Cheers,
  Trond

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Sep 2003 19:12:58 -0500

> upgrade/downgrade the open stateid if there is already a lock stateid

For a posix locking scheme, that doesn't apply.  The first close() for
a file will release all file locks (if they are left) and therefore
the lock stateid is no longer needed.

> a much less fragile scheme...

For the Solaris client, this is unnecessary.  Therefore, it is one
open_stateid per open_owner per file.

Spencer

---

From: eric kustarz <[email-masked]>
Date: Thu, 11 Sep 2003 17:39:31 -0700

Right, please see the connectathon presentation from last year we did on
this very subject:

http://www.connectathon.org/talks03/eric.pdf

(it even starts with a picture of Mike Eisler :) )

eric

---

From: Trond Myklebust <[email-masked]>
Date: 11 Sep 2003 20:49:42 -0400

>> upgrade/downgrade the open stateid if there is already a lock
    >> stateid hanging off it?

     > For a posix locking scheme, that doesn't apply.  The first
     > close() for a file will release all file locks (if they are
     > left) and therefore the lock stateid is no longer needed.

Are you perhaps assuming that open_owner == posix file handle (as per
the recommendation in the RFC). That is a concept which doesn't appear
to scale on the server side. We have already hit problems with it
under testing against existing commercial NFSv4 servers which quickly
run out of resources...

For that reason, we'd prefer to use a limited number of open_owners
(say related to the number of allowed simultaneous RPC requests on the
wire) which we share among all the users.

Now I am curious, though. Where in the RFC does it say explicitly that
OPEN will invalidate an existing stateid, and why should it be necessary?

Cheers,
  Trond

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Sep 2003 21:40:50 -0500

> wire) which we share among all the users.

We have yet to come across a resource issue with our own work to this
point.  It is good for a client to conserve but if the number of
clients active against a particular grows, the server will have the
same issues.  Of course, if a server is offering delegations, then the
presumably the number of open_owners the server sees will be very much
lower since the client will not have to do the over the wire OPENs.

Anyway, that really isn't the issue...

The approach you suggest will work and is allowed.  I believe my point
still holds however for posix based systems.  For locks to be present
at the server, there still needs to be an active open or
open_owner/stateid for that file.  In the design you suggest where
there are fewer open_owners, the open_owner will exist and when a
CLOSE or OPEN_DOWNGRADE is done, there shouldn't be any issues with
respect to an appropriately associated open_owner/stateid.

> OPEN will invalidate an existing stateid, and why should it be necessary?

The RFC doesn't require that a stateid change values on operations
like OPEN (upgrade) or OPEN_DOWNGRADE but it makes the server
implementation much easier with respect to tracking a single valid
stateid for a file.  The amount of state that needs to be kept is a
little less, the server can easily determine if pending reads/writes
fall under the appropriate stateid, and when debugging client and
server interaction it is easier to track the transitions that occur
with stateids.

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: 11 Sep 2003 23:55:54 -0400

> The approach you suggest will work and is allowed.  I believe
     > my point still holds however for posix based systems.  For
     > locks to be present at the server, there still needs to be an
     > active open or open_owner/stateid for that file.  In the design
     > you suggest where there are fewer open_owners, the open_owner
     > will exist and when a CLOSE or OPEN_DOWNGRADE is done, there
     > shouldn't be any issues with respect to an appropriately
     > associated open_owner/stateid.

I was really referring to the case in which 2 different processes OPEN
a file using the same open_owner:

Assume process 1 opens the file for write only, and sets a byte range
lock (defining a lock_stateid in the process). What happens when
process 2 opens the same file for reading? Is the lock_stateid
invalidated together with the open_stateid?

Furthermore, is process 1 then required to serialize its writes
w.r.t. the OPEN call of process 2? How does it know a priori that the
OPEN call actually refers to the same file on a server that allows
hard links and/or renames of open files?

     > The RFC doesn't require that a stateid change values on
     > operations like OPEN (upgrade) or OPEN_DOWNGRADE but it makes
     > the server implementation much easier with respect to tracking
     > a single valid stateid for a file.  The amount of state that
     > needs to be kept is a little less, the server can easily
     > determine if pending reads/writes fall under the appropriate
     > stateid, and when debugging client and server interaction it is
     > easier to track the transitions that occur with stateids.

Right, but given that we have a hierarchy of state (see the above
example), can this be made consistent with the model I have suggested?

Cheers,
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: 15 Sep 2003 13:20:00 -0400

>> Now I am curious, though. Where in the RFC does it say
    >> explicitly that OPEN will invalidate an existing stateid, and
    >> why should it be necessary?

     > easier to track the transitions that occur with stateids.

As I said earlier, the only case where this is a problem is OPEN,
since there the client does not a priori know what the filehandle is,
and so cannot predict which stateid is liable to be invalidated.

Would be consistent with RFC3530 to recommend as a "best practice"
that servers should send an OPEN_CONFIRM request in the particular
case where the result of the OPEN would be to invalidate an existing
stateid?

Since the original OPEN reply returns the filehandle, this would allow
the client to put its house in order before accepting the upgraded
stateid for the given file.

Cheers,
  Trond

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 15 Sep 2003 20:09:48 -0500

> invalidated together with the open_stateid?

For OPEN("upgrade"), no.  For OPEN_DOWNGRADE, potentially.  For CLOSE,
yes.  Once a lock_stateid is instantiated, an OPEN("upgrade") will not
invalidate that lock_stateid.

For a Posix based client that wants to have a small number of
open_owners, this is okay.  The client just has to reference count the
open_owner and OPEN access such that a OPEN_DOWNGRADE or CLOSE is not
done until locks have been released.  For Posix, the release of locks
occurs on the first close().

> hard links and/or renames of open files?

Technically, it doesn't have to.  The server will return
NFS4ERR_OLDSTATEID and the client can retry the request (presumably a
READ/WRITE) with the updated stateid.

As mentioned previously, this type of behavior can be mitigated in
the presence of delegations.

> example), can this be made consistent with the model I have suggested?

Yes.  I believe it can based on the protocol and the implementation
work our team has done.

Spencer