Subject: [nfsv4] protocol question about delegation stateid

From: Olga Kornievskaia <[email-masked]>
Date: Wed, 28 Jan 2015 21:10:02 -0500

Hi folks,

Can somebody clarify if the server is *required* to update the sequence
number of the delegation stateid when it receives an OPEN for which it has
already gave out a delegation before? Client sends one OPEN and server
gives out a delegation, then client sends another OPEN and server is
choosing returning a delegation again.

If the server is not required to update the delegation stateid sequence,
there is a problem with a racing DELEGRETURN and another OPEN from the same
file. Client is in the process of returning a delegation (due to resource
constraints) and a different thread on the client is opening the same file
it (had) the delegation for (at this point the client doesn't see the
delegation and thus sends the open request to the server). The server can
get the OPEN before the DELEGRETURN and from server's perspective it knows
the client has a delegation. If it does not increment the seqnumber when it
returns a delegation in OPEN, when the server receives a DELEGRETURN, the
server assumes the client is returning the delegation. However, the client
finishes processing the reply from the OPEN and stores the delegation. The
use of this delegation later results in BAD_STATEID error.

Let me state that server does have an option of not returning a delegation
to the client in this case. However, then the new open would lack the
benefits of a delegation and perhaps this should be acceptable solution to
deal with the race.

There is a general wording from the definition of the stateid that implies
that seqnumber should be incremented since delegation stateid is a stateid.
Handling of open stateids and need for incrementing is more clearly
specified in the spec.

from rfc 3530 (same wording in rfc 5661)
stateid4

                  struct stateid4 {
                    uint32_t        seqid;
                    opaque          other[12];
                  };

   This structure is used for the various state sharing mechanisms
   between the client and server.  For the client, this data structure
   is read-only.  The starting value of the seqid field is undefined.
   The server is required to increment the seqid field monotonically at
   each transition of the stateid.

What should the server behavior be when it receives an OPEN for which it
already gave out a delegation?

Thank you.

---

From: David Noveck <[email-masked]>
Date: Thu, 29 Jan 2015 12:25:45 -0500

> already gave out a delegation before?

I don't think it is required.  It might be a good idea if the server is
sending the same delegation to the client multiple times, but I don't think
servers should do that.

> Client sends one OPEN and server gives out a delegation, then client
sends another OPEN and server is choosing returning a delegation again.

Why would it choose to do that?  Although the text is not as clear as it
might be, the idea is that by returning a delegation in connection with the
open response, the server is bestowing a delegation.  I don't see why a
server would send the delegation back with the open response when there is
no transfer of a delegation to the client.  In that case, it would be using
this field to remind the client of the existing delegation.

> If the server is not required to update the delegation stateid sequence ..

I see your point.  If you are sending these delegations back in the open
response as a reminder, sequencing issues can arise.  if you increment
seqid, as you suggest, the DELEGRETURN would fail, in the case you are
worried about.  On the other hand most clients would return the delegation
with a zero seqid and not worry about the issue.

> Let me state that server does have an option of not returning a
delegation to the client in this case. However, then the new open would
lack the benefits of > a delegation and perhaps this should be acceptable
solution to deal with the race.

I assume that the client  knows about the delegation and does not require a
reminder in the open response, to get the benefits.  The delegation is a
promise to the client by the server and the server has received
the promise.

> There is a general wording from the definition of the stateid that
implies that seqnumber should be incremented since delegation stateid is a
stateid.
specified in the spec.

It has to increment the seqid when there is a "transition of the stateid'.
Adding another open (or downgrading) an open stateid is a transition, as is
changing the set of lock represented by a lock stateid.  These are
clearly transitions since they affect what the stateid represents.  In the
delegation case, there is no change in content.  The delegation represents
the same promise as previously and so there is no transition.

On Wed, Jan 28, 2015 at 9:10 PM, Olga Kornievskaia <[email-masked]>
wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>

---

From: "Adamson, Andy" <[email-masked]>
Date: Thu, 29 Jan 2015 18:16:26 +0000

> On Jan 29, 2015, at 12:25 PM, David Noveck <[email-masked]> wrote:
> 
> > Can somebody clarify if the server is *required* to update the sequence number of the delegation stateid when it receives an OPEN for which it has 
> I don't think it is required.  It might be a good idea if the server is sending the same delegation to the client multiple times, but I don't think servers should do that.
> 
> > Client sends one OPEN and server gives out a delegation, then client sends another OPEN and server is choosing returning a delegation again.
> 
> Why would it choose to do that?  Although the text is not as clear as it might be, the idea is that by returning a delegation in connection with the open response, the server is bestowing a delegation.  I don't see why a server would send the delegation back with the open response when there is no transfer of a delegation to the client.  In that case, it would be using this field to remind the client of the existing delegation.
> 
> I see your point.  If you are sending these delegations back in the open response as a reminder, sequencing issues can arise.  if you increment
> seqid, as you suggest, the DELEGRETURN would fail, in the case you are worried about.  On the other hand most clients would return the delegation with a zero seqid and not worry about the issue.
> 
> > Let me state that server does have an option of not returning a delegation to the client in this case. However, then the new open would lack the benefits of > a delegation and perhaps this should be acceptable solution to deal with the race.
> 
> I assume that the client  knows about the delegation and does not require a reminder in the open response, to get the benefits.

The issue we are hitting is the following race:

The client has a write delegation on the file. Do to resource limitations, the client reaps some inodes, choosing the inode with the write delegation to evict which triggers a DELEGRETURN.  Then the application opens the file, and since there is no longer a write delegation, the client sends an OPEN to the server instead of just performing a local open.  The OPEN hits the server before the DELEGRETURN, and the server replies to the OPEN with the delegation stateid that it had already given to the client, the delegation that the client is returning.  Then the DELEGRETURN hits, and the server removes the delegation that it just granted to the OPEN.

The client uses the just-granted delegation for I/O and receives an NFS4ERR_BAD_STATEID which ends up as an EIO to the application, as the client perceives that the server has reneged on the write delegation promise, so all bets are off.

>  The delegation is a promise to the client by the server and the server has received the promise.  
> 
> > There is a general wording from the definition of the stateid that implies that seqnumber should be incremented since delegation stateid is a stateid. 
> > Handling of open stateids and need for incrementing is more clearly specified in the spec. 
> 
> It has to increment the seqid when there is a "transition of the stateid'.  Adding another open (or downgrading) an open stateid is a transition, as is changing the set of lock represented by a lock stateid.  These are clearly transitions since they affect what the stateid represents.  In the delegation case, there is no change in content.  The delegation represents the same promise as previously and so there is no transition.

In the above race, in the client’s point of view, the delegation returned by the server is a NEW promise which the server then unilaterally withdraws.

A solution would be for the server to treat CLAIM_NULL OPENs on a file with an existing delegation as a transition, and return the same delegation stateid with the stateid seqid incremented.

Another solution would be for the server to not return a delegation stateid on a CLAIM_NULL OPEN on a file that already has a delegation.

—>Andy

> 
> On Wed, Jan 28, 2015 at 9:10 PM, Olga Kornievskaia <[email-masked]> wrote:
> Hi folks,
> 
> Can somebody clarify if the server is *required* to update the sequence number of the delegation stateid when it receives an OPEN for which it has already gave out a delegation before? Client sends one OPEN and server gives out a delegation, then client sends another OPEN and server is choosing returning a delegation again.
> 
> If the server is not required to update the delegation stateid sequence, there is a problem with a racing DELEGRETURN and another OPEN from the same file. Client is in the process of returning a delegation (due to resource constraints) and a different thread on the client is opening the same file it (had) the delegation for (at this point the client doesn't see the delegation and thus sends the open request to the server). The server can get the OPEN before the DELEGRETURN and from server's perspective it knows the client has a delegation. If it does not increment the seqnumber when it returns a delegation in OPEN, when the server receives a DELEGRETURN, the server assumes the client is returning the delegation. However, the client finishes processing the reply from the OPEN and stores the delegation. The use of this delegation later results in BAD_STATEID error.
> 
> Let me state that server does have an option of not returning a delegation to the client in this case. However, then the new open would lack the benefits of a delegation and perhaps this should be acceptable solution to deal with the race.
> 
> There is a general wording from the definition of the stateid that implies that seqnumber should be incremented since delegation stateid is a stateid. Handling of open stateids and need for incrementing is more clearly specified in the spec. 
> 
> What should the server behavior be when it receives an OPEN for which it already gave out a delegation?  
> 
> Thank you.
>

---

From: Rick Macklem <[email-masked]>
Date: Thu, 29 Jan 2015 18:00:39 -0500 (EST)

Andy Adamson wrote:
> > > Can somebody clarify if the server is *required* to update the
> > > sequence number of the delegation stateid when it receives an
> > > OPEN for which it has
> > I don't think it is required.  It might be a good idea if the
> > server is sending the same delegation to the client multiple
> > times, but I don't think servers should do that.
> > 
> > > Client sends one OPEN and server gives out a delegation, then
> > > client sends another OPEN and server is choosing returning a
> > > delegation again.
> > 
> > Why would it choose to do that?  Although the text is not as clear
> > as it might be, the idea is that by returning a delegation in
> > connection with the open response, the server is bestowing a
> > delegation.  I don't see why a server would send the delegation
> > back with the open response when there is no transfer of a
> > delegation to the client.  In that case, it would be using this
> > field to remind the client of the existing delegation.
> > 
> > I see your point.  If you are sending these delegations back in the
> > open response as a reminder, sequencing issues can arise.  if you
> > increment
> > seqid, as you suggest, the DELEGRETURN would fail, in the case you
> > are worried about.  On the other hand most clients would return
> > the delegation with a zero seqid and not worry about the issue.
> > 
> > > delegation to the client in this case. However, then the new
> > > open would lack the benefits of > a delegation and perhaps this
> > > should be acceptable solution to deal with the race.
> > 
> > I assume that the client  knows about the delegation and does not
> > require a reminder in the open response, to get the benefits.
> 
> The client has a write delegation on the file. Do to resource
> limitations, the client reaps some inodes, choosing the inode with
> the write delegation to evict which triggers a DELEGRETURN.  Then
> the application opens the file, and since there is no longer a write
> delegation, the client sends an OPEN to the server instead of just
> performing a local open.  The OPEN hits the server before the
> DELEGRETURN, and the server replies to the OPEN with the delegation
> stateid that it had already given to the client, the delegation that
> the client is returning.  Then the DELEGRETURN hits, and the server
> removes the delegation that it just granted to the OPEN.
> 
> The client uses the just-granted delegation for I/O and receives an
> NFS4ERR_BAD_STATEID which ends up as an EIO to the application, as
> the client perceives that the server has reneged on the write
> delegation promise, so all bets are off.
> 
I believe that the client must serialize DELEGRETURN for an FH with
any new OPEN for the same FH. Or, put another way, the client must delay
any OPENs on the file until after the DELEGRETURN reply is received
from the server. (In the FreeBSD client, a delegation return locks
out new Ops on the file for some time, since it must acquire
Opens against the server for all local opens on the file before doing
the DELEGRETURN op. Then, only after receiving the DELEGRETURN reply
does it do new opens, since it might have acquired an Open for the
FH, OpenOwner prior to the DELEGRETURN. It may be possible to implement
some parallelism here, but it's not going to be easy to get right?)

Since use of the "seqid" in a stateid for a delegation is "underspecified"
in the RFCs, I don't think a client can make any assumptions w.r.t. when
it will get incremented. 

For example, if you chose to expect a delegation stateid to be incremented
when a new delegation is issued for the same file-->the server would have
to remember that it had previously issued a delegation for the file *forever*
I think?
rick
ps: I can't even remember if the FreeBSD server resends a delegation
    stateid for a subsequent Open of the same file already delegated
    to the client or not (if you need to know, I can take a look at
    the code and let you know).

> > > There is a general wording from the definition of the stateid
> > > that implies that seqnumber should be incremented since
> > > delegation stateid is a stateid.
> > It has to increment the seqid when there is a "transition of the
> > stateid'.  Adding another open (or downgrading) an open stateid is
> > a transition, as is changing the set of lock represented by a lock
> > stateid.  These are clearly transitions since they affect what the
> > stateid represents.  In the delegation case, there is no change in
> > content.  The delegation represents the same promise as previously
> > and so there is no transition.
> 
> In the above race, in the client’s point of view, the delegation
> returned by the server is a NEW promise which the server then
> unilaterally withdraws.
> 
> A solution would be for the server to treat CLAIM_NULL OPENs on a
> file with an existing delegation as a transition, and return the
> same delegation stateid with the stateid seqid incremented.
> 
> Another solution would be for the server to not return a delegation
> stateid on a CLAIM_NULL OPEN on a file that already has a
> delegation.
> 
> —>Andy
> 
> > 
> > Can somebody clarify if the server is *required* to update the
> > sequence number of the delegation stateid when it receives an OPEN
> > for which it has already gave out a delegation before? Client
> > sends one OPEN and server gives out a delegation, then client
> > If the server is not required to update the delegation stateid
> > sequence, there is a problem with a racing DELEGRETURN and another
> > OPEN from the same file. Client is in the process of returning a
> > delegation (due to resource constraints) and a different thread on
> > the client is opening the same file it (had) the delegation for
> > (at this point the client doesn't see the delegation and thus
> > sends the open request to the server). The server can get the OPEN
> > before the DELEGRETURN and from server's perspective it knows the
> > client has a delegation. If it does not increment the seqnumber
> > when it returns a delegation in OPEN, when the server receives a
> > DELEGRETURN, the server assumes the client is returning the
> > delegation. However, the client finishes processing the reply from
> > the OPEN and stores the delegation. The use of this delegation
> > later results in BAD_STATEID error.
> > 
> > delegation to the client in this case. However, then the new open
> > would lack the benefits of a delegation and perhaps this should be
> > acceptable solution to deal with the race.
> > 
> > implies that seqnumber should be incremented since delegation
> > stateid is a stateid. Handling of open stateids and need for
> > incrementing is more clearly specified in the spec.
> >

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 29 Jan 2015 18:28:55 -0500

On Thu, Jan 29, 2015 at 6:00 PM, Rick Macklem <[email-masked]> wrote:

> > > > sequence ..

Rick,

I feel that the issue is precisely that that "use of seqid in a stateid for
delegation is underspecified" that it stating that the client must
serialize the OPEN with the DELEGRETURN in unfair.

Dave,

I hope Andy's reiteration of the example of the possible sequence of events
clarifies (a) linux client is not ignoring a delegation and simply sending
another open but that it already queued up a delegreturn to be sent (b) so
the client does need the "new" delegation to still be valid. I would argue
that this case should represent a "transition of stateid" for the
delegation stateid.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 29 Jan 2015 20:54:49 -0500 (EST)

Olga Kornievskaia wrote:
> 
> 
> 
> 
> 
> 
> On Thu, Jan 29, 2015 at 6:00 PM, Rick Macklem < [email-masked]
> > wrote:
> 
> 
> 
> 
> Andy Adamson wrote:
> > 
> > > On Jan 29, 2015, at 12:25 PM, David Noveck < [email-masked]
> > > >
> > > wrote:
> > > 
> I feel that the issue is precisely that that "use of seqid in a
> stateid for delegation is underspecified" that it stating that the
> client must serialize the OPEN with the DELEGRETURN in unfair.
> 
Well, if this was 2003 and no servers had shipped, it could be
"clarified" and all servers could conform to it. Since NFSv4.0,4.1 servers
have been shipping and are out there running, I can't see how you
can "require a semantic on the stateid.seqid" (for NFSv4.0,4.1 at least)
unless all extant servers can be shown to provide the semantic anyhow.

I am in the camp that doesn't see that the "state" of the delegation
has changed because a client has chosen to do an Open/Claim_null against
the server for the file instead of doing the Open locally.

Just FYI, if another new delegation for the same file is returned by
the FreeBSD server, the stateid.other field will change. However, I
do not believe that this is required by the RFCs and I do not believe
that you can assume this will be the case.

One other way the client might safely handle this is to not use the new
delegation sent in the open reply and then do another DELEGRETURN for the
stateid. (If it isn't a newly created one, the server will probably
generate an error reply for this DELEGRETURN, but the client doesn't
care if the DELEGRETURN fails, does it?)

rick
> 
> Dave,
> 
> 
> I hope Andy's reiteration of the example of the possible sequence of
> events clarifies (a) linux client is not ignoring a delegation and
> simply sending another open but that it already queued up a
> delegreturn to be sent (b) so the client does need the "new"
> delegation to still be valid. I would argue that this case should
> represent a "transition of stateid" for the delegation stateid.
> 
> 
> 
> 
> 
> 
> 
> 
> > > On Wed, Jan 28, 2015 at 9:10 PM, Olga Kornievskaia
> > > < [email-masked] > wrote:
> > > Hi folks,
> > > 
> > > OPEN
> > > that

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Jan 2015 07:49:23 -0500

> from the server. (In the FreeBSD client, a delegation return locks

That's unacceptably inflexible. 4.1 clients expect to be able to
parallelise opens and closes. Your belief would break that ability.

Cheers
  Trond

---

From: "Adamson, Andy" <[email-masked]>
Date: Fri, 30 Jan 2015 13:22:38 +0000

> On Jan 30, 2015, at 7:49 AM, Trond Myklebust <[email-masked]> wrote:
> 

Yes, I agree.

Considering that the described race/issue is a corner case, I think that the most appropriate  solution would be for the server to not return a delegation stateid on a CLAIM_NULL OPEN when the server already has a matching delegation on that file.  While the benefits of a delegation would be lost, it is a corner case, and for NFSv4.1, there is always WANT_DELEGATION.

—>Andy

---

From: David Noveck <[email-masked]>
Date: Fri, 30 Jan 2015 13:22:41 -0500

> Considering that the described race/issue is a corner case,

I agree that it is very unlikely to arise.

>I think that the most appropriate  solution would be for the server to not
return a delegation stateid on a CLAIM_NULL OPEN when the server already
> has a matching delegation on that file.

Agree.

Now let's look at what has to happen for this situation to arise:

   - a client has to have accumulated so many delegation that it is a
   resource issue for the client.
   - the client picks a particular delegation to return, presumably
   because, it feels it is unlikely to be needed in the future.  For example,
   it might not have been used in a long time.
   - Very soon after it makes that decision, it needs to open the file
   previously covered by that delegation.
   - That happens so quickly that there is a chance that the OPEN gets to
   the server before the DELEGRETURN, even though the OPEN occurred after the
   decision to return the delegation was made.  If the OPEN got to the server
   after the DELEGRETURN was received, this would be the case of a new
   delegation, and the server would be perfectly justified in returning a new
   stateid for it.

> While the benefits of a delegation would be lost,

I don't think any benefits are "lost" in this case.  There are no benefits
from the delegation, because the delegation was returned.  The client chose
to return it based on a guess about future use, and that guess was wrong.
As you point out, WANT_DELEGATION can be used to get an after-the-open
delegation in v4.1.

On Fri, Jan 30, 2015 at 8:22 AM, Adamson, Andy <[email-masked]>
wrote:

> Considering that the described race/issue is a corner case, I think that
> the most appropriate  solution would be for the server to not return a
> delegation stateid on a CLAIM_NULL OPEN when the server already has a
> matching delegation on that file.  While the benefits of a delegation would
> be lost, it is a corner case, and for NFSv4.1, there is always
> WANT_DELEGATION.
>
> —>Andy
>
> >
> > Cheers
> >  Trond
>

---

From: Rick Macklem <[email-masked]>
Date: Fri, 30 Jan 2015 21:25:27 -0500 (EST)

David Noveck wrote:
> >I think that the most appropriate solution would be for the server
> >to not return a delegation stateid on a CLAIM_NULL OPEN when the
> >server already
Well, it happens that the FreeBSD server does this. (ie. It does not
return a delegation stateid in an Open/Claim_null reply unless the
delegation is newly issued for the client/file for that Open.)

However, since this behaviour isn't clearly stated in the RFCs, servers
that do return a delegation for this case are not broken, imho.
The question is:
  Are there any extant NFSv4 servers that do return a delegation in an
  claim_null open reply when the client already has a delegation for the
  file?
If the answer to this is Yes, then it is too late to make this incorrect
server behaviour.

>     * a client has to have accumulated so many delegation that it is
>     a resource issue for the client.
>     * the client picks a particular delegation to return, presumably
>     because, it feels it is unlikely to be needed in the future. For
>     example, it might not have been used in a long time.
>     * Very soon after it makes that decision, it needs to open the
>     file previously covered by that delegation.
>     * That happens so quickly that there is a chance that the OPEN
>     gets to the server before the DELEGRETURN, even though the OPEN
>     occurred after the decision to return the delegation was made.
>     If the OPEN got to the server after the DELEGRETURN was
>     received, this would be the case of a new delegation, and the
>     server would be perfectly justified in returning a new stateid
>     for it.
> 
> 
> I don't think any benefits are "lost" in this case. There are no
> benefits from the delegation, because the delegation was returned.
> The client chose to return it based on a guess about future use, and
> that guess was wrong. As you point out, WANT_DELEGATION can be used
> to get an after-the-open delegation in v4.1.
> 
> 
> 
> 
> On Fri, Jan 30, 2015 at 8:22 AM, Adamson, Andy <
> [email-masked] > wrote:
> 
> 
> 
> 
> 
> > On Jan 30, 2015, at 7:49 AM, Trond Myklebust < [email-masked] >
> > wrote:
> > 
> > On Thu, Jan 29, 2015 at 6:00 PM, Rick Macklem <
> > [email-masked] > wrote:
> >> Andy Adamson wrote:
> >>> 
> >>>> On Jan 29, 2015, at 12:25 PM, David Noveck <
> >>>> [email-masked] >
> >>>> wrote:
> >>>> 
Why?

The only time Opens would be blocked (maybe I shouldn't have used the
term serialize) is the case where DELEGRETURN is in progress.
Parallel opens/closes could normally be done since, as others have noted,
doing the DELEGRETURN is a rare case.

I also noted that another possible fix is for the client to do a DELEGRETURN
of the delegation in the reply to the Claim_null Open for this case and
just ignore any error replied to the DELEGRETURN by the server.

rick

> Yes, I agree.
> 
> Considering that the described race/issue is a corner case, I think
> that the most appropriate solution would be for the server to not
> return a delegation stateid on a CLAIM_NULL OPEN when the server
> already has a matching delegation on that file. While the benefits
> of a delegation would be lost, it is a corner case, and for NFSv4.1,
> there is always WANT_DELEGATION.
> 
> —>Andy
> 
> 
> 
> > 
> > Cheers
> > Trond
>

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Jan 2015 23:26:28 -0500

> If the answer to this is Yes, then it is too late to make this incorrect

By that particular argument, it would also be too late to declare this
to be incorrect client behaviour, as there is no explicit requirement
in the RFC that the client must serialise OPEN and DELEGRETURN, and
there are plenty of clients out there that are coded to that
non-requirement too.

So you now have an impasse situation.

> term serialize) is the case where DELEGRETURN is in progress.

A rose by any other name...

> doing the DELEGRETURN is a rare case.

The client doesn't necessarily know that it is opening the same file,
so in order to entirely get rid of the race, you'd have to serialise
against _all_ non-filehandle based OPENs. Why add that burden when it
is much easier to not send the delegation in the first place?

> just ignore any error replied to the DELEGRETURN by the server.

Now you are adding a new requirement that the client "remember" the
delegation after it has been returned just so that it can return it a
second time. Again why do this just in order to avoid a server
behavioural change?

The point here is that it is completely senseless for the client to be
asking for the same delegation twice. Unlike the case for layouts,
there is no tolerance in the spec for a forgetful client. It is
required to send the DELEGRETURN when it is done using the delegation,
and only then does it make sense for it to start sending ordinary
OPENs again.
So what would a server be trying to achieve by sending the same
delegation a second time? It too knows that the client already holds
the delegation. How does it make any sense whatsoever to require a
change the behaviour all those clients just in order to preserve a
pointless server behaviour?

Trond

---

From: Rick Macklem <[email-masked]>
Date: Sat, 31 Jan 2015 12:08:53 -0500 (EST)

> there are plenty of clients out there that are coded to that
Well, I would say that a server reply of NFS4ERR_BAD_STATEID that
translates to EIO being returned by a system call as a broken client
implementation and I don't see how you could argue that a client
using a delegation stateid after it has DELEGRETURN'd it can be
correct client behaviour.

I chose to block Opens while doing DELEGRETURN for the FreeBSD client,
but I've already suggested one other possible way for the client to
deal with the problem. (That one might get a NFS4ERR_BAD_STATEID reply
to a second DELEGRETURN, but in this case I think it is harmless.)

The protocol would be broken if there was no way for the client to
avoid the NFS4ERR_BAD_STATEID, but I don't believe that is the case here.

rick

> >> > On Jan 30, 2015, at 7:49 AM, Trond Myklebust < [email-masked]
> >> > >
> >> > wrote:
> >> >
> >> >>>> I see your point. If you are sending these delegations back
> >> >>>> in
> >> >>>> the
> >> >>>> seqid, as you suggest, the DELEGRETURN would fail, in the
> >> >>>> case
> >> >>>> you
> >> >>> stateid that it had already given to the client, the
> >> >>> delegation
> >> >>> that
> >> >>> The client uses the just-granted delegation for I/O and
> >> >>> receives
> >> >>> an
> >> >>> NFS4ERR_BAD_STATEID which ends up as an EIO to the
> >> >>> application,
> >> >>> as
> >> >> any new OPEN for the same FH. Or, put another way, the client
> >> >> must
> >> >> delay
Yea, in FreeBSD it is a single lock which is normally acquired with
shared semantics and only acquired with exclusive semantics for
Delegreturn (plus server crash recovery).
(This was actually done because the code that does Opens
against the server via delegate_current for local opens when a delegation
is to be returned was rather intricate and I could only easily get it
right by single threading it. If the code only had to do 4.1 it would
have been much easier, since all Opens could be on the same open_owner4
since the client always does DENY_NONE. It was the ugly because of the
seqid for the open_owner4 used to serialize opens for 4.0. But this is
implementation trivia.;-)

However, even if you can only use one lock...
- Open acquires shared lock and releases it when done.
- Delegreturn acquires exclusive lock (on same lock) and releases
  it when done.

--> If Delegreturn is a rare case, why would this not allow parallel
    Opens normally?

> second time. Again why do this just in order to avoid a server
Not exactly. What I meant was return the "new" (which happens to be the same)
delegation that shows up in the Open reply received after the first Delegreturn 
for it has been sent to the server.
However, I haven't worked through this and I suspect the client might have
to remember that it doesn't want a delegation for a given FH for a while
and should just DELEGRETURN any delegation stateid in Open replies for a
given FH and not use them.
(I think the "while" is until the reply from the first DELEGRETURN is received,
 since after that point in time, any delegation in an Open reply really is a
 "new" one.)

rick

> asking for the same delegation twice.
I agree that it would have been nice if whether or not there should
be a delegation stateid in the reply to Open when the server has already
sent the delegation stateid to the client in a previous (w.r.t. the server's
ordering) Open reply or not was stated in RFC-3530.

But it wasn't. I vaguely recall asking what the server should do in this
case long long ago, but I don't recall the reply. As such, there can be
servers out there in production use doing either and I can't see how that
can be changed now (except maybe for a future minor version of the protocol).

I can only assume some server chooses to do this, or the race would have never
been identified in the first place?

rick

> Unlike the case for layouts,