Subject: [nfsv4] Need for clarification of CLAIM_DELEGATE_PREV

From: <[email-masked]>
Date: Thu, 26 Jan 2012 13:08:32 -0500

I've just been reading the spec about this, and what is actually in 
RFCs 3530 and 5661 about this is less clear than what I (thought I) 
remembered.  Bad sign :-(.

The issue is that that the server has the ability to keep delegations 
across client reboot, which is a good thing in the exciting world of
persistent caching using flash, but there is no way to indicate, when
you get the delegation, whether it merits this persistent treatment.

  "How do I feel about the Future?"

  "Hey, man.  The future's not here, yet"

                 -- The Firesign Theater.
                 -- I Think We're all Bozos on this Bus.

The problem is that it is looking like the future will be here petty soon.

One possibility is that if you support CLAIM_DELEGATE_PREV, it is expected 
you do so for all delegation but this would be overkill.  Still, since the
spec doesn't say anything else ...

The "right" way to fix this is to specify this via a new value for the
open_delegation_type4 enum, which might make sense for v4.x, for x >= 2,
But we need to do something about the cases in which x < 2.

So my suggestion is that we make it clear that if you support this, the 
normal expectation would be that it would be supported for OPEN_DELEGATE_WRITE
delegations.  The recent clarifications about the usage of write delegations
in the latest rev of RFC3530bis would fit well with this, in that those writing
but not doing persistent write-back caching could use OPEN_DELEGATE_READ to
indicate their lack of need for a persistent delegation.  If there is a
subsequent rev of RFC3530bis, it would be nice to include something addressing 
this, but it is not a "stop the presses"  issue.

Are there other suggestions about how to handle this?

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 26 Jan 2012 15:16:17 -0500

> you get the delegation, whether it merits this persistent treatment.

I'm a little confused about which way you're suggesting the "indication"
needs to go.  Do you mean, the server would indicate this to the client?

> indicate their lack of need for a persistent delegation.

OK, so the hint goes in the other direction.

Uh, maybe that makes more sense, I'm not sure.

> If there is a

I'm a little confused about how persistent delegations help the client
do persistent caching.

In the write case if you fetch the change attribute after each write
then you can still revalidate your cache after you reboot, can't you?
No, I suppose that's not good enough if the server reboots mid-write, or
if you have multiple outstanding writes.

Whereas if you have a read delegation on a mostly-static file, your
cache should be easy to revalidate on reboot.

So the ability to reclaim write delegations does seem a lot more useful
than the ability to reclaim read delegations.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 26 Jan 2012 21:07:29 -0500 (EST)

> you get the delegation, whether it merits this persistent treatment.
I don't quite understand this. If the client reboots, the server (which
I will presume has not crashed/rebooted) will find this out when the
client does a fresh SetClientID/confirm. At that point, I believe the
client is expected to hold onto "old delegations" for that client, so
they can be recovered by the client via CLAIM_DELEGATE_PREV. (I also
don't see any indication that this is an "optional feature", so a server
that doesn't implement this doesn't conform to RFC3530, imho.)

The place where I can see a "hint from the client" as being useful would
be if the client's lease expired, due to it taking too long to reboot.
(The hint could be used by the server to handle delegations for this
 client as "courtesy locks" and keep them for as long as practical, even
 after the client's lease has expired.)

Btw, I have an experimental patch for the FreeBSD client I call Packrats,
which does do "on-disk caching of delegated files" and seems to work
pretty well. It is still experimental mainly because I haven't yet
written the client crash recovery code for it. I am hoping to have that
written and ready for testing for the June Ann Arbor bakeathon (assuming
that it happens).

> spec doesn't say anything else ...
Except for the issue of how the server handles the delegations after a
client's lease has expired, I don't see why it needs to do anything
different with them than any other delegation? (If you are talking
about this specific case then, yes, I can see a "hint" from the client
as useful.)

> delegations.
> The recent clarifications about the usage of write
> delegations
Well, I would agree that, if a server implementation needs to choose
which delegations to keep after the client's lease has expired, the
write ones would be the better ones to keep. At least for packrats,
updates to the file on the server may be delayed a long time, so
losing the delegation after a client reboot will hurt file consistency.
(I'm actually not sure how I will handle this for Packrats. The choice
 is to write the changes back to the server with the potential of
 overwriting changes done by another client vs throwing the changes
 away. Maybe these case will require manual intervention on the client,
 but I haven't thought about that yet.)

> Are there other suggestions about how to handle this?
For NFSv4.0, I can't think of any "trick" the client could use to give
the server a "hint" that it was planning on-disk caching for a delegation
that might be issued for an Open. As such, all I can suggest is basically
what you mentioned, along the lines of:
- Once a client's lease has expired, the server should keep the delegations
  for that client as long as practicable, with a bias towards write
  delegations over read delegation.

rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 27 Jan 2012 09:06:43 -0500

> that doesn't implement this doesn't conform to RFC3530, imho.)

	"A server MAY support a claim type of CLAIM_DELEGATE_PREV"

> be if the client's lease expired, due to it taking too long to reboot.

I guess I was assuming that would be the normal case.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 27 Jan 2012 10:51:06 -0500 (EST)

> "A server MAY support a claim type of CLAIM_DELEGATE_PREV"
Oops, sorry. My mistake. You are correct. The above clearly indicates that
it is optional. (I should have taken a look at the RFC before I replied. I
had never noticed that.)

> I guess I was assuming that would be the normal case.
Yep.

Actually, since CLAIM_DELEGATE_PREV is optional, it would be nice if the client knew that
so that it could decide whether or not to cache the file/delegation in
stable storage or not.

Although this sounds a little scary...
Maybe a client could try a CLAIM_DELEGATE_PREV for some file
in the mount point, after it did the SetClientConfirm. If it gets
a NFS4ERR_NOTSUPP reply, it would assume that CLAIM_DELEGATE_PREV
is not supported by the server. If it gets some other error or
succeeds, getting an open with a delegation, then it could assume
it is supported. (For the succeed case, it could just return the
delegation and close the file. If it does the above with a "file"
it knows it didn't hold a delegation for, this shouldn't happen,
but the client might not always know if it held a delegation for
the file before the crash/reboot.)

Sorry about the noise w.r.t. it not being optional, rick

---

From: Casey Bodley <[email-masked]>
Date: Fri, 27 Jan 2012 11:05:33 -0500

> the file before the crash/reboot.)

You might use the fact that DELEGPURGE is required for servers that
support CLAIM_DELEGATE_PREV. By sending a DELEGPURGE after client
confirm, you can both probe for support and tell the server to clean
up any of those delegations that the client doesn't remember.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 27 Jan 2012 11:38:15 -0500

> so that it could decide whether or not to cache the file/delegation in

Couldn't a persistent data cache turn out to be useful even in the
absence of any delegation?

> the file before the crash/reboot.)

Also, the protocol trusts clients to only reclaim delegations it held
before, so if it succeeds when it shouldn't have you could end up
denying some other client some state that it shouldn't be able to
reclaim.

So you'd have to somehow cook up a CLAIM_DELEGATE_PREV that would be
guaranteed to fail, and then hope that the server chooses to return the
NOTSUPP error in preference to any other.

Doesn't sound like something you'd want to rely on.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 27 Jan 2012 11:39:22 -0500

> up any of those delegations that the client doesn't remember.

Casey wins.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 27 Jan 2012 11:40:18 -0500 (EST)

Casey Bodley wrote:
> >> > > I've just been reading the spec about this, and what is
> >> > > actually
> >> > > in
> >> > > RFCs 3530 and 5661 about this is less clear than what I
> >> > > (thought
> >> > > I)
> >> > > across client reboot, which is a good thing in the exciting
> >> > > world
> >> > > of
> >> > > persistent caching using flash, but there is no way to
> >> > > indicate,
> >> > > when
> >> > client is expected to hold onto "old delegations" for that
> >> > client,
> >> > so
> >> > The place where I can see a "hint from the client" as being
> >> > useful
> >> > would
I like this idea. Good suggestion.

The only problem I see with it is that there could be a server that
supports DELEGPURGE (probably just returns NFS_OK without doing anything)
but not CLAIM_DELEGATE_PREV.

However, a reply of NFS4ERR_NOTSUPP tells the client that CLAIM_DELEGATE_PREV
isn't supported and a successful reply "hints" that it is. And this is
much simpler and safer than what I suggested.

rick

---

From: Casey Bodley <[email-masked]>
Date: Fri, 27 Jan 2012 11:48:25 -0500

> but not CLAIM_DELEGATE_PREV.

You should be able to rely on this: "The server that supports
CLAIM_DELEGATE_PREV MUST support the DELEGPURGE operation." (RFC3530
section 9.2.1)

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 27 Jan 2012 14:51:35 -0500

> CLAIM_DELEGATE_PREV MUST support the DELEGPURGE operation." (RFC3530

Yeah, Rick was just pointing out there's no guarantee of the converse.

--b.

---

From: <[email-masked]>
Date: Fri, 27 Jan 2012 17:18:18 -0500

> Yeah, Rick was just pointing out there's no guarantee of the converse.

Yes, but I'm wondering whether there should be such a guarantee/assurance.

Does it make sense for a server to support DELEGPURGE if it doesn't support
CLAIM_DELEGATE_PREV?

My understanding of DELEGPURGE is that it is a way for the client to say
"I'm done recovering delegations (i.e. using CLAIM_DELEGATE_PREV) and so any
that have not been so recovered should be disposed of".  A server that did
not support CLAIM_DELEGATE_PREV would be expected to dispose of all delegations 
as soon as STCLIENTID_CONFIRM is done, since the set of delegation that will 
not be recovered is identical to the whole set of delegations, since there is 
no way to recover any.

So a client that would like to recover delegations in this case will find out 
via getting NOTSUPP when using CLAIM_DELEGATE_PREV.  At that point, the
client should know that DELEGPURGE is not required, and that all such delegations
are gone.

It is  possible a client has no delegations to recover and thus no occasion to
find out that there is no persistent delegation support.  I can imagine three
sorts of servers: 

1) the server responds NFS_OK as if to say "I have gotten rid of all the 
   remaining delegations (none) that you asked me to dispose of.  Where's my 
   tip?"

2) the server responds NOTSUPP as if to say, "I don't hold with this newfangled
   persistent delegations stuff and so there are no delegations to dispose of.  Give
   me a break".

3) the Mitt Romney server which sometimes acts like 1) and other times like 2).

The spec needn't explicitly disallow 3), but it clearly doesn't make sense.

1) is more polite but I think 2) gives people information that they need without
causing any harm so perhaps the spec can embrace the converse implication.

In that case a server would support CLAIM_DELEGATE_PREV iff it supported
DELEGPURGE. 
   
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of J. Bruce Fields
Sent: Friday, January 27, 2012 2:52 PM
To: Casey Bodley
Cc: Rick Macklem; [email-masked]
Subject: Re: [nfsv4] Need for clarification of CLAIM_DELEGATE_PREV

--b.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 27 Jan 2012 19:03:53 -0500 (EST)

> In that case a server would support CLAIM_DELEGATE_PREV iff it
I think this would be nice. Maybe if no extant server does 1), it could
be changed for 3530bis?

Btw, theoretically the FreeBSD server supports both DELEGPURGE and
CLAIM_DELEGATE_PREV. I say "theoretically" since I don't believe it
has ever been tested:-)

So, what do other servers do for DELEGPURGE?

rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 27 Jan 2012 19:16:11 -0500

> So, what do other servers do for DELEGPURGE?

It's not supported, on Linux.  (And CLAIM_DELEGATE_PREV opens also get
NOTSUPP, of course.)

--b.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 27 Jan 2012 19:31:20 -0500 (EST)

> absence of any delegation?
Maybe, but I haven't looked at it? 

Packrats was intended for a fairly specific case, which
was better performance over WANs. For the WAN case, I didn't want the
client waiting for a lot of RPC replies, since the RPC RTT is much larger.
Without a delegation, coherency checks (Getattr of the change attribute)
need to be done frequently, along with Opens and write backs upon Close,
and I wanted to avoid those "synchronous" RPCs. (By "synchronous" I mean
RPCs where the client has to wait for the RPC reply before the related system
call can complete.)

A typical scenario I envision would be:
- Working on a document (a few files of well under a Gbyte) on a your laptop
  for a couple of hours when "on the road". Since no one else would be accessing
  these files, there's a pretty good chance the client will get a write delegation
  (at least for 4.1, where there doesn't have to be a separate working callback path).
  Packrat threads would then copy the entire file to the local disk cache. After
  that copy completes, the client would only have to keep the lease valid, where
  RRT doesn't matter. At the end of the session, the changes would be written back
  during the "umount", before the delegations gets returned.
  A record of the delegation in the local disk cache (not yet implemented, so it is
  just an empty file at this time:-), could be used to "hopefully" re-acquire the
  delegation via CLAIM_DELEGATE_PREV if the laptop crashes or gets network
  partitioned for long enough that the lease expires and the delegation is lost.
  (Admittedly, if servers retain the delegation as in a "courtesy lock", then the
   CLAIM_DELEGATE_PREV isn't needed for the "network partition" case.)

Packrats also waits a long time to write changes back. Unless the delegation
is returned (recalled by the server or LRU replaced when there are more than
20000 delegations), it currently only does a write back when the file system
is unmounted. (As such, "umount" can take many seconds with the current
implementation. I did say it was "experimental".:-) Although I feel write back
needs to be done more agressively than the above, I don't want to be forced to
do so upon Close and only after long enough that it is unlikely that it is a
temporary file that will soon be removed. (Removal before write back avoids the
write back, since the file data is simply discarded.)

Probably way too long winded, but just in case you were curious, rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 30 Jan 2012 08:48:20 -0500

> write back, since the file data is simply discarded.)

You can skip flushing on close, but I don't think it's safe to do so on
sync.

Data that's been sync'd is supposed to survive a crash, and counting on
a write delegation for that sounds iffy.  Even with a server that's
generous with "courtesy" locks and supports CLAIM_DELEGATE_PREV, there's
a sufficiently long network partition or reboot could still easily lose
you your delegation.

And I think applications tend to sync a lot, don't they?

--b.

---

From: Rick Macklem <[email-masked]>
Date: Mon, 30 Jan 2012 10:32:50 -0500 (EST)

> And I think applications tend to sync a lot, don't they?
Well, in general I think I agree. For Packrats, a copy of the entire file
exists in a directory on the client's local disk. As such, loss of the
delegation and/or a client crash/reboot doesn't mean loss of the file
changes.

Unfortunately, without the delegation there is no guarantee that another
client hasn't modified the file. So, the tradeoff is "re-open the file
and write the changes back, running the risk of overwriting another client's
changes" or "losing a significant part of the performance gain by doing the
writeback upon sync". Since files being concurrently written by multiple
clients is a fairly rare occurrence, my current thinking is that doing
the "re-open..." is good enough to optionally allow. (Packrats will only
be enabled optionally and write-back upon sync could also be optional,
although too many options just confuse users, so...)

Good point, though. I think I'll implement a sysctl to enable/disable
writeback upon sync and do a bit of performance comparison to see how
much effect it has.

rick

---

From: <[email-masked]>
Date: Mon, 30 Jan 2012 11:17:47 -0500

> You can skip flushing on close, but I don't think it's safe to do so on

It depends what "flush" means in this context (or do I mean "these 
contexts"?).

Let's say:
   
   pflush -- written to some reasonably reliable (but not necessarily 
             totally reliable) persistent storage.
   sflush -- written to a server where it is at least as safe as with 
             pflush but where the changes are visible to anyone who 
             wants to read the file.

I think these contexts are "interesting", precisely because 
pflush != sflush.

> Data that's been sync'd is supposed to survive a crash,
 
but not a head-crash or for SSD's an omigod-even-with-all-that-wear-
leveling-it-can-still-wear-out event.

> and counting on a write delegation for that sounds iffy.  

Somewhere between iffy and insane.

> Even with a server that's

In the courtesy locks case, a conflicting open can cause loss of the lock/
delegation.  So if you hold modified data but do not actually write
without a lock/delegation, you have two sorts of situations:

If the other open is for read, he will either see the old (previous sflush)
or the new version but not something halfway between.  You don't
get the normal close-to-open semantic guarantee that close normally gets
you, that people will se the new version but presumably by waking this
approach, you've waived that.

The case in which you lose the delegation in the middle of writing changes
is even more "interesting".

The case in which you lose your lock because of conflicting open for write
will be caught if you check the change attribute when you re-establish 
(rather than reclaim) a delegation.  The only thing to do is get the user 
involved trying to merge changes to the file. 

> And I think applications tend to sync a lot, don't they?

Which means that if sync implies sflush and the server is many 
light-milliseconds away, even with the best possible network 
connection.  One light-ms equals 200 miles, but with
RPC's you have to figure one ms of latency for every 100 miles.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of J. Bruce Fields
Sent: Monday, January 30, 2012 8:48 AM
To: Rick Macklem
Cc: [email-masked]
Subject: Re: [nfsv4] Need for clarification of CLAIM_DELEGATE_PREV

--b.

---

From: <[email-masked]>
Date: Mon, 30 Jan 2012 11:29:44 -0500

Thanks for the discussion everybody.

So here is my proposal for clarification/corrections that I'd like the 
group to consider:
    
    Explicitly state DELEGPURGE supported iff CLAIM_DELEGATE_PREV is
    supported.

    CLAIM_DELEGATE_PREV is primarily to deal with the case in which the 
    client (only) restarts.

    When the server indicates support of CLAIM_DELEGATE_PREV (implicitly)
    by returning NFS_OK to DELEGPURGE, a client with a write delegation,
    use write-back caching for data to be written to the server, deferring 
    the write-back, until such time as the delegation is recalled, possibly
    after intervening client restarts.  Similarly, when the server indicates
    support of CLAIM_DELEGATE_PREV, a client with a read delegation and
    an open-for-write subordinate to that delegation, may be sure of the 
    integrity of its persistently cached copy of the file after a client 
    restart without specific verification of the change attribute.

    Case in which you have a series of client and server failures where
    there is no restart of both at the same time, are dealt with via a
    combination of CLAIM_DELEGATE_PREV and CLAIM_DELEGATE_CUR reclaims.  
    Persistent storage is needed only on the client.  For each server 
    failure, a CLAIM_DELEGATE_CUR reclaim is done, while for each client 
    restart, CLAIM_DELEGATE_PREV reclaim is done.

    To deal with the possibility of simultaneous failure of client and 
    server (power outages, etc.), the server MAY persistently store 
    delegation information so that it can respond to a CLAIM_DELEGATE_PREV 
    reclaim request which it receives from a restarting client.  A server 
    is not required to store this information, but if it does do so, it s
    should do so for write delegations and for read delegations, during the 
    pendency of which (across multiple client and/or server instances), some 
    open-for-write was done as part of delegation. When the space to 
    persistently record such information is limited, server should recall 
    delegations in this class in preference to keeping them active without 
    persistent storage recording.

    TBD: Review the "edge condition" text in light of persistent delegations.

Comments/corrections appreciated.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Mon, 30 Jan 2012 19:35:36 +0000

Hi Dave,

>     CLAIM_DELEGATE_PREV is primarily to deal with the case in which the 

What about the case in which the client/server experience a network
partition and/or the various edge condition cases in section 8.6.3 (rfc
3530)? Isn't the client going to want to attempt a CLAIM_DELEGATE_PREV
in order to recover those delegations that the server may have cached on
its behalf?

>     restart without specific verification of the change attribute.

How does the client distinguish between support of CLAIM_DELEGATE_PREV
and other error conditions? Does the error return NFS4ERR_RECLAIM_BAD
when it supports CLAIM_DELEGATE_PREV, but cannot satisfy the reclaim vs.
NFS4ERR_INVAL if it simply does not satisfy CLAIM_DELEGATE_PREV?

>     restart, CLAIM_DELEGATE_PREV reclaim is done.

Since CLAIM_DELEGATE_CUR is for dealing with delegation recalls (i.e.
converting a delegation into a stateful OPEN), did you actually mean
CLAIM_PREVIOUS above?

>     TBD: Review the "edge condition" text in light of persistent delegations.

I'm also unclear on how CLAIM_DELEGATE_PREV avoids the need for open
confirmation. The server states that the CLAIM_DELEGATE_PREV attempt
will only occur after a client reboot, and so open owners are not
recycled, however after a network partition, the client is not going to
declare a reboot if it held locks prior to the partition (in case the
server is holding "courtesy locks"). In such a situation, if the client
attempts to CLAIM_DELEGATE_PREV, the server may find itself dealing with
reused open owners.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Rick Macklem <[email-masked]>
Date: Mon, 30 Jan 2012 19:55:09 -0500 (EST)

> in order to recover those delegations that the server may have cached
A client implementation might very well find doing this useful. I don't
see why a client implementation cannot choose to "pretend to crash/reboot"
whenever it likes. A client might very well choose to do the equivalent of
a "crash/reboot" recovery when a server replies NFS4ERR_EXPIRED. From
the server's perspective (or what goes on the wire, if you prefer), it
would be indistinguishable from a client crash/reboot. (I'm assuming that
"pretend to crash/reboot" includes a Setclientid, Setclientidconfirm being
done just before the Opens with CLAIM_DELEGATE_PREV. See below.)

If you are going to allow clients to do CLAIM_DELEGATE_PREV whenever they
feel like and not just right after a Setclientid, Setclientidconfirm, that
probably is a new case that hasn't been thought through before? (Again, see
below.)

> NFS4ERR_INVAL if it simply does not satisfy CLAIM_DELEGATE_PREV?
Wouldn't a server reply NFS4ERR_NOTSUPP if it isn't supported? (Both
Delegourge and Open have this error in their error lists.)

Or, did I not understand what you were asking?

> attempts to CLAIM_DELEGATE_PREV, the server may find itself dealing
Hmm, interesting question. I had assumed that CLAIM_DELEGATE_PREV could
only be used immediately after a Setclientid, Setclientidconfirm.
(Exchange_id, Create_session for 4.1, maybe? I haven't read RFC5661
 in detail yet.;-)
The last para of RFC3530 sec. 14.2.34:

   SETCLIENTID_CONFIRM does not establish or renew a lease.  However, if
   SETCLIENTID_CONFIRM removes relevant leased client state, and that
   state does not include existing delegations, the server MUST allow
   the client a period of time no less than the value of lease_time
   attribute, to reclaim, (via the CLAIM_DELEGATE_PREV claim type of the
   OPEN operation) its delegations before removing unreclaimed
   delegations.

seems to say that the client should do the CLAIM_DELEGATE_PREV right after
Setclientidconfirm. (Within a lease duration.) However, it doesn't explicitly
say that a server can't allow them at other times.

> > > > Actually, since CLAIM_DELEGATE_PREV is optional, it would be
> > > > nice if
> > > > the client knew that
> > > > so that it could decide whether or not to cache the
> > > > file/delegation
> > > > in
> > > > stable storage or not.
> > >
> > > guaranteed to fail, and then hope that the server chooses to
> > > return
> > > the

---

From: <[email-masked]>
Date: Tue, 31 Jan 2012 10:09:52 -0500

To help avoid nested-quotation-ad-infinitum syndrome, let me address all your 
issues here.  Later I'll present an updated proposal with changes keyed to which
comments they address

=== T1 ===

> partition 

I don't believe CLAIM_DELEGATE_PREV is meant to address that, and that if you
try to make it do so, you will wind up dealing with the fact that it is bad fit
because the CLAIM_DELEGATE_PREV model is that the client restarts (i.e. changes
the nfs4-client-id instance).

I think 'courtesy locks" is/are meant to address this.  

> and/or the various edge condition cases in section 8.6.3 (RFC

The edge conditions are those in which the server cannot guarantee that the locks
he would give you are valid.  If he can't give you that guarantee as to locks,
Why should you trust him regarding delegations?

=== T2 ===

> NFS4ERR_INVAL if it simply does not satisfy CLAIM_DELEGATE_PREV?

I would think NOTSUPP (the operation of reclaiming a persistent delegation being
the operation not supported.

LOCK_NOTSUPP makes sense but we would have to expand the definition.

INVAL does say "unsupported argument" but the context/example is something the 
protocol does not support (i. e. READLINK of something not a symbolic link) 
and not something a particular server does not support.  

=== T3 === 
> did you actually mean CLAIM_PREVIOUS above?

Duh!  Yes.

>     delegation   

==== T4 ====

> confirmation. 

It doesn't.  Should it?

> The server states that the CLAIM_DELEGATE_PREV attempt
> server is holding "courtesy locks"). 

I agree.

> In such a situation, 

By "such a situation", I take it you mean "after a network partition".
Correct?

> if the client
> attempts to CLAIM_DELEGATE_PREV, 

Which I think he shouldn't do.

> the server may find itself dealing with
> reused open owners.

Which wouldn't be good.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Monday, January 30, 2012 2:36 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Need for clarification of CLAIM_DELEGATE_PREV

Hi Dave,

NetApp

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 31 Jan 2012 15:31:42 +0000

> probably is a new case that hasn't been thought through before? (Again, see

That's not really what I was saying. I was saying that a recovery
situation where the client has lost its lease doesn't necessarily mean
that we're in a client (or server) reboot situation.

Basically, if the client has outstanding writes, and either gets a
NO_GRACE error when it attempts to apply reboot recovery, or knows that
it is in a network partition scenario, then why shouldn't it try a
CLAIM_DELEGATE_PREV as a last-ditch strategy to recover its delegation?

> Delegourge and Open have this error in their error lists.)

NFS4ERR_NOTSUPP means "Operation is not supported". I'd interpret that
to mean that the NFS server doesn't support the OPEN operation.

> say that a server can't allow them at other times.

True. Is CLAIM_DELEGATE_PREV allowed in the grace period? I think it
should be, since the server should know that there are no conflicts, but
if it isn't allowed for some reason, then the client should be able to
defer the CLAIM_DELEGATE_PREV attempt.

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Tue, 31 Jan 2012 11:21:19 -0500 (EST)

> On Mon, 2012-01-30 at 19:55 -0500, Rick Macklem wrote:
[good stuff snipped for brevity]
> > Delegpurge and Open have this error in their error lists.)
> 
A few comments here:
1 - A server that doesn't support Open is an "interesting" server.
2 - The following snippet from RFC3530 Sec. 14.2.16 (the Open description)

   If the server can not support these exclusive create semantics,
   possibly because of the requirement to commit the verifier to stable
   storage, it should fail the OPEN request with the error,
   NFS4ERR_NOTSUPP.

   Uses NFS4ERR_NOTSUPP for a particular case of Open, so it doesn't
   indicate Open is not supported "in general".

3 - I think the error returned by Delegpurge is more important than
   what Open returns, since that will be used by the client to
   determine iff CLAIM_DELEGATE_PREV is supported.

However, I agree that this should be clarified, if possible.

[more good stuff snipped for brevity]
> defer the CLAIM_DELEGATE_PREV attempt.
Yes, I don't think it is clear and needs to be clarified. If it is allowed
during Grace, then the error that indicates "not supported" is important,
since the client will want to test to see if it can be done and switch to
using CLAIM_PREVIOUS if not supported.

Btw, another piece of the CLAIM_DELEGATE_PREV puzzle that isn't clear to
me is what the server returns for OPEN4resok.stateid (the open stateid)?
Since CLAIM_DELEGATE_PREV is reclaiming a delegation, it doesn't necessarily
imply acquiring an Open, but since OPEN4resok.stateid is always in the
reply, what should that be?
I can only see 2 answers:
1 - It always replies with an open as well as a delegation. This is
    consistent with the other claim types, but not really what the
    claim type is said to do.
OR
2 - It replies with a special stateid (all 0s or all 1s) to indicate
    that there is no open in the reply, just the delegation.

At the moment I suspect you must assume #1, but maybe #2 makes more sense?

Have fun with it, rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 31 Jan 2012 17:43:44 +0000

I think part of my confusion stems from the issue of whether or not
CLAIM_DELEGATE_PREV delegations are allowed to be persistent across
server reboots.

I agree that if the delegation is only granted until the server dies,
then there is no need for clarification on most of the issues that I
have raised. Basically, the CLAIM_DELEGATE_PREV is there in order to
ensure that the courtesy locks of type "delegation" survive the client
sending a SETCLIENTID with a reboot notification.

If, however, the CLAIM_DELEGATE_PREV have persistence properties that
exceed those of courtesy locks, then I'm still confused...

NetApp

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 31 Jan 2012 18:26:55 +0000

I keep on seeing references to RFC 3530. Could we instead be quoting
3530bis? That at least gets more eyes on that document and might catch any
mistakes there before we enter last call. Plus, if any changes arise from
this discussion, it will be in that document.

---

From: <[email-masked]>
Date: Tue, 31 Jan 2012 15:47:07 -0500

Good point.  For example, looking at the new network partition text
was a big help to me in trying to deal with the issues Trond raised.

---

From: <[email-masked]>
Date: Tue, 31 Jan 2012 15:50:11 -0500

If, however, the CLAIM_DELEGATE_PREV have persistence properties that
exceed those of courtesy locks, then I'm still confused.

For my answer to this question, see lines marked "T5>" in the new proposal.

-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Tuesday, January 31, 2012 12:44 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [nfsv4] Need for clarification of CLAIM_DELEGATE_PREV

NetApp

---

From: <[email-masked]>
Date: Tue, 31 Jan 2012 16:35:55 -0500

So here is my updated proposal for clarification/corrections that 
I'd like the group to consider.  The T* notes correspond to Trond's
Comments.

..>     Explicitly state DELEGPURGE supported iff CLAIM_DELEGATE_PREV is
T2>     supported.  Either both will return NFS4ERR_NOTSUPP or neither will.
T2>     Note that NFS4ERR_NOTSUPP is already used to indicate that exclusive
T2>     OPEN is not supported even though OPEN as a whole is.
..> 
T1>     CLAIM_DELEGATE_PREV may only be used after a SETCLIENTID_CONFIRM and before
T1>     DELEGPURGE.  When it is used outside of this context, the server,
T1>     if it supports CLAIM_DELEGATE_PREV, will return the error 
T1>     NFS4ERR_INVAL
..>     CLAIM_DELEGATE_PREV is primarily to deal with the case in which the 
..>     client (only) restarts.
T1>     Another case in which the client may use CLAIM_DELEGATE_PREV is
T1>     after receiving NFS4ERR_EXPIRED indicating a network partition
T1>     which resulted in loss of lease state.  In such a situation, a
T1>     server MAY honor a CLAIM_DELEGATE_PREV reclaim, even though it 
T1>     is not supporting "courtesy locks".
..>
T5>     One good way to think about persistent delegations is that for the most
T5>     part, they function like "courtesy locks", with a special semantic
T5>     adjustments to allow them to be retained across a client restart, which
T5>     cause all other sorts of locks to be freed.  Such locks are generally 
T5>     not retained across a server restart.  The one exception is the
T5>     case of simultaneous failure of the client and server and is discussed
T5>     below.
..>
..>     When the server indicates support of CLAIM_DELEGATE_PREV (implicitly)
..>     by returning NFS_OK to DELEGPURGE, a client with a write delegation,
..>     use write-back caching for data to be written to the server, deferring 
..>     the write-back, until such time as the delegation is recalled, possibly
..>     after intervening client restarts.  Similarly, when the server indicates
..>     support of CLAIM_DELEGATE_PREV, a client with a read delegation and
..>     an open-for-write subordinate to that delegation, may be sure of the 
..>     integrity of its persistently cached copy of the file after a client 
..>     restart without specific verification of the change attribute.
..>
T2>     When a server supports CLAIM_DELEGATE_PREV and there is no
T2>     persistent delegation for the specified file, known to the server,
T2>     NFS4ERR_BAD_RECLAIM is returned.  
..>
..>     Case in which you have a series of client and server failures where
..>     there is no restart of both at the same time, are dealt with via a
T3>     combination of CLAIM_DELEGATE_PREV and CLAIM_PREVIOUS reclaims.  
..>     Persistent storage is needed only on the client.  For each server 
T3>     failure, a CLAIM_PREVIOUS reclaim is done, while for each client 
..>     restart, CLAIM_DELEGATE_PREV reclaim is done.
..>
..>     To deal with the possibility of simultaneous failure of client and 
..>     server (power outages, etc.), the server MAY persistently store 
..>     delegation information so that it can respond to a CLAIM_DELEGATE_PREV 
T5>     reclaim request which it receives from a restarting client.   This
T5>     is the one case in which persistent delegation state can be retained
T5>     across a server restart.  A server  is not required to store this 
..>     information, but if it does do so, it should do so for write 
..>     delegations and for read delegations, during the pendency of which
..>     (across multiple client and/or server instances), some open-for-write 
..>     was done as part of delegation. When the space to persistently
..>     record such information is limited, the server should recall 
..>     delegations in this class in preference to keeping them active without 
..>     persistent storage recording.
..>
T4>     In all the situations in which CLAIM_DELEGATE_PREV is validly done,
T4>     there should have been some situation which caused locking state 
T4>     relevant to the client (with the exception of that relating to persistent
T4>     delegations) to be reset:  Either a client restart will be seen by the
T4>     server or the server will have reset all lease state and returned 
T4>     NFS4ERR_EXPIRED.  This would include lokowner state.
T4>
T4>     Note that when there is a network partition and some lock state is 
T4>     preserved (in the form of "courtesy locks"), locking state is not reset
T4>     and use of CLAIM_DELEGATE_PREV is not needed since any still-valid 
T4>     delegations will still be available as "courtesy locks".
..> 
..>     TBD: Review the "edge condition" text in light of persistent delegations.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 31 Jan 2012 23:19:03 +0000

The text looks good to me, but I have a few more questions.

1) In NFSv4 minor version 0, should we perhaps recommend that all
clients issue DELEGPURGE after they have finished state recovery (i.e.
make doing so a SHOULD). It seems that doing so can considerably reduce
the resource burden on the server.

2) Is CLAIM_DELEGATE_PREV allowed during the grace period? Both Rick and
I seem to assume that it must be if the server is to support reclaims
after a power failure, etc.
        If so, why could we not just let RECLAIM_COMPLETE act as an
        implicit DELEGPURGE and just deprecate the latter altogether?
        Are there any existing client implementations out there that
        would suffer if we did this?

Cheers
  Trond

NetApp

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 31 Jan 2012 23:28:27 +0000

Thanks for driving this, Dave.  When we seem to have consensus,
I would like to have a "clean" version of the text prep'd for
WG approval so that we can push it into the errata mechanism.

Spencer

> Sent: Tuesday, January 31, 2012 3:19 PM
> To: [email-masked]
> 1) In NFSv4 minor version 0, should we perhaps recommend that all clients
> issue DELEGPURGE after they have finished state recovery (i.e.
> 2) Is CLAIM_DELEGATE_PREV allowed during the grace period? Both Rick and I
> seem to assume that it must be if the server is to support reclaims after
> a power failure, etc.
> > So here is my updated proposal for clarification/corrections that I'd
> > like the group to consider.  The T* notes correspond to Trond's
> > Comments.
> >
> > T3>     failure, a CLAIM_PREVIOUS reclaim is done, while for each
> > T3> client
> > T5>     across a server restart.  A server  is not required to store
> > T5> this

---

From: Rick Macklem <[email-masked]>
Date: Tue, 31 Jan 2012 19:13:36 -0500 (EST)

This text reads well to me, except for a couple of trivial typos
that I've flagged below, rick.

----- Original Message -----
> ..> use write-back caching for data to be written to the server,
There seems to be a word or two missing here. Maybe "may use write-back caching..."?

> ..> the write-back, until such time as the delegation is recalled,
Do you want the comma after write-back?

> T2> NFS4ERR_BAD_RECLAIM is returned.
It's actually NFS4ERR_RECLAIM_BAD, at least in RFC3530.

> ..> delegations and for read delegations, during the pendency of which
Is "pendency" a word? (If so, it's just my poor english, but I don't
follow what "during the pendency..." is saying?

> T4> NFS4ERR_EXPIRED. This would include lokowner state.
                                         lockowner

---

From: Rick Macklem <[email-masked]>
Date: Tue, 31 Jan 2012 19:23:48 -0500 (EST)

> implicit DELEGPURGE and just deprecate the latter altogether?

Don't we want to keep DELEGPURGE so that a client can use it to
get the "hint" as to whether or not CLAIM_DELEGATE_PREV is supported?

Also, it might be less confusing if DELEGPURGE is retained with the
same semantics for all NFSv.n? It is only one extra Op, which could
be put in the same compound as RECLAIM_COMPLETE after recovery, I think?

> would suffer if we did this?
Not at this point for the FreeBSD one. I haven't gotten to the point
of doing CLAIM_DELEGATE_PREV yet, rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 1 Feb 2012 00:43:13 +0000

> get the "hint" as to whether or not CLAIM_DELEGATE_PREV is supported?

If it needs that information, can't it just try to
OPEN(CLAIM_DELEGATE_PREV) on the file that it wants to cache?

According to Dave's proposed text, the server will return
NFS4ERR_NOT_SUPP in the case where there is no support at all, or
something along the lines of NFS4ERR_RECLAIM_BAD if there is support but
the delegation is being reclaimed at the wrong moment.

> be put in the same compound as RECLAIM_COMPLETE after recovery, I think?

If the intention is that CLAIM_DELEGATE_PREV can only be performed as
part of the state recovery process, then I think it is clearer if we
just have a single operation that designates "end of recovery". Once we
start allowing "end of recovery, except....", then we're on a slippery
slope to unnecessary complexity and confusion.

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Tue, 31 Jan 2012 20:28:01 -0500 (EST)

Trond Myklebust wrote:
> > > 2) Is CLAIM_DELEGATE_PREV allowed during the grace period? Both
> > > Rick
> > > and
It could, but an OPEN is a lot more complicated to do than DELEGPURGE.
I'd just like to do this right at mount time to set a flag on the
mount point, before I have a valid name in a directory FH or a valid
FH for a regular file to do the OPEN on.
(Also, if the OPEN succeeds, the client needs to close the
OPEN stateid and, maybe, DELEGRETURN the delegation. It shouldn't
succeed, but if a server does allow it...)

It does, however, bring up the question of whether DELEGPURGE can be
done before or after RECLAIM_COMPLETE, if it is retained for NFSv4.1?

I can't find anything in RFC5661 that answers this. The description for
RECLAIM_COMPLETE refers to lock recovery and lock ops. (see snippet below)
Since a delegation isn't a "lock", I'd say the current words allow DELEGPURGE
either before or after RECLAIM_COMPLETE. (An Open is a form of lock,
but a delegation isn't, so you could interpret this either way, I think?)
I think the same applies to OPEN(CLAIM_DELEGATE_PREV) w.r.t. can it be
done before/after RECLAIM_COMPLETE?

When I read Dave's current words, it seems to allow a server to retain
old delegations for a long time, if it wishes to do so. As such, I can
see it allowing an OPEN(CLAIM_DELEGATE_PREV) after RECLAIM_COMPLETE and
after Grace is over. If that is allowed, then DELEGPURGE needs to be
retained and be allowed after RECLAIM_COMPLETE.

> slope to unnecessary complexity and confusion.
Well, here's a snippet from the RECLAIM_COMPLETE description:

   Once a RECLAIM_COMPLETE is done, there can be no further reclaim
   operations for locks whose scope is defined as having completed
   recovery.  Once the client sends RECLAIM_COMPLETE, the server will
   not allow the client to do subsequent reclaims of locking state for
   that scope and, if these are attempted, will return NFS4ERR_NO_GRACE.

   Whenever a client establishes a new client ID and before it does the
   first non-reclaim operation that obtains a lock, it MUST send a
   RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
   locks to reclaim.  If non-reclaim locking operations are done before
   the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned.

It always seems to refer to recovery of locks. However, I can see an
argument for restricting DELEGPURGE and OPEN(CLAIM_DELEGATE_PREV)
to before a RECLAIM_COMPLETE, to keep it simpler.

rick

---

From: <[email-masked]>
Date: Tue, 31 Jan 2012 21:04:58 -0500

> the resource burden on the server.

Here are the cases we've talked about:

     client reboot : has some CLAIM_DELEGATE_PREV and then DELEGPURGE,
                     but no state recovery.
     Server reboot : has state recovery including CLAIM_PREVIOUS reclaim of
                     delegation, and no DELEGPURGE.  Should there be a DELEGPURGE 
                     and why?
     Both reboot: has some CLAIM_DELEGATE_PREV which you can construe as state
                  recovery and a DELEGPURGE.
     Network partition with EXPIRED: may have some CLAIM_DELEGATE_PREV and
                                     A DELEGPURGE.  You may re-establish locks
                                     but not by doing reclaims but rather redoing
                                     opens/locks if it lets you. 

I think DELEGPURGE should stay associated with CLAIM_DELEGATE_PREV reclaims only.

I think that for the cases in which you can do CLAIM_DELEGATE_PREV, one SHOULD 
do a DELEGPURGE. 

> after a power failure, etc.

I've been assuming that as well but I agree it should be mentioned.
I think that the text should warn people, that during the grace
Period, no locks should be given out that conflict with persistent delegations
That have not been released by DELEGPURGE.

>       implicit DELEGPURGE and just deprecate the latter altogether?

1) DELEGPURGE now tells people that CLAIM_DELEGATE_PREV is supported so that
You can act based on the fact that the facility is available.

Also these are two separte sorts of recovery and the fact that you can do both at 
The same time does not mean that you have to.

>       would suffer if we did this?.

No, but Tom and I would suffer because of all the editing that this would 
cause.

-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Tuesday, January 31, 2012 6:19 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [nfsv4] Need for clarification of CLAIM_DELEGATE_PREV

Cheers
  Trond

NetApp