Subject: Re: [nfsv4] [PATCH 2/2] nfs41: handle BLK_LAYOUT CB_RECALL_ANY

From: <[email-masked]>
Date: Tue, 1 Nov 2011 10:59:49 -0400

As I recall, the primary focus of CB_RECALL ANY was in dealing with recall of delegations. 

With delegations the server is in no position to, if needs to get some number back, pick
ones that are not being used, since it is in the nature of delegations that the client may
be using them with great benefit with the server not hearing anything about them.  So the
logic there was that if the client didn't do the picking and just did nothing the server would 
have to pick for himself, and would almost certainly do a poor job, and the client might not
like the server's choices.

For other recallable objects, the logic my be different.  In the case of layouts, the 
data servers will know if they are being used but the MDFS will not.  So if the client won't
act on this, and the server picks badly, it can't say "there was no way I could have picked 
better ones" as it can with delegations.

The spec should make it clear whether the client is at fault in this case or the server 
should have done its best by incorporating data server information on layout use, but ....

   The chances of client and server implementors agreeing on this are fairly low.
   So ambiguity/confusion on this issue may persist for a while.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Welch, Brent
Sent: Monday, October 31, 2011 5:43 PM
To: Jim Rees; Trond Myklebust
Cc: [email-masked]; nfsv4 list
Subject: Re: [nfsv4] [PATCH 2/2] nfs41: handle BLK_LAYOUT CB_RECALL_ANY

CB_RECALL_ANY was added based on Panasas experience with its own protocols.   When you have
100's (or 1000's) of clients and millions of files for which you maintain state, it is
very much worth optimizing how the server can reclaim this state.
	Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jim Rees
Sent: Monday, October 31, 2011 11:32 AM
To: Trond Myklebust
Cc: [email-masked]; nfsv4 list
Subject: Re: [nfsv4] [PATCH 2/2] nfs41: handle BLK_LAYOUT CB_RECALL_ANY

Trond Myklebust wrote:

  I don't necessarily disagree with what you are saying, but I have yet to
  see a single server side implementation of CB_RECALL_ANY, let alone any
  numbers that indicate performance or responsiveness problems resulting
  from our existing client-side implementation.
  
  I therefore find it hard to understand why optimising this particular
  code is such a high priority, or why a patch that is adding per-file
  layoutreturns to initiate_bulk_draining() is going to help anything at
  all.

Testing between the linux block layout client and the EMC block layout
server revealed a deadlock when the server had handed out some number of
layouts and couldn't hand out any more.  So now the EMC block layout server
implements CB_RECALL_ANY.  So yes, this solves a real world problem, and
yes, there is a server that implements this.

We had some discussions at the time, and I don't remember if those were on
the linux-nfs list or in some other forum.  We decided that the client was
in the best position to decide which layouts were no longer needed, so we
needed some way for the server to tell the client to return some layouts
without specifying which ones.  CB_RECALL_ANY seemed custom-made for this
purpose, so we used it.

I don't think it would be appropriate for the server to recall all layouts
when it only needs some of them back.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Tue, 1 Nov 2011 01:29:27 -0400

Hi, Trond,

> From: Trond Myklebust [mailto:[email-masked]]
> Sent: Tuesday, November 01, 2011 2:40 AM
> To: Jim Rees
> Cc: Benny Halevy; Peng Tao; [email-masked]; Peng, Tao; nfsv4 list
> Subject: Re: [PATCH 2/2] nfs41: handle BLK_LAYOUT CB_RECALL_ANY
> 
> On Mon, 2011-10-31 at 14:31 -0400, Jim Rees wrote:
> > Trond Myklebust wrote:
> >
> As I said previously: the current client implementation deals with
> CB_RECALL_ANY by calling initiate_file_draining(), which forgets _all_
> layouts. If that is what we want to continue to use, then sending
> layoutreturn with LAYOUTRETURN4_ALL is appropriate.
> Otherwise, please fix the client to be more selective (in which case
> LAYOUTRETURN4_FILE may be more appropriate) and please remember to
> provide performance numbers to justify the need for optimisation.
OK, fair enough. I will work on the selective layoutreturn approach when I have the time slot. And I agree it is of low priority.

Thanks,
Tao