Subject: [nfsv4] pNFS - bad devices

From: Marc Eshel <[email-masked]>
Date: Tue, 1 Nov 2005 11:21:54 -0800

When one of the data server crashes the metadata server uses 
CB_LAYOUTRECALL to recall all layouts that were included the now dead 
node. The client will than go and try to commit its data to the dead 
node(device) before returning the layout which will cause at the minimum 
big delays. Maybe we can add more information to the layout recall to 
include either the list of bad devices (don't even try to do I/O to those 
devices) or just a way to advice the client to flush/commit the data using 
the MDS  to complete the recall.
Marc. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Halevy, Benny" <[email-masked]>
Date: Thu, 3 Nov 2005 16:08:04 -0500

Yeah, I agree that having two flags would cover more use cases.

sync_thru_mds  new_layout_available
false          false                 conflict type, client is expected to
sync dirty data
                                     using current layout and return it.

false          true                  I'm not sure what scenario would cause
this combination - maybe
                                     a sync point for snapshots.  (*)
                                     The client would optionally sync using
existing layout, return it,
                                     and possibly get a new one (which it
should expect to be exactly
                                     the same).
true           false                 This will move the client immediately
out of "pnfs mode" back
                                     into NFSv4.
true           true                  Layout changed, client is expected to
sync data through the MDS
                                     and may immediately get a new layout.

(*) To make the layout return semantics a bit stronger and more useful
for achieving a consistent sync point I suggest to change the word "should"
in section 10.1 to "MUST" (see below).

background:
a. in section 3.4 we write
   Once a layout has been returned, the client should no longer issue
   I/Os to the storage devices for the file, byte range, and iomode
   represented by the returned layout.  
b. in section 10.1 we write
   The client should complete any in-flight I/O operations using the
           ^^^ MUST ^^^
   recalled layout(s) before returning it/them via LAYOUTRETURN.  If the
   client has buffered dirty data, it may chose to write it directly to
   storage before calling LAYOUTRETURN, or to write it later using
   normal NFSv4 WRITE operations to the metadata server.

The ramifications of this addition is that the client may have to wait
until in-flight ops to a data server time out in cases like a network
partitioning before returning a layout.  This may cause the server
to time out the callback recall process and optionally fence the client.
Hopefully, by doing so the server will discover the failure and handle it.

Benny

> -----Original Message-----
> From: Marc Eshel [mailto:[email-masked]]
> Sent: Wednesday, November 02, 2005 8:02 PM
> To: Garth Goodson
> Cc: Halevy, Benny; Noveck, Dave; [email-masked]
> Subject: Re: [nfsv4] pNFS - bad devices
> 
> 
> Sure two flags will be even better.
> Marc.
> 
> Garth Goodson <[email-masked]> wrote on 11/02/2005 
> 09:28:58 AM:
> 
> > I think two types of recalls have been hit upon.  First, 
> there is the 
> > 'layout being recalled due to conflict, but the layout is 
> not changing - 
> 
> > flush dirty data using existing layout'.  Second, there is 
> the 'layout 
> > recalled due to bad device - flush data through mds'.
> > 
> > My question is: do we need one or two flags?  One for 
> 'flush through the 
> 
> > MDS', and one for 'attempt to reget layout after recall is 
> complete'. 
> > Or does one imply the other?  My preference is to be clear 
> rather than 
> > overly concise.
> > 
> > -Garth
> > 
> > Marc Eshel wrote:
> > > The draft suggest that the client might chose to flush dirty data 
> before 
> > > returning the layout, so I think that we do need the flag Benny 
> suggested. 
> > > Not only to get the new layout in one call and save an 
> RPC, but more 
> > > importantly tell the client NOT to flush using the old layout.
> > > Marc.
> > > 
> > > "Noveck, Dave" <[email-masked]> wrote on 
> 11/02/2005 05:37:11 
> AM:
> > > 
> > > 
> > >>Well if this flag didn't exist, what should a client do after
> > >>returning the layout?
> > >>
> > >>Try to get a new layout I would assume.  And if he got 
> one, he would
> > >>be exactly where he wants to be. 
> > >>
> > >>On the other hand if he didn't get it, he would have wasted one
> > >>RPC.  So if this flag existed, it would be a hint (if 
> zero) telling
> > >>the client that it was likely he would not be able to reget the 
> > >>layout.  I think that would be helpful but it is a minor 
> optimization.
> > >>
> > >>-----Original Message-----
> > >>From: Halevy, Benny [mailto:[email-masked]]
> > >>Sent: Wednesday, November 02, 2005 6:14 AM
> > >>To: 'Marc Eshel'; Goodson, Garth
> > >>Cc: [email-masked]
> > >>Subject: RE: [nfsv4] pNFS - bad devices
> > >>
> > >>
> > >>In fact, there's no consistency related reason for the 
> client to flush
> > >>any data in this scenario, and since layouts are 
> orthogonal to cache
> > >>consistency a better approach for the client might be to 
> return the
> > >>old layout and get a new one (can be in the same compound) while 
> keeping
> > >>the dirty data in its cache and sync it later with the new layout
> > >>or via the MDS at the client's convenience.
> > >>
> > >>This can be achieved by adding a flag to CB_LAYOUTRECALL 
> indicating
> > >>that the layout is just invalidated - i.e. there's a good 
> chance for 
> the
> > >>client to get a new layout immediately after returning 
> the current 
> one.
> > >>
> > >>Benny
> > >>
> > >>
> > >>>-----Original Message-----
> > >>>From: [email-masked] 
> > >>>[mailto:[email-masked]]On Behalf Of
> > >>>Marc Eshel
> > >>>Sent: Tuesday, November 01, 2005 9:22 PM
> > >>>To: Garth Goodson
> > >>>Cc: [email-masked]
> > >>>Subject: [nfsv4] pNFS - bad devices
> > >>>
> > >>>

---

From: "Halevy, Benny" <[email-masked]>
Date: Wed, 23 Nov 2005 07:40:15 -0500

Looks good.

Benny

> From: Garth Goodson [mailto:[email-masked]]
> Sent: Wednesday, November 23, 2005 3:56 AM
> To: Halevy, Benny
> Cc: 'Marc Eshel'; [email-masked]
> Ok, this is my suggestion.  I want to keep these changes simple and 
> limited to only the most necessary.  I added a 
> 'layoutchanged' bool to 
> the recall structure.  Any comments?
> 
> -Garth
> 
> 
> 10.1  CB_LAYOUTRECALL
> 
>     SYNOPSIS
> 
>       layout_type, iomode, layoutchanged, layoutrecall -> -
> 
>     ARGUMENT
> 
>     ...
> 
>       struct CB_LAYOUTRECALLargs {
>               pnfs_layouttype4        layout_type;
>               pnfs_layoutiomode4      iomode;
>               bool                    layoutchanged;
>               layoutrecall4           layoutrecall;
>       };
> 
>     RESULT
> 
>       struct CB_LAYOUTRECALLres {
>               nfsstat4        status;
>       };
> 
>     DESCRIPTION
> 
>     The CB_LAYOUTRECALL operation is used to begin the process of
>     recalling a layout, a portion thereof, or all layouts 
> pertaining to a
>     particular file system (FSID).  If RECALL_FILE is specified, the
>     offset and length fields specify the portion of the layout to be
>     returned.  The iomode specifies the set of layouts to be returned.
>     An iomode of ANY specifies that all matching layouts, 
> regardless of
>     iomode, must be returned; otherwise, only layouts that 
> exactly match
>     the iomode must be returned.
> 
>     If the layoutchanged field is TRUE, then the client 
> SHOULD not flush
>     its dirty data to the devices specified by the layout 
> being recalled.
>     Instead, it is preferable for the client to flush the dirty data
>     through the metadata server.  Alternatively, the client 
> may attempt
>     to obtain a new layout.  Note: in order to obtain a new layout the
>     client must first return the old layout.  Since obtaining a new
>     layout is not guaranteed to succeed, the client must be ready to
>     flush its dirty data through the metadata server.
> 
> ...
> 
> Halevy, Benny wrote:
> > any data in this scenario, and since layouts are orthogonal to cache
> > consistency a better approach for the client might be to return the
> > old layout and get a new one (can be in the same compound) 
> while keeping
> > This can be achieved by adding a flag to CB_LAYOUTRECALL indicating
> chance for the
> > client to get a new layout immediately after returning the 
> current one.
> > 
> > Benny
> > 
> >