Subject: RE: [nfsv4] pNFS - bad devices

From: "Halevy, Benny" <[email-masked]>
Date: Wed, 2 Nov 2005 06:13:35 -0500

In fact, there's no consistency related reason for the client to flush
any data in this scenario, and since layouts are orthogonal to cache
consistency a better approach for the client might be to return the
old layout and get a new one (can be in the same compound) while keeping
the dirty data in its cache and sync it later with the new layout
or via the MDS at the client's convenience.

This can be achieved by adding a flag to CB_LAYOUTRECALL indicating
that the layout is just invalidated - i.e. there's a good chance for the
client to get a new layout immediately after returning the current one.

Benny

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of
> Marc Eshel
> Sent: Tuesday, November 01, 2005 9:22 PM
> To: Garth Goodson
> Cc: [email-masked]
> Subject: [nfsv4] pNFS - bad devices
> 
> 
> When one of the data server crashes the metadata server uses 
> CB_LAYOUTRECALL to recall all layouts that were included the now dead 
> node. The client will than go and try to commit its data to the dead 
> node(device) before returning the layout which will cause at 
> the minimum 
> big delays. Maybe we can add more information to the layout recall to 
> include either the list of bad devices (don't even try to do 
> I/O to those 
> devices) or just a way to advice the client to flush/commit 
> the data using 
> the MDS  to complete the recall.
> Marc. 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Tue, 22 Nov 2005 17:55:37 -0800

Ok, this is my suggestion.  I want to keep these changes simple and 
limited to only the most necessary.  I added a 'layoutchanged' bool to 
the recall structure.  Any comments?

-Garth

10.1  CB_LAYOUTRECALL

    SYNOPSIS

      layout_type, iomode, layoutchanged, layoutrecall -> -

    ARGUMENT

    ...

      struct CB_LAYOUTRECALLargs {
              pnfs_layouttype4        layout_type;
              pnfs_layoutiomode4      iomode;
              bool                    layoutchanged;
              layoutrecall4           layoutrecall;
      };

    RESULT

      struct CB_LAYOUTRECALLres {
              nfsstat4        status;
      };

    DESCRIPTION

    The CB_LAYOUTRECALL operation is used to begin the process of
    recalling a layout, a portion thereof, or all layouts pertaining to a
    particular file system (FSID).  If RECALL_FILE is specified, the
    offset and length fields specify the portion of the layout to be
    returned.  The iomode specifies the set of layouts to be returned.
    An iomode of ANY specifies that all matching layouts, regardless of
    iomode, must be returned; otherwise, only layouts that exactly match
    the iomode must be returned.

    If the layoutchanged field is TRUE, then the client SHOULD not flush
    its dirty data to the devices specified by the layout being recalled.
    Instead, it is preferable for the client to flush the dirty data
    through the metadata server.  Alternatively, the client may attempt
    to obtain a new layout.  Note: in order to obtain a new layout the
    client must first return the old layout.  Since obtaining a new
    layout is not guaranteed to succeed, the client must be ready to
    flush its dirty data through the metadata server.

...

---

From: Marc Eshel <[email-masked]>
Date: Wed, 23 Nov 2005 08:27:42 -0800

This will work too. The client will just have to find out if a new layout 
is available before deciding what to do next. 
What about the following text? it should be dropped or changed.
section 10.1: 
"The client should complete any in-flight I/O operations using the
   recalled layout(s) before returning it/them via LAYOUTRETURN.  If the
   client has buffered dirty data, it may chose to write it directly to
   storage before calling LAYOUTRETURN, or to write it later using
   normal NFSv4 WRITE operations to the metadata server."

---

From: Garth Goodson <[email-masked]>
Date: Mon, 28 Nov 2005 11:30:49 -0800

I'll change that wording appropriately.  But, I would like the preferred 
option to be that the client write out dirty data before returning the 
layout (using the layout or through the MDS).

-Garth

Marc Eshel wrote:
>>    recalling a layout, a portion thereof, or all layouts pertaining to 
> 
> a
> 
>>    its dirty data to the devices specified by the layout being 
> 
> recalled.
> 
>>>old layout and get a new one (can be in the same compound) while 
> 
> keeping
> 
>>>that the layout is just invalidated - i.e. there's a good chance for 
> 
> the
> 
>>>client to get a new layout immediately after returning the current 
> 
> one.
> 
>>>Benny
>>>
>>>
>>>

---

From: Marc Eshel <[email-masked]>
Date: Mon, 28 Nov 2005 12:59:44 -0800

> layout (using the layout or through the MDS).

This is right if layoutchanged is false. If layoutchanged is true the 
client should first try to get a new layout and if the get new layout call 
fails than write the dirty data through the MDS.
Marc.