Subject: [nfsv4] Rework of error handling for LAYOUTRETURN

From: Thomas Haynes <[email-masked]>
Date: Mon, 11 Apr 2011 10:24:01 -0500

I've edited the Section on error handling for LAYOUTRETURN to match the
slides presented in Prague. Rather than publish a new draft, I thought
I'd post it here for comments.

2.  pNFS LAYOUTRETURN Error Handling

2.1.  Introduction

   In the pNFS description provided in [2], the client is not enabled to
   relay an error code from the DS to the MDS.  In the specification of
   the Objects-Based Layout protocol [3], use is made of the opaque
   lrf_body field of the LAYOUTRETURN argument to do such a relaying of
   error codes.  In this section, we define a new data structure to
   enable the passing of error codes back to the MDS and provide some
   guidelines on what both the client and MDS should expect in such
   circumstances.

Haynes                  Expires October 13, 2011                [Page 5]
^L
Internet-Draft                   NFSv4.2                      April 2011

   There are two broad classes of errors, transient and persistent.  The
   client SHOULD strive to only use this new mechanism to report
   persistent errors.  It MUST be able to deal with transient issues by
   itself.  Also, while the client might consider an issue to be
   persistent, it MUST be prepared for the MDS to consider such issues
   to be persistent.  A prime example of this is if the MDS fences off a
   client from either a stateid or a filehandle.  The client will get an
   error from the DS and might relay either NFS4ERR_ACCESS or
   NFS4ERR_STALE_STATEID back to the MDS, with the belief that this is a
   hard error.  The MDS on the other hand, is waiting for the client to
   report such an error.  For it, the mission is accomplished in that
   the client has returned a layout that the MDS had most likley
   recalled.

2.2.  Changes to Operation 51: LAYOUTRETURN

   The existing LAYOUTRETURN operation is extended by introducing a new
   data structure to report errors, layoutreturn_device_error4.  Also,
   layoutreturn_device_error4 is introduced to enable an array of errors
   to be reported.

2.2.1.  ARGUMENT

   The ARGUMENT specification of the LAYOUTRETURN operation in section
   18.44.1 of [2] is augmented by the following XDR code [11]:

   struct layoutreturn_device_error4 {
           deviceid4       lrde_deviceid;
           nfsstat4        lrde_status;
           nfs_opnum4      lrde_opnum;
   };

   struct layoutreturn_error_report4 {
           layoutreturn_device_error4      lrer_errors<>;
   };

2.2.2.  RESULT

   The RESULT of the LAYOUTRETURN operation is unchanged; see section
   18.44.2 of [2].

2.2.3.  DESCRIPTION

   The following text is added to the end of the LAYOUTRETURN operation
   DESCRIPTION in section 18.44.3 of [2].

   When a client used LAYOUTRETURN with a type of LAYOUTRETURN4_FILE,
   then if the lrf_body field is NULL, it indicates to the MDS that the

Haynes                  Expires October 13, 2011                [Page 6]
^L
Internet-Draft                   NFSv4.2                      April 2011

   client experienced no errors.  If lrf_body is non-NULL, then the
   field references error information which is layout type specific.
   I.e., the Objects-Based Layout protocol can continue to utilize
   lrf_body as specified in [3].  For both Files-Based Layouts, the
   field references a layoutreturn_device_error4, which contains an
   array of layoutreturn_device_error4.

   Each individual layoutreturn_device_error4 descibes a single error
   associated with a DS, which is identfied via lrde_deviceid.  The
   operation which returned the error is identified via lrde_opnum.
   Finally the NFS error value (nfsstat4) encountered is provided via
   lrde_status and may consist of the following error codes:

   NFS4_OKAY:  No issues were found for this device.

   NFS4ERR_NXIO:  The client was unable to establish any communication
      with the DS.

   NFS4ERR_*:  The client was able to establish communication with the
      DS and is returning one of the allowed error codes for the
      operation denoted by lrde_opnum.

2.2.4.  IMPLEMENTATION

   The following text is added to the end of the LAYOUTRETURN operation
   IMPLEMENTATION in section 18.4.4 of [2].

   A client that expects to use pNFS for a mounted filesystem SHOULD
   check for pNFS support at mount time.  This check SHOULD be performed
   by sending a GETDEVICELIST operation, followed by layout-type-
   specific checks for accessibility of each storage device returned by
   GETDEVICELIST.  If the NFS server does not support pNFS, the
   GETDEVICELIST operation will be rejected with an NFS4ERR_NOTSUPP
   error; in this situation it is up to the client to determine whether
   it is acceptable to proceed with NFS-only access.

   Clients are expected to tolerate transient storage device errors, and
   hence clients SHOULD NOT use the LAYOUTRETURN error handling for
   device access problems that may be transient.  The methods by which a
   client decides whether an access problem is transient vs. persistent
   are implementation-specific, but may include retrying I/Os to a data
   server under appropriate conditions.

   When an I/O fails to a storage device, the client SHOULD retry the
   failed I/O via the MDS.  In this situation, before retrying the I/O,
   the client SHOULD return the layout, or the affected portion thereof,
   and SHOULD indicate which storage device or devices was problematic.
   If the client does not do this, the MDS may issue a layout recall

Haynes                  Expires October 13, 2011                [Page 7]
^L
Internet-Draft                   NFSv4.2                      April 2011

   callback in order to perform the retried I/O.

   The client needs to be cognizant that since this error handling is
   optional in the MDS, the MDS may silently ignore this functionality.
   Also, as the MDS may consider some issues the client reports to be
   expected (see Section 2.1), the client might find it difficult to
   detect a MDS which has not implemented error handling via
   LAYOUTRETURN.

   If an MDS is aware that a storage device is proving problematic to a
   client, the MDS SHOULD NOT include that storage device in any pNFS
   layouts sent to that client.  If the MDS is aware that a storage
   device is affecting many clients, then the MDS SHOULD NOT include
   that storage device in any pNFS layouts sent out.  Clients must still
   be aware that the MDS might not have any choice in using the storage
   device, i.e., there might only be one possible layout for the system.

   Another interesting complication is that for existing files, the MDS
   might have no choice in which storage devices to hand out to clients.
   The MDS might try to restripe a file across a different storage
   device, but clients need to be aware that not all implementations
   have restriping support.

   An MDS SHOULD react to a client return of layouts with errors by not
   using the problematic storage devices in layouts for that client, but
   the MDS is not required to indefinitely retain per-client storage
   device error information.  An MDS is also not required to
   automatically reinstate use of a previously problematic storage
   device; administrative intervention may be required instead.

   A client MAY perform I/O via the MDS even when the client holds a
   layout that covers the I/O; servers MUST support this client
   behavior, and MAY recall layouts as needed to complete I/Os.