Subject: Re: [nfsv4] [Fwd: Question about pNFS OSD layout draft]

From: Benny Halevy <[email-masked]>
Date: Mon, 03 Mar 2008 17:59:17 +0200

Rick, thanks for your comment!
Please see my detailed answer below.

Benny

On Mar. 01, 2008, 15:24 +0200, Jim Zelenka <[email-masked]> wrote:
> 
> ------------------------------------------------------------------------
> 
> Subject:
> Question about pNFS OSD layout draft
> From:
> Rick McNeal <[email-masked]>
> Date:
> Fri, 29 Feb 2008 13:07:01 -0700
> To:
> [email-masked], [email-masked]
> 
> To:
> [email-masked], [email-masked]
> 
> 
> Hello,
> 
> I've been looking at the OSD layout draft and have a question  
> regarding error conditions. Nothing is mentioned in the draft around  
> how recovery is done for RAID4, RAID5, or RAIDP+Q. Are you handling  
> this condition through the control protocol with copy on write which  
> is rolled back in the layout commit isn't received from the client? If  
> so, why isn't this mentioned in the draft?

All I/O errors, regardless of RAID level, are handled and recovered by the
metadata server.  In case of RAID{4,5,P+Q} the client reports the error
via the pnfs_osd_ioerr4 structure at LAYOUTRETURN time (there's a minor
phrasing issue in draft -05 with that regards which I'll fix in the next draft)
and the server analyzes the error and repairs the affected stripes.
The client can then retry the write operation by either fetching a new layout
or writing to the MDS.

Since this is not apparent from the draft, I propose the following changes:

diff -Nurp draft-ietf-nfsv4-pnfs-obj-05.xml draft-ietf-nfsv4-pnfs-obj-06.xml
--- draft-ietf-nfsv4-pnfs-obj-05.xml	2008-03-03 16:17:37.000000000 +0200
+++ draft-ietf-nfsv4-pnfs-obj-06.xml	2008-03-03 17:56:14.000000000 +0200
@@ -206,7 +206,7 @@ This scheme is described in more detail 
    fail to pass the integrity check verification.
  </t>
  <t>
-   To recover from this condition the client should report the error via LAYOUTCOMMIT,
+   To recover from this condition the client should report the error and
    return the layout using LAYOUTRETURN, and invalidate all the device address mappings
    associated with this layout.
    The client can then ask for a new layout if it wishes using LAYOUTGET and
@@ -911,14 +911,15 @@ the lou_body opaque value is defined by 
   <figure>
    <artwork>
 struct pnfs_osd_layoutupdate4 {
-    pnfs_osd_deltaspaceused4    delta_space_used;
+    pnfs_osd_deltaspaceused4    lou_delta_space_used;
+    bool                        lou_ioerr;
 };
    </artwork>
   </figure>
 
   <t>
 Object-Based pNFS clients are not allowed to modify the layout.
-"delta_space_used" is used to convey capacity usage
+"lou_delta_space_used" is used to convey capacity usage
 information back to the metadata server.
   </t>
 
@@ -927,7 +928,7 @@ information back to the metadata server.
     <artwork>
 union pnfs_osd_deltaspaceused4 switch (bool valid) {
     case TRUE:
-        int64_t     delta;  /* Bytes consumed by write activity */
+        int64_t     dsu_delta;  /* Bytes consumed by write activity */
     case FALSE:
         void;
 };
@@ -946,12 +947,63 @@ so it can accurately track quota.  The p
 trust this information coming from the clients and therefore
 avoid querying the OSDs at the time of LAYOUTCOMMIT.
 If the client is unable to obtain this information from the OSD,
-it simply returns invalid delta_space_used.
+it simply returns invalid lou_delta_space_used.
    </t>
   </section>
+
+  <t>
+The "lou_ioerr" flag is used when I/O errors were encountered while writing
+the file.  The client MUST report the errors using the pnfs_osd_ioerr4
+structure (See <xref target="pnfs_osd_errno4" />) at LAYOUTRETURN time.
+  <t>
+If the client updated the file successfully before hitting the
+I/O errors it MAY use LAYOUTCOMMIT to update the metadata server as
+described above.  Typically, in the error-free case, the server MAY
+turn around and update the file's attributes on the storage devices.
+However, if I/O errors were encountered the server better not attempt
+to write the new attributes on the storage devices until it receives
+the I/O error report, therefore the client MUST set the lou_ioerr flag
+to true.
+Note that in this case, the client SHOULD send both the
+LAYOUTCOMMIT and LAYOUTRETURN operations in the same COMPOUND RPC.
+  </t>
  </section>
 </section>
 
+<section title="Recovering from Client I/O Errors">
+ <t>
+The pNFS client may encounter errors when directly accessing
+the object storage devices.
+However, it is the responsibility of the metadata server to
+handle the I/O errors.
+When the LAYOUT4_OSD2_OBJECTS layout type is used, the client
+MUST report the I/O errors to the server at LAYOUTRETURN time
+using the pnfs_osd_ioerr4 structure (See <xref targte="pnfs_osd_errno4" />).
+ </t>
+ <t>
+The metadata server analyzes the error and determines the required
+recovery operations such as repairing any parity inconsistencies,
+recovering media failures, or reconstructing missing objects.
+ </t>
+ <t>
+The metadata server SHOULD recall any outstanding layouts to allow it
+exclusive write access to the stripes being recovered and to prevent other
+clients from hitting the same error condition.
+In these cases, the server MUST complete recovery before handing out
+any new layouts to the affected byte ranges.
+ </t>
+ <t>
+Although is it MAY be acceptable for the client to propagate a
+corresponding error to the application that initiated the I/O operation
+and drop any unwritten data, the client SHOULD attempt to retry the original
+I/O operation by requesting a new layout using LAYOUTGET and retry the
+I/O operation(s) using the new layout or the client MAY just retry the
+I/O operation(s) using regular NFS READ or WRITE operations via the metadata
+server.  The client SHOULD attempt to retrieve a new layout and retry the I/O operation using OSD commands first and only if the error persists, retry
+the I/O operation via the metadata server.
+ </t>
+</section>
+
 <section title="Object-Based Layout Return">
  <t>
 layoutreturn_file4 is used in the LAYOUTRETURN operation
@@ -1098,9 +1150,13 @@ and byte range within the component obje
 "iswrite" is set to "true" if the failed OSD operation was data modifying, and
 "errno" represents the type of error.
    </t>
+   <t>
+Component byte ranges in the optional pnfs_osd_ioerr4 structure are
+used for recovering the object and MUST be set by the client to cover all
+failed I/O operations to the component.
+   </t>
   </section>
- 
-</section>
+ </section>
 </section>
 
 <section title="Object-Based Creation Layout Hint">
@@ -1254,9 +1310,6 @@ LAYOUTCOMMIT is only used by object-base
 hints and/or to report I/O errors to the MDS.
 Therefore, the offset and length in LAYOUTCOMMIT4args are reserved for future
 use and should be set to 0.
-However, component byte ranges in the optional pnfs_osd_ioerr4 structure are
-used for recovering the object and MUST be set by the client to cover all
-failed I/O operations to the component.
   </t>
  </section>
 </section>
@@ -1702,7 +1755,8 @@ The XDR description, with the sentinel s
 ///};
 ///
 ///struct pnfs_osd_layoutupdate4 {
-///    pnfs_osd_deltaspaceused4    delta_space_used;
+///    pnfs_osd_deltaspaceused4    lou_delta_space_used;
+///    bool                        lou_ioerr;
 ///};
 ///
 ////*

> 
> Thanks,
> ----
> Rick McNeal
> [email-masked]
> 
> “The government solution to any problem is usually at least as bad as  
> the problem.” —Milton Friedman
> 

-- 
Benny Halevy
Software Architect
Tel/Fax: +972-3-647-8340
Mobile: +972-54-802-8340
US:      +1-412-203-3187
[email-masked]
 
Panasas, Inc.
The Leader in Parallel Storage
www.panasas.com
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Rick McNeal <[email-masked]>
Date: Mon, 3 Mar 2008 09:20:57 -0700

I should have been a little more verbose in my description since I  
still see a hole in the error processing. Take the following setup. 1  
MDS and 5 OSD devices in a RAID5 configuration. A client wishes to  
perform a write operation and the MDS returns a layout which covers a  
full stripe plus parity. The client starts the writes and crashes. At  
this point the server only knows that the client started a write, but  
doesn't know what state the stripe is in. That stripe is now corrupt.

I see two possible solutions.

     (1) Before a layout is giving to a client the server copies the  
information from
         the DS's involved (data and parity). If the MDS fails to  
receive a layout
         commit from the client it can then rewrite the data and parity.

     (2) The client is required to read the data and store the  
information someplace
         in the event of a crash. That location must be global.

The first solution wouldn't require a modification to the current  
draft since the server would handle data protection in the background.  
The problem would be that the server must read the data before each  
write operation, for all clients, creating a bottleneck which pNFS is  
trying to solve.

The second solution would require an extension to the OSD  
specification that tells the client where it should write a copy of  
the data first. The server would control where that information is  
stored and be able to recover from a client crash.

It's been awhile since I've read the T10 OSD specification. Is there a  
copy on write or snapshot attribute for the objects? If so, I guess  
the server could set that attribute before the layout is given and if  
the client fails to issue a layout commit, the server to perform a  
recovery operation that way.

On Mar 3, 2008, at 8:59 AM, Benny Halevy wrote:

> +server.  The client SHOULD attempt to retrieve a new layout and  
> retry the I/O operation using OSD commands first and only if the  
> error persists, retry

----
Rick McNeal
[email-masked]

---

From: Benny Halevy <[email-masked]>
Date: Tue, 04 Mar 2008 11:41:05 +0200

> doesn't know what state the stripe is in. That stripe is now corrupt.

The server can just scrub the parity and then the end state is no different
than a client crashing while writing several blocks in parallel to the MDS
using traditional WRITEs.  There's no reason to roll back any data.

Benny

---

From: Rick McNeal <[email-masked]>
Date: Tue, 4 Mar 2008 07:55:30 -0700

> using traditional WRITEs.  There's no reason to roll back any data.

There's still the problem that if one of the devices is also lost  
during that time you couldn't regenerate that device since the parity  
and remaining data devices wouldn't XOR the original data. If the  
chances are sufficiently remote of this occurring maybe this condition  
should just be treated like a second device failure for RAID5. Should  
this be noted in the specification?

----
Rick McNeal
[email-masked]

I say to myself "That's it God! I have got to get through here before  
the sun sets! No more pics unless I see something totally amazing!!!"

Word to the wise; don't challenge God when you are on a quest for  
spiritual enlightenment. 300 feet later, just around the bend, I enter  
"Monument Valley."

---

From: Benny Halevy <[email-masked]>
Date: Tue, 04 Mar 2008 17:42:39 +0200

> and remaining data devices wouldn't XOR the original data. If the  

True.

> should just be treated like a second device failure for RAID5. 

Yup.  FWIW, this is what Panasas are doing today.

> Should

I'll add a paragraph about that to the recovery section in draft #6.
This issue is one of the motivations for having the error report...

Benny