Subject: [nfsv4] [VER 01] draft-ietf-nfsv4-rfc5664bis 00=>01 diff

From: Boaz Harrosh <[email-masked]>
Date: Fri, 29 Apr 2011 13:14:31 +0300

git diff --stat -p -M master -- draft-ietf-nfsv4-rfc5664bis.txt
 draft-ietf-nfsv4-rfc5664bis.txt |  586 +++++++++++++++++++++++----------------
 1 files changed, 349 insertions(+), 237 deletions(-)

diff --git a/draft-ietf-nfsv4-rfc5664bis.txt b/draft-ietf-nfsv4-rfc5664bis.txt
index 1a8c1f3..658acae 100644
--- a/draft-ietf-nfsv4-rfc5664bis.txt
+++ b/draft-ietf-nfsv4-rfc5664bis.txt
@@ -4,12 +4,12 @@
 NFSv4                                                          B. Halevy
 Internet-Draft                                                B. Harrosh
 Intended status: Standards Track                                B. Welch
-Expires: May 21, 2011                                            Panasas
-                                                       November 17, 2010
+Expires: October 31, 2011                                        Panasas
+                                                          April 29, 2011

               Object-Based Parallel NFS (pNFS) Operations
-                     draft-ietf-nfsv4-rfc5664bis-00
+                     draft-ietf-nfsv4-rfc5664bis-01
 
 Status of this Memo
 
@@ -32,11 +32,11 @@ Status of this Memo
    The list of Internet-Draft Shadow Directories can be accessed at
    http://www.ietf.org/shadow.html.
 
-   This Internet-Draft will expire on May 21, 2011.
+   This Internet-Draft will expire on October 31, 2011.
 
 Copyright Notice
 
-   Copyright (c) 2010 IETF Trust and the persons identified as the
+   Copyright (c) 2011 IETF Trust and the persons identified as the
    document authors.  All rights reserved.
 
    This document is subject to BCP 78 and the IETF Trust's Legal
@@ -52,9 +52,9 @@ Abstract

-Halevy, et al.            Expires May 21, 2011                  [Page 1]
+Halevy, et al.          Expires October 31, 2011                [Page 1]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    allow clients to directly access file data on the storage used by the
@@ -92,48 +92,48 @@ Table of Contents
      5.3.  Data Mapping Schemes . . . . . . . . . . . . . . . . . . . 15
        5.3.1.  Simple Striping  . . . . . . . . . . . . . . . . . . . 15
        5.3.2.  Nested Striping  . . . . . . . . . . . . . . . . . . . 16
-       5.3.3.  Mirroring  . . . . . . . . . . . . . . . . . . . . . . 17
-     5.4.  RAID Algorithms  . . . . . . . . . . . . . . . . . . . . . 18
-       5.4.1.  PNFS_OSD_RAID_0  . . . . . . . . . . . . . . . . . . . 18
-       5.4.2.  PNFS_OSD_RAID_4  . . . . . . . . . . . . . . . . . . . 18
-       5.4.3.  PNFS_OSD_RAID_5  . . . . . . . . . . . . . . . . . . . 19
-       5.4.4.  PNFS_OSD_RAID_PQ . . . . . . . . . . . . . . . . . . . 19
-       5.4.5.  RAID Usage and Implementation Notes  . . . . . . . . . 20
-   6.  Object-Based Layout Update . . . . . . . . . . . . . . . . . . 20
-     6.1.  pnfs_osd_deltaspaceused4 . . . . . . . . . . . . . . . . . 21
-     6.2.  pnfs_osd_layoutupdate4 . . . . . . . . . . . . . . . . . . 21
-   7.  Recovering from Client I/O Errors  . . . . . . . . . . . . . . 22
-   8.  Object-Based Layout Return . . . . . . . . . . . . . . . . . . 22
-     8.1.  pnfs_osd_errno4  . . . . . . . . . . . . . . . . . . . . . 23
-
-
-
-Halevy, et al.            Expires May 21, 2011                  [Page 2]
+       5.3.3.  Mirroring  . . . . . . . . . . . . . . . . . . . . . . 18
+     5.4.  RAID Algorithms  . . . . . . . . . . . . . . . . . . . . . 19
+       5.4.1.  PNFS_OSD_RAID_0  . . . . . . . . . . . . . . . . . . . 19
+       5.4.2.  PNFS_OSD_RAID_4  . . . . . . . . . . . . . . . . . . . 19
+       5.4.3.  PNFS_OSD_RAID_5  . . . . . . . . . . . . . . . . . . . 20
+       5.4.4.  PNFS_OSD_RAID_PQ . . . . . . . . . . . . . . . . . . . 21
+       5.4.5.  RAID Usage and Implementation Notes  . . . . . . . . . 22
+   6.  Object-Based Layout Update . . . . . . . . . . . . . . . . . . 22
+     6.1.  pnfs_osd_deltaspaceused4 . . . . . . . . . . . . . . . . . 23
+     6.2.  pnfs_osd_layoutupdate4 . . . . . . . . . . . . . . . . . . 23
+   7.  Recovering from Client I/O Errors  . . . . . . . . . . . . . . 24
+   8.  Object-Based Layout Return . . . . . . . . . . . . . . . . . . 24
+     8.1.  pnfs_osd_errno4  . . . . . . . . . . . . . . . . . . . . . 25
+
+
+
+Halevy, et al.          Expires October 31, 2011                [Page 2]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

-     8.2.  pnfs_osd_ioerr4  . . . . . . . . . . . . . . . . . . . . . 24
-     8.3.  pnfs_osd_layoutreturn4 . . . . . . . . . . . . . . . . . . 25
-   9.  Object-Based Creation Layout Hint  . . . . . . . . . . . . . . 25
-     9.1.  pnfs_osd_layouthint4 . . . . . . . . . . . . . . . . . . . 25
-   10. Layout Segments  . . . . . . . . . . . . . . . . . . . . . . . 27
-     10.1. CB_LAYOUTRECALL and LAYOUTRETURN . . . . . . . . . . . . . 27
-     10.2. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . . 28
-   11. Recalling Layouts  . . . . . . . . . . . . . . . . . . . . . . 28
-     11.1. CB_RECALL_ANY  . . . . . . . . . . . . . . . . . . . . . . 28
-   12. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . . 29
-   13. Security Considerations  . . . . . . . . . . . . . . . . . . . 29
-     13.1. OSD Security Data Types  . . . . . . . . . . . . . . . . . 30
-     13.2. The OSD Security Protocol  . . . . . . . . . . . . . . . . 31
-     13.3. Protocol Privacy Requirements  . . . . . . . . . . . . . . 32
-     13.4. Revoking Capabilities  . . . . . . . . . . . . . . . . . . 32
-   14. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 33
-   15. References . . . . . . . . . . . . . . . . . . . . . . . . . . 33
-     15.1. Normative References . . . . . . . . . . . . . . . . . . . 33
-     15.2. Informative References . . . . . . . . . . . . . . . . . . 34
-   Appendix A.  Acknowledgments . . . . . . . . . . . . . . . . . . . 35
-   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 35
+     8.2.  pnfs_osd_ioerr4  . . . . . . . . . . . . . . . . . . . . . 26
+     8.3.  pnfs_osd_layoutreturn4 . . . . . . . . . . . . . . . . . . 27
+   9.  Object-Based Creation Layout Hint  . . . . . . . . . . . . . . 27
+     9.1.  pnfs_osd_layouthint4 . . . . . . . . . . . . . . . . . . . 27
+   10. Layout Segments  . . . . . . . . . . . . . . . . . . . . . . . 29
+     10.1. CB_LAYOUTRECALL and LAYOUTRETURN . . . . . . . . . . . . . 29
+     10.2. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . . 30
+   11. Recalling Layouts  . . . . . . . . . . . . . . . . . . . . . . 30
+     11.1. CB_RECALL_ANY  . . . . . . . . . . . . . . . . . . . . . . 30
+   12. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . . 31
+   13. Security Considerations  . . . . . . . . . . . . . . . . . . . 31
+     13.1. OSD Security Data Types  . . . . . . . . . . . . . . . . . 32
+     13.2. The OSD Security Protocol  . . . . . . . . . . . . . . . . 33
+     13.3. Protocol Privacy Requirements  . . . . . . . . . . . . . . 34
+     13.4. Revoking Capabilities  . . . . . . . . . . . . . . . . . . 34
+   14. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 35
+   15. References . . . . . . . . . . . . . . . . . . . . . . . . . . 35
+     15.1. Normative References . . . . . . . . . . . . . . . . . . . 35
+     15.2. Informative References . . . . . . . . . . . . . . . . . . 36
+   Appendix A.  Acknowledgments . . . . . . . . . . . . . . . . . . . 37
+   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 37

@@ -164,9 +164,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                  [Page 3]
+Halevy, et al.          Expires October 31, 2011                [Page 3]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

 1.  Introduction
@@ -220,9 +220,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                  [Page 4]
+Halevy, et al.          Expires October 31, 2011                [Page 4]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    sh extract.sh < spec.txt > pnfs_osd_prot.x
@@ -276,9 +276,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                  [Page 5]
+Halevy, et al.          Expires October 31, 2011                [Page 5]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    ///  *   FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
@@ -293,7 +293,7 @@ Internet-Draft                pnfs objects                 November 2010
    ///  *   IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
    ///  *   ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    ///  *
-   ///  * This code was derived from draft-ietf-nfsv4-rfc5664bis-00.
+   ///  * This code was derived from draft-ietf-nfsv4-rfc5664bis-01.
    [[RFC Editor: please insert RFC number if needed]]
    ///  * Please reproduce this note if possible.
    ///  */
@@ -332,9 +332,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                  [Page 6]
+Halevy, et al.          Expires October 31, 2011                [Page 6]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    the GETDEVICELIST and GETDEVICEINFO operations.  The deviceid4 data
@@ -388,9 +388,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                  [Page 7]
+Halevy, et al.          Expires October 31, 2011                [Page 7]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

 3.3.  pnfs_osd_object_cred4
@@ -444,9 +444,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                  [Page 8]
+Halevy, et al.          Expires October 31, 2011                [Page 8]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

       the input_message to the GSS_unwrap() function.  Note that to
@@ -466,10 +466,10 @@ Internet-Draft                pnfs objects                 November 2010
 3.4.  pnfs_osd_raid_algorithm4
 
    /// enum pnfs_osd_raid_algorithm4 {
-   ///     PNFS_OSD_RAID_0     = 1,
-   ///     PNFS_OSD_RAID_4     = 2,
-   ///     PNFS_OSD_RAID_5     = 3,
-   ///     PNFS_OSD_RAID_PQ    = 4     /* Reed-Solomon P+Q */
+   ///     PNFS_OSD_RAID_0  = 1,
+   ///     PNFS_OSD_RAID_4  = 2,
+   ///     PNFS_OSD_RAID_5  = 3,
+   ///     PNFS_OSD_RAID_PQ = 4     /* Reed-Solomon P+Q */
    /// };
    ///
 
@@ -500,9 +500,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                  [Page 9]
+Halevy, et al.          Expires October 31, 2011                [Page 9]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    In some situations, SCSI target discovery may need to be driven based
@@ -556,9 +556,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 10]
+Halevy, et al.          Expires October 31, 2011               [Page 10]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

 4.2.  pnfs_osd_deviceaddr4
@@ -612,9 +612,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 11]
+Halevy, et al.          Expires October 31, 2011               [Page 11]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    Fibre Channel Framing and Signaling (FC-FS) [17].  These are
@@ -668,9 +668,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 12]
+Halevy, et al.          Expires October 31, 2011               [Page 12]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    enum layouttype4 {
@@ -724,9 +724,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 13]
+Halevy, et al.          Expires October 31, 2011               [Page 13]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    reached its final stripe width.  The file length in this case MUST be
@@ -780,9 +780,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 14]
+Halevy, et al.          Expires October 31, 2011               [Page 14]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    array within the full array of components that comprise the file.
@@ -811,54 +811,58 @@ Internet-Draft                pnfs objects                 November 2010
    component object C and object-specific offset O is defined by the
    following equations:
 
-   L = logical offset into the file
-   W = total number of components
-   S = W * stripe_unit
-   N = L / S
-   C = (L-(N*S)) / stripe_unit
-   O = (N*stripe_unit)+(L%stripe_unit)
+   L: logical offset into the file
 
-   In these equations, S is the number of bytes in a full stripe, and N
-   is the stripe number.  C is an index into the array of components, so
-   it selects a particular object storage device.  Both N and C count
-   from zero.  O is the offset within the object that corresponds to the
-   file offset.  Note that this computation does not accommodate the
-   same object appearing in the olo_components array multiple times.
-
-   For example, consider an object striped over four devices, <D0 D1 D2
-   D3>.  The stripe_unit is 4096 bytes.  The stripe width S is thus 4 *
-   4096 = 16384.
+   W: stripe width
+       W = size of olo_components array
 
+   S: number of bytes in a stripe
+       S = W * stripe_unit
 
+   N: stripe number
+       N = L / S
 
+   C: component index corresponding to L
+      C = (L % S) / stripe_unit
 
+   O: The component offset corresponding to L
+      O = (N * stripe_unit) + (L % stripe_unit)
 
+   Note that this computation does not accommodate the same object
+   appearing in the olo_components array multiple times.  Therefore the
+   server may not return layouts with the same object appearing multiple
+   times.  If needed the server can return multiple layout segments each
+   covering a single instance of the object.

-Halevy, et al.            Expires May 21, 2011                 [Page 15]
+Halevy, et al.          Expires October 31, 2011               [Page 15]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

+   For example, consider an object striped over four devices, <D0 D1 D2
+   D3>.  The stripe_unit is 4096 bytes.  The stripe width S is thus 4 *
+   4096 = 16384.
+
    Offset 0:
      N = 0 / 16384 = 0
-     C = 0-0/4096 = 0 (D0)
+     C = (0 % 16384) /4096 = 0 (D0)
      O = 0*4096 + (0%4096) = 0
 
    Offset 4096:
      N = 4096 / 16384 = 0
-     C = (4096-(0*16384)) / 4096 = 1 (D1)
+     C = (4096 % 16384) / 4096 = 1 (D1)
      O = (0*4096)+(4096%4096) = 0
 
    Offset 9000:
      N = 9000 / 16384 = 0
-     C = (9000-(0*16384)) / 4096 = 2 (D2)
+     C = (9000 % 16384) / 4096 = 2 (D2)
      O = (0*4096)+(9000%4096) = 808
 
    Offset 132000:
      N = 132000 / 16384 = 8
-     C = (132000-(8*16384)) / 4096 = 0 (D0)
+     C = (132000 % 16384) / 4096 = 0 (D0)
      O = (8*4096) + (132000%4096) = 33696
 
 5.3.2.  Nested Striping
@@ -872,68 +876,114 @@ Internet-Draft                pnfs objects                 November 2010
    computations map from the logical offset L to the component index C
    and offset relative O within that component object.
 
-   L = logical offset into the file
-   W = total number of components
-   S = stripe_unit * group_depth * W
-   T = stripe_unit * group_depth * group_width
-   U = stripe_unit * group_width
-   M = L / S
-   G = (L - (M * S)) / T
-   H = (L - (M * S)) % T
-   N = H / U
-   C = (H - (N * U)) / stripe_unit + G * group_width
-   O = L % stripe_unit + N * stripe_unit + M * group_depth * stripe_unit
 
-   In these equations, S is the number of bytes striped across all
-   component objects before the pattern repeats.  T is the number of
-   bytes striped within a group of component objects before advancing to
-   the next group.  U is the number of bytes in a stripe within a group.
-   M is the "major" (i.e., across all components) stripe number, and N

-Halevy, et al.            Expires May 21, 2011                 [Page 16]
+
+
+
+
+
+
+
+
+
+
+
+
+Halevy, et al.          Expires October 31, 2011               [Page 16]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011
+
+
+   L: logical offset into the file
+
+   FW: total number of components
+       FW = size of olo_components array
+
+   W: stripe width
+      W = group_width, if not zero, else FW
+
+   group_count: number of groups
+      group_count = FW / group_width, if group_width is not zero, else 1
+
+   D: number of data devices in a stripe
+      D = W
+
+   U: number of data bytes in a stripe within a group
+      U =  D * stripe_unit
+
+   T: number of bytes striped within a group of component objects
+      (before advancing to the next group)
+      T = U * group_depth
+
+   S: number of bytes striped across all component objects
+      (before the pattern repeats)
+      S = T * group_count
+
+   M: The "major" (i.e., across all components) cycle number
+      M = L / S
+
+   G: group number from the beginning of the major cycle
+      G = (L % S) / T
 
+   H: byte offset within the last group
+      H = (L % S) % T
 
-   is the "minor" (i.e., across the group) stripe number.  G counts the
-   groups from the beginning of the major stripe, and H is the byte
-   offset within the group.
+   N: The "minor" (i.e., across the group) stripe number
+      N = H / U
+
+   C: component index corresponding to L
+      C = (G * D) + ((H % U) / stripe_unit)
+
+   O: The component offset corresponding to L
+      O = (M * group_depth * stripe_unit) + (N * stripe_unit) +
+          (L % stripe_unit)
 
    For example, consider an object striped over 100 devices with a
    group_width of 10, a group_depth of 50, and a stripe_unit of 1 MB.
    In this scheme, 500 MB are written to the first 10 components, and
    5000 MB are written before the pattern wraps back around to the first
+
+
+
+Halevy, et al.          Expires October 31, 2011               [Page 17]
+
+Internet-Draft                pnfs objects                    April 2011
+
+
    component in the array.
 
    Offset 0:
      W = 100
-     S = 1 MB * 50 * 100 = 5000 MB
-     T = 1 MB * 50 * 10 = 500 MB
+     group_count = 100 / 10 = 10
+     D = 10
      U = 1 MB * 10 = 10 MB
+     T = 10 MB * 50 = 500 MB
+     S = 500 MB * 10 = 5000 MB
      M = 0 / 5000 MB = 0
-     G = (0 - (0 * 5000 MB)) / 500 MB = 0
-     H = (0 - (0 * 5000 MB)) % 500 MB = 0
+     G = (0 % 5000 MB) / 500 MB = 0
+     H = (0 % 5000 MB) % 500 MB = 0
      N = 0 / 10 MB = 0
-     C = (0 - (0 * 10 MB)) / 1 MB + 0 * 10 = 0
-     O = 0 % 1 MB + 0 * 1 MB + 0 * 50 * 1 MB = 0
+     C = (0 * 10) + ((0 % 10 MB) / 1 MB) = 0
+     O = (0 * 50 * 1 MB) + (0 * 1 MB) + (0 % 1 MB) = 0
 
    Offset 27 MB:
      M = 27 MB / 5000 MB = 0
-     G = (27 MB - (0 * 5000 MB)) / 500 MB = 0
-     H = (27 MB - (0 * 5000 MB)) % 500 MB = 27 MB
+     G = (27 MB % 5000 MB) / 500 MB = 0
+     H = (27 MB % 5000 MB) % 500 MB = 27 MB
      N = 27 MB / 10 MB = 2
-     C = (27 MB - (2 * 10 MB)) / 1 MB + 0 * 10 = 7
-     O = 27 MB % 1 MB + 2 * 1 MB + 0 * 50 * 1 MB = 2 MB
+     C = (0 * 10) + ((27 MB % 10 MB) / 1 MB) = 7
+     O = (0 * 50 * 1 MB) + (2 * 1 MB) + (27 MB % 1 MB) = 2 MB
 
    Offset 7232 MB:
      M = 7232 MB / 5000 MB = 1
-     G = (7232 MB - (1 * 5000 MB)) / 500 MB = 4
-     H = (7232 MB - (1 * 5000 MB)) % 500 MB = 232 MB
+     G = (7232 MB % 5000 MB) / 500 MB = 4
+     H = (7232 MB % 5000 MB) % 500 MB = 232 MB
      N = 232 MB / 10 MB = 23
-     C = (232 MB - (23 * 10 MB)) / 1 MB + 4 * 10 = 42
-     O = 7232 MB % 1 MB + 23 * 1 MB + 1 * 50 * 1 MB = 73 MB
+     C = (4 * 10) + ((232 MB % 10 MB) / 1 MB) = 42
+     O = (1 * 50 * 1 MB) + (23 * 1 MB) + (7232 MB % 1 MB) = 73 MB
 
 5.3.3.  Mirroring
 
@@ -945,21 +995,25 @@ Internet-Draft                pnfs objects                 November 2010
    Note that mirroring can be defined over any RAID algorithm and
    striping pattern (either simple or nested).  If odm_group_width is
    also non-zero, then the size of the olo_components array MUST be a
+   multiple of odm_group_width * (odm_mirror_cnt+1).  Note that
+   odm_group_width does not account for mirrors.  Replicas are adjacent
+   in the olo_components array, and the value C produced by the above
+   equations is not a direct index into the olo_components array.
+   Instead, the following equations determine the replica component
+   index RCi, where i ranges from 0 to odm_mirror_cnt.

-Halevy, et al.            Expires May 21, 2011                 [Page 17]
+Halevy, et al.          Expires October 31, 2011               [Page 18]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

-   multiple of odm_group_width * (odm_mirror_cnt+1).  Replicas are
-   adjacent in the olo_components array, and the value C produced by the
-   above equations is not a direct index into the olo_components array.
-   Instead, the following equations determine the replica component
-   index RCi, where i ranges from 0 to odm_mirror_cnt.
+   FW = size of olo_components array / (odm_mirror_cnt+1)
 
    C = component index for striping or two-level striping
+       as calculated using above equations
+
    i ranges from 0 to odm_mirror_cnt, inclusive
    RCi = C * (odm_mirror_cnt+1) + i
 
@@ -994,27 +1048,31 @@ Internet-Draft                pnfs objects                 November 2010
    Or the client can replay the READ against the pNFS server that will
    presumably perform the reconstructed read on the client's behalf.
 
-   When parity is present in the file, then there is an additional
-   computation to map from the file offset L to the offset that accounts
-   for embedded parity, L'.  First compute L', and then use L' in the
-   above equations for C and O.
+   When parity is present in the file, then the number of parity devices
+   is taken into account in the above equations when calculating (D),
+   the number of data devices in a stripe, as follows:
+
+
+

 
-Halevy, et al.            Expires May 21, 2011                 [Page 18]
+Halevy, et al.          Expires October 31, 2011               [Page 19]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

-   L = file offset, not accounting for parity
-   P = number of parity devices in each stripe
-   W = group_width, if not zero, else size of olo_components array
-   N = L / (W-P * stripe_unit)
-   L' = N * (W * stripe_unit) +
-        (L % (W-P * stripe_unit))
+   P: number of parity devices in each stripe
+      P = 1
+
+   D: number of data devices in a stripe
+      D = W - P
+
+   I: parity device index
+      I = D
 
 5.4.3.  PNFS_OSD_RAID_5
 
@@ -1024,51 +1082,100 @@ Internet-Draft                pnfs objects                 November 2010
    the second stripe, the next-to-last component holds the parity, and
    so on.  In this scheme, all stripe units are rotated so that I/O is
    evenly spread across objects as the file is read sequentially.  The
-   rotated parity layout is illustrated here, with numbers indicating
-   the stripe unit.
+   rotated parity layout is illustrated here, with hexadecimal numbers
+   indicating the stripe unit.
 
    0 1 2 P
    4 5 P 3
    8 P 6 7
    P 9 a b
 
-   To compute the component object C, first compute the offset that
-   accounts for parity L' and use that to compute C. Then rotate C to
-   get C'.  Finally, increase C' by one if the parity information comes
-   at or before C' within that stripe.  The following equations
-   illustrate this by computing I, which is the index of the component
-   that contains parity for a given stripe.
-
-   L = file offset, not accounting for parity
-   W = odm_group_width, if not zero, else size of olo_components array
-   N = L / (W-1 * stripe_unit)
-   (Compute L' as describe above)
-   (Compute C based on L' as described above)
-   C' = (C - (N%W)) % W
-   I = W - (N%W) - 1
-   if (C' <= I) {
-     C'++
-   }
+   Note that RAID_5 is exactly as RAID_4 only that individual stripes
+   get rotated backwards and devices change their position in the
+   stripe.  So proceed exactly as equation above for RAID_4, then
+   compute the rotation as below.  Also note that the raid cycle always
+   start at a group boundary.  So the first stripe in a group has its
+   parity at device D.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Halevy, et al.          Expires October 31, 2011               [Page 20]
+
+Internet-Draft                pnfs objects                    April 2011
+
+
+   P: number of parity device in each stripe
+      P = 1
+
+   PC: Parity Cycle
+       PC = W
+
+   R: The parity rotation index
+      (N is as computed in above equations)
+      R = N % PC
+
+   I: parity device index
+      I = (W + W - (R + 1) * P) % W
+
+   Cr: The rotated device index
+       (C is as computed in the above equations)
+       Cr = (W + C - (R * P)) % W
+
+   Note: W is added above to avoid negative numbers modulo math.
 
 5.4.4.  PNFS_OSD_RAID_PQ
 
    PNFS_OSD_RAID_PQ is a double-parity scheme that uses the Reed-Solomon
    P+Q encoding scheme [19].  In this layout, the last two component
    objects hold the P and Q data, respectively.  P is parity computed
-   with XOR, and Q is a more complex equation that is not described
-   here.  The equations given above for embedded parity can be used to
+   with XOR.  The Q computation is described in detail by Anvin [20].
+   The same polynomial "x^8+x^4+x^3+x^2+1" and Galois field size of 2^8
+   are used here.  Clients may simply choose to read data through the
+   metadata server if two or more components are missing or damaged.
 
+   The equations given above for embedded parity can be used to map a
+   file offset to the correct component object by setting the number of
+   parity components (P) to 2 instead of 1 for RAID5 and computing the
+   Parity Cycle length as the Lowest Common Multiple [21] of
+   odm_group_width and P, devided by P, as described below.  Note: This
+   algorithm can be used also for RAID_5 where P=1.
 
+   P: number of parity devices
+      P = 2
 
-Halevy, et al.            Expires May 21, 2011                 [Page 19]
-
-Internet-Draft                pnfs objects                 November 2010
+   PC: Parity cycle:
+       PC = LCM(W, P) / P
 
+   Q: The device index holding the Q component
+      (I is as computed in the above equations for RAID_5)
+      Qdev = (I + 1) % W
+
+
+
+
+
+
+Halevy, et al.          Expires October 31, 2011               [Page 21]
+
+Internet-Draft                pnfs objects                    April 2011
 
-   map a file offset to the correct component object by setting the
-   number of parity components to 2 instead of 1 for RAID4 or RAID5.
-   Clients may simply choose to read data through the metadata server if
-   two components are missing or damaged.
 
 5.4.5.  RAID Usage and Implementation Notes
 
@@ -1081,6 +1188,15 @@ Internet-Draft                pnfs objects                 November 2010
    requirements such as this.  For example, the metadata server may do
    so by not granting overlapping write layouts within mirrored objects.
 
+   Many alternative encoding schemes exist for P>=2 [22].  These involve
+   P or Q equations different than those used in PNFS_OSD_RAID_PQ.
+   Thus, if one of these schemes is to be used in the future, a distinct
+   value must be added to pnfs_osd_raid_algorithm4 for it.  While Reed-
+   Solomon codes are well understood, recently discovered schemes such
+   as Liberation codes are more computationally efficient for small
+   group_widths, and Cauchy Reed-Solomon codes are more computationally
+   efficient for higher values of P.
+
 
 6.  Object-Based Layout Update
 
@@ -1112,13 +1228,9 @@ Internet-Draft                pnfs objects                 November 2010

-
-
-
-
-Halevy, et al.            Expires May 21, 2011                 [Page 20]
+Halevy, et al.          Expires October 31, 2011               [Page 22]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

 6.1.  pnfs_osd_deltaspaceused4
@@ -1172,9 +1284,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 21]
+Halevy, et al.          Expires October 31, 2011               [Page 23]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    COMPOUND RPC.
@@ -1228,9 +1340,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 22]
+Halevy, et al.          Expires October 31, 2011               [Page 24]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    struct layoutreturn_file4 {
@@ -1284,9 +1396,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 23]
+Halevy, et al.          Expires October 31, 2011               [Page 25]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    o  PNFS_OSD_ERR_EIO indicates the operation failed because the object
@@ -1340,9 +1452,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 24]
+Halevy, et al.          Expires October 31, 2011               [Page 26]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    objects that generated errors during data transfers.  These are hints
@@ -1396,9 +1508,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 25]
+Halevy, et al.          Expires October 31, 2011               [Page 27]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    /// };
@@ -1452,9 +1564,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 26]
+Halevy, et al.          Expires October 31, 2011               [Page 28]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    optional so the client can give values for only the parameters it
@@ -1508,9 +1620,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 27]
+Halevy, et al.          Expires October 31, 2011               [Page 29]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

 10.2.  LAYOUTCOMMIT
@@ -1564,9 +1676,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 28]
+Halevy, et al.          Expires October 31, 2011               [Page 30]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    recalled layouts.  In response, the client SHOULD return layouts of
@@ -1620,9 +1732,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 29]
+Halevy, et al.          Expires October 31, 2011               [Page 31]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    capabilities allowing it I/O access to the specified objects
@@ -1676,9 +1788,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 30]
+Halevy, et al.          Expires October 31, 2011               [Page 32]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    The security types are opaque to the generic layers of the pNFS
@@ -1732,9 +1844,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 31]
+Halevy, et al.          Expires October 31, 2011               [Page 33]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    the request to the storage device, and that request is signed with
@@ -1788,9 +1900,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 32]
+Halevy, et al.          Expires October 31, 2011               [Page 34]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    object at any time, for any reason (with the understanding that there
@@ -1844,9 +1956,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 33]
+Halevy, et al.          Expires October 31, 2011               [Page 35]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

          Representation Standard (XDR) Description", RFC 5662,
@@ -1900,9 +2012,9 @@ Internet-Draft                pnfs objects                 November 2010

-Halevy, et al.            Expires May 21, 2011                 [Page 34]
+Halevy, et al.          Expires October 31, 2011               [Page 36]
 
-Internet-Draft                pnfs objects                 November 2010
+Internet-Draft                pnfs objects                    April 2011

    [17]  T11 1619-D, "Fibre Channel Framing and Signaling - 2
@@ -1914,6 +2026,18 @@ Internet-Draft                pnfs objects                 November 2010
    [19]  MacWilliams, F. and N. Sloane, "The Theory of Error-Correcting
          Codes, Part I", 1977.
 
+   [20]  Anvin, H., "The Mathematics of RAID-6", May 2009,
+         <http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf>.
+
+   [21]  The free encyclopedia, Wikipedia., "Least common multiple",
+         April 2011,
+         <http://en.wikipedia.org/wiki/Least_common_multiple>.
+
+   [22]  Plank, James S., and Luo, Jianqiang and Schuman, Catherine D.
+         and Xu, Lihao and Wilcox-O&qtHearn, Zooko, XXX., "A Performance
+         Evaluation and Examination of Open-source Erasure Coding
+         Libraries for Storage", 2007.
+
 
 Appendix A.  Acknowledgments
 
@@ -1936,15 +2060,6 @@ Authors' Addresses
    URI:   http://www.panasas.com/

-   Boaz Harrosh
-   Panasas, Inc.
-   1501 Reedsdale St. Suite 400
-   Pittsburgh, PA  15233
-   USA
-
-   Phone: +1-412-323-3500
-   Email: [email-masked]
-   URI:   http://www.panasas.com/

@@ -1953,12 +2068,20 @@ Authors' Addresses

+Halevy, et al.          Expires October 31, 2011               [Page 37]
+
+Internet-Draft                pnfs objects                    April 2011

+   Boaz Harrosh
+   Panasas, Inc.
+   1501 Reedsdale St. Suite 400
+   Pittsburgh, PA  15233
+   USA
 
-Halevy, et al.            Expires May 21, 2011                 [Page 35]
-
-Internet-Draft                pnfs objects                 November 2010
+   Phone: +1-412-323-3500
+   Email: [email-masked]
+   URI:   http://www.panasas.com/

    Brent Welch
@@ -2001,16 +2124,5 @@ Internet-Draft                pnfs objects                 November 2010

-
-
-
-
-
-
-
-
-
-
-
-Halevy, et al.            Expires May 21, 2011                 [Page 36]
+Halevy, et al.          Expires October 31, 2011               [Page 38]

---

From: Boaz Harrosh <[email-masked]>
Date: Fri, 29 Apr 2011 13:11:24 +0300

I'm Sending in the revised pnfs-objects RFC with the most important fixes needed for a
more final/correct draft.

Attached is the draft-ietf-nfsv4-rfc5664bis-01.txt file. It incorporates My fixes
to the maths, and Benny's review.
[As reply to this mail I'll send the .diff file of -01.txt from the previous -00.txt,
 for easy review]

I have also published the rfc git tree on open-osd.org website. The *master* branch
is still on rfc5664bis-00 version. Checkout the *rfc5664bis-01* for this version.
It is all in gitweb here:
http://git.open-osd.org/gitweb.cgi?p=ietf-rfc5664.git;a=shortlog;h=refs/heads/rfc5664bis-01
The git URL is:
git clone git://git.open-osd.org/ietf-rfc5664.git

I will move master to the new branch only after Brent Welch's review! Brent?
(You can edit the txt directly and send me the .diff from the original version, I can
 easily transfer the changes to the xml.tmpl file. Though now it is easy to change the
 xml.tmpl directly and even compile it, by git cloning the above tree and running make)

These are the changes added to the *rfc5664bis-01* branch:

commit b8c1feb836c186830500994cc2339fba4663ff5b
Author: Boaz Harrosh <[email-masked]>
Date:   Fri Apr 29 00:33:25 2011 +0300

    01: draft-ietf-nfsv4-rfc5664bis-01 output files
    
    modified after the new version 01 change:
       draft-ietf-nfsv4-rfc5664bis.html
       draft-ietf-nfsv4-rfc5664bis.txt
       draft-ietf-nfsv4-rfc5664bis.x
    
    Signed-off-by: Boaz Harrosh <[email-masked]>

commit fe81d532f702d527972ccce8b2c60e330ecea3d4
Author: Boaz Harrosh <[email-masked]>
Date:   Fri Feb 18 17:52:03 2011 -0800

    01: Fixes to the striping and Raid Math
    
    The raid4/5/6 maths was totally out of wak. Fix it.
    Add comment about raid rotation starting on a group boundary.
    Actually explain better the rotating raid6 that we do.
    Add reference to two raid6 papers.
    
    Increment to version 01
    
    Signed-off-by: Benny Halevy <[email-masked]>
    Signed-off-by: Boaz Harrosh <[email-masked]>

commit 3ebeda56bbaf7b850343b5fe24fdb2583ca13bf1
Author: Boaz Harrosh <[email-masked]>
Date:   Fri Apr 29 10:52:47 2011 +0300

    Change version release policy and git usage
    
    Rename rfc5664bis-00.xml.tmpl => rfc5664bis.xml.tmpl
    
    Change Makefile and file names so there is no longer any
    file-names renaming when incrementing a version.
    
    All files have their base -rfc5664bis.XXX name in git, Incrementing
    of the version number at Makefile.info will insert the version into
    files (as before) and the Makefile will also produce
    -rfc5664bis-$VER.{txt,html} for convenience of sending them via email,
    with the version attached to the name.
    
    Added -rfc5664bis.html to the git because I will reference it from
    the open-osd.org website. It will be displayed directly from git.
    
    Added -rfc5664bis.x -rfc5664bis.h rfc5664bis_xdr.c to git. Even though
    they are output files they might be used directly from gitweb without
    the need for the xml2rfc tool-chain.
    
    Removed *.xml output files from git. All other target files have
    usage other then during the make process, Not so the *.xml files.
    
    Add a .gitignore file, so git will not complain about above
    -rfc5664bis-$VER.{txt,html} and *.xml output files.
    
    Signed-off-by: Boaz Harrosh <[email-masked]>

commit 0a515925fe6c6bd8f35b25240e82c8cedfed010f
Author: Boaz Harrosh <[email-masked]>
Date:   Fri Apr 29 10:51:38 2011 +0300

    xmlrfc: Revert back to the version that works
    
    Our rfc xml files are not compateble with xml2rfc-1.35. Revert
    back to xml2rfc-1.34 that works.
    
    Remove the version from the name, change Makefile to directly
    work on the git version
    
    Signed-off-by: Boaz Harrosh <[email-masked]>

commit b7b641054b7fd4da4693049f6cbff63553524bf3
Author: Boaz Harrosh <[email-masked]>
Date:   Tue Apr 26 16:59:49 2011 +0300

    DELETE all obsolete files (They are still available in git)
    
    We have changed version progression policy. From now on we'll do
    'git mv OLD_VER.* NEW_VER.*' and always have a single latest copy
    of the files checked out. Old versions with their old names can
    be checked-out from git history if needed.
    
    Signed-off-by: Boaz Harrosh <[email-masked]>

Cheers
Boaz