Subject: [nfsv4] FW: New Version Notification for draft-faibish-nfsv4-pnfs-lustre-layout-00.txt

From: "faibish, sorin" <[email-masked]>
Date: Fri, 5 Oct 2012 13:49:06 -0400

Spencer,

As you probably saw I posted the pNFS Lustre Layout draft. I would like to have an agenda item discussion on the draft in Atlanta. Thank you

/Sorin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, October 05, 2012 11:47 AM
To: faibish, sorin
Cc: Peng, Tao
Subject: New Version Notification for draft-faibish-nfsv4-pnfs-lustre-layout-00.txt

A new version of I-D, draft-faibish-nfsv4-pnfs-lustre-layout-00.txt
has been successfully submitted by Sorin Faibish and posted to the
IETF repository.

Filename:	 draft-faibish-nfsv4-pnfs-lustre-layout
Revision:	 00
Title:		 pNFS Lustre Layout Operations
Creation date:	 2012-10-05
WG ID:		 Individual Submission
Number of pages: 19
URL:             http://www.ietf.org/internet-drafts/draft-faibish-nfsv4-pnfs-lustre-layout-00.txt
Status:          http://datatracker.ietf.org/doc/draft-faibish-nfsv4-pnfs-lustre-layout
Htmlized:        http://tools.ietf.org/html/draft-faibish-nfsv4-pnfs-lustre-layout-00

Abstract:
   Parallel NFS (pNFS) extends Network File System version 4.1 (NFSv4.1)
   to allow clients to directly access file data on the storage used by
   the NFSv4.1 server. This ability to bypass the server for data access
   can increase both performance and parallelism, but requires
   additional client functionality for data access, some of which is
   dependent on the class of storage used, a.k.a. the Layout Type. The
   main pNFS operations and data types in NFSv4 Minor version 1 specify
   a layout-type-independent layer; layout-type-specific information is
   conveyed using opaque data structures whose internal structure is
   further defined by the particular layout type specification. This
   document specifies the NFSv4.1 Lustre pNFS Layout Type as a companion
   to the main NFSv4 Minor version 1 specification.

The IETF Secretariat

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 5 Oct 2012 18:00:07 +0000

Yes.  Thanks.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of faibish, sorin
Sent: Friday, October 5, 2012 10:49 AM
To: Spencer Shepler
Cc: [email-masked]
Subject: [nfsv4] FW: New Version Notification for draft-faibish-nfsv4-pnfs-lustre-layout-00.txt

Spencer,

/Sorin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, October 05, 2012 11:47 AM
To: faibish, sorin
Cc: Peng, Tao
Subject: New Version Notification for draft-faibish-nfsv4-pnfs-lustre-layout-00.txt

Abstract:

The IETF Secretariat

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Haynes, Tom" <[email-masked]>
Date: Sat, 6 Oct 2012 16:52:05 +0000

Sorin,

This is editorial only, I use "you" and I mean the work, not the
authors.

Thanks for understanding,
Tom

1) Page 1: The Copyright is for 2011, that is off.

2) Figure 1 is from 5661, please cite that source.

3) Page 3: You define OSS as both a Lustre storage
device and an object storage server.

I think you want a table of definitions.

4) In many places, you reintroduce the meaning of
an acroynym:

   Lustre uses block devices (SCSI LUNs) for file data (called OST) and
   metadata storages (MDT) and each block device can be managed by only

and

   An MDS (metadata server) provides metadata services. One MDS manages
   one metadata target (MDT). Each MDT (SCSI LUN) stores file metadata,

Please do so only once.

5) It gets confusing whether you mean a pNFS MDS or a Lustre MDS.
I would suggest that you preface a MDS with Lustre if that is what
you mean. Else it is by default a pNFS MDS.

6) I don't understand this:

  The Lustre protocol specifies several operations on objects,
  including OPEN, READ, WRITE, GET ATTRIBUTES, SET ATTRIBUTES, CREATE,
  and DELETE. However, using the Lustre layout the client only uses the
  OPEN, READ, WRITE and GET ATTRIBUTES commands. The other commands are
  only used by the pNFS server.

What client? An NFSv4.1 one or a Lustre one?

What pNFS server? The MDS?

There is more, but I'll wait for clarification.

(By the way, this is indented differently than the rest of the
document.)

7) Repeated sentence:

   Essentially, EA describes the mapping
   between file object id and its corresponding OSTs. Essentially, EA
   describes the mapping between file object identifier and its
   corresponding OSTs.

8) Page 4

Okay, the Introduction is over and I have no clue as to what
your use case is here. Why do we want to consider this new layout?

What are the pros and cons? I'm pretty sure in Paris we gave
you some feedback.

9) Page 5,

   /// * Copyright (c) 2010 IETF Trust and the persons identified

It is 2012.

10) Page 6

   /// /*
   /// * pnfs_lustre_prot.x
   /// */

Fix the indentation

10.1) While 5662 does produce nfs4_prot.x, I would suggest that
you point out there is potential confusion between using the
nfs4_prot.x from 3530, 3530bis, and 5662:

   ///
   /// %#include <nfs4_prot.x>
   ///

11) Page 6

   Lustre uses two magic numbers to identify different lov_mds_md
   versions.

What is the lov_mds_md and why do we care about it?

11.1)

   /// enum pnfs_lov_magic {
   ///   LOV_MAGIC_V1 = 0x0BD10BD0 /* to identify lov_mds_md_v1 */
   ///   LOV_MAGIC_V3 = 0x0BD30BD0 /* to identify lov_mds_md_v3 */
   /// };

Need a comma after the first enum value.

12) Page 7

   (The format value is
   0x0BD10BD0 for version V1 capability.  However, it seems most robust
   to call out the magic number explicitly.)

Robust in what way?

12) Page 7

   /// struct pnfs_los_object_cred4 {
   ///     pnfs_los_objid4         lc_object_id;
   ///     pnfs_los_cap_key_sec4   lc_cap_key_sec;
   ///     opaque                  lc_capability_key<>;
   ///     opaque                  lc_capability<>;
   /// };
   ///

Where is pnfs_los_objid4 defined?

Mike Eisler put together a simple way to test your XDR file, you
can grab a copy of the NFSv4.2 (or 3530bis) git tree and adapt
the Makefile to strip the XDR out of this file and compile the
XDR. Look at 'make testx'.

13) The convention we have adopted for structure names would have
these as ploc_object_id. First letter of each word in the structure
name. Please follow convention here and throughout.

14) Page 7

   Lustre ptlrpc supports gss authentication. So pnfs_los_object_cred4
   is put inside pnfs_los_layout4 so that if network requires security,
   credentials can be passed around.

What is ptlrpc?

There is a lot of assumption that the reader is more than passing
familiar with Lustre.

15) Page 8

4.3. data stripping algorithms

Caps are nice.

Are we stripping data from the files or are we striping it across
the DSes?

16) You define RAID later.

17)

/// const LOV_PATTERN_RAID0 = 0x001 /* stripes are used round-robin */
   /// const LOV_PATTERN_RAID1 = 0x002 /* stripes are mirrors of each
   other */

Indentation is off and the second cannot span multiple lines. If
width is an issue, consider putting the comments above each line
and not on the line.

18) Page 7 and 8, more reintroductions of acronyms.

19) Page 8

The following enum specifies the manner in which a OST (OSS target) can
be specified. The target can be specified by the network access
protocol type used.

/// enum pnfs_los_targetid_type4 {
///     LOS_TARGET_TCP  = 1,
///     LOS_TARGET_IB   = 2
/// };

5.2. pnfs_los_deviceaddr4

The specification for an object device address is as follows:

/// union pnfs_los_targetid4 switch(pnfs_los_targetid_type4 oti_type) {
///     case OSS_TARGET_TCP:
///         netaddr4    tcp_addr<>;
///
///     case OSS_TARGET_IB:
///         netaddr4    ib_addr<>;
///
///     default:
///         void;
/// };
///
/// struct pnfs_los_deviceaddr4 {
///     pnfs_los_targetid4      lda_targetid;
///     opaque                  lda_ossname<>;
/// };

While it is glaringly obvious what TCP and IB refer to, you might want
to spell it out.

Are there other transports?

Why these two?

ou have LOS_TARGET_TCP and OSS_TARGET_TCP, which indicates again you
need to 'make testx'.

But even more importantly, why are you wrapping the oti_type over
the netaddr4? Doesn't that already handle this?

Provide a reference to this:

   [15]  Eisler, M., "IANA Considerations for Remote Procedure Call
         (RPC) Network Identifiers and Universal Address Formats",
         RFC 5665, January 2010.

By the way, again, oti_type, tcp_addr, and ib_addr do not follow
the naming conventions.

Also, these lines are indented inconsistently with the rest of the XDR.

20) Page 9

5.2.1.  OSS Target Identifier

When "lda_targetid" is specified as an OSS_TARGET_TCP, if the TCIP
network protocol is used or as the OSS_TARGET_IB if Infiniband protocol
is used.

When "lda_targetid" is specified the opaque field MUST be formatted as
the OSS name.

Isn't it always specified?

Anyway, why is the desciption of the target id down here and not in 5.1?

21) You have used NFSv4.1 before this:

specifies a TCP/IP or IB based endpoint (as specified in NFSv4.1 [3]).
When given, the client SHOULD use it to probe for the OSS device at
the given network address.  The client MAY still use other discovery

Faibish et al.          Expires April 5, 2013                  [Page 9]
^L
Internet-Draft      pNFS Lustre Layout Operations          October 2012

   mechanisms to locate the device using the lda_targetid.  In
   particular, such an external name service SHOULD be used when the
   devices may be attached to the network using multiple connections,
   and/or multiple storage fabrics (e.g., TCP or IB).

6. Lustre-Based Layout

The layout4 type is defined in the NFSv4.1 [3] as follows:

But not cited it - now all of a sudden, every reference to NFSv4.1
has to cite it?

22) Indentation changes across pages:

5.2.2.  Device Network Address

The network address is given with the netaddr4 type, which
specifies a TCP/IP or IB based endpoint (as specified in NFSv4.1 [3]).
When given, the client SHOULD use it to probe for the OSS device at
the given network address.  The client MAY still use other discovery

Faibish et al.          Expires April 5, 2013                  [Page 9]
^L
Internet-Draft      pNFS Lustre Layout Operations          October 2012

   mechanisms to locate the device using the lda_targetid.  In
   particular, such an external name service SHOULD be used when the
   devices may be attached to the network using multiple connections,
   and/or multiple storage fabrics (e.g., TCP or IB).

But more importantly, what external name service?

23) Page 10:

The layout4 type is defined in the NFSv4.1 [3] as follows:

enum layouttype4 {
    LAYOUT4_NFSV4_1_FILES   = 1,
    LAYOUT4_OSD2_OBJECTS    = 2,
    LAYOUT4_BLOCK_VOLUME    = 3
    LAYOUT4_OSS_OBJECTS     = 4,
};

Missing a comma after 3.

But please, RFC5661 does not define LAYOUT4_OSS_OBJECTS.

You are going to need an IANA Considerations section asking to
be assigned a new value. Please refer to Section 3.3.13 of 5661.

For now, like Benny does, you should not be defining a value
in the assigned space.

24) Page 9:

/// struct pnfs_lov_ost_data { /* per-stripe data structure */
///     __u64 l_object_id;  /* OST object ID */
///     __u64 l_object_seq; /* OST object seq number */
///     __u32 l_ost_gen;    /* generation of this l_ost_idx */

Where is __u64, __32, __u16, etc defined? Please use standard
XDR types or define these.

And please follow standard naming conventions.

25) And again, indentation is off:

/// struct pnfs_lov_mds_md_v1 { /* LOV EA mds/wire data */
///     __u32 lmm_pattern;   /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */
///     __u64 lmm_object_id;    /* LOV object ID */
///     __u64 lmm_object_seq;   /* LOV object seq number */
///     __u32 lmm_stripe_size;  /* size of stripe in bytes */
///     __u16 lmm_stripe_count; /* num stripes in use for this object
*/
///     __u16 lmm_layout_gen;     /* layout generation number */
/// struct pnfs_lov_ost_data lmm_objects[0]; /* per-stripe data */
/// };

You don't need to say "struct pnfs_lov_ost_data". You can just use
"pnfs_lov_ost_data".

By the way, another convention would have these be pnfs_lov_ost_data4.

26) Can not split lines:

///       char  lmm_pool_name[LOV_MAXPOOLNAME]; /* must be 32bit
aligned */

So, with the tool to produce XDR, the second line is not output
into your .x file. That means this does not compile.

More importantly, when and where is LOV_MAXPOOLNAME defined?

27) You can't have the same name twice for a field:

/// union pnfs_lov_mds_md switch (pnfs_lov_magic lmm_magic) {
///     case LOV_MAGIC_V1:
///         pnfs_lov_mds_md_v1      mds_md;
///     case LOV_MAGIC_V3:
///         pnfs_lov_mds_md_v3      mds_md;
///     default:
///         void;
/// };

28) This section goes heavily into needing to know Lustre. Who
is going to review it externally?

The server MAY grow the file by adding more components to the stripe
while clients hold valid layouts until the file has reached its final
stripe width.  The file length in this case MUST be limited to the
number of bytes in a full stripe.

This is confusing.

What "server"?

How does it add components to the stripe? How does it communicate
it to the client?

29) Wow, costly?

Clients need to obtain layout generation before IO and check layout
generation after IO. If layout generation is changed, client need to
redo the operations.

"the client needs to"

Is this related to the "server" growing the file?

30) Names are not consistent:

/// struct pnfs_los_layout4 {
///     pnfs_lov_magic     lmm_magic;
///     pnfs_lov_mds_md    lov_mds_md;
///     uint32_t                llo_comps_index;
///     pnfs_los_object_cred4   llo_components<>;
/// };
///

Why is lmm_magic present?

lov_mds_mdd already has it present.

31) Why 0?

The "llo_components" is an array of "pnfs_los_object_cred4", containing
credentials that client need to use to connect to OSS's. The
"llo_components" may present all credentials that the client needs to
access each object of the file, in which case "llo_comps_index" is set
to zero. Otherwise if a file has multiple layout segments and different
stripping patterns, "llo_comp_index" is set to the index of the object
composing the file, and "llo_components" MUST have exactly one entry.

What is the llo_comps_index used for?

If llo_components MUST have exactly one entry, then won't llo_comp_index
always be one then? Or actually, zero by convention?

How then do you differentiate from the other case?

Very confusing here.

Note that you refer to it as "llo_comps_index" and "llo_comp_index".

32)  Which metadata server?

Note that the layout depends on the file size, which the client
learns, by doing GETATTR commands to the metadata server.  The client
uses the file
size to decide if it should fill holes with zeros or return a short
read. Striping patterns can cause cases where component objects are
shorter than other components because a hole happens to correspond to
the last part of the component object.

Which client? And how can it return short reads?

What is a hole? Never defined.

Again, confusing.

Is this related to the fact that not all DSes are allocated?

33) Use ""

In the above equation obj_id is the object identifier of a file
fragment on the ost p, "stripe size" is the size of each file segment
on one OST and "stripe width" is the number of OST's used. So if the
"stripe size" is 1MB, and the "stripe width" is 3, then this would mean
t

Need " voer obj_id to be consistent with your use.

34) What is a LSM?

Before reading the file, client will query the MDS and be informed that
it should talk to <ost p, ost q, ost r> for this operation. This
information is structured in so-called LSM, and client side LOV
(logical object volume) is to interpret this information so client can

What does talk mean?

"in so-called LSM" - this is a protocol standard, the language is not
precise.

35) 6.4. RAID Algorithms

Finally, RAID is introduced.

36) Where is this defined?

If a component object is marked as PNFS_OST_MISSING, the pNFS
client MUST either return an I/O error if this component is attempted
to be read or, alternatively, it can retry the READ against the pNFS
server.

PNFS_OST_MISSING that is.

And where is it marked?

What server is the pNFS server? The MDS?

37) are?

In this case the component objects are data bytes are
still located by the above equations for C and O.

Which are?

38) 7.1. pnfs_los_layouthint4

You need to define what the hints mean.

39) I don't see a mirroring hint.

 e.g. it can provide a hint for the desired number of mirrored
components, regardless of the RAID algorithm selected for the file.

So how does the client do that?

Please enumerate all of the hints than can be provided.

Thanks,
Tom

---

From: "faibish, sorin" <[email-masked]>
Date: Sat, 6 Oct 2012 14:06:39 -0400

Thank you for the review and very good comments. Hopefully i or WE will have a new draft addressing your comments before the meeting. There are some comments that will need discussions in Atlanta.  I will be there and i hope we will have an agenda item to discuss this.

/Sorin

Sent from my iPhone

---

From: "faibish, sorin" <[email-masked]>
Date: Fri, 19 Oct 2012 18:39:16 -0400

Tom,

I finally got a new draft that addressed most of your comments in version 01.
http://www.ietf.org/internet-drafts/draft-faibish-nfsv4-pnfs-lustre-layout-01.txt

Some of the issues are not yet addressed as my co-author had a tragedy in the family and couldn't include his changes. I tried to address them myself but might need another version before Atlanta. See within my reply to your comments.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Haynes, Tom
Sent: Saturday, October 06, 2012 12:52 PM
To: faibish, sorin
Cc: nfsv4 list
Subject: Re: [nfsv4] New Version Notification for draft-faibish-nfsv4-pnfs-lustre-layout-00.txt

> 1) Page 1: The Copyright is for 2011, that is off.
[SF] Fixed in draft 01.

> 2) Figure 1 is from 5661, please cite that source.
[SF] Fixed in draft 01.

>
> 3) Page 3: You define OSS as both a Lustre storage device and an
> object storage server.
>
[SF] Fixed in draft 01.

>
> 4) In many places, you reintroduce the meaning of an acroynym:
>
[SF] Fixed in draft 01.

> I would suggest that you preface a MDS with Lustre if that is what you
> mean. Else it is by default a pNFS MDS.
>
[SF] Fixed in draft 01.

> What pNFS server? The MDS?
[SF] Fixed in draft 01.

> There is more, but I'll wait for clarification.

[SF] Peng will address this issue when he will be available.

>   describes the mapping between file object identifier and its
[SF] Fixed in draft 01.

>
> 8) Page 4
>
> Okay, the Introduction is over and I have no clue as to what your use
> case is here. Why do we want to consider this new layout?
>
> What are the pros and cons? I'm pretty sure in Paris we gave you some
> feedback.
>

[SF] Need discussion in Atlanta. We will put a slide together to discuss.

>   /// * Copyright (c) 2010 IETF Trust and the persons identified
[SF] Fixed in draft 01.

>
> 10) Page 6
>
>
>   /// /*
>   /// * pnfs_lustre_prot.x
>   /// */
>
> Fix the indentation
>
> 10.1) While 5662 does produce nfs4_prot.x, I would suggest that you
> point out there is potential confusion between using the nfs4_prot.x
> from 3530, 3530bis, and 5662:

[SF] Need to more details. Could you explain what I need to do in more details please?

> What is the lov_mds_md and why do we care about it?

[SF] Peng will address this

> Need a comma after the first enum value.
[SF] Fixed in draft 01.

>   to call out the magic number explicitly.)
[SF] Peng will address this issue.

> Where is pnfs_los_objid4 defined?

[SF] Fixed in draft 01.

>
> Mike Eisler put together a simple way to test your XDR file, you can
> grab a copy of the NFSv4.2 (or 3530bis) git tree and adapt the
> Makefile to strip the XDR out of this file and compile the XDR. Look
> at 'make testx'.
>

[SF] Peng will do this test before resubmitting 02.

> name. Please follow convention here and throughout.
[SF] Fixed in draft 01.

> There is a lot of assumption that the reader is more than passing

[SF] Fixed in draft 01.

> 4.3. data stripping algorithms

[SF] Ha, ha, ha. Good laugh. I must have been tired when I wrote that.

> Are we stripping data from the files or are we striping it across the
> DSes?
[SF] Addressed in draft 01.

> Indentation is off and the second cannot span multiple lines. If width
> is an issue, consider putting the comments above each line and not on
> the line.
[SF] Fixed in draft 01.

> 18) Page 7 and 8, more reintroductions of acronyms.
[SF] Fixed in draft 01.

>
> 19) Page 8
>
> The following enum specifies the manner in which a OST (OSS target)
> can be specified. The target can be specified by the network access
> protocol type used.
>
[SF] Fixed in draft 01.

> Are there other transports?

[SF] I will leave this to Peng to answer as he is more knowledgeable in Lustre than me.

> ou have LOS_TARGET_TCP and OSS_TARGET_TCP, which indicates again you

[SF] Fixed in draft 01.

>
> But even more importantly, why are you wrapping the oti_type over the
> netaddr4? Doesn't that already handle this?

[SF] This is another good question for Peng to address

>         (RPC) Network Identifiers and Universal Address Formats",
[SF] Fixed in draft 01.

>
> By the way, again, oti_type, tcp_addr, and ib_addr do not follow the
> naming conventions.
>

[SF] Fixed in draft 01.

> network protocol is used or as the OSS_TARGET_IB if Infiniband
> protocol is used.
>

[SF] Fixed in draft 01.

> Anyway, why is the desciption of the target id down here and not in 5.1?

[SF] Peng to address

> But not cited it - now all of a sudden, every reference to NFSv4.1 has
> to cite it?
>
[SF] Fixed in draft 01.

> The network address is given with the netaddr4 type, which specifies a
> TCP/IP or IB based endpoint (as specified in NFSv4.1 [3]).

[SF] Again Peng needs to validate this statement:
Replace "such an external name service SHOULD" with "an external name service (external to the data protocol coming from LNET)  SHOULD"

>    LAYOUT4_OSD2_OBJECTS    = 2,
[SF] Fixed in draft 01.

> You are going to need an IANA Considerations section asking to be
> assigned a new value. Please refer to Section 3.3.13 of 5661.
>
> For now, like Benny does, you should not be defining a value in the
> assigned space.
[SF] Fixed in draft 01.

> Where is __u64, __32, __u16, etc defined? Please use standard XDR
> types or define these.
>
[SF] Fixed in draft 01.

> /// struct pnfs_lov_ost_data lmm_objects[0]; /* per-stripe data */ ///
> };
>
[SF] Fixed in draft 01.

> By the way, another convention would have these be pnfs_lov_ost_data4.

[SF] Peng to address.

> 26) Can not split lines:

[SF] Fixed in draft 01.

> So, with the tool to produce XDR, the second line is not output into
> your .x file. That means this does not compile.
>

[SF] Peng to address this.

> ///         pnfs_lov_mds_md_v3      mds_md;

[SF] Fixed in draft 01.

> 28) This section goes heavily into needing to know Lustre. Who is
> going to review it externally?

[SF] Andreas Dilger and Oleg Drokin.

> number of bytes in a full stripe.
[SF] Fixed in draft 01.

>
> How does it add components to the stripe? How does it communicate it
> to the client?
>

[SF] Peng will Explain based on the document Lustre internals (ref [1]).

> Is this related to the "server" growing the file?

[SF] Peng will address in next draft.

> Why is lmm_magic present?

[SF] Peng to clarify in next draft.

> lov_mds_mdd already has it present.

[SF] Fixed in draft 01.

>
> 31) Why 0?
>
> The "llo_components" is an array of "pnfs_los_object_cred4",
> containing credentials that client need to use to connect to OSS's.
> The "llo_components" may present all credentials that the client needs
> to access each object of the file, in which case "llo_comps_index" is
> set to zero. Otherwise if a file has multiple layout segments and
> different stripping patterns, "llo_comp_index" is set to the index of
> the object composing the file, and "llo_components" MUST have exactly one entry.
>

[SF] Peng will address and fix if wrong and will explain the logic.

>
> If llo_components MUST have exactly one entry, then won't
> llo_comp_index always be one then? Or actually, zero by convention?
>

[SF] Peng will clarify the logic and fix if wrong.

> uses the file size to decide if it should fill holes with zeros or
> return a short read. Striping patterns can cause cases where component
> objects are shorter than other components because a hole happens to
> correspond to the last part of the component object.
>

[SF] Peng will clarify what short read is in next draft.

> What is a hole? Never defined.

[SF] Fixed in draft 01.
Peng needs to validate my definition: hole is represented with a single layout extent.

> Is this related to the fact that not all DSes are allocated?

[SF] Peng to address.

> "stripe size" is 1MB, and the "stripe width" is 3, then this would
> mean t
>
[SF] Fixed in draft 01.

>
> 34) What is a LSM?
>
> Before reading the file, client will query the MDS and be informed
> that it should talk to <ost p, ost q, ost r> for this operation. This
[SF] Fixed in draft 01.

> Finally, RAID is introduced.

[SF] Should I introduce it sooner in 4.3?

> If a component object is marked as PNFS_OST_MISSING, the pNFS client
> MUST either return an I/O error if this component is attempted to be
> read or, alternatively, it can retry the READ against the pNFS server.
>

[SF] Peng will fix and define if true.

> In this case the component objects are data bytes are still located by
> the above equations for C and O.
>
> Which are?
>

[SF] Fixed in draft 01.
Peng needs to address where OST_MISSING is used and what it is?

> You need to define what the hints mean.

[SF] Fixed in draft 01.
Peng to validate that hint as an indicator?

> Please enumerate all of the hints than can be provided.

[SF] Peng will provide the list of hints. We will include a table in next draft.
Thanks for your patience

/Sorin