Subject: Re: [nfsv4] New Version Notification for draft-bhalevy-nfs-obj-00.txt

From: "Haynes, Tom" <[email-masked]>
Date: Sat, 6 Oct 2012 05:31:53 +0000

Benny,

Could you please rename this to 

draft-bhalevy-nfsv4-obj

as a bare minimum? That will enable us to easily find it at:

http://datatracker.ietf.org/wg/nfsv4/

Thanks,
Tom

On Aug 31, 2012, at 6:17 AM, Benny Halevy <[email-masked]> wrote:

> Folks,
> 
> I've submitted a proposed extension to the pnfs objects layout
> that introduces the use of NFS filers as object storage data servers.
> 
> The following features are proposed as well:
> * OSD Multi-path: The OSD address may provide an array of network addresses (ooa_netaddrs).
> * I/O Statistics: The client may report I/O stats on LAYOUTRETURN
> 
> The motivation for adding support for NFS comes from:
> 1. The desire to use an ubiquitous, standard protocol to access the data servers.
> The T10 OSD protocol, although standard, lacks wide adoption, while NFS and in particular
> NFSv3 is very popular and widely available.
> 
> 2. Encourage best-of-breed solution.
> In contrast to the files layout, the object layout does not require a proprietary
> back-end protocol, hence the proposed layout allows one to mix and match a metadata
> server and data servers from different vendors.
> A simple security control method is proposed to achieve that:
> The metadata server controls the file ownership and permissions of the objects
> stored on data servers and the client is handed a corresponding RPC credential
> on LAYOUTGET a-la OSD capabilities.  Outstanding credentials are unilaterally
> revoked by the MDS by modifying the objects user or group owner.
> 
> Benny
> 
> -------- Original Message --------
> Subject: New Version Notification for draft-bhalevy-nfs-obj-00.txt
> Date: Fri, 31 Aug 2012 05:52:12 -0700
> From: [email-masked]
> To: [email-masked]
> 
> 
> A new version of I-D, draft-bhalevy-nfs-obj-00.txt
> has been successfully submitted by Benny Halevy and posted to the
> IETF repository.
> 
> Filename:	 draft-bhalevy-nfs-obj
> Revision:	 00
> Title:		 Object-Based Parallel NFS (pNFS) Operations - Version 2
> Creation date:	 2012-08-31
> WG ID:		 Individual Submission
> Number of pages: 42
> URL:             http://www.ietf.org/internet-drafts/draft-bhalevy-nfs-obj-00.txt
> Status:          http://datatracker.ietf.org/doc/draft-bhalevy-nfs-obj
> Htmlized:        http://tools.ietf.org/html/draft-bhalevy-nfs-obj-00
> 
> 
> Abstract:
>   Parallel NFS (pNFS) extends Network File System version 4 (NFSv4) to
>   allow clients to directly access file data on the storage used by the
>   NFSv4 server.  This ability to bypass the server for data access can
>   increase both performance and parallelism, but requires additional
>   client functionality for data access, some of which is dependent on
>   the class of storage used, a.k.a. the Layout Type.  The main pNFS
>   operations and data types in NFSv4 Minor version 1 specify a layout-
>   type-independent layer; layout-type-specific information is conveyed
>   using opaque data structures whose internal structure is further
>   defined by the particular layout type specification.  This document
>   specifies the NFSv4.1 Object-Based pNFS Layout Type as a companion to
>   the main NFSv4 Minor version 1 specification.  Version 2 of this
>   layout type introduces the use of the Network File System protocol as
>   an object-storage protocol in addition to the T-10 OSD protocol.
> 
> 
> 
> 
> The IETF Secretariat
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 7 Oct 2012 22:38:45 +0000

4.2.3.  NFS Device Identifier

   The PNFS_OBJ_NFS pnfs_obj_type4 is used to identify NFS filer
   devices.  In this case, ona_version and ona_minorversion represent
   the NFS protocol version to be used to access the NFS filer.  Either
   the "ona_netaddrs" or "ona_fqdn" fields are used to locate the
   device. "ona_netaddrs" MAY be set to a list holding one or more of
   the device network addresses.  Alternatively, "ona_fqdn" MAY be set
   to the NFS device fully qualified domain name that can be resolved by
   the client to locate the NFS device network address (See Domain Names
   [22]).  The server MUST provide either one of "ona_netaddrs" or
   "ona_fqdn", but it MUST NOT provide both.

   "ona_path" MUST be set by the server to an exported path on the
   device.  The path provided MUST exist and be accessible to the
   client.  If the path does not exist, the client MUST ignore this
   device information and any layouts referring to the respective
   deviceid until valid device information is acquired.

I think this is an end run (*) over LAYOUT4_NFSV4_1_FILES.

If I were a customer, I would be nervous about the certification method used here. OSD components
fall naturally into the Block Based Layout of RFC 5664 because they are basically disks, not
filesystems. These NFS mounts are filesystems and questions are easy to see about directory
consistency, file consistency, filesystem consistency, etc.

>From a standards viewpoint, this is no longer work in the realm of OSD. The title is misleading
and a client which understands LAYOUT4_OBJECTS_V2 would be able to access a layout
which could be comprised solely of NFS devices. Where is the beef?

I would be perfectly fine with a proposed layout which allowed the use of NFS exported paths as
targets. But I am not fine with this proposal.

By the way, fine with such a proposal does not mean I think it would be wise to implement. These
NFS devices presented here have to be very dumb because of the lack of a control protocol between
them and the MDS. I don't see how they handle locks, stateids, and any state in general. Once we
throw in sparse files in NFSv4.2, I don't see how they would be communicated.

In short, these NFS devices are pretty dumb and do not give the benefits of a DS.

And Section 13.5 makes it pretty clear that there is no control protocol.

---

(*) For those whom American English slang might be confusing, please see http://www.thefreedictionary.com/end-run

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 7 Oct 2012 22:38:39 +0000

Benny,

Editorial, "you" means the document. :->

Thanks,
Tom

On Aug 31, 2012, at 8:17 AM, Benny Halevy <[email-masked]<mailto:[email-masked]>> wrote:

Folks,

Benny

1) You need to put the above into the Introduction. I.e., if you have to set the stage in an
email, then you also need to in the document.

I also find it very strange that there is no mention of RFC 5664 in this document. Considering
a great deal of this document is based off of that document, there should be a reference.

Is the intent that this obsoletes RFC 5664? Given that LAYOUT4_OSD2_OBJECTS is
not OBSOLETED, my guess is not.

2) Even with the stage set, what are the use cases?

Why is the version 1 of the objects layout insufficient?

3) Provide a reference

In pNFS,

Also, you haven't defined the term.

4) Period at the end:

   This document describes the layouts used with object-based
   storage devices (OSDs) that are accessed according to the OSD storage
   protocol standard (ANSI INCITS 400-2004 [1]) or the NFS protocol
   (RFC1813 [14], RFC3530 [15], RFC5661 [2])

5)

The Object Storage protocol

How is this related to the OSD storage protocol?

Pretty sure they are the same, but why are you using different terms?

6)

   The Object Storage protocol specifies
   several operations on objects, including READ, WRITE, FLUSH, GET
   ATTRIBUTES, SET ATTRIBUTES, CREATE, and DELETE.  However, using the
   object-based layout the client only uses the READ, WRITE, GET
   ATTRIBUTES, and FLUSH commands, or in the NFS case, the READ, WRITE,
   GETATTR, and COMMIT operations.  The other commands are only used by
   the pNFS server.

The way this is written, it sound like the pNFS server is used even in the non-NFS case.

BTW - which pNFS server? MDS or DS?

7) Need a comma:

With NFS filers used for object storage devices the object's owner,

After devices…

8) Mismatch between this:

   Note that the XDR code contained in this document depends on types
   from the NFSv4.1 nfs4_prot.x file ([5]).

and this

   /// %#include <nfs4_prot.x>

9) Is this always true:

   Creation and
   management of partitions is outside the scope of this document, and
   is a facility provided by the object-based storage file system.

While such acts are outside the scope, is there a standard in place which defines the requirement for
such filesystems?

BTW - what are the requirements for NFS devices?

10) Used before definition:

   However, it MUST
   refer to a device identifed as an NFS device, represented as
   oda_obj_type equal to PNFS_OBJ_NFS.

So PNFS_OBJ_NFS is in Section 3.3. I do not see a oda_obj_type defined.

Okay, it is here:

/// union pnfs_obj_deviceaddr4 switch (pnfs_obj_type4 oda_obj_type) {

I think it is cleaner to use pnfs_obj_type4 here and also in Section 3.1:

   The device MUST by identifed as an
   OSD device, represented as oda_obj_type equal to PNFS_OBJ_OSD_V1 or
   PNFS_OBJ_OSD_V2.

BTW - these are again used before defined. I would suggest that you rearrange your sections such that
3.3 is 3.1.

10.1)

   The NFS equivalent of pnfs_obj_osd_objid4 identifies the object using
   a NFS filehandle (See RFC1813 [14], RFC3530 [15], or RFC5661 [2]).

The implication here is that NFSv3, NFSv4, and NFSv4.1 filehandles are valid to use in this proposed
extension.

Given that the only valid operations are:

   An "object" is a container for data and attributes, and files are
   stored in one or more objects.  The Object Storage protocol specifies
   several operations on objects, including READ, WRITE, FLUSH, GET
   ATTRIBUTES, SET ATTRIBUTES, CREATE, and DELETE.  However, using the
   object-based layout the client only uses the READ, WRITE, GET
   ATTRIBUTES, and FLUSH commands, or in the NFS case, the READ, WRITE,
   GETATTR, and COMMIT operations.  The other commands are only used by
   the pNFS server.

then OPEN is not allowed. Since the "pNFS server" must perforce do this for the client, then how does
the pNFS server present the stateid needed for NFSv4 and NFSv4.1 operations?

   struct READ4args {
           /* CURRENT_FH: file */
           stateid4        stateid;
           offset4         offset;
           count4          count;
   };

11)  Field names do not follow the NFSv4.1 standard of being derived from the structure name:

   /// struct pnfs_obj_nfs_objid4 {
   ///     deviceid4       nid_device_id;
   ///     opaque          nid_fhandle<>;
   /// };
   ///

pono_fhandle would be consistent.

12) Rogue period:

   The second generation OSD protocol (SNIA T10/1729-D [16]). has
   additional proposed features to support more robust error recovery,
   snapshots, and byte-range capabilities.

13)

   pnfs_obj_type4 is used to indicate the object storage protocol type
   and version or whether an object is missing (i.e., unavailable).
   Some of the object-based layout- supported RAID algorithms encode

Why the '-' after 'layout'?

If this was C code, I'd say that the precedence rules were being violated.

How about:

   pnfs_obj_type4 is used to indicate whether the object was

   missing (i.e., unavailable) or the object storage protocol type

   and version.
   Some of the object-based layout- supported RAID algorithms encode

And if you have to explain "missing", perhaps a better enum would be PNFS_OBJ_UNAVAILABLE.

14) Is this needed?

   (This information can also be deduced by looking inside the
   capability type at the format field, which is the first byte.  The
   format value is 0x1 for an OSD v1 capability.  However, it seems most
   robust to call out the version explicitly.)

Where is the format field? What is the capability type?

What is it for a OSD v2 capability?

I'd ditch this altogether, but if you need to say anything, just mention while it can be
derived from OSD, this just explicitly states it.

15) How can I tell which type of object id I am looking at:

   /// struct pnfs_obj_osd_cred4 {
   ///     pnfs_obj_osd_objid4         ooc_object_id;
   ///     pnfs_obj_osd_cap_key_sec4   ooc_cap_key_sec;
   ///     opaque                      ooc_capability_key<>;
   ///     opaque                      ooc_capability<>;
   /// };
   ///
   /// struct pnfs_obj_nfs_cred4 {
   ///     pnfs_obj_nfs_objid4         onc_object_id;
   ///     opaque_auth                 onc_auth;
   /// };

Both ooc_object_id and onc_object_id appear to be the same type of object based
on the name. But they are vastly different.

16)  Redundant:

   "oc_obj_type" represents the object storage
   device protocol type and version, or whether that component is
   unavailable.

Point them to Section 3.3.

17) 3.4 is very hard to follow.

I would suggest sub-sections:

Missing
Object
NFS

You have Object and NFS covered, but no coverage of what is to go into oc_missing_obj_id.

I'd also split the chunk of XDR at the start of 3.4 into these subsections as needed.

You are hurt by embedding the XDR directly into the document. The flow needed to
compile is different from the flow needed to present.

17) Used before definition:

   When oc_obj_type indicates PNFS_OBJ_OSD_V1 or
   PNFS_OBJ_OSD_V2, the "ooc_object_id" field identifies the component
   object, the "ooc_capability" and "ooc_capability_key" fields, along

   with the "ooa_systemid" from the pnfs_obj_deviceaddr4, provide the
   OSD security credentials needed to access that object.

What is a pnfs_obj_deviceaddr4? See Section 4.2…

Also defined in Section 4 before use.

Going back to my point about flow, I'm having a difficult time at this point seeing how
given a pnfs_obj_comp4, do I get the pnfs_obj_deviceaddr4?

Also, this sentence confuses me:

   The pnfs_obj_comp4 union is used to identify each component
   comprising the file.

What is a component here? Will I have an array of pnfs_obj_comp4s?

Yes, you have used component here without defining it. I think you need a prior section
which paints the big picture.

18) Confusing:

   These appear
   in the pnfs_obj_deviceaddr4 type below under the "ooa_systemid" and
   "oda_osdname" fields.

below under

I'd say

   These appear
   in the pnfs_obj_deviceaddr4 type presented in Section 4.2

   as the "ooa_systemid" and "oda_osdname" fields.

19) By the way, Section 4 provides the overview I am asking for in 17 above for this section…

20)  Get rid of comma aster "as a SCSI Name, or"

21) What is OBJ_TARGET_ANON used for?

22) Why the change in naming scheme for the enum?

   /// enum pnfs_osd_targetid_type4 {
   ///     OBJ_TARGET_ANON             = 1,
   ///     OBJ_TARGET_SCSI_NAME        = 2,
   ///     OBJ_TARGET_SCSI_DEVICE_ID   = 3
   /// };

As per the other uses you present, this should be PNFS_OBJ_*.

23) Formatting is off:

  The specification for an object device address is as follows:

Halevy                    Expires March 4, 2013                [Page 12]
Internet-Draft                pnfs objects                   August 2012

/// union pnfs_obj_targetid4 switch (pnfs_osd_targetid_type4 oti_type) {

Break the long lines up rather than changing the start column.

24) While NetApp appreciates the free publicity, what is a filer here? :->

   device.

Are there any restrictions as to what version of NFS is to be used here?

24.1)

Why isn't this a fatal error to the application?

What gets returned to the application? Is it supposed to wait forever?

How does the client inform the MDS that there is an issue?

How does the control protocol from the MDS to the DS interact?

Finally, I'm a NFS server administrator.

a) What do I expect to see if I access this path? Data files? How are they named? How do I fsck this
path?

b) Am I allowed to let other clients access this path via protocols other than NFSv4.1?

c) What about with another layout?

25) NFSv4.1 does not define layouttype4 as such:

The layout4 type is defined in the NFSv4.1 [2] as follows:

   enum layouttype4 {
       LAYOUT4_NFSV4_1_FILES   = 1,
       LAYOUT4_OSD2_OBJECTS    = 2,
       LAYOUT4_BLOCK_VOLUME    = 3,
       LAYOUT4_OBJECTS_V2      = 0x08010004       /* Tentatively */
   };

Just mention you want to add LAYOUT_OBJECTS_V2. By the way, why the change
in format from OSD2_OBJECTS? Are these no longer OSD objects? :->

I see you have an IANA Considerations in Section 14. You need to make a reference to it
here. And that section is wrong. You are asking for them to assign you a globally unique
value. As it is, your current value is private.

BTW, 5661 defines this enum as:

   enum layouttype4 {
           LAYOUT4_NFSV4_1_FILES   = 0x1,
           LAYOUT4_OSD2_OBJECTS    = 0x2,
           LAYOUT4_BLOCK_VOLUME    = 0x3
   };

You have left off the 0x for the first 3.

26)

   "odm_num_comps" is the number of component objects the file is
   striped over.  The server MAY grow the file by adding more components
   to the stripe while clients hold valid layouts until the file has
   reached its final stripe width.  The file length in this case MUST be
   limited to the number of bytes in a full stripe.

Such an innocent paragraph and so many questions.

This is a radical departure from pNFS, which assumes that a layout has a fixed number of components.
How is the change communicated from the server (btw, I assume the MDS is meant here) to the client?
Is the layout recalled?

What happens if the client tries to write with a different view of the number of components?

Are you planning on  using byte ranges for the layout or is the file restriped when this addition of
components occurs?

27) What happens if it is not?

   The "odm_group_width" and "odm_group_depth" parameters allow a nested
   striping pattern (see Section 5.3.2 for details).  If there is no
   nesting, then odm_group_width and odm_group_depth MUST be zero.  The
   size of the components array MUST be a multiple of odm_group_width.

So if there is no nesting, then the size of the components array must be zero?

If there is nesting, then what error is returned if the size of the components array is not a multiple of
the odm_group_width?

Also, what error for:

   The "odm_mirror_cnt" is used to replicate a file by replicating its
   component objects.  If there is no mirroring, then odm_mirror_cnt
   MUST be 0.  If odm_mirror_cnt is greater than zero, then the size of
   the component array MUST be a multiple of (odm_mirror_cnt+1).

28) What error is returned here?

   The data
   placement algorithm that maps file data onto component objects
   assumes that each component object occurs exactly once in the array
   of components.  Therefore, component objects MUST appear in the
   olo_components array only once.

29) We don't use structure notation:

   the olo_components array is
   equal to olo_map.odm_num_comps.

How about odm_num_comps in olo_map?

30) olo_comps_index

   The components array may represent
   all objects comprising the file, in which case "olo_comps_index" is
   set to zero and the number of entries in the olo_components array is
   equal to olo_map.odm_num_comps.  The server MAY return fewer
   components than odm_num_comps, provided that the returned components
   are sufficient to access any byte in the layout's data range (e.g., a
   sub-stripe of "odm_group_width" components).  In this case,
   olo_comps_index represents the position of the returned components
   array within the full array of components that comprise the file.

What is olo_comps_index supposed to do?

An assumption here is that if it is a sub-stripe, it can only occur with components on the upper range of

the array? I.e., you define the start point, but not the end point. They must be adjacent, etc.

What determines sufficient? And how is an error communicated back to the MDS if this is not sufficient?

Is this to make up for the fact that odm_num_comps can grow?

31) What client?

   The client uses the file
   size to decide if it should fill holes with zeros or return a short
   read.

What are holes?

How does the client return short reads? I.e., I expect a server to return short reads.

32) Redundant:

   The odm_mirror_cnt is used to replicate a file by replicating its
   component objects.  If there is no mirroring, then odm_mirror_cnt
   MUST be 0.  If odm_mirror_cnt is greater than zero, then the size of
   the olo_components array MUST be a multiple of (odm_mirror_cnt+1).

Already provided in Section 5.1.

33) Another (see Section 3.5) of use of RAID before definition:

   Note that mirroring can be defined over any RAID algorithm and
   striping pattern (either simple or nested).

34) 5.4 RAID Alogorithms.

   Note: The term "RAID" (Redundant Array of Independent
   Disks) is used in this document to represent an array of component
   objects that store data for an individual file.

Really needs to be defined much earlier. This is central to what you are trying to define.

35) IANA or a version bump?

   Thus, if one of these schemes is to be used in the future, a distinct
   value must be added to pnfs_obj_raid_algorithm4 for it.

36) What is (*)

   In this protocol, the OSD returns the capacity consumed by a
   write (*),

37) pnfs_obj_layoutupdate4 is used before defined

   Object-Based pNFS clients are not allowed to modify the layout.
   Therefore, the information passed in pnfs_obj_layoutupdate4 is used
   only to update the file's attributes.

38) Used before defined and no reference target:

   If the lora_layout_type layout type is LAYOUT4_OBJECTS_V2, then the
   lrf_body opaque value is defined by the pnfs_obj_layoutreturn4 type.

39) Which paragraph 4?

13.5.  Security Considerations over NFS

   When NFS is used for storage protocol, obviously the T10 security
   mechanism cannot be implemented.  Instead, the server uses the NFS
   owner and group identifiers, in combination with the RPC credentials
   provided with the layout (as described in Paragraph 4), to simulate

40) Yes, you need them to assign a value for LAYOUT4_OBJECTS_v2...

14.  IANA Considerations

   As described in NFSv4.1 [2], new layout type numbers have been
   assigned by IANA.  This document defines the protocol associated with
   the existing layout type number, LAYOUT4_OBJECTS_V2, and it requires
   no further actions for IANA.

This is a cut and paste error from RFC5664.

41) And another cut and paste error.

Appendix A.  Acknowledgments

   Todd Pisek was a co-editor of the initial versions of this document.
   Daniel E. Messinger, Pete Wyckoff, Mike Eisler, Sean P. Turner, Brian
   E. Carpenter, Jari Arkko, David Black, and Jason Glasgow reviewed and
   commented on this document.

42) Fencing

12.  Client Fencing

   In cases where clients are uncommunicative and their lease has
   expired or when clients fail to return recalled layouts within a
   lease period at the least (see "Recalling a Layout"[2]), the server
   MAY revoke client layouts and/or device address mappings and reassign
   these resources to other clients.  To avoid data corruption, the
   metadata server MUST fence off the revoked clients from the
   respective objects as described in Section 13.4.

Should be Sections 13.4 and 13.5.

43)

13.5.  Security Considerations over NFS

   When NFS is used for storage protocol, obviously the T10 security
   mechanism cannot be implemented.  Instead, the server uses the NFS
   owner and group identifiers, in combination with the RPC credentials
   provided with the layout (as described in Paragraph 4), to simulate
   the OSD CAPKEY security model.  The file's mode or ACL are set to
   provide the file's owner READ and WRITE permissions, to the file's
   group READ only permissions, and no permissions to others.
   Respecively, the client is provided with the respective credentials
   to provided READ/WRITE or READ only access for the respective layout
   lo_iomode.

   Fencing off a client over NFS is achieved by modifying the respective
   files' ownership attributes.  This will implictly revoke the
   outstanding credentials and will require the client to ask the server
   for new layouts.

Fencing in terms of pNFS occurs on the DS and not the MDS. I.e., the MDS has revoked the stateid/layout and access,
so how does the DS do this on compounds which have left the client before the revocation from the MDS
reach the client?

For file-based layouts, the control protocol is used for the MDS to inform the DS to fence.

So with this proposal here, if the client ignores the ownership attributes, then it can still write to the NFS
device. It has no means to prevent this - it has no knowledge of either the layout or stateids.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 09 Oct 2012 12:45:48 +0200

> http://datatracker.ietf.org/wg/nfsv4/

Sure.  No problem.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 14:24:46 +0200

> Editorial, "you" means the document. :->

Sure :)  Thanks for your thorough review!

> 
> Thanks,
> Tom
> 
> On Aug 31, 2012, at 8:17 AM, Benny Halevy <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> Folks,
>>

OK

> a great deal of this document is based off of that document, there should be a reference.

Will do.

> not OBSOLETED, my guess is not.

Correct.  The intent is to extend RFC5664 not to obsolete it.

> 2) Even with the stage set, what are the use cases?

The use cases go from clustering filers for HPC, e.g. scaling capacity and
bandwidth, through Big Data - e.g. exporting Hadoop clusters over pNFS,
to NAS virtualization.

> Why is the version 1 of the objects layout insufficient?

a. It cannot aggregate existing NFS filers.
b. The OSD protocol is not widely supported and its future development has practically stopped.

> Also, you haven't defined the term.

OK

>    (RFC1813 [14], RFC3530 [15], RFC5661 [2])

OK

> Pretty sure they are the same, but why are you using different terms?

I'm trying to distinguish a generic Object Storage protocol/model from the
specific T10-OSD model, which becomes one embodiment of the model.

> The way this is written, it sound like the pNFS server is used even in the non-NFS case.

Will clarify.

> BTW - which pNFS server? MDS or DS?

The operations are to the data server, which, if NFSv4.1 is in use, will be used
as a vanilla NFSv4.1 server with no pNFS requirements (neither MDS nor DS).

> With NFS filers used for object storage devices the object's owner,

OK

>    /// %#include <nfs4_prot.x>

RFC5662 seems to generate nfs4_prot.x.
What mismatch are you referring to?

> BTW - what are the requirements for NFS devices?

The MDS doesn't have to partition the DS and doing so is implementation specific
so there's no requirement from the storage device to support any particular standard.
(though SMI-S comes into mind)

>    OSD device, represented as oda_obj_type equal to PNFS_OBJ_OSD_V1 or

True.  Will fix.

> BTW - these are again used before defined. I would suggest that you rearrange your sections such that

OK. I'll look into this.

> the pNFS server present the stateid needed for NFSv4 and NFSv4.1 operations?

Oops, this is a leftover from an earlier internal draft.
OPEN needs to be supported, otherwise the client will have to revert to using
only special stateless stateids.

> pono_fhandle would be consistent.

OK.

> 12) Rogue period:

What do you think need changing?

> Why the '-' after 'layout'?

English, I guess (Ask Brent :)

> If this was C code, I'd say that the precedence rules were being violated.

:)

>    Some of the object-based layout- supported RAID algorithms encode

Looks good.

> And if you have to explain "missing", perhaps a better enum would be PNFS_OBJ_UNAVAILABLE.

I'd rather leave the RFC5664 terminology.

> What is it for a OSD v2 capability?

The capability data structure and enumerated values are specified in the T-10 OSD spec.

> derived from OSD, this just explicitly states it.

I tend to agree since the capability is opaque to the client which
does not have to be able to parse and verify it.

Brent, what do you think?

> on the name. But they are vastly different.

The former is typed ..._osd_objid4 and the latter ..._nfs_objid4

Am I missing anything?

> Point them to Section 3.3.

OK

> I would suggest sub-sections:

OK, that makes sense.

> compile is different from the flow needed to present.

Yup. A short introductory paragraph should help a human reader of the document.

> What is a pnfs_obj_deviceaddr4? See Section 4.2…

Thanks

> given a pnfs_obj_comp4, do I get the pnfs_obj_deviceaddr4?

The deviceid4 is embedded in both pnfs_obj_{nfs,osd}_objid4.
In theory it can be extracted into pnfs_obj_comp4 but it is needed
in the *objid4 structure where it is referenced elsewhere (pnfs_obj_ioerr4).

> What is a component here? Will I have an array of pnfs_obj_comp4s?

Yes, olo_components<> is an array.

> which paints the big picture.

OK.

>    as the "ooa_systemid" and "oda_osdname" fields.

OK

> 19) By the way, Section 4 provides the overview I am asking for in 17 above for this section…

I'll see if it makes sense to reverse the order.

> 20)  Get rid of comma aster "as a SCSI Name, or"

Sure

> 21) What is OBJ_TARGET_ANON used for?

As said at the bottom of 4.2.1:
   The OBJ_TARGET_ANON pnfs_osd_targetid_type4 MAY be used for providing
   no target identification.  In this case, only the OSD System ID, and
   optionally the provided network address, are used to locate the
   device.

I'll mention it briefly in 4.1 too.

> As per the other uses you present, this should be PNFS_OBJ_*.

No problem.  This was carried verbatim from RFC5664.

> Break the long lines up rather than changing the start column.

xml2rfc did this for me (us :)... I'll try breaking the long lines
and hope that the resulting .x will compile.

> 24) While NetApp appreciates the free publicity, what is a filer here? :->

Sure.  Will s/filer/Data Server/
> Are there any restrictions as to what version of NFS is to be used here?

Yes.  >= 3.  Needs to be specified as such...

>    deviceid until valid device information is acquired.

As discussed with Bruce Fields, this requirement will be changed to SHOULD.

> Why isn't this a fatal error to the application?

ona_path is visible only to the pNFS client, not to the application.

> How does the client inform the MDS that there is an issue?

It doesn't, it will not use pNFS for any layouts referring to this device.

> How does the control protocol from the MDS to the DS interact?

A MDS/DS control protocol is outside the scope of this document.

> a) What do I expect to see if I access this path? Data files? How are they named? How do I fsck this

There are going to be a bunch of objects which content is opaque to the user.
The internal organization of the Data Server namespace is implementation specific
so it is up to the (filesystem running on the) MDS.

> b) Am I allowed to let other clients access this path via protocols other than NFSv4.1?

Better not allow WRITE access.
READ access might be useful for data protection.

> c) What about with another layout?

NFSv4.1 allows that.

> Just mention you want to add LAYOUT_OBJECTS_V2

OK.

> By the way, why the change

Not T-10 OSD anymore.

> here.

OK

> And that section is wrong. You are asking for them to assign you a globally unique

True.

> You have left off the 0x for the first 3.

Oops, will fix.

> How is the change communicated from the server (btw, I assume the MDS is meant here) to the client?

If needed.  Since the layout is limited to the byte range, adding components may not necessarily
require layout recall if it does not effect the mapping of the bytes in the byte range
that was handed out.

> What happens if the client tries to write with a different view of the number of components?

Again, for the given byte range the layout is fine.
Adding components dynamically is applicable within the first stripe.
Once the layout wraps around the stripe (or group) width is set.

> Are you planning on  using byte ranges for the layout or is the file restriped when this addition of

Planning on using byte ranges.

> So if there is no nesting, then the size of the components array must be zero?

Duh. Will fix.

> If there is nesting, then what error is returned if the size of the components array is not a multiple of

returned by whom?
The server MUST provide a valid layout otherwise the client is free to ditch it and revert
to plain ol' NFSv4.1

>    the component array MUST be a multiple of (odm_mirror_cnt+1).

ditto

>    olo_components array only once.

ditto

> How about odm_num_comps in olo_map?

Sounds good.  Using unique prefix for structure members helps here.

> the array? I.e., you define the start point, but not the end point. They must be adjacent, etc.

It is the start point and yes, the components are adjacent in the array.

> 
> 
> What determines sufficient?

All bytes in the layout's byte range must map to the provided components.
or as it is said here: "sufficient to access any byte in the layout's data range"
It seems clear enough to me, apparently it is not :)

> And how is an error communicated back to the MDS if this is not sufficient?

No error, see above.

> Is this to make up for the fact that odm_num_comps can grow?

Partly, and part for returning sub-stripes.

>    size to decide if it should fill holes with zeros or return a short

The NFS client.

> What are holes?

Byte ranges which map to unallocated areas on the Data Servers.
I.e. they have no component object existing or to short objects where
the hole's byte range maps beyond the component object EOF (as explained
later in the paragraph)

   Striping patterns can cause cases where component objects are
   shorter than other components because a hole happens to correspond to
   the last part of the component object.

> How does the client return short reads? I.e., I expect a server to return short reads.

This is client implementation specific.  A NFS client implementation might
return a status representing fewer bytes than asked by the application when
issuing a posix compliant "read" call to the operating system.

> Already provided in Section 5.1.

OK. Will provide a XREF instead.

>    striping pattern (either simple or nested).

Seems like this also suggests a general introduction section
before diving into the XDR section.

> Really needs to be defined much earlier. This is central to what you are trying to define.

OK

>    value must be added to pnfs_obj_raid_algorithm4 for it.

Good catch.

>    In this protocol, the OSD returns the capacity consumed by a

Good question.  Carried from RFC5664, but I'm not sure what the original intent was.
Brent, can you help?

>    only to update the file's attributes. 

I assume the reader can and will use text search to find the definition
to forward usage cases.  Does RFC5661 follow the no-use-before-definition
rule by the way?

>    lrf_body opaque value is defined by the pnfs_obj_layoutreturn4 type.

Which exactly?

>    provided with the layout (as described in Paragraph 4), to simulate

Hmm, I use <xref target="RPC Credentials Usage" /> which
is defined as <t anchor="RPC Credentials Usage">.
I guess having a section for it would work better.

> This is a cut and paste error from RFC5664.

True.  Will fix.

>    commented on this document.

Yes. Already fixed in my repository.

> Should be Sections 13.4 and 13.5.

OK

> so how does the DS do this on compounds which have left the client before the revocation from the MDS

In this case fencing is done by manipulating the object (file on DS) ownership or ACL.

> So with this proposal here, if the client ignores the ownership attributes, then it can still write to the NFS

If the device will let it do that and ignores the RPC auth credentials the client provides.

> It has no means to prevent this - it has no knowledge of either the layout or stateids.

There is no intention to follow the file-based layout security model, but rather to propose
a simpler model that could use "loosely coupled" Data Server using no proprietary back-end
protocol at the basic level.  Such a back-end protocol can be developed as an enhancement.

Benny

---

From: "Welch, Brent" <[email-masked]>
Date: Thu, 11 Oct 2012 18:04:10 +0000

You have a few questions in-line for me below.  A couple are minor that we'll try to address in
our upcoming 5664bis draft we are trying to complete (like the use of hyphens and the (*).

The one substantial question you raise about exposing the format of the capability is a good
question.  The capability is really opaque to the client.  It is generated by the MDS and passed
along to the OSD, which verifies and enforces it.  Of course an "NFS Object" would have something
completely different in the capability.

I have one suggestion for Benny.  I think new layout type should be
LAYOUT4_NFS_OBJECTS rather than LAYOUT4_OBJECTS_V2" 
This captures your intent of using an NFS server as an object store,
and I don't want it to be construed as replacing LAYOUT4_OSD2_OBJECTS,
which is using T10 OSDv2 objects as the object store.

Yours would be a new RFC, not a replacement for 5664

Also, as you can tell from the mailing list discussion with Trond, Boaz, etc., the new RFC needs to have
a lot more background a motivation that sets it in context of the other layouts and RFCs.

 --
Brent

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, October 11, 2012 5:25 AM
To: Haynes, Tom
Cc: nfsv4 list; Welch, Brent
Subject: Re: [nfsv4] New Version Notification for draft-bhalevy-nfs-obj-00.txt

OK

Will do.

OK

OK

Will clarify.

> After devices...

OK

True.  Will fix.

OK. I'll look into this.

OK.

:)

Looks good.

Am I missing anything?

OK

OK, that makes sense.

> What is a pnfs_obj_deviceaddr4? See Section 4.2...

Thanks

OK.

OK

> 19) By the way, Section 4 provides the overview I am asking for in 17 above for this section...
> 

Sure

NFSv4.1 allows that.

OK.

Not T-10 OSD anymore.

OK

True.

Oops, will fix.

Duh. Will fix.

returned by whom?

ditto

ditto

No error, see above.

The NFS client.

OK

Good catch.

Which exactly?

True.  Will fix.

OK

Benny

---

From: "Haynes, Tom" <[email-masked]>
Date: Thu, 11 Oct 2012 18:58:23 +0000

> a lot more background a motivation that sets it in context of the other layouts and RFCs.

Sorry, old eyes.

>>   The second generation OSD protocol (SNIA T10/1729-D [16]). has

16]) has

> The former is typed ..._osd_objid4 and the latter ..._nfs_objid4

Elsewhere you use osd and nfs in the names,

so occ_osd_object_id and onc_nfs_object_id.

Nit I'm willing to let go. It just gets hard when reading the text.

>   optionally the provided network address, are used to locate the

My point would be that the description in 4.2.1 belongs here.

> A MDS/DS control protocol is outside the scope of this document.

Then this document does not describe a pNFS implementation as described by 5661. (see below)

> The server MUST provide a valid layout otherwise the client is free to ditch it and revert

Then just state that if this (or the below) is violated, the client MUST return the layout
and use NFSv4.1 with the MDS.

>>   size to decide if it should fill holes with zeros or return a short

I know what holes are, my point is that you are using it without definition.

> return a status representing fewer bytes than asked by the application when

Ahh, I would reword the sentence to indicate that this is something the
client layer is communicating with the application.

> to forward usage cases.  Does RFC5661 follow the no-use-before-definition

No clue, I wasn't involved. But I do get dinged on that with stuff I produce. :->

>>   lrf_body opaque value is defined by the pnfs_obj_layoutreturn4 type.

No clue, slept since then. Assume them all? :->

> protocol at the basic level.  Such a back-end protocol can be developed as an enhancement.

There are requirements that need to be met. 

12.2.6.  Control Protocol

   As noted in Figure 1, the control protocol is used by the exported
   file system between the metadata server and storage devices.
   Specification of such protocols is outside the scope of the NFSv4.1
   protocol.  Such control protocols would be used to control activities
   such as the allocation and deallocation of storage, the management of
   state required by the storage devices to perform client access
   control, and, depending on the storage protocol, the enforcement of
   authentication and authorization so that restrictions that would be
   enforced by the metadata server are also enforced by the storage
   device.

   A particular control protocol is not REQUIRED by NFSv4.1 but
   requirements are placed on the control protocol for maintaining
   attributes like modify time, the change attribute, and the end-of-
   file (EOF) position.  Note that if pNFS is layered over a clustered,
   parallel file system (e.g., PVFS [51]), the mechanisms that enable
   clustering and parallelism in that file system can be considered the
   control protocol.

Again, I don't read this as being strictly a requirement for the file-based
layout, but for all layout types.

I believe the proposal has to address the issues on maintaining the
attributes.

Also, once NFSv4.2 drops into place, how will this proposal handle
Labeled NFS?

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Oct 2012 19:07:42 +0000

>From Brent:
>>> Also, as you can tell from the mailing list discussion with Trond, Boaz, etc., the new RFC needs to have a lot more background a motivation that sets it in context of the other layouts and RFCs.

Just to be clear, at this point, the discussion is about a personal I-D not an RFC.  Benny has stated the desire to include this work in the working group charter.

For inclusion within the working group, there needs to be some form of momentum or clear interest within the WG.  As mentioned elsewhere what would help this are things like: clear statement of goals, a use case or two, and the reasons why this work needs to be taken on by the working group or why it is an interest to others.  Note that not all of these things need to be captured in an I-D.

Spencer

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Welch, Brent
> Sent: Thursday, October 11, 2012 11:04 AM
> To: Benny Halevy; Haynes, Tom
> Cc: nfsv4 list
> You have a few questions in-line for me below.  A couple are minor that we'll
> try to address in our upcoming 5664bis draft we are trying to complete (like the
> use of hyphens and the (*).
> 
> The one substantial question you raise about exposing the format of the
> capability is a good question.  The capability is really opaque to the client.  It is
> generated by the MDS and passed along to the OSD, which verifies and
> enforces it.  Of course an "NFS Object" would have something completely
> different in the capability.
> 
> This captures your intent of using an NFS server as an object store, and I don't
> want it to be construed as replacing LAYOUT4_OSD2_OBJECTS, which is using
> T10 OSDv2 objects as the object store.
> 
> Also, as you can tell from the mailing list discussion with Trond, Boaz, etc., the
> new RFC needs to have a lot more background a motivation that sets it in
> context of the other layouts and RFCs.
> 
>  --
> Brent
> 
> 
> >> I've submitted a proposed extension to the pnfs objects layout that
> >> introduces the use of NFS filers as object storage data servers.
> >>
> >> In contrast to the files layout, the object layout does not require a
> >> proprietary back-end protocol, hence the proposed layout allows one
> >> to mix and match a metadata server and data servers from different
> vendors.
> >> The metadata server controls the file ownership and permissions of
> >> the objects stored on data servers and the client is handed a
> >> corresponding RPC credential on LAYOUTGET a-la OSD capabilities.
> >> Outstanding credentials are unilaterally revoked by the MDS by modifying
> the objects user or group owner.
> >>
> >> Benny
> >>
> >
> > 1) You need to put the above into the Introduction. I.e., if you have
> > to set the stage in an email, then you also need to in the document.
> 
> OK
> 
> >
> > I also find it very strange that there is no mention of RFC 5664 in
> > this document. Considering a great deal of this document is based off of that
> document, there should be a reference.
> 
> Will do.
> 
> >
> > Is the intent that this obsoletes RFC 5664? Given that
> > LAYOUT4_OSD2_OBJECTS is not OBSOLETED, my guess is not.
> 
> bandwidth, through Big Data - e.g. exporting Hadoop clusters over pNFS, to NAS
> virtualization.
> 
> >
> > While such acts are outside the scope, is there a standard in place
> > which defines the requirement for such filesystems?
> >
> The MDS doesn't have to partition the DS and doing so is implementation
> specific so there's no requirement from the storage device to support any
> particular standard.
> > The implication here is that NFSv3, NFSv4, and NFSv4.1 filehandles are
> > valid to use in this proposed extension.
> >
> > then OPEN is not allowed. Since the "pNFS server" must perforce do
> > this for the client, then how does the pNFS server present the stateid needed
> for NFSv4 and NFSv4.1 operations?
> >
> >    struct READ4args {
> >            /* CURRENT_FH: file */
> >            stateid4        stateid;
> >            offset4         offset;
> >            count4          count;
> >    };
> 
> 
> > I'd ditch this altogether, but if you need to say anything, just
> > mention while it can be derived from OSD, this just explicitly states it.
> >
> 
> I tend to agree since the capability is opaque to the client which does not have
> to be able to parse and verify it.
> 
> > Both ooc_object_id and onc_object_id appear to be the same type of
> > object based on the name. But they are vastly different.
> 
> > You are hurt by embedding the XDR directly into the document. The flow
> > needed to compile is different from the flow needed to present.
> >
> 
> > Going back to my point about flow, I'm having a difficult time at this
> > point seeing how given a pnfs_obj_comp4, do I get the
> pnfs_obj_deviceaddr4?
> >
> 
> In theory it can be extracted into pnfs_obj_comp4 but it is needed in the
> *objid4 structure where it is referenced elsewhere (pnfs_obj_ioerr4).
> 
> > Yes, you have used component here without defining it. I think you
> > need a prior section which paints the big picture.
> 
> OK.
> 
> >
> > 18) Confusing:
> >
> >    These appear
> xml2rfc did this for me (us :)... I'll try breaking the long lines and hope that the
> resulting .x will compile.
> 
> >
> > a) What do I expect to see if I access this path? Data files? How are
> > they named? How do I fsck this path?
> 
> The internal organization of the Data Server namespace is implementation
> specific so it is up to the (filesystem running on the) MDS.
> 
> >
> > I see you have an IANA Considerations in Section 14. You need to make
> > a reference to it here.
> 
> OK
> 
> > And that section is wrong. You are asking for them to assign you a
> > globally unique value. As it is, your current value is private.
> 
> True.
> 
> >
> If needed.  Since the layout is limited to the byte range, adding components
> may not necessarily require layout recall if it does not effect the mapping of
> the bytes in the byte range that was handed out.
> 
> >
> > Are you planning on  using byte ranges for the layout or is the file
> > restriped when this addition of components occurs?
> 
> > If there is nesting, then what error is returned if the size of the
> > components array is not a multiple of the odm_group_width?
> 
> returned by whom?
> The server MUST provide a valid layout otherwise the client is free to ditch it
> and revert to plain ol' NFSv4.1
> 
> >
> > Also, what error for:
> >
> I.e. they have no component object existing or to short objects where the hole's
> byte range maps beyond the component object EOF (as explained later in the
> paragraph)
> 
> Seems like this also suggests a general introduction section before diving into
> the XDR section.
> 
> >
> >
> I assume the reader can and will use text search to find the definition to
> forward usage cases.  Does RFC5661 follow the no-use-before-definition rule by
> the way?
> 
> >
> >
> Hmm, I use <xref target="RPC Credentials Usage" /> which is defined as <t
> anchor="RPC Credentials Usage">.
> > Fencing in terms of pNFS occurs on the DS and not the MDS. I.e., the
> > MDS has revoked the stateid/layout and access, so how does the DS do
> > this on compounds which have left the client before the revocation from the
> MDS reach the client?
> 
> > So with this proposal here, if the client ignores the ownership
> > attributes, then it can still write to the NFS device.
> 
> There is no intention to follow the file-based layout security model, but rather
> to propose a simpler model that could use "loosely coupled" Data Server using
> no proprietary back-end protocol at the basic level.  Such a back-end protocol
> can be developed as an enhancement.
> 
> Benny
> 
> >
> >

---

From: "faibish, sorin" <[email-masked]>
Date: Thu, 11 Oct 2012 15:19:24 -0400

And BTW same will be for my personal draft for pNFS for Lustre. I assume I will get 15 min as well. Thanks

/Sorin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Spencer Shepler
Sent: Thursday, October 11, 2012 3:08 PM
To: Welch, Brent; Benny Halevy; Haynes, Tom
Cc: nfsv4 list
Subject: Re: [nfsv4] New Version Notification for draft-bhalevy-nfs-obj-00.txt

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Oct 2012 20:13:10 +0000

Your item is on the agenda.

> From: faibish, sorin [mailto:[email-masked]]
> Sent: Thursday, October 11, 2012 12:19 PM
> To: Spencer Shepler; Welch, Brent; Benny Halevy; Haynes, Tom
> Cc: nfsv4 list
> Subject: RE: [nfsv4] New Version Notification for draft-bhalevy-nfs-obj-00.txt
> 
> >>> Also, as you can tell from the mailing list discussion with Trond, Boaz,
> etc., the new RFC needs to have a lot more background a motivation that
> sets it in context of the other layouts and RFCs.
> 
> 
> Just to be clear, at this point, the discussion is about a personal I-D not an
> RFC.  Benny has stated the desire to include this work in the working group
> charter.
> 
> For inclusion within the working group, there needs to be some form of
> momentum or clear interest within the WG.  As mentioned elsewhere what
> would help this are things like: clear statement of goals, a use case or two,
> and the reasons why this work needs to be taken on by the working group or
> why it is an interest to others.  Note that not all of these things need to be
> captured in an I-D.
> 
> Spencer
> 
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Welch, Brent
> > You have a few questions in-line for me below.  A couple are minor
> > that we'll try to address in our upcoming 5664bis draft we are trying
> > to complete (like the use of hyphens and the (*).
> >
> > The one substantial question you raise about exposing the format of
> > the capability is a good question.  The capability is really opaque to
> > the client.  It is generated by the MDS and passed along to the OSD,
> > which verifies and enforces it.  Of course an "NFS Object" would have
> > something completely different in the capability.
> >
> > Also, as you can tell from the mailing list discussion with Trond,
> > Boaz, etc., the new RFC needs to have a lot more background a
> > motivation that sets it in context of the other layouts and RFCs.
> >
> >  --
> > Brent
> >
> >
> > >> 1. The desire to use an ubiquitous, standard protocol to access the
> > >> data
> > servers.
> > >> In contrast to the files layout, the object layout does not require
> > >> a proprietary back-end protocol, hence the proposed layout allows
> > >> one to mix and match a metadata server and data servers from
> > >> different
> > vendors.
> > > 1) You need to put the above into the Introduction. I.e., if you
> > > have to set the stage in an email, then you also need to in the document.
> >
> > OK
> >
> > >
> > The use cases go from clustering filers for HPC, e.g. scaling capacity
> > and bandwidth, through Big Data - e.g. exporting Hadoop clusters over
> > pNFS, to NAS virtualization.
> >
> > >
> > I'm trying to distinguish a generic Object Storage protocol/model from
> > the specific T10-OSD model, which becomes one embodiment of the
> model.
> >
> > >
> > > 6)
> > >
> > > The way this is written, it sound like the pNFS server is used even
> > > in the non-
> > NFS case.
> >
> > Will clarify.
> >
> > >
> > The operations are to the data server, which, if NFSv4.1 is in use,
> > will be used as a vanilla NFSv4.1 server with no pNFS requirements (neither
> MDS nor DS).
> >
> > >
> > > 7) Need a comma:
> > >
> > The MDS doesn't have to partition the DS and doing so is
> > implementation specific so there's no requirement from the storage
> > device to support any particular standard.
> > > The implication here is that NFSv3, NFSv4, and NFSv4.1 filehandles
> > > are valid to use in this proposed extension.
> > >
> > OPEN needs to be supported, otherwise the client will have to revert
> > to using only special stateless stateids.
> >
> > >
> > >
> > >
> > > 11)  Field names do not follow the NFSv4.1 standard of being derived
> > > from
> > the structure name:
> > >
> > > You have Object and NFS covered, but no coverage of what is to go
> > > into
> > oc_missing_obj_id.
> > >
> > > I'd also split the chunk of XDR at the start of 3.4 into these
> > > subsections as
> > needed.
> > >
> > > You are hurt by embedding the XDR directly into the document. The
> > > flow needed to compile is different from the flow needed to present.
> > >
> >
> > > Going back to my point about flow, I'm having a difficult time at
> > > this point seeing how given a pnfs_obj_comp4, do I get the
> > pnfs_obj_deviceaddr4?
> > >
> >
> > > 19) By the way, Section 4 provides the overview I am asking for in
> > > 17 above
> > for this section...
> > >
> >
> > xml2rfc did this for me (us :)... I'll try breaking the long lines and
> > hope that the resulting .x will compile.
> >
> > >
> > > a) What do I expect to see if I access this path? Data files? How
> > > are they named? How do I fsck this path?
> >
> > The internal organization of the Data Server namespace is
> > implementation specific so it is up to the (filesystem running on the) MDS.
> >
> > >
> > > b) Am I allowed to let other clients access this path via protocols
> > > other than
> > NFSv4.1?
> > >
> >
> > > I see you have an IANA Considerations in Section 14. You need to
> > > make a reference to it here.
> >
> > OK
> >
> > > This is a radical departure from pNFS, which assumes that a layout
> > > has a fixed
> > number of components.
> > > How is the change communicated from the server (btw, I assume the
> > > MDS is
> > meant here) to the client?
> > > Is the layout recalled?
> >
> > If needed.  Since the layout is limited to the byte range, adding
> > components may not necessarily require layout recall if it does not
> > effect the mapping of the bytes in the byte range that was handed out.
> >
> > >
> > > What happens if the client tries to write with a different view of
> > > the number
> > of components?
> > >
> >
> > The server MUST provide a valid layout otherwise the client is free to
> > ditch it and revert to plain ol' NFSv4.1
> >
> > >
> > > Also, what error for:
> > >
> > > the array? I.e., you define the start point, but not the end point.
> > > They must be
> > adjacent, etc.
> >
> > the hole's byte range maps beyond the component object EOF (as
> > explained later in the
> > paragraph)
> >
> > > How does the client return short reads? I.e., I expect a server to
> > > return short
> > reads.
> >
> > This is client implementation specific.  A NFS client implementation
> > might return a status representing fewer bytes than asked by the
> > application when issuing a posix compliant "read" call to the operating
> system.
> >
> > >
> > > 32) Redundant:
> > >
> > Seems like this also suggests a general introduction section before
> > diving into the XDR section.
> >
> > >
> > >
> > > Really needs to be defined much earlier. This is central to what you
> > > are trying
> > to define.
> >
> > OK
> >
> > >
> > I assume the reader can and will use text search to find the
> > definition to forward usage cases.  Does RFC5661 follow the
> > no-use-before-definition rule by the way?
> >
> > >
> > >
> > Hmm, I use <xref target="RPC Credentials Usage" /> which is defined as
> > <t anchor="RPC Credentials Usage">.
> > > For file-based layouts, the control protocol is used for the MDS to
> > > inform the
> > DS to fence.
> > >
> > > It has no means to prevent this - it has no knowledge of either the
> > > layout or
> > stateids.
> >
> > There is no intention to follow the file-based layout security model,
> > but rather to propose a simpler model that could use "loosely coupled"
> > Data Server using no proprietary back-end protocol at the basic level.
> > Such a back-end protocol can be developed as an enhancement.
> >
> > Benny
> >
> > >
> > >

---

From: Benny Halevy <[email-masked]>
Date: Sun, 14 Oct 2012 13:22:23 +0200

On 2012-10-11 20:04, Welch, Brent wrote:
> along to the OSD, which verifies and enforces it.

Therefore I'd be reluctant to require the client to verify its version.

> Of course an "NFS Object" would have something

We propose a discriminated union that would provide a T-10 OSD capability only
for OSD objects, so not to reuse the present structure members in a hackish way.
The MDS provides a RPC auth structure instead for NFS objects.

> which is using T10 OSDv2 objects as the object store.

No problem.

> Yours would be a new RFC, not a replacement for 5664

A new independent RFC for sure, and not obsoleting RFC5664, especially
since I propose wire format changes.

True.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Sun, 14 Oct 2012 14:30:37 +0200

On 2012-10-11 20:58, Haynes, Tom wrote:

<snip>

>> What mismatch are you referring to?

:-)

<snip>

>>>   The second generation OSD protocol (SNIA T10/1729-D [16]). has

Ah, good point.

Thanks for clarifying, I was thinking period as in interval of time
and it didn't make much sense :)

<snip>

> Nit I'm willing to let go. It just gets hard when reading the text.

No no, that sounds helpful and I'm all for improving readability.

<snip>

> Then this document does not describe a pNFS implementation as described by 5661. (see below)

I disagree.
(discussion below)

<snip>

OK

<snip>

> I know what holes are, my point is that you are using it without definition.

I presumed it's common knowledge in the storage community but I can take a stab
at defining holes more clearly.

<snip>

No problem.

>> issuing a posix compliant "read" call to the operating system.

<snip>

> No clue, slept since then. Assume them all? :->

All names can be looked up, some are indeed used before definition.

<snip>

> There are requirements that need to be met. 

In our case, we essentially propose the NFS protocol as a control protocol
and this point apparently needs to be clarified.

>    such as the allocation and deallocation of storage,

CREATE/OPEN, REMOVE

> the management of
>    control

SETATTR + out of band manipulation of NFS exports properties.

> , and, depending on the storage protocol, the enforcement of

relying on NFS for that.

>    file (EOF) position.

As previously discussed and sort of captured in Errata 2505 for RFC5661,
that can achieved without a particular control protocol by relying on LAYOUTCOMMIT.

> Note that if pNFS is layered over a clustered,

I agree, though the block layout has no control protocol and is still considered valid.
The caveats (and there caveats in the blocks case) are documented there.

OK

> Also, once NFSv4.2 drops into place, how will this proposal handle

It is optional for the server to support labeled NFS
draft-ietf-nfsv4-minorversion2-15 puts the burden on the client:
 7.3.2. Permission Checking:
   NFSv4 should defer permission checking on this attribute to the host
   system.  These checks are performed in addition to existing DAC and
   ACL checks outlined in the NFSv4 protocol

 7.3.5. Label Changes:
   Consider a system in which the clients enforce MAC checks and and the
   server has a very simple security system which just stores the
   labels.  In this system, the MAC label check always allows access,
   regardless of the subject label.

 7.4. pNFS Considerations:
 7.4.1. MAC Label Checks
   The new FATTR4_SEC_LABEL attribute is metadata information and as
   such the DS is not aware of the value contained on the MDS.
   Fortunately, the NFSv4.1 protocol [2] already has provisions for
   doing access level checks from the DS to the MDS.  In order for the
   DS to validate the subject label presented by the client, it SHOULD
   utilize this mechanism.

This ability is specific to the files layout and to the proprietary
control protocol.  What can be said in our case is that when there is
such a protocol in place, it SHOULD be used to enforce MAC labels checks.

Benny

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 15 Oct 2012 18:01:53 +0000

> at defining holes more clearly.

I think the biggest disconnect for me was that the text shifted to the client
interacting with the application without an introduction. It was not clear
who was generating the holes and who they were being sent to.

Anyway, you should point out that this behavior is specific to the OSD
data servers as holes cannot be sent along earlier versions of NFS.

<snip>

> SETATTR + out of band manipulation of NFS exports properties.

And depending on the implementation of caching of the exports,
this can impose performance penalties on the other clients.

The larger problem with this is that the access control needed are
 per file, not per filesystem. And unless the NFS server supports
per file exports, this approach fails to meet requirements.

<snip>

> It is optional for the server to support labeled NFS

For NFSv4.2 it is optional, it could be made mandatory for NFSv4.3.

> such a protocol in place, it SHOULD be used to enforce MAC labels checks.

Why is it specific to the files layout?

This is the kernel of my contention with the way pNFS as described by
5661 works with different layout models. 5661 has requirements placed
outside of the layout model - the requirement for a Control Protocol is
one such.

I've been assuming all along that the blocks and object based layouts
took care of this issue by the storage devices being part of an array 
or cluster. And they communicated to solve these issues.

If they do not, then I do not consider them to be compliant to 5661.