Subject: [nfsv4] Seperation of generic pNFS and layout encoding/decoding

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Thu, 17 Nov 2005 15:42:24 -0500

hi brent, garth

as we investigate the construction of the pNFS client on linux, we are driven 
to implement a clear separation between the generic and layout specific 
portions of pNFS, with an API to join them. i think that part of this 
separation is the encoding/decoding of layout type specific data: e.g. device 
lists and layout data.

with this in mind, i'm curious as to why there are union declarations that 
switch(layout type) in the pNFS XDR. switching on layout type with the case 
LAYOUT_NFSV4_FILES as one case while all other layout types will be opaque<>. 
the opaque data for non LAYOUT_NFSV4_FILES will be encoded somehow. are new 
layout types required to add a LAYOUT_XXX type to all switch(layout type) xdr 
declarations? or are we to implement one way of processing layouts for 
LAYOUT_NFSV4_FILES (xdr then hand off to layout driver) and non 
LAYOUT_NFSV4_FILES (hand off opaque data to layout driver).

it would simplify the API between the generic and layout-specific pNFS to have 
one method. we propose that instead of using a union with a switch on layout 
type, that all layout data is opaque, with each layout type encoding the 
layout data inside the opaque data.

what i am suggesting can be viewed as architecturally similar to RPCSEC_GSS; 
with the RPC (a generic portion), multiple security mechanisms (the layout 
specific portion), and GSSAPI, (the API to join them). the GSSAPI 
initialContextTokens have a generic portion which includes a mechanism OID, 
and an internal token that is opaque to the GSSAPI, and encoded per mechanism. 
kerberos has it's own encoding, SPKM uses ASN.1, whatever.

anyway, i've attached suggested changes, each with a CURRENT and PROPOSED 
section, for the three places where the union and switch(layout type) is used.

6.3 pnfs_deviceaddr4 (read the NOTE in the PROPOSED section)
6.5 pnfs_layout
6.6 pnfs_layoutupdate4 (LAYOUTCOMMIT)

-->Andy
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Thu, 17 Nov 2005 13:17:37 -0800

I'm open to other ways of representing these structures.  My original 
hope was that the BLOCKS and OBJECT definitions would also be cases 
within those unions.  However, this probably depends upon their timeline 
for spec completion.

A change I might suggest to your structures is that the layoutdata in 
LAYOUTCOMMIT (for layoutupdate) should be renamed to reflect that it 
holds data which may be different than the layout (e.g., for FILES it 
holds nothing), thus I would prefer the name is different.  The same 
goes for the structure used by the LAYOUTHINT attribute.

-Garth

William A.(Andy) Adamson wrote:
> ------------------------------------------------------------------------
> 
> CURRENT:
> 
> 6.3  pnfs_deviceaddr4
> 
>      union pnfs_deviceaddr4 switch (pnfs_layouttype4 layout_type) {
>             case LAYOUT_NFSV4_FILES:
>                    string           r_netid<>;   /* network ID */
>                    string           r_addr<>;    /* universal address */
>             default:
>                    opaque           device_addr<>; /* Other layouts */
>      };
> 
>    The device address is used to set up a communication channel with the
>    storage device.  Different layout types will require different types
>    of structures to define how they communicate with storage devices.
>    The union is switched on the layout type.  Currently, the only device
>    address defined is that for the NFSv4 file layout, which identifies a
>    storage device by network IP address and port number.  This is
>    sufficient for the clients to communicate with the NFSv4 storage
>    devices, and may also be sufficient for object-based storage drivers
>    to communicate with OSDs.
> 
>    [NOTE: other device address types will be added as the respective
>    specifications mature.  It has been suggested that a separate
>    device_type is used as a switch to the pnfs_deviceaddr4 structure
> 
> 
> 
> Welch, et al.            Expires March 19, 2006                [Page 39]
> ^L
> Internet-Draft               pNFS Operations              September 2005
> 
> 
>    (e.g., if multiple types of addresses exist for the same layout
>    type).  Until such a time as a real case is made and the respective
>    layout types have matured, the device address structure will be left
>    as is.]
> 
> 6.4  pnfs_devlist_item4
> 
>      struct pnfs_devlist_item4 {
>             pnfs_deviceid4          id;
>             pnfs_deviceaddr4        addr;
>      };
> 
>    An array of these values is returned by the GETDEVICELIST operation.
>    They define the set of devices associated with a file system.
> 
> 
> PROPOSED:
> 
> 6.4  pnfs_devlist_item4
> 
>      struct pnfs_devlist_item4 {
>             pnfs_deviceid4          id;
> 	    pnfs_layoutttype4       layout_type;
>             opaque		    device_addr<>;
>      };
> 
>    An array of these values is returned by the GETDEVICELIST operation.
>    They define the set of devices associated with a file system.
>    The device_addr is used to set up a communication channel with the
>    storage device and is specific to the layout_type.  Currently, the only 
>    device_addr defined is that for the NFSv4 file layout, which identifies a
>    storage device by network IP address and port number. see Section 5.x 
>    for its definition. 
> 
>    [NOTE: other device address types will be added as the respective
>    specifications mature. Since the device_addr is opaque with each
>    layout_type defining the device_addr interal structure, features such 
>    as multiple types of addresses for the same layout type can be
>    expressed without any changes to this structure.]
> 
> 5.x pnfs_file_address4
> 
> 	struct pnfs_file_address4 {
>                    string           r_netid<>;   /* network ID */
>                    string           r_addr<>;    /* universal address */
> 	}
> 
>    The pnfs_file_address4 is used by the the NFSv4 file layout, which 
>    identifies a storage device by network IP address and port number.
>    This is sufficient for the clients to communicate with the NFSv4 storage
>    devices, and may also be sufficient for object-based storage drivers
>    to communicate with OSDs.
> 
> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> 
> CURRENT:
> 
> 6.5  pnfs_layout4
> 
>      union pnfs_layouttypes4 switch (pnfs_layouttype4 layout_type) {
>             case LAYOUT_NFSV4_FILES:
>                    nfsv4_file_layouttype4 file_layout;
>             default:
>                    opaque           layout_data<>;
>      };
> 
>      struct pnfs_layout4 {
>             offset4                 offset;
>             length4                 length;
>             pnfs_layoutiomode4      iomode;
>             pnfs_layouttypes4       layout;
>      };
> 
> 
>    The pnfs_layout4 structure defines a layout for a file.  The
>    pnfs_layouttypes4 union contains the portion of the layout specific
>    to the layout type.  Currently, only the NFSv4 file layout type is
>    defined; see Section 5.1 for its definition.  Since layouts are sub-
>    dividable, the offset and length (together with the file's
>    filehandle, the clientid, iomode, and layout type), identifies the
>    layout.
> 
>    [OPEN ISSUE: it has been suggested that the layout type include a
>    generic striping layer, as defined in Section 5.1.  This has not yet
>    been done.]
>                                                               2262,6        64%
> PROPOSED: 
> 
> 6.x  pnfs_layoutdata4
> 
>      struct pnfs_layoutdata4 {
>              pnfs_layouttype4 layout_type;
>              opaque           layout_data<>;
>      };
>    
>    The pnfs_layoutdata4 structure contains the portion of the layout
>    specific to the layout type.  Currently, only the NFSv4 file layout 
>    type is defined; see Section 5.1 for its definition.  
> 
> 6.5  pnfs_layout4
> 
>      struct pnfs_layout4 {
>             offset4                 offset;
>             length4                 length;
>             pnfs_layoutiomode4      iomode;
> 	    pnfs_layoutdata4	    layout;
>      };      
> 
>    The pnfs_layout4 structure defines a layout for a file.
>    Since layouts are subdividable, the offset 
>    and length (together with the file's filehandle, the clientid, 
>    iomode, and layout type), identifies the layout.
> 
>    [OPEN ISSUE: it has been suggested that the layout type include a
>    generic striping layer, as defined in Section 5.1.  This has not yet
>    been done.]
> 
> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
> 
> CURRENT:
> 
> 6.6  pnfs_layoutupdate4
> 
>      union pnfs_layoutupdate4 switch (pnfs_layouttype4 layout_type) {
>             case LAYOUT_NFSV4_FILES:
>                    void;
>             default:
>                    opaque           layout_data<>;
>      };
> 
>    The pnfs_layoutupdate4 structure is used by the client to return
>    'updated' layout information to the metadata server at LAYOUTCOMMIT
>    time.  This provides a channel to pass layout type specific
>    information back to the metadata server.  E.g., for block layout
>    types this could include the list of reserved blocks that were
>    written.  The contents of the structure are determined by the layout
>    type and are defined in their context.
> 
> 		........
> 
> 9.2  LAYOUTCOMMIT - Commit writes made using a layout
> 
>    SYNOPSIS
> 
>      (cfh), clientid, offset, length, lastbytewritten,
>      time_modify, time_access, layoutupdate -> newsize
> 
> 
>    ARGUMENT
> 
>      union newtime4 switch (bool timechanged) {
>              case TRUE:
>                      nfstime4           time;
>              case FALSE:
>                      void;
>      };
> 
>      union newsize4 switch (bool sizechanged) {
>              case TRUE:
>                      length4            size;
>              case FALSE:
>                      void;
>      };
> 
>      struct LAYOUTCOMMIT4args {
>              /* CURRENT_FH: file */
>              clientid4               clientid;
>              offset4                 offset;
>              length4                 length;
>              length4                 lastbytewritten;
>              newtime4                time_modify;
>              newtime4                time_access;
>              pnfs_layoutupdate4      layoutupdate;
>      };
> 
>                  ...........
> 
>    The "layoutupdate" argument to LAYOUTCOMMIT provides a mechanism for
>    a client to provide layout specific updates to the metadata server.
>    For example, the layout update can describe what regions of the
>    original layout have been used and what regions can be deallocated.
>    There is no NFSv4 file layout specific layoutupdate structure.
> 
>                  ...........
> 
> PROPOSED
> 
> 9.2  LAYOUTCOMMIT - Commit writes made using a layout
> 
>    SYNOPSIS
> 
>      (cfh), clientid, offset, length, lastbytewritten,
>      time_modify, time_access, layoutupdate -> newsize
> 
> 
>    ARGUMENT
> 
>      union newtime4 switch (bool timechanged) {
>              case TRUE:
>                      nfstime4           time;
>              case FALSE:
>                      void;
>      };
> 
>      union newsize4 switch (bool sizechanged) {
>              case TRUE:
>                      length4            size;
>              case FALSE:
>                      void;
>      };
> 
>      struct LAYOUTCOMMIT4args {
>              /* CURRENT_FH: file */
>              clientid4               clientid;
>              offset4                 offset;
>              length4                 length;
>              length4                 lastbytewritten;
>              newtime4                time_modify;
>              newtime4                time_access;
>              pnfs_layoutdata4        layoutupdate;
>      };
> 
>                  ...........
> 
>    The "layoutupdate" argument to LAYOUTCOMMIT provides a mechanism for
>    a client to provide layout specific updates to the metadata server.
>    For example, the layout update can describe what regions of the
>    original layout have been used and what regions can be deallocated.
>    The contents of the structure are determined by the layout
>    type and are defined in their context.
> 
>                  ...........

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 17 Nov 2005 16:30:44 -0500

> within those unions.  However, this probably depends upon their timeline 

Adding them to that union removes the 4-byte length field that would
otherwise precede their contents.  So if you do that whenever a new
layout type is standardized, you change the xdr in an incompatible way
each time.

It makes more sense just be consistent and declare them all to be
opaques.  As Andy says, this is exactly what the RPC RFC does, for
example.  It declares the credential and verifier to be opaque types and
also gives xdr definitions for their contents in a few cases.

--b.

---

From: Brent Welch <[email-masked]>
Date: Thu, 17 Nov 2005 13:40:10 -0800

I spoke with Andy about this, and I support the opaque approach.
There will be more than just files, blocks, and objects in the
long run, as folks introduce experimental back-ends.  For example,
the very first PNFS system is backed by PVFS - a user-level
distributed file system for clusters.

I think the draft standard should still have the XDR types for
the files back-end, and we'll supply the XDR types for the
objects back-end as well.  But it does feel cleaner to me that
there is just an opaque that is passed through the client API.

It turns out the Solaris folks are also thinking about the
internal client API between the generic NFSv4 client and the
layout drivers, and they are also working on files and objects.
So, whatever model we work out should be similar between the
two code bases, to the degree practical.

>>>Garth Goodson said:
 > > as we investigate the construction of the pNFS client on linux, we are 
dri
     ven 
 > > separation is the encoding/decoding of layout type specific data: e.g. 
dev
     ice 
      
 > > switch(layout type) in the pNFS XDR. switching on layout type with the 
cas
     e 
 > > LAYOUT_NFSV4_FILES as one case while all other layout types will be 
opaque
     <>. 
 > > the opaque data for non LAYOUT_NFSV4_FILES will be encoded somehow. 
are ne
     w 
 > > layout types required to add a LAYOUT_XXX type to all switch(layout 
type)
     xdr 
 > > it would simplify the API between the generic and layout-specific pNFS 
to
     have 
 > > one method. we propose that instead of using a union with a switch on 
layo
     ut 
 > > what i am suggesting can be viewed as architecturally similar to 
RPCSEC_GS
     S; 
      
 > > initialContextTokens have a generic portion which includes a mechanism 
OID
     , 
 > > and an internal token that is opaque to the GSSAPI, and encoded per 
mechan
     ism. 
 > > section, for the three places where the union and switch(layout type) 
is u
     sed.
      
 > >    device_addr defined is that for the NFSv4 file layout, which 
identifies
      a
 > >    This is sufficient for the clients to communicate with the NFSv4 
storag
     e
 > >                                                               2262,6   

       64%

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Garth Goodson <[email-masked]>
Date: Tue, 22 Nov 2005 15:30:12 -0800

Here are my proposed changes based on Andy's suggestion.  Some 
differences include: Keeping a separate pnfs_deviceaddr4 structure as it 
is used in both GETDEVICELIST and GETDEVICEINFO; Merging the layout 
structure with the opaque portion; Keeping separate layout_update and 
layout_hint structures (for cleanliness).

If there are no disagreements I will send to Spencer for inclusion.

-Garth

Proposed changes:
-----------------

6.3  pnfs_deviceaddr4

      struct pnfs_netaddr4 {

      struct pnfs_deviceaddr4 {
               pnfs_layouttype4 type;
               opaque           device_addr<>;
      };

    The device address is used to set up a communication channel with the
    storage device.  Different layout types will require different types
    of structures to define how they communicate with storage devices.
    The opaque device_addr field must be interpreted based on the
    specified layout type.

...

6.5  pnfs_layout4

      struct pnfs_layout4 {
             offset4                 offset;
             length4                 length;
             pnfs_layouttype4        type;
             opaque                  layout<>;
      };

    The pnfs_layout4 structure defines a layout for a file.  The layout
    type specific data is opaque within this structure and must be
    interepreted based on the layout type.  Currently, only the NFSv4
    file layout type is defined; see Section 5.1 for its definition.
    Since layouts are sub-dividable, the offset and length together with
    the file's filehandle, the clientid, iomode, and layout type,
    identifies the layout.

...

6.6  pnfs_layoutupdate4

      struct pnfs_layoutupdate4 {
             pnfs_layouttype4        type;
             opaque                  layoutupdate_data<>;
      };

    The pnfs_layoutupdate4 structure is used by the client to return
    'updated' layout information to the metadata server at LAYOUTCOMMIT
    time.  This structure provides a channel to pass layout type specific
    information back to the metadata server.  E.g., for block/volume
    layout types this could include the list of reserved blocks that were
    written.  The contents of the opaque layoutupdate_data argument are
    determined by the layout type and are defined in their context.  The
    NFSv4 file-based layout does not use this structure, thus the
    update_data field should have a zero length.

6.7  pnfs_layouthint4

      struct pnfs_layouthint4 {
             pnfs_layouttype4      type;
             opaque                layouthint_data<>;
      };

    The pnfs_layouthint4 structure is used by the client to pass in a
    hint about the type of layout it would like created for a particular
    file.  It is the structure specified by the FILE_LAYOUT_HINT
    attribute described below.  The metadata server may ignore the hint,
    or may selectively ignore fields within the hint.  This hint should
    be provided at create time as part of the initial attributes within
    OPEN.  The NFSv4 file-based layout uses the "nfsv4_file_layouthint"
    structure as defined in Section 5.1.