Subject: Re: [nfsv4] Agenda items for the Paris meeting

From: Spencer Shepler <[email-masked]>
Date: Tue, 20 Mar 2012 01:24:26 +0000

Yes, we are meeting.  Agenda query was long enough ago that we have collectively forgotten. :-)

Any others I need to know about; will be posting draft agenda in couple of days...

Spencer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, March 19, 2012 6:15 PM
To: Spencer Shepler
Cc: [email-masked]
Subject: Agenda items for the Paris meeting

Spencer,

I don't think I saw any call for the agenda items for IETF 83 in Paris, France. Are we going to have a meeting? Is there anybody attending?

I have 1 agenda item: New pNFS layout for Lustre.
I would also like to discuss the LNFS status of the draft in 4.2 if there are any interested parties attending.

Thank you for your patience and understanding

/Sorin

---

From: <[email-masked]>
Date: Mon, 19 Mar 2012 21:14:40 -0400

Spencer,

/Sorin

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 20 Mar 2012 16:18:32 +0000

On 3/19/12 8:14 PM, "[email-masked]" <[email-masked]> wrote:

>Spencer,
>
>I don't think I saw any call for the agenda items for IETF 83 in Paris,
>France. Are we going to have a meeting? Is there anybody attending?
>
>_______________________________________________
>nfsv4 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/nfsv4

Sorin,

For the LNFS, I've asked the WG chairs to put the requirements draft into
Last Call.

For the pNFS layout for Lustre, is there going to be a delta from the
presentation you gave at ConnectAThon 2012?
(which can be found here:
http://nfsv4bat.org/Documents/ConnectAThon/2012/pNFS4Lustre-cthon-2012.pdf)

I'd prefer to see discussion first on the mailing list and an executive
summary as a presentation.

I.e., I don't think we make good use of our limited meeting time when we
introduce new material and slides that people have to read. If we have
something new, we should be prepping the attendees ahead of time.

Thanks,
Tom

---

From: Chuck Lever <[email-masked]>
Date: Tue, 20 Mar 2012 14:00:16 -0400

Hi-

> Any others I need to know about; will be posting draft agenda in couple of days...

The FedFS DNS SRV draft appears to be coming along nicely.  Will there be an update on the FedFS NSDB draft and the NSDB schema changes due with the next version?  Any new editions of the FedFS ADMIN draft expected?

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: <[email-masked]>
Date: Tue, 20 Mar 2012 15:37:35 -0400

-----Original Message-----
From: Haynes, Tom [mailto:[email-masked]] 
Sent: Tuesday, March 20, 2012 12:19 PM
To: faibish, sorin; Spencer Shepler
Cc: nfsv4
Subject: Re: [nfsv4] Agenda items for the Paris meeting

Sorin,

[SF] So far my way of introducing a new draft was to do a presentation to the IETF and after that to start a discussion if the committee agrees that it is interested. I did it so far for at least 4 drafts some accepted some rejected. You may want to say this is not right; you may be right but it worked for me so far and I will continue this. My intention is to present the new layout idea and possible some pseudo-code. And no the attached presentation is not what I will present. I am working on it right now with my team in China and I will send the slides before the end of the week. I only need 15 min to present it. Yet the slides are good to introduce the problem if the group wants to understand what I will propose.

Thanks,
Tom

---

From: David Quigley <[email-masked]>
Date: Wed, 21 Mar 2012 06:05:23 -0400

On 03/19/2012 21:14, [email-masked] wrote:
> Spencer,
>
> I don't think I saw any call for the agenda items for IETF 83 in
> Paris, France. Are we going to have a meeting? Is there anybody
> attending?
>
> I would also like to discuss the LNFS status of the draft in 4.2 if
> there are any interested parties attending.
>

Unfortunately I will not be able to make it to Paris however I should 
be able to hop on the audio stream and in the chatroom to participate 
remotely.

Dave

---

From: <[email-masked]>
Date: Mon, 26 Mar 2012 13:58:46 -0400

In preparation for the discussion on a new layout for Lustre I want to bring some pertinent discussion points to the nfsv4 list.

(i) Lustre layout is similar yet different from 5664:

From rfc5664, I think the most important differences are about how object layout maps the file to pNFS object layout and how it makes use of standard SCSI OSD/OSD-2 commands. As it is known from 5664 the workflow of pNFS object layout is like following:
- send layoutget to MDS to retrieve pnfs_osd_layout4 structure.
- map IO pages to different objects according to pnfs_osd_data_map4
- for each related objects, send proper OSD/OSD-2 commands to accomplish the IO requests.

In summary if we compare the 2 layouts Lustre and pNFS object layout we can see similarities and differences that would be able to justify a new layout.

Similarities:
1. Both Lustre and pNFS object layout use layout information to map large file onto several object files residing on OST/OSD.
2. By design, they both support several RAID algorithms for data redundancy.

Differences:
1. They use different data path protocols. Lustre uses PtlRPC and Lustre protocol to send/receive data, while object layout is tight with OSD/OSD-2 commands.
2. Lustre file extent locks are decoupled and managed by OSTs, while pNFS use layouts to manage read/write permissions and they are managed solely by MDS.

(ii) Lustre layout is similar yet different from pNFS file layout in 5661:

As it is known from 5661 the workflow of pNFS file layout is like the following:

- layoutget to retrieve layout information from server and possibly getdeviceinfo to retrieve ds_list for the file
- when a page need to read/write, calculate corresponding data server via stripping and ds_list information
- use corresponding file handle to read/write to data servers via NFSv41

According to 5661 pNFS file layout supports byte range locks. It's just that POSIX semantics is not guaranteed because of nfs close-to-open semantics. And it is left as an implementation choice because protocol-wise RFC5661 allows for POSIX like shared access.

Similarities:
1. Both Lustre and pNFS file maintain file layout information on MDS and use layout information to map file data to DS (OST for Lustre)

Differences:
1. Lustre layout can support OST level data redundancy like RAID. But pnfs file layout can't because per RFC5661, one unit can be mapped to only one DS list.
2. Data path protocol is different as well as the control protocol between MDS and OSS/OST.
3. Implementation wise, Lustre layout supports POSIX while Linux pNFS only supports close-to-open semantics. (applies to object comparison as well.) 

In summary the pertinent points for introduction of a new Lustre pNFS layout:

1. Similar architectures are used (same as comparison with object layout).
2. Lustre decouples extents IO permission to DS, and pNFS controls it in MDS (same as comparison with object layout). Or if only close-to-open semantics is required (POSIX may break), pNFS file and object layout can both allow shared IO without caring about data lost (e.g., write on one client and read on another).
3. pnfs file layout supports MDS/DS multipathing via NFSv41 trunking. Similarly, Lustre supports failover-pairs of MDS/OSS.
4. Per nfsv4_1_file_layout_ds_addr4, each unit is mapped to one DS (if no failure). Therefore pnfs file layout cannot support any DS level data redundancy (such as RAID1, RAID5 etc.), although RAID0 can be supported and DS can have built-in RAID. On the other hand, both Lustre and  object layout can support different RAID algorithms on DS level but the client is involved in the RAID in the case of objects.

This is my opinion shared with Peng Tao after a lot of reading on the Lustre layout and locking semantics. For example I include an example of Lustre layout to help with your understanding.

In Lustre each file is composed of multiple data objects striped on one or more OSTs (data containers as different from OSS that are object servers). A file object’s layout information is defined in the extended attribute (EA) of the inode that describes the mapping between file object id and its corresponding OSTs. This information is also known as striping EA.

For example, if file A has a stripe count of three , then its EA might look like:

 EA ---> <obj id x, ost p>
         <obj id y, ost q>
         <obj id z, ost r>
         stripe size and stripe width

So if the stripe size is 1MB, then this would means that [0,1M]), [4M,5M) … are stored as object x, which is on OST p; [1M, 2M), [5M, 6M) … are stored as object y, which is on OST q; [2M,3M), [6M,7M) …
are stored as object z, which is on OST r.

This is all I have for the discussion on Wednesday.  Thank you for your patience and understanding.

Sent from my iPad

---

From: <[email-masked]>
Date: Mon, 26 Mar 2012 14:07:23 -0400

Adding Peng Tao. Sorry for the miss.

Sent from my iPad

---

From: <[email-masked]>
Date: Tue, 27 Mar 2012 08:52:13 -0400

Slides for tomorrow discussion are attached.

/Sorin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, March 26, 2012 1:59 PM
To: [email-masked]
Cc: [email-masked]
Subject: [nfsv4] PNFS Lustre layout justification discussion pre-meeting

Similarities:

Differences:

Similarities:

Differences:

Sent from my iPad

---

From: "Welch, Brent" <[email-masked]>
Date: Tue, 27 Mar 2012 15:50:05 +0000

Can you guys put a "Standards" agenda item into the Lustre layout discussion?

I think it's fine for the Lustre community to define a pnfs layout backend, but I'm not sure it fits into an IETF RFC because no other part of the Lustre protocols are defined anywhere, except the code.  In contrast, with the NFSv4, SCSI/block, and iSCSI/OSDv2 backends, those interfaces are covered by a standard and there is an implied commitment to adhere to them.  As an example, Panasas has had to do real work to support the OSDv2 standard, which was codified several years after we started shipping an iSCSI/OSD component in our system.  We put in the effort to get OSD standardized, not to mention pNFS.

So, to put it precisely, what is the normative reference to the Lustre back end?  Version 1.8.1?  Version 2.1?

--
Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, March 27, 2012 5:52 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] PNFS Lustre layout justification discussion pre-meeting

/Sorin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, March 26, 2012 1:59 PM
To: [email-masked]
Cc: [email-masked]
Subject: [nfsv4] PNFS Lustre layout justification discussion pre-meeting

Similarities:

Differences:

Similarities:

Differences:

Sent from my iPad

---

From: "Spencer Shepler" <[email-masked]>
Date: Tue, 27 Mar 2012 08:55:53 -0700

Thanks for raising the issue, Brent.  I am not sure that we defined
such a requirement when allowing for separately defined NFSv4.1 layout types.
There is a requirement that the layout type be captured in a standards-track
RFC that is approved/reviewed by the working group.  However, I believe 
there is a lack of definition about how "deep" that definition requirement
is to be followed.

So, good question.  As said, we should discuss this and determine what
would work best for the community.

Spencer

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Welch, Brent
> Sent: Tuesday, March 27, 2012 8:50 AM
> I think it's fine for the Lustre community to define a pnfs layout
> backend, but I'm not sure it fits into an IETF RFC because no other part
> of the Lustre protocols are defined anywhere, except the code.  In
> contrast, with the NFSv4, SCSI/block, and iSCSI/OSDv2 backends, those
> interfaces are covered by a standard and there is an implied commitment to
> adhere to them.  As an example, Panasas has had to do real work to support
> the OSDv2 standard, which was codified several years after we started
> shipping an iSCSI/OSD component in our system.  We put in the effort to
> get OSD standardized, not to mention pNFS.
> 
> In preparation for the discussion on a new layout for Lustre I want to
> bring some pertinent discussion points to the nfsv4 list.
> 
> From rfc5664, I think the most important differences are about how object
> layout maps the file to pNFS object layout and how it makes use of
> standard SCSI OSD/OSD-2 commands. As it is known from 5664 the workflow of
> pNFS object layout is like following:
> In summary if we compare the 2 layouts Lustre and pNFS object layout we
> can see similarities and differences that would be able to justify a new
> layout.
> 
> Similarities:
> 1. Both Lustre and pNFS object layout use layout information to map large
> file onto several object files residing on OST/OSD.
> 1. They use different data path protocols. Lustre uses PtlRPC and Lustre
> protocol to send/receive data, while object layout is tight with OSD/OSD-2
> commands.
> 2. Lustre file extent locks are decoupled and managed by OSTs, while pNFS
> use layouts to manage read/write permissions and they are managed solely
> by MDS.
> 
> - layoutget to retrieve layout information from server and possibly
> getdeviceinfo to retrieve ds_list for the file
> According to 5661 pNFS file layout supports byte range locks. It's just
> that POSIX semantics is not guaranteed because of nfs close-to-open
> semantics. And it is left as an implementation choice because protocol-
> wise RFC5661 allows for POSIX like shared access.
> 
> Similarities:
> 1. Both Lustre and pNFS file maintain file layout information on MDS and
> use layout information to map file data to DS (OST for Lustre)
> 
> Differences:
> 1. Lustre layout can support OST level data redundancy like RAID. But pnfs
> file layout can't because per RFC5661, one unit can be mapped to only one
> DS list.
> 3. Implementation wise, Lustre layout supports POSIX while Linux pNFS only
> supports close-to-open semantics. (applies to object comparison as well.)
> 
> 
> 2. Lustre decouples extents IO permission to DS, and pNFS controls it in
> MDS (same as comparison with object layout). Or if only close-to-open
> semantics is required (POSIX may break), pNFS file and object layout can
> both allow shared IO without caring about data lost (e.g., write on one
> client and read on another).
> 3. pnfs file layout supports MDS/DS multipathing via NFSv41 trunking.
> Similarly, Lustre supports failover-pairs of MDS/OSS.
> 4. Per nfsv4_1_file_layout_ds_addr4, each unit is mapped to one DS (if no
> failure). Therefore pnfs file layout cannot support any DS level data
> redundancy (such as RAID1, RAID5 etc.), although RAID0 can be supported
> and DS can have built-in RAID. On the other hand, both Lustre and  object
> layout can support different RAID algorithms on DS level but the client is
> involved in the RAID in the case of objects.
> 
> This is my opinion shared with Peng Tao after a lot of reading on the
> Lustre layout and locking semantics. For example I include an example of
> Lustre layout to help with your understanding.
> 
> In Lustre each file is composed of multiple data objects striped on one or
> more OSTs (data containers as different from OSS that are object servers).
> A file object’s layout information is defined in the extended attribute
> (EA) of the inode that describes the mapping between file object id and
> its corresponding OSTs. This information is also known as striping EA.
> 
> So if the stripe size is 1MB, then this would means that [0,1M]), [4M,5M)
> … are stored as object x, which is on OST p; [1M, 2M), [5M, 6M) … are
> stored as object y, which is on OST q; [2M,3M), [6M,7M) … are stored as
> object z, which is on OST r.
>

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 27 Mar 2012 16:39:00 +0000

> So, to put it precisely, what is the normative reference to the Lustre back end?  Version 1.8.1?  Version 2.1?

The question boils down to this: are we trying to define a layout that
exists simply for the benefit of allowing multiple clients to operate
against a single server vendor, or are we trying to define something
that will allow for competing server implementations too?

If the former, then the whole exercise is a lot less interesting IMO.
Why would we want to work on multiple client implementations, instead of
just standardising on the existing native Lustre client?

Cheers
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: <[email-masked]>
Date: Tue, 27 Mar 2012 13:37:58 -0400

Sent from my iPad

> that will allow for competing server implementations too?
There is no single Lustre server. There is an open source project that any vendor can use to build their own Lustre cluster/server. The pNFS client will allow different distress and clients OS to work with any server of any vendor including yours and ours and anybody else.

> If the former, then the whole exercise is a lot less interesting IMO.
As I explain it is the former.

> just standardising on the existing native Lustre client?
Well the answer to this question you will get on Monday during the discussion on Lustre client to kernel at LSF meeting from Peng. IMHO pNFS for Lustre will be easier accepted by the Linux community than Lustre client in the kernel.  But you made a very good point that a pNFS client for Lustre can be implemented on different OS, BSD, Wndows, Solaris, etc. I think this alone worth to add a Lustre pNFS layout. As about the transport I think that the current object layout was build as one flavor but doesn't exclude other mechanisms which are closer to file layout than to object. Not to mention that the goal of separate pNFS layouts protocol was exactly intended for additional transports, Lustre was one of the possibilities.

/Sorin

---

From: <[email-masked]>
Date: Tue, 27 Mar 2012 13:41:43 -0400

Sent from my iPad

> There is no single Lustre server. There is an open source project that any vendor can use to build their own Lustre cluster/server. The pNFS client will allow different distress and clients OS to w…
Distros ?!!? iPad idiot.

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 27 Mar 2012 14:00:51 -0400 (EDT)

Inline

But...

----- "faibish sorin" <[email-masked]> wrote:

> backend, but I'm not sure it fits into an IETF RFC because no other
> part of the Lustre protocols are defined anywhere, except the code. 
> In contrast, with the NFSv4, SCSI/block, and iSCSI/OSDv2 backends,
> those interfaces are covered by a standard and there is an implied
> commitment to adhere to them.  As an example, Panasas has had to do
> real work to support the OSDv2 standard, which was codified several
> years after we started shipping an iSCSI/OSD component in our system. 
> We put in the effort to get OSD standardized, not to mention pNFS.
> >> 
> >> So, to put it precisely, what is the normative reference to the
> Lustre back end?  Version 1.8.1?  Version 2.1?
> > 
> There is no single Lustre server. There is an open source project that
> any vendor can use to build their own Lustre cluster/server. The pNFS
> client will allow different distress and clients OS to work with any
> server of any vendor including yours and ours and anybody else.

Isn't it potentially more plausible that Lustre is part of a -class- of layout implementation which could share
some extended features?  Then we could discuss what those were.

For example, in implementing pnfs files with Ceph we came up with a number of areas where
we needed more flexibility.

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Tue, 27 Mar 2012 18:18:19 +0000

So, you are proposing a new layout type. However, some of the differences could speak (preferably) for changes to the base protocol:

>1. pNFS object use layouts to manage read/write permissions managed solely by MDS.

This asks for metadata striping. Look at Mike's proposal couple of years ago. I'd hate to have another layout type doing this differently, when I think the base protocol can address this for all layout types.

>2. Lustre supports POSIX while pNFS only supports close-to-open

How do you propose even addressing this with new layout?

Basically, other than solving the Lustre file system support with pNFS, do we expect to add anything to the protocol?

> Sent: Tuesday, March 27, 2012 10:42 AM
> To: [email-masked]
> Cc: [email-masked]; Myklebust, Trond; [email-masked];
> [email-masked]
> backend, but I'm not sure it fits into an IETF RFC because no other part of the
> Lustre protocols are defined anywhere, except the code.  In contrast, with
> the NFSv4, SCSI/block, and iSCSI/OSDv2 backends, those interfaces are
> covered by a standard and there is an implied commitment to adhere to
> them.  As an example, Panasas has had to do real work to support the OSDv2
> standard, which was codified several years after we started shipping an
> iSCSI/OSD component in our system.  We put in the effort to get OSD
> standardized, not to mention pNFS.
> >>>
> >> The question boils down to this: are we trying to define a layout
> >> that exists simply for the benefit of allowing multiple clients to
> >> operate against a single server vendor, or are we trying to define
> >> something that will allow for competing server implementations too?
> > There is no single Lustre server. There is an open source project that any
> vendor can use to build their own Lustre cluster/server. The pNFS client will
> allow different distress and clients OS to work with any server of any vendor
> including yours and ours and anybody else.
> Distros ?!!? iPad idiot.
> 
> >
> >>
> >> Why would we want to work on multiple client implementations, instead
> >> of just standardising on the existing native Lustre client?
> > Well the answer to this question you will get on Monday during the
> discussion on Lustre client to kernel at LSF meeting from Peng. IMHO pNFS
> for Lustre will be easier accepted by the Linux community than Lustre client
> in the kernel.  But you made a very good point that a pNFS client for Lustre
> can be implemented on different OS, BSD, Wndows, Solaris, etc. I think this
> alone worth to add a Lustre pNFS layout. As about the transport I think that
> the current object layout was build as one flavor but doesn't exclude other
> mechanisms which are closer to file layout than to object. Not to mention
> that the goal of separate pNFS layouts protocol was exactly intended for
> additional transports, Lustre was one of the possibilities.
> >
> > /Sorin
> >
> >>
> >> Cheers
> >> Trond
> >>
> >>> --
> >>> Brent
> >>>
> >>>
> >>> From rfc5664, I think the most important differences are about how
> object layout maps the file to pNFS object layout and how it makes use of
> standard SCSI OSD/OSD-2 commands. As it is known from 5664 the workflow
> of pNFS object layout is like following:
> >>> In summary if we compare the 2 layouts Lustre and pNFS object layout
> we can see similarities and differences that would be able to justify a new
> layout.
> >>>
> >>> Similarities:
> >>> 1. Both Lustre and pNFS object layout use layout information to map
> large file onto several object files residing on OST/OSD.
> >>> 2. Lustre file extent locks are decoupled and managed by OSTs, while
> pNFS use layouts to manage read/write permissions and they are managed
> solely by MDS.
> >>>
> >>> - when a page need to read/write, calculate corresponding data
> >>> server via stripping and ds_list information
> semantics. And it is left as an implementation choice because protocol-wise
> RFC5661 allows for POSIX like shared access.
> >>>
> >>> Similarities:
> >>> 1. Both Lustre and pNFS file maintain file layout information on MDS
> >>> and use layout information to map file data to DS (OST for Lustre)
> >>>
> >>> Differences:
> >>> 1. Lustre layout can support OST level data redundancy like RAID. But
> pnfs file layout can't because per RFC5661, one unit can be mapped to only
> one DS list.
> >>> 3. Implementation wise, Lustre layout supports POSIX while Linux
> >>> pNFS only supports close-to-open semantics. (applies to object
> >>> comparison as well.)
> >>>
> >>>
> both allow shared IO without caring about data lost (e.g., write on one client
> and read on another).
> >>> 4. Per nfsv4_1_file_layout_ds_addr4, each unit is mapped to one DS (if
> no failure). Therefore pnfs file layout cannot support any DS level data
> >>> In Lustre each file is composed of multiple data objects striped on one
> or more OSTs (data containers as different from OSS that are object servers).
> A file object’s layout information is defined in the extended attribute (EA)
> of the inode that describes the mapping between file object id and its
> corresponding OSTs. This information is also known as striping EA.
> >>>
> >>> So if the stripe size is 1MB, then this would means that [0,1M]),
> >>> [4M,5M) … are stored as object x, which is on OST p; [1M, 2M), [5M, 6M)
> … are stored as object y, which is on OST q; [2M,3M), [6M,7M) … are stored
> as object z, which is on OST r.
> >>>

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 27 Mar 2012 14:24:43 -0400 (EDT)

Hi,

----- "Pranoop Erasani" <[email-masked]> wrote:

> So, you are proposing a new layout type. However, some of the
> differences could speak (preferably) for changes to the base
> protocol:
> 
> This asks for metadata striping. Look at Mike's proposal couple of
> years ago. I'd hate to have another layout type doing this
> differently, when I think the base protocol can address this for all
> layout types.

There are very useful ideas in that draft, IIRC.  I'm would certainly like
to see a process which considered tradeoffs from several directions (implementations).
I wasn't clear that draft had advanced past a very early stage?

Matt

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Tue, 27 Mar 2012 18:27:48 +0000

> There are very useful ideas in that draft, IIRC.  I'm would certainly like to see
> a process which considered tradeoffs from several directions
> (implementations).

It hasn't been reviewed. However, this keeps coming from time to time.
I'm very much interested in taking that forward, if there is value in this context.

- Pranoop

---

From: "Welch, Brent" <[email-masked]>
Date: Tue, 27 Mar 2012 20:22:36 +0000

Metadata clustering (i.e., striping) is orthogonal to the current pNFS data layout protocol.
You can have one or multiple metadata services behind the pNFS interface, which is what we do.
However, you suffer a hop in the metadata path that you in turn optimize with a big cache
at the NFSv4.1 server that represents the first hop.

But, to the topic at hand, a new layout type for Lustre doesn't demand metadata striping, and
the pNFS object-based layout back-end doesn't include specific support for metadata striping either.
--
Brent

---

From: Rick Macklem <[email-masked]>
Date: Tue, 27 Mar 2012 17:08:47 -0400 (EDT)

Spencer Shepler wrote:
>From my point of view, it would be nice if the RFC went deep enough
that a client implementation could be done without needing to look at
Lustre sources. (Reusing sources would be nice, but in the FreeBSD
world, anything GPL'd is a "no go".)

I would also hope that the client->DS protocol would not be a "moving target",
as can easily happen in a "our sources only" situation.

At some point, having a pNFS client able to use Lustre servers would
seem to me to be a useful goal, imho.

rick

> > Subject: Re: [nfsv4] PNFS Lustre layout justification discussion
> > pre-
> > meeting
> >
> > part
> > [email-masked]
> > Subject: Re: [nfsv4] PNFS Lustre layout justification discussion
> > pre-
> > meeting
> >
> > [email-masked]
> > object
> > - for each related objects, send proper OSD/OSD-2 commands to
> > accomplish
> > the IO requests.
> >
> > we
> > large
> > pNFS
> > As it is known from 5661 the workflow of pNFS file layout is like
> > the
> > following:
> >
> > and
> > 2. Data path protocol is different as well as the control protocol
> > between
> > MDS and OSS/OST.
> > pNFS only
> > In summary the pertinent points for introduction of a new Lustre
> > pNFS
> > layout:
> >
> > For example, if file A has a stripe count of three , then its EA
> > might
> > look like:
> >
> > [4M,5M)
> > This is all I have for the discussion on Wednesday. Thank you for
> > your
> > patience and understanding.
> >
> >
> >
> > Sent from my iPad
> >
> > On Mar 20, 2012, at 2:24 AM, "Spencer Shepler"
> > <[email-masked]>
> > wrote:
> >
> > >
> > > Any others I need to know about; will be posting draft agenda in
> > > couple
> > of days...
> > >
> > > Spencer
> > >
> > > Paris,

---

From: <[email-masked]>
Date: Tue, 27 Mar 2012 21:14:19 -0400

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rick Macklem
Sent: Tuesday, March 27, 2012 5:09 PM
To: Spencer Shepler
Cc: Brent' 'Welch; [email-masked]; [email-masked]; faibish, sorin
Subject: Re: [nfsv4] PNFS Lustre layout justification discussion pre-meeting

>> there is a lack of definition about how "deep" that definition
.>> 
> world, anything GPL'd is a "no go".)
This is my clear intention. I did the reading of the Lustre code and architecture so the users of pNFS Lustre layout will not have to.

> As can easily happen in a "our sources only" situation.
After 10 years in the making I most certainly hope so. Yet it didn't change for the last few years.

> seem to me to be a useful goal, imho.
Ack.

/Sorin

rick

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 28 Mar 2012 01:51:09 -0700

As David Black said in the WG meeting, this is the crux of the issues
around this proposal.

It seems to me that a pNFS layout type for a clustered file system is
going to "reach" in to the file systems internals, using internal
interfaces that are might not stable today. For example, when the Lustre
file system software is revised, is it required that each node in cluster
be upgraded all at once or is it a rolling upgrade? If the former, that
implies unstable internal interfaces. Supporting a Lustre layout type
would require the Lustre community to either commit to stablizing internal
interfaces, or commit to supporting a data protocol to specifically
support pNFS that is guaranteed to be forward compatible with future
revisions of the Lustre file system.

One thing that has not been clear to me is why can the files-layout
(LAYOUT4_NFSV4_1_FILES) be used for Lustre?

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: <[email-masked]>
Date: Wed, 28 Mar 2012 05:37:49 -0400

Hi Mike,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Mike Eisler
> Sent: Wednesday, March 28, 2012 4:51 PM
> To: Rick Macklem; [email-masked]
You are right that Lustre cannot use files-layout directly because from DS side files-layout uses NFS while Lustre uses its own protocol.

Yet there is a PoC in Lustre community to implement file layout servers on top of Lustre servers. That's probably why you hear files-layout can be used for Lustre. The basic idea of the PoC is to export Lustre MDS as file layout MDS, and Lustre OSS as file layout DS, based on the fact that Lustre client can re-export Lustre mount points as nfs exports. As you can get the picture, there is one Lustre client running on each Lustre OSS to turn it into file layout DS. And change has made to Lustre MDS so that it can be exported as pNFS file layout MDS. And on client side it just uses unchanged file layout clients. However it is just a PoC and the performance overhead can be large compared with original Lustre data protocol.

Thanks,
Tao
> >> > A file objectâ€™s layout information is defined in the extended
> >> > attribute
> >> > â€¦ are stored as object x, which is on OST p; [1M, 2M), [5M, 6M) â€¦
> >> > are
> >> > stored as object y, which is on OST q; [2M,3M), [6M,7M) â€¦ are stored
> >> > as

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 28 Mar 2012 14:11:14 +0000

> (LAYOUT4_NFSV4_1_FILES) be used for Lustre?

Or the objects layout, which should be a closer fit to the Lustre
OSD-based model...

> >> > A file objectâs layout information is defined in the extended
> >> > attribute
> >> > â¦ are stored as object x, which is on OST p; [1M, 2M), [5M, 6M) â¦
> >> > are
> >> > stored as object y, which is on OST q; [2M,3M), [6M,7M) â¦ are stored
> >> > as

NetApp

---

From: <[email-masked]>
Date: Wed, 28 Mar 2012 11:01:05 -0400

Sent from my iPad

> Or the objects layout, which should be a closer fit to the Lustre
If you read my presentation you will see Lustre layout model it is further away from OSD than from file. There are more common parts with file than with object. My presentation shows it.

/Sorin