Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 17 Nov 2004 18:04:24 -0500

Interesting points.
 
> Please feel free to table my comments for now if it increases the
confusion at this stage.
 
I'm not sure how increasing the confusion is possible right now,
although I will probably be
proved wrong any minute now.
 
I think all interesting ideas need to be explored.

	-----Original Message-----
	From: Chenggong Charles Fan [mailto:[email-masked]] 
	Sent: Wednesday, November 17, 2004 5:15 PM
	To: Noveck, Dave
	Cc: Marc Eshel; Nicolas Williams; Mike Eisler; Brent Welch;
[email-masked]; Trond Myklebust
	Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

	In addition to load balancing, more applications may benefit
from this "pNFS layout-based file-level referral".  For example,
tier-storage management -- we can use the pNFS layout mechanism an an
HSM stub.  An HSM application can be built on top to move certain files
(for example, older or less valuable ones), according to some policies,
from primary file server to a secondary file server.  Clients continue
access the primary server, and with pNFS in place, the primary server
can behave like a pNFS meta-data server for those files that have been
moved and "refer" (or give a layout that point) the client to the
secondary file server.
	
	The fact the layout can be recalled at any time by the metadata
server is actually very nice, for the case when the location of a file
changes, meta-data server can recall the layout to update it.
	
	I know these are not the primary applications pNFS is designed
for.  I share them to show that potentially more applications (than HPC
I/O parallelization) may benefit from the pNFS mechanisms.  Please feel
free to table my comments for now if it increases the confusion at this
stage.
	
	Charles
	
	Noveck, Dave wrote:

		I'm going to object once more to the "file-level
referral" terminology (since the meta-data server is retaining ultimate
authority), but I suspect I am fighting a losing battle.
		 
		What it seems like Charles is asking for here is that
the target metadata server has a data delegation for the file and that
the meta-data server informs the client of this fact in the layout, so
the client could then cache this fact and then direct requests for
file-level access control to that server.  Multiple target data servers
would be possible if the data servers had a read delegation and so you
could then have load balancing.  In a striped configuration, there would
be multiple data servers so data delegation clearly couldn't be given
for write.  I need to think a little more about the multi-stripe read
delegation case. 

			-----Original Message-----
			From: Chenggong Charles Fan
[mailto:[email-masked]] 
			Sent: Tuesday, November 16, 2004 8:59 PM
			To: Noveck, Dave
			Cc: Marc Eshel; Nicolas Williams; Mike Eisler;
Brent Welch; [email-masked]; Trond Myklebust
			Subject: Re: [nfsv4] comments on
draft-welch-pnfs-ops-00.txt

			I echo Marc and Dave.  It will be interesting to
have file-level referal as one of the use cases for pNFS, in the case
actual storage server is also NFSv4 and {fs_location, FH', byte-ranges(
0 - infinity )}  I think as long as pNFS spec doesn't prevent an
implemention where meta-data server provides only location/layout
information but performs no file-level access control, and actual access
control is only performed on the "backend" NFSv4 storage server, we'll
get the file-level referal feature for free.  
			
			Charles
			
			Noveck, Dave wrote:

				It will give you many of the benefits of
file-level reverral but it
				isn't exactly a referral in that the
metadata server maintains
				control of the metadata and can recall
the layout at any time.
				
				The interesting point is that while much
of the pnfs discussion
				discusses the benfits of parallelism in
terms of striping large
				files among data repositories, there is
also a benefit for 
				environments in which file sizes are
more moderate and only consist 
				of a single stipe.  In that case there
can be parallelsim due 
				to the fact that files from the same fs
are distributed among
				a set of data servers.
				
				-----Original Message-----
				From: Marc Eshel
[mailto:[email-masked]]
				Sent: Tuesday, November 16, 2004 6:17 PM
				To: Nicolas Williams
				Cc: Noveck, Dave; Mike Eisler;
[email-masked]; Trond Myklebust; Brent
				Welch
				Subject: Re: [nfsv4] comments on
draft-welch-pnfs-ops-00.txt

				

				Nicolas Williams
<[email-masked]> wrote on 11/16/2004 02:17:43
				PM:

				On Tue, Nov 16, 2004 at 05:13:40PM
-0500, Trond Myklebust wrote:

				ty den 16.11.2004 Klokka 12:30 (-0600)
skreiv Nicolas Williams:

				On Tue, Nov 16, 2004 at 01:14:46PM
-0500, Noveck, Dave wrote:

				So the question is, how do you create a
meaningful requirement?
				OK, it's mandatory to provide a
NFSv4-style layout.  Nothing could
				be simpler.  The layout for file handle
FH says that the data is
				at FH' and IP address X (one of the
servers IP addresses) and your
				done.  Is that OK?  The performance of
accessing the data using
				that layout will be no better than
accessing it over standard
				NFSv4, but do we want to get in the
business of making performance
				requirements?

				Huh?  Why couldn't the meta-data server
say
				
				[{fs_location1, FH', byte-ranges},
{fs_location2, FH', byte-ranges},

				..,

				 {fs_locationN, FH', byte-ranges}]

				Yech. That is potentially an *unbounded*
list.

				No, it's not.  It's clearly bounded
*and* this is exactly the sort of
				thing that was presented last week at
the meeting, only there the
				emphasis was on blocks.

				Nico
				--

				This is great we get file level referral
with {fs_location, FH',
				byte-ranges( 0 - infinity )}
				which is good among other things for
load balancing.
				
				Marc.

_______________________________________________
				nfsv4 mailing list
				[email-masked]
	
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
				nfsv4 mailing list
				[email-masked]
	
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Gordon Waidhofer" <[email-masked]>
Date: Wed, 17 Nov 2004 17:39:23 -0800

MessageI'm a believer in file-level referral.

I never thought about it in terms of pNFS or stripping.

It does seem to have merit for global name space management
and file migration.

In short, file-level referral should be explored in a broader
context. And not in haste.

For what it's worth.

Regards,
    -gww
  -----Original Message-----
  From: [email-masked] [mailto:[email-masked]]On Behalf Of
Noveck, Dave
  Sent: Wednesday, November 17, 2004 3:04 PM
  To: Chenggong Charles Fan
  Cc: Nicolas Williams; Mike Eisler; Brent Welch; Marc Eshel;
[email-masked]; Trond Myklebust
  Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

  Interesting points.

confusion at this stage.

  I'm not sure how increasing the confusion is possible right now, although
I will probably be
  proved wrong any minute now.

    -----Original Message-----
    From: Chenggong Charles Fan [mailto:[email-masked]]
    Sent: Wednesday, November 17, 2004 5:15 PM
    To: Noveck, Dave
    Cc: Marc Eshel; Nicolas Williams; Mike Eisler; Brent Welch;

    In addition to load balancing, more applications may benefit from this
"pNFS layout-based file-level referral".  For example, tier-storage
management -- we can use the pNFS layout mechanism an an HSM stub.  An HSM
application can be built on top to move certain files (for example, older or
less valuable ones), according to some policies, from primary file server to
a secondary file server.  Clients continue access the primary server, and
with pNFS in place, the primary server can behave like a pNFS meta-data
server for those files that have been moved and "refer" (or give a layout
that point) the client to the secondary file server.

    The fact the layout can be recalled at any time by the metadata server
is actually very nice, for the case when the location of a file changes,
meta-data server can recall the layout to update it.

    I know these are not the primary applications pNFS is designed for.  I
share them to show that potentially more applications (than HPC I/O
parallelization) may benefit from the pNFS mechanisms.  Please feel free to
table my comments for now if it increases the confusion at this stage.

    Charles

    Noveck, Dave wrote:

      I'm going to object once more to the "file-level referral" terminology
(since the meta-data server is retaining ultimate authority), but I suspect
I am fighting a losing battle.

      What it seems like Charles is asking for here is that the target
metadata server has a data delegation for the file and that the meta-data
server informs the client of this fact in the layout, so the client could
then cache this fact and then direct requests for file-level access control
to that server.  Multiple target data servers would be possible if the data
servers had a read delegation and so you could then have load balancing.  In
a striped configuration, there would be multiple data servers so data
delegation clearly couldn't be given for write.  I need to think a little
more about the multi-stripe read delegation case.
        -----Original Message-----
        From: Chenggong Charles Fan [mailto:[email-masked]]
        Sent: Tuesday, November 16, 2004 8:59 PM
        To: Noveck, Dave
        Cc: Marc Eshel; Nicolas Williams; Mike Eisler; Brent Welch;

        I echo Marc and Dave.  It will be interesting to have file-level
referal as one of the use cases for pNFS, in the case actual storage server
is also NFSv4 and {fs_location, FH', byte-ranges( 0 - infinity )}  I think
as long as pNFS spec doesn't prevent an implemention where meta-data server
provides only location/layout information but performs no file-level access
control, and actual access control is only performed on the "backend" NFSv4
storage server, we'll get the file-level referal feature for free.

        Charles

        Noveck, Dave wrote:

It will give you many of the benefits of file-level reverral but it
isn't exactly a referral in that the metadata server maintains
control of the metadata and can recall the layout at any time.

The interesting point is that while much of the pnfs discussion
discusses the benfits of parallelism in terms of striping large
files among data repositories, there is also a benefit for
environments in which file sizes are more moderate and only consist
of a single stipe.  In that case there can be parallelsim due
to the fact that files from the same fs are distributed among
a set of data servers.

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]]
Sent: Tuesday, November 16, 2004 6:17 PM
To: Nicolas Williams
Cc: Noveck, Dave; Mike Eisler; [email-masked]; Trond Myklebust; Brent
Welch

Nicolas Williams <[email-masked]> wrote on 11/16/2004 02:17:43
PM:

  On Tue, Nov 16, 2004 at 05:13:40PM -0500, Trond Myklebust wrote:
    ty den 16.11.2004 Klokka 12:30 (-0600) skreiv Nicolas Williams:
      On Tue, Nov 16, 2004 at 01:14:46PM -0500, Noveck, Dave wrote:
        So the question is, how do you create a meaningful requirement?
OK, it's mandatory to provide a NFSv4-style layout.  Nothing could
be simpler.  The layout for file handle FH says that the data is
at FH' and IP address X (one of the servers IP addresses) and your
done.  Is that OK?  The performance of accessing the data using
that layout will be no better than accessing it over standard
NFSv4, but do we want to get in the business of making performance
requirements?
          Huh?  Why couldn't the meta-data server say

[{fs_location1, FH', byte-ranges}, {fs_location2, FH', byte-ranges},
        ..,
   {fs_locationN, FH', byte-ranges}]
        Yech. That is potentially an *unbounded* list.

  No, it's not.  It's clearly bounded *and* this is exactly the sort of
thing that was presented last week at the meeting, only there the
emphasis was on blocks.

  Nico
--

This is great we get file level referral with {fs_location, FH',
byte-ranges( 0 - infinity )}
which is good among other things for load balancing.

Marc.

---

From: Jon Haswell <[email-masked]>
Date: Thu, 18 Nov 2004 09:15:15 -0800

Yes, let me add my voice as well as being interested in file level
referral.

My primary interest in it would be load balancing, I believe it useful for
situations such as video rendering where a cluster of machines are
typically all targetted to work on 'today's production, which will have all
it's data kept in a single directory, for the convenience of the human
animators.  None of the files is enormously big or requires excessive
bandwidth but one does want to be able to spread the load of serving the
files from this one directory across a farm of NFS servers without having
to reorganize the directory structure. I've seen this as a real world
customer request and also seen similar requests from some other HPC type
applications

Thanks

Jon Haswell
IBM Research
Tel 408 927 2074 Tie 457 2074
Fax 408 927 3030 Tie 457 3030

[email-masked] wrote on 11/17/2004 05:39:23 PM:

> I'm a  believer in file-level referral.
>

> In addition to load  balancing, more applications may benefit from
> this "pNFS layout-based  file-level referral".  For example, tier-
> storage management -- we can  use the pNFS layout mechanism an an
> HSM stub.  An HSM application can  be built on top to move certain
> files (for example, older or less valuable  ones), according to some
> policies, from primary file server to a secondary  file server.
> Clients continue access the primary server, and with pNFS  in place,
> the primary server can behave like a pNFS meta-data server for
> those files that have been moved and "refer" (or give a layout that
> point)  the client to the secondary file server.
>
> server is actually very nice, for the  case when the location of a
> file changes, meta-data server can recall the  layout to update it.
>
> for.  I share them to show that potentially more  applications (than
> HPC I/O parallelization) may benefit from the pNFS  mechanisms.
> I'm going to object once more to the "file-level referral"
> terminology (since the meta-data server is retaining ultimate
> metadata server has a data delegation for the file and that the
> meta-data  server informs the client of this fact in the layout, so
> file-level access  control to that server.  Multiple target data
> servers would be  possible if the data servers had a read delegation
> and so you could then  have load balancing.  In a striped
> configuration, there would be  multiple data servers so data
> delegation clearly couldn't be given  for write.  I need to think a
> little more about the multi-stripe read  delegation case.
[email-masked]
> ; Trond  Myklebust
> referal as one of  the use cases for pNFS, in the case actual
> storage server is also NFSv4  and {fs_location, FH', byte-ranges( 0
> - infinity )}  I think as  long as pNFS spec doesn't prevent an
> information but performs no file-level  access control, and actual
> access control is only performed on the  "backend" NFSv4 storage
> server, we'll get the file-level referal feature  for free.
>
> Charles
>
> Noveck, Dave wrote:
>
> nfsv4 mailing
[email-masked]://www1.ietf.org/mailman/listinfo/nfsv4
> nfsv4 mailing
[email-masked]://www1.ietf.org/mailman/listinfo/nfsv4