Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 17 Nov 2004 15:49:06 -0500

I'm going to object once more to the "file-level referral" terminology
(since the meta-data server is retaining ultimate authority), but I
suspect I am fighting a losing battle.
 
What it seems like Charles is asking for here is that the target
metadata server has a data delegation for the file and that the
meta-data server informs the client of this fact in the layout, so the
client could then cache this fact and then direct requests for
file-level access control to that server.  Multiple target data servers
would be possible if the data servers had a read delegation and so you
could then have load balancing.  In a striped configuration, there would
be multiple data servers so data delegation clearly couldn't be given
for write.  I need to think a little more about the multi-stripe read
delegation case. 

	-----Original Message-----
	From: Chenggong Charles Fan [mailto:[email-masked]] 
	Sent: Tuesday, November 16, 2004 8:59 PM
	To: Noveck, Dave
	Cc: Marc Eshel; Nicolas Williams; Mike Eisler; Brent Welch;
[email-masked]; Trond Myklebust
	Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

	I echo Marc and Dave.  It will be interesting to have file-level
referal as one of the use cases for pNFS, in the case actual storage
server is also NFSv4 and {fs_location, FH', byte-ranges( 0 - infinity )}
I think as long as pNFS spec doesn't prevent an implemention where
meta-data server provides only location/layout information but performs
no file-level access control, and actual access control is only
performed on the "backend" NFSv4 storage server, we'll get the
file-level referal feature for free.  
	
	Charles
	
	Noveck, Dave wrote:

		It will give you many of the benefits of file-level
reverral but it
		isn't exactly a referral in that the metadata server
maintains
		control of the metadata and can recall the layout at any
time.
		
		The interesting point is that while much of the pnfs
discussion
		discusses the benfits of parallelism in terms of
striping large
		files among data repositories, there is also a benefit
for 
		environments in which file sizes are more moderate and
only consist 
		of a single stipe.  In that case there can be
parallelsim due 
		to the fact that files from the same fs are distributed
among
		a set of data servers.
		
		-----Original Message-----
		From: Marc Eshel [mailto:[email-masked]]
		Sent: Tuesday, November 16, 2004 6:17 PM
		To: Nicolas Williams
		Cc: Noveck, Dave; Mike Eisler; [email-masked]; Trond
Myklebust; Brent
		Welch
		Subject: Re: [nfsv4] comments on
draft-welch-pnfs-ops-00.txt

		

		Nicolas Williams <[email-masked]> wrote on
11/16/2004 02:17:43
		PM:

			On Tue, Nov 16, 2004 at 05:13:40PM -0500, Trond
Myklebust wrote:

				ty den 16.11.2004 Klokka 12:30 (-0600)
skreiv Nicolas Williams:

				On Tue, Nov 16, 2004 at 01:14:46PM
-0500, Noveck, Dave wrote:

				So the question is, how do you create a
meaningful requirement?
				OK, it's mandatory to provide a
NFSv4-style layout.  Nothing could
				be simpler.  The layout for file handle
FH says that the data is
				at FH' and IP address X (one of the
servers IP addresses) and your
				done.  Is that OK?  The performance of
accessing the data using
				that layout will be no better than
accessing it over standard
				NFSv4, but do we want to get in the
business of making performance
				requirements?

				Huh?  Why couldn't the meta-data server
say
				
				[{fs_location1, FH', byte-ranges},
{fs_location2, FH', byte-ranges},

		..,

				 {fs_locationN, FH', byte-ranges}]

				Yech. That is potentially an *unbounded*
list.

			No, it's not.  It's clearly bounded *and* this
is exactly the sort of
			thing that was presented last week at the
meeting, only there the
			emphasis was on blocks.

			Nico
			--

		This is great we get file level referral with
{fs_location, FH',
		byte-ranges( 0 - infinity )}
		which is good among other things for load balancing.
		
		Marc.

			_______________________________________________
			nfsv4 mailing list
			[email-masked]
			https://www1.ietf.org/mailman/listinfo/nfsv4

		
		_______________________________________________
		nfsv4 mailing list
		[email-masked]
		https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Chenggong Charles Fan <[email-masked]>
Date: Wed, 17 Nov 2004 14:15:12 -0800

In addition to load balancing, more applications may benefit from this 
"pNFS layout-based file-level referral".  For example, tier-storage 
management -- we can use the pNFS layout mechanism an an HSM stub.  An 
HSM application can be built on top to move certain files (for example, 
older or less valuable ones), according to some policies, from primary 
file server to a secondary file server.  Clients continue access the 
primary server, and with pNFS in place, the primary server can behave 
like a pNFS meta-data server for those files that have been moved and 
"refer" (or give a layout that point) the client to the secondary file 
server.

The fact the layout can be recalled at any time by the metadata server 
is actually very nice, for the case when the location of a file changes, 
meta-data server can recall the layout to update it.

I know these are not the primary applications pNFS is designed for.  I 
share them to show that potentially more applications (than HPC I/O 
parallelization) may benefit from the pNFS mechanisms.  Please feel free 
to table my comments for now if it increases the confusion at this stage.

Charles

Noveck, Dave wrote:

> file-level access control to that server.  Multiple target data 
> servers would be possible if the data servers had a read delegation 
> and so you could then have load balancing.  In a striped 
> configuration, there would be multiple data servers so data delegation 
> clearly couldn't be given for write.  I need to think a little more 
> about the multi-stripe read delegation case. 
>
>     *From:* Chenggong Charles Fan [mailto:[email-masked]]
>     *Sent:* Tuesday, November 16, 2004 8:59 PM
>     *To:* Noveck, Dave
>     *Cc:* Marc Eshel; Nicolas Williams; Mike Eisler; Brent Welch;
>     *Subject:* Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt
>
>     referal as one of the use cases for pNFS, in the case actual
>     storage server is also NFSv4 and {fs_location, FH', byte-ranges( 0
>     - infinity )}  I think as long as pNFS spec doesn't prevent an
>     implemention where meta-data server provides only location/layout
>     information but performs no file-level access control, and actual
>     access control is only performed on the "backend" NFSv4 storage
>     server, we'll get the file-level referal feature for free. 
>
>     Charles
>
>     Noveck, Dave wrote:
>
>>It will give you many of the benefits of file-level reverral but it
>>isn't exactly a referral in that the metadata server maintains
>>control of the metadata and can recall the layout at any time.
>>
>>The interesting point is that while much of the pnfs discussion
>>discusses the benfits of parallelism in terms of striping large
>>files among data repositories, there is also a benefit for 
>>environments in which file sizes are more moderate and only consist 
>>of a single stipe.  In that case there can be parallelsim due 
>>to the fact that files from the same fs are distributed among
>>a set of data servers.
>>
>>Cc: Noveck, Dave; Mike Eisler; [email-masked]; Trond Myklebust; Brent
>>Welch
>>Nicolas Williams <[email-masked]> wrote on 11/16/2004 02:17:43
>>PM:
>>
>>  
>>
>>>On Tue, Nov 16, 2004 at 05:13:40PM -0500, Trond Myklebust wrote:
>>>    
>>>
>>>>ty den 16.11.2004 Klokka 12:30 (-0600) skreiv Nicolas Williams:
>>>>      
>>>>
>>>>>On Tue, Nov 16, 2004 at 01:14:46PM -0500, Noveck, Dave wrote:
>>>>>        
>>>>>
>>>>>>So the question is, how do you create a meaningful requirement?
>>>>>>OK, it's mandatory to provide a NFSv4-style layout.  Nothing could
>>>>>>be simpler.  The layout for file handle FH says that the data is
>>>>>>at FH' and IP address X (one of the servers IP addresses) and your
>>>>>>done.  Is that OK?  The performance of accessing the data using
>>>>>>that layout will be no better than accessing it over standard
>>>>>>NFSv4, but do we want to get in the business of making performance
>>>>>>requirements?
>>>>>>          
>>>>>>
>>>>>Huh?  Why couldn't the meta-data server say
>>>>>
>>>>>[{fs_location1, FH', byte-ranges}, {fs_location2, FH', byte-ranges},
>>>>>        
>>>>>
>>..,
>>  
>>
>>>>Yech. That is potentially an *unbounded* list.
>>>>      
>>>>
>>
>>  
>>
>>>No, it's not.  It's clearly bounded *and* this is exactly the sort of
>>>thing that was presented last week at the meeting, only there the
>>>emphasis was on blocks.
>>>    
>>>
>>
>>  
>>
>>>Nico
>>>--
>>>    
>>>
>>
>>This is great we get file level referral with {fs_location, FH',

---

From: Erblichs <[email-masked]>
Date: Wed, 17 Nov 2004 16:11:33 -0800

Dave Noveck,

	I don't quite understand...

	"Multiple target data servers would be possible if the data servers had
a read delegation and so you could then have load balancing."

	OR

	"The interesting point is that while much of the pnfs discussion
       discusses the benfits of parallelism in terms of striping large
       files among data repositories"

	 "In that case there can be parallelsim due 
       to the fact that files from the same fs are distributed among
       a set of data servers."

	Thus....

	Is this for readers from different hosts accessing
	different files on different servers?

	If it was different hosts accessing parts of the same
	file, then read-ahead/read-cacheing performance 
	improvements would be minimized.

	If the same host was randomly accessing multiple
	files, then again no read improvements.

	If it was the same host accessing different files,
	then completely different connections would be
	generated and each of them would have (TCP, etc)
	slow-start start-up costs involved.

	So, I am trying to come up with a load scenario
	where 1 data server is overloaded, and splitting
	it up could get a improvement. A sample case is a
	hi-def video. If the data was stripped then COMPRESSION
	(yes, a tradeoff of CPU Mhz vs Mbits) should be
	used (unless 0 loss), and the full link-bandwidth 
	could be consumed. If you stripped the data, you
	could theorectically consume N-stripe bandwidths.

	However, is the overhead of re-merging the stripped
	data at the host worth it?

	Also, with the current and future generation of
	drives, I can't see the real reason for striping.
	Yes, mirroring of data allows a higher level of
	availabliity assuming network failures, WAN access
	delays, etc.

	So, in my opinion, if you say you had a very large
	file, why wouldn't you split it in say X size chunks, and
	have each chunk accessible by a server IP address,
	FH, and byte/block range? This way you could read each
	"segment" as needed and remove the striping behaviour.
	Then the only question is whether you support holes
	and reservations on the FS to gurantee that a full
	segment on a FS can exist.

	Mitchell Erblich
	--------------------------

> "Noveck, Dave" wrote:
>      I echo Marc and Dave.  It will be interesting to have
>      file-level referal as one of the use cases for pNFS, in the
>      case actual storage server is also NFSv4 and {fs_location,
>      FH', byte-ranges( 0 - infinity )}  I think as long as pNFS
>      spec doesn't prevent an implemention where meta-data server
>      provides only location/layout information but performs no
>      file-level access control, and actual access control is only
>      performed on the "backend" NFSv4 storage server, we'll get
>      the file-level referal feature for free.
> 
>      Charles
> 
>      Noveck, Dave wrote:
> 
>     ---------------------------------------------------------------