Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 18 Nov 2004 14:53:29 -0500

I don't quite understand...

Maybe I don't either.  Let's walk through it.

	"Multiple target data servers would be possible if the data
servers had
       a read delegation and so you could then have load balancing."

I could be a little slick and just note that I said it was possible and 
not that it was desirable, but would I do such a thing? :-)

Anyway, your message has prompted me to reconsider this issue and I have

to say that the circumstances would have to be very unusual for the
same file to be delegated to multiple servers simultaneously.

	OR

	"The interesting point is that while much of the pnfs discussion
       discusses the benfits of parallelism in terms of striping large
       files among data repositories"

	 "In that case there can be parallelsim due 
       to the fact that files from the same fs are distributed among
       a set of data servers."

I guess it is implicit in the above that I thought there were benefits
also in the non-striping case.  I continue to think that even though 
the case in which a single file is read on multiple servers would not
be likely to give much benefit. 

	Thus....

	Is this for readers from different hosts accessing
	different files on different servers?

This is the most interesting case.

	If it was different hosts accessing parts of the same
	file, then read-ahead/read-cacheing performance 
	improvements would be minimized.

	If the same host was randomly accessing multiple
	files, then again no read improvements.

I'm not sure what you are saying here.  No read improvments
relative to what?  Anyway, in cases in which a single
server does not have the horsepower to satisfy all of the
load for files within a single fs, splitting it up and 
allowing some to be served from one server and some from 
another would improve overall read performance.

	If it was the same host accessing different files,
	then completely different connections would be
	generated and each of them would have (TCP, etc)
	slow-start start-up costs involved.

Yes, different connections would be generated, but that is
a one time cost.  The connections would stay around and if
that client were accessing many more files than servers
(as would be expected), those start-up costs would be 
amortized fairly quickly, or at least they would, anywhere
you would want to deploy such a configuration.

> 	So, I am trying to come up with a load scenario
>	where 1 data server is overloaded, and splitting
>	it up could get a improvement. A sample case is a
>	hi-def video. If the data was stripped then COMPRESSION
>	(yes, a tradeoff of CPU Mhz vs Mbits) should be
>	used (unless 0 loss), and the full link-bandwidth 
>	could be consumed. If you stripped the data, you
>	could theorectically consume N-stripe bandwidths.

Yup.  For me, that's the pnfs value proposition.  

Now, I'm confused, you are talking about striping.  Is it 
that you are talking  about striping at the application level?  
If so, I agree that that doesn't make a lot of sense compared
to pnfs, which striped below that.

If you are reading large files as in the hi-def video case,
then you want to stripe.  However there are many cases in
which people have many many fairly small files that they 
want to read.  I know it seems wierd but there are some 
just crazy software build and similar environments out 
there where the requirements for bandwidth to a set of 
fairly small files get intense.  You can call these people
insane if you want to, but my inclination is to give them
what they want if they're willing to pay for it. 

> 	However, is the overhead of re-merging the stripped
>	data at the host worth it?

If you are talking about remerging at the application level,
then it is only worth it when you are desperate.

If you are talking about pnfs-style striping, then it is
indeed worth it since there is essentially no remerge overhead
just as there isn't reading a local file that on a large
raide group.

> 	Also, with the current and future generation of
>	drives, I can't see the real reason for striping.
>	Yes, mirroring of data allows a higher level of
>	availabliity assuming network failures, WAN access
>	delays, etc.

Despite continuing disk drive improvements, I find people
who always want more than is available from a single drive
and also people who want more bandwidth than is available
from a single server with files striped among many striped 
drives and who need to stripe among servers.  

	So, in my opinion, if you say you had a very large
	file, why wouldn't you split it in say X size chunks, and
	have each chunk accessible by a server IP address,
	FH, and byte/block range? This way you could read each
	"segment" as needed and remove the striping behaviour.

I think you have just described striping behavior, but the
striping is performed by the client rather than the application
(if I have understood what you are saying).

	Then the only question is whether you support holes
	and reservations on the FS to gurantee that a full
	segment on a FS can exist.

If that were the only question, we'd be just about done.

And I understand Spencer said this would take years to get a 
spec done.  

I think you are right in principle; this is a fundamentally 
simple idea, but Spencer is unfortunately right in practice.
When you add in security issues, dealing with errors, server
and clients going up and down, accommodating the possibility
of additional storage protocols (and possibly new security 
models), it takes a long time.  I think the working group 
needs to figure out a subset that can be done in a reasonable
time, which is still going to take a while to get to the 
Proposed Standard stage.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Erblichs <[email-masked]>
Date: Thu, 18 Nov 2004 15:30:32 -0800

Yes,

	I understand.. I have just in the past used to
	"fast-tracking" everything to get it done. yes,
	their are always many issues to identify and
	resolve when working with a new spec..

	To expand on only one item..

(ME)     If it was the same host accessing different files,
        then completely different connections would be
        generated and each of them would have (TCP, etc)
        slow-start start-up costs involved.

not-me : Yes, different connections would be generated, but that is

	Multiple servers
Me again --- Close but no.. If multiple TCP connections are made
	to multiple servers vs using just a few connections to
	a few servers.  Most of the connections would go "idle"
	and thus not be network limited. With application
	limited connections or idleness, TCP development is 
	to decrease the available bandwidth in these 
	connections towards a slow-start behaviour. So when
	a application burst is required their are no longer
	as many acks present in each connection, and we are basicly
	back to slow start. We may not even avg 10% of the
	available bandwidth per connection.

	This is to prevent the idle / delay removing the app from
	allowing a burst of data when, this delay timeframe might
	have informed us of congestion forming in the route and
	seeing drops of segments if we had stayed in congestion 
	avoidance.

	One server
	------------
	With one server we can cheat if we have multiple
	connections (ex: http 1.1) and peak to see if one of
	our connections is seeing loss. If it is not, we can
	somewhat avoid slow-start on new connections or other
	existing connections.

	Thus, sorry to say but I would think people would want
	network limited connections, as long as new data entering
	the path has minimal delay. This, keeps the number of
	acks at near max and gives us a fast-rexmit when segs are
	dropped. This will also maximimze our achievable bandwidth
	and minimize the delay for any random large read. Thus,
	our best performance could be close to the the media
	bandwidth capability.

	Mitchell Erblich
	---------------------