Subject: RE: more NFS over RPC transport stuff

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 10 Dec 2002 10:03:03 -0800

> > Sure.  But what that shows is simply that the application can process
> > data faster than it can be fetched over the network connection.  Let's
> > assume Brent's example of a T1-speed connection accessing a server
> > across the country.  Let's say the network latency is 100 ms.  In that 
> > environment, if the application is processing data faster that 180KB/s, 
> > it is going to wind up waiting for its data.  But the problem is not the 
> > latency but the bandwidth of the connection.  If we moved the server to 
> > the next room and the network latency went to < 1 ms, you would still be 
> > waiting for the data and you would only be able to get it at 180KB/s.  
> > Of course, if your definition of "latency" included the network transfer 
> > time, then it would be the case that the process was affected by the latency.
> 
> I'll agree with all of this, although I'll note that most any application
> will process file data faster than something like 180Kbps, so you'll be
> stuck when bandwidth is limited.

180 KB/s was an extreme example but the principle holds.  You get the 
bandwidth that you pay for and there is (unfortunately) no free OC3. 

> (When I referred to latency, I was thinking
> of the whole ball of wax. Time for request to get to the server + delay at
> server for a response + time for response to return to client.) I would
> agree that for the case of a network interconnect with 100ms trip time and
> 180Kbps, that the lack of bandwidth is going to be the bottleneck. (If you
> instead have a 1Gbps transcontinental link with a 250ms trip delay, then
> I think the trip delay may become significant?)

The trip delay is always significant but the significance varies depending
on what you are doing.  The long delay hurts you on requests are unpredictable;
requests other than reads and writes, random reads, direct writes, initial
reads of a file, and the time finishing off a close.  With sequential
reads and writes the latency can be covered by read-ahead and write-behind.
In your example, of a high-delay high-bandwidth link, the delay corresponds
to 32MB of data in flight if you are to cover the latency.  The problem
will be that most files are not that long, and even for those that are, the
startup time of 250 ms is a significant problem.  With 32KB requests, that
means that you need something like a throusand read-aheads outstanding.
I doubt that clients can do that, but that is what would be necessary to
handle such a link effectively.  The cost of such a link would probably
suggest that using it straight (no proxy caching, no delegation) isn't
a very good idea.  With the money you would spend, it is probable that 
you could put together something better, once you understood the 
structure of the application. 

> > Where the asynchronous behavior comes in is in the fact that with
> > the IO being done asynchronously, the latency of the connection does
> > not prevent you from bringing down the IO size, if you need to do that for
> > other reasons (interference with short requests).  You can bring the IO
> > size down and still (with more requests outstanding) cover the latency of
> > a (for example) cross-country link.
>
> Ok, now I think I see what your argument is and it sounds interesting.
> Are you thinking that the I/O size might be dynamically adjusted via some
> information from the transport layer, such as TCP send window size, or
> just that the user would find small read/write size settings on the mount
> more appropriate or...?

What I was thinking of when I wrote that was the latter, but I realize that 
users will often get things wrong.  Dynamic adjustment is a desirable
but would be a lot of work.  One possiblity is a quick test to pick the
size at mount time, if not specified by the user.  If we are talking 
about very low-bandwidth links, then I think you can recognize with a
test at mount time and make the default transfer sizes a bit more
reasonable than the typical defaults.  I don't think you need to truly 
optimize things, but rather to avoid really bad choices, such as the
32K or greater that Brent was talking about that would be kind of bad
news on T1. (at least bad news for delayed random gettar's and such).

Of course, dynamic adjustment is better, and as long as someone else
is doing the work, I'm all for it :-)  

> > I think the write-behind case is precisely parallel.
>
> Yep, except that I think write traffic tends to be burstier, although I'm
> not sure if that's relevant here. (Maybe more likely to saturate a net
> connection during a burst?)

> > That's true.  However, there is a clear benefit in giving the smaller
> > requests priority (in the absence of any other information establishing
> > an inherent priority for these requests).  If I give a small request 
> > priority, I don't delay the large request very much at all, while the 
> > reverse decision will delay the small request a lot.  So if you don't
> > have any other priority information, small first is good.  The process
> > may have to wait when he does a large request, but still handling the
> > requests with little data first is a net win.   This is especially true
> > since reads and writes comes in bunches (read-ahead/write-behind to 
> > optimize things).
>
> Well, I'd agree that there's probably cases where the above is true.
> A user doing a large software build in one window can still "ls -l"
> with a decent response time in another window, might be one.
> However, I still think there's a catch-22 here.
>
> If the network connection
> isn't anywhere near saturation, then I'd agree that prioritizing the
> requests will result in higher throughput (assuming you've defined it as
> RPCs/unit time) and should work just fine. Since the network connection
> isn't near saturation, I'm not sure the effect will be a big win, though.

I wasn't saying higher throughput.  If the network isn't anywhere near
saturation, I don't think throughput will be much better.

> On the other hand...
> If the network connection is near saturation, you run the risk of introducing
> large delays to the large requests, to the point where they will appear to
> be completely unresponsive. We had an amusing case of this just a couple
> of weeks ago here at Guelph.

For most NFS workloads, I have a hard time imagining a situation in which 
the small requests and responses themselves will saturate the link.  If the
small and the large together saturate the link, then you are in the area where
I think favoring the small would help.  But if the small are so numerous that
they saturate the link (or even close to it), then you have a problem.  I hadn't
though of that as a very likely situtation, but it could happen (and I suppose
anything possible will eventualy happen), then that argues for some scheme in
which larger requests can age out of low-priority handling.

> We have approximately 5000 students living on campus, each with LAN connections.
> Friday afternoon a lot of complaints came into the network help line of slow
> response for web access, so one of the net guys decided to up the priority of
> http packets (they recently got new Cisco switches, so they became capable
> of controlling priority of packets recently). Shortly thereafter, the campus
> email hub started to act strangely, with several hundred sendmail connections
> hanging around.
> Basically the change resulted in the campus email system coming to a grinding
> halt, since sendmail packets were being delayed indefinitely while the http
> packets saturated our internet connection, since they had priority.
> (Of course, when the guy who runs the campus mail server asked the net
> guys "Is there anything wrong with the network? You haven't changed anything,
> have you?", they responded with "Nope, aok and nothing's changed", which made
> the eventual resolution of this rather entertaining for me, given I wasn't
> a part of it:-)

I'm guessing that some pretty large messages were included in those getting
priority, so that makes it much more likely that http packets as a class
can saturate the network themselves.

> The catch is that lower priority packets wait a loonnnggg time (theoretically
> approaching infinity) as the net connection approaches saturation. As such,
> I'm a bit shy of any priority scheme.

Fair enough.  But I would say that this might make you skeptical about strict
priority schemes.  Modified priority schemes that include a limit on how long
something can be delayed, are more complicated but are worth looking at.

Reads and writes, besides have long messages associated are also those that
through read-ahead and write-behind (at least in the sequential case) have a
way of avoiding the effects of a short additional delay.  So that makes them
good candidates for at least a modest additional delay when small request 
traffic is also present.

> Well, definitely interesting. Have a good week, rick