Subject: more NFS over RPC transport stuff

From: [email-masked]
Date: Mon, 9 Dec 2002 18:21:32 -0500 (EST)

> Sure.  But what that shows is simply that the application can process
> data faster than it can be fetched over the network connection.  Let's
> assume Brent's example of a T1-speed connection accessing a server
> across the country.  Let's say the network latency is 100 ms.  In that 
> environment, if the application is processing data faster that 180KB/s, 
> it is going to wind up waiting for its data.  But the problem is not the 
> latency but the bandwidth of the connection.  If we moved the server to 
> the next room and the network latency went to < 1 ms, you would still be 
> waiting for the data and you would only be able to get it at 180KB/s.  
> Of course, if your definition of "latency" included the network transfer 
> time, then it would be the case that the process was affected by the latency.

I'll agree with all of this, although I'll note that most any application
will process file data faster than something like 180Kbps, so you'll be
stuck when bandwidth is limited. (When I referred to latency, I was thinking
of the whole ball of wax. Time for request to get to the server + delay at
server for a response + time for response to return to client.) I would
agree that for the case of a network interconnect with 100ms trip time and
180Kbps, that the lack of bandwidth is going to be the bottleneck. (If you
instead have a 1Gbps transcontinental link with a 250ms trip delay, then
I think the trip delay may become significant?)

> Where the asynchronous behavior comes in is in the fact that with
> the IO being done asynchronously, the latency of the connection does
> not prevent you from bringing down the IO size, if you need to do that for
> other reasons (interference with short requests).  You can bring the IO
> size down and still (with more requests outstanding) cover the latency of
> a (for example) cross-country link.

Ok, now I think I see what your argument is and it sounds interesting.
Are you thinking that the I/O size might be dynamically adjusted via some
information from the transport layer, such as TCP send window size, or
just that the user would find small read/write size settings on the mount
more appropriate or...?

> I think the write-behind case is precisely parallel.

Yep, except that I think write traffic tends to be burstier, although I'm
not sure if that's relevant here. (Maybe more likely to saturate a net
connection during a burst?)

> That's true.  However, there is a clear benefit in giving the smaller
> requests priority (in the absence of any other information establishing
> an inherent priority for these requests).  If I give a small request 
> priority, I don't delay the large request very much at all, while the 
> reverse decision will delay the small request a lot.  So if you don't
> have any other priority information, small first is good.  The process
> may have to wait when he does a large request, but still handling the
> requests with little data first is a net win.   This is especially true
> since reads and writes comes in bunches (read-ahead/write-behind to 
> optimize things).

Well, I'd agree that there's probably cases where the above is true.
A user doing a large software build in one window can still "ls -l"
with a decent response time in another window, might be one.
However, I still think there's a catch-22 here.

If the network connection
isn't anywhere near saturation, then I'd agree that prioritizing the
requests will result in higher throughput (assuming you've defined it as
RPCs/unit time) and should work just fine. Since the network connection
isn't near saturation, I'm not sure the effect will be a big win, though.

On the other hand...
If the network connection is near saturation, you run the risk of introducing
large delays to the large requests, to the point where they will appear to
be completely unresponsive. We had an amusing case of this just a couple
of weeks ago here at Guelph.

We have approximately 5000 students living on campus, each with LAN connections.
Friday afternoon a lot of complaints came into the network help line of slow
response for web access, so one of the net guys decided to up the priority of
http packets (they recently got new Cisco switches, so they became capable
of controlling priority of packets recently). Shortly thereafter, the campus
email hub started to act strangely, with several hundred sendmail connections
hanging around.
Basically the change resulted in the campus email system coming to a grinding
halt, since sendmail packets were being delayed indefinitely while the http
packets saturated our internet connection, since they had priority.
(Of course, when the guy who runs the campus mail server asked the net
guys "Is there anything wrong with the network? You haven't changed anything,
have you?", they responded with "Nope, aok and nothing's changed", which made
the eventual resolution of this rather entertaining for me, given I wasn't
a part of it:-)

The catch is that lower priority packets wait a loonnnggg time (theoretically
approaching infinity) as the net connection approaches saturation. As such,
I'm a bit shy of any priority scheme.

Well, definitely interesting. Have a good week, rick

---

From: Brent Callaghan <[email-masked]>
Date: Mon, 09 Dec 2002 16:21:51 -0800

[email-masked] wrote:
 >  :

Yes, there are priority schemes that can get you into trouble.

Though a scheme that prioritizes packets according to their
size seems eminently fair.  That's essentially what you have
with multiple connections: even though the bandwidth is shared
and finite, shorter packets can slip by the longer ones, simply
because long packets take longer to transmit.

	Brent