Subject: re: NFSv4 over SCTP (actually wrt TCP)

From: [email-masked]
Date: Fri, 6 Dec 2002 15:57:43 -0500 (EST)

Good, interesting post, although I'll choose to be skeptical about a couple
of your statements:-)

> ...
> Similarly,
> if 32K is too large, then smaller might be better, especially since
> the large requests are typically handled asynchronously, making
> the latency issue less pressing.
  
The experiments I did (admittedly quite a while ago), didn't really show
a lot of asynchronous behaviour (at least in the sense that the process
wasn't affected by the latency of asynchronous I/O).

For reading, I found that clients usually caught up with read-aheads
(at the time I tried a read-ahead of up to 4 blocks) and still ended up
waiting for the reads to complete. (Small files were just a couple of
reads, but the process still had to wait for them.)

For write-behinds, the process usually hummed along just fine until it
did a close(2), then it was forced to wait for all the delayed writes.
(BSD and I think most Unix clients did this so that write errors would
 be reported back to the process on the client.)

Now, having said the above, with machines that can afford to keep very
large caches (including on local disks) and with V4's delegation, it's
a whole new ball game. (I hope to explore this someday, once there is
a solid BSD client and server to play with?)

> A series of large messages generated
> by read-ahead or write-behind can effectively block the
> link to more interactive traffic.

I would argue that most processes have to wait for reads and writes anyhow.
(I'm assuming a typical desktop system with one user running a relatively
 small number of processes, not an old fashioned 100 user time sharing system.)
I think the problem here is that it is still interactive vs non-interactive
processes that might benefit from prioritizing, not large vs small NFS packets.
(Unfortunately, I think it is very difficult for the NFS client layer to
 differentiate between a non-interactive process doing something like a backup
 vs a make that the user is waiting for. Both of these would do a lot of reads
 and writes. It's even hard for the NFS layer to recognize something like
 "ls -lR", which seems should be given higher priority for its NFS
 ops than a backup or make.)

I would actually conjecture that the fuller (larger average send queue size)
a TCP connection, the more efficiently the data can be transferred without
congestion. (By efficiently here, I am suggesting a fair use of the bottleneck
link(s) without congestion occurring.) If this is the case, I see an argument
for one TCP connection being used for all traffic between a given client and
server? (Van Jacobson's algorithms detect congestion via observed packet loss
and this will only happen when transmission is active.)

I would also be concerned that constraints, such as record marks, may be
an impediment on algorithms like Van's congestion avoidance strategy.

Anyhow, thanks for the interesting post, rick

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 9 Dec 2002 14:04:12 -0800

> Good, interesting post, although I'll choose to be skeptical about a couple

I'll return the compliment and also the skepticism :-)

> wasn't affected by the latency of asynchronous I/O).

Whether this is so, depends on your interpretation of "latency".  See
below.

> reads, but the process still had to wait for them.)

Sure.  But what that shows is simply that the application can process
data faster than it can be fetched over the network connection.  Let's
assume Brent's example of a T1-speed connection accessing a server
across the country.  Let's say the network latency is 100 ms.  In that 
environment, if the application is processing data faster that 180KB/s, 
it is going to wind up waiting for its data.  But the problem is not the 
latency but the bandwidth of the connection.  If we moved the server to 
the next room and the network latency went to < 1 ms, you would still be 
waiting for the data and you would only be able to get it at 180KB/s.  
Of course, if your definition of "latency" included the network transfer 
time, then it would be the case that the process was affected by the latency.

Where the asynchronous behavior comes in is in the fact that with
the IO being done asynchronously, the latency of the connection does
not prevent you from bringing down the IO size, if you need to do that for
other reasons (interference with short requests).  You can bring the IO
size down and still (with more requests outstanding) cover the latency of
a (for example) cross-country link.

> be reported back to the process on the client.)

I think the write-behind case is precisely parallel.

> I would argue that most processes have to wait for reads and writes anyhow.

That's true.  However, there is a clear benefit in giving the smaller
requests priority (in the absence of any other information establishing
an inherent priority for these requests).  If I give a small request 
priority, I don't delay the large request very much at all, while the 
reverse decision will delay the small request a lot.  So if you don't
have any other priority information, small first is good.  The process
may have to wait when he does a large request, but still handling the
requests with little data first is a net win.   This is especially true
since reads and writes comes in bunches (read-ahead/write-behind to 
optimize things).

> ops than a backup or make.)

I agree with you.  Doing this on the basis of the user's actual 
preferences (for interactivity and other reasons) would be the best.
But that is, as you note, hard.  We're in a situation in which the cost
of cpu cycles and bytes of memory has continued its asymptotic march to
zero, making prioritization of those resources less of an issue in the
environments you are talking about.  However, there are many cases in 
which a network connection is the important contended-for resource, for
which prioritized treatment would be very beneficial (like the one
I'm typing on now) but clients are not set up to do that.  I suspect
this is partly because prioritization in general has become less of an
issue as CPU becomes virtually free, and partly because OS's have not
been structured to make the necessary information easily available to 
sub-systems (like NFS) judged peripheral.  I don't know whether the 
greater need for this will cause change in this area, or whether people 
will just wait for networks to get faster.  Maybe some of each.

> and this will only happen when transmission is active.)

Certainly if efficiency is the goal, I agree.  The thing is that it doesn't
take into account user perception.  Even without going to message scheduling
based on interactivity, the ability to send a short message without waiting 
for a long one to complete, is going to improve perceived response, although 
not efficiency, although I'd hope it wouldn't hurt efficiency.