Subject: idea - listio

From: "Wittle, Mark" <[email-masked]>
Date: Thu, 15 Mar 2001 20:12:05 -0800

The Batch IO mechanism defines 3 new operations:
BATCH_SUBMIT, BATCH_PROGRESS_REPORT, and BATCH_COMPLETION.

BATCH_SUBMIT:

The BATCH_SUBMIT operation is used to initiate a set of read
and write requests from and to regular files, in a generalized
version of the Posix listio functionality.  This can be used
as both a synchronous mechanism, as well as an asynchronous
mechanism if call backs are supported on the request path
request.  In asynchronous mode, either completions are reported
by server initiated BATCH_COMPLETION callbacks, or
the client requests completions with a subsequent
BATCH_REPORT_PROGRESS operation.

Each individual i/o request is encapsulated in a
"read_write_request".  Each request is tagged with a
request_id that allows a client to match up completion
notifications to previously-issued requests.

Each read_write_request contains two lists, one of file
regions and another of memory buffers.  The number of elements
in those two lists does not need to match, nor does the size
of the file regions need to match the size of the individual
memory buffers, but the total number of bytes specified in the
file region list must match the total number of bytes
specified in the memory list.  These represent a combined
scatter-gather list, either moving data from the file to the
specified addresses in the case of a read, or from the
addresses to a file in the case of a write.  The server can
decompose the two lists to a single list of transfers of
memory to file regions, and is free to perform any
optimizations.  The decomposed list can be thought of as the
list of transfers defined by dividing all the data to be moved
for a given filehandle at every point that separates one memory
buffer from another or one file region from another.

This list can easily be generated by running a cursor through
each of the file list and memory buffer list provided.  The
condensed description using two lists, however, makes certain
optimizations trivial.  Two such examples are the case of I/O
from a single file region into multiple memory buffers, or
from a single memory buffer into discontinuous segments of a
file.

If the "synchronous" flag is set in the Batch_Submit_Args
structure, then the server treats the request as a single
synchronous operation to be performed immediately.  Once
the entire batch or requests completes (either successfully
or in error), the server sends a single reply containing the
status of each operation.

Otherwise, if "synchronous" is not set, then the procedure
becomes an asynchronous batch submit.  The server immediately
replies to acknowledge receipt of the request list.  The
reply should not contain any completions.  In this case, the
server may use the "msec_window" parameter as a hint as to
the window of time (in milliseconds) within which the batched
requests are expected to be completed.  The client can also
set "num_completions" as a hint to tell the server how many
completions it would like reported at once.  This parameter is
ignored in the synchronous case.

The server is not obligated to complete the requests in the
batch in any specific order or with any atomicity obligations.
Also, if a client submits multiple asynchronous batch
operations, the server may coalesce them and report
completions from multiple batches in the same completion
notification.

BATCH_COMPLETION:

The BATCH_COMPLETION is used by the server to notify the
client that one or more request batch operations has
completed.

For each completed i/o request, the server returns the
request_id as specified by the client along with the
completion status of the operation.  For successfully
completed requests, the server also returns the number of
bytes transferred; for unsuccessfully completed requests,
the server returns 0.

The returned completions may have been initiated by any
number of prior submit operations; they need not come from
a single batch operation.  The server may use this callback
operation to report any number of outstanding completions;
the client may user BATCH_REPORT_PROGRESS to indicate how
many completions it would like to receive in a batch, but
the server is not obligated to that number.  In all cases,
if all outstanding batch IO requests have been completed,
the server should not delay reporting those completions,
regardless of the client's desired batching number.

BATCH_REPORT_PROGRESS:

The BATCH_REPORT_PROGRESS operation allows the client to
request completion notifications from the server, either
synchronously or asynchronously.  The "num_completions"
parameter indicates how many i/o requests the server should
wait to complete before notifying the client.  If
"num_completions" is set to 0, then the server should
immediately report any i/o requests have been completed.

As with BATCH_SUBMIT, if the "synchronous" flag is set,
then the server waits for the appropriate number of
completions before reporting them in the response, or
replies with fewer if no further operations are outstanding. 
If "synchronous" is unset, the server immediately replies
on the operation channel, and later reports the completions
asynchronously using a callback operation.

Rationale
---------
There are three goals for the Batch IO mechanism:
1) support a generalized Posix style listio capability to
support a variety of client-side buffer cache architectures.
2) allow the client to influence the completion rate of batched
asynchronous i/o requests to provide improved buffer management. 
3) provide hints and batch semantics that allow the server to
optimize the overall throughput characteristics of a data 
stream.

Thus, the "num_completions" argument is a way for the client to
create and smooth the i/o request flow rate.  The "msec_window" 
argument is a hint to the server about the throughput rate
requirements that he can optimize within.

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 15 Mar 2001 23:13:26 -0800

"Wittle, Mark" wrote:
>  :

I'm not sure what problem this is addressing.

Currently, NFS clients can "batch" operations via
reads-ahead and writes-behind. The client gets
completion messages via normal RPC replies.
Congestion can be controlled by TCP.

	Brent