Subject: [rddp] Simple implementation of "RDMA mode model 1"

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 17 Jun 2003 14:49:45 -0400

Of the three options David Black listed, the first,

>I think there are three "models" of how to get into RDMA mode
>under discussion:
> 
>(1) Enter immediately on startup, possibly with an initial frame
>    containing private data required to setup the connection.

has a rather easy solution:

Upon initial endpoint (socket) creation, each side limits its TCP
receive window to a small value, say 512 bytes. This is supported
by most systems through use of the setsockopt(SO_RCVBUF) api.
(Some systems however do enforce larger minimum buffers, such as
several KB.)

Then, provided that the initial "frame" is exactly this many bytes, it
is trivial for each peer to convey this payload before entering RDMA
mode. After consuming the frame and using any contents to initialize
the RDMA endpoint, the receive buffer is raised to a more reasonable
value by a subsequent setsockopt() at each end.

There is the issue of new data arriving between the time that the "frame"
is consumed but before RDMA mode has been entered, however this is
a local interface issue and in any case wouldn't be a problem for software
RDMA implementations.

Also, it requires prior knowledge of the peer's intention to send such
a frame, which of course implies a well-known port or other prior contract.
But, it will work and is a reasonable design point, in my opinion.

By the way, on some systems, a tiny initial window of 1 or 2 bytes, or
even 0, is permitted. It's within the realm of possibility that this
technique could be used to initiate RDMA mode without any exchange
whatsoever.

I have a test program which demonstrates this over the sockets api if
anyone wants to try it out.

Tom.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 17 Jun 2003 15:04:23 -0500

On 6/17/03, Talpey, Thomas wrote:

>Upon initial endpoint (socket) creation, each side
>limits its TCP receive window to a small value, say
>512 bytes. This is supported by most systems through
>use of the setsockopt(SO_RCVBUF) api. (Some systems
>however do enforce larger minimum buffers, such as
>several KB.)
>
>Then, provided that the initial "frame" is exactly
>this many bytes, it is trivial for each peer to convey
>this payload before entering RDMA mode. After
>consuming the frame and using any contents to
>initialize the RDMA endpoint, the receive buffer is
>raised to a more reasonable value by a subsequent
>setsockopt() at each end.
>
>There is the issue of new data arriving between the
>time that the "frame" is consumed but before RDMA mode
>has been entered, however this is a local interface
>issue and in any case wouldn't be a problem for
>software RDMA implementations.
>
>Also, it requires prior knowledge of the peer's
>intention to send such a frame, which of course
>implies a well-known port or other prior contract.
>By the way, on some systems, a tiny initial window of
>1 or 2 bytes, or even 0, is permitted. It's within the
>realm of possibility that this technique could be used
>to initiate RDMA mode without any exchange whatsoever.
>
>I have a test program which demonstrates this over the
>sockets api if anyone wants to try it out.
>

This is very close to what I was envisioning, except
that it could be combined with the MPA Mode "proof"
and it would be at most 512 bytes (or some other small
size), but could be smaller when the ULP has nothing
worth saying before establishing the full RDMA connection.

Of course, we can take InfiniBand's CM protocol as proof
that having a small fixed size is clearly acceptable to
application developers.

Caitlin Bestler
http://asomi.com/CaitlinBestler/