Subject: RE: [rddp] Simple implementation of "RDMA mode model 1"

From: "Williams, Jim" <[email-masked]>
Date: Wed, 18 Jun 2003 11:07:53 -0400

I would add that if you are intending to 
transfer the connection mid-stream from
the software TCP stack to a hardware
accelerated implementation, it is likely
you are already modifying the
software TCP, so tweaking
the windowing should not be much incremental
effort.

There is one annoying detail that could
be a problem, though.  If the TCP connection
negotiates window scaling, which may be
needed for high performance, the actual window
size will have some quantization error resulting
in a few extra (or too few) bytes being received.
For instance, if you want 512 bytes and receive
312, you need to acknowledge the 312 and set
the new window to 200.  But if the window
scale factor is 16, this needs to be rounded
up to 208 which is a multiple of 16, or down
to 192.

> has a rather easy solution:
> 
> Upon initial endpoint (socket) creation, each side limits its TCP
> receive window to a small value, say 512 bytes. This is supported
> by most systems through use of the setsockopt(SO_RCVBUF) api.
> (Some systems however do enforce larger minimum buffers, such as
> several KB.)
> 
> Then, provided that the initial "frame" is exactly this many bytes, it
> is trivial for each peer to convey this payload before entering RDMA
> mode. After consuming the frame and using any contents to initialize
> the RDMA endpoint, the receive buffer is raised to a more reasonable
> value by a subsequent setsockopt() at each end.
> 
> There is the issue of new data arriving between the time that 
> the "frame"
> is consumed but before RDMA mode has been entered, however this is
> a local interface issue and in any case wouldn't be a problem 
> for software
> RDMA implementations.
> 
> Also, it requires prior knowledge of the peer's intention to send such
> a frame, which of course implies a well-known port or other 
> prior contract.
> But, it will work and is a reasonable design point, in my opinion.
> 
> By the way, on some systems, a tiny initial window of 1 or 2 bytes, or
> even 0, is permitted. It's within the realm of possibility that this
> technique could be used to initiate RDMA mode without any exchange
> whatsoever.
> 
> I have a test program which demonstrates this over the sockets api if
> anyone wants to try it out.
> 
> Tom.
> 
> 
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Stephen Bailey <[email-masked]>
Date: Sun, 22 Jun 2003 11:11:34 -0400

> the windowing should not be much incremental

While I generally agree this may be possible, I still don't think we
should be messing around with this type of stuff.

A ULP that does not adopt the `turn on immediately' model probably
already has some handshaking, so connection setup is already not
`lightweight' without RDDP/TCP.

It would be nice if the RDDP/TCP handshake could be specified in such
a way that it piggybacks the ULP handshake.  That's what we tried to
do with the original framing drafts: define what must be exchanged,
and how a peer controls exactly when the first RDDPed (framed) data
will come to it.  However, if y'all think it's problematic to allow
this (I understand why), we shouldn't mess with it.  Declare victory
and move on for now.

Steph

---

From: Caitlin Bestler <[email-masked]>
Date: Sun, 22 Jun 2003 11:14:29 -0500

> this (I understand why), we shouldn't mess with it.  Declare victory

There are two categories of negotiation that are fairly simple to deal 
with.

a) turn on immediately. Endpoint configuration is not session specific.
      This can be done simply,  and presented to the ULP in a 
LLP-neutral way.

b) turns on after extensive ULP specific negotiations that are required 
for
     ULP-specific reasons. Once we fully document the rules on the state
    of the connection at the delayed entry point (i.e., what can be 
in-flight,
    what must have already been acked, etc.) this will be fairly simple 
as
    well. The descriptions already floated on this type of procedure are
    fine, they just fail to state all of the simplifying conditions as 
flat rules.

    This can be made LLP-neutral if the sole criteria on whether to enter
    RDMA mode is capability. That is an endpoint can be non-RDMA,
    RDMA-only  or support both. Everything works  as expected, with
    the caveat that if both ends are RDMA capable  then the association
    will be RDMA capable.

There is a third category of program that is relevant though:

c) Session-specific negotiations are required. Further,  these 
negotiations
     are RDMA specific and/or this is a RDMA-only protocol.

RDMA specific  factors that have the  potential of being 
session-specific
include  the number of initial send credits  and the number of allowed
RDMA Read operations.

I believe that c) will be a common model.  In fact it is the *only* 
model
that was developed by the DAT Collaborative. It is also  the model
used in other RDMA protocols such as InfiniBand.

Properly supporting c) under MPA/TCP requires either  including
Private Data in the MPA Mode Frame, or defining  a pre-MPA "frame"
that can be captured by the stack (rather than by the ULP). This
reduces the  ULP interaction to supplying and accepting a "private
data" that accompanies a connection request or response.

Caitlin Bestler
http://asomi.com/