Subject: RE: [rddp] Transport Neutral "Connections"

From: [email-masked]
Date: Fri, 6 Jun 2003 14:46:28 -0600

Caitlin,

I do not understand why you say: 
Secondly, TCP is not well suited to this purpose. In all of
the current proposals for pre-RDMA-mode negotiations, the
two sides have to agree *precisely* on how many bytes of
Private Data will be exchanged in each direction.

There is no need for the two sides to agree on the size of the pre-RDMA mode message(s). It is entirely possible to define a pre-RDMA negotiation that allows for variable size Private Data if that is desired. All that is necessary is that one have some means of knowing when one has reached the end of the not MPA data so that one can start MPA/DDP/RDMAP.

One already needs to support operation over TCP directly and operation over the iWARP stack over TCP. If ther has to be a point when MPA is operating but DDP/RDMAP is not, then we have to support and test an additional operating mode. I don't see what is gained by it.

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Thursday, June 05, 2003 9:46 AM
To: Culley, Paul
Cc: [email-masked]
Subject: RE: [rddp] Transport Neutral "Connections"

On 6/5/03, Culley, Paul wrote:

>Regarding "private data";
>
>DDP/MPA are currently expected to be enabled together,
>after creating the QP (and associating it with a PD).  The
>method we expected to use to deal with private data was to
>have DDP/MPA's ULP (perhaps a bit of code in the IT-API
>interface for example) be in charge of starting the TCP
>connections and then, after doing any negotiations needed
>between endpoints (using streaming mode TCP data), create
>the QP and transition the connection to DDP/MPA mode.
>
>The "private data" would be exchanged in TCP streaming
>mode (probably using standard sockets calls under the
>API).  When the Application/API had any necessary stuff
>completed (like figuring out the correct PD), the QP can
>be created, buffers posted, and the TCP "connection
>context" passed to the iWARP stack to start DDP/MPA mode
>processing.  The Verbs document you referred to supports
>this type of startup.
>
>I think that using the TCP streaming mode prior to MPA
>enablement is preferred to adding additional features to
>MPA as you suggest. The difference in flow over your
>connection establishment sequence would only be that MPA
>is not involved until the "bind" step.
>
>The IT-API group would have to publish a small draft
>describing the use of the streaming mode and expected data
>format prior to iWARP startup, but this is little
>different than other applications or APIs such as SDP etc.
>

Requiring Private Data negotiations to be done with the LLP
forces the ULP to be aware of what the ULP is. With the
change proposed, the ULP could be indifferent as to whether
the LLP was SCTP or TCP.

Do you agree this would be of value?

Secondly, TCP is not well suited to this purpose. In all of
the current proposals for pre-RDMA-mode negotiations, the
two sides have to agree *precisely* on how many bytes of
Private Data will be exchanged in each direction.

ULPs that need this data *frequently* have variable
information needs, or even *optional* private data.

Including Private Data in the MPA Mode frame solves this
problem very easily. An MPA frame already has a length.
A frame to introduce MPA Mode is already called for.

Further, there is no need to use this Private Data.
Applications that wanted to use LLP specific methods
of exchanging private data would be free to do so.
This proposal *enables* transport neutral connection
establishment.

The only benefit I can identify to use of LLP-specific
streaming mode to negotiate connections is that such
a negotiation could last several rounds.

However, neither DAT or the IT-API found multi-round
negotiations to be useful. The InfiniBand CM protocol
does not support multiple-rounds of ULP negotiation either.

The bottom line goal of pre-RDMA negotiations is to enable
the correct Protection Domain to be selected. This has
two important characteristics:

-- Frequenty there is no need to constrain the selection
   of the Protection Domain. Meaning that many connections
   require zero Private Data. Incorporating the Private Data
   as an optional variable length field supports zero length
   Private Data very inexspensively. Testing the TCP stream
   to see *if* there is an MPA mode frame present would be
   difficult. I will point out that the current verbs
   proposed in draft-hilland do not support *checking* to
   see if the TCP stream is ready to enter MPA mode.
   
-- One message exchange is all that is required to determine
   any and all Protection Domain and QP/endpoint needs.
   Other session details can be negotiated after the
   QP/endpoints are linked.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 6 Jun 2003 16:25:04 -0500

On 6/6/03, [email-masked] wrote:

>>Secondly, TCP is not well suited to this purpose. In all
>>of the current proposals for pre-RDMA-mode negotiations,
>>the two sides have to agree *precisely* on how many bytes
>>of Private Data will be exchanged in each direction.

MPA mode must be entered at a precise byte within the TCP
stream. Further, this cannot be correctly done based upon
timing, boundaries of TCP segments, or other artifacts
no matter how likely they are.

It is totally reasonable for one side to send its final
pre-RDMA mode negotiation data, and then send its MPA 
Mode frame followed by its first DDP segment. They could
even end up being in the same TCP segment.

The proposed iWarp verbs in draft-hilland even provide
a single verb to send a last streaming mode message and
then enter MPA mode. So bundling of at least two of those
operations is in fact likely.

The ULP would have to use the plain-TCP stack to consume
*exactly* the correct number of bytes. Then it has to
tell the stack to shift to MPA mode, consuming the MPA
Mode frame as confirmation, and *then* MPA processing
is fully enabled. The various complaints about "three modes"
all seem to ignore that it is the existence of the MPA Mode
Frame that forces this, the presence of Private Data within
the frame changes nothing.

Anyway, how does the receiving end know how many bytes
to consume? After extensive discussion within the
DAT Collaborative and the Opengroup's ICSC it was
agreed that the only method that worked with the current
definitions was for the ULP to *read* the Private Data
a TCP stream -- and that simulating this interface
for SCTP and InfiniBand would be unnatural.

More importantly, it probably increases the number of
kernel/user transitions. The ULP is awoken with notice
of the TCP connection. It then has to read the TCP
connection, and *then* bind the TCP connection to an
RDMA endpoint/QP.

The procedure I proposed would provide a single notice
to the ULP that an iWARP connection had been requested
(whether on a pair of SCTP Streams or on a TCP connection).
That notice would include any collected Private Data.
The ULP would then either accept or reject the request.
If accepted the endpoint/QP is specified, and optional
response Private Data is supplied.

This is *always?* two user/kernel interactions. Compare
that with the "consume the TCP stream yourself" approach:

-- If the ULP knows there is no Private Data, it's a
   breakeven. Two user/kernel interactions will work.
-- If the ULP knows there will be a fixed amount of
   Private Data it can read that, then respond. That's
   three user/kernel interactions. An increase of one.
-- If the Private Data has a variable length, the ULP
   must first read the minimum length, and *then*
   read the additional length. That's four user/kernel
   interactions, or an increase of two.

A standard protocol, whether integrated into MPA or not,
would allow the kernel/hardware to complete the entire
process of accepting the the TCP connection, reading the
Private Data, and presenting a notice to the ULP. Further,
that same interface could be supported for SCTP.

The ULP requires fewer interactions with the kernel
*and* maintains transport neutrality.

The current MPA solution only looks simple if you
assume applications have nothing negotiate AND that
they are written over TCP. That is *not* an acceptable
assumption to make in designing iWARP. 

>
>There is no need for the two sides to agree on the size of
>the pre-RDMA mode message(s). It is entirely possible to
>define a pre-RDMA negotiation that allows for variable
>size Private Data if that is desired. All that is
>necessary is that one have some means of knowing when one
>has reached the end of the not MPA data so that one can
>start MPA/DDP/RDMAP.
>
But suggest a negotiation would be a) ULP specific and
b) LLP specific. Neither is desirable. Neither matches
the procedures that a ULP would use to establish a
connection between RDMA endpoints over other transports.

The impact on MPA and the propoed verbs is actually quite
slight. Variable length Private Data must be added to an
MPA Mode Frame that will be required anyway. Secondly
there is a difference between enabling for MPA receive
and final binding to the iWARP endpoint/QP.

Such a two-step enabling process is already described
in draft-hilland.

Caitlin Bestler