Subject: RE: [rddp] Transport Neutral "Connections"

From: "Culley, Paul" <[email-masked]>
Date: Thu, 5 Jun 2003 09:33:24 -0500

Regarding "private data";

DDP/MPA are currently expected to be enabled together, after creating
the QP (and associating it with a PD).  The method we expected to use to
deal with private data was to have DDP/MPA's ULP (perhaps a bit of code
in the IT-API interface for example) be in charge of starting the TCP
connections and then, after doing any negotiations needed between
endpoints (using streaming mode TCP data), create the QP and transition
the connection to DDP/MPA mode.

The "private data" would be exchanged in TCP streaming mode (probably
using standard sockets calls under the API).  When the Application/API
had any necessary stuff completed (like figuring out the correct PD),
the QP can be created, buffers posted, and the TCP "connection context"
passed to the iWARP stack to start DDP/MPA mode processing.  The Verbs
document you referred to supports this type of startup.

I think that using the TCP streaming mode prior to MPA enablement is
preferred to adding additional features to MPA as you suggest.
The difference in flow over your connection establishment sequence would
only be that MPA is not involved until the "bind" step.

The IT-API group would have to publish a small draft describing the use
of the streaming mode and expected data format prior to iWARP startup,
but this is little different than other applications or APIs such as SDP
etc.

Paul R. Culley
HP Senior Fellow
281-514-5543

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Wednesday, June 04, 2003 11:07 AM
To: [email-masked]
Subject: [rddp] Transport Neutral "Connections"

It is highly preferable if applications can be indifferent
as to which underlying reliable transport is used as the LLP for
RDMAP/DDP.

Even beyond that, there is a desire by application
developers to apply the same application logic for
RDMA-aware applications no matter which RDMA transport is
used. This has led to the development of the Direct Access Transport
(DAT) Collaboriative's uDAPL and kDAPL, and to the Open Group's ICSC
IT-API.

Support of these goals can be enhanced by a relatively minor change to
both the SCTP adaptation and in the definition of the MPA Mode Frame.

With this change, ULPs would be able to engage in the same message
exchanges prior to having a fully operational "connection" between two
RDMA-enabled endpoints, whether using RDMAP/DDP over MPA/TCP, over SCTP
or even a non-IP transport such as InfiniBand.

This type of uniformity will enable more rapid development
of RDMA-optimized ULPs.

The first issue to be addressed is the need for
pre-connection data exchange.

An RDMA Endpoint (referred to as a Queue Pair, or QP, in InfiniBand and
the draft-hilland proposed iWARP verbs) must be associated with a
Protection Domain. No receive operations may be posted to the endpoint
before it is associated with a Protection Domain. Indeed under both the
InfiniBand and proposed iWARP verbs an endpoint/QP is created within a
Protection Domain.

There are some applications where the choice of Protection Domain is
dependent upon the identity of the remote ULP client. For example, if a
user session requires multiple connections, it is highly desireable for
all of those connections to use a single Protection Domain.

InfiniBand, the DAT APIs and the IT-API all provide for
the active side ULP to provide Provide Data when requesting
a connection. This data is passed to the ULP to allow it
to determine whether to accept the connection, and if so
with which endpoint (and implicitly which Protection
Domain).

The Private Data can also be used to ensure that both ends
of the connection have configured their RDMA endpoints compatibly on
such matters as the RDMA Read capacity. Further ULP-specific uses are
also presumed, such as establishing the identity of the client.

Private Data is also allowed for when accepting the
connection, to allow completion of any negotiation on
RDMA resources and for other ULP reasons.

What is proposed is that iWARP standardize on allowing a
single ULP message to be sent in each direction in the
process of establishing an LLP Connection.

For SCTP: the stream reset message would allow variable
length Private Data at the end of the chunk. The local interface SHOULD
report this data to the ULP when reporting the stream's state
transition.

An additional Adaptation Layer Control function code would
be defined to explicitly terminate, but not re-enable,
a stream. The Local interface SHOUD report this, and any included
Private Data, to the ULP when reporting the stream's state transition.

For MPA/TCP: the MPA Mode Frame would be defined to be an
MPA Frame, which allowed the same variable length Private
Data at the end of the frame. The local interface SHOULD
report this data to the ULP when reporting the connections state
transition.

For SCTP, a "connection" could be established as follows:

- The passive side would listen on the destination port,
  to indicate its readiness to accept an RDMAP/DDP mode
  association.
- The active side ULP would request a connection from a
  RDMA endpoint to a destination address and port.
- If there was no association already established to
  that SCTP endpoint, the SCTP association would be
  established as with normal SCTP associations, except
  that the Adaptation indication would specifcy RDMAP/DDP.
- The active side would select an unused Stream, and
  send a Stream Reset Data Chunk on that stream, including
  any ULP supplied Private Data.
- The passive side listener would receive the reset chunk.
  It would notify the ULP of the requested "connection",
  including any passed Private Data.
- To accept the connection request, the ULP would bind
  the SCTP stream to an RDMA endpoint, and provide
  the Private Data Response. An Adaptatio Layer Control
  data chunk with a Function Code of DDP Stream Reset would
  be sent to re-initialize the reverse stream and accept
  the "connection" request.
- To reject the connection request, the passive side
  would send an Adaptation Layer Control data chunk
  with a Function Code of DDP Stream Terminate, including
  any ULP supplied Private Data.

For MPA/TCP, a connection could be established in virtually
the same way:

- The passive side would listen on the destination port,
  to indicate its readiness to accept a connection.
- The active side ULP would request a connection from a
  RDMA endpoint to a destination address and port.
- The active side would initiate a TCP connection to the
  destination port. Acceptance/rejection of the connection
  would proceed as per normal TCP connection establishment.
  This would allow normal TCP gatekeepers, such as INETD
  and TCPserver, to exercise their normal safeguard/logging
  functions.
- Both sides would indicate their expectation to receive
  an MPA Mode frame, and thereby enter MPA mode.
- The active side would send an MPA Mode Frame including
  any ULP supplied Private Data.
- The passive side listener would receive the MPA Mode
  Frame. Notice of this state transisiton would be reported
  to the Consumer, including the supplied Private Data.
- To accept the connection request, the ULP would bind
  the TCP connections to an RDMA endpoint, and provide
  the Private Data Response. An MPA Mode Frame would
  be sent to confirm that this was to be an MPA mode
  connection, along with any ULP supplied Private data.
- To reject the connection request, the passive side
  would send an incompatible MPA Mode Frame response, 
  including any ULP supplied Private Data.
  
Note that under both protocols exchange of Private Data is strictly
optional. Neither the Adaptation Layer Control data chunk or the MPA
Mode Frame is required to contain Private Data if the ULP has no desire
to send it. When the ULP has no desire to exchange Private Data, the ALC
DDP Stream Reset or MPA Mode Frames may be sent immediately by both
ends. There is no need to wait for the active side "request" to arrive
first.

The alternative to this approach is the conduct these negotiations using
the LLP alone. SCTP, even in RDMAP/DDP adaptation mode, can still send
and receive user messages. A TCP connection can still send and receive
byte streams.

This approach presents a challenge, however. The ULP using
TCP must consume the correct number of "negotiation" bytes
*before* transitioning to MPA and binding the TCP connection with an
RDMA endpoint. If the ULP would prefer a variable length, or even
optional, Private Data exchange this can become a very tricky process.

More importantly it is a *different* process than the
exchanges required under SCTP or any other RDMA transport.

I believe the benefits of allowing uniform "connection" establishment
across reliable IP transports justify the addition of variable length
Private Data fields to the DDP Stream Reset ALC data chunk and MPA Mode
Frame.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 5 Jun 2003 11:45:44 -0500

On 6/5/03, Culley, Paul wrote:

>DDP/MPA are currently expected to be enabled together,
>after creating the QP (and associating it with a PD).  The
>method we expected to use to deal with private data was to
>have DDP/MPA's ULP (perhaps a bit of code in the IT-API
>interface for example) be in charge of starting the TCP
>connections and then, after doing any negotiations needed
>between endpoints (using streaming mode TCP data), create
>the QP and transition the connection to DDP/MPA mode.
>
>The "private data" would be exchanged in TCP streaming
>mode (probably using standard sockets calls under the
>API).  When the Application/API had any necessary stuff
>completed (like figuring out the correct PD), the QP can
>be created, buffers posted, and the TCP "connection
>context" passed to the iWARP stack to start DDP/MPA mode
>processing.  The Verbs document you referred to supports
>this type of startup.
>
>I think that using the TCP streaming mode prior to MPA
>enablement is preferred to adding additional features to
>MPA as you suggest. The difference in flow over your
>connection establishment sequence would only be that MPA
>is not involved until the "bind" step.
>
>The IT-API group would have to publish a small draft
>describing the use of the streaming mode and expected data
>format prior to iWARP startup, but this is little
>different than other applications or APIs such as SDP etc.
>

Requiring Private Data negotiations to be done with the LLP
forces the ULP to be aware of what the ULP is. With the
change proposed, the ULP could be indifferent as to whether
the LLP was SCTP or TCP.

Do you agree this would be of value?

Secondly, TCP is not well suited to this purpose. In all of
the current proposals for pre-RDMA-mode negotiations, the
two sides have to agree *precisely* on how many bytes of
Private Data will be exchanged in each direction.

ULPs that need this data *frequently* have variable
information needs, or even *optional* private data.

Including Private Data in the MPA Mode frame solves this
problem very easily. An MPA frame already has a length.
A frame to introduce MPA Mode is already called for.

Further, there is no need to use this Private Data.
Applications that wanted to use LLP specific methods
of exchanging private data would be free to do so.
This proposal *enables* transport neutral connection
establishment.

The only benefit I can identify to use of LLP-specific
streaming mode to negotiate connections is that such
a negotiation could last several rounds.

However, neither DAT or the IT-API found multi-round
negotiations to be useful. The InfiniBand CM protocol
does not support multiple-rounds of ULP negotiation either.

The bottom line goal of pre-RDMA negotiations is to enable
the correct Protection Domain to be selected. This has
two important characteristics:

-- Frequenty there is no need to constrain the selection
   of the Protection Domain. Meaning that many connections
   require zero Private Data. Incorporating the Private Data
   as an optional variable length field supports zero length
   Private Data very inexspensively. Testing the TCP stream
   to see *if* there is an MPA mode frame present would be
   difficult. I will point out that the current verbs
   proposed in draft-hilland do not support *checking* to
   see if the TCP stream is ready to enter MPA mode.
   
-- One message exchange is all that is required to determine
   any and all Protection Domain and QP/endpoint needs.
   Other session details can be negotiated after the
   QP/endpoints are linked.

Caitlin Bestler