Subject: RE: [rddp] Transport Neutral "Connections"

From: "Culley, Paul" <[email-masked]>
Date: Thu, 5 Jun 2003 14:26:24 -0500

Embedded...

Paul R. Culley
HP Senior Fellow
281-514-5543

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Thursday, June 05, 2003 11:46 AM
To: Culley, Paul
Cc: [email-masked]
Subject: RE: [rddp] Transport Neutral "Connections"

Requiring Private Data negotiations to be done with the LLP forces the
ULP to be aware of what the ULP is. With the change proposed, the ULP
could be indifferent as to whether the LLP was SCTP or TCP.

Do you agree this would be of value?
<prc> I agree that the ultimate ULP (the Application) should not see any
difference; that is the point of a good API (to hide differences in HW
or SW).  However, I believe that OS's generally use some "shim" code to
mask differences in the various interfaces that are below it.  I doubt
very much that we can (or should) hide all of the differences between
iWARP and InfiniBand (for instance) in the MPA layer, and am proposing
that we should not even try to hide all of the differences between TCP
and SCTP, at least not in MPA.  I could see someone creating a rather
simple ietf draft doing what you describe using TCP streaming, before
converting to iWARP mode, if the IT-API guys don't want to write it.
<prc>

Secondly, TCP is not well suited to this purpose. In all of
the current proposals for pre-RDMA-mode negotiations, the
two sides have to agree *precisely* on how many bytes of Private Data
will be exchanged in each direction.

ULPs that need this data *frequently* have variable
information needs, or even *optional* private data.

Including Private Data in the MPA Mode frame solves this problem very
easily. An MPA frame already has a length. A frame to introduce MPA Mode
is already called for.
<prc> The mode frame I am currently thinking about is simply
declarative; it tells MPA how to operate (or whether to close the
connection).  What you describe requires that the ULP also get involved
between the reception of the MPA mode frame, and the full activation of
DDP/MPA.  The ULP would have to see the frame's private data (if any),
decide what to do, and then somehow migrate the "partial MPA" mode to a
full iWARP connection and QP.  It creates a new mode of MPA (not really
related to full iWARP mode) just for this purpose.
<prc>

Further, there is no need to use this Private Data. Applications that
wanted to use LLP specific methods of exchanging private data would be
free to do so. This proposal *enables* transport neutral connection
establishment.
<prc> agree with the first part; but must point out that we will then
have to deal with three modes (Streaming mode at start, "pre MPA", and
full iWARP) and the fun of describing the methods of switching between
these modes when apps will be using some combination of them.  I prefer
that for iWARP, we stick with just two modes (streaming, and iWARP) and
let another draft describe the startup needed with the IT-API.
<prc>

The only benefit I can identify to use of LLP-specific streaming mode to
negotiate connections is that such a negotiation could last several
rounds.

However, neither DAT or the IT-API found multi-round negotiations to be
useful. The InfiniBand CM protocol does not support multiple-rounds of
ULP negotiation either.
<prc> There is at least one application in the works that does use a
multi-round negotiation prior to iWARP startup...  Of course it is
written to use TCP streaming mode on a legacy port for this.
<prc>

The bottom line goal of pre-RDMA negotiations is to enable
the correct Protection Domain to be selected. This has
two important characteristics:

-- Frequenty there is no need to constrain the selection
   of the Protection Domain. Meaning that many connections
   require zero Private Data. Incorporating the Private Data
   as an optional variable length field supports zero length
   Private Data very inexspensively. Testing the TCP stream
   to see *if* there is an MPA mode frame present would be
   difficult. I will point out that the current verbs
   proposed in draft-hilland do not support *checking* to
   see if the TCP stream is ready to enter MPA mode.
<prc> The current verbs leaves it to the ULP to perform such checks; for
some applications a streaming mode setup or negotiation is used, for
others outside knowledge (port number, name service etc.) will be used
to determine that DDP/MPA is to be used.

I mostly agree with what you are saying, I just don't want to add this
functionality to MPA; I believe it belongs in one of MPA's ULPs, for the
APIs you describe that would be a thin shim between the Application and
the Verbs.
<prc>

-- One message exchange is all that is required to determine
   any and all Protection Domain and QP/endpoint needs.
   Other session details can be negotiated after the
   QP/endpoints are linked.

Caitlin Bestler
http://asomi.com/CaitlinBestler/

On 6/5/03, Culley, Paul wrote:

>Regarding "private data";
>
>DDP/MPA are currently expected to be enabled together,
>after creating the QP (and associating it with a PD).  The method we 
>expected to use to deal with private data was to have DDP/MPA's ULP 
>(perhaps a bit of code in the IT-API interface for example) be in 
>charge of starting the TCP connections and then, after doing any 
>negotiations needed between endpoints (using streaming mode TCP data), 
>create the QP and transition the connection to DDP/MPA mode.
>
>The "private data" would be exchanged in TCP streaming
>mode (probably using standard sockets calls under the
>API).  When the Application/API had any necessary stuff completed (like

>figuring out the correct PD), the QP can be created, buffers posted, 
>and the TCP "connection context" passed to the iWARP stack to start 
>DDP/MPA mode processing.  The Verbs document you referred to supports
>this type of startup.
>
>I think that using the TCP streaming mode prior to MPA enablement is 
>preferred to adding additional features to MPA as you suggest. The 
>difference in flow over your connection establishment sequence would 
>only be that MPA is not involved until the "bind" step.
>
>The IT-API group would have to publish a small draft describing the use

>of the streaming mode and expected data format prior to iWARP startup, 
>but this is little different than other applications or APIs such as 
>SDP etc.
>
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 5 Jun 2003 21:12:03 -0500

On 6/5/03, Culley, Paul wrote:

>
>Requiring Private Data negotiations to be done with the
>LLP forces the ULP to be aware of what the ULP is. With
>the change proposed, the ULP could be indifferent as to
>whether the LLP was SCTP or TCP.
>
><prc> I agree that the ultimate ULP (the Application)
>should not see any difference; that is the point of a good
>API (to hide differences in HW or SW).  However, I believe
>that OS's generally use some "shim" code to mask
>differences in the various interfaces that are below it. 
>I doubt very much that we can (or should) hide all of the
>differences between iWARP and InfiniBand (for instance) in
>the MPA layer, and am proposing that we should not even
>try to hide all of the differences between TCP and SCTP,
>at least not in MPA.  I could see someone creating a
>rather simple ietf draft doing what you describe using TCP
>streaming, before converting to iWARP mode, if the IT-API
>guys don't want to write it. <prc>
>
A middleware API cannot "hide the difference" without
either creating fixed size Private Data messages (losing
flexibility), defining a protocol to define the length
(and running the risk that each ULP will adopt a different
protocol), or dynamically involve the ULP.

The natural, and existing, API is to exchange one message
in each direction. The fuzziness of "message" under TCP
is the core of the problem.

>Secondly, TCP is not well suited to this purpose. In all
>of the current proposals for pre-RDMA-mode negotiations,
>the two sides have to agree *precisely* on how many bytes
>of Private Data will be exchanged in each direction.
>
>Including Private Data in the MPA Mode frame solves this
>problem very easily. An MPA frame already has a length. A
>frame to introduce MPA Mode is already called for.

><prc> The mode frame I am currently thinking about is
>simply declarative; it tells MPA how to operate (or
>whether to close the connection).  What you describe
>requires that the ULP also get involved between the
>reception of the MPA mode frame, and the full activation
>of DDP/MPA.  The ULP would have to see the frame's private
>data (if any), decide what to do, and then somehow migrate
>the "partial MPA" mode to a full iWARP connection and QP. 
>It creates a new mode of MPA (not really related to full
>iWARP mode) just for this purpose. <prc>
>
It would stll be a declarative frame.

The steps do not change that much either.

The passive side does the following:

1)  Listen for a TCP connection on the port.
2)  When the socket is delivered, enable MPA receive.
3)  Wait for a transition/event indicating that an
    MPA mode frame has been received (or an error if
    it was not, but then you teardown and quit).
    This notice/event includes any Private Data that
    had been within the MPA Mode frame.
4)  Select/configure the endpoint/QP, supplying any
    response Private Data. This fully enters MPA mode
    and binds the TCP connection to the endpoint/QP.
    The MPA Mode Frame is sent with the Private Data.

The active side is even simpler:

1)  Initiate the TCP connection.
2)  Once the TCP connection is established, bind it
    to endpoint/QP supplying the Private Data. The
    MPA Mode Frame is sent with any supplied Private Data.
3)  Wait for the MPA Mode Frame to confirm. Report any
    Private Data received to the ULP.

The currently proposed iWARP verbs already describe two
stage entry to MPA mode.

If a ULP does not require Private Data, then each side can
simply send their MPA Mode Frames and their off and running.
There is no penalty for allowing for Private Data.

This proposal also deals with *optional* Private Data
naturally. This is not something that can be dealt with
using pre-MPA mode TCP exchanges. Actual bytes must be
sent to say "sorry no Private Data supplied" or the
ULP might accidentally consume the MPA Mode frame in
normal streaming mode.

Remember that stack cannot prevent the ULP from consuming
the MPA Mode Frame in streaming mode. There is no way to
know that this is not valid streaming mode data.

>
>However, neither DAT or the IT-API found multi-round
>negotiations to be useful. The InfiniBand CM protocol does
>not support multiple-rounds of ULP negotiation either.

><prc> There is at least one application in the works that
>does use a multi-round negotiation prior to iWARP
>startup...  Of course it is written to use TCP streaming
>mode on a legacy port for this. <prc>

Which is *exactly* what we need to avoid. An iWARP
application SHOULD NOT be aware of which LLP is in use.

The need to configure an endpoint/QP properly is intrinsic
to iWARP. iWARP should provide the means to negotiate it.

Note that there are *NOT* three states for the TCP
connection. There is a pre-MPA mode, and a post-MPA mode.
In both cases a single MPA frame is the delimiter. The
only question is whether Private Data that is directly
related to enabling RDMA mode communication is communicated
within that delimiting packet, or is communicated prior to
MPA mode in a way that undermines LLP transparency.

Caitlin Bestler