Subject: RE: [rddp] Transport Neutral "Connections"

From: "Carrier, John" <[email-masked]>
Date: Thu, 5 Jun 2003 16:40:52 -0700

Hi Caitlin,

I do see the value of what you are suggesting, but don't see how the ULP can
be involved with MPA directly.  A mode frame can easily be inserted as the
first MPA frame in an RDMA connection, but for the options discussed on this
list so far, the information exchanged is strictly between MPA peers.  I
don't see any way for MPA to pass any data up to the ULP directly.  

As Paul points out, in the current RDMAP/DDP model there is streaming mode
and then iWARP mode.  What you are suggesting is a third, intermediate mode.
The problem is that MPA talks directly to DDP and there is no provision for
a ULP to receive data from DDP unless the ULP has pre-posted a receive
buffer.  Pre-posting a receive buffer implies that the RDMA endpoint has
already been created for the ULP.

InfiniBand sends its private data out-of-band via CM messages before the
RDMA connection is established.  Since there isn't a similar out-of-band
mechanism for iWARP, I think the private data exchange should take place
in-band using the LLP connection in streaming mode, before the transition to
iWARP mode, regardless of the whether the LLP is SCTP or TCP.  

--jc

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Thursday, June 05, 2003 9:46 AM
> To: Culley, Paul
> Cc: [email-masked]
> Subject: RE: [rddp] Transport Neutral "Connections"
> 
> 
> On 6/5/03, Culley, Paul wrote:
> 
> >Regarding "private data";
> >
> >DDP/MPA are currently expected to be enabled together,
> >after creating the QP (and associating it with a PD).  The
> >method we expected to use to deal with private data was to
> >have DDP/MPA's ULP (perhaps a bit of code in the IT-API
> >interface for example) be in charge of starting the TCP
> >connections and then, after doing any negotiations needed
> >between endpoints (using streaming mode TCP data), create
> >the QP and transition the connection to DDP/MPA mode.
> >
> >The "private data" would be exchanged in TCP streaming
> >mode (probably using standard sockets calls under the
> >API).  When the Application/API had any necessary stuff
> >completed (like figuring out the correct PD), the QP can
> >be created, buffers posted, and the TCP "connection
> >context" passed to the iWARP stack to start DDP/MPA mode
> >processing.  The Verbs document you referred to supports
> >this type of startup.
> >
> >I think that using the TCP streaming mode prior to MPA
> >enablement is preferred to adding additional features to
> >MPA as you suggest. The difference in flow over your
> >connection establishment sequence would only be that MPA
> >is not involved until the "bind" step.
> >
> >The IT-API group would have to publish a small draft
> >describing the use of the streaming mode and expected data
> >format prior to iWARP startup, but this is little
> >different than other applications or APIs such as SDP etc.
> >
> 
> Requiring Private Data negotiations to be done with the LLP
> forces the ULP to be aware of what the ULP is. With the
> change proposed, the ULP could be indifferent as to whether
> the LLP was SCTP or TCP.
> 
> Do you agree this would be of value?
> 
> Secondly, TCP is not well suited to this purpose. In all of
> the current proposals for pre-RDMA-mode negotiations, the
> two sides have to agree *precisely* on how many bytes of
> Private Data will be exchanged in each direction.
> 
> ULPs that need this data *frequently* have variable
> information needs, or even *optional* private data.
> 
> Including Private Data in the MPA Mode frame solves this
> problem very easily. An MPA frame already has a length.
> A frame to introduce MPA Mode is already called for.
> 
> Further, there is no need to use this Private Data.
> Applications that wanted to use LLP specific methods
> of exchanging private data would be free to do so.
> This proposal *enables* transport neutral connection
> establishment.
> 
> The only benefit I can identify to use of LLP-specific
> streaming mode to negotiate connections is that such
> a negotiation could last several rounds.
> 
> However, neither DAT or the IT-API found multi-round
> negotiations to be useful. The InfiniBand CM protocol
> does not support multiple-rounds of ULP negotiation either.
> 
> The bottom line goal of pre-RDMA negotiations is to enable
> the correct Protection Domain to be selected. This has
> two important characteristics:
> 
> -- Frequenty there is no need to constrain the selection
>    of the Protection Domain. Meaning that many connections
>    require zero Private Data. Incorporating the Private Data
>    as an optional variable length field supports zero length
>    Private Data very inexspensively. Testing the TCP stream
>    to see *if* there is an MPA mode frame present would be
>    difficult. I will point out that the current verbs
>    proposed in draft-hilland do not support *checking* to
>    see if the TCP stream is ready to enter MPA mode.
>    
> -- One message exchange is all that is required to determine
>    any and all Protection Domain and QP/endpoint needs.
>    Other session details can be negotiated after the
>    QP/endpoints are linked.
>    
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 5 Jun 2003 21:16:56 -0500

On 6/5/03, Carrier, John wrote:

>As Paul points out, in the current RDMAP/DDP model there
>is streaming mode and then iWARP mode.  What you are
>suggesting is a third, intermediate mode. The problem is
>that MPA talks directly to DDP and there is no provision
>for a ULP to receive data from DDP unless the ULP has
>pre-posted a receive buffer.  Pre-posting a receive buffer
>implies that the RDMA endpoint has already been created
>for the ULP.

The ULP has authorizes entry to MPA mode and tells the
MPA/stack to expect a confirming MPA Mode frame. A buffer
can easily be provided at that time.

With or without Private Data, any TCP byte stream that
converts to MPA has a series of bytes before MPA mode,
an MPA Mode Frame, and then an MPA mode series of bytes.

That is not changed by the inclusion of Private Data
within the MPA Mode Frame.

Caitlin Bestler