Subject: RE: [rddp] Transport Neutral "Connections"

From: "Culley, Paul" <[email-masked]>
Date: Fri, 6 Jun 2003 10:45:46 -0500

Below...

Paul R. Culley
HP Senior Fellow
281-514-5543

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]] 
> Sent: Thursday, June 05, 2003 9:12 PM
> To: Culley, Paul
> Cc: [email-masked]
> Subject: RE: [rddp] Transport Neutral "Connections"
> 
> 
> On 6/5/03, Culley, Paul wrote:
> 
> >
> >Requiring Private Data negotiations to be done with the
> >LLP forces the ULP to be aware of what the ULP is. With
> >the change proposed, the ULP could be indifferent as to 
> whether the LLP 
> >was SCTP or TCP.
> >
> >Do you agree this would be of value?
> ><prc> I agree that the ultimate ULP (the Application)
> >should not see any difference; that is the point of a good
> >API (to hide differences in HW or SW).  However, I believe that OS's 
> >generally use some "shim" code to mask differences in the various 
> >interfaces that are below it. I doubt very much that we can 
> (or should) 
> >hide all of the differences between iWARP and InfiniBand 
> (for instance) 
> >in the MPA layer, and am proposing that we should not even
> >try to hide all of the differences between TCP and SCTP,
> >at least not in MPA.  I could see someone creating a
> >rather simple ietf draft doing what you describe using TCP
> >streaming, before converting to iWARP mode, if the IT-API
> >guys don't want to write it. <prc>
> >
> A middleware API cannot "hide the difference" without
> either creating fixed size Private Data messages (losing 
> flexibility), defining a protocol to define the length (and 
> running the risk that each ULP will adopt a different 
> protocol), or dynamically involve the ULP.
If the API (or another draft standard) "defines a protocol to define the
length" then each ULP that uses that API will use the same protocol.
Those that don't use the API may also use the standard (or not as
desired).  I just believe that MPA is not the right place to make that
standard, because it adds functionality to MPA that is not always needed
and adds extra stuff to the proposed "verbs". 
> 
> The natural, and existing, API is to exchange one message
> in each direction. The fuzziness of "message" under TCP
> is the core of the problem.
> 
> 
> 
> >Secondly, TCP is not well suited to this purpose. In all
> >of the current proposals for pre-RDMA-mode negotiations,
> >the two sides have to agree *precisely* on how many bytes
> >of Private Data will be exchanged in each direction.
> >
> >ULPs that need this data *frequently* have variable 
> information needs, 
> >or even *optional* private data.
> >
> >Including Private Data in the MPA Mode frame solves this 
> problem very 
> >easily. An MPA frame already has a length. A frame to introduce MPA 
> >Mode is already called for.
> 
> ><prc> The mode frame I am currently thinking about is
> >simply declarative; it tells MPA how to operate (or
> >whether to close the connection).  What you describe
> >requires that the ULP also get involved between the
> >reception of the MPA mode frame, and the full activation
> >of DDP/MPA.  The ULP would have to see the frame's private data (if 
> >any), decide what to do, and then somehow migrate the "partial MPA" 
> >mode to a full iWARP connection and QP. It creates a new mode of MPA 
> >(not really related to full iWARP mode) just for this purpose. <prc>
> >
> It would stll be a declarative frame.
> 
> The steps do not change that much either.
> 
> The passive side does the following:
> 
> 1)  Listen for a TCP connection on the port.
> 2)  When the socket is delivered, enable MPA receive.
> 3)  Wait for a transition/event indicating that an
>     MPA mode frame has been received (or an error if
>     it was not, but then you teardown and quit).
>     This notice/event includes any Private Data that
>     had been within the MPA Mode frame.
> 4)  Select/configure the endpoint/QP, supplying any
>     response Private Data. This fully enters MPA mode
>     and binds the TCP connection to the endpoint/QP.
>     The MPA Mode Frame is sent with the Private Data.
> 
> The active side is even simpler:
> 
> 1)  Initiate the TCP connection.
> 2)  Once the TCP connection is established, bind it
>     to endpoint/QP supplying the Private Data. The
>     MPA Mode Frame is sent with any supplied Private Data.
> 3)  Wait for the MPA Mode Frame to confirm. Report any
>     Private Data received to the ULP.
Does this mean that the active side cannot send its first messages until
the private data is returned?  Or that the active side does not actually
need the private data to send the first message?

In the former case, you are mandating a full two way exchange (a round
trip) prior to the active side's first iWARP message.  The passive side
can send after receiving the mode frame.  This is probably a new topic
worthy of separate discussion...
> 
> The currently proposed iWARP verbs already describe two
> stage entry to MPA mode.
> 
> If a ULP does not require Private Data, then each side can 
> simply send their MPA Mode Frames and their off and running. 
> There is no penalty for allowing for Private Data.
> 
> This proposal also deals with *optional* Private Data 
> naturally. This is not something that can be dealt with using 
> pre-MPA mode TCP exchanges. Actual bytes must be sent to say 
> "sorry no Private Data supplied" or the ULP might 
> accidentally consume the MPA Mode frame in normal streaming mode.
I disagree here; it does not matter if MPA describes how we deal with
the private data length, or some other draft that moves bytes over
streaming mode.  Keep in mind that I like your basic idea, I just don't
think it needs to be part of MPA...
> 
> Remember that stack cannot prevent the ULP from consuming
> the MPA Mode Frame in streaming mode. There is no way to
> know that this is not valid streaming mode data.
This is a problem in all cases where any streaming data is allowed prior
to entering iWARP mode; that is why the verbs specification is so
careful to describe the one way that MUST be supported by an interface.

> 
> 
> 
> >
> >However, neither DAT or the IT-API found multi-round 
> negotiations to be 
> >useful. The InfiniBand CM protocol does not support 
> multiple-rounds of 
> >ULP negotiation either.
> 
> ><prc> There is at least one application in the works that
> >does use a multi-round negotiation prior to iWARP
> >startup...  Of course it is written to use TCP streaming
> >mode on a legacy port for this. <prc>
> 
> Which is *exactly* what we need to avoid. An iWARP
> application SHOULD NOT be aware of which LLP is in use.
> 
> The need to configure an endpoint/QP properly is intrinsic
> to iWARP. iWARP should provide the means to negotiate it.
It does; a streaming mode, prior to iWARP, where you can do anything...
> 
> Note that there are *NOT* three states for the TCP
> connection. There is a pre-MPA mode, and a post-MPA mode.
> In both cases a single MPA frame is the delimiter. The
> only question is whether Private Data that is directly
> related to enabling RDMA mode communication is communicated 
> within that delimiting packet, or is communicated prior to 
> MPA mode in a way that undermines LLP transparency.
Semantics... The iWARP stack would have to have three modes; pre-MPA,
MPA-Frame with private data reception, and full iWARP.
> 
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 6 Jun 2003 11:09:23 -0500

On 6/6/03, Culley, Paul wrote:

>Does this mean that the active side cannot send its first
>messages until the private data is returned?  Or that the
>active side does not actually need the private data to
>send the first message?
>
That is the current definition for both the DAT and IT-API.
This is of course based upon the need to be transport
neutral. You cannot do your first send under InfiniBand
until the CM response is recieved for the simple reason
that you don't know the destination address.

There is no reason for iWARP to impose that restriction
however. Each side would be able to send immediately after
sending their MPA Mode Frame if their ULP feels no need
to receive the Private Data.

I'm already working on how to evolve the DAT/IT-API model
to improve the API interactions to be compatible with this
iWARP capability while still working for InfiniBand. It
will probably involve allowing the ULP to post writes in
this stage, but not guaranteeing they will be sent before
getting the response from the other side. [Actually, because
of flow-control, there is never a guarantee on when things
will be sent, so that will probably work fairly well.]

>In the former case, you are mandating a full two way
>exchange (a round trip) prior to the active side's first
>iWARP message.  The passive side can send after receiving
>the mode frame.  This is probably a new topic worthy of
>separate discussion...

Agreed. Applications that do not require a two-way Private
Data exchange should not be forced into doing so merely
because InfiniBand required a two-way handshake.

As an objective, the recomended iWARP connection
establishment procedures should work for ULPs that:

-- require NO exchange of Private Data.
-- require Private Data only from the Active side.
-- require an exchange of Private Data.

>I disagree here; it does not matter if MPA describes how
>we deal with the private data length, or some other draft
>that moves bytes over streaming mode.  Keep in mind that I
>like your basic idea, I just don't think it needs to be
>part of MPA...

Then where else would you recomend solving it? An additional
mapping protocol for "Connection Establishment Messages"
to be mapped to both SCTP and TCP?

Wouldn't including a Private Data field in existing messages
be much simpler?

>This is a problem in all cases where any streaming data is
>allowed prior to entering iWARP mode; that is why the
>verbs specification is so careful to describe the one way
>that MUST be supported by an interface.

But it is simpler when Private Data is included in the
existing adaptation layer messages. The ULPs merely have
to ensure that they have reach a quiet state on the TCP
connection before beginning work.

Without Private Data Support, the ULPs must perform this
shift while having messages in flight. Indeed, it is
likely that the Private Data and the trailing MPA Mode
Frame could end up within the very same TCP segment.

>Semantics... The iWARP stack would have to have three
>modes; pre-MPA, MPA-Frame with private data reception, and
>full iWARP.

Are you stating that MPA cannot be enabled distinctly from
iWARP? Isn't that a severe layering violation?

Caitlin Bestler