Subject: RE: [rddp] Stream Inititation (8.1)

From: "Julian Satran" <[email-masked]>
Date: Sun, 29 Sep 2002 05:28:14 +0300

Caitlin,

Implementation might involve some complexity but I am not sure that the 
RNIC must understand the ULP messages.
What I had in mind is that the local ULP will take care to notify the RNIC 
about transition.

If the stream is "read ahead" then the RNIC might have passed some RDMA 
frames without knowing and some additional things might be required
(pacing, some predefined/empty RDMA frames).

Julo

Caitlin Bestler <[email-masked]>
29/09/02 03:24

        To:     [email-masked], Julian Satran/Haifa/IBM@IBMIL
        cc:     "Shah, Hemal" <[email-masked]>, [email-masked]
        Subject:        RE: [rddp] Stream Inititation (8.1)

On 9/29/02, Julian Satran wrote:

>As the transition to RDMA mode is an agreement between a
>sender and a receiver Hemal's outline is the only thing
>needed. The question still lingering (aggravated by the
>unqualified use of stream initiation is there is apparent
>requirement for reaching the RDMA "stae" on both
>directions. This is hard and not clearly necessary.
>
>If "entering RDMA state is viewed a "half-connection"
>event there is no issue anymore.
>
>For this to happen you need 2 elements:
>
> an agreement to transition a transition message for the
> "half connection" by which the sender announces
> transition to the receiver (transition does not have to
> happen on both hals at the same time)
>
>

A 'half connection' approach would indeed be an improvement.
However, it does not solve the problem of a ULP defined 
transition packet. The receipt of the RDMA init message does
NOT trigger  RDMA mode, rather it is an inherently
asynchronous response by the receiving ULP. This creates
inherent problems for any scenario other than the two ULPs
agreeing to establish RDMA mode "instantly" on connection
establishment.

To avoid these problems one of three things is required:

-- The NIC must know that the message triggers RDMA mode.
   This is incompatible with a true LLP-mode for the stream.
-- An ack is required from the ULP itself.
-- There must be a method to "undeliver" data to the
   non-RDMA stack when the ULP finally announces the
   shift to RDMA mode ex post facto.
 
The latter scenario is very difficult, and likely requires
the NIC to hold the original L2 frames for a short period.

A different approach that would also work is to eliminate
the concept that a stream can be used for general purpose
LLP traffic and then be converted.

A socket would be created either in LLP mode, configured
RDMA mode or unconfigured RDMA mode.

LLP mode would never be allowed to convert to other modes.
This stream can be handled by legacy stacks without any
special internal interfaces being required.

Configured RDMA mode is the case already covered, where the
ULP peers both comply with an agreement as to the nature of
the RDMA connection established prior to the connection
(whether by out-of-band negotiations or RFC).

Unconfigured RDMA mode would allow each side to send *one*
untagged message (which must fit in a single LLP segment or
message). The ULP MUST respond to this message by either
configuring the RDMA parameters or breaking its
half-connection.

Caitlin Bestler
http://asomi.com/CaitlinBestler/

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 28 Sep 2002 21:50:33 -0500

On 9/29/02, Julian Satran wrote:

>Caitlin,
>
>Implementation might involve some complexity but I am not
>sure that the RNIC must understand the ULP messages. What
>I had in mind is that the local ULP will take care to
>notify the RNIC about transition.
>
>If the stream is "read ahead" then the RNIC might have
>passed some RDMA frames without knowing and some
>additional things might be required (pacing, some
>predefined/empty RDMA frames).
>
>Julo
>

Conventional NICs pass L2 frames to a software stack.
It is then parsed, and buffered until the receiving socket
requests the data. Hence "read ahead" is not an "if", it
is almost a certainty if the next frame arrives before the
ULP has had time to respond.

Pacing / empty frames / etc. are just hacks attempting
to finesse the absence of an ack. They are placing a
real-time constraint on the combination of the non-RDMA IP
stack and the ULP. Almost any such timing constraint will
either artificially slow down connection establishment or
fail on certain OSs under certain conditions (such as heavy
thrashing of applications to from swap disk).

Wouldn't it make more sense to define a solution that does
not require finessing a race condition?

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp