Subject: RE: [rddp] Stream Inititation (8.1)

From: "Julian Satran" <[email-masked]>
Date: Sun, 29 Sep 2002 00:22:40 +0300

As the transition to RDMA mode is an agreement between a sender and a
receiver Hemal's outline is the only thing needed.
The question still lingering (aggravated by the unqualified use of stream
initiation is there is apparent requirement for reaching the RDMA "stae"
on both directions. This is hard and not clearly necessary.

If "entering RDMA state is viewed a "half-connection" event there is no
issue anymore.

For this to happen you need 2 elements:

   an agreement to transition
   a transition message for the "half connection" by which the sender
   announces transition to the receiver (transition does not have to happen
   on both hals at the same time)

Julo

                      "Shah, Hemal"                                                                                                         
                      <hemal.shah@intel        To:       "'Caitlin Bestler'" <[email-masked]>, [email-masked], [email-masked] 
                      .com>                    cc:                                                                                          
                      Sent by:                 Subject:  RE: [rddp] Stream Inititation (8.1)                                                
                      [email-masked]                                                                                                     
                      rg                                                                                                                    

                      28/09/02 01:40                                                                                                        

Caitlin,

With switch PDU, the end which sent the switch PDU does not send any more
data in stream mode. Here is one way to do the negotiation that can cleanly
transition both ends from Stream to RDMA Mode:

          End1                                    End2

            |                                          |
            |                                          |   Stream
            |Switch Msg (last msg sent in stream mode) |   Mode
Stream      |    -------------------------->           |
Mode        |                                          |
            |                                          |

Wait for    |                                          |
Switch      |                                          |
Message     |                                          |
            |                                          |
            |                                          |
            |Switch Msg (last msg sent in stream mode) |
            |    <-------------------------            |______Switch to
RDMA
Transition  |                                          | RDMA Mode
to RDMA Mode|                                          |
____________|                                          | Wait for
   RDMA Mode|                                          | RDMA Hello
            |                                          | Message
            |           RDMA Hello                     |
            |    -------------------------->           |
            |                                          |
            |(both sides have switched to RDMA Mode)   |
            //   <-------------------------->         //
                       RDMA Traffic

Hemal

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Friday, September 27, 2002 2:59 PM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Stream Inititation (8.1)

On 9/27/02, [email-masked] wrote:

>Caitlin,
>
>The method for the transition should be left up to the ULP.
>
>One method is:
>
>Both sides negotiate agreement to use RDMAP. When the ULP
>is ready to start transmitting RDMAP (as defined by the
>ULP's negotiation method), the ULP transmits a switch PDU
>which indicates it is the last non-RDMA PDU. When the
>other ULP receives the switch PDU, it turns on RDMAP for
>the receive stream.
>
>There is no need for the transmitter to wait for TCP acks
>to coordinate this.
>
>Regards,
>Pat
>

I'm not following how it can be that simple.

If data is transmitted *after* the switching PDU, what
receive queue will it be placed in? The ULP cannot process
the switch-mode request *instantly*, therefore the following
data must be fenced against this transition. Otherwise it
would be at risk of being processed to the non-RDDP TCP
receive queue rather than by the RDMA engine.

The ULP interpretation of the switch-mode request is
asynchronous. Further, because it is ULP specific, the NIC
has no clue that this is in any way special TCP traffic. The
NIC *should* be optimizing placement of the following bytes
by whatever mechanisms it has for non-TCP placement.

At the minimum, the NIC / stack would need the ability to
recall those bytes, and re-interpret them via the DDP
engine. Such a detailed requirement on non RDMA mode
handling of TCP streams would seem to be a heavy burden on
implementations that would probably hope to simply re-use
existing TCP stacks. In particular, it could be very tricky
for a NIC implemented DDP stack to revoke delivery of bytes
to a software TCP/IP stack.

So, which is preferable, forcing ALL NICs to implement both
RDMA and non-RDMA modes at the same level, or requiring that
the following tranwsmit be fenced with regards to the ack?

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 28 Sep 2002 20:24:41 -0500

On 9/29/02, Julian Satran wrote:

>As the transition to RDMA mode is an agreement between a
>sender and a receiver Hemal's outline is the only thing
>needed. The question still lingering (aggravated by the
>unqualified use of stream initiation is there is apparent
>requirement for reaching the RDMA "stae" on both
>directions. This is hard and not clearly necessary.
>
>If "entering RDMA state is viewed a "half-connection"
>event there is no issue anymore.
>
> an agreement to transition a transition message for the
> "half connection" by which the sender announces
> transition to the receiver (transition does not have to
> happen on both hals at the same time)
>
>

A 'half connection' approach would indeed be an improvement.
However, it does not solve the problem of a ULP defined 
transition packet. The receipt of the RDMA init message does
NOT trigger  RDMA mode, rather it is an inherently
asynchronous response by the receiving ULP. This creates
inherent problems for any scenario other than the two ULPs
agreeing to establish RDMA mode "instantly" on connection
establishment.

To avoid these problems one of three things is required:

-- The NIC must know that the message triggers RDMA mode.
   This is incompatible with a true LLP-mode for the stream.
-- An ack is required from the ULP itself.
-- There must be a method to "undeliver" data to the
   non-RDMA stack when the ULP finally announces the
   shift to RDMA mode ex post facto.
   
The latter scenario is very difficult, and likely requires
the NIC to hold the original L2 frames for a short period.

A different approach that would also work is to eliminate
the concept that a stream can be used for general purpose
LLP traffic and then be converted.

A socket would be created either in LLP mode, configured
RDMA mode or unconfigured RDMA mode.

LLP mode would never be allowed to convert to other modes.
This stream can be handled by legacy stacks without any
special internal interfaces being required.

Configured RDMA mode is the case already covered, where the
ULP peers both comply with an agreement as to the nature of
the RDMA connection established prior to the connection
(whether by out-of-band negotiations or RFC).

Unconfigured RDMA mode would allow each side to send *one*
untagged message (which must fit in a single LLP segment or
message). The ULP MUST respond to this message by either
configuring the RDMA parameters or breaking its
half-connection.

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Mon, 30 Sep 2002 06:59:42 -0700

>    This is incompatible with a true LLP-mode for the stream.

This would require a RNIC to know all ULPs operating over it and for all 
ULPs to agree on a scheme from the start - that would greatly expand the 
scope and the complexity of an implementation.

>-- An ack is required from the ULP itself.

Making it a ULP implementation issue.

>    shift to RDMA mode ex post facto.

This is clearly an implementation issue and does not impact the RDMA / DDP 
/ MPA specifications.

>configuring the RDMA parameters or breaking its

As stated by many, there are many approaches but nothing here begs a change 
in the specifications at hand.

Mike