Subject: RE: [rddp] Stream Inititation (8.1)

From: "Culley, Paul" <[email-masked]>
Date: Sun, 29 Sep 2002 10:12:58 -0500

How about the following:

1) Active side ULP LLP-sends the ULP specific switch msg.

2) Passive side ULP LLP-receives it.

3) Passive side enables RDMA mode, for RECEIVE only.

4) Passive side then sends a streaming mode response, saying "ok lets do RDMA, I'm in receive mode".  Alternately, the passive can say, in streaming mode, "no, no RDMA today", and not enable RDMA receive.

5) Active side ULP LLP-receives the response switch msg.

6) Active side enables RDMA mode immediately (both
send/recv).

7) Active side performs first RDMA mode transfer. This
serves as an ack of the Passive sides switch msg.

8) When passive side gets RDMA message, it enables the Transmit half of its RDMA connection.

9) Both sides are in RDMA mode and may continue the conversations there.

By splitting the passive side RDMA enable into Transmit and Receive halves, you eliminate the potential races.  (This is what will be proposed as the upper control interface for DDP/RDMA connection management, when we get around to it...).

As you say the "Start with TCP connection" is conceptually a bit easier, as both ends know ahead of time to start that way.

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Saturday, September 28, 2002 7:49 PM
To: Shah, Hemal
Cc: [email-masked]; [email-masked]
Subject: RE: [rddp] Stream Inititation (8.1)

On 9/27/02, Shah, Hemal wrote:

>Caitlin,
>
>With switch PDU, the end which sent the switch PDU does
>not send any more data in stream mode. Here is one way to
>do the negotiation that can cleanly transition both ends
>from Stream to RDMA Mode:
>
>
>          End1                                    End2
>
>            |                                          |   
>            |                                          |   Stream
>            |Switch Msg (last msg sent in stream mode) |   Mode  
>Stream      |    -------------------------->           |     
>Mode        |                                          |   
>            |                                          |
>
>Wait for    |                                          |          
>Switch      |                                          | 
>Message     |                                          |
>            |                                          |
>            |                                          |
>            |Switch Msg (last msg sent in stream mode) |
>            |    <-------------------------            |______Switch to RDMA
>Transition  |                                          | RDMA Mode
>to RDMA Mode|                                          |
>____________|                                          | Wait for
>   RDMA Mode|                                          | RDMA Hello   
>            |                                          | Message  
>            |           RDMA Hello                     |    
>            |    -------------------------->           |
>            |                                          |
>            |(both sides have switched to RDMA Mode)   |
>            //   <-------------------------->         //
>                       RDMA Traffic
>

There are a few problems with this:

First, there is no 'RDMA Hello' message defined in either
the DDP or RDMAP drafts.

More fundamentally, it will not work if the only mandate is
that the ULP "enables RDMA mode".

End2 (I'll presume this is the Passive end), sends the
'Switch Msg' and *then* switches to RDMA mode. But this
step is taken asynchroously by the ULP. It is possible,
however unlikely, that End1 could have received the switch
message and already responded.

Unlikely? Not if the ULP is swapped out immediately after
sending the Switch Msg. In any event, connection
establishment should not rely upon luck in process
scheduling.

With some refinement in the local interface, a similar
scheme could be made to work. Sending and Receiving must be
enabled seperately, and tied to specific points in the LLP
stream:

1) Active side ULP LLP-sends the ULP specific switch msg.

2) Passive side ULP LLP-receives it.

3) Passive side enables RDMA mode, for transmit *after* the
following Switch Msg and immediately on the receive side.

4) Passive side ULP LLP-sends its response switch msg.

5) Active side ULP LLP-receives the response switch msg.

6) Active side enables RDMA mode immediately (both
send/recv).

7) Active side performs first RDMA mode transfer. This
serves as an ack of the Passive sides switch msg.

It is important to remember that this is merely *one* way
to enable RDMA mode. The more common method will be merely
to have the ULP peers pre-agree that the connection will be
in RDMA mode with certain characteristics, and simpy so
configure it as soon as the LLP-connection is established
without any LLP exchange of data.

However, I do not believe it is possible to define a robust
method of exchanging stream specific parameters at the LLP
level *without* refining the requirement for the local
interface.

Further, there is still the issue of the difficulty of
handling retransmits of the passive side's switch msg. It
will be difficult to switch back and forth between a
processor based L3/L4 stack and a NIC based DDP stack,
especially if there is a strong desire to leave the
existing processor based stack intact.

A more realistic implementation would switch
*responsibility* immediately on the passive side, but then
have the DDP NIC send its Switch Msg in non-RDMA format.
The DDP NIC be responsible for any retransmits. This would
considerably ease implementation, with the only restriction
being that the ULP would be restricted to a single LLP
segment/message for completing the transition.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Sun, 29 Sep 2002 10:41:42 -0500

On 9/29/02, Culley, Paul wrote:

>4) Passive side then sends a streaming mode response,
>saying "ok lets do RDMA, I'm in receive mode". 
>Alternately, the passive can say, in streaming mode, "no,
>no RDMA 
>today", and not enable RDMA receive.
>
>By splitting the passive side RDMA enable into Transmit
>and Receive halves, you eliminate the potential races. 
>(This is what will be proposed as the upper control
>interface for DDP/RDMA connection management, when we get
>around to it...).
>

Your sequence is slightly better than the one I outlined.
What they share is a) splitting the enabling of receive
and transmit and b) allowing RDMA mode transmit to be
enabled by the first RDMA mode receive.

It may be sufficient for the specification to merely require
those capabilities, suggest one or two possible algorithsm,
and then make it clear that ULPs are free to negotiate any
solution that they care to.

>As you say the "Start with TCP connection" is conceptually
>a bit easier, as both ends know ahead of time to start
>that way.
>
This could also be made to serve virtually all of the time
if we explicitly required that the local interface allow for
deferred configuration of the RDMA capabilities.

Then the ULPs could simply enter RDMA immediately, and then
negotiate using untagged messages of a pre-defined size (or
pre-defined maximum size for SCTP). This benefit is that it
avoids transitioning the connection between two stacks.

There are, however, TCP entry guarding daemons that use the
connection themselves before turning it over to the
designated daemon, and such daemons would require the
conventional stack. An example is Dan Bernstein's tcpserver.
It may be acceptable to simply require an alternate version
be used, but I believe we need to justify all restrictions
placed on establishing an RDMAP enabled connection.

The problems of migrating an active connection between two
stacks is certainly a good start to that justification. It
may even be sufficient.

Caitlin Bestler

---

From: Caitlin Bestler <[email-masked]>
Date: Sun, 29 Sep 2002 12:49:22 -0500

>4) Passive side then sends a streaming mode response,
saying "ok lets do RDMA, I'm in 
>receive mode".  Alternately, the passive can say, in
streaming mode, "no, no RDMA 
>serves as an ack of the Passive sides switch msg.

This can be made even better by requiring that the Active
side not transmit its first RDMA mode message until it has
LLP acked the last pre-RDMA mode message.

This would allow the Passive side to perform a clean
switchover to RDMA transmits without having to worry about
the possible retransmission of pre-RDMA segments.

Caitlin Bestler