Subject: RE: [rddp] Stream Inititation (8.1)

From: "Shah, Hemal" <[email-masked]>
Date: Fri, 27 Sep 2002 16:40:38 -0700

Caitlin,

With switch PDU, the end which sent the switch PDU does not send any more
data in stream mode. Here is one way to do the negotiation that can cleanly
transition both ends from Stream to RDMA Mode:

          End1                                    End2

            |                                          |   
            |                                          |   Stream
            |Switch Msg (last msg sent in stream mode) |   Mode  
Stream      |    -------------------------->           |     
Mode        |                                          |   
            |                                          |

Wait for    |                                          |          
Switch      |                                          | 
Message     |                                          |
            |                                          |
            |                                          |
            |Switch Msg (last msg sent in stream mode) |
            |    <-------------------------            |______Switch to RDMA
Transition  |                                          | RDMA Mode
to RDMA Mode|                                          |
____________|                                          | Wait for
   RDMA Mode|                                          | RDMA Hello   
            |                                          | Message  
            |           RDMA Hello                     |    
            |    -------------------------->           |
            |                                          |
            |(both sides have switched to RDMA Mode)   |
            //   <-------------------------->         //
                       RDMA Traffic

Hemal

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Friday, September 27, 2002 2:59 PM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Stream Inititation (8.1)

On 9/27/02, [email-masked] wrote:

>Caitlin,
>
>The method for the transition should be left up to the ULP. 
>
>One method is:
>
>Both sides negotiate agreement to use RDMAP. When the ULP
>is ready to start transmitting RDMAP (as defined by the
>ULP's negotiation method), the ULP transmits a switch PDU
>which indicates it is the last non-RDMA PDU. When the
>other ULP receives the switch PDU, it turns on RDMAP for
>the receive stream.
>
>There is no need for the transmitter to wait for TCP acks
>to coordinate this.
>
>Regards,
>Pat
>

I'm not following how it can be that simple.

If data is transmitted *after* the switching PDU, what
receive queue will it be placed in? The ULP cannot process
the switch-mode request *instantly*, therefore the following
data must be fenced against this transition. Otherwise it
would be at risk of being processed to the non-RDDP TCP
receive queue rather than by the RDMA engine.

The ULP interpretation of the switch-mode request is
asynchronous. Further, because it is ULP specific, the NIC
has no clue that this is in any way special TCP traffic. The
NIC *should* be optimizing placement of the following bytes
by whatever mechanisms it has for non-TCP placement.

At the minimum, the NIC / stack would need the ability to
recall those bytes, and re-interpret them via the DDP
engine. Such a detailed requirement on non RDMA mode
handling of TCP streams would seem to be a heavy burden on
implementations that would probably hope to simply re-use
existing TCP stacks. In particular, it could be very tricky
for a NIC implemented DDP stack to revoke delivery of bytes
to a software TCP/IP stack.

So, which is preferable, forcing ALL NICs to implement both
RDMA and non-RDMA modes at the same level, or requiring that
the following tranwsmit be fenced with regards to the ack?

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 28 Sep 2002 19:49:16 -0500

On 9/27/02, Shah, Hemal wrote:

>Caitlin,
>
>With switch PDU, the end which sent the switch PDU does
>not send any more data in stream mode. Here is one way to
>do the negotiation that can cleanly transition both ends
>from Stream to RDMA Mode:
>
>
>          End1                                    End2
>
>            |                                          |   
>            |                                          |   Stream

There are a few problems with this:

First, there is no 'RDMA Hello' message defined in either
the DDP or RDMAP drafts.

More fundamentally, it will not work if the only mandate is
that the ULP "enables RDMA mode".

End2 (I'll presume this is the Passive end), sends the
'Switch Msg' and *then* switches to RDMA mode. But this
step is taken asynchroously by the ULP. It is possible,
however unlikely, that End1 could have received the switch
message and already responded.

Unlikely? Not if the ULP is swapped out immediately after
sending the Switch Msg. In any event, connection
establishment should not rely upon luck in process
scheduling.

With some refinement in the local interface, a similar
scheme could be made to work. Sending and Receiving must be
enabled seperately, and tied to specific points in the LLP
stream:

1) Active side ULP LLP-sends the ULP specific switch msg.

2) Passive side ULP LLP-receives it.

3) Passive side enables RDMA mode, for transmit *after* the
following Switch Msg and immediately on the receive side.

4) Passive side ULP LLP-sends its response switch msg.

5) Active side ULP LLP-receives the response switch msg.

6) Active side enables RDMA mode immediately (both
send/recv).

7) Active side performs first RDMA mode transfer. This
serves as an ack of the Passive sides switch msg.

It is important to remember that this is merely *one* way
to enable RDMA mode. The more common method will be merely
to have the ULP peers pre-agree that the connection will be
in RDMA mode with certain characteristics, and simpy so
configure it as soon as the LLP-connection is established
without any LLP exchange of data.

However, I do not believe it is possible to define a robust
method of exchanging stream specific parameters at the LLP
level *without* refining the requirement for the local
interface.

Further, there is still the issue of the difficulty of
handling retransmits of the passive side's switch msg. It
will be difficult to switch back and forth between a
processor based L3/L4 stack and a NIC based DDP stack,
especially if there is a strong desire to leave the
existing processor based stack intact.

A more realistic implementation would switch
*responsibility* immediately on the passive side, but then
have the DDP NIC send its Switch Msg in non-RDMA format.
The DDP NIC be responsible for any retransmits. This would
considerably ease implementation, with the only restriction
being that the ULP would be restricted to a single LLP
segment/message for completing the transition.

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Mon, 30 Sep 2002 06:55:43 -0700

>First, there is no 'RDMA Hello' message defined in either

Correct.

>establishment should not rely upon luck in process

I do not see any change proposed for the RDMA / DDP / MPA specs in the 
below exchange.  Given there are no wire protocol changes proposed, then 
aside from an implementation education for ULP, it appears this is a 
non-issue from a specification perspective.

>without any LLP exchange of data.

Again, implementation specific not a specification change.

>level *without* refining the requirement for the local

Aka, a verbs specification which is not what the three specifications in 
question provide.

>existing processor based stack intact.

Again, a verbs / implementation issue and not a specification issue.

>segment/message for completing the transition.

Variety of ways to solve but none put forth as yet require a change in the 
specifications being discussed.

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 30 Sep 2002 10:56:05 -0500

On 9/30/02, Michael Krause wrote:

>
>I do not see any change proposed for the RDMA / DDP / MPA
>specs in the below exchange.  Given there are no wire
>protocol changes proposed, then aside from an
>implementation education for ULP, it appears this is a

You are correct that there is no required change to any of
the wire protocol.

However, both the ddp and rdma drafts place requirements on
the local interface: specifically that the ULP can enter
RDMA mode at any any time after the stream is connected
(especially immediately).

As pointed out in the prior posts, this local interface is
inadequate. It either requires the ULPs to engage in timing
to avoid race conditions, or to have a complex interface
between the RDMA stack and traditional stack.

Based upon Paul Culley's refinement of my suggestion, it is
clear that the *only* refinements to the local interface are
as follows:

1) The ULP may enable RDMA send and RDMA receive modes
seperately.

2) The ULP may *fence* the first RDMA send pending receipt
of the first RDMA message. (So that the passive side can be
assured that the active side has completed enabling RDMA).

That is the *only* specification change required, and it
only deals with the very minimal "verb" description in
section 8.1.

I would argue that this very minor increase in the
reuqirements for the local interface is preferable to ULP
negotiations that have inherent race conditions in them
and/or complex "delivery revocation" interfaces with
software L2-up stacks.

It is also necessary that these capabilities be specified in
the draft. Later ULPs cannot define their own application
specific peer-to-peer unless the local interface
capabilities they rely upon are documented in the base
standard. There is no need to specify an API, or even a
verb, just that this capability is available to the ULP.

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Mon, 30 Sep 2002 10:24:09 -0700

>only deals with the very minimal "verb" description in

In other words, it is a verbs issue and there should be a separate document 
that describes ULP / verb interactions to ensure there is a fence 
capability in the verbs spec.

>verb, just that this capability is available to the ULP.

I don't disagree with having this as something that should be supported by 
the verbs but don't see a need to describe this within any of the three 
specs being discussed at the moment.  If one does not want to rely upon a 
verbs spec, then I might suggest that a separate informational draft be 
produced to provide guidance to ULP providers.

Mike