Subject: RE: [rddp] Transport Neutral "Connections"

From: [email-masked]
Date: Tue, 17 Jun 2003 10:32:39 -0400

John,

>> I think there are three "models" of how to get into RDMA mode 
>> under discussion: 
>>   
>> (1) Enter immediately on startup, possibly with an initial frame 
>>     containing private data required to setup the connection. 
>> (2) Enter at some subsequent point, "quiescing" the connection first. 
>> (3) Enter at some subsequent point, on the fly without a hiccup. 
>>   
>> I read Caitlin's posts as saying that (3) is hard to support in 
>> hardware; in order to obtain the flexibility of entering RDMA mode 
>> at some subsequent point the result is likely to require something 
>> that looks like a handshake with the associated latency.  This doesn't 
>> rule out (2), it just says that those who want that flexibility are 
>> going to wind up paying some synchronization latency (Aside: it may 
>> be that the structure of the ULP requires that this be paid anyway, 
>> but that depends on the ULP).  Viewed from another angle, those who 
>> want to minimize elapsed time from startup to blasting away in RDMA 
>> mode will want to use model (1), and the proposed private data 
>> mechanism may help make that model more applicable.  Those who wish 
>> the generality of switching into RDMA mode later can use (2), with 
>> the knowledge that some sort of handshake is needed. 
>> 
>> I don't think I see a problem here - the main point is that the 
>> flexibility of deferred startup may come with a price, i.e., ULP 
>> designers should be thinking about (2), not (3).

> I think I agree with your statement, but I am not quite sure
> what on the fly means.  Or exactly what "quiescing" means in this case.

The problematic scenario is that one side sends a stream of data and
expects the other side to switch into RDMA mode in the middle (that's
"on-the-fly".  The implementation problem is that by the time the
receiver ULP figures out that it's supposed to switch into RDMA mode,
some of the RDMA-mode traffic has already been processed in non-RDMA
mode.  Quiescing is needed to avoid this.

> If for instance we are saying that everything is brought to a
> half duplex like mode, where side A is waiting for side B to send
> a finial message before entering RDMA mode?  If so, then I think
> that would be OK.
> 
> This might mean, for example that: 
> 1. Side B will need to have an atomic method of sending the
> last message in LLP mode, and then transiting into RDMA mode, and 
> 2. After side A has received the last LLP message from B, it may
> then transit into RDMA mode.
> 
> Would you say that is quiescing and entering RDMA mode?

Only for that half of the connection, side A still has to cleanly
enter RDMA mode, so there's another pair of steps involving a wait
for B:

3.  Side A transitions into RDMA mode and sends its first RDMA-mode
	message to B.
4.  B waits for receipt of that message before sending any RDMA-mode
	traffic to A (i.e., B cannot send any traffic to A after
	sending its "last LLP message" until it receives an RDMA-mode
	message indicating that A has switched).

Notice that both directions of traffic pause as part of this -
A has to wait for the last non-RDMA message from B (step 2) before
sending any RDMA traffic, and B has to wait for the first RDMA
message from A (step 4) before sending RDMA traffic.

Side B's method does not need to be atomic - B can:
a) Transition receive side of connection into RDMA mode
b) Send last message to A in non-RDMA mode
c) Transition send side of connection into RDMA mode
In contrast, A can transition both sides of the connection into
RDMA mode at the same time because B's final non-RDMA message says
that the response from A is to be in RDMA mode.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 17 Jun 2003 13:52:46 -0500

On 6/17/03, [email-masked] wrote:

>> I think I agree with your statement, but I am not
>> quite sure what on the fly means.  Or exactly what
>> "quiescing" means in this case.
>
>The problematic scenario is that one side sends a
>stream of data and expects the other side to switch
>into RDMA mode in the middle (that's "on-the-fly". 
>The implementation problem is that by the time the
>receiver ULP figures out that it's supposed to switch
>into RDMA mode, some of the RDMA-mode traffic has
>already been processed in non-RDMA mode.  Quiescing is
>needed to avoid this.

Exactly.

The off-load stack  wants to "place" TCP segments
immediately upon receipt. This is true even when in
"non-RDMA" mode. It's just that they are being "placed"
to the conventional stack, and probably as Ethernet
frames rather than as TCP segments.

It wouldn't make sense for the off-load stack to have
to buffer TCP segments recieved out-of-order when that
is what it was designed *not* to do. Let the conventional
stack handle putting conventional TCP segments back in
order. But to do that we can't have any pre-mature
placements to the conventional stack. If segment N
is going to cause the receiver to shift the stream
to RDMA mode, then we can't handle N+1.

>> 1. Side B will need to have an atomic method of
>> sending the last message in LLP mode, and then
>> transiting into RDMA mode, and 
>> 2. After side A has received the last LLP message
>> from B, it may then transit into RDMA mode.
>> 
>Only for that half of the connection, side A still has
>to cleanly enter RDMA mode, so there's another pair of
>steps involving a wait for B:
>

If I understand John's example correctly, A is quiescent
because B is in the passive mode and will not send the
first RDMA-mode message until A does the first RDMA-mode
transmit. But that does need to be explicitly stated.

Caitlin Bestler
http://asomi.com/CaitlinBestler/