Subject: RE: [rddp] Transport Neutral "Connections"

From: John Hufferd <[email-masked]>
Date: Sat, 14 Jun 2003 10:57:55 -0700

Caitlin,
I can not determine from your note whether you are now objecting to the 
application being able to perform its own negotiation before entering RDMA 
mode (via an Atomic last message operation) as is currently defined; or 
are you agreeing with that, but ALSO want a LLP neutral method to enter 
the RDMA mode?

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Caitlin Bestler <[email-masked]>
06/14/2003 07:33 AM

        To:     John Hufferd/San Jose/IBM@IBMUS
        cc:     "Williams, Jim" <[email-masked]>, [email-masked], 
[email-masked]
        Subject:        RE: [rddp] Transport Neutral "Connections"

On 6/13/03, John Hufferd wrote:

>Jim,
>I am not sure of what your statement would prevent, but as
>I mentioned to Caitlin, and she agreed, we have important
>applications that MUST first initiate in LLP mode, and
>after finishing with negotiations may then switch into
>RDMA mode.  So I wonder if the thing you talk about in the
>first part of your note needs to be possible anyway,  I am
>not sure that it is an issue. 
>

There is a great deal of difference between a model that
allows *either* application model, and one that *requires*
the the ULP to set up the LLP itself.

I read Jim's statement as matching my concern -- the
problem is that the current proposals are only friendly
for LLP aware applications.

Vague suggestions that some middleware can sweep everything
under the rug ring very hollow with me. I have major doubts
as to how a TCP stack can be an efficient TCP stack AND
handle the last step of negotiating RDMA entry efficiently.

Not only is the ULP using a byte *stream* for the
final pre-RDMA negotiation message unnatural (and fail to
match any other transport), it has the nasty effect of
requiring a hybrid stack to parse a single TCP
segment in two different stacks. (In particular pulling
bytes from the NIC only after the ULP has actually read them
is a very inefficient way to run a normal TCP stream. So if
the bytes have already been flushed from the NIC
"prematurely we'd have to "give them back". Neither
slow-fetch or push-back are optimal or elegant solutions.
And both get far uglier when you have to retransmit across a
network that doesn't preserve TCP segment boundaries).

That won't be reliable. What it will lead to is applications
"playing it safe" by delaying MPA entry long enough for
the TCP connection to have gone idle. Effectively
creating the very round-trip delay that we agreed that
the MPA mode frame was not supposed to require.

Efficient RDMA Mode entry requires capture of the last
pre-RDMA message *and* entry into MPA mode to be an
atomic step that can be performed *without* the ULP
being directly involved. We're supposed to be minimizing
the number of required ULP interactions, right?

Caitlin Bestler
http://asomi.com/CaitlinBestler/

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 14 Jun 2003 13:38:05 -0500

On 6/14/03, John Hufferd wrote:

>Caitlin,
>I can not determine from your note whether you are now
>objecting to the application being able to perform its
own
>negotiation before entering RDMA mode (via an Atomic
last
>message operation) as is currently defined; or are you
>agreeing with that, but ALSO want a LLP neutral method
to
>enter the RDMA mode?

The latter.

But I don't view the LLP neutral as the "additional"
method. It should be the primary method. Conditional
negotiation is a special need for certain applications.
The LLP SHOULD be invisible to the ULP. If the ULP
wants to deal with the ULP, well then there certainly
isn't any reason to forbid it. I just don't want to
force the ULP to deal with the LLP - or even suggest
that it ought to. I would prefer to state forthrightly
that the ULP SHOULD NOT deal with the LLP.

The current proposals for MPA do exactly that. Vague
unspecified middleware to parse a "message" above the
stack does not solve the problem. It is a) still the
ULP, and b) unrealistically complex.

This middleware would be expected to consume the
*precise* number of bytes required, and leave the
remaining byte sitting for transfer to the MPA stack
*instantly* when the request to assign the TCP Stream
to an RDMA Endpoint/QP is made. This would be true even
if the MPA Mode frame had been included in the very
same TCP segment as the final pre-RDMA "message".

This creates a nasty problem for at least some valid
implementation strategies.

Assume that at least a few sites will not be eager to
turn over all TCP connections to a new stack.

Therefore there will be a non-MPA stack still running
on the host CPU, regardless of whatever
RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC.

The RNIC is also the Ethernet NIC. I am going to take a
bold guess here and venture that at least a few sites
will wish to preserve the current IP stack. The
existing IP stack is known to all of the other existing
applications that don't need this "RDMA" feature yet.
The same stack that has already been tested and audited
with all of those applications. New Ethernet NICs are
much simpler to test and validate than new IP stacks.
So at least some sites will want the RNIC to present
itself as a conventional NIC for the benefit of the
existing IP stack.

And Ethernet NICs handoff entire frames, not bytes from
TCP frames as the ULP reads them from sockets.

Now this handoff can be solved, but it is tricky.
Involving the ULP in the middle of it will only make
things far worse.

"Atomic" may be too strong of a word. But I do believe
that receipt of the final "negotiation" message and
entry to MPA mode needs to be co-ordinated at the
kernel and/or hardware layer. The ULP should only be
involved for policy issues -- not the mechanics. The
mechanics are messy *and* LLP specific.

To encourage, not just allow, the ULP to be LLP-neutral
we need to define a procedure where the ULP can exchange
messages that lead to the assignment of RDMA endpoints
(or QPs if you insist on callling them that, but if you
think about it, they no longer have two queues).

Further, certain of those messages are naturally trigger
events directly related to session state changes
(request, accept and terminate). Those semantic are
best recognized as they are parsed, not after a bounce
through the ULP that only delays things.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp