Subject: RE: [rddp] Transport Neutral "Connections"

From: [email-masked]
Date: Mon, 16 Jun 2003 18:35:11 -0400

Folks,
 
I think there are three "models" of how to get into RDMA mode
under discussion:
 
(1) Enter immediately on startup, possibly with an initial frame
    containing private data required to setup the connection.
(2) Enter at some subsequent point, "quiescing" the connection first.
(3) Enter at some subsequent point, on the fly without a hiccup.
 
I read Caitlin's posts as saying that (3) is hard support in
hardware; in order to obtain the flexibility of entering RDMA mode
at some subsequent point the result is likely to require something
that looks like a handshake with the associated latency.  This doesn't
rule out (2), it just says that those who want that flexibility are
going to wind up paying some synchronization latency (Aside: it may
be that the structure of the ULP requires that this be paid anyway,
but that depends on the ULP).  Viewed from another angle, those who
want to minimize elapsed time from startup to blasting away in RDMA
mode will want to use model (1), and the proposed private data
mechanism may help make that model more applicable.  Those who wish
the generality of switching into RDMA mode later can use (2), with
the knowledge that some sort of handshake is needed.
 
I don't think I see a problem here - the main point is that the
flexibility of deferred startup may come with a price, i.e., ULP
designers should be thinking about (2), not (3).
 
Thanks,
--David

-----Original Message-----
From: John Hufferd [mailto:[email-masked]] 
Sent: Saturday, June 14, 2003 5:29 PM
To: Caitlin Bestler
Cc: [email-masked]
Subject: RE: [rddp] Transport Neutral "Connections"

Well I thought we were almost together, but I can not accept the statement
that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive
for some very important applications that must start their functional
negotiations in TCP/IP mode, and then when RDMA mode is agreed upon, switch
into that mode.  This permits the application to gradually move into RDMA
mode across a network of nodes that implement RDMA at different rates (or
not at all).   

Now the above statement should not indicate that there should not be a way
to go directly to RDMA mode, it just says that we need an Overt method to
accommodate the above type of negotiation whether or not we come up with a
direct mode for entering RDMA, independent of the LLP. 

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked] 

	Caitlin Bestler <[email-masked]> 

06/14/2003 11:38 AM 

        To:        John Hufferd/San Jose/IBM@IBMUS, [email-masked] 
        cc:         
        Subject:        RE: [rddp] Transport Neutral "Connections" 

On 6/14/03, John Hufferd wrote:

>Caitlin,
>I can not determine from your note whether you are now
>objecting to the application being able to perform its
own
>negotiation before entering RDMA mode (via an Atomic
last
>message operation) as is currently defined; or are you
>agreeing with that, but ALSO want a LLP neutral method
to
>enter the RDMA mode?

The latter.

But I don't view the LLP neutral as the "additional"
method. It should be the primary method. Conditional
negotiation is a special need for certain applications.
The LLP SHOULD be invisible to the ULP. If the ULP
wants to deal with the ULP, well then there certainly
isn't any reason to forbid it. I just don't want to
force the ULP to deal with the LLP - or even suggest
that it ought to. I would prefer to state forthrightly
that the ULP SHOULD NOT deal with the LLP.

The current proposals for MPA do exactly that. Vague
unspecified middleware to parse a "message" above the
stack does not solve the problem. It is a) still the
ULP, and b) unrealistically complex.

This middleware would be expected to consume the
*precise* number of bytes required, and leave the
remaining byte sitting for transfer to the MPA stack
*instantly* when the request to assign the TCP Stream
to an RDMA Endpoint/QP is made. This would be true even
if the MPA Mode frame had been included in the very
same TCP segment as the final pre-RDMA "message".

This creates a nasty problem for at least some valid
implementation strategies.

Assume that at least a few sites will not be eager to
turn over all TCP connections to a new stack.

Therefore there will be a non-MPA stack still running
on the host CPU, regardless of whatever
RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC.

The RNIC is also the Ethernet NIC. I am going to take a
bold guess here and venture that at least a few sites
will wish to preserve the current IP stack. The
existing IP stack is known to all of the other existing
applications that don't need this "RDMA" feature yet.
The same stack that has already been tested and audited
with all of those applications. New Ethernet NICs are
much simpler to test and validate than new IP stacks.
So at least some sites will want the RNIC to present
itself as a conventional NIC for the benefit of the
existing IP stack.

And Ethernet NICs handoff entire frames, not bytes from
TCP frames as the ULP reads them from sockets.

Now this handoff can be solved, but it is tricky.
Involving the ULP in the middle of it will only make
things far worse.

"Atomic" may be too strong of a word. But I do believe
that receipt of the final "negotiation" message and
entry to MPA mode needs to be co-ordinated at the
kernel and/or hardware layer. The ULP should only be
involved for policy issues -- not the mechanics. The
mechanics are messy *and* LLP specific.

To encourage, not just allow, the ULP to be LLP-neutral
we need to define a procedure where the ULP can exchange
messages that lead to the assignment of RDMA endpoints
(or QPs if you insist on callling them that, but if you
think about it, they no longer have two queues).

Further, certain of those messages are naturally trigger
events directly related to session state changes
(request, accept and terminate). Those semantic are
best recognized as they are parsed, not after a bounce
through the ULP that only delays things.

Caitlin Bestler
http://asomi.com/CaitlinBestler/

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 17 Jun 2003 10:33:10 -0500

On 6/16/03, [email-masked] wrote:

>Folks,
> 
>I think there are three "models" of how to get into
>RDMA mode under discussion:
> 
>(1) Enter immediately on startup, possibly with an
> initial frame containing private data required to
> setup the connection.

This is what I see as the main model, but it needs
to allow the passive side to receive the Private Data
in the initial frame, then select/configure its endpoint/QP.

I favor making the initiat frame also serve as the MPA
mode frame. However any frame that can be parsed without
active ULP involvement, after the ULP selects this mode,
is fine with me.

I want the ULP to interpret the message, not be involved
in determining when the entire message has arrived. The
meaning of the message can be LLP independent, the mechanics
of collecting it cannot.

>(3) Enter at some subsequent point, on the fly without
My real concern here is how to define "quiescing". What
I fear is that implemenetations will interpret "message"
as being "all data that arrives in the first TCP segment
or n microseconds thereafter."

What is really scary about that definition is that it will
almost always work.

Bugs that hide well are always the hardest to kill.

TCP segments are not guaranteed to be preserved. So there
has to be some method of determining when the "message"
has been collected that does not rely upon TCP segment
boundaries or timing.

The best implementation-independent definition of
"quiescing" that I can come up with is that all prior
bytes have been acked, and both sides are in full agreement
that RDMA mode should be established.

Each side could then send a single TCP segment any last
ULP initialization message and the MPA initialization,
while expecting the same. [The single TCP segment is
a measure of efficiency and succintness, I'm not proposing
that as a protocol requirement, just something that should
be achievable by an optimized implementation].

In an active/active scenario these messages could pass
in flight. For active/passive the active side would send,
the passive side would receive, interpret, select/configure
its endpoint and then respond.

My assumption here is that the RNIC will be expected to
support non-RDMA TCP connections, and may be required/
expected to do so with a conventional IP stack. So unless
I'm intending to run BSD on my RNIC, the only way I'll
satisfy a really thorough security concious network admin
is to pass the TCP connections I am not using through to
an existing IP stack. To do that efficiently, I want to
"place" those received "non-RDMA" IP datagrams to the
convention IP stack just as immediately as I place the
data from RDMA connections.

I can achieve that if the "MPA Entry Sequence" is something
that can be entirely handled by the RNIC after being enabled
by the ULP while the connection is quiescent. Most ULPs
should figure out that the easiest time to have a quiescent
connection is right when its established.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Mon, 23 Jun 2003 10:49:47 -0700 (PDT)

Sorry for the delay, I tried to send this last week, but it silently
disapeared.

> hardware; in order to obtain the flexibility of entering RDMA mode

While I agree that it's not easy to support now, I've been looking at some
of the OS infrastructure proposals for adding full tcp offload support,
and I think it won't be too hard.

The offload proposals I've seen talk about having the host OS handle TCP
connection setup, then hand off an existing connection. The way handoff
works is that the host OS signals the NIC that it will be getting the
connection, so it sets up filters to intercept packets. The host OS then
finishes processing any pending packets that had been delivered to it,
then uploads the connection state to the NIC. The NIC then starts handling
the connection.

Yes, that's TCP. But handling RDMA on top of that isn't hard. The same
procedure can happen if the host OS has a software RDMA stack. The
software stack processes any pending mis-labeled TCP packets (really RDMA
but we didn't know we were doing RDMA at that point). That will advance
RDMA connection state, then the host OS offloads the RDMA processing to
the card.

So if a software/host OS stack is available, (3) won't be too hard. It
also won't require more complexity than TCP connection offload will
require.

Take care,

Bill

---

From: [email-masked]
Date: Mon, 23 Jun 2003 15:10:13 -0400 (EDT)

>
> The offload proposals I've seen talk about having the
> host OS handle TCP connection setup, then hand off an
> existing connection. The way handoff works is that the
> host OS signals the NIC that it will be getting the
> connection, so it sets up filters to intercept packets.
> The host OS then finishes processing any pending packets
> that had been delivered to it, then uploads the
> connection state to the NIC. The NIC then starts
> handling the connection.
>

I would be interested in hearing any explanations that
can be made publicly as to how you allow the RNIC to
accept TCP connections and still keep the RNIC current
on anti-DoS attacks.

It is easy when you're running BSD on the RNIC, but in
the long run I presume the goal is to avoid requiring
an RNIC to have a full processor of its own.

I suppose you could view TCP-connection grooming as a
feature of the RNIC, but then how do you keep it current?

This approach also runs into some nasty problems with
accepting SCTP INIT requests for multi-homed endpoints.
How do the two RNICs avoid race conditions?

My presumption has been that at least some designs will
want the RNIC to take control of RDMA, not of connection
grooming. Therefore they will want to exercise full control
over accepting connections, including logging.

How do you propose to perform the safeguard/logging
functions currently performed by INETD, TCPSERVER and
similiar connection management daemons?

> Yes, that's TCP. But handling RDMA on top of that isn't
> hard. The same procedure can happen if the host OS has a
> software RDMA stack. The software stack processes any
> pending mis-labeled TCP packets (really RDMA but we
> didn't know we were doing RDMA at that point). That will
> advance RDMA connection state, then the host OS offloads
> the RDMA processing to the card.
>
> So if a software/host OS stack is available, (3) won't
> be too hard. It also won't require more complexity than
> TCP connection offload will require.
>
> Take care,
>
> Bill

------------
Caitlin Bestler
[email-masked]
http://asomi.com/

---

From: [email-masked]
Date: Mon, 23 Jun 2003 15:19:26 -0700 (PDT)

> accept TCP connections and still keep the RNIC current

You don't. You let the host OS accept the connection, and then
the host tells the NIC to take it over. That way you keep using
the host OS security policies; you don't try to map N different
host OS policies to M different NIC infrastructures. Also, doing
things that way (the host OS stays in charge) answers a lot of
questions about how to handle policy; you just let the host do
it. :-)

Note, part of this idea is that the NIC looks like a simple NIC in
addition to everything else. :-)

> an RNIC to have a full processor of its own.

I think the above class of solution will help with that. All the
RNIC needs is to be told all the TCP state for a connection, and
it can do the TCP offload. Since you also have to offload routing
and ARP info, there's a whole hierarchy. So getting back to RNICs,
it's no biggie to add RDMA as another layer on top. ;-)

> How do the two RNICs avoid race conditions?

As above, the RNICs would be reacting to host OS direction, so
the host OS would have to have chosen one of them. :-)

> over accepting connections, including logging.

My gut feel is that ALL implementations should do this. By letting
the host OS do it, you remove a lot of issues. :-)

> similar connection management daemons?

Note: I'm not proposing this. I'm noting what I've seen (in a vague
conceptual manner :-)

Either the host OS would accept the connection, move it to the RNIC, then
have inetd stop it, or the host OS would do whatever it does with inetd
and say tcpwrappers, and only move it to the NIC if it passes.

> > TCP connection offload will require.

Just to be clear, I'm not saying folks shouldn't do David's (2), but that
(3) will probably be feasible.

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 25 Jun 2003 10:59:45 -0500

On Monday, June 23, 2003, [email-masked] wrote:

>The offload proposals I've seen talk about having the host OS
>handle TCP connection setup, then hand off an existing
>connection. The way handoff works is that the host OS signals
>the NIC that it will be getting the connection, so it sets up
>filters to intercept packets. The host OS then finishes
>processing any pending packets that had been delivered to it,
>then uploads the connection state to the NIC. The NIC then
>starts handling the connection.

My first read of what you were saying was wrong. Having read
your clarification, I still don't accept that it is an easy
solution.

The "connection state" that you are suggesting simply be handed
back to the offload engine would have to include:

-- All prior portions of the TCP stream that have not been consumed.
-- Notations on which portions of that have been acked. Keep in mind
   that portions may have been SACKed.
-- All prior transmits that have not been acked.
-- and of course the obvious items: window size, scaling, congestion...

And all of the above has to work with varying scale options.

I believe that the model of doing a TCP specific negotiation and
*then* deciding to enter RDMA mode as a by-product can and should
be supported. But we need to determine what the real requirements
are, and explicitly state all of the simplifying assumptions.
Examples would include "all prior TCP segments must have been
acked before the final streaming mode message may be sent."

I also believe we can support an equivalent capability for
SCTP, *if* the sole criteria for entry to RDMA mode is capability.
I have not seen any scenarios for optional-RDMA session establishment
that call for not establishing it even when both sides are capable.
So I do not anticipate that this would be an issue, particularly with
SCTP where there is not even a marker-penalty for using RDMA mode.

Caitlin Bestler - [email-masked] - http://asomi.com/