Subject: RE: [rddp] Transport Neutral "Connections"

From: John Hufferd <[email-masked]>
Date: Fri, 13 Jun 2003 23:38:56 -0700

Jim,
I am not sure of what your statement would prevent, but as I mentioned to 
Caitlin, and she agreed, we have important applications that MUST first 
initiate in LLP mode, and after finishing with negotiations may then 
switch into RDMA mode.  So I wonder if the thing you talk about in the 
first part of your note needs to be possible anyway,  I am not sure that 
it is an issue. 

Caitlin's point, as I understand it, is not to prevent that type of 
operation, but instead, permit applications that KNOW they are going to 
enter RDMA mode, to be able to start out that way, and as a result be 
independent of the underpinning LLP. 

Now it is possible that I misunderstood your point, but if not, 
these are two different needs.  Not one versus the other.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

"Williams, Jim" <[email-masked]>
Sent by: [email-masked]
06/12/2003 12:24 PM

        To:     [email-masked]
        cc: 
        Subject:        RE: [rddp] Transport Neutral "Connections"

I strongly agree with the sentiments
from Caitlin below.

I have great concerns about that model that requires
the ULP to set up an LLP connection, optionally
negotiate RDDP, and then change connection mode to
RDDP.

My concern is about how the lines of responsibility
are drawn between the host OS vendor,
the NIC vendor, and the ULP developer.

If the ULP developer ends up with a number of 
different APIs equal to the product of the number
of NIC vendors supported and the number of OS's supported,
this becomes unworkable.

Likwise, the NIC vendors need to be able to treat
host OS's as closed black boxes.  Otherwise
schedules become open ended and impossible
to depend on.  Also costs to port to new
OS's become prohibitive.

There may be a way to resolve these issues, but
the complexity of the verbs draft, the close
coupling with the host OS, and the need for
ULP vendors to work with existing host OS
networking stacks leaves me pessimistic.

The idea of a transport neutral API that 
can be used for RDMA avoids most of these problems.
NIC vendors provide the entire API, and ULP
implementors write to it.

The other important piece is a software implementation
of RDMA that allows RDMA aware ULPs to use the RDMA API,
but communicate through the host OS network stack,
and do so with no loss (or minimal) loss of efficiency
of what could be done if the ULP used the host
OS network stack directly in non-RDMA mode.

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Saturday, June 07, 2003 10:15 PM
> To: John Hufferd; [email-masked]
> Subject: RE: [rddp] Transport Neutral "Connections"
> 
> 
> 
> An application which forms a TCP connection, and *then*
> decides whether or not to enable RDMA mode, obviously
> must be aware that it is using TCP.
> 
> Indeed, for reasons I'll go into in a moment, that model
> does not work well for SCTP.
> 
> The model I was working from was one where the use of
> RDMA is presumed in advance. The ULP is RDMA specific.
> Their may be a non-RDMA equivalent, but it is running
> on a different port (or perhaps using a different IP
> address).
> 
> Not surprisingly this is the model that has dominated
> discussions within both the DAT Collaborative and
> the OpenGroup's ICSC. Their objective, after all, is
> to define transport neutral APIs for RDMA-aware
> applications.
> 
> So to rephrase more precisely, it should be possible
> for a transport neutral application to communicate
> the Private Date required to select a Protection Domain
> and configure the RDMA Read credits in a single message
> sent *before* the RDMA endpoint is selected -- and to
> do so in a transport neutral way.
> 
> To achieve this goal, either InfiniBand and SCTP must
> simulate a "stream" of pre-RDMA endpoint data, or TCP
> must provide a method of sending "one message".
> 
> I believe the best way to do so is to include a variable
> length Private Data field in the MPA Mode frame. The
> ability to send one Private Data message before selecting
> the RDMA endpoint is already present in InfiniBand and
> is easily defined for SCTP.
> 
> Alternatively, a protocol could be defined to delimit such
> a message over TCP independently of the shift into MPA mode.
> Either method  works.
> 
> In either case, the procedure is not imposed on any
> applications. Those that want to establish a TCP connection
> and then dynamically decide to enter RDMA mode are free to
> do so. If a Private Data field is defined, there will be
> no requirement to use it, and no prohibition against
> negotiating in plain TCP before its use.
> 
> But there are a larger group of ULPs that will be RDMA
> specific from the very beginning. Indeed, for some it would
> make no sense to have a non-RDMA version. Would you run
> SDP over plain TCP?
> 
> Even with less extreme cases, optimum use of RDMA is best
> achieved when the ULP is fully aware of RDMA's capabilities.
> While there will undoubtedly be servers that have valid
> reasons for dynamically deciding whether to enable RDMA,
> there will be at least as large a group that prefer to
> run the two distinctly different sets of logic through
> different daemons running on different ports. The server
> advertises two different sevices (whether by SLP, in
> the URL, or by whatever method), and the client chooses
> which to invoke.
> 
> 
> 
> There is an issue on the dynamic "in-session" switch to
> RDMA model as applied to SCTP. The current SCTP draft
> will not support such a model.
> 
> The rationale is that sharing control of streams within an
> single SCTP association between a non-RDMA stack and a RDMA
> stack would be very difficult. A likely adoption strategy
> would have the RDMA stack implemented on an RNIC, with the
> non-RDMA stack remaining "as is" on the processor.
> 
> Segregating SCTP associations between the
> RDMA-implementation and the current stack allows RDMA to be
> deployed without disrupting current stacks.
> 
> This same issue exists with TCP connections. But TCP only
> requires a relatively clean handoff from the conventional
> stack to the MPA/iWARP stack (relatively clean, because
> responsiblity for retransmitting might be split right at
> the shift point).
> 
> An SCTP implementation cannot shift just one Stream.
> Then handling of TSNs, SACKs and all other control chunks
> is common across all streams.
> 
> So is the correct response:
> 
> a) Don't worry about it. The "dynamic in-session" model is
>    highly specialized and inherently related to existing
>    servers that deal with long-haul connections where the
>    clients have little a priori knowledge of the server.
>    There is little need to support this model for SCTP.
> b) Require an iWARP capable SCTP stack to also support
>    non-iWARP streams. The trick is that non-iWARP streams
>    can have *ordered* Data Chunks. The handling of which
>    is both more complex and more consuming of memory
>    bandwidth than for an iWARP stream (which only uses
>    unordered data chunks).
> 
> If dynamic in-session switching is a general model that
> should be supported in an LLP fashion then the SCTP draft
> should find a way to accomodate it -- preferably in a way
> that doesn't make an implemetnation that much more
> difficult.
> 
> 
> 
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 14 Jun 2003 09:33:19 -0500

On 6/13/03, John Hufferd wrote:

>Jim,
>I am not sure of what your statement would prevent, but as
>I mentioned to Caitlin, and she agreed, we have important
>applications that MUST first initiate in LLP mode, and
>after finishing with negotiations may then switch into
>RDMA mode.  So I wonder if the thing you talk about in the
>first part of your note needs to be possible anyway,  I am
>not sure that it is an issue. 
>

There is a great deal of difference between a model that
allows *either* application model, and one that *requires*
the the ULP to set up the LLP itself.

I read Jim's statement as matching my concern -- the
problem is that the current proposals are only friendly
for LLP aware applications.

Vague suggestions that some middleware can sweep everything
under the rug ring very hollow with me. I have major doubts
as to how a TCP stack can be an efficient TCP stack AND
handle the last step of negotiating RDMA entry efficiently.

Not only is the ULP using a byte *stream* for the
final pre-RDMA negotiation message unnatural (and fail to
match any other transport), it has the nasty effect of
requiring a hybrid stack to parse a single TCP
segment in two different stacks. (In particular pulling
bytes from the NIC only after the ULP has actually read them
is a very inefficient way to run a normal TCP stream. So if
the bytes have already been flushed from the NIC
"prematurely we'd have to "give them back". Neither
slow-fetch or push-back are optimal or elegant solutions.
And both get far uglier when you have to retransmit across a
network that doesn't preserve TCP segment boundaries).

That won't be reliable. What it will lead to is applications
"playing it safe" by delaying MPA entry long enough for
the TCP connection to have gone idle. Effectively
creating the very round-trip delay that we agreed that
the MPA mode frame was not supposed to require.

Efficient RDMA Mode entry requires capture of the last
pre-RDMA message *and* entry into MPA mode to be an
atomic step that can be performed *without* the ULP
being directly involved. We're supposed to be minimizing
the number of required ULP interactions, right?

Caitlin Bestler