Subject: RE: [rddp] Transport Neutral "Connections"

From: John Hufferd <[email-masked]>
Date: Sat, 7 Jun 2003 18:14:17 -0700

Catlin,
Paul said the following:

><prc> There is at least one application in the works that
>does use a multi-round negotiation prior to iWARP
>startup...  Of course it is written to use TCP streaming
>mode on a legacy port for this. <prc>

And you replied with

"Which is *exactly* what we need to avoid. An iWARP
application SHOULD NOT be aware of which LLP is in use."

I find that answer less than satisfying.  I also know of a application 
that must have a number of multi-round negotiations before they go into 
RDMA mode.  This has to do with understanding all the capabilities common 
between the sides, before actually deciding on RDMA mode, or continuing in 
TCP/IP mode.  That application does NOT want to stop the connection if the 
other side does not want to enter RDMA mode.  In other words it wants to 
fall back to a previously defined TCP/IP protocol and continue operations.

Perhaps I misunderstood you, but if you do not want to support this 
multi-round negotiations, then I do not think we are covering all the 
requirements. 

Did I misunderstand you?

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 7 Jun 2003 21:14:55 -0500

On 6/7/03, John Hufferd wrote:

>I find that answer less than satisfying.  I also know of a
>application that must have a number of multi-round
>negotiations before they go into RDMA mode.  This has to
>do with understanding all the capabilities common between
>the sides, before actually deciding on RDMA mode, or
>continuing in TCP/IP mode.  That application does NOT want
>to stop the connection if the other side does not want to
>enter RDMA mode.  In other words it wants to fall back to
>a previously defined TCP/IP protocol and continue
>operations.
>
>Perhaps I misunderstood you, but if you do not want to
>support this multi-round negotiations, then I do not think
>we are covering all the requirements. 
>
>Did I misunderstand you?
>

No, I was dealing with a different set of applications.

An application which forms a TCP connection, and *then*
decides whether or not to enable RDMA mode, obviously
must be aware that it is using TCP.

Indeed, for reasons I'll go into in a moment, that model
does not work well for SCTP.

The model I was working from was one where the use of
RDMA is presumed in advance. The ULP is RDMA specific.
Their may be a non-RDMA equivalent, but it is running
on a different port (or perhaps using a different IP
address).

Not surprisingly this is the model that has dominated
discussions within both the DAT Collaborative and
the OpenGroup's ICSC. Their objective, after all, is
to define transport neutral APIs for RDMA-aware
applications.

So to rephrase more precisely, it should be possible
for a transport neutral application to communicate
the Private Date required to select a Protection Domain
and configure the RDMA Read credits in a single message
sent *before* the RDMA endpoint is selected -- and to
do so in a transport neutral way.

To achieve this goal, either InfiniBand and SCTP must
simulate a "stream" of pre-RDMA endpoint data, or TCP
must provide a method of sending "one message".

I believe the best way to do so is to include a variable
length Private Data field in the MPA Mode frame. The
ability to send one Private Data message before selecting
the RDMA endpoint is already present in InfiniBand and
is easily defined for SCTP.

Alternatively, a protocol could be defined to delimit such
a message over TCP independently of the shift into MPA mode.
Either method  works.

In either case, the procedure is not imposed on any
applications. Those that want to establish a TCP connection
and then dynamically decide to enter RDMA mode are free to
do so. If a Private Data field is defined, there will be
no requirement to use it, and no prohibition against
negotiating in plain TCP before its use.

But there are a larger group of ULPs that will be RDMA
specific from the very beginning. Indeed, for some it would
make no sense to have a non-RDMA version. Would you run
SDP over plain TCP?

Even with less extreme cases, optimum use of RDMA is best
achieved when the ULP is fully aware of RDMA's capabilities.
While there will undoubtedly be servers that have valid
reasons for dynamically deciding whether to enable RDMA,
there will be at least as large a group that prefer to
run the two distinctly different sets of logic through
different daemons running on different ports. The server
advertises two different sevices (whether by SLP, in
the URL, or by whatever method), and the client chooses
which to invoke.

There is an issue on the dynamic "in-session" switch to
RDMA model as applied to SCTP. The current SCTP draft
will not support such a model.

The rationale is that sharing control of streams within an
single SCTP association between a non-RDMA stack and a RDMA
stack would be very difficult. A likely adoption strategy
would have the RDMA stack implemented on an RNIC, with the
non-RDMA stack remaining "as is" on the processor.

Segregating SCTP associations between the
RDMA-implementation and the current stack allows RDMA to be
deployed without disrupting current stacks.

This same issue exists with TCP connections. But TCP only
requires a relatively clean handoff from the conventional
stack to the MPA/iWARP stack (relatively clean, because
responsiblity for retransmitting might be split right at
the shift point).

An SCTP implementation cannot shift just one Stream.
Then handling of TSNs, SACKs and all other control chunks
is common across all streams.

So is the correct response:

a) Don't worry about it. The "dynamic in-session" model is
   highly specialized and inherently related to existing
   servers that deal with long-haul connections where the
   clients have little a priori knowledge of the server.
   There is little need to support this model for SCTP.
b) Require an iWARP capable SCTP stack to also support
   non-iWARP streams. The trick is that non-iWARP streams
   can have *ordered* Data Chunks. The handling of which
   is both more complex and more consuming of memory
   bandwidth than for an iWARP stream (which only uses
   unordered data chunks).
   
If dynamic in-session switching is a general model that
should be supported in an LLP fashion then the SCTP draft
should find a way to accomodate it -- preferably in a way
that doesn't make an implemetnation that much more
difficult.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Stephen Bailey <[email-masked]>
Date: Fri, 13 Jun 2003 19:16:33 -0400

> that doesn't make an implemetnation that much more

I believe that would be wise.

SCTP will also have protocols that you may want to run with or without
RDDP.  It seems pretty clear (from other examples, e.g.  SSL) that
allowing in-band negotiation our best bet to make RDDP easy to use.
It may cause a touch of additional implementation complexity, but
that'll be a drop in the bucket compared to the total complexity of
any RDDP implementation.

Steph

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 13 Jun 2003 18:42:15 -0500

On 6/13/03, Stephen Bailey wrote:

>SCTP will also have protocols that you may want to run
>with or without RDDP.  It seems pretty clear (from other
>examples, e.g.  SSL) that allowing in-band negotiation our
>best bet to make RDDP easy to use. It may cause a touch of
>additional implementation complexity, but that'll be a
>drop in the bucket compared to the total complexity of any
>RDDP implementation.
>
>Steph

There's actually two approaches that are theoretically
simple, but I fear are quite disruptive.

a) Define a range of Payload Protocol Identifiers which
will be passed through the DDP stack to a non-RDDP
SCTP stack. This stack could also originate said Data
Chunks with those Payload Protocol Identifiers.

Very simple, conceptually, except for the fact that
its an interface that no current SCTP stack has --
and otherwise the adaptation runs very cleanly on
top of existing SCTP stacks (more so than for MPA/TCP).

b) Simply mandate that an SCTP Stream pair be reassingable
to a new DDP Endpoint at any time.

The "pre" negotiations could be done with untagged messages
and one Protection Domain (with no STags enabled). When
full RDMA capabilities are enabled the stream can be handed
off to a new Endpoint with the negotiated credentials.

Again it is very simple conceptually, except that nobody
makes endpoints like that. For InfiniBand, DAT, the IT
API and RDMAC proposed iWARP verbs an endpoint (QP)
is created *within* a Protection Domain.

That's why I'm still favor the single-message model.
Of course I'm more than willing to update the draft
to match whatever the consensus favors. The ultimate
goal is enable a transport neutral API that works over
both MPA/TCP and SCTP (and InfiniBand would be nice too)
that does not lower them all to the lowest common
denominator.

Caitlin Bestler