Subject: [rddp] Transport Neutral "Connections"

From: Caitlin Bestler <[email-masked]>
Date: Wed, 4 Jun 2003 11:06:37 -0500

It is highly preferable if applications can be indifferent
as to which underlying reliable transport is used as the LLP
for RDMAP/DDP.

Even beyond that, there is a desire by application
developers to apply the same application logic for
RDMA-aware applications no matter which RDMA transport is
used. This has led to the development of the Direct Access
Transport (DAT) Collaboriative's uDAPL and kDAPL, and to
the Open Group's ICSC IT-API.

Support of these goals can be enhanced by a relatively minor
change to both the SCTP adaptation and in the definition of
the MPA Mode Frame.

With this change, ULPs would be able to engage in the same
message exchanges prior to having a fully operational
"connection" between two RDMA-enabled endpoints, whether
using RDMAP/DDP over MPA/TCP, over SCTP or even a non-IP
transport such as InfiniBand.

This type of uniformity will enable more rapid development
of RDMA-optimized ULPs.

The first issue to be addressed is the need for
pre-connection data exchange.

An RDMA Endpoint (referred to as a Queue Pair, or QP, in
InfiniBand and the draft-hilland proposed iWARP verbs)
must be associated with a Protection Domain. No receive
operations may be posted to the endpoint before it is
associated with a Protection Domain. Indeed under both
the InfiniBand and proposed iWARP verbs an endpoint/QP
is created within a Protection Domain.

There are some applications where the choice of Protection
Domain is dependent upon the identity of the remote ULP
client. For example, if a user session requires multiple
connections, it is highly desireable for all of those
connections to use a single Protection Domain.

InfiniBand, the DAT APIs and the IT-API all provide for
the active side ULP to provide Provide Data when requesting
a connection. This data is passed to the ULP to allow it
to determine whether to accept the connection, and if so
with which endpoint (and implicitly which Protection
Domain).

The Private Data can also be used to ensure that both ends
of the connection have configured their RDMA endpoints
compatibly on such matters as the RDMA Read capacity.
Further ULP-specific uses are also presumed, such as
establishing the identity of the client.

Private Data is also allowed for when accepting the
connection, to allow completion of any negotiation on
RDMA resources and for other ULP reasons.

What is proposed is that iWARP standardize on allowing a
single ULP message to be sent in each direction in the
process of establishing an LLP Connection.

For SCTP: the stream reset message would allow variable
length Private Data at the end of the chunk. The local
interface SHOULD report this data to the ULP when reporting
the stream's state transition.

An additional Adaptation Layer Control function code would
be defined to explicitly terminate, but not re-enable,
a stream. The Local interface SHOUD report this, and any
included Private Data, to the ULP when reporting the
stream's state transition.

For MPA/TCP: the MPA Mode Frame would be defined to be an
MPA Frame, which allowed the same variable length Private
Data at the end of the frame. The local interface SHOULD
report this data to the ULP when reporting the connections
state transition.

For SCTP, a "connection" could be established as follows:

- The passive side would listen on the destination port,
  to indicate its readiness to accept an RDMAP/DDP mode
  association.
- The active side ULP would request a connection from a
  RDMA endpoint to a destination address and port.
- If there was no association already established to
  that SCTP endpoint, the SCTP association would be
  established as with normal SCTP associations, except
  that the Adaptation indication would specifcy RDMAP/DDP.
- The active side would select an unused Stream, and
  send a Stream Reset Data Chunk on that stream, including
  any ULP supplied Private Data.
- The passive side listener would receive the reset chunk.
  It would notify the ULP of the requested "connection",
  including any passed Private Data.
- To accept the connection request, the ULP would bind
  the SCTP stream to an RDMA endpoint, and provide
  the Private Data Response. An Adaptatio Layer Control
  data chunk with a Function Code of DDP Stream Reset would
  be sent to re-initialize the reverse stream and accept
  the "connection" request.
- To reject the connection request, the passive side
  would send an Adaptation Layer Control data chunk
  with a Function Code of DDP Stream Terminate, including
  any ULP supplied Private Data.

For MPA/TCP, a connection could be established in virtually
the same way:

- The passive side would listen on the destination port,
  to indicate its readiness to accept a connection.
- The active side ULP would request a connection from a
  RDMA endpoint to a destination address and port.
- The active side would initiate a TCP connection to the
  destination port. Acceptance/rejection of the connection
  would proceed as per normal TCP connection establishment.
  This would allow normal TCP gatekeepers, such as INETD
  and TCPserver, to exercise their normal safeguard/logging
  functions.
- Both sides would indicate their expectation to receive
  an MPA Mode frame, and thereby enter MPA mode.
- The active side would send an MPA Mode Frame including
  any ULP supplied Private Data.
- The passive side listener would receive the MPA Mode
  Frame. Notice of this state transisiton would be reported
  to the Consumer, including the supplied Private Data.
- To accept the connection request, the ULP would bind
  the TCP connections to an RDMA endpoint, and provide
  the Private Data Response. An MPA Mode Frame would
  be sent to confirm that this was to be an MPA mode
  connection, along with any ULP supplied Private data.
- To reject the connection request, the passive side
  would send an incompatible MPA Mode Frame response, 
  including any ULP supplied Private Data.
  
Note that under both protocols exchange of Private Data is
strictly optional. Neither the Adaptation Layer Control
data chunk or the MPA Mode Frame is required to contain
Private Data if the ULP has no desire to send it. When the
ULP has no desire to exchange Private Data, the ALC DDP
Stream Reset or MPA Mode Frames may be sent immediately
by both ends. There is no need to wait for the active
side "request" to arrive first.

The alternative to this approach is the conduct these
negotiations using the LLP alone. SCTP, even in RDMAP/DDP
adaptation mode, can still send and receive user messages.
A TCP connection can still send and receive byte streams.

This approach presents a challenge, however. The ULP using
TCP must consume the correct number of "negotiation" bytes
*before* transitioning to MPA and binding the TCP connection
with an RDMA endpoint. If the ULP would prefer a variable
length, or even optional, Private Data exchange this can
become a very tricky process.

More importantly it is a *different* process than the
exchanges required under SCTP or any other RDMA transport.

I believe the benefits of allowing uniform "connection"
establishment across reliable IP transports justify the
addition of variable length Private Data fields to the
DDP Stream Reset ALC data chunk and MPA Mode Frame.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Kanevsky, Arkady" <[email-masked]>
Date: Thu, 5 Jun 2003 08:35:47 -0700

But API can not define protocol only ULP can.
So, ITWG or DAT can not define a protocol for
connection management nor the one for private data.
Arkady

Arkady Kanevsky          email: [email-masked]
Network Appliance        phone: 781-768-5395
375 Totten Pond Rd.      Fax:   781-895-1195
3rd Floor                http: www.netapp.com
Waltham, MA 02451-2010   general phone: 781-768-5300

> -----Original Message-----
> From: Culley, Paul [mailto:[email-masked]]
> Sent: Thu, June 05, 2003 10:33 AM
> To: Caitlin Bestler; [email-masked]
> Subject: RE: [rddp] Transport Neutral "Connections"
> 
> 
> Regarding "private data";
> 
> DDP/MPA are currently expected to be enabled together, after creating
> the QP (and associating it with a PD).  The method we 
> expected to use to
> deal with private data was to have DDP/MPA's ULP (perhaps a 
> bit of code
> in the IT-API interface for example) be in charge of starting the TCP
> connections and then, after doing any negotiations needed between
> endpoints (using streaming mode TCP data), create the QP and 
> transition
> the connection to DDP/MPA mode.
> 
> The "private data" would be exchanged in TCP streaming mode (probably
> using standard sockets calls under the API).  When the Application/API
> had any necessary stuff completed (like figuring out the correct PD),
> the QP can be created, buffers posted, and the TCP 
> "connection context"
> passed to the iWARP stack to start DDP/MPA mode processing.  The Verbs
> document you referred to supports this type of startup.
> 
> I think that using the TCP streaming mode prior to MPA enablement is
> preferred to adding additional features to MPA as you suggest.
> The difference in flow over your connection establishment 
> sequence would
> only be that MPA is not involved until the "bind" step.
> 
> The IT-API group would have to publish a small draft 
> describing the use
> of the streaming mode and expected data format prior to iWARP startup,
> but this is little different than other applications or APIs 
> such as SDP
> etc.
> 
> Paul R. Culley
> HP Senior Fellow
> 281-514-5543
> 
> 
> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]] 
> Sent: Wednesday, June 04, 2003 11:07 AM
> To: [email-masked]
> Subject: [rddp] Transport Neutral "Connections"
> 
> 
> as to which underlying reliable transport is used as the LLP for
> RDMAP/DDP.
> 
> used. This has led to the development of the Direct Access Transport
> (DAT) Collaboriative's uDAPL and kDAPL, and to the Open Group's ICSC
> IT-API.
> 
> Support of these goals can be enhanced by a relatively minor change to
> both the SCTP adaptation and in the definition of the MPA Mode Frame.
> 
> With this change, ULPs would be able to engage in the same message
> exchanges prior to having a fully operational "connection" between two
> RDMA-enabled endpoints, whether using RDMAP/DDP over MPA/TCP, 
> over SCTP
> or even a non-IP transport such as InfiniBand.
> 
> InfiniBand and
> the draft-hilland proposed iWARP verbs) must be associated with a
> Protection Domain. No receive operations may be posted to the endpoint
> before it is associated with a Protection Domain. Indeed 
> under both the
> InfiniBand and proposed iWARP verbs an endpoint/QP is created within a
> Protection Domain.
> 
> There are some applications where the choice of Protection Domain is
> dependent upon the identity of the remote ULP client. For 
> example, if a
> user session requires multiple connections, it is highly 
> desireable for
> all of those connections to use a single Protection Domain.
> 
> of the connection have configured their RDMA endpoints compatibly on
> such matters as the RDMA Read capacity. Further ULP-specific uses are
> also presumed, such as establishing the identity of the client.
> 
> interface SHOULD
> report this data to the ULP when reporting the stream's state
> transition.
> 
> a stream. The Local interface SHOUD report this, and any included
> Private Data, to the ULP when reporting the stream's state transition.
> 
> report this data to the ULP when reporting the connections state
> transition.
> 
> Note that under both protocols exchange of Private Data is strictly
> optional. Neither the Adaptation Layer Control data chunk or the MPA
> Mode Frame is required to contain Private Data if the ULP has 
> no desire
> to send it. When the ULP has no desire to exchange Private 
> Data, the ALC
> DDP Stream Reset or MPA Mode Frames may be sent immediately by both
> ends. There is no need to wait for the active side "request" to arrive
> first.
> 
> negotiations using
> the LLP alone. SCTP, even in RDMAP/DDP adaptation mode, can still send
> and receive user messages. A TCP connection can still send and receive
> byte streams.
> 
> *before* transitioning to MPA and binding the TCP connection with an
> RDMA endpoint. If the ULP would prefer a variable length, or even
> optional, Private Data exchange this can become a very tricky process.
> 
> I believe the benefits of allowing uniform "connection" establishment
> across reliable IP transports justify the addition of variable length
> Private Data fields to the DDP Stream Reset ALC data chunk 
> and MPA Mode
> Frame.
> 
>  
> 
> Caitlin Bestler

---

From: "Culley, Paul" <[email-masked]>
Date: Fri, 6 Jun 2003 10:59:23 -0500

Below...

Paul R. Culley
HP Senior Fellow
281-514-5543

> Sent: Friday, June 06, 2003 10:13 AM
> To: Shah, Hemal
> Cc: Carrier, John; [email-masked]
> On 6/6/03, Shah, Hemal wrote:
> 
> >I strongly prefer that we have to deal with only two
> >modes: Stream mode and iWARP mode. How the Private Data is exchanged 
> >and CRC on/off capability is communicated is out of the scope of MPA 
> >specification.
> 
> So you are arguing that you cannot enter MPA mode without
> fully associating it with an RDMAP/DDP endpoint?
> 
> Wouldn't that mean that MPA could *only* be used with RDMAP/DDP?
That is the way that the current MPA draft is written...
> 
> The idea of shortening the CRC to cover only the header,
> as opposed to omitting it, was rejected because we 
> considered it inappropriate for MPA to know how long
> the RDMAP/DDP Header was.
> 
> But why wouldn't we if the only way to enable MPA
> was to associate it with an RDMAP/DDP endpoint?
Quite another topic...
> 
> There is obviously a desire to preserve the draft
> RDMAC iWARP verbs. But at least as far as that draft
> deals with enabling an LLP Stream, it has failed to
> deal with SCTP and fails to cleanly layer DDP over
> the LLP.
We freely admit that SCTP interface has not been thoroughly thought out,
mostly because there were no SCTP experts available to author those
sections...
> 
> The proposal to allow Private Data in the MPA Frame
> mode was an attempt to *minimize* changes to those verbs,
> but merely requiring that MPA be supported *before*
> enabling RDMAP/DDP would also solve the problem.
> 
> Either solution would work.
I agree...
> 
> I do not think it is appropriate to define MPA in a way
> which a) forces iWARP applications to directly use the
> LLP themselves for iWARP related functions, and
> b) assumes the local interface will horrendously
> rototill its *interface* to mandate that MPA, DDP
> and RDMAP all be enabled at the same time with a
> single user action.
A) I don't expect that 'applications' using standard startup APIs that
expect to use private data will have a problem; the API (and some other
draft standard) will deal with it.
B) For what it is worth, the vendors that helped in authoring the
various specs seemed to prefer the "big switch" approach to starting
iWARP; the idea of starting each layer at a different time was just not
very interesting.

---

From: "Williams, Jim" <[email-masked]>
Date: Thu, 12 Jun 2003 15:24:27 -0400

I strongly agree with the sentiments
from Caitlin below.

I have great concerns about that model that requires
the ULP to set up an LLP connection, optionally
negotiate RDDP, and then change connection mode to
RDDP.

My concern is about how the lines of responsibility
are drawn between the host OS vendor,
the NIC vendor, and the ULP developer.

If the ULP developer ends up with a number of 
different APIs equal to the product of the number
of NIC vendors supported and the number of OS's supported,
this becomes unworkable.

Likwise, the NIC vendors need to be able to treat
host OS's as closed black boxes.  Otherwise
schedules become open ended and impossible
to depend on.  Also costs to port to new
OS's become prohibitive.

There may be a way to resolve these issues, but
the complexity of the verbs draft, the close
coupling with the host OS, and the need for
ULP vendors to work with existing host OS
networking stacks leaves me pessimistic.

The idea of a transport neutral API that 
can be used for RDMA avoids most of these problems.
NIC vendors provide the entire API, and ULP
implementors write to it.

The other important piece is a software implementation
of RDMA that allows RDMA aware ULPs to use the RDMA API,
but communicate through the host OS network stack,
and do so with no loss (or minimal) loss of efficiency
of what could be done if the ULP used the host
OS network stack directly in non-RDMA mode.

> Sent: Saturday, June 07, 2003 10:15 PM
> To: John Hufferd; [email-masked]
> An application which forms a TCP connection, and *then*
> decides whether or not to enable RDMA mode, obviously
> must be aware that it is using TCP.
> 
> Indeed, for reasons I'll go into in a moment, that model
> does not work well for SCTP.
> 
> The model I was working from was one where the use of
> RDMA is presumed in advance. The ULP is RDMA specific.
> Their may be a non-RDMA equivalent, but it is running
> on a different port (or perhaps using a different IP
> address).
> 
> Not surprisingly this is the model that has dominated
> discussions within both the DAT Collaborative and
> the OpenGroup's ICSC. Their objective, after all, is
> to define transport neutral APIs for RDMA-aware
> applications.
> 
> So to rephrase more precisely, it should be possible
> for a transport neutral application to communicate
> the Private Date required to select a Protection Domain
> and configure the RDMA Read credits in a single message
> sent *before* the RDMA endpoint is selected -- and to
> do so in a transport neutral way.
> 
> To achieve this goal, either InfiniBand and SCTP must
> simulate a "stream" of pre-RDMA endpoint data, or TCP
> must provide a method of sending "one message".
> 
> I believe the best way to do so is to include a variable
> length Private Data field in the MPA Mode frame. The
> ability to send one Private Data message before selecting
> the RDMA endpoint is already present in InfiniBand and
> is easily defined for SCTP.
> 
> Alternatively, a protocol could be defined to delimit such
> a message over TCP independently of the shift into MPA mode.
> Either method  works.
> 
> In either case, the procedure is not imposed on any
> applications. Those that want to establish a TCP connection
> and then dynamically decide to enter RDMA mode are free to
> do so. If a Private Data field is defined, there will be
> no requirement to use it, and no prohibition against
> negotiating in plain TCP before its use.
> 
> But there are a larger group of ULPs that will be RDMA
> specific from the very beginning. Indeed, for some it would
> make no sense to have a non-RDMA version. Would you run
> SDP over plain TCP?
> 
> Even with less extreme cases, optimum use of RDMA is best
> achieved when the ULP is fully aware of RDMA's capabilities.
> While there will undoubtedly be servers that have valid
> reasons for dynamically deciding whether to enable RDMA,
> there will be at least as large a group that prefer to
> run the two distinctly different sets of logic through
> different daemons running on different ports. The server
> advertises two different sevices (whether by SLP, in
> the URL, or by whatever method), and the client chooses
> which to invoke.
> 
> 
> 
> There is an issue on the dynamic "in-session" switch to
> RDMA model as applied to SCTP. The current SCTP draft
> will not support such a model.
> 
> The rationale is that sharing control of streams within an
> single SCTP association between a non-RDMA stack and a RDMA
> stack would be very difficult. A likely adoption strategy
> would have the RDMA stack implemented on an RNIC, with the
> non-RDMA stack remaining "as is" on the processor.
> 
> Segregating SCTP associations between the
> RDMA-implementation and the current stack allows RDMA to be
> deployed without disrupting current stacks.
> 
> This same issue exists with TCP connections. But TCP only
> requires a relatively clean handoff from the conventional
> stack to the MPA/iWARP stack (relatively clean, because
> responsiblity for retransmitting might be split right at
> the shift point).
> 
> An SCTP implementation cannot shift just one Stream.
> Then handling of TSNs, SACKs and all other control chunks
> is common across all streams.
> 
> So is the correct response:
> 
> a) Don't worry about it. The "dynamic in-session" model is
>    highly specialized and inherently related to existing
>    servers that deal with long-haul connections where the
>    clients have little a priori knowledge of the server.
>    There is little need to support this model for SCTP.
> b) Require an iWARP capable SCTP stack to also support
>    non-iWARP streams. The trick is that non-iWARP streams
>    can have *ordered* Data Chunks. The handling of which
>    is both more complex and more consuming of memory
>    bandwidth than for an iWARP stream (which only uses
>    unordered data chunks).
>    
> If dynamic in-session switching is a general model that
> should be supported in an LLP fashion then the SCTP draft
> should find a way to accomodate it -- preferably in a way
> that doesn't make an implemetnation that much more
> difficult.
> 
> 
> 
> 
> Caitlin Bestler

---

From: John Hufferd <[email-masked]>
Date: Sat, 14 Jun 2003 14:28:45 -0700

Well I thought we were almost together, but I can not accept the statement 
that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive for some very important applications 
that must start their functional negotiations in TCP/IP mode, and then 
when RDMA mode is agreed upon, switch into that mode.  This permits the 
application to gradually move into RDMA mode across a network of nodes 
that implement RDMA at different rates (or not at all). 

Now the above statement should not indicate that there should not be a way 
to go directly to RDMA mode, it just says that we need an Overt method to 
accommodate the above type of negotiation whether or not we come up with a 
direct mode for entering RDMA, independent of the LLP.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Caitlin Bestler <[email-masked]>
06/14/2003 11:38 AM

        To:     John Hufferd/San Jose/IBM@IBMUS, [email-masked]
        cc: 
        Subject:        RE: [rddp] Transport Neutral "Connections"

On 6/14/03, John Hufferd wrote:

>Caitlin,
>I can not determine from your note whether you are now
>objecting to the application being able to perform its
own
>negotiation before entering RDMA mode (via an Atomic
last
>message operation) as is currently defined; or are you
>agreeing with that, but ALSO want a LLP neutral method
to
>enter the RDMA mode?

The latter.

But I don't view the LLP neutral as the "additional"
method. It should be the primary method. Conditional
negotiation is a special need for certain applications.
The LLP SHOULD be invisible to the ULP. If the ULP
wants to deal with the ULP, well then there certainly
isn't any reason to forbid it. I just don't want to
force the ULP to deal with the LLP - or even suggest
that it ought to. I would prefer to state forthrightly
that the ULP SHOULD NOT deal with the LLP.

The current proposals for MPA do exactly that. Vague
unspecified middleware to parse a "message" above the
stack does not solve the problem. It is a) still the
ULP, and b) unrealistically complex.

This middleware would be expected to consume the
*precise* number of bytes required, and leave the
remaining byte sitting for transfer to the MPA stack
*instantly* when the request to assign the TCP Stream
to an RDMA Endpoint/QP is made. This would be true even
if the MPA Mode frame had been included in the very
same TCP segment as the final pre-RDMA "message".

This creates a nasty problem for at least some valid
implementation strategies.

Assume that at least a few sites will not be eager to
turn over all TCP connections to a new stack.

Therefore there will be a non-MPA stack still running
on the host CPU, regardless of whatever
RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC.

The RNIC is also the Ethernet NIC. I am going to take a
bold guess here and venture that at least a few sites
will wish to preserve the current IP stack. The
existing IP stack is known to all of the other existing
applications that don't need this "RDMA" feature yet.
The same stack that has already been tested and audited
with all of those applications. New Ethernet NICs are
much simpler to test and validate than new IP stacks.
So at least some sites will want the RNIC to present
itself as a conventional NIC for the benefit of the
existing IP stack.

And Ethernet NICs handoff entire frames, not bytes from
TCP frames as the ULP reads them from sockets.

Now this handoff can be solved, but it is tricky.
Involving the ULP in the middle of it will only make
things far worse.

"Atomic" may be too strong of a word. But I do believe
that receipt of the final "negotiation" message and
entry to MPA mode needs to be co-ordinated at the
kernel and/or hardware layer. The ULP should only be
involved for policy issues -- not the mechanics. The
mechanics are messy *and* LLP specific.

To encourage, not just allow, the ULP to be LLP-neutral
we need to define a procedure where the ULP can exchange
messages that lead to the assignment of RDMA endpoints
(or QPs if you insist on callling them that, but if you
think about it, they no longer have two queues).

Further, certain of those messages are naturally trigger
events directly related to session state changes
(request, accept and terminate). Those semantic are
best recognized as they are parsed, not after a bounce
through the ULP that only delays things.

Caitlin Bestler

---

From: John Hufferd <[email-masked]>
Date: Tue, 17 Jun 2003 00:58:25 -0700

David,
I think I agree with your statement, but I am not quite sure what on the 
fly means.  Or exactly what "quiescing" means in this case.

If for instance we are saying that everything is brought to a half duplex 
like mode, where side A is waiting for side B to send a finial message 
before entering RDMA mode?  If so,  then I think that would be OK. 

This might mean, for example that:
1. Side B will need to have an atomic method of sending the last message 
in LLP mode, and then transiting into RDMA mode, and
2. After side A has received the last LLP message from B, it may then 
transit into RDMA mode.

Would you say that is quiescing and entering RDMA mode?
.
.
John L. Hufferd

[email-masked]
Sent by: [email-masked]
06/16/2003 03:35 PM

        To:     John Hufferd/San Jose/IBM@IBMUS, [email-masked]
        cc:     [email-masked]
        Subject:        RE: [rddp] Transport Neutral "Connections"

Folks,
 
I think there are three "models" of how to get into RDMA mode
under discussion:
 
(1) Enter immediately on startup, possibly with an initial frame
    containing private data required to setup the connection.
(2) Enter at some subsequent point, "quiescing" the connection first.
(3) Enter at some subsequent point, on the fly without a hiccup.
 
I read Caitlin's posts as saying that (3) is hard support in
hardware; in order to obtain the flexibility of entering RDMA mode
at some subsequent point the result is likely to require something
that looks like a handshake with the associated latency.  This doesn't
rule out (2), it just says that those who want that flexibility are
going to wind up paying some synchronization latency (Aside: it may
be that the structure of the ULP requires that this be paid anyway,
but that depends on the ULP).  Viewed from another angle, those who
want to minimize elapsed time from startup to blasting away in RDMA
mode will want to use model (1), and the proposed private data
mechanism may help make that model more applicable.  Those who wish
the generality of switching into RDMA mode later can use (2), with
the knowledge that some sort of handshake is needed.
 
I don't think I see a problem here - the main point is that the
flexibility of deferred startup may come with a price, i.e., ULP
designers should be thinking about (2), not (3).
 
Thanks,
--David
 
-----Original Message-----
From: John Hufferd [mailto:[email-masked]] 
Sent: Saturday, June 14, 2003 5:29 PM
To: Caitlin Bestler
Cc: [email-masked]
Subject: RE: [rddp] Transport Neutral "Connections"

.
.
John L. Hufferd

own
last
to

The latter.

Caitlin Bestler

---

From: John Hufferd <[email-masked]>
Date: Tue, 17 Jun 2003 13:20:24 -0700

David,
I agree with your statement about the extra step that must be done, and 
that side B can not send an RDMA message to A until it receives an RDMA 
message from A.

I think, however, more amplification is need  with regards to your last 
statement ...

"Side B's method does not need to be atomic - B can:
a) Transition receive side of connection into RDMA mode
b) Send last message to A in non-RDMA mode
c) Transition send side of connection into RDMA mode
In contrast, A can transition both sides of the connection into
RDMA mode at the same time because B's final non-RDMA message says
that the response from A is to be in RDMA mode."

From a verbs/API interface, this is what I am calling Atomic, I think it 
is up to the RNIC (or RDMA Stack) implementation to determine how this is 
actually accomplished, but the point is, that the application should see 
this as Atomic. This means that the implementation must make sure that it 
can handle, ACKs, and errors from the last non RDMA mode message while 
being able to handle any new RDMA incoming messages, etc. 

.
.
John L. Hufferd

[email-masked]
06/17/2003 07:32 AM

        To:     John Hufferd/San Jose/IBM@IBMUS
        cc:     [email-masked]
        Subject:        RE: [rddp] Transport Neutral "Connections"

John,

>> I read Caitlin's posts as saying that (3) is hard to support in 

> I think I agree with your statement, but I am not quite sure
> what on the fly means.  Or exactly what "quiescing" means in this case.

The problematic scenario is that one side sends a stream of data and
expects the other side to switch into RDMA mode in the middle (that's
"on-the-fly".  The implementation problem is that by the time the
receiver ULP figures out that it's supposed to switch into RDMA mode,
some of the RDMA-mode traffic has already been processed in non-RDMA
mode.  Quiescing is needed to avoid this.

> If for instance we are saying that everything is brought to a
> half duplex like mode, where side A is waiting for side B to send
> a finial message before entering RDMA mode?  If so, then I think
> that would be OK.
> 
> 1. Side B will need to have an atomic method of sending the
> last message in LLP mode, and then transiting into RDMA mode, and 
> 2. After side A has received the last LLP message from B, it may
> then transit into RDMA mode.
> 

Only for that half of the connection, side A still has to cleanly
enter RDMA mode, so there's another pair of steps involving a wait
for B:

3.  Side A transitions into RDMA mode and sends its first RDMA-mode
                 message to B.
4.  B waits for receipt of that message before sending any RDMA-mode
                 traffic to A (i.e., B cannot send any traffic to A after
                 sending its "last LLP message" until it receives an 
RDMA-mode
                 message indicating that A has switched).

Notice that both directions of traffic pause as part of this -
A has to wait for the last non-RDMA message from B (step 2) before
sending any RDMA traffic, and B has to wait for the first RDMA
message from A (step 4) before sending RDMA traffic.

Side B's method does not need to be atomic - B can:
a) Transition receive side of connection into RDMA mode
b) Send last message to A in non-RDMA mode
c) Transition send side of connection into RDMA mode
In contrast, A can transition both sides of the connection into
RDMA mode at the same time because B's final non-RDMA message says
that the response from A is to be in RDMA mode.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp