Subject: [rddp] Stream Inititation (8.1)

From: Caitlin Bestler <[email-masked]>
Date: Tue, 24 Sep 2002 18:07:17 -0500

In section 8.1 the RDMAP draft specifies that the ULP peers
must agree on the send credits, rdma read credits, permitted
modes and configure the RDMAP/DDP layers as agreed before
RDMA mode is enabled.

This matches prior proposals.

It also states that that "RDMAP Stream initialization occurs
after the LLP Stream has been created (e.g. for DDP/MPA over
TCP the first TCP Segment after the SYN, SYN/ACK exchange)"

Wouldn't that make it impossible to negotiate connection
specific resource levels? There would still be protocol
specific, but negotiating connection specific would require
communicating *before* the connection was established.

Establishing a pair of connections between two NATted nodes
without protocol specific support in the firewall is
difficult to impossible.

In order to be truly transparent to all NAT boxes, doesn't
RDMAP have to be negotiated in a manner that allows it to
function over a single LLP connection without resorting to
UDP exchanges or multiple connections?

Is there a specific reason for not allowing a single TCP
segment exchange to establish stream-specific parameters?

This is less of a concern for SCTP, because the ULP could
easily use a different stream to negotiate RDMAP parameters
on the other streams.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Barron, Dwight" <[email-masked]>
Date: Thu, 26 Sep 2002 17:54:00 -0500

I'm a strong proponent of (b), which implies good ULP practice. However, there has to be mutual agreement between the transmitter and receiver at what point in the byte stream the transition is to occur and the RMDAP receiver (hw or sw) made aware of this transition point. I would think that the receiver would need to be able to detect data to the left of the transition point and at least raise an exception. Otherwise you are open to misinterpretation of TCP data vs. RDMA header. 

Regards,
Dwight

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Thursday, September 26, 2002 4:23 PM
> To: [email-masked]; [email-masked]
> Subject: RE: [rddp] Stream Inititation (8.1)
> 
> 
> Caitlin,
> 
> I believe the answer should be a version of b) - the ULP
> is responsible for determining the point in the stream
> at which the transition takes place and making sure
> both sides of the connection expect/perform the
> transition at that point.  ACKed vs. Un-ACKed seems
> like an issue for an implementation that doesn't
> grok the concept of "DDP was enabled at this point
> in the stream" when doing (re)transmits - a ULP dealing
> with such a <expletive deleted> implementation may
> need to resort to measures as extreme as a).
> 
> Thanks,
> --David
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 249-6449            FAX: +1 (508) 497-8018
> [email-masked]       Mobile: +1 (978) 394-7754
> ---------------------------------------------------
> 
> > -----Original Message-----
> > From: Caitlin Bestler [mailto:[email-masked]]
> > Sent: Thursday, September 26, 2002 5:04 PM
> > To: THALER,PAT (A-Roseville,ex1); [email-masked]
> > Subject: RE: [rddp] Stream Inititation (8.1)
> > 
> > 
> > On 9/26/02, THALER,PAT (A-Roseville,ex1) wrote:
> > 
> > >Caitlin,
> > >
> > >The statement is accurate but perhaps misleading taken
> > >alone. "After" was not meant to mean "immediately after".
> > >It can be "sometime after". The next sentences should
> > >clarify that meaning: "The ULP is responsible for
> > >transitioning the LLP Stream into RDMA enabled mode. Once
> > >in RDMA enabled mode, an implementation MUST send only
> > >RDMA Messages across the transport Stream until the RDMAP
> > >Stream is torn down."  
> > >
> > >The intent is that you can bring up a connection with
> > >RDMAP off, negotiate over the connection and then
> > >transition to RDMAP on.
> > >
> > >Pat
> > >
> > 
> > What should be specified as to any anacked portion of
> > outbound stream at the time of enabling RMDA?
> > 
> > a) It's not allowed. At the time of transition all
> > previously sent bytes must be fully acked. The next
> > TCP segment received is assumed to be in RDMA mode.
> > 
> > b) The transition is understood to take place after all
> > previously transmitted bytes. Co-ordinating this with the
> > peer is the ULP's responsibility. The first segment after n
> > bytes (as specified by the local ULP) will be assumed to be
> > in RDMA mode.
> > 
> > I would assume option A, but that also should be specified.
> > 
> > 
> > 
> > Caitlin Bestler

---

From: "Culley, Paul" <[email-masked]>
Date: Mon, 30 Sep 2002 07:52:28 -0500

embedded...

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Sunday, September 29, 2002 12:49 PM
To: Culley, Paul; [email-masked]
Subject: RE: [rddp] Stream Inititation (8.1)

On 9/29/02, Culley, Paul wrote:

>How about the following:
>
>1) Active side ULP LLP-sends the ULP specific switch msg.
>
>2) Passive side ULP LLP-receives it.
>
>3) Passive side enables RDMA mode, for RECEIVE only.
>
>4) Passive side then sends a streaming mode response,
saying "ok lets do RDMA, I'm in 
>receive mode".  Alternately, the passive can say, in
streaming mode, "no, no RDMA 
>today", and not enable RDMA receive.
>
>5) Active side ULP LLP-receives the response switch msg.
>
>6) Active side enables RDMA mode immediately (both
>send/recv).
>
>7) Active side performs first RDMA mode transfer. This
>serves as an ack of the Passive sides switch msg.
>

This can be made even better by requiring that the Active
side not transmit its first RDMA mode message until it has
LLP acked the last pre-RDMA mode message.

This would allow the Passive side to perform a clean
switchover to RDMA transmits without having to worry about
the possible retransmission of pre-RDMA segments.

<prc> This should not be a problem, even without requiring to Ack first; the response from Passive to Active that triggers the Active switch to RDMA must have been completely "delivered" to the application before the app could have enabled RDMA.  The DDP/RDMA delivery/completion guarantees require that the completed messages and all prior are all present.  Accordingly, the LLP, even if the Ack was not yet sent, must be ignoring "old" messages.  So even if a repeated TCP segment in non-RDMA mode arrives, TCP should drop it and RDMA ignore it.
<prc>

Caitlin Bestler

---

From: "Renato Recio" <[email-masked]>
Date: Mon, 30 Sep 2002 12:21:57 -0500

Caitlin,

In my view, the first paragraph of section 8.1 of the RDMA specification is
sufficient.  Specifically the second and third sentences:  "The ULP is
responsible for transitioning the LLP Stream into RDMA enabled mode. Once
in RDMA enabled mode, an implementation MUST send only RDMA Messages across
the transport Stream until the RDMAP Stream is torn down."

In my view, the second sentence covers your point.  In other words, if the
passive side ULP wants to wait for the active side to send an RDMA Message
it can do so.  On the otherhand, if the active/passive side immediately
start off in RDMA mode, the passive side can immediately start sending RDMA
Messages.

BTW, I created a presentation during the development of the .95 draft that
outlines a set of connection establishment options that are possible for
RDMAP.  I tried posting it to this reflector, but the e-mail was too long.
If you are interested in it,  let me know and I'll be happy to send it to
you.

Thanks,

Renato J Recio
Distinguished Engineer
IBM I/O Architecture and Technology
Tel 512-838-3685, T/L 678-3685

                      Caitlin Bestler                                                                                   
                      <[email-masked]        To:       [email-masked]                                                  
                      i.net>                   cc:       Michael Krause <[email-masked]>                             
                      Sent by:                 Subject:  RE: [rddp] Stream Inititation (8.1)                            
                      [email-masked]                                                                                 
                      rg                                                                                                

                      09/30/2002 10:56                                                                                  
                      AM                                                                                                

On 9/30/02, Michael Krause wrote:

>
>I do not see any change proposed for the RDMA / DDP / MPA
>specs in the below exchange.  Given there are no wire
>protocol changes proposed, then aside from an
>implementation education for ULP, it appears this is a
>non-issue from a specification perspective.
>

You are correct that there is no required change to any of
the wire protocol.

However, both the ddp and rdma drafts place requirements on
the local interface: specifically that the ULP can enter
RDMA mode at any any time after the stream is connected
(especially immediately).

As pointed out in the prior posts, this local interface is
inadequate. It either requires the ULPs to engage in timing
to avoid race conditions, or to have a complex interface
between the RDMA stack and traditional stack.

Based upon Paul Culley's refinement of my suggestion, it is
clear that the *only* refinements to the local interface are
as follows:

1) The ULP may enable RDMA send and RDMA receive modes
seperately.

2) The ULP may *fence* the first RDMA send pending receipt
of the first RDMA message. (So that the passive side can be
assured that the active side has completed enabling RDMA).

That is the *only* specification change required, and it
only deals with the very minimal "verb" description in
section 8.1.

I would argue that this very minor increase in the
reuqirements for the local interface is preferable to ULP
negotiations that have inherent race conditions in them
and/or complex "delivery revocation" interfaces with
software L2-up stacks.

It is also necessary that these capabilities be specified in
the draft. Later ULPs cannot define their own application
specific peer-to-peer unless the local interface
capabilities they rely upon are documented in the base
standard. There is no need to specify an API, or even a
verb, just that this capability is available to the ULP.

Caitlin Bestler