Subject: [nfsv4] CREATE_SESSION4_FLAG_CONN_RDMA

From: Karen Deitke <[email-masked]>
Date: Thu, 15 Nov 2012 13:17:26 -0700

Can someone please explain what it means to "step up" to RDMA mode on a connection?  My understanding is that the
connection is either an RDMA connection over IB, a TCP connection over IP or a TCP connection over IB.  I'm not
really sure how to "step up" to RDMA on a connection.

Thanks,

Karen

CREATE_SESSION4_FLAG_CONN_RDMA:

          If CREATE_SESSION4_FLAG_CONN_RDMA is set in csa_flags, and if
          the connection over which the CREATE_SESSION operation arrived
          is currently in non-RDMA mode but has the capability to operate
          in RDMA mode, then the client is requesting that the server
          "step up" to RDMA mode on the connection.  If the server
          agrees, it sets CREATE_SESSION4_FLAG_CONN_RDMA in the result
          field csr_flags.  If CREATE_SESSION4_FLAG_CONN_RDMA is not set

Shepler, et al.              Standards Track                  [Page 515]

    <http://tools.ietf.org/html/rfc5661#page-516>
RFC 5661  <http://tools.ietf.org/html/rfc5661>                          NFSv4.1                    January 2010

          in csa_flags, then CREATE_SESSION4_FLAG_CONN_RDMA MUST NOT be
          set in csr_flags.  Note that once the server agrees to step up,
          it and the client MUST exchange all future traffic on the
          connection with RPC RDMA framing and not Record Marking ([8  <http://tools.ietf.org/html/rfc5661#ref-8>]).

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 15 Nov 2012 20:45:06 +0000

Hi Karen,

If I remember correctly, it means that the server will start listening for NFSv4.1 requests on an RDMA channel over IB after creating the new session (see section 2.10.7.4). IOW: it “converts” the non-RDMA (i.e. TCP) channel that was used to create the session into an RDMA channel.  You can still use BIND_CONN_TO_SESSION to bind new TCP connections to this session (section 20.10.3.1) but the original TCP connection should not be reused (see section 18.36.4).

Cheers
  Trond

From: [email-masked] [mailto:[email-masked]] On Behalf Of Karen Deitke
Sent: Thursday, November 15, 2012 3:17 PM
To: [email-masked]
Subject: [nfsv4] CREATE_SESSION4_FLAG_CONN_RDMA

Thanks,

Karen

RFC 5661<http://tools.ietf.org/html/rfc5661>                         NFSv4.1                    January 2010

         connection with RPC RDMA framing and not Record Marking ([8<http://tools.ietf.org/html/rfc5661#ref-8>]).

---

From: Karen Deitke <[email-masked]>
Date: Thu, 15 Nov 2012 14:57:56 -0700

How is a channel defined as TCP or RDMA?  Can't a fore channel for 
example can contain both a TCP connection and an RDMA connection?    If 
create_session is received on a TCP connection over IB, how can that 
connection be "convered" to RDMA?

Thanks,

Karen

On 11/15/12 1:45 PM, Myklebust, Trond wrote:
>
> Hi Karen,
>
> If I remember correctly, it means that the server will start listening 
> for NFSv4.1 requests on an RDMA channel over IB after creating the new 
> session (see section 2.10.7.4). IOW: it “converts” the non-RDMA (i.e. 
> TCP) channel that was used to create the session into an RDMA channel. 
>  You can still use BIND_CONN_TO_SESSION to bind new TCP connections to 
> this session (section 20.10.3.1) but the original TCP connection 
> should not be reused (see section 18.36.4).
>
> Cheers
>
> Trond
>
> *From:[email-masked] [mailto:[email-masked]] *On 
> Behalf Of *Karen Deitke
> *Sent:* Thursday, November 15, 2012 3:17 PM
> *To:* [email-masked]
> *Subject:* [nfsv4] CREATE_SESSION4_FLAG_CONN_RDMA
>

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 16 Nov 2012 00:00:54 +0000

On Thu, 2012-11-15 at 14:57 -0700, Karen Deitke wrote:
> example can contain both a TCP connection and an RDMA connection?
> If create_session is received on a TCP connection over IB, how can
> that connection be "convered" to RDMA?  

Argh... I rechecked the RFCs, and it looks as if I'm remembering this
incorrectly. Ccing Tom Talpey to make sure that I get it right this
time...

So if I'm reading this correctly, you don't have a step-up phase for
Infiniband. Infiniband appears to be one of the "no streaming phase"
cases in section 2.10.7.4 of RFC5661.
I assume therefore, that you can just connect to port 20049 on the
server using ordinary RDMA/IB, just like you would for NFSv4 (see
RFC5666 and RFC5667) and then send an EXCHANGE_ID+CREATE_SESSION without
setting the CREATE_SESSION4_FLAG_CONN_RDMA flag.
Note that you should also be able to set up the EXCHANGE_ID
+SESSION_CREATE using TCP and IPoIB to port 2049 (again without
specifying CREATE_SESSION4_FLAG_CONN_RDMA), then attach an RDMA
connection to that session. Again, you'd use ordinary RDMA/IB to set up
a connection to port 20049, on which you then send a
BIND_CONN_TO_SESSION.

The "stepping up" operation is for use with a protocol like iWARP, that
conforms to RFC5040. In that case, you would connect using standard TCP
and send your RPC requests using standard record-marking based RPC/TCP
(streaming phase). Then once the reply to the CREATE_SESSION RPC request
is received, both client and server will fire up iWARP using the _same_
TCP connection that the CREATE_SESSION used, and they will both start
using RDMA.
So the CREATE_SESSION w/CREATE_SESSION4_FLAG_CONN_RDMA is acting as the
ULP "trigger" to start using RDMA as described in section 6.1 of
RFC5040.

For BIND_CONN_TO_SESSION on iWARP, I'm assuming that the
bctsa_use_conn_in_rdma_mode and bctsr_use_conn_in_rdma_mode take the
role of CREATE_SESSION4_FLAG_CONN_RDMA to induce the 'step up to RDMA'
phase. However the text actually references something which looks like
an obsolete version of that flag: csa_use_conn_in_rdma_mode and
csr_use_conn_in_rdma_mode. Tom, this needs an errata!

> > If I remember correctly, it means that the server will start
> > listening for NFSv4.1 requests on an RDMA channel over IB after
> > creating the new session (see section 2.10.7.4). IOW: it “converts”
> > the non-RDMA (i.e. TCP) channel that was used to create the session
> > into an RDMA channel.  You can still use BIND_CONN_TO_SESSION to
> > bind new TCP connections to this session (section 20.10.3.1) but the
> > original TCP connection should not be reused (see section 18.36.4).
> > 
> >  
> > 
> > Cheers
> > 
> >   Trond
> > 
> >  
> > 
> > RFC 5661                         NFSv4.1                    January 2010
> >  
> >  
> >          connection with RPC RDMA framing and not Record Marking ([8]).
> 

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Tom Talpey <[email-masked]>
Date: Thu, 15 Nov 2012 23:09:10 -0500

That is correct - the optional session "step-up" exchange was designed
to support iWARP RDMA operation without requiring a new port, new
firewall rules, or RDMA listens. This handshake can be used from TCP
mode on port 2049, when both client and server agree using the
CREATE_SESSION4_FLAG_CONN_RDMA flag. (A similar handshake is defined by
iSER to support iSCSI over iWARP RDMA, on port 3260 by negotiating the
RDMAExtensions key at login.)

Use of this exchange is not required, however. When the server listens
on port 20049, RDMA mode is implicit without negotiation. Infiniband
always does this, because it does not support a streaming-to-RDMA
transition. NFSv[234]/RDMA use 20049, because they lack the upper
layer negotiation. For simplicity, even NFSv4.1/iWARP implementations
do the same.

The bctsa_use_conn_in_rdma_mode is different, that's a similar
negotiation, but describing the backchannel mode when a new connection
is made for one. I'll need to look into the csa_ ones; their usage (or
non-usage) may be left over from some earlier spec revisions.

Tom.