Subject: [nfsv4] pNFS file and use of RPC-over-RDMA to access an NFSv4.1 data server

From: David Noveck <[email-masked]>
Date: Wed, 18 Apr 2018 09:51:57 -0400

As far as I can determine, there has not been significant discussion of the
question of how a client which receives a pNFS file layout is to determine
whether the connection to the data server is to be established using TCP or
RPC-over-RDMA.  As a result, the Linux NFS client, when it receives a file
layout, unconditionally establishes a TCP connection to the data server.
I’ve had to disable pNFS support on the server for my testing, so that the
NFS-over-RDMA data paths can get tested but I’d prefer if we could arrive
at an approach which allows interoperable implementations supporting both
pNFS file and RPC-over-RDMA to be developed and tested.

I have discussed this issue with some people and arrived at an initial list
of ways in which this choice might be made.   Additions are possible but
our principal task is arriving at a compatible subset which can serve as a
basis for use in testing events and eventual standardization and
deployment.  One critical criterion governing our choice is the expected
penalty for making the wrong choice.   When this is high, the need is
strong for a correct decision, while, if we could reduce the penalty, a
reasonable basis for a guess could be an acceptable choice.  In this
connection, it is worth noting that currently Linux’s RoCE support for
RPC-over-RDMA winds up waiting 60 seconds (i.e. about 10 million times the
typical round-trip inter-node delay time) before deciding that
RPC-over-RDMA service is not available.  Until and unless that is changed,
we would need a quite accurate/reliable means of determining when such
support is available.

In any case, here is the list of approaches I have come up with so far.
I’d like the working group to augment the list as new approaches arise and
then decide on a way forward that we can proceed on for use in future
testing events and as a basis for standardization.

   - Deciding that, when the metadata server connection is RDMA, data
   connections are, by default, presumed to be RDMA.   This is a basis for a
   guess and would not be acceptable if the penalty for a wrong guess is
   high.   There will certainly be cases in which RDMA support is present on
   the MDS but might not be available on one or more DS’s.
   - Using the port specified in the layout with 20049 indicating that RDMA
   is to be attempted.   I don’t see any practical issue with this but it is
   kind of hacky.
   - Using a value of   “rdma’ or “rdma6” instead of “tcp” or “tcp6”, in
   the netaddr field of the device  info.  I’m not sure how hard or
   time-consuming it would be to have a new netaddr assigned in the case in
   which we use existing universal address formats.   We need to investigate
   this.
   - Expecting the client to connect, at first, using a tcp connection and
   then interrogating the fs_locations_info attribute and using the
   FSLI4TF_RDMA flag as an indication that the client is to access this fs
   using an RDMA-capable transport .
   - Expecting  the client to connect, at first, using a tcp connection and
   then do a CREATE_SESSION specifying CREATE_SESSION4_FLAG_CONN_RDMA, or to
   address transports in which a within-connection step-up is not possible i.e
   . RoCE or Infiniband, defniing a new similar create-session flag to support
   creation of a new connection to obtain RDMA support on the session using a
   BIND_CONN_TO_SESSION on the new connection.

---

From: Chuck Lever <[email-masked]>
Date: Wed, 18 Apr 2018 11:05:14 -0400

Hi Dave-

Thanks for bringing this up. I agree it should be explored.
A few comments below.

> On Apr 18, 2018, at 9:51 AM, David Noveck <[email-masked]> wrote:
> 
> As far as I can determine, there has not been significant discussion of the question of how a client which receives a pNFS file layout is to determine whether the connection to the data server is to be established using TCP or RPC-over-RDMA.  As a result, the Linux NFS client, when it receives a file layout, unconditionally establishes a TCP connection to the data server.   I’ve had to disable pNFS support on the server for my testing, so that the NFS-over-RDMA data paths can get tested but I’d prefer if we could arrive at an approach which allows interoperable implementations supporting both pNFS file and RPC-over-RDMA to be developed and tested.

We have found that there is a similar transport selection issue
that occurs when following NFSv4 referrals.

> I have discussed this issue with some people and arrived at an initial list of ways in which this choice might be made.   Additions are possible but our principal task is arriving at a compatible subset which can serve as a basis for use in testing events and eventual standardization and deployment.  One critical criterion governing our choice is the expected penalty for making the wrong choice.   When this is high, the need is strong for a correct decision, while, if we could reduce the penalty, a reasonable basis for a guess could be an acceptable choice.  In this connection, it is worth noting that currently Linux’s RoCE support for RPC-over-RDMA winds up waiting 60 seconds (i.e. about 10 million times the typical round-trip inter-node delay time) before deciding that RPC-over-RDMA service is not available.  Until and unless that is changed, we would need a quite accurate/reliable means of determining when such support is available.

Not a broad survey by any means, but I've consulted a few people
about the delay in rdma_resolve_addr when detecting end-to-end
RDMA capability on certain fabrics. It is not expected and
implementers should study it further.

There are many possible scenarios where bridging and routing can
be employed between fabrics that can break end-to-end RDMA
capability. I don't believe either a server or client by itself
can know for certain whether an end-to-end capability exists.
For example, the server can suggest the use of RDMA even when an
end-to-end capability does not exist.

This is similar in nature to the issue of assessing end-to-end
path bandwidth when a client attempts to select the most
performant network path to a server, though the consequences are
somewhat more existential.

I believe that whatever other mechanisms are put into place, this
issue must be resolved as well to ensure smooth interoperation on
all fabric types.

> In any case, here is the list of approaches I have come up with so far.   I’d like the working group to augment the list as new approaches arise and then decide on a way forward that we can proceed on for use in future testing events and as a basis for standardization.
> 
> 
> 	• Deciding that, when the metadata server connection is RDMA, data connections are, by default, presumed to be RDMA.   This is a basis for a guess and would not be acceptable if the penalty for a wrong guess is high.   There will certainly be cases in which RDMA support is present on the MDS but might not be available on one or more DS’s. 

In those cases the client should select a TCP connection to the
MDS, IMO, and the client implementation should use the same
transport capability for both. One could argue that a client
administrator would expect that the same transport protocol is
used for accessing the MDS and any DS when a pNFS file layout is
in effect.

> 	• Using the port specified in the layout with 20049 indicating that RDMA is to be attempted.   I don’t see any practical issue with this but it is kind of hacky.

I don't have a problem with this mechanism. A precedent has
already been established with non-pNFS connections, and the port
number is codified in standards-track RFCs.

> 	• Using a value of   “rdma’ or “rdma6” instead of “tcp” or “tcp6”, in the netaddr field of the device  info.  I’m not sure how hard or time-consuming it would be to have a new netaddr assigned in the case in which we use existing universal address formats.   We need to investigate this.

rdma and rdma6 are already IANA-assigned netids. See:

https://www.iana.org/assignments/rpc-netids/rpc-netids.xhtml

> 	• Expecting the client to connect, at first, using a tcp connection and then interrogating the fs_locations_info attribute and using the FSLI4TF_RDMA flag as an indication that the client is to access this fs using an RDMA-capable transport .

This seems like a reliable long-term solution, and it has the
benefit of providing not one but a list of interfaces and
transport types.

However, it falls prey to the issue above, where both ends could
support RDMA, but the end-to-end network path might not.

> 	• Expecting  the client to connect, at first, using a tcp connection and then do a CREATE_SESSION specifying CREATE_SESSION4_FLAG_CONN_RDMA, or to address transports in which a within-connection step-up is not possible i.e . RoCE or Infiniband, defniing a new similar create-session flag to support creation of a new connection to obtain RDMA support on the session using a BIND_CONN_TO_SESSION on the new connection.

The ultimate issue is path discovery. Here are other ideas:

- We invent a control plane protocol that allows a network
  administrator to specify, or some autonomous agent to discover,
  the true end-to-end capabilities of the storage fabric. Have
  clients consult this information before attempting to
  establish connections.

- A client should attempt to establish connections to all
  entries in an fs_locations result in parallel. The server
  can report the availability of RDMA-enabled interfaces,
  and for those entries, the client can try both RDMA and
  TCP.

  The first connection that completes is used, and the others
  are discarded as they complete or time out.

  This is also a good strategy for discovering end-to-end IPv6
  capability.

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Wed, 18 Apr 2018 16:07:17 +0000

> -----Original Message-----
> From: nfsv4 <[email-masked]> On Behalf Of Chuck Lever
> Sent: Wednesday, April 18, 2018 11:05 AM
> To: David Noveck <[email-masked]>
> Cc: NFSv4 <[email-masked]>
> Subject: Re: [nfsv4] pNFS file and use of RPC-over-RDMA to access an NFSv4.1
> data server
> 
> Hi Dave-
> 
> layout, unconditionally establishes a TCP connection to the data server.   I’ve

<all good stuff, sniped>
 
> > 	• Expecting the client to connect, at first, using a tcp connection and
> then interrogating the fs_locations_info attribute and using the FSLI4TF_RDMA
> flag as an indication that the client is to access this fs using an RDMA-capable
> transport .
> 

I just want to interject that this is what is done by the server side of SMB3
multichannel, which additionally provides an address and link speed. The
client then sorts by type and speed, chooses some preferences and performs
a routing lookup to assess possible connectivity for each type. It then proceeds
to attempt one or more additional connections, moving traffic to them if/when
they succeed.

> support RDMA, but the end-to-end network path might not.

Absolutely yes. And worse, if the connection succeeds but for some reason
cannot sustain heavy traffic. We see this All The Time with RoCE protocols,
which depend on accurate PFC and sometimes ECN behaviors to be
performed by each and every switch hop in the path. The symptoms are
maddening, and the solution even more so. We hope the RoCE vendors
will step up to address this over time with diagnosibility information, and
further refinement of the RoCE protocol requirements. I don't think that
pNFS or NFSv4 should attempt to address this.

Bottom line, simple is best at the pNFS/NFSv4 layer. I guarantee human
intervention will always be required, somewhere along the way.

Tom.

---

From: David Noveck <[email-masked]>
Date: Tue, 24 Apr 2018 09:34:02 -0400

Thanks to Tom and Chuck for their responses.  I'll respond to both  below.

First of all, let me say how my server will behave on this issue at future
testing events.

I intend to specify both rdma/rdma6 netaddr and a port number of 20049 on
the devinfos associated layouts when the data server supports RDMA.
Clients can then key off either of those two indications or they can use
the fact that the MDS connection is RDMA, if they choose.

I also intend to support use of  the FSLI4TF_RDMA flag as an indication
that an RDMA connection can be used but probably won't have that support on
my intial server.

Whatever the working group deciides to do on this issue, we need to
document the choice in some way.  Maybe we will need a short RFC on this
specfic issue rather than trying to update RFC8267 to include this.   This
short RFC would probably be marked as updating RFC5661, since it does
significantly revise chapter 13 of that document.  I have given some
thought to including this stuff in draft-ietf-nfsv4-mv1-msns-update but so
far it doesn't seem to fit very well.

With regard to Chuck's comments about the analogous problems for referrals,
I have some questions:

   - I assume the same problem can arise with regard to migrations,
   although, for that case it is a reasonable guess, in the absence of any
   other indications that the destination is of the same connection type as
   the source.  Are migration and referral essentially the same in this regard?
   - How useful is  the FSLI4TF_RDMA flag in addressing this problem?  Do
   we need anything else for satisfactory support?
   - What is the situation with regard to NFSv4.0?  How  can  RDMA
   connection be established for migration/referrals in the absence of
   fs_locations_info? Can we regard this as a can't-get-there-from-here
   situation?  If not, how do we address this?

I think that whatever is decided on these questions, this needs to be
addressed in draft-ietf-nfsv4-mv1-msns-update.  Given that this document is
presenting a major respecification of this area, it needs to do so in light
of the fact that multiple connection types are on the agenda now.  Similar
logic might apply in the case of draft-ietf-nfsv4-mv0-trunking-update as
well.  I intend to add discussion of these issues to
draft-ietf-nfsv4-migration-issues-15.

I agree with Tom that human internention is going to be required to deal
with switching/routing problems/incompatibilities although I am not joining
in his "guarantee".   Even though no protocol additions seem likely, we
have to be aware of these difficulties as we draft and review documents in
this area.  As Chuck indicates, the server's and client's knowledge that
they have support are not sufficient to conclude that it is appropriate
that it be used.  This probably means that we will need to avoid most uses
of RFC2119 terms as we describe this area.  It also seems to me that it
would be better if we avoided those terms' lower-case homonyms as well.

With regard to Tom's discussion of SMB3 facilities for presentation of
server interface characteristics, this has been discussed for NFSv4 but it
didn't seem to fit into NFSv4.1.   It is possible to provide such a
facility as an optional extension to NFSv4.2 if someone is interested in
working on this.  Unless there are objections, I will add a discussion of
this possibility to draft-ietf-nfsv4-migration-issues-15.

On Wed, Apr 18, 2018 at 12:07 PM, Tom Talpey <[email-masked]> wrote:

> FSLI4TF_RDMA

---

From: Chuck Lever <[email-masked]>
Date: Wed, 25 Apr 2018 13:30:17 -0600

> On Apr 24, 2018, at 7:34 AM, David Noveck <[email-masked]> wrote:
> 
> With regard to Chuck's comments about the analogous problems for referrals, I have some questions:
> 	• I assume the same problem can arise with regard to migrations, although, for that case it is a reasonable guess, in the absence of any other indications that the destination is of the same connection type as the source.  Are migration and referral essentially the same in this regard?

There are several scenarios that I will put in the category of
"automounted," for lack of a better term. The distinctive
characteristic of these scenarios is that a mount occurs, but
there is no clear opportunity to provide parameters for the
behavior of that mount; especially in this case, an option to
determine which transport is to be used.

+ cd'ing into an automounted directory
+ crossing a server FSID boundary
+ following a referral
+ recovering from a migration event
+ failing over to a replica

We might also place into this category any explicit mount
request where the transport type was not specified by the
administrator.

Client implementations typically have a set of policies which
determine the operating parameters of such mounts. On Linux
for example, proto=tcp is the default. The Solaris client tries
proto=rdma first, then falls back to proto=tcp.

For automounts, Linux currently tries proto=rdma only if the
parent directory uses proto=rdma (inheritance) and falls back
to TCP if RDMA is not available. Solaris will always try
proto=rdma first, then try proto=tcp.

The salient difference between these two implementations seems
to be that Linux supports RoCE and iWARP, where there are
scenarios in which it would be difficult to automatically
determine whether end-to-end RDMA is available; currently
Solaris supports only InfiniBand, which technically can speak
only to other IB hosts and thus determining whether end-to-end
RDMA is available is expected to be reliable.

> 	• How useful is  the FSLI4TF_RDMA flag in addressing this problem?  Do we need anything else for satisfactory support?

Because NFS is IP-based, there necessarily is a translation from
an IP address to a GUID for use by RoCE or InfiniBand. That is
handled by the client's Connection Manager implementation and
the HCA/NICs on the endpoints. iWARP goes through the same
motions but I don't believe an address translation is required.

Usually the CM can determine quickly that there is an
end-to-end RDMA path available. The issue is how frequently
there will be a combination of fabric and NIC that prevents
quick determination of RDMA capability.

> 	• What is the situation with regard to NFSv4.0?  How  can  RDMA connection be established for migration/referrals in the absence of fs_locations_info? Can we regard this as a can't-get-there-from-here situation?  If not, how do we address this?

See above: there are ad hoc implementations of transport class
negotiation that work on NFSv4.0.

NFSv4.1 has a "gear shift" capability where, after establishment
of a TCP connection, the client and server detect that an RDMA
connection is possible. At this point, I'm not aware of any
implementation of this capability. Are we as a community deciding
to abandon this approach for NFS?

> I think that whatever is decided on these questions, this needs to be addressed in draft-ietf-nfsv4-mv1-msns-update.  Given that this document is presenting a major respecification of this area, it needs to do so in light of the fact that multiple connection types are on the agenda now.  Similar logic might apply in the case of draft-ietf-nfsv4-mv0-trunking-update as well.  I intend to add discussion of these issues to draft-ietf-nfsv4-migration-issues-15.

The specific area of interoperability concern is probably limited
to exactly how a server advertises its RDMA capability, if we
conclude that is useful or necessary.

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Thu, 26 Apr 2018 08:23:31 -0400

> to abandon this approach for NFS?

I hope not! Note, the iSCSI/RDMA mapping, iSER, has a similar
capability in its "RDMAExtensions" key at logon.

But there are some challenges. RoCE and InfiniBand only support an
RDMA connection, and can't perform the step-up. And while the iWARP
(specifically, MPAv2) protocol does, the local APIs to drive the shift
are immature.

I believe the iSER and NFS/RDMA communities could work together to
enable this type of interaction. But as you point out, these are
implementation questions. The protocols are in place.

Tom.

---

From: "Black, David" <[email-masked]>
Date: Thu, 26 Apr 2018 15:40:17 +0000

> (specifically, MPAv2) protocol does, the local APIs to drive the shift

When the revised iSER RFC 7145 was published in 2014, there were no iSER implementations that supported this "gear shift" functionality.

The original iSER RFC 5046 required "gear shift" ... but implementations ignored that requirement.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Tom Talpey <[email-masked]>
Date: Fri, 27 Apr 2018 08:50:59 -0400

> The original iSER RFC 5046 required "gear shift" ... but implementations ignored that requirement.

Yes, that's a valid point, and the iSCSI "Portal Group" could be a
framework to reference. It provides some of the necessary cookies
for the client/initiator to select a protocol.

However, it does require external side protocols for discovery, and
most importantly requires a single port to serve a single protocol.
This leads to requiring more than one logical or physical interface,
when a target/server wishes to serve both RDMA and other transports.
Step-up/gear shift mode supports a single interface, seamlessly.

Tom.

---

From: David Noveck <[email-masked]>
Date: Sat, 28 Apr 2018 09:30:34 -0400

> to abandon this approach for NFS?

I don't see any reason to do so.  There are a bunch of things in RFC 5661
for which there is no current implementation.  In this case, it is not
surprising that this has not been implemented so far given the delay in
getting
RDMA-capable implementations in general.    The fact that this is an
iWarp-only
capability is an impediment to investing resources in this feature so
further
delays are likely.  However, there is no time limit for feature
implementations, so
I don't see any reason to abandon this.

>> I think that whatever is decided on these questions, this needs to be
addressed
>> in draft-ietf-nfsv4-mv1-msns-update.  Given that this document is
presenting a
>> major respecification of this area, it needs to do so in light of the
fact that multiple
>> connection types are on the agenda now.  Similar logic might apply in
the case of
>> draft-ietf-nfsv4-mv0-trunking-update as well.  I intend to add
discussion of these
>> issues to draft-ietf-nfsv4-migration-issues-15.

>> conclude that is useful or necessary.

Those matters are fairly clear for v4.0 and v4.1:

For v4.0, there is no way to do this and the issue is probably not so
serious that we should provide a correction (as provided for by RFC 8178),
so, while useful and deisrable, it is pretty much impossible at this point.

For v4.1, there is a way to advertise RDMA capability (using
FSLI4TF_RDMA) specified in RFC5661, and I certainly hope that

is useful, necessary and adequate.

It is possible that other, more extensive, facilities such as Tom mentioned
that
SMB3 has might be desirable as potential extensions in NFSv4.2.

Beyond the server capability, there needs to be discussion of possible
client responses.   In the context of trunking, a client TCP connection to
address X is distinct from the RDMA connection to that same address, so
the trunking descriptons, at least for v4.1, will need to worked over to
reflect
that fact.

For example, we have to cover the case in which the client discovers the
RDMA
capability using fs_locations_info, finds out after using EXCHANGE_ID that
the
connections are session-trunkble and binds that new connection to the
existing
session.

In any case, draft-ietf-nfsv4-migration-issues15 will mention areas that
might
need to be addressed to reflect the fact the combination client and server
address
does not identify a connection and that connection type needs to be
involved as
well.

On Wed, Apr 25, 2018 at 3:30 PM, Chuck Lever <[email-masked]> wrote:

> >       • I assume the same problem can arise with regard to migrations,
> >       • How useful is  the FSLI4TF_RDMA flag in addressing this
> problem?  Do we need anything else for satisfactory support?
>
> >       • What is the situation with regard to NFSv4.0?  How  can  RDMA
> addressed in draft-ietf-nfsv4-mv1-msns-update.  Given that this document
> is presenting a major respecification of this area, it needs to do so in
> light of the fact that multiple connection types are on the agenda now.
> Similar logic might apply in the case of draft-ietf-nfsv4-mv0-trunking-update
> as well.  I intend to add discussion of these issues to