Subject: [nfsv4] NFS/RDMA next steps

From: Chuck Lever <[email-masked]>
Date: Mon, 31 Jul 2017 14:34:17 -0400

Hi-

During the nfsv4 WG meeting at IETF 99, I presented some slides on possible
next steps for NFS/RDMA and related protocols.

https://datatracker.ietf.org/meeting/99/materials/slides-99-nfsv4-nfsrdma-next-steps-chuck-lever

There are many areas that could use some attention, yet only a handful of
engineering resources are available. Slides 11 - 18 describe the directions
that could IMO be fruitful individual next steps.

I hoped my talk could frame a conversation about where we think the highest
priority is, but it was decided that there were enough interested and
informed people who were not present that such a conversation should be
moved to this mailing list and continued after IETF 99.

Note that RFC 5667bis is now "Submitted to IESG for Publication". The work
items in the slides assume that this document will be completed and published
as planned.

The slides split the possibilities into three somewhat orthogonal groupings.
A fourth grouping arose during discussion in the room, which I'll add as
"Grouping Zero" below. We can choose any or all of these approaches. Opinions
are welcome as to what order, whether something is left out, or what might be
removed from this list. Groupings Two and Three could introduce new Working
Group documents, and thus have implications for our charter milestone count.

Grouping Zero:
Focus on improving existing implementations of RPC-over-RDMA and NFS/RDMA. No
IETF action needed, which is why I didn't include this on the slides. There
are substantial improvements that can be made to existing base implementations,
but these would be done by many of the same folks who would be working on new
protocol.

Grouping One:
Enable greater transport parallelism in NFS. This includes multipathing and
use of pNFS with RDMA. No changes to RPC-over-RDMA or NFS/RDMA are necessary,
and this would bring important performance capabilities to NFS, especially
by enabling very low latency client access to Storage Class Memory.

Grouping Two:
Incrementally improve RPC-over-RDMA version 1. The main idea here is to
introduce a per-connection transport property negotiation mechanism to replace
CCP. This would enable variable size (ie larger) inline thresholds and the use
of Remote Invalidation in some instances with existing deployments.

Grouping Three:
Pursue RPC-over-RDMA version 2. This would open a variety of avenues by which
many of the perceived shortcomings of RPC-over-RDMA version 1 could be
addressed.

IMO Zero and One are where we can get the greatest bang for the buck in the
near term.

Latency to access Storage Class Memory is substantially shorter than the
latency of traversing the NFS and RPC stack on just the client. Thus
bypassing RPC entirely (eg by using an RDMA layout type) seems like the best
strategy we have for tapping the potential of this new variety of durable
storage.

The current proposal for Grouping Two (draft-cel-nfsv4-rpcrdma-cm-pvt-msg) is
controversial. Grouping Three would be an immense amount of work to generalize
some things for less gain than we might see with work in Grouping Zero or One.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 31 Jul 2017 16:04:35 -0400

> items in the slides assume that this document will be completed and
published

It's been that way for seventeen days.  I expect it to be done in about six
months but RFC 8154 took about a year.  Sigh!

> The slides split the possibilities into three somewhat orthogonal
groupings.

You had to divide these up to produce slides, but everyone will divide
things
differently.  In any case, I don't see these divisions as all that
helpful.  I think we
need to decide what documents are ready to become working group
documents based on their maturity and promise and believe that is the
appropriate way to frame the discussion.

> Opinions

Note that Grouping One does as well.  See below.

> IETF action needed, which is why I didn't include this on the slides.

For the same reason it might not be all that appropriate to this
discussion.  The balance
between iimplementation and specfication is a decision for individual
companies to
make.  In my case, the work is concerned with the development of a new
implementation
but the same principle applies.  My employer will have opinions about this
question and
I'm going to pay attenntion to my company's needs.  It is not up to me how
Oracle people
spend their time or up to Oracle about how mine is spent.

> There

True.  If people are so busy doing implementation that they have no time to
work on
specs we will have to adjust.

> Grouping One:
> Enable greater transport parallelism in NFS.

This strikes me as a highly unnatural grouping to use as a basis for
decision-making.
It might make sense in a slide preentation.

> This includes multipathing

I'm working on this.  I mentioned prodcucing
draft-dnoveck-nfsv4-mv1-msns-update at the meeting.

My understanding is that Chuck will submit draft-cel-mv0-trunking-update
which addresses
trunking/multipathing in the v4.0 context.

> and

I agree with doing this but what we heard at ietf99 was a very early
draft.  This is a long-term effort which should be pursued but this
document is quite a ways from being ready to become
a working group document.

> of Remote Invalidation in some instances with existing deployments.

Implementation of this is important and will be proceding.  I intend to
implement it
regardlersss of working group decisions about priorities or the pospect of
controversy.
Perhaps this belongs in group Zero.  The work to get this
already completed specfication work through the IETF process is relatively
small although there may be controversies that need to be resolved.  See
below.

> IMO Zero and One are where we can get the greatest bang for the buck in
the

Two issues with this:

   - I believe Zero is out of scope for this discussion for reasons already
   given.
   - The pNFS-RDMA layout type is not a near-term item.  I agree we should
   pursue it, but it is not going to happen soon, especially if much of the
   working group is busy doing implementation work.  In any case, when
   Christoph thinks this is ready to be a working group item, we should
   definitely consider this seriously.

> controversial.

if Tom, or anyone else, has objections to proceeding with this, we need to
discuss this on the list and work to arrive at a reasonable resolution.  My
understaning is that Tom objects to this being standards-track.  While I
believe it should be standards-track, I can live with an Informatinal RFC.
  I just don't want ths to be an undocumented de facto standard.

> Grouping Three would be an immense amount of work to generalize

I don't think "immense" is the right word but let's not argue about that.
In any case, we need
relef from the major performance issue that were somehow baked into Version
One.  We can
live with Version One if we have the implementation corresponding to
cm-pvt-msg.  I'm not sure whether this is Grouping Two (as above) or Zero
(because we are are talking about existing implementations).  In any case
we have running code and should take advantage of Chuck's good work in
specifying this regardless of any potential controversy.

On Mon, Jul 31, 2017 at 2:34 PM, Chuck Lever <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>