Subject: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt

From: David Noveck <[email-masked]>
Date: Mon, 5 Jun 2017 11:15:04 -0400

Submitted with a small set of revisions to avoid expiration:

   - New contact information
   - Updated references
   - Changed some terminology.  "Send-based DDP" has become "send-based
   data placement", while "DDP" is limited to data placement using explicit
   RDMA operations.  Since there are corresponding XDR changes, this makes the
   diff large, but the actual extensions is pretty much unchanged

---------- Forwarded message ----------
From: <[email-masked]>
Date: Mon, Jun 5, 2017 at 10:58 AM
Subject: New Version Notification for
draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt
To: David Noveck <[email-masked]>

A new version of I-D, draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt
has been successfully submitted by David Noveck and posted to the
IETF repository.

Name:           draft-dnoveck-nfsv4-rpcrdma-rtrext
Revision:       02
Title:          RPC-over-RDMA Extensions to Reduce Internode Round-trips
Document date:  2017-06-05
Group:          Individual Submission
Pages:          44
URL:            https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-
rpcrdma-rtrext-02.txt
Status:         https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-
rpcrdma-rtrext/
Htmlized:       https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-
rtrext-02
Htmlized:       https://datatracker.ietf.org/doc/html/draft-dnoveck-nfsv4-
rpcrdma-rtrext-02
Diff:           https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-
rpcrdma-rtrext-02

Abstract:
   It is expected that a future version of the RPC-over-RDMA transport
   will allow protocol extensions to be defined.  This would provide for
   the specification of OPTIONAL features allowing participants who
   implement such features to cooperate as specified by that extension,
   while still interoperating with participants who do not support that
   extension.

   A particular extension is described herein, whose purpose is to
   reduce the latency due to inter-node round-trips needed to effect
   operations which involve direct data placement or which transfer RPC
   messages longer than the fixed inline buffer size limit.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 6 Jun 2017 08:42:52 +0200

Hi David,

send based data placement in it's current form only makes sense for
transfers from the client to the server (e.g. NFSv4 WRITE).  For transfers
from the server to the client (e.g. NFSv4 READ) the client has to place
the data into arbitrary buffers not controller by the NFS client,
so send-based data placement will lead to data copies on the client,
which will kill performance.  I'm speaking from experience because we
actually implemented this to verify the performance overhead for copies
in early NVMe over Fabrics prototypes.

That being said I've started drafting and prototyping a Verbs extension
for a tagged RECV WR.  Right now it's just a software implementation,
and there is conѕiderable hardware complexity if we want to implement
it in HCA.  I could talk about this idea a bit in Prague, but I'm not
sure about the venue - it's clearly not NFSv4 WG material, and there
doesn't seem to be a follow on to the STORM WG.  Ccing Dave Black if
he has any idea.

Also I find the idea to split requests into multiple sends just to
avoid RDMA ops very questionable - the latency advantage of SEND
from the client to the server vs RDMA read from the server mostly
matters to reduce the latency for small I/Os.  For large overheads
the registration (or in fact usually unregistration) overhead does
not matter in the grand scheme.  Last but not least modern RDMA
protocols have show us that we should simplify credit management
instead of making it even more complex as that has been one of the
major sources of bugs.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---end quoted text---

---

From: David Noveck <[email-masked]>
Date: Tue, 6 Jun 2017 10:32:50 -0400

> For transfers
> which will kill performance.

I'm clear that copies will kill performance, but I think you are wrong
about this
requiring copies.  The idea here is that the client will post as read
buffers, aligned
cache buffers not recently used.  The RNIC is free to pick one of those
buffers
when a send comes in.  so the buffers are not precisely controlled, as they
are
with DDP, but they are not arbitrary either.  They are buffers that the
client has
assigned to receiving block-aligned data.   This is basically the same as
the case in
wich the client picks less-recently used buffer to use when reading from
disk.  The
only difference is that, to accommodate the RNIC, the client is not in
control of which
read goes with which buffer (among the set of pending reads).   The client
can find
out which buffer got which block and map them to user addresses the same
way it does
with data read from block devices.

> sure about the venue -

Ultimately, you might wind up discussing this in a blue-sheet-less
environment.  For example, over a beer.

If you can send out the draft, I'd be interested in seeing it.  Others
probably
would as well.

> it's clearly not NFSv4 WG material,

Probably not, and given that the schedule is tight, there isn;t much point
trying
that.  Still, I think some people in the working group would be interested.

> and there

I there should be some follow-on.  Maybe A BOF could be thought about for a
future meeting.

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 6 Jun 2017 17:42:21 +0200

On Tue, Jun 06, 2017 at 10:32:50AM -0400, David Noveck wrote:
> cache buffers not recently used.

What cache buffer?  The destination of a READ I/O might be any piece
of memory in the system, including user memory.  And even for the page
cache most modern OSes need to allocate the page first before doing
I/O as it's the synchronization object.

Your idea might work in the context of a system that only uses buffered
I/O, uses a fixed buffer cache like a Unix system designed in the 70s,
and lets you heavily hack^H^H^H^Hoptimize the core code for your
super specialized use case.  It ain't gonna happen in practice.

> disk.

It's nothing at all like a page cache in the sense how most OSes use
it - we usually don't directly recycle 'buffers' but pages from the
general system allocator.  And they are page size (usually 4k or 64k(
and might not fit your I/O size at all.

If you want send based data placement to work you need tagged read
buffers so that the HCA uses exactly the read buffer your want, something
that is not supported by the current Verbs API, and not by any HCA
I know off.