Subject: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-xcharext-00.txt

From: David Noveck <[email-masked]>
Date: Mon, 11 Apr 2016 10:44:49 -0400

This is a "low-hanging fruit" extension, building on the extension
mechanism defined in draft-cel-nfsv4-rpcrdma-version-two-00.

The principal benefits are:

   - The ability to use inline buffer sizes larger than the 4K default in
   draft-cel-nfsv4-rpcrdma-version-two.
   - The ability to decide on use of remote invalidation.

---------- Forwarded message ----------
From: <[email-masked]>
Date: Mon, Apr 11, 2016 at 10:24 AM
Subject: New Version Notification for
draft-dnoveck-nfsv4-rpcrdma-xcharext-00.txt
To: David Noveck <[email-masked]>

A new version of I-D, draft-dnoveck-nfsv4-rpcrdma-xcharext-00.txt
has been successfully submitted by David Noveck and posted to the
IETF repository.

Name:           draft-dnoveck-nfsv4-rpcrdma-xcharext
Revision:       00
Title:          RPC-over-RDMA Extension to Manage Transport Characteristics
Document date:  2016-04-11
Group:          Individual Submission
Pages:          22
URL:
https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-rpcrdma-xcharext-00.txt
Status:
https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-rpcrdma-xcharext/
Htmlized:
https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-xcharext-00

Abstract:
   It is expected that the RPC-over-RDMA transport will, at some point,
   allow protocol extensions to be defined.  This would provide for the
   specification of OPTIONAL features, allowing participants who
   implement the OPTIONAL features, to cooperate as specified by that
   extension, while still interoperating with participants who do not
   support that extension.

   A particular extension is described herein, whose purpose is to allow
   RPC/RDMA Endpoints to specify and manage their transport
   characteristics in order to allow the other participant to optimize
   message transfer in light of the characteristics communicated by the
   initial sender.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Chuck Lever <[email-masked]>
Date: Mon, 11 Apr 2016 11:47:34 -0400

> On Apr 11, 2016, at 10:44 AM, David Noveck <[email-masked]> wrote:
> 
> This is a "low-hanging fruit" extension, building on the extension mechanism defined in draft-cel-nfsv4-rpcrdma-version-two-00.
> 
> 
> 	• The ability to use inline buffer sizes larger than the 4K default in draft-cel-nfsv4-rpcrdma-version-two.

Before choosing a protocol mechanism for increasing the
inline threshold, there are some issues to consider.
This was about where our phone call ended a couple of
weeks ago.

From an implementer's perspective, I like the idea of
each peer advertising its receive buffer size. This is
a simple way for implementations to support larger
inline thresholds.

However I'd like to see use cases for anything more
complicated like negotiating inline threshold in the
midst of operation. IMO there is currently no association
between a particular ULP and a particular inline
threshold. There is no exposure of the threshold to
upper layers in current implementations.

Mechanically speaking, a receive buffer can be replaced
only when it is consumed, and the ULP has no control over
which receive buffers the hardware chooses to consume to
satisfy an incoming RDMA Send. It might be difficult for
a receiver to tell when it can safely receive larger
messages, unless it tears down all posted Receives and
reposts with larger buffers.

If an inline threshold increase involves replacing
receive buffers, I'd like to see one or two plausible
implementation mechanisms for replacing receive buffers
without a connection loss (either mandated or accidental).

If a receiver implements SRQ, then the receive buffers
for all connections are the same size. Changing the
inline threshold on one connection must not adversely
effect other connections.

Given the above I would be hesitant to attempt an
implementation that requires receive buffer size changes.
I might provide a range of supported inline thresholds,
but use a fixed buffer size. This is simple and safe, but
might not be memory efficient.

Instead of altering the inline threshold, message chaining
would allow the transport to convey requests larger than
the inline threshold using only RDMA Send, and would also
possibly allow DDP using only RDMA Send (RDMA_MSGP). It
also would not require any changes to receive buffer size.

The only downside seems to be increased complexity of credit
accounting (each chained Send consumes one credit). Today
most implementations assume an RPC always consumes exactly
one credit, and this would have to change.

IIRC SMB Direct allows message chaining and uses a single
fixed receive buffer size.

> 	• The ability to decide on use of remote invalidation.

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Mon, 11 Apr 2016 18:03:00 +0000

Chuck all good points about the difficulties of renegotiating the receive

> fixed receive buffer size.

Correct, the SMB Direct protocol uses relatively small buffers (1364 bytes
in current implementations), and allows senders to chain multiple messages
into a logical single datagram. These fragments are subject to the usual
crediting, etc, and their relatively small size has proven very convenient
resource-wise.

Personally I would suggest being VERY careful in encouraging upper layers,
particularly NFS, to use "large" messages, which I'd define as >4KB. It is
uncommon that all messages, in both directions, would require such a
buffer, and there can be many issues providing resources in support of
this. However, flexibility in the transport layer is good.

Finally, I will mention that RPC itself has precedent for a similar chaining
behavior. The RPC/TCP record marker defines the high-order bit of the
length field as a "more fragments" indicator. Multiple TCP segments
can be merged into a single logical message with it. However, I don't
believe many NFS/RPC/TCP implementations actually send it.

Tom.

> -----Original Message-----
> From: nfsv4 [mailto:[email-masked]] On Behalf Of Chuck Lever
> Sent: Monday, April 11, 2016 8:48 AM
> To: [email-masked]
> Subject: Re: [nfsv4] New Version Notification for draft-dnoveck-nfsv4-
> rpcrdma-xcharext-00.txt
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.i
> etf.org%2fmailman%2flistinfo%2fnfsv4&data=01%7c01%7cttalpey%40micros
> oft.com%7c3b385126a63d4942a2d208d36220a030%7c72f988bf86f141af91ab2
> d7cd011db47%7c1&sdata=lp1%2b7ZqyBTjOQqEu6XY2aWO7UzM7IWvgzWfpX
> lr7plk%3d