Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5666bis-04.txt

From: [email-masked]
Date: Fri, 04 Mar 2016 08:00:39 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Network File System Version 4 of the IETF.

        Title           : Remote Direct Memory Access Transport for Remote Procedure Call
        Authors         : Charles Lever
                          William Allen Simpson
                          Tom Talpey
	Filename        : draft-ietf-nfsv4-rfc5666bis-04.txt
	Pages           : 61
	Date            : 2016-03-04

Abstract:
   This document specifies a protocol for conveying Remote Procedure
   Call (RPC) messages on physical transports capable of Remote Direct
   Memory Access (RDMA).  It requires no revision to application RPC
   protocols or the RPC protocol itself.  This document obsoletes RFC
   5666.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5666bis/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5666bis-04

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-rfc5666bis-04

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Chuck Lever <[email-masked]>
Date: Fri, 4 Mar 2016 08:10:44 -0800

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5666bis-04.txt
> Date: March 4, 2016 at 8:00:39 AM PST
> To: <[email-masked]>
> Cc: [email-masked]
> 
> 

In addition to a number of clean-ups, the new revision has
two substantive changes:

- Dave Noveck's changes to enable XDR extensibility have been
integrated

- A section explaining the conventions for the operation of
RPCSEC_GSS on RPC-over-RDMA has been added

Let's use rfc5666bis-04 as a fulcrum for discussion during
the upcoming WG design phone call. During this call I'd like
to visit the following topics:

- Identify and discuss remaining controversies in rfc5666bis

- Agree on a plan for getting rfc5666bis through WGLC

- Define a process for rfc5667 document updates

- Assess post-rfc5666bis extensions for RPC-over-RDMA

--
Chuck Lever

---

From: William Allen Simpson <[email-masked]>
Date: Mon, 7 Mar 2016 23:56:44 -0500

> - Dave Noveck's changes to enable XDR extensibility have been
Chuck gave me a head's up during Connectathon.  I had not seen
them until then.  I'm opposed to these additions.  I'd hoped to
completely re-write them before posting, but Chuck decided that
in the spirit of openness we should post this previously private
contribution (secret from me) and then replace it.

> RPCSEC_GSS on RPC-over-RDMA has been added
This seems to me a much needed section, based on discussed
difficulties in implementation.

However, IMnsHO, after so much effort to merely document
Version One for posterity, this delves into the realm of
entirely new material.

Wouldn't this be better in Version Two?

> - Assess post-rfc5666bis extensions for RPC-over-RDMA
We have learned that no Version One implementation is extensible,
because no shipping Version One implementation checks for new
message header types (Procedure Numbers).  All of them fall through
and pass the unknown data up as NFS XDR.  Since anticipated
extensions don't have the same data fields, this is unusable,
potentially causing massive failures.

While mine (non-shipping Ganesha/nTI-RPC) was the only
implementation checking Procedure Numbers, it has the problem that
it closes the connection immediately.  This is a result of telling
Ganesha that the Call failed, so that it doesn't try to build a
Reply.  This will operate properly as the client can open a new
connection, but it would be faster to simply report the error and
await another Call.  (Such requires a library API change and
significant restructuring.)

Therefore, for any extensions, we must define Version Two.

Fortunately, every implementation now tests its Version Number!

It seems to me that our time would be better spent declaring that
Version One is deprecated and working on Version Two.  I do not
feel a moral imperative to "fully document" a deprecated version.

I propose expanding and publishing Chuck's excellent implementation
experience document, and making RFC5666bis be Version Two, recycling
at Proposed Standard.

---

From: David Noveck <[email-masked]>
Date: Tue, 8 Mar 2016 06:32:03 -0500

> but Chuck decided that

It seems like you are saying that Chuck posted a -04, with the intention
that
a major change in it be replaced, did not inform the working group of
that, and then proceeded to tell the working group that that draft should be
the fulcrum of discussion in our forthcoming conference call.

This doesn't accord with the presumed "spirit of openness" that you cite
above or with what I've come to expect from Chuck as far as actual
openness/honesty.

If Chuck does propose extensibility-related changes to -04, I'll assume
that he reached the conclusion that these are needed after his 3/4 message
to the working group.

BTW, the implementation problems with servers not checking message
header types  (and implications for extensibility) were discussed on the
working group list on 2/4.

On Mon, Mar 7, 2016 at 11:56 PM, William Allen Simpson <
[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever <[email-masked]>
Date: Tue, 8 Mar 2016 12:51:24 -0500

> On Mar 7, 2016, at 11:56 PM, William Allen Simpson <[email-masked]> wrote:
> 

Dave's review comments on -03 were posted to the nfsv4
mailing list on February 1, 2016, and are archived on
the datatracker.

Bill, please fix your mail sorting rules and your paranoia.
This is not the first time you have missed pertinent
email items.

> Wouldn't this be better in Version Two?

No. There has been a working Version One implementation
of RPCSEC_GSS for years, making it a current practice.

Instead, I would consider removing RPCSEC from rfc5666bis
and into a separate I-D, documented as an extension to V1
that does not change the V1 XDR. That would depend on
whether reviewers think there needs to be additional
security specified in the same document that obsoletes
RFC 5666.

RFC 5531, Section 14, paragraph 3 states "All non-trivial
RPC programs developed in the future should implement
RPCSEC_GSS-based security appropriately." The current RFC
5666 does not enable RPCSEC_GSS-based security at all.

My experience with the IESG is that one or more of them
are well aware of the above paragraph, and thus might
require explicit RPCSEC support in any document that
obsoletes and replaces a current RFC, especially
considering existing IETF politics around Internet
security.

The current remarks in RFC5666 about preferring a link-
layer protection scheme would likely be found wanting in
a freshly-issued Proposed Standard. In that light I
prefer that the RPCSEC text be left in rfc5666bis.

Likewise, I would consider removing bi-directional RPC
from rfc5666bis and moving the existing separate BiRPC
I-D forward as an extension to V1 that does not change
the V1 XDR.

XDR extensibility, as Dave proposed, does indeed change
the V1 XDR, and also is not a current implementation
practice. In that regard doing XDR extensibility in a V2
rather than as part of rfc5666bis makes sense.

But the objection all along to this approach has been
that a V2 enables and even invites broader changes and
thus could take a very long time to complete. So we
attempted to add XDR extensibility to V1.

> We have learned that no Version One implementation is extensible,

They are extensible. It would require some bug fixes
and some potentially odd conventions, which we would
have to live with forever after. That may be
aesthetically displeasing, but it is workable.

> potentially causing massive failures.

We have a workaround for this issue. The only problem
with this workaround is it is ugly.

> Therefore, for any extensions, we must define Version Two.

A Version Two would make extensibility cleaner, but
so far there has been no compelling technical
imperative to create a V2 to enable detection of
optional features.

> Fortunately, every implementation now tests its Version Number!

The Solaris server may test the version number, but it
drops requests with unrecognized versions, and does not
reply to them.

But for the record, RFC 5666 very explicitly describes
the correct response for this case. We can say verily
that the Solaris server does not comply with existing
specification, which has no technical error here. That
server implementation should be fixed.

The practical reality is that even bumping the version
number is not a perfectly clean technical solution.

> feel a moral imperative to "fully document" a deprecated version.

There are two important reasons why we need a working
V1 specification.

1. There are deployed systems using V1. A V2 would not
interoperate with these.

2. Any extensions to V1, including a V2, would need to
have a foundational V1 specification. RFC 5666 is not
adequate for that because of its ambiguities and a few
missing pieces.

> experience document, and making RFC5666bis be Version Two, recycling

Changing rfc5666bis to specify a V2 would mean it no
longer updates the RFC that specifies V1. What you are
thus proposing is that rfc5666bis be scrapped and a new
document be created.

It would be foolish to walk away from V1 now. We are
very close to finishing rfc5666bis. If the controversial
features above are removed, IMO rfc5666bis can be moved
through WGLC quickly.

The good news is I can't think of another feature that
needs to be added to rfc5666bis. It's just a matter of
deciding what to keep and what to jettison before the
I-D is moved forward.

I believe all of us want to have a robust RPC-over-RDMA
extensibility mechanism as soon as possible. The issue
has always been the prospect of a long V2 design and
prototyping process. In other words, we are attempting
to work around a WG process problem, not a technical
issue.

If the preference is to avoid invasive changes to V1 and
rely on the existing versioning mechanism in the protocol,
how can V2 be made quickly attainable?

One answer to that is to limit, at the outset, the set of
new features in V2, and then stick to it. Suppose, for
instance, that V2 contains only one new feature: XDR
extensibility.

We would need a specification of what features are not
changing, and that's rfc5666bis. There is already text
and a framework for XDR extension that can be recycled
from rfc5666bis-04. In a V2, certain XDR changes that
were constrained in V1 would be allowed. What remains
is polishing the protocol and creating interoperating
prototypes.

Then implementations can choose V1 or V2, can interoperate
with existing deployed servers and clients, and new
features can subsequently be added without needing to
join a batch of other features.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 8 Mar 2016 14:49:26 -0500

> especially

I've had some experience with current IESG attitudes in dealing with the
review of draft-ietf-nfsv4-rfc3530-migration-update.  My advice is that if
you think something might be a problem, it is very likely to be a problem,
and if
you have no idea  why something might be a problem, there's a pretty good
chance it will be one as well.

> I-D forward as an extension to V1 that does not change

So it appears that you are proposing that 5666bis, as defining all of
RPC/RDMA.
provide for extension by means of defining optional features, as long as
those
features make no change to the XDR.

I think this is a good direction but the problem is that if you either
explicitly say
or by default allow people to assume that any XDR change implies a version
number change, then you have just foreclosed the option of an XDR extension
model in a later version.

I think there are two possible ways around this issue:

   - Keep everything that is now in section 9 as applying to all future
   RPC/RDMA versions as it is, but modify section 9.2 to say that no
   XDR extensions may be made in RPC/RDMA Version One.
   - Say that any incompatible change or change to the Version One XDR will
   cause a Version number change but that future versions (e.g., version Two)
   are free to define their own rules regarding the version number.

> > I do not

It hasn't been "deprecated" (in that sense) yet.  Actually, I have
deprecated
it (in the normal sense) a number of times.  my formulation was that it was
"moribund on arrival".   In fact, if I were the thriager, I would have been
tempted to let the thing die and start afresh.  I think Chuck's judgment,
to
try to save this protocol has been vindicated.

> 1. There are deployed systems using V1. A V2 would not

A random V2 would not, but I think we can design a V2 which is capable of
interoperating with existing V1 implementations and allowing
implementations
of the new version to incrementally incorporate extensions.

> adequate for that because of its ambiguities and a few

Those gaps were the basis of my "moribund on arrival" comment, although the
situation was worse than I knew given some things I now know about the
implementations.

> They are extensible.

I think we want to distinguish here between being extensible and being
capable
of interacting with other implementations that need to find out that they
have no
extensions.

> It would require some bug fixes

Making the servers actually extensible would require both of these but
making
existing ("old-school") servers capable of (unextended) interoperation
would
only require the latter.

> with this workaround is it is ugly.

Let me explain the workaround to those who haven't been following this in
detail.
I agree that it is ugly but I think that is the only appropriate response
to the ugly
situation we are in.

The workaround is to send an NFS NULL using a known-to-be-invalid message
header type.  If you get a response to the NFS NULL indicating that the
server
treated the invalid header type as RDMA_MSG, then it is an "old-school"
server
(i.e a buggy implementation of RFC5666 rather than a correct implementation
of rfc5666bis) and thus not extensible.  This is true even if the server
does not
return an error when it gets an incorrect version number.

Some possible reserved header types to use for testing

   - 4294967295
   - 3141592605
   - 666

---

From: William Allen Simpson <[email-masked]>
Date: Wed, 9 Mar 2016 18:59:39 -0500

On 3/8/16 6:32 AM, David Noveck wrote:
> It seems like you are saying that Chuck posted a -04, with the intention that
I've known Chuck a very long time.  I didn't understand why he'd
bother to include a section "for discussion" that he knows I've
ardently opposed.  I'm writing here to clarify that as far as I'm
concerned, this is only for the purpose of formally being rejected
and removed.

> working group list on 2/4.
"Detecting loss of connection with RPC-over-RDMA"

Pardon me that I cannot fathom how that discussion became fodder
for checking message header types -- and that I've missed the
wording therein somehow.

---

From: David Noveck <[email-masked]>
Date: Wed, 9 Mar 2016 21:13:28 -0500

> working group list on 2/4.

It was actually discussed on 2/2.  Sorry about that.  The subject line was
"re: [nfsv4] rfc5666bis-03.txt extensibility" and it was part of a
discussion in which you, Chuck, and I were involved.

---

From: William Allen Simpson <[email-masked]>
Date: Thu, 10 Mar 2016 04:34:26 -0500

tl;dr -- misunderstanding about how Standards Track works....

Let's talk next week.  If we can jettison all the
controversial language, and agree on immediate WGLC, ....

> mailing list on February 1, 2016, and are archived on
Searching my archives for pertinent words, I find:

[...]
   A set of such new protocol elements may be introduced by a Standards-trac=
k document and are together consiered an OPTIONAL feature.  Working group a=
nd IESG review, together with appropriate testing of prototype implementati=
ons, should ensure </span></font></font><span style=3D"font-size:12.8px;whi=
te-space:pre-wrap;font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">=
continued interoperation with </span><span style=3D"font-size:12.8px;white-=
space:pre-wrap;font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">exi=
sting implementations.</span></pre></blockquote><blockquote style=3D"margin=
:0px 0px 0px 40px;border:none;padding:0px"><pre style=3D"word-wrap:break-wo=
rd"><span style=3D"font-size:12.8px;white-space:pre-wrap;font-family:arial,=
helvetica,sans-serif;color:rgb(0,0,0)">For the sender to determine whether =
a receiver supports a particular extension, he can issue a simple test requ=
est using the specified header type:</span></pre></blockquote><blockquote s=
tyle=3D"font-size:12.8px;margin:0px 0px 0px 40px;border:none;padding:0px"><=
pre style=3D"white-space:pre-wrap;word-wrap:break-word"><ul><li><span style=
=3D"color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:12.8p=
x">In RPC-over-RDMA Version One, a receiver can indicate that it does not r=
ecognize an rdma_proc enum value only by returning an RDMA_ERROR procedure =
with the rdma_err field set to RDMA_ERR_BAD (see Section 5.5.2) and with an=
  xid field that matches the one sent.  Although this is indistinguishable f=
rom a situation where the receiver does indeed support the procedure, but t=
he XDR is malformed, this fact should pose no difficulty in practice.  If t=
he sender always tests for receiver support using a simple instance of the =
header type to be tested, such an error at this point inicates that the sen=
der and receiver have no prospect of using the protocol element interoperab=
ility and so a lack of feature support can be reasonably assumed.</span><br=
 ></li><li><span style=3D"color:rgb(0,0,0);font-family:arial,helvetica,sans-=
serif;font-size:12.8px">The extension specification needs to specify some b=
ehavior that the receiver can implement that the original sender can use to=
  determine that support is in fact present.  This may include sending a res=
ponse header, which may itself be an extension such that the original sende=
r by sending his test header, indicates that he is capable of accepting and=
  using the response header.  The original header and the response can be ti=
ed together using the xid field.</span></li></ul></pre></blockquote><blockq=
uote style=3D"font-size:12.8px;margin:0px 0px 0px 40px;border:none;padding:=
0px"><pre style=3D"white-space:pre-wrap;word-wrap:break-word"><font face=3D=
"arial, helvetica, sans-serif"><font color=3D"#000000">Documents describing=
  extension to Version One should contain in addition to an explanation of t=
he purpose and use of each protocol element:</font></font></pre></blockquot=
e></div></div><div><div><blockquote style=3D"font-size:12.8px;margin:0px 0p=
x 0px 40px;border:none;padding:0px"><pre style=3D"white-space:pre-wrap;word=
-wrap:break-word"><ul><li><span style=3D"color:rgb(0,0,0);font-family:arial=
,helvetica,sans-serif;font-size:12.8px">An XDR description and a script to =
[...]

There was also a .pdf, which was base64 encoded.

Pardon me, I didn't bother to parse such nonsense.  I've
instructed my Thunderbird software to never compose with html,
as do all experienced Internauts.

And I've selected the option to post inline, not top-post, as do
all persons with even the slightest regard for their readers.

I should thank you for the thankless task of trying to understand,
re-format, and edit it.  You are a better man than I.

> This is not the first time you have missed pertinent
My email sorting rules work just fine.  My internet draft
announcements have gone into the same folder since 1992.

I don't read them every day, or even every week.  Do you?

I do try to read RDMA related material in nfsv4 several times
per week.  But as you've found, the internet-drafts folks don't
always send their announcements to the nfsv4 list.  Until I've
seen your forward of the announcement (with details), I'm
usually unaware it has been issued.

> of RPCSEC_GSS for years, making it a current practice.
Have there been 2 working implementations?  If not, then it
could not be incorporated into a Draft/Final Standard.

We have a conflict of vision here.  It's appropriate to add
for a document that will be published as Historical.

> security specified in the same document that obsoletes
We can have multiple documents obsoleting one RFC, and
have plenty of examples of such splits.

See RFC 1134 -> 1171, 1172 -> 1331, 1332 -> 1548, ... ->
1661 et seq.

> 5666 does not enable RPCSEC_GSS-based security at all.
Correct.

> considering existing IETF politics around Internet
IESG is certainly a concern.

> prefer that the RPCSEC text be left in rfc5666bis.
I agree about depending on link-layer protection.

IETF PPP link-layer protection was opposed by Steve Kent
going back to 1990, and was never finished.

IEEE 802 link-layer protection has a terrible track record.

> I-D forward as an extension to V1 that does not change
It does have the advantage of being much shorter.

Personally, I thought it was OK to incorporate because
it seemed more informative than specification, and it has
multiple implementations.

> rather than as part of rfc5666bis makes sense.
Yes.  Instead, let's just document the few fields and
error codes that MUST be implemented for all versions.

That way is easier, simpler, and moves us forward.

> attempted to add XDR extensibility to V1.
I invite broader V2 changes.

> aesthetically displeasing, but it is workable.
As you know, I firmly disagree.  It requires changing the
semantics of an error return, among other bogosities.

Rather, there are several completely conformant implementations,
that are conformant only because the original RFC authors
failed to specify MUST and SHOULD.

As you know, I'm a strong advocate of more mandatory language.

And eschewing odd contortions and conventions.

> with this workaround is it is ugly.
The slow workaround is not without problems, causes potential
crashes -- and is definitely ugly..... :(

> imperative to create a V2 to enable detection of
Moved to next message.

> drops requests with unrecognized versions, and does not
Ganesha server has the same problem.  The code to send
the version error reply is present, but never called.

> server implementation should be fixed.
Verily, Ganesha will also be fixed.

> number is not a perfectly clean technical solution.
But the result is cleanly backward compatible.

> 1. There are deployed systems using V1. A V2 would not
It would interoperate to the extent that it could always fall
back to V1, because in V1 there exists an error code that
facilitates fallback (properly implemented).

> adequate for that because of its ambiguities and a few
Woah, Nelly!

We are not extending V1.  It is not extensible in the field.

V2 will be fully specified.

There seems to be a misunderstanding about how Standards Track
works....

V1 is currently Proposed Standard.  5666bis V1 would be Standard.

But 5666ter V2 would be Proposed Standard.  It would not be an
extension of a Standard.  It would recycle on Proposed.

Thus, it makes no sense to do V1 and V2 in parallel.  In the usual
case, we'd mark 5666 as Historical, and 5666bis V2 as Proposed.
We'd never bother finishing V1 only to abandon it.

There's only so much WG time and effort available.

> thus proposing is that rfc5666bis be scrapped and a new
Scrapped as in s/One/Two/g.

It would Obsolete 5666, not Update 5666.  That's the usual
process, done hundreds of times to date.

See RFC 1134 -> 1171, 1172 -> 1331, 1332 -> 1548, ... ->
1661 et seq.

> deciding what to keep and what to jettison before the
Let's talk next week.  If we can jettison all the
controversial language, and agree on immediate WGLC, ....

But all I've seen is weeks of quibbling about arcane
methods of extending V1.  I'd rather spend my time
working on polishing V2.

---

From: William Allen Simpson <[email-masked]>
Date: Thu, 10 Mar 2016 04:37:22 -0500

> imperative to create a V2 to enable detection of
Here's my compelling technical narrative.

* (A) Inline Data *

Version One is slow. It has Send buffers that are 1K. This was
entirely reasonable in a world where the usual page size was 1K,
and where the usual network segment was 1500 bytes.

It is not best practice in a world with 4K page sizes and up, with
jumbo gram data segments.

QED. We want to extend the protocol to negotiate more inline data.

* (B) Error Recovery *

Error recovery in Version One is poorly specified. Improved error
recovery is at least as important as extending the protocol.

Since we need at least one optional feature, we need an error code
indicating an optional feature is not supported.

Since there there may be more than one future optional feature, we
also MUST indicate _which_ optional feature is not supported.

This error code MUST be repeatable for multiple errors. We do not
want to try the connection over and over to detect and remove
individual optional features one-at-a-time.

Moreover, there is only one (1) existing such error code now named
RDMA_ERR_BADHEADER (nee RDMA_ERR_CHUNK). This code MUST place the
value 4 in the first word (RDMA_ERROR) and value 1 in the second
word (enum errcode RDMA_ERR_BADHEADER). Currently, there is no
third word.

Therefore, I propose adding a third word in Version Two (ignored
by Version One): rdma_position. Just as rdma_position is used
for read_segment, this indicates exactly where the error occurs.

This position will also be useful for debugging unsupported values
of optional features, by pointing at specific value fields.

This position will be far better than the current definition for
debugging chunk errors. The current definition does not indicate
where the actual error occurs.

Current Version One implementations terminate the connection. For
backward compatibility, this can be treated exactly the same as
removing optional features one-at-a-time. No changes to Version
One implementations are required.

* (C) Chunk ordering *

The protocol also has other problems that affect performance.

Write chunks are dependent upon the client, and MUST be used,
rather than intelligently determined by a smarter server that
knows where the data is actually long or short. The Linux
client persists in assigning long Write chunks of odd sizes,
when we only need to write a few bytes inline.

This can be fixed by making the Write chunk definitions the
same as the Read chunk definitions, so that Write chunks can be
associated with specific positions in the Reply.

This has the advantage of being much more logically consistent,
much easier to implement, and easier to document.

Likewise, the order of the chunks in the Call is Read, Write,
Reply. But in the Reply, the order is NULL, Write, Reply.

This requires copying the Write and Reply chunks.  Not a big
deal as it's not a lot of data, but also requires that the Call
and Reply have to be in different buffers.

Worse, the Call processing has to find the Reply (it needs to
know whether the optional Reply is present, to determine the
kind of buffer the Reply will be written into), then process
Read buffers, while updating Write buffers.

If the chunks were ordered Reply, Write, Read on both the
Call and Reply, the same buffer could be used. Values could be
updated in place (without copying). And in my implementation
this would reduce the pthread locking and allocations by half
on average, which is significant.

We shouldn't be stuck with an historical design that wasn't well
crafted for actual speedy implementation.

---

From: William Allen Simpson <[email-masked]>
Date: Thu, 10 Mar 2016 04:59:44 -0500

On 3/8/16 2:49 PM, David Noveck wrote:
> Let me explain the workaround to those who haven't been following this in detail.
> I agree that it is ugly but I think that is the only appropriate response to the ugly
> situation we are in.
>
> header type.  If you get a response to the NFS NULL indicating that the server
> treated the invalid header type as RDMA_MSG, then it is an "old-school" server
> of rfc5666bis) and thus not extensible.  This is true even if the server does not
>   * 4294967295
>   * 3141592605
>   * 666
For those of you who have not been following this in detail,
this is a bad idea.

Noveck's proposal, V1-only:

RDMA_V1, RDMA_666 followed by NFS NULL ->
<- OK (or crashes)
RDMA_V1, RDMA_MSG ->

Noveck's proposal, V2:

RDMA_V1, RDMA_666, followed by NFS NULL ->
<- terminate connection
RDMA_V2, RDMA_Optional_Proc ->
<- OK
RDMA_V2, RDMA_MSG ->

Simpson's proposal, V1-only:

RDMA_V2, RDMA_Optional_Proc, RDMA_MSG ->
<- RDMA_ERROR(version 1 supported)
RDMA_V1, RDMA_MSG ->

Simpson's proposal, V2:

RDMA_V2, RDMA_Optional_Proc, RDMA_MSG ->
<- OK

Note in both cases, my proposal is much cleaner, and doesn't
involve any potential crashes.

Moreover, in the case we actually want V2, it just works!
No extra round trips of probing.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 10 Mar 2016 11:12:15 -0500

> On Mar 10, 2016, at 4:37 AM, William Allen Simpson <[email-masked]> wrote:
> 

No one is arguing we don't want larger inline thresholds.
The question is how to get there. I personally think it was
valuable to consider V1 extension, and to investigate how
the current implementations might support it. We have
determined that there would be significant issues, and are
now turning to other approaches.

And note that ULPs need to work no matter what inline
threshold is in effect. Increasing the threshold is an
optimization, not an issue for fixing interoperation, which
is what rfc5666bis is for.

Lastly, today Solaris and Linux NFS/RDMA run plenty fast
with 1KB receive buffers, faster than NFS on IPoIB (faster
than the bear).

> where the actual error occurs.

What, pray tell, would the receiving end do with the XDR
offset of the problem? The point of adding such a field
to the protocol is that it would provide additional
recourse for error recovery in real time. I can't imagine
how a client might use this information.

If better debugging visibility is what you want, your
implementation is free to display the problem offset
in your log. Or better yet, it could report a human
readable error message that describes the issue.

> when we only need to write a few bytes inline.

I'm not sure if you mean "small READs" or "short READs". A
short NFS READ has always been defined as when the server
returns less data than the client requested. Short READs
are exceptionally rare. Handling them is not considered
performance critical. This is because the client almost
always has an idea how much data needs to be read before
it issues an NFS READ operation.

So I'm guessing you mean a small READ. Let's define a
"small" READ as one where the reply can fit inline.

There isn't much difference between returning the reply
with an RDMA Write followed by an RDMA Send, and returning
the reply with an RDMA Send only.

Servers don't have to wait for the RDMA Write to complete
before they emit the RDMA Send. Both can be chained and
posted together, in fact.

Because the HCA will send these operations one after another
without waiting, the difference in wire latency is just a
microsecond or two. This is unlike how V1 uses RDMA Read,
where there is indeed an additional round trip.

Compared to the amount of time it takes for the client to
build an NFS READ request, register a Write chunk that
the server would ignore, the server to process the READ
request, the client to handle the reply, and the client to
invalidate the unused Write chunk, a microsecond or two is
small potatoes.

The Linux client always uses a Write list because the Linux
server had a bug that resulted in data corruption when a
READ reply was returned inline. Both inline and using a Write
list are valid protocol, and servers MUST support when a
client uses either approach.

It's certainly possible to change the Linux client to avoid
using a Write list for small READs, and that seems to be the
root of your complaint.

In other words, I don't believe this is a significant
performance issue, and it can be solved by an implementation
change.

> much easier to implement, and easier to document.

I think we'd need to see a prototype implementation before
deciding whether it was easier to implement or even useful.

> and Reply have to be in different buffers.

With modern HCAs on both peers, the RPC-over-RDMA header
is not ever going to be large. Copying the header from the
receive buffer to the send buffer starts with a memcpy of
just a handful of bytes, then the server updates the three
or four words that have to change.

I'm not sure how the protocol "requires the Call and Reply
to be in different buffers". That seems like an
implementation choice to me.

> Read buffers, while updating Write buffers.

The header was parsed when the RPC Call was received. Save
the location of the Write list and Reply chunk.

> on average, which is significant.

Sure, but now you've penalized Call header parsing to find
the Read list. It has to walk past the Reply and Write lists.
This is trading six for a half-dozen.

> crafted for actual speedy implementation.

It sounds to me as if you are asking the rest of us to change
our implementations to fix a locking problem on your server.
What would be our incentive to support such a protocol change?

--
Chuck Lever

---

From: Matt Benjamin <[email-masked]>
Date: Thu, 10 Mar 2016 16:39:13 -0500 (EST)

Hi,

inline

----- Original Message -----
> From: "Chuck Lever" <[email-masked]>
> To: "William Allen Simpson" <[email-masked]>
> Cc: "NFSv4" <[email-masked]>
> Sent: Thursday, March 10, 2016 11:12:15 AM
> Subject: Re: [nfsv4] rfc5666ter features
> 
<snips>
> with 1KB receive buffers, faster than NFS on IPoIB (faster

That's gratifying, but I think that the actual success criteria should be to permit implementations that reach the available performance of the RDMA transport.

> performance issue, and it can be solved by an implementation

As above.

Regards,

Matt

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

---

From: Chuck Lever <[email-masked]>
Date: Thu, 10 Mar 2016 17:57:41 -0500

> On Mar 10, 2016, at 4:39 PM, Matt Benjamin <[email-masked]> wrote:
> 
> Hi,
> 
> inline
> 

Here's how I'd like to do that:

The current proposal is to introduce an RPC-over-RDMA
V2 that has one feature: XDR extension; and then write
a simple extension that adds inline threshold
negotiation, and another that adds remote invalidation
negotiation, and so on.

If you want larger inline thresholds immediately, the
default V2 inline threshold can be increased to 4KB or
some other arbitrary number. Negotiation is icing on
that cake.

XDR extension is preferred here because the protocol
specification process is very lengthy. It takes a long
time because of feature batching. All features in a new
protocol version have to wait for the one or two that is
controversial or hard to prototype. (NFSv4.2 and S2SC
is a very fresh example of this phenomena).

As soon as RPC-over-RDMA V2 is open to arbitrary changes,
that makes the specification process take much longer.

XDR extension is a way of dividing this process up and
making it parallel. Each extension can be specified
separately and does not have to wait for others.

>> performance issue, and it can be solved by an implementation

Bill's premise is that a server may want to return
small READ replies inline when a client has provided
a Write chunk, because the server "knows better".

It's a false premise: An NFS client almost always
knows how much data the server is about to return
in a READ, and can prepare the correct chunk
resources for an optimal reply.

If the client is providing the wrong resources for
READ replies, then the fix is to adjust the client's
behavior.

There's nothing to fix in the protocol!

--
Chuck Lever