Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5666bis-01.txt

From: [email-masked]
Date: Mon, 14 Dec 2015 12:40:04 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Network File System Version 4 Working Group of the IETF.

        Title           : Remote Direct Memory Access Transport for Remote Procedure Call
        Authors         : Charles Lever
                          William Allen Simpson
                          Tom Talpey
	Filename        : draft-ietf-nfsv4-rfc5666bis-01.txt
	Pages           : 51
	Date            : 2015-12-14

Abstract:
   This document specifies a protocol for conveying Remote Procedure
   Call (RPC) messages on physical transports capable of Remote Direct
   Memory Access (RDMA).  The RDMA transport binding enables efficient
   bulk-data transport over high-speed networks with minimal change to
   RPC applications.  It requires no revision to application RPC
   protocols or the RPC protocol itself.  This document obsoletes RFC
   5666.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5666bis/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5666bis-01

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-rfc5666bis-01

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Chuck Lever <[email-masked]>
Date: Mon, 14 Dec 2015 16:05:03 -0500

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5666bis-01.txt
> Date: December 14, 2015 at 3:40:04 PM EST
> To: <[email-masked]>
> Cc: [email-masked]
> 
> 

This revision addresses many of the comments from David
Noveck last week, and revises several other sections. I
have also added Bill Simpson as a co-author.

One comment remains unresolved.

I am sympathetic to Dave's concerns about the use of the
term "deprecated." Most of the instances of the term have
been purged from the portion of the document Dave has
already commented on, as he recommended.

However, it seems to me that deprecation is exactly what
we are doing to RDMA_MSGP.

I think there are other protocol elements in a similar
state of ill-definition that we happen to be keeping, so
I did not retain that part of Dave's suggested replacement
text.

Current receivers do recognize RDMA_MSGP. We are telling
senders not to use it. That seems to fit Dave's definition
of deprecation as describing a protocol element that
remains a recognized part of the protocol but is in
transition out of use.

AIUI Dave's larger concern is that we are removing a
protocol element that could be extended at some future
time to improve performance of RPC-over-RDMA Version One.
I am hesitant to expand the role of this document to more
than simply recording the state of current implementations
of RPC-over-RDMA Version One, as that will add time needed
to finish this effort. Bill has privately suggested changes
to RDMA_MSGP that might be difficult to express in Version
One without XDR changes, which make RDMA_MSGP improvements
interesting to consider for a Version Two of the protocol.

I consider this item still open for active discussion,
and welcome your thoughts.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 14 Dec 2015 21:40:04 -0500

BTW, thanks for the quick turnaround.

> state of ill-definition

Really?  I'm not sure what protocol elements you are referring to.

Note that RDMA_MSGP is not just confusingly defined.  It is
essentially not defined
at all.  The use of  two fields  in the header that are added for RDMA_MSGP
is not defined at all.

> that we happen to be keeping, so

Not sure what you are referring to

> remains a recognized part of the protocol but is in

Close but you have to consider what happened here:

You have a protocol element that was in rfc5666.

But it was not implemented as defined and couldn't be implemented as
defined because it was never defined.

So people implemented it as doing something else, which had been defined.

So we have decided to maintain the existing non-implementation for any
receivers and tell people not to send it.

I don't think there is a single English word that describes that
situation.  Perhaps there's a very long German one, but if there is, I
don't know it.

> time to improve performance of RPC-over-RDMA Version One.

That's not it.  Although I wish RDMA_MSGP had been more successful, I don't
believe it can be resurrected and certainly not in Version One.  I think
the most that is possible is a kind of reincarnation, with the same idea
embodied in a replacement feature.

The thing I object to is the following text:

Implementers of RPC-over-RDMA Version One have neglected to make use of the
RDMA_MSGP message type. Therefore RDMA_MSGP is deprecated.

Although it says RDMA_MSGP "is deprecated", it appears to deprecate the
presumably neglectful implementers who simply didn't implement what never
made sense in rfc5666.

Also, the fact that it never made sense is the reason for
removal/deprecation rather than non-implementation.

Similarly, those people who didn't implement the stringprep nonsense in
rfc3530 weren't neglectful.  Instead, they were sensible and we decided to
remove the stringprep stuff and base rfc7530 on what those implementers did.

> to finish this effort.

Understood, but, even if we don;t discuss specific additional features, we
need an extensibility strategy and reading section 9, I'm not so clear what
it is.

> Bill has privately suggested changes
> interesting to consider

Interesting but outside the scope of rfc5666bis presumably.

> for a Version Two of the protocol.

My reading of section 9.1 (possibly incorrect) suggests that optional
features could be implemented in an optional extensions to Version One.
Nowhere does it say that an compatible extension requires the version
number to be changed.

I think that any real implementation of this would wait until Version Two
but the reason is not the need for an XDR change.  The reason is that
Version One seems wedded to a 1K message size.

On Mon, Dec 14, 2015 at 4:05 PM, Chuck Lever <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 15 Dec 2015 09:32:30 +0100

> Version One seems wedded to a 1K message size.

How would we add compatible extensions to the protocol?  Adding support
for remote invalidations seems to be very important for performance
when not taking shortcuts like allphys registrations, and it might make
be a good idea to get started on that ASAP in parallel to 5666bis.

---

From: David Noveck <[email-masked]>
Date: Tue, 15 Dec 2015 07:02:27 -0500

> How would we add compatible extensions to the protocol?

Note that this answer is based on my reading of section 9.1 in the -00 and
a quick glance at the -01.  I could be wrong, and Section 9 in rfc5666bis
could change in future revisions.

Right now, there is no prohibition of XDR changes,in the document, despite
some earlier discussion on the mailing list about  there being a
"consensus" about XDR changes not being allowed in Version One (or perhaps
it was in rfc5666bis).

Right now, the prohibition is only on changes that would cause an
implementation to be unable to interoperate with
existing Version One implementations.

The document states:

The value of the RPC-over-RDMA header's version field MUST be changed

   o  Whenever the on-the-wire format of the RPC-over-RDMA header is
      changed in a way that prevents interoperability with current
      implementations

Given that, one would define a new MSGP_SETPARM which allowed the parties
to request such things
as use of remote invalidation, and in the response indicate that the
request was granted.  An existing receiver implementation that did not
support this extension would report this as an XDR error and the
sender could conclude that the request was effectively denied and
interoperate without any further
attempt to use the extension.

Currently, Section 9.1 of -01 has the following text that might be an
impediment to dealing with the remote invalidation issue in  Version One.
The document states:

The value of the RPC-over-RDMA header's version field MUST be changed

...

   o  Whenever the set of abstract RDMA operations that may be used is
      changed

My review of section 9 will suggest that this be changed but I don't think
it is worth delaying rfc5666bis by having a big argument about this.  The
distinction between a Version One extension and a modest-sized Version Two
is small, as long as people  make efforts to interoperate with existing
Version One implementations.  I think one could write an I-D that
side-stepped the version number issue and allowed experiments with
extensions to be performed.  Such a document would be a good candidate for
working group document status after WGLC for rfc5666bis concluded.   At
that point one could decide the version number issue based on rfc5666bis as
it is then.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 15 Dec 2015 10:50:36 -0500

> On Dec 14, 2015, at 9:40 PM, David Noveck <[email-masked]> wrote:
> 
> Note that RDMA_MSGP is not just confusingly defined.  It is essentially not defined at all.  The use of  two fields  in the header that are added for RDMA_MSGP is not defined at all.
> 
> But it was not implemented as defined and couldn't be implemented as defined because it was never defined.
> 
> So we have decided to maintain the existing non-implementation for any receivers and tell people not to send it.
> 
> I don't think there is a single English word that describes that situation.  Perhaps there's a very long German one, but if there is, I don't know it.

Deprecation seems close enough to me. It means
"don't use it." Is there a strong reason to
additionally carry the baggage of the history
of the thing?

> That's not it.  Although I wish RDMA_MSGP had been more successful, I don't believe it can be resurrected and certainly not in Version One.  I think the most that is possible is a kind of reincarnation, with the same idea embodied in a replacement feature.
> 
> Implementers of RPC-over-RDMA Version One have neglected to make use of the RDMA_MSGP message type. Therefore RDMA_MSGP is deprecated.
> 
> 
> Although it says RDMA_MSGP "is deprecated", it appears to deprecate the presumably neglectful implementers who simply didn't implement what never made sense in rfc5666.
> 
> Also, the fact that it never made sense is the reason for removal/deprecation rather than non-implementation.
>  
> Similarly, those people who didn't implement the stringprep nonsense in rfc3530 weren't neglectful.  Instead, they were sensible and we decided to remove the stringprep stuff and base rfc7530 on what those implementers did.

I'll revisit that text in rfc5666bis and see if
I can craft something better.

> Understood, but, even if we don;t discuss specific additional features, we need an extensibility strategy and reading section 9, I'm not so clear what it is.
> 
> My reading of section 9.1 (possibly incorrect) suggests that optional features could be implemented in an optional extensions to Version One. Nowhere does it say that an compatible extension requires the version number to be changed.

Optional features could be implemented, yes. I'm
suggesting here that the changes needed to make
an RDMA_MSGP-like protocol element useful would
very likely result in some incompatible XDR
changes. And agreed, the inline threshold will
need more flexibility than is practical in
Version One.

But above you suggest a resurrection in Version
Two of RDMA_MSGP. That agrees with my preferred
strategy.

> I think that any real implementation of this would wait until Version Two but the reason is not the need for an XDR change.  The reason is that Version One seems wedded to a 1K message size.

The default inline threshold is 1KB, and it may be
increased for new connections by some unspecified
mechanism. This provides ample room to experiment,
and does not limit the use of future protocol
extensions to make this happen.

There is no mechanism in Version One for exposing
a peer's inline threshold to a remote. Maybe that's
a mistake in protocol design, but that's what we
have today.

Version One implementations are not prepared to
change the threshold size while a connection is
active. I'm not even sure if that's possible,
given the way RDMA Receive works.

If we want to change the inline threshold behavior
to be useful by allowing negotiation of inline
threshold per connection, we will need prototyping
and protocol design that I feel is not in the
scope of rfc5666bis.

The simplest way to make something like this work
is for both sides to advertise their inline
threshold values when a connection is established.
Both sides then take the smaller value of the two.

--
Chuck Lever

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 15 Dec 2015 16:57:47 +0100

> Both sides then take the smaller value of the two.

I think the easiest way to specify either an inline limit and
remote invalidation support is to support private data during
the RDMA/CM connect message exchange.

This can be trivially done as a separate RFC not impacting the main
protocol.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 15 Dec 2015 11:27:04 -0500

> On Dec 15, 2015, at 10:57 AM, Christoph Hellwig <[email-masked]> wrote:
> 

It depends on whether existing implementations behave
consistently when confronted with an unrecognized
protocol element.

Dave assumed they would return an XDR error. That may
not always be the case.

Writing an I-D is the easy part, but figuring out how
to make this interoperate with existing implementations
could be non-trivial. We also need to craft rfc5666bis
so that it doesn't limit our ability to create such
extensions.

--
Chuck Lever

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 15 Dec 2015 17:42:04 +0100

> so that it doesn't limit our ability to create such

The beauty of the RDMA/CM private data is that it's outside
the normal XDR scope.  An RDMA-extensions client would set the
newly defined private data to ask for a larger inline size or
remote invalidations.  Now there are three basic ways the server
can react to that:

 (a) it ignores the private data and will not send private
     data in the reply.  The client knows the server doesn't
     understand the extensions and can proceed (That's for
     example what the existing Linux server would do)
 (b) the server explicitly checks for private data and rejects
     a connection attempt.  The client would have to re-try
     the connection without private data
 (b) the server understand the private data and returns values
     that allow negotiating the supported feature set

---

From: David Noveck <[email-masked]>
Date: Tue, 15 Dec 2015 11:59:40 -0500

Yes, but Chuck is right to point out that what rfc5666bis says on the
matter is not irrelevant.  For example, language regarding what an endpoint
may validly assume as to the other side's send/receive buffer size must not
be written in a fashion that interferes with implementations' ability to
use the information obtained from the private data exchange.

---

From: David Noveck <[email-masked]>
Date: Tue, 15 Dec 2015 12:08:13 -0500

I prefer the rfc2119 terms to say "don't use it" but there is no point in
continuing the argument.

I think we can avoid the history, but I object to giving an erroneous
version of history as when the implementers are alleged to have "neglected"
to implement RDMA_MSGP.
On Dec 15, 2015 10:50 AM, "Chuck Lever" <[email-masked]> wrote:

> > Note that RDMA_MSGP is not just confusingly defined.  It is essentially
> not defined at all.  The use of  two fields  in the header that are added
> for RDMA_MSGP is not defined at all.
> >
> > That's not it.  Although I wish RDMA_MSGP had been more successful, I
> don't believe it can be resurrected and certainly not in Version One.  I
> think the most that is possible is a kind of reincarnation, with the same
> idea embodied in a replacement feature.
> >
> > Implementers of RPC-over-RDMA Version One have neglected to make use of
> the RDMA_MSGP message type. Therefore RDMA_MSGP is deprecated.
> >
> >
> > Understood, but, even if we don;t discuss specific additional features,
> we need an extensibility strategy and reading section 9, I'm not so clear
> what it is.
> >
> > I think that any real implementation of this would wait until Version
> Two but the reason is not the need for an XDR change.  The reason is that

---

From: Chuck Lever <[email-masked]>
Date: Tue, 15 Dec 2015 12:35:31 -0500

> On Dec 15, 2015, at 12:08 PM, David Noveck <[email-masked]> wrote:
> 
> I prefer the rfc2119 terms to say "don't use it" but there is no point in continuing the argument.
> 
> I think we can avoid the history, but I object to giving an erroneous version of history as when the implementers are alleged to have "neglected" to implement RDMA_MSGP.

AFAICT that language was removed in -01, which now says:

   o  Support for sending the RDMA_MSGP message type has been
      deprecated.  This document instructs senders not to use it, but
      receivers must continue to recognize it.

      RDMA_MSGP has no benefit for RPC programs that place bulk payload
      items at positions other than at the end of their argument or
      result lists, as is common with NFSv4 COMPOUND RPCs [RFC7530].
      Similarly it is not beneficial when a connection's inline
      threshold is significantly smaller than the system page size, as
      is typical for RPC-over-RDMA Version One implementations.

How about reducing this to:

The RDMA_MSGP message type is no longer supported. Receivers
must continue to recognize it, but senders should not use it.
The specification of RDMA_MSGP in [RFC5666] and current
implementations of it are incomplete.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 15 Dec 2015 12:43:53 -0500

I'm OK with either formulation.
On Dec 15, 2015 12:35 PM, "Chuck Lever" <[email-masked]> wrote:

> essentially not defined at all.  The use of  two fields  in the header that
> are added for RDMA_MSGP is not defined at all.
> > >
> > > Implementers of RPC-over-RDMA Version One have neglected to make use
> of the RDMA_MSGP message type. Therefore RDMA_MSGP is deprecated.
> > >
> > >
> > > Although it says RDMA_MSGP "is deprecated", it appears to deprecate
> the presumably neglectful implementers who simply didn't implement what
> never made sense in rfc5666.
> > >
> > > Similarly, those people who didn't implement the stringprep nonsense
> in rfc3530 weren't neglectful.  Instead, they were sensible and we decided
> to remove the stringprep stuff and base rfc7530 on what those implementers
> did.
> >
> > > Understood, but, even if we don;t discuss specific additional
> features, we need an extensibility strategy and reading section 9, I'm not
> so clear what it is.
> > >

---

From: Chuck Lever <[email-masked]>
Date: Tue, 15 Dec 2015 14:47:12 -0500

> On Dec 15, 2015, at 11:59 AM, David Noveck <[email-masked]> wrote:
> 
> Yes, but Chuck is right to point out that what rfc5666bis says on the matter is not irrelevant.  For example, language regarding what an endpoint may validly assume as to the other side's send/receive buffer size must not be written in a fashion that interferes with implementations' ability to use the information obtained from the private data exchange.

To accommodate a private data exchange as Christoph
describes, I've updated two paragraphs in Section
4.3.1, "Initial Connection State".

The bullet that defines Inline Threshold now states:

"A connection's "inline threshold" value is the largest
message size (in bytes) that can be exchanged via an
RDMA Send/Receive combination on that connection.
The receiver pre-posts receive buffers of at least this
size."

The second paragraph in that bullet is unchanged.

The final paragraph of Section 4.3.1 now reads:

"Implementations MUST support an inline threshold of 1024
bytes, but MAY support larger inline thresholds values.
A mechanism for discovering a peer's inline threshold value
before a connection is established may be used to optimize
the connection's inline threshold. In the absense of such a
mechanism, senders MUST assume a receiver's inline threshold
is 1024 bytes."

This language distinguishes the inline threshold from
the actual buffer size. The two sides would advertise their
maximum threshold values (eg, the actual size of their
receive buffers) and the smaller of the two values would
be used as the inline threshold for that connection.

Before a connection is made, the ULP can request that
larger buffers be used on its end of the connection. After
a connection is made, it's difficult to make changes to
receive buffer size without disconnecting. Changing the
inline threshold before messages are exchanged should be
straightforward as long as a buffer size change is not
necessary.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 15 Dec 2015 15:08:18 -0500

> Dave assumed they would return an XDR error. That may not always be the
case.

I certainly hope it is.

Here's what draft-ietf-nfsv4-rfc5666bis-01 says about the matter:

   When a peer receives an RPC-over-RDMA message, it MUST perform basic
   validity checks on the header and chunk contents.  If such errors are
   detected in the request, an RDMA_ERROR reply MUST be generated.

   Two types of errors are defined, version mismatch and invalid chunk
   format.

   o  When a responder detects an RPC-over-RDMA header version that it
      does not support (currently this document defines only Version
      One), it replies with an error code of ERR_VERS, and provides the
      low and high inclusive version numbers it does, in fact, support.
      The version number in this reply MUST be any value otherwise valid
      at the receiver.

   o  When a responder detects other decoding errors in the header or
      chunks, one of the following errors MUST be returned: either an
      RPC decode error such as RPC_GARBAGEARGS, or the RPC-over-RDMA
      error code ERR_CHUNK.

I'm guessing that there are no implementations that return ERR_CHUNK if the
header opcode is
unrecognized, and that rfc5666bs-01 is correct about current implementation
behavior.

If we find implementations that behave differently, the specification can
adjust,  As long as there are
no implementations which ignore the error, drop the connection, or launch
weapons of mass destruction, one can implement an optional feature along
these lines.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 15 Dec 2015 15:42:59 -0500

> On Dec 15, 2015, at 3:08 PM, David Noveck <[email-masked]> wrote:
> 
> > Dave assumed they would return an XDR error. That may not always be the case.
> 
> I certainly hope it is.
> 

This text isn't exactly consistent with the "RDMA_ERROR
reply MUST be generated." above.

I copied a lot of this from RFC 5666. It does have to be
verified against the current implementations. I think
RPC_GARBAGEARGS is the way implementations do it these
days, which is problematic (in general). Clients have
their own unique ways of dealing with GARBAGEARGS.

But there should be just a single way for the responder
to report a problem decoding an RPC-over-RDMA header.
There should be an ERR_HEADER enum to make this work
well.

> I'm guessing that there are no implementations that return ERR_CHUNK if the header opcode is
> unrecognized, and that rfc5666bs-01 is correct about current implementation behavior.
> 
> If we find implementations that behave differently, the specification can adjust,  As long as there are 
> no implementations which ignore the error, drop the connection, or launch weapons of mass destruction, one can implement an optional feature along these lines.  
> 
> On Tue, Dec 15, 2015 at 11:27 AM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 15 Dec 2015 17:06:23 -0500

> reply MUST be generated." above.

The only way an implementation could be consistent with all the statements
in this section is to return ERR_VERS or ERR_CHUNK on all header XDR decode
errors, which doesn't make a whole lot of sense,

> verified against the current implementations.

Or at least all the ones you can find and test against :-(

> I think

I don't think that is problematic in practice in this case, since the RPC
client will never see this error.  Although the layering is messed up, the
transport layer can figure out that a particular opcode is not supported by
the peer.

> to report a problem decoding an RPC-over-RDMA header.

Other than in the case in which there is a version mismatch, yes.

> There should be an ERR_HEADER enum to make this work

That's the right way to deal with this.

Unfortunately our hands are tied by some bis-related constraints

   - By the existing implementations.
   - By the fact that adding ERR_HEADER is an XDR change.

So Version One, as specified in rfc5666bis, will not work well, but I think
it will work OK.

Trying to make this change in a Version Two does seem possible, but if
Version Two implementations are to interoperate with Version One
implementations that use optional features, Version Two implementations
will still have to return  GARBAGEARGS if there is a decode error in a
Version One header.  Sigh!.

On Tue, Dec 15, 2015 at 3:42 PM, Chuck Lever <[email-masked]> wrote:

> > no implementations which ignore the error, drop the connection, or
> launch weapons of mass destruction, one can implement an optional feature
> along these lines.
> >

---

From: David Noveck <[email-masked]>
Date: Thu, 17 Dec 2015 06:08:48 -0500

I'm replying to myself because I was wrong.  This is not evidence of
Multiple Personality Syndrome, but, then again, neither of the David
Novecks involved in this discussion is licensed to practice medicine :-)

> Unfortunately our hands are tied by some bis-related constraints

But not all that tightly.  See below.

>      o By the existing implementations.

We have to accommodate existing implementations by allowing them to return
GARBAGEARGS, but I think it is valid to say that they SHOULD NOT do this
and that responders MUST accept this as a potential indication that the
header is not XDR-decodable.

Even apart from the internal contradiction. the fact that you can't
distinguish XDR-decode problems in the header from those in the RPC message
is a clear technical error in RFC5666, even in the absence of an errata
being filed.  Fixing stuff like that is a valid thing to do in a bis
document.

>      o By the fact that adding ERR_HEADER is an XDR change.

This may be an issue  sometime since there are people who look askance at
XDR changes without looking at the substance of the issue.  Nevertheless,,
given that what you are doing is doing is adding an error code, I suggest
making the change.  The worst that can happen is that you have to undo it.

> So Version One, as specified in rfc5666bis, will not work well, but I
think it will work OK.

"barely OK" but worth fixing.  The only case in which it works is the one
we were originally discussing, in which you are testing a header to see if
the responder accepts it, knowing that there is no RPC call to get an XDR
decode error on.  But, for general RPC use, it kind of sucks.

On Tue, Dec 15, 2015 at 5:06 PM, David Noveck <[email-masked]> wrote:

> So Version One, as specified in rfc5666bis, will not work well, but I
> think it will work OK.
>

---

From: Chuck Lever <[email-masked]>
Date: Thu, 17 Dec 2015 11:45:43 -0500

> On Dec 17, 2015, at 6:08 AM, David Noveck <[email-masked]> wrote:
> 
> I'm replying to myself because I was wrong.  This is not evidence of Multiple Personality Syndrome, but, then again, neither of the David Novecks involved in this discussion is licensed to practice medicine :-)
> 
> Anyway, I wrote:
> 
> We have to accommodate existing implementations by allowing them to return GARBAGEARGS, but I think it is valid to say that they SHOULD NOT do this and that responders MUST accept this as a potential indication that the header is not XDR-decodable.
> 
> Even apart from the internal contradiction. the fact that you can't distinguish XDR-decode problems in the header from those in the RPC message is a clear technical error in RFC5666, even in the absence of an errata being filed.  Fixing stuff like that is a valid thing to do in a bis document.
> 
> This may be an issue  sometime since there are people who look askance at XDR changes without looking at the substance of the issue.  Nevertheless,, given that what you are doing is doing is adding an error code, I suggest making the change.  The worst that can happen is that you have to undo it.
> 
> > So Version One, as specified in rfc5666bis, will not work well, but I think it will work OK.
> 
> "barely OK" but worth fixing.  The only case in which it works is the one we were originally discussing, in which you are testing a header to see if the responder accepts it, knowing that there is no RPC call to get an XDR decode error on.  But, for general RPC use, it kind of sucks.

I tested both the Linux NFS/RDMA server and a Solaris 11.3
server. Both of them treat an out-of-range rdma_proc value
like RDMA_MSG. No error is returned, and the RPC message
is processed normally.

Now the question is, which is right: RFC 5666, or the
implementations, which match each other, but do not comply
with RFC 5666?

If this were any other XDR implementation, we'd say that it
was broken to not return an error. Clearly an out-of-range
enum value is an error. Thus the implementations should be
fixed to return ERR_CHUNK.

rfc5666bis should take an approach like you outline below,
but I think the discussion can be expanded a little. There
seem to be several categories of error when parsing a Call
message:

  - The header XDR has a syntax error. Eg, the XDR
    itself is invalid. Responder MUST return ERR_CHUNK.
    The requester should terminate the RPC if it
    recognizes the XID.

  - The header XDR has a semantic error. Eg, RDMA_NOMSG
    is used but there are no chunk lists. Responder MUST
    return ERR_CHUNK. The requester should terminate
    the RPC if it recognizes the XID.

  - The XID fields don't match. Which one is correct,
    and should be used to send the reply? The best that
    can be done is the responder should use the
    RPC-over-RDMA header XID and return ERR_CHUNK. No
    XID checking is done today, AFAICT.

  - The version field is not 1. Responder MUST return
    ERR_VERS. The requester should terminate the RPC if
    it recognizes the XID (note: rdma_vers != 1 could
    mean the Call message is screwed up somehow).

  - The values in the chunk lists contain incorrect or
    invalid values. This typically results in a Remote
    Access error, which is fatal to the connection. But
    how does the responder return an error after the
    connection is gone? This leads to an upper layer
    retransmit loop.

  - The upper layer can't parse the arguments. This might
    include mismatches between the chunk sizes or offsets
    and the contents of the RPC XDR stream. Responder MUST
    return RPC_GARBAGEARGS. The requester should terminate
    the RPC.

I believe this is close to what is implemented, though
implementations likely do not check some of the above
cases today.

A similar but simpler analysis applies for Reply messages,
where the requester cannot "reply" to the responder that
it sent a bogus Reply message.

Now, as for adding ERR_HEADER: the above bullet points
suggest there is little choice for the requester but to
terminate the RPC whose XID matches the reply, no matter
what the problem was. After thinking through it more, I'm
not sure there's much value in having the responder make
the distinction, and perhaps one error return value is
all that is necessary.

ERR_CHUNK may be somewhat inappropriately named, but it
does what we need it to.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 17 Dec 2015 12:11:38 -0500

The only change I'd make is to put the ERR_VERS section first to indicate
that if the version is not correct, that error should be returned,
irrespective of whether you (also) have an XDR decode error.

> ERR_CHUNK may be somewhat inappropriately named, but it does what we need
it to.

Yes. :-(

On Thu, Dec 17, 2015 at 11:45 AM, Chuck Lever <[email-masked]>
wrote:

> > We have to accommodate existing implementations by allowing them to
> return GARBAGEARGS, but I think it is valid to say that they SHOULD NOT do
> this and that responders MUST accept this as a potential indication that
> the header is not XDR-decodable.
> >
> > This may be an issue  sometime since there are people who look askance
> at XDR changes without looking at the substance of the issue.
> Nevertheless,, given that what you are doing is doing is adding an error
> code, I suggest making the change.  The worst that can happen is that you
> have to undo it.
> >
> > "barely OK" but worth fixing.  The only case in which it works is the
> one we were originally discussing, in which you are testing a header to see
> if the responder accepts it, knowing that there is no RPC call to get an
> XDR decode error on.  But, for general RPC use, it kind of sucks.
>
>

---

From: David Noveck <[email-masked]>
Date: Thu, 17 Dec 2015 12:38:13 -0500

> like RDMA_MSG. No error is returned, and the RPC message

It occurs to me that perhaps this means that our conclusion about receivers
(sort of) supporting RDMA_MSGP was wrong.

Were there any cases of implementations* sending *RDMA_MSGP?

---

From: Chuck Lever <[email-masked]>
Date: Thu, 17 Dec 2015 14:02:31 -0500

> On Dec 17, 2015, at 12:38 PM, David Noveck <[email-masked]> wrote:
> 
> It occurs to me that perhaps this means that our conclusion about receivers (sort of) supporting RDMA_MSGP was wrong.

You can look at the Linux server code and see that there
is support to parse the header and retrieve the alignment
information, but the information is not used.

But I'm not sure I understand. RDMA_MSGP is a _valid_
rdma_proc value. I was talking about _invalid_ values
below.

> Were there any cases of implementations sending RDMA_MSGP?

The Linux client used to have logic to send RDMA_MSGP,
but it was left disabled for a long while. I removed
it because it no longer worked and didn't seem worth
fixing for the reasons I've given elsewhere.

> On Thu, Dec 17, 2015 at 11:45 AM, Chuck Lever <[email-masked]> wrote:
> 
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 17 Dec 2015 14:12:55 -0500

> But I'm not sure I understand.

I'm absolutely sure I don't understand.

> RDMA_MSGP is a _valid_ rdma_proc value. I was talking about _invalid_
values below.

Before you told me, I had no way of knowing that the Linux server's view of
what was a valid rdma_proc value was any more reliable that it's handling
of XDR decode errors in the header.  So, being the cynical person I am, I
just assumed it might have gotten that wrong too.  Thanks for the info.

On Thu, Dec 17, 2015 at 2:02 PM, Chuck Lever <[email-masked]> wrote:

> > It occurs to me that perhaps this means that our conclusion about
> receivers (sort of) supporting RDMA_MSGP was wrong.
>