Subject: Re: [nfsv4] rfc5666bis section 7

From: David Noveck <[email-masked]>
Date: Wed, 23 Mar 2016 13:44:18 -0400

> The original idea behind rpcrdma was that it was a platform for any rpc
> protocol.

It was but we have to face the fact that has never been used for any non-NFS
protocol, and that it has issues, which we will address in documents to be
written,
in it being used used for NFSv4.1 and NFSv4.2.

> Writing all these MUSTs based on the NFS/RDMA choices
> is a significant limitation of that goal.

Yes but right now those are the only choices that exist.  there is no ULB
for
any other protocol and I think we are safe in assuming that we will publish
the
Version Two RFC, if not in July, then before there is any chance that a ULB
for a non-NFS protocol is published.  Realistically, any such future ULBs
will
be relevant to Version Two.

I think putting these restrictions in the ULB is the right thing,
structurally. If Chuck
believes it is more expedient to address the issue differently in Version
One, I'm
inclined to defer to his judgment.

On Wed, Mar 23, 2016 at 11:24 AM, Tom Talpey <[email-masked]> wrote:

> On 3/23/2016 11:08 AM, Chuck Lever wrote:
>
>>
>> On Mar 23, 2016, at 9:37 AM, David Noveck <[email-masked]> wrote:
>>>
>>> After our discussion, I read section 7 and came to the conclusion that
>>> the various section within section 7 might be treated differently.  In
>>> particular, during our discussion, we referred to "conventions for
>>> bidirectional operation"as if this was all of Section 7, when in fact this
>>> only applies to Section 7.3.  Looking at the subsections, I note the
>>> following:
>>>
>>>         • Sections 7.1 and 7.2 would make sense where they are or they
>>> could be moved to a separate document.  The only specific issue I have is
>>> section 7.1.3 saying "or only backward direction operations".  It is
>>> unclear how or anybody could choose to do this and, if a protocol were to
>>> do so, the only place that selection could reasonably be addressed is the
>>> ULB.
>>>         • With regard to section 7.3, the specific choices (no RDMA
>>> operations, limited operation size) really belong in 5667bis.  I think Tom
>>> is right to point out that, inherently, the protocol could support other
>>> options. The only section that doesn't belong in 5667bis is 7.3.1 which
>>> should be changed to basically say that direction choices for each ULP are
>>> a responsibility of the ULB.
>>>         • As written now, section 7.4, should be removed.  It basically
>>> says that because we have decided that nothing in the reverse direction is
>>> DDP-eligible, no ULB is necessary.  It is better to leave the decision as
>>> to lack of DDP-eligibility, which is reasonable in the v4.1 case, to the
>>> ULB.  Also, matters relating to the callback program don't belong in a
>>> separate callback program ULB as suggested there.  This stuff belongs in
>>> 5667bis.
>>>
>>
>> I'm uncomfortable relegating part of the specification
>> of Version One backward direction operation to ULBs.
>>
>> The transport can make valuable simplifications if it
>> knows there will never be chunks in the backward direction.
>> Most of the design of the Version One backward conventions
>> is to keep those simplifications for all RPC programs.
>>
>
> I don't understand this statement. So what if there are chunks?
> The transport doesn't process them, the upper layer does. So
> in my opinion, the presence of chunks means nothing to RPC/RDMA.
> They are entirely the domain of the upper layer.
>
> The purpose of having a copy of the XID in the RPC-over-RDMA
>> header is the big red flag here: obviously it was the
>> intention of the authors of RFC 5666 to make it cheap to
>> find the value in this field in the face of varying header
>> length. Finding the RPC message's direction field in the
>> presence of chunk lists would add a per-message cost that
>> is undesirable, IMO.
>>
>
> Ok, that's a point to consider. But again, I don't see why this
> is a protocol specification issue. It's at best a note to the
> implementor.
>
> In particular, this cost is incurred on every RPC message
>> in the forward direction of operation. To eliminate it
>> properly, a message direction field should be added to the
>> header to keep receive processing straightforward while
>> enabling chunks in the backward direction. This can be done
>> in a future version of RPC-over-RDMA.
>>
>
> Sure, if it's useful it should be considered, for rpc-rdma-next.
>
> My bottom line opinion is this: just because nothing is doing it
> today doesn't mean the specification must say MUST NOT. The original
> idea behind rpcrdma was that it was a platform for any rpc
> protocol. Writing all these MUSTs based on the NFS/RDMA choices
> is a significant limitation of that goal.
>
> Tom.
>
>
>
>
> More practically, there are already two implementations of
>> backward direction operation on RPC-over-RDMA Version One
>> based on these conventions. I'm not aware of anyone who is
>> interested in prototyping backward direction chunks at this
>> time. As a rule of thumb, I think we should not write a
>> specification for something we don't plan to prototype at
>> least once.
>>
>> The current limited backward direction mode of operation
>> serves the needs of existing ULPs and implementations. The
>> proposed increase in inline threshold in RPC-over-RDMA
>> Version Two means that backward direction operation can
>> convey 4x larger backward direction messages, and still no
>> chunks are needed.
>>
>> Outside of RPC message size, chunks are appropriate to use
>> when there are significant data payload sizes expected. I'm
>> not aware of any particular ULP backward direction operation
>> that would carry a significant payload that could benefit
>> from DDP. (Feel free to educate me!)
>>
>> So, yes, we _could_ allow chunks in the backward direction
>> in RPC-over-RDMA Version One, but I see no driver to add
>> the complication of backward direction chunks at this time.
>> I would rather spend our energy on more pressing needs,
>> and address backward chunk lists when there is a well-
>> defined requirement for them, and in a context where there
>> is more design leeway to reduce impact on the efficiency of
>> forward channel operation.
>>
>> I still plan to remove all of rfc5666bis Section 7 in favor
>> of advancing the separate existing rpcrdma-bidrection I-D,
>> as discussed yesterday. We should carry this discussion
>> forward relative to that document.
>>
>>
>> --
>> Chuck Lever
>>
>>
>>
>>
>>
>>
>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 23 Mar 2016 15:52:16 -0400

Hi Tom-

> On Mar 23, 2016, at 11:24 AM, Tom Talpey <[email-masked]> wrote:
> 
>>> After our discussion, I read section 7 and came to the conclusion that the various section within section 7 might be treated differently.  In particular, during our discussion, we referred to "conventions for bidirectional operation"as if this was all of Section 7, when in fact this only applies to Section 7.3.  Looking at the subsections, I note the following:
>>> 
>>> 	• Sections 7.1 and 7.2 would make sense where they are or they could be moved to a separate document.  The only specific issue I have is section 7.1.3 saying "or only backward direction operations".  It is unclear how or anybody could choose to do this and, if a protocol were to do so, the only place that selection could reasonably be addressed is the ULB.
>>> 	• With regard to section 7.3, the specific choices (no RDMA operations, limited operation size) really belong in 5667bis.  I think Tom is right to point out that, inherently, the protocol could support other options. The only section that doesn't belong in 5667bis is 7.3.1 which should be changed to basically say that direction choices for each ULP are a responsibility of the ULB.
>>> 	• As written now, section 7.4, should be removed.  It basically says that because we have decided that nothing in the reverse direction is DDP-eligible, no ULB is necessary.  It is better to leave the decision as to lack of DDP-eligibility, which is reasonable in the v4.1 case, to the ULB.  Also, matters relating to the callback program don't belong in a separate callback program ULB as suggested there.  This stuff belongs in 5667bis.
>> 

I would argue that the upper layer in this case (NFSv4)
doesn't care either.

As rfc5666bis is written, if an XDR data item is not
explicitly mentioned as being DDP-eligible, then it is
not eligible by default.

Since there are no DDP eligible data items in NFSv4.[012]
CB operations, there is no need to consider DDP-eligibility
of CB arguments or results at all in 5667 or its bis.

I'd rather not take the effort to specify the transport
behavior of backward direction chunks in a ULB that
doesn't actually need to use them.

So that conclusion can change if there are specific NFSv4
CB arguments or results that someone feels should be DDP
eligible. Are there any?

If not, the only use for chunks is to send Long Call and
Reply messages. That seems to me to be entirely in the
realm, then, of the transport.

> is a protocol specification issue. It's at best a note to the

Suppose the spec language allows chunks in backward
direction calls. Stated something like this:

If an implementation does not support backward
direction chunks, then the ULP MUST advertise a
size limit on backward direction operations such that
they always fit inline.

That doesn't completely exclude the use of chunks,
though. What happens if a peer does send a backward
Call that has a chunk (or sends a Long backward Call)
to a non-chunk-supporting receiver?

MUST the receiver return an RDMA_ERROR message? Then a
non-chunk-supporting receiver still has to parse the
whole header to figure out that a chunk happens to be
present on a backward Call.

MUST the receiver drop the message? No more forward
progress. Eventually the server gives up and revokes
the delegation or layout; undesired consequences.

Likewise if the receiver drops the connection.

IMO the safest design here is to state flatly that
backward chunks are not allowed in Version One. If they
are used, your implementation is not guaranteed to
work. The only alternative is to require full support
for them, and we have no current ULP that needs it.

> is a significant limitation of that goal.

If I understand you correctly, your concern is about
not disallowing future capability in present language.
I agree with that concern.

Just to note that these MUST NOTs are meant to refer
only to the Version One extension described in
rpcrdma-bidirection. They are here to make the
transport protocol work effectively. NFSv4 CB ULP
requirements happen to fit in the window of what
that specification allows.

I can restructure the language to clarify which
normative requirements do not apply to future Versions
of RPC-over-RDMA. The whole document needs that treatment,
I suspect (though honestly I haven't looked at it in a
while).

--
Chuck Lever

---

From: Matt Benjamin <[email-masked]>
Date: Thu, 24 Mar 2016 10:29:08 -0400 (EDT)

Hi,

inline

----- Original Message -----
> From: "Chuck Lever" <[email-masked]>
> To: "Tom Talpey" <[email-masked]>
> Cc: "NFSv4" <[email-masked]>
> Sent: Wednesday, March 23, 2016 3:52:16 PM
> Subject: Re: [nfsv4] rfc5666bis section 7
> 
> Hi Tom-
> 
> 
> >>> bidirectional operation"as if this was all of Section 7, when in fact
> >>> this only applies to Section 7.3.  Looking at the subsections, I note
> >>> the following:
> >>> 
> >>> 	• Sections 7.1 and 7.2 would make sense where they are or they could be
> >>> 	moved to a separate document.  The only specific issue I have is
> >>> 	unclear how or anybody could choose to do this and, if a protocol were
> >>> 	to do so, the only place that selection could reasonably be addressed
> >>> 	is the ULB.
> >>> 	• With regard to section 7.3, the specific choices (no RDMA operations,
> >>> 	limited operation size) really belong in 5667bis.  I think Tom is right
> >>> 	to point out that, inherently, the protocol could support other
> >>> 	should be changed to basically say that direction choices for each ULP
> >>> 	are a responsibility of the ULB.
> >>> 	• As written now, section 7.4, should be removed.  It basically says
> >>> 	that because we have decided that nothing in the reverse direction is
> >>> 	DDP-eligible, no ULB is necessary.  It is better to leave the decision
> >>> 	as to lack of DDP-eligibility, which is reasonable in the v4.1 case, to
> >>> 	the ULB.  Also, matters relating to the callback program don't belong
> >>> 	in a separate callback program ULB as suggested there.  This stuff
> >>> 	belongs in 5667bis.
> >> 

These seem like consequences of limiting decisions in the design of the Version One protocol, some of which are being fixed even in Version One.  We should strive to eliminate problems we can easily describe and solve, in a Version Two.

> for them, and we have no current ULP that needs it.

The latter argument is circular, more below.

> > is a significant limitation of that goal.

I share Tom's view that a version one description of the RPC RDMA backchannel should not constrain the behavior of the protocol in later versions.  It seems to me that you've addressed that concern in your response here.

Having said that, I do strongly intuit that in general NFSv4 is better served by provision up-front with fully symmetric capabilities in the fore and back channels.  As proved to be the case with backchannel security (in AFS3 and again in NFS prior to version 4), symmetry with respect to DDP permits stronger backchannel operations to be defined, whereas lack of the capability closes that off.  Reasoning just from the needs of existing operations is circular--no operation which would benefit from chunk operations (e.g., server-directed bulk transfer) exists in the current protocol, by definition.

> >> from DDP. (Feel free to educate me!)

As above.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
> 

Matt

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

---

From: David Noveck <[email-masked]>
Date: Thu, 24 Mar 2016 15:37:17 -0400

>  forward relative to that document.

OK.  You at least got me to read it.

The first issue is that its intended status is experimental.  I'm not sure
exactly what that means but I'm assuming that if this is to be
clustered-with rfc5666bis, and essentially serves to complete Version One,
this will be changed to Proposed Standard.

Another issue is that in a number of places it says "The rdma_vers field
MUST contain the value one". I think this has to go, since:

   - If this document only applies to Version One it is superfluous.
   - If this document becomes standards-track and does not apply only to
   Version One, then it effectively prevents all future versions from
   implementing backward direction RDMA.
   - If this document is not standards track, it shouldn't be making these
   normative statements.

I think we clearly need to specify that this document only applies to
Version One.

Note that section 3.3, without clearly stating that this applies only to
Version One, definite leans in that direction:

To address the limitations described in this section in the long run,

a new version of the RPC-over-RDMA protocol would be required.  The

use of the conventions described in this document to enable backward

direction operation is thus a transitional approach that is

appropriate only while RPC-over-RDMA Version One is the predominantly

deployed version of the RPC-over-RDMA protocol.

It may be that the 4K inline size limit will address this concern.

Another approach is for Version Two is to at least make use of backward
direction chunks an
OPTIONAL feature. I don't think there should be a REQUIREMENT to support
that as that
would break compatibility with Version One, but if Version Two simply
refrains from reiterating
the language that prevents this, an extension could provide the means
whereby the participants could
determne whether the other side had the appropriate support.

On Thu, Mar 24, 2016 at 10:29 AM, Matt Benjamin <[email-masked]>
wrote:

> could be
> operations,
> says
> These seem like consequences of limiting decisions in the design of the
> Version One protocol, some of which are being fixed even in Version One.
> We should strive to eliminate problems we can easily describe and solve, in
> a Version Two.
>
> >
> I share Tom's view that a version one description of the RPC RDMA
> backchannel should not constrain the behavior of the protocol in later
> versions.  It seems to me that you've addressed that concern in your
> response here.
>
> Having said that, I do strongly intuit that in general NFSv4 is better
> served by provision up-front with fully symmetric capabilities in the fore
> and back channels.  As proved to be the case with backchannel security (in
> AFS3 and again in NFS prior to version 4), symmetry with respect to DDP
> permits stronger backchannel operations to be defined, whereas lack of the
> capability closes that off.  Reasoning just from the needs of existing
> operations is circular--no operation which would benefit from chunk
> operations (e.g., server-directed bulk transfer) exists in the current
> protocol, by definition.
>
> >

---

From: Chuck Lever <[email-masked]>
Date: Thu, 24 Mar 2016 16:26:29 -0400

> On Mar 24, 2016, at 3:37 PM, David Noveck <[email-masked]> wrote:
> 
> The first issue is that its intended status is experimental.  I'm not sure exactly what that means but I'm assuming that if this is to be clustered-with rfc5666bis, and essentially serves to complete Version One, this will be changed to Proposed Standard.

This I-D can be moved to the PS track, as long as
others are OK with that too. 

> Another issue is that in a number of places it says "The rdma_vers field MUST contain the value one".   I think this has to go, since:
> 	• If this document only applies to Version One it is superfluous.
> 	• If this document becomes standards-track and does not apply only to Version One, then it effectively prevents all future versions from implementing backward direction RDMA.
> 	• If this document is not standards track, it shouldn't be making these normative statements.
> I think we clearly need to specify that this document only applies to Version One.
> 
> Note that section 3.3, without clearly stating that this applies only to Version One, definite leans in that direction:

I was hoping to re-use this transitional bi-direction
convention in the base Version Two protocol.

It should be possible to write the bidirection I-D in a
way that allows it to be used with other versions of
RPC-over-RDMA, but without limiting the ability to add
backward chunks (or even a whole different way to
exchange backward direction operations) later.

> Another approach is for Version Two is to at least make use of backward direction chunks an
> OPTIONAL feature.  I don't think there should be a REQUIREMENT to support that as that
> would break compatibility with Version One, but if Version Two simply refrains from reiterating 
> the language that prevents this, an extension could provide the means whereby the participants could

The extensibility feature proposed for RPC-over-RDMA Version
Two is exactly for the purpose of introducing this kind of
enhancement. So an OPTIONAL capability is OK with me.

> On Thu, Mar 24, 2016 at 10:29 AM, Matt Benjamin <[email-masked]> wrote:
> Hi,
> 
> inline
> 

--
Chuck Lever