Subject: [nfsv4] review of draft-ietf-nfsv4-rpcrdma-bidirection-03

From: David Noveck <[email-masked]>
Date: Wed, 4 May 2016 13:10:59 -0400

*General Comments*

*Overall Evaluation*

This document is in good shape and the changes necessary to make it ready
to go forward are small.

A lot of my comments are with regard to inevitable small editorial issues.
A lot of these are matters of preference rather than of correctness.

Another category of comments has to do with the rough edges that have
resulted from this document's history.  It has successively been a set of
conventions, an optional extension, a part of version One and finally a
cross-version guide to bidirectional operation.  I'll leave it to Chuck to
determine the level of sanding/filing/buffing that he feels comfortable
with.

*Important Issues*

The only issue that I think needs to be addressed to go forward is the
treatment of ULBs for bidirectional protocols, such as NFSv4.  Chuck and I
have already discussed this so I think the necessary changes can be made
quickly.

Look at *7. Backward Direction Upper Layer Binding* for details. It appears
that these changes address the issue of contradictions between this
document and draft-ietf-nfsv4-rfc5666bis-05, discussed earlier on the list.

*Comments by Section*

*1.  Introduction*

In the second sentence of the first paragraph, suugest replacing "in
particular pNFS" by "in particular, of pNFS.

In the first sentence of the second paragraph, the phrase "protocol
described in this document" is a problem.  This document doesn't describe a
protocol.  It uses, quite properly, the protocol described in rfc5666bis
(and implicitly future versions).  Suggest replacing "protocol" by
"approach".

In the last sentence of the second paragraph, the word "Therefore" is
flat-out wrong.  Documents which do not change the XDR can and do update
documents that contain XDR without changing the XDR.  Some examples:

   - RFC7530 obsoletes RFC3530 without changing the v4.0 XDR.
   - draft-ietf-nfsv4-rfc3530-migration-update will, when it is finally
   approved, update RFC7530, without changing the v4.0 XDR.
   - draft-ietf-nfsv4rfc5666bis will, when it is approved, obsolete
   RFC5666.  It may make some cosmetic XDR changes to the XDR, but that has
   nothing to do with whether it obsoletes RFC5666.

Some possible ways to address this issue:

   - Just delete the "Therefore".
   - Replace it by "This document does not modify the XDR for RPC-over-RDMA
   Version One [rfc5666bis] or for any future RPC-over-RDMA version."
   - Replace it by "This document does not modify the protocol described by
   [rfc5666bis] or any future RPC-over-RDMA version."

While the statement in the last paragraph is true, it seems incomplete.
Given that we know whose responsibility this is, we should say so, rather
than making the reader guess. I suggest replacing the paragraph by the
following:

This document does not provide an Upper Layer Binding for NFSv4.x
callback operations or any other form of backward direction RPC.
Issues related to ULBs for backward direction operation are dealt with
in Section 7.

*2.  Understanding RPC Direction*

In the first sentence of the first paragraph, I don't understand what
the word "fundamentally" adds in this context.  I suggest deleting it.

In the first sentence of the fourth paragraph, suggest replacing
"generates" by "results in".

In the last sentence of the fifth paragraph, suggest replacing "wait
passively for" by "passively await".

A general problem is that the basic idea stated in the fifth
paragraph, that the client is responsible for establishing transport
connections, appears in sections 2, 2.1, 2.2, and 2.3.

As part of cleaning that up, I am suggesting the following replacement
for the fifth paragraph:

RPC-over-RDMA is a connection-oriented RPC transport.  When a
connection-oriented transport is used, ONC RPC client endpoints are
responsible for initiating transport connections, while ONC RPC
service endpoints passively await incoming connection requests.  This
same division of responsibility applies in cases in which the client
is the Requester (Section 2.1), the Responder (Section 2.2), or both
(Section 2.3).

In the last paragraph suggest replacing "considered" by "addressed".

*2.1.  Forward Direction*

It's not clear what "traditional" means here.  I'm going to suggest
the following revision of the first paragraph

Normally, an ONC RPC client acts as a Requester, while an ONC RPC
service acts as a Responder.  This form of ONC RPC message passing is
referred to as operation in the "forward direction."

In line with the suggestion in the section above, I suggest deleting
the second paragraph.

*2.2.  Backward Direction*

In line with the suggestion in *2.  Understanding RPC Direction*
above, I suggest deleting the second paragraph.

*2.3.  Bi-directional Operation*

In line with the suggestion in *2.  Understanding RPC Direction*
above, I suggest deleting the second sentence of the second paragraph.

*3.  Rationale For Bi-Directional RPC-over-RDMA*

In the section title, I suggest replacing "Rationale For" by "Uses of".

*3.1.  NFSv4.0 Callback Operation*

In the last paragraph, suggest the following changes:

   - In the first sentence, suggest replacing "are fully functional"
by "can function".
   - In the last sentence, suggest replacing "functional correctness"
by "correctness".

*3.2.  NFSv4.1 Callback Operation*

In the first sentence of the penultimate paragraph,suggest replacing
"Some implementations" by "Implementations often".

Suggest replacing the last sentence of the last paragraph by the following:

The pNFS feature raises additional issues.  Because NFSv4.1 relies on
callbacks to help manage pNFS layout state, pNFS operation is not
possible without a backchannel.

*4.  Flow Control*

Suggest the following changes in the first sentence of the first paragraph:

   - replacing "work" by "work properly".
   - replacing "in which to land" by "into which to transfer"

In the second sentence of the first paragraph, suggest replacing
"land" by "accommodate"

Suggest replacing the last sentence of the second paragraph by the following:

In the case of Version One, this is discussed in Section 4.3 of
[I-D.ietf-nfsv4-rfc5666bis].

*5.  Protocol For Backward Operation*

Suggest replacing "Protocol For" by "Effecting".

Suggest the following rewrite of the only paragraph in this section:

Performing backward direction ONC RPC operations over an RPC-over-RDMA
transport connection can be accomplished by using the RPC-over-RDMA
protocol in the way described in the following subsections.  It may be
helpful to refer to the XDR description of the appropriate version of
RPC-over-RDMA.  In the case of Version One, this is contained in
Section 5.1 of [I-D.ietf-nfsv4-rfc5666bis].

*6.  In the Absence of Backward Direction Support*

Suggest moving the current last paragraph so it follows the current
third paragraph.

Suggest prepending "For example, " to the first sentence of the
current fourth paragraph.

*7.  Backward Direction Upper Layer Binding*

The basic problem in this section is that it concerns itself with ULBs
for the backward direction case while the more important case of
bidirectional operation is not explicitly addressed (except in *1.
Introduction*) where it is declared out-of-scope).  I'm going to
propose changes assuming that the necessary material will be added to
this section, although a separate section is possible.

Suggest changing the section title to "Upper Layer Binding for
Backward Direction and Bi-directional Operation"

Suggest adding the following new sentence to the end of the first paragraph:

This applies to both backward direction and bi-directional operation.

Given Chuck's choice regarding how to deal with this issue, I suggest
rewriting the third paragraph as follows:

By default, no data items in a ULP are DDP-eligible.  If there are no
DDP-eligible data items to document, the specification of DDP-eligible
items need not be part of the Upper Layer Binding for an Upper Layer
Protocol that operates only in the backward direction.

I suggest adding a new paragraph after the existing third paragraph,
reading as follows:

An Upper Layer Protocol that operates on RPC-over-RDMA transports and
incorporates operation in both  directions may have DDP-eligible data
items for transfers in either or both directions.  All of these are
specified in an Upper Layer Binding document, as they would be for a
protocol which only operates in a single direction. This applies even
when, as in the case of NFSv4. the XDR is constructed so that each
direction of operation is defined as part of a separate XDR program.

In the last paragraph, suggest replacing "else may be contained" by "is
required".

---

From: Chuck Lever <[email-masked]>
Date: Thu, 5 May 2016 10:07:35 -0400

> On May 4, 2016, at 1:10 PM, David Noveck <[email-masked]> wrote:
> 
> General Comments
> 
> Overall Evaluation
> 
> This document is in good shape and the changes necessary to make it ready to go forward are small. 
> 
> A lot of my comments are with regard to inevitable small editorial issues.  A lot of these are matters of preference rather than of correctness.
> 
> Another category of comments has to do with the rough edges that have resulted from this document's history.  It has successively been a set of conventions, an optional extension, a part of version One and finally a cross-version guide to bidirectional operation.  I'll leave it to Chuck to determine the level of sanding/filing/buffing that he feels comfortable with.
> 
> Important Issues
> 
> The only issue that I think needs to be addressed to go forward is the treatment of ULBs for bidirectional protocols, such as NFSv4.  Chuck and I have already discussed this so I think the necessary changes can be made quickly.
> 
> Look at 7.  Backward Direction Upper Layer Binding for details.  It appears that these changes address the issue of contradictions between this document and draft-ietf-nfsv4-rfc5666bis-05, discussed earlier on the list.
> 
> 
> Comments by Section
> 
> 1.  Introduction
> 
> In the second sentence of the first paragraph, suugest replacing "in particular pNFS" by "in particular, of pNFS.
> 
> In the first sentence of the second paragraph, the phrase "protocol described in this document" is a problem.  This document doesn't describe a protocol.  It uses, quite properly, the protocol described in rfc5666bis (and implicitly future versions).  Suggest replacing "protocol" by "approach".
> 
> In the last sentence of the second paragraph, the word "Therefore" is flat-out wrong.  Documents which do not change the XDR can and do update documents that contain XDR without changing the XDR.  Some examples:
> 	• RFC7530 obsoletes RFC3530 without changing the v4.0 XDR.
> 	• draft-ietf-nfsv4-rfc3530-migration-update will, when it is finally approved, update RFC7530, without changing the v4.0 XDR.
> 	• draft-ietf-nfsv4rfc5666bis will, when it is approved, obsolete RFC5666.  It may make some cosmetic XDR changes to the XDR, but that has nothing to do with whether it obsoletes RFC5666.
> 	• Just delete the "Therefore".
> 	• Replace it by "This document does not modify the XDR for RPC-over-RDMA Version One [rfc5666bis] or for any future RPC-over-RDMA version."  
> 	• Replace it by "This document does not modify the protocol described by [rfc5666bis] or any future RPC-over-RDMA version."
> While the statement in the last paragraph is true, it seems incomplete.  Given that we know whose responsibility this is, we should say so, rather than making the reader guess. I suggest replacing the paragraph by the following:
> This document does not provide an Upper Layer Binding for NFSv4.x callback operations or any other form of backward direction RPC.  Issues related to ULBs for backward direction operation are dealt with in Section 7. 
> 2.  Understanding RPC Direction
> In the first sentence of the first paragraph, I don't understand what the word "fundamentally" adds in this context.  I suggest deleting it. 
> In the first sentence of the fourth paragraph, suggest replacing "generates" by "results in".
> In the last sentence of the fifth paragraph, suggest replacing "wait passively for" by "passively await".
> A general problem is that the basic idea stated in the fifth paragraph, that the client is responsible for establishing transport connections, appears in sections 2, 2.1, 2.2, and 2.3.  
> As part of cleaning that up, I am suggesting the following replacement for the fifth paragraph:
> RPC-over-RDMA is a connection-oriented RPC transport.  When a connection-oriented transport is used, ONC RPC client endpoints are responsible for initiating transport connections, while ONC RPC service endpoints passively await incoming connection requests.  This same division of responsibility applies in cases in which the client is the Requester (Section 2.1), the Responder (Section 2.2), or both (Section 2.3).
> 2.1.  Forward Direction
> It's not clear what "traditional" means here.  I'm going to suggest the following revision of the first paragraph
> Normally, an ONC RPC client acts as a Requester, while an ONC RPC service acts as a Responder.  This form of ONC RPC message passing is referred to as operation in the "forward direction."
> In line with the suggestion in the section above, I suggest deleting the second paragraph.
> 2.2.  Backward Direction
> In line with the suggestion in 2.  Understanding RPC Direction above, I suggest deleting the second paragraph.
> 2.3.  Bi-directional Operation
> In line with the suggestion in 2.  Understanding RPC Direction above, I suggest deleting the second sentence of the second paragraph.
> 3.  Rationale For Bi-Directional RPC-over-RDMA
> 3.1.  NFSv4.0 Callback Operation
> 	• In the first sentence, suggest replacing "are fully functional" by "can function".
> 	• In the last sentence, suggest replacing "functional correctness" by "correctness".
> 3.2.  NFSv4.1 Callback Operation
> In the first sentence of the penultimate paragraph,suggest replacing "Some implementations" by "Implementations often".
> The pNFS feature raises additional issues.  Because NFSv4.1 relies on callbacks to help manage pNFS layout state, pNFS operation is not possible without a backchannel.
> 4.  Flow Control
> 	• replacing "work" by "work properly".
> 	• replacing "in which to land" by "into which to transfer"
> In the second sentence of the first paragraph, suggest replacing "land" by "accommodate"
> In the case of Version One, this is discussed in Section 4.3 of [I-D.ietf-nfsv4-rfc5666bis].
> 5.  Protocol For Backward Operation
> Performing backward direction ONC RPC operations over an RPC-over-RDMA transport connection can be accomplished by using the RPC-over-RDMA protocol in the way described in the following subsections.  It may be helpful to refer to the XDR description of the appropriate version of RPC-over-RDMA.  In the case of Version One, this is contained in Section 5.1 of [I-D.ietf-nfsv4-rfc5666bis].
> 6.  In the Absence of Backward Direction Support
> Suggest moving the current last paragraph so it follows the current third paragraph.
> Suggest prepending "For example, " to the first sentence of the current fourth paragraph.
> 7.  Backward Direction Upper Layer Binding
> The basic problem in this section is that it concerns itself with ULBs for the backward direction case while the more important case of bidirectional operation is not explicitly addressed (except in 1.  Introduction) where it is declared out-of-scope).  I'm going to propose changes assuming that the necessary material will be added to this section, although a separate section is possible.
> Suggest changing the section title to "Upper Layer Binding for Backward Direction and Bi-directional Operation"
> Given Chuck's choice regarding how to deal with this issue, I suggest rewriting the third paragraph as follows:
> By default, no data items in a ULP are DDP-eligible.  If there are no DDP-eligible data items to document, the specification of DDP-eligible items need not be part of the Upper Layer Binding for an Upper Layer Protocol that operates only in the backward direction.
> I suggest adding a new paragraph after the existing third paragraph, reading as follows:
> An Upper Layer Protocol that operates on RPC-over-RDMA transports and incorporates operation in both  directions may have DDP-eligible data items for transfers in either or both directions.  All of these are specified in an Upper Layer Binding document, as they would be for a protocol which only operates in a single direction. This applies even when, as in the case of NFSv4. the XDR is constructed so that each direction of operation is defined as part of a separate XDR program.
> In the last paragraph, suggest replacing "else may be contained" by "is required".

Thanks, these look straightforward. Barring any responses
to these comments, I'll fold them in before the next
revision.

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Wed, 18 May 2016 15:44:51 -0400

Hi Dave-

> On May 5, 2016, at 10:07 AM, Chuck Lever <[email-masked]> wrote:
> 
> 
>> The basic problem in this section is that it concerns itself with ULBs for the backward direction case while the more important case of bidirectional operation is not explicitly addressed (except in 1.  Introduction) where it is declared out-of-scope).

Maybe mark it up to my progressing years, but as I'm
looking back at these comments, I find that I don't
understand them at all.

Where is "bi-directional operation declared out of scope?"
Section 1 appropriately says that an NFSv4.x Upper Layer
Binding is out of scope. It says nothing about
bi-directional operation of RPC-over-RDMA. If needed I
can clarify Section 1.

>> I'm going to propose changes assuming that the necessary material will be added to this section, although a separate section is possible.

A separate section for what?

I guess I don't understand why backward operation by
itself is any different than backward operation on
the same transport as forward operation.

>> This applies to both backward direction and bi-directional operation.

I don't understand why you believe this is necessary to
state here.

>> By default, no data items in a ULP are DDP-eligible.  If there are no DDP-eligible data items to document, the specification of DDP-eligible items need not be part of the Upper Layer Binding for a…

That's my recollection as well.

>> An Upper Layer Protocol that operates on RPC-over-RDMA transports and incorporates operation in both  directions may have DDP-eligible data items for transfers in either or both directions.  All o…

OK, but: don't NFSv4.1 callback operations also use a
different RPC program than NFSv4.1 forward channel
operations? Since there is a separate XDR definition,
and a unique RPC Program number, that makes these two
separate and distinct RPC Programs (and thus are
different ULPs), IMO.

So what is the essential difference we want to call out
here? Thinking aloud:

If a backward-only ULP is defined by itself, it has to
have a statement, even a simple one, about DDP-eligible
data items.

If a backward-only ULP is defined along side a forward
only ULP, it can fall back on the default DDP-eligibility
stated in rfc5666bis (or the equivalent normative
standard that specifies the RPC-over-RDMA version that
is being used).

If a ULP defines operations that can flow in either
direction, then what? Note: we don't have one of these
yet, AFAIK.

I don't see why these distinctions are necessary.

How about: Before any operation can flow in either
direction, it MUST be covered by a DDP-eligibility
statement.

If there is an associated forward-only ULP that already
has a binding, the backward operations are covered by
that binding (meaning: if the backward operations are
not mentioned, they have the default DDP eligibility).

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 18 May 2016 17:13:01 -0400

> Where is "bi-directional operation declared out of scope?"

I agree that it doesn't say that.

> bi-directional operation of RPC-over-RDMA

Note that this comment was made in the context of section 7 which concerns
itself with ULBs and section one does say that the ULB for the only
protocol that supports bidirectional operation is out of the scope of this
document.  I agree that that is a valid statement, BTW.

> If needed I

I think the clarification I am looking for is one that says where the ULB
will be provided, rather than where it is not provided.
This could be an informative reference to rfc5667bis or a simple statement
that the issue is addressed in section 7.

> >> I'm going to propose changes assuming that the necessary material will
be added to this section, although a separate section is possible.

> A separate section for what?

A separate section for ULBs for bidirectional operation, as opposed to the
existing section for backlward operation.

We both agree that this is undesirable, but I need to answer your question.

> the same transport as forward operation.

The question is whether backward operation of a single protocol is
different from bidirectional operation of a single protocol.  I don't think
it is any different but given that this distinction is made, it needs to be
made clear that the same ULB rules apply to both.

> I don't understand why you believe this is necessary to

I don't know that it is necessary but I do think it would helpful.

In many places you make the distinction between backward and bi-directional
operation, so it would helpful, if this section is to cover both, to state
that it does cover both.

> OK but: don't NFSv4.1 callback operations also use a
> operations?

Yes they do.

> Since there is a separate XDR definition,
> separate and distinct RPC Programs

They are distinct but they  are not separate.  They function as a
single protocol.

RFC5661 does not say it describes two protocols.  It says it describes a
single protocol, NFSv4.1, which operates in both directions.

>  (and thus are

I don't think it is actually your opinion that forward and backward
direction NFSv4 operations are part of two separate protocols.  If it is,
this is a very new opinion.

> have a statement, even a simple one, about DDP-eligible

Right, but my concern about section 7 was that it seemed that was the only
case that was addressed there and, as I read it, bidirectional operation
was not addressed.

> standard that specifies the RPC-over-RDMA version that

Yes, but as the bidirection docment is now, you also have to make provision
for the non-default case, in which there might be backward DDP-eligible
items.

So I guess you are saying, and it might even be your opinion that forward
and backward direction are two protocols, described "alongside" one
another.  If that's your opinion and you want to say it that way OK, but it
might be confusing to people.  If you are going to adopt that usage, you
should make it clear by citing the case of NFSv4.1, that that is what you
mean.

> direction, then what? Note: we don't have one of these

In that case, the ULB is responsible for defining DDP-eligibility for
transmission in both directions.

there isn't a case yet, but neither is there a case of backward direction
chunks.   This sort of spec needs to address these kinds of future
possibilities the best that it can.

Fortunately, in this case, that isn't all that hard.

> I don't see why these distinctions are necessary.

They aren't necessary, but they have been made and given that they have
been made, you have to deal with the consequences of having made them.

In this case, if you want to treat bidirectional and backward direction
operation the same, from the ULB viewpoint, you have to state that that is
what you are doing.

> direction, it MUST be covered by a DDP-eligibility

OK, except for the "MUST".

> not mentioned, they have the default DDP eligibility).

I guess that works but I think that you are adding confusion
by trying to treat NFSv4.1 as two protocols.  If you want to do
that, you need to make clear, in the introduction if necessary,
that that is what you are doing

On Wed, May 18, 2016 at 3:44 PM, Chuck Lever <[email-masked]> wrote:

> bidirectional operation is not explicitly addressed (except in 1.
> Introduction) where it is declared out-of-scope).
>
> >> I'm going to propose changes assuming that the necessary material will
> be added to this section, although a separate section is possible.
>
> incorporates operation in both  directions may have DDP-eligible data items
> for transfers in either or both directions.  All of these are specified in
> an Upper Layer Binding document, as they would be for a protocol which only
> operates in a single direction. This applies even when, as in the case of
> NFSv4. the XDR is constructed so that each direction of operation is
> defined as part of a separate XDR program.
>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 18 May 2016 17:25:40 -0400

> On May 18, 2016, at 5:13 PM, David Noveck <[email-masked]> wrote:
> 
> Note that this comment was made in the context of section 7 which concerns itself with ULBs and section one does say that the ULB for the only protocol that supports bidirectional operation is out of the scope of this document.  I agree that that is a valid statement, BTW. 
> 
> > If needed I
> > can clarify Section 1.
> 
> I think the clarification I am looking for is one that says where the ULB will be provided, rather than where it is not provided.
> This could be an informative reference to rfc5667bis or a simple statement that the issue is addressed in section 7.
> 
> A separate section for ULBs for bidirectional operation, as opposed to the existing section for backlward operation.
> 
> The question is whether backward operation of a single protocol is different from bidirectional operation of a single protocol.  I don't think it is any different but given that this distinction is made, it needs to be made clear that the same ULB rules apply to both.
> 
> In many places you make the distinction between backward and bi-directional operation, so it would helpful, if this section is to cover both, to state that it does cover both. 
> 
> RFC5661 does not say it describes two protocols.  It says it describes a single protocol, NFSv4.1, which operates in both directions.
> 
> >  (and thus are
> > different ULPs), IMO.
> 
> I don't think it is actually your opinion that forward and backward direction NFSv4 operations are part of two separate protocols.  If it is, this is a very new opinion.

How do we define Upper Layer Protocol? An Upper Layer
Binding applies to an Upper Layer Protocol, which is
defined as a distinct RPC Program and Version. See
Section 7 rfc5666bis:

> Each RPC Program and Version tuple that utilizes RPC-over-RDMA
> Version One needs to have an Upper Layer Binding specification.

RFC 5661/2 define two separate RPC Programs, thus
separate ULBs are needed, according to rfc5666bis.

This is the way the I-Ds are written now. I am
willing to entertain alternatives.

> Right, but my concern about section 7 was that it seemed that was the only case that was addressed there and, as I read it, bidirectional operation was not addressed.

I don't see how backward-only and bidirectional operation
are addressed separately, except for the separate definitions
in Section 2. Can you give other examples of where they are
separately treated?

> Yes, but as the bidirection docment is now, you also have to make provision for the non-default case, in which there might be backward DDP-eligible items.
> 
> So I guess you are saying, and it might even be your opinion that forward and backward direction are two protocols, described "alongside" one another.  If that's your opinion and you want to say it that way OK, but it might be confusing to people.  If you are going to adopt that usage, you should make it clear by citing the case of NFSv4.1, that that is what you mean.  
> 
> In that case, the ULB is responsible for defining DDP-eligibility for transmission in both directions.
> 
> there isn't a case yet, but neither is there a case of backward direction chunks.   This sort of spec needs to address these kinds of future possibilities the best that it can.
> 
> They aren't necessary, but they have been made and given that they have been made, you have to deal with the consequences of having made them.
> 
> In this case, if you want to treat bidirectional and backward direction operation the same, from the ULB viewpoint, you have to state that that is what you are doing.
> 

It's the same MUST that applies to other ULPs, stated
in rfc5666bis Section 7.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 18 May 2016 22:21:10 -0400

> defined as a distinct RPC Program and Version. See

I think it would be helpful if the defnition of "Upper Layer Protocol"
matched the way the word "protocol" is normally used.  And NFSv4.1
is, quite reasonably, described as a protocol, and not as a set of
two protocols described alongside one another.

> willing to entertain alternatives.

OK.  First of all it might be stated that when two separate RPC
programs are defined together and used together as a  single protocol
capable of bidirectional operation, they can be
considered as a single protocol with a ULB,covering both
directions of operation.

> in Section 2.

The introduction to the subject defines forward operation,
backward operation, and bi-directional operation, all on the
same level, as the three classes of RPC directionality.

> Can you give other examples of where they are
> separately treated?

No I can't and part of the reason is that,  in most contexts, you are
talking about a request or response that part of a forward or backward RPC.
 there are no bidirectional RPC's even though there are (or may
be) bidirectional protocols.

It is quite natural in the context to treat backward an bidirectional
together.   The issue is that if you introduce these three classes and then
choose to treat two of them together, you need to tell the reader
that you are gong to do so.  You may think it is natural and obvious to
treat these together but you can't assume the reader will
naturally share that view, given he has been told that these are separate
things.

Of course,, we now have to deal with the irony that arises from the fact
that you originally began this work to enable the
NFSv4.1 protocol to operate bidirectionally, and having done that, you have
(magically :-) made the NFSv4.1 protocol disappear  and with
it bidirectional operation.  If I understand your position correctly, we
now no longer have a single protocol operating bidirectionally.  Instead,
we have one ULP operating in the forward direction and another ULP, defined
alongside it, operating in the backward direction.  If this is the case,
then bidirectional protocols  do not currently exist but might be defined
in the future.

> It's the same MUST that applies to other ULPs, stated

It is the same word but it is not used in the same way:

There are the only three uses of "MUST" in section 7:

Senders MUST NOT reduce data items that are not DDP-eligible.

In the first case, a responder MUST NOT process the Call message.

Both types of violations MUST be reported as described in Section 5.5.2.

These are all straightforward directions to implementers.

I know there is uncertainty/dispute about whether RFC2119 terms are
appropriately directed to spec authors.

I found your statement above troubling in that it tied
an implementation-level event together with a specification level event.

On Wed, May 18, 2016 at 5:25 PM, Chuck Lever <[email-masked]> wrote:

> > Note that this comment was made in the context of section 7 which
> concerns itself with ULBs and section one does say that the ULB for the
> only protocol that supports bidirectional operation is out of the scope of
> this document.  I agree that that is a valid statement, BTW.
> >
> > > If needed I
> > > can clarify Section 1.
> >
> > I think the clarification I am looking for is one that says where the
> ULB will be provided, rather than where it is not provided.
> > This could be an informative reference to rfc5667bis or a simple
> statement that the issue is addressed in section 7.
> >
> > > >> I'm going to propose changes assuming that the necessary material
> will be added to this section, although a separate section is possible.
> >
> > A separate section for ULBs for bidirectional operation, as opposed to
> the existing section for backlward operation.
> >
> > In many places you make the distinction between backward and
> bi-directional operation, so it would helpful, if this section is to cover
> both, to state that it does cover both.
> >
> > Right, but my concern about section 7 was that it seemed that was the
> only case that was addressed there and, as I read it, bidirectional
> operation was not addressed.
>
> > Yes, but as the bidirection docment is now, you also have to make
> provision for the non-default case, in which there might be backward
> DDP-eligible items.
> >
> > So I guess you are saying, and it might even be your opinion that
> forward and backward direction are two protocols, described "alongside" one
> > there isn't a case yet, but neither is there a case of backward
> direction chunks.   This sort of spec needs to address these kinds of
> future possibilities the best that it can.
> >
> > >> The basic problem in this section is that it concerns itself with
> ULBs for the backward direction case while the more important case of
> > >> I'm going to propose changes assuming that the necessary material
> will be added to this section, although a separate section is possible.
> >

---

From: Chuck Lever <[email-masked]>
Date: Thu, 19 May 2016 10:13:21 -0400

Hi Dave-

> On May 18, 2016, at 10:21 PM, David Noveck <[email-masked]> wrote:
> 

But we do make a distinction between NLM, NFSACL,
MOUNT, NSM, and NFS protocols, referring to NFS
as the "main protocol" and the others as "side-band
protocols."

In NFSv4.0, forward operation and callback are
always used on distinct connections, and they have
distinct RPC Program numbers and separate XDR
definitions. Forward operation can occur without
a callback channel. Are those considered one
protocol, or two?

Is NFSv3 the same ULP as NFSv4.0 or NFSv4.1? If so,
why are there separate discussions in RFC 5667 for
v3 and v4.0, and why are we not attempting to claim
that RFC 5667 or rfc5667bis are normative for all
future versions of NFS?

Thus I don't consider the colloquial usage of the
term 'protocol' to be adequate in this context.

> programs are defined together and used together as a  single protocol capable of bidirectional operation, they can be 

Except that last time I included a mention of call
direction in rfc5666bis, you objected to that. :-)

I still prefer the way rfc5666bis is written now.
It is unambiguous about what needs to have a
Binding. The boundaries defined by Program and
Version number are historical, simple to
understand, and well-defined.

(And, this is a valuable conversation, since we
as a WG should be striving to get this right).

> No I can't and part of the reason is that,  in most contexts, you are talking about a request or response that part of a forward or backward RPC.  there are no bidirectional RPC's even though there are (or may be) bidirectional protocols.
> 
> together.   The issue is that if you introduce these three classes and then choose to treat two of them together, you need to tell the reader
> that you are gong to do so.  You may think it is natural and obvious to treat these together but you can't assume the reader will
> naturally share that view, given he has been told that these are separate things.

It seems natural to me because, logically, you
can't have bi-directional operation without
backward direction operation. I can review the
text with your comment in mind, however.

Besides the sentence you proposed in Section 7,
are there other places in the document where I
should add "and bi-directional" or its equivalent?
I'm open to suggestion.

> Of course,, we now have to deal with the irony that arises from the fact that you originally began this work to enable the NFSv4.1 protocol to operate bidirectionally, and having done that, you have (magically :-) made the NFSv4.1 protocol disappear  and with it bidirectional operation.  If I understand your position correctly, we now no longer have a single protocol operating bidirectionally.

When it is broken down, the RPC transport, not the
protocol, operates bi-directionally. For example:

A "uni-directional" protocol such as NFSv3 or NLM
can be used bi-directionally, if two connection
endpoints act as both clients and servers.

NFSv4.1 implementations can set up distinct and
separate connections for forward and backward
operation. In that case, no bi-directional
operation is occurring on either connection (after
the BC2S operation on the callback connection).

> Instead, we have one ULP operating in the forward direction and another ULP, defined alongside it, operating in the backward direction.

In addition, the set of procedures that may be used
in the forward direction and the set of procedures
that may be used in the backward direction are
disjoint.

> If this is the case, then bidirectional protocols  do not currently exist but might be defined in the future.

I agree with that.

What the I-D creates is the ability to send RPC
requests in both directions on a transport. How
ULPs use this capability is rather up to them.

> I know there is uncertainty/dispute about whether RFC2119 terms are appropriately directed to spec authors.
> 
> I found your statement above troubling in that it tied an implementation-level event together with a specification level event.

Oh, I see. "MUST be covered by a DDP-eligibility
statement" is directed to spec authors, not to
implementers, thus an RFC 2119 term is not
appropriate. Got it.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 19 May 2016 11:52:35 -0400

> backward direction operation.

That's likely to be the case, but note that, you can't have bi-directional
operation
without forward direction operation either, and it somehow doesn't seem
natural
to identify forward and bi-directional operation.  I think logic does not
have as
much to do with this as you assume.

> > If this is the case, then bidirectional protocols  do not currently
exist but might be defined in the future.

> I agree with that.

Yes, but it seems we disagree about whether they currently exist because we
use the word "protocol" in
different ways.  This is not a big problem right now but I'm worried about
a situation in which two of the
documents that NFSv4 implementers refer to use such a basic term in such
different ways.

On Thu, May 19, 2016 at 10:13 AM, Chuck Lever <[email-masked]>
wrote:

> talking about a request or response that part of a forward or backward
> RPC.  there are no bidirectional RPC's even though there are (or may be)
> bidirectional protocols.
> >
> > together.   The issue is that if you introduce these three classes and
> then choose to treat two of them together, you need to tell the reader
> that you originally began this work to enable the NFSv4.1 protocol to
> operate bidirectionally, and having done that, you have (magically :-) made
> the NFSv4.1 protocol disappear  and with it bidirectional operation.  If I
> understand your position correctly, we now no longer have a single protocol
> operating bidirectionally.
>
> > Instead, we have one ULP operating in the forward direction and another
> ULP, defined alongside it, operating in the backward direction.
>
> > If this is the case, then bidirectional protocols  do not currently
> exist but might be defined in the future.
>
> I agree with that.
>
> > I found your statement above troubling in that it tied an
> implementation-level event together with a specification level event.
>
> > > RFC5661 does not say it describes two protocols.  It says it describes
> a single protocol, NFSv4.1, which operates in both directions.
> > >
> > > >  (and thus are
> > > > different ULPs), IMO.
> > >
> > > They aren't necessary, but they have been made and given that they
> have been made, you have to deal with the consequences of having made them.
> > >
> > > In this case, if you want to treat bidirectional and backward
> direction operation the same, from the ULB viewpoint, you have to state
> that that is what you are doing.
> > >
> > > >> Given Chuck's choice regarding how to deal with this issue, I
> suggest rewriting the third paragraph as follows:
> > > >> By default, no data items in a ULP are DDP-eligible.  If there are
> no DDP-eligible data items to document, the specification of DDP-eligible
> > > >> An Upper Layer Protocol that operates on RPC-over-RDMA transports
> and incorporates operation in both  directions may have DDP-eligible data
> protocol which only operates in a single direction. This applies even when,
> as in the case of NFSv4. the XDR is constructed so that each direction of
> operation is defined as part of a separate XDR program.
> > >

---

From: Chuck Lever <[email-masked]>
Date: Fri, 20 May 2016 11:34:32 -0400

> On May 19, 2016, at 11:52 AM, David Noveck <[email-masked]> wrote:
> 
> That's likely to be the case, but note that, you can't have bi-directional operation
> without forward direction operation either, and it somehow doesn't seem natural
> to identify forward and bi-directional operation.  I think logic does not have as 
> Yes, but it seems we disagree about whether they currently exist because we use the word "protocol" in 
> different ways.  This is not a big problem right now but I'm worried about a situation in which two of the 
> documents that NFSv4 implementers refer to use such a basic term in such different ways.

The language in Section 7 of -03 was indeed problematic.
It contradicted, for example, our long-standing claim
that RFC 5667 is missing DDP-eligibility statements for
callback operations. (I think part of the reason for
the problematic language was the desire to grandfather
NFSv4 callback operations into existing RFC 5667 language,
and perhaps that is not consistent with what has been
written and discussed so far).

I've rewritten Section 7:

7.  Considerations For Upper Layer Bindings

   An Upper Layer Protocol that operates on RPC-over-RDMA transports may
   have procedures that include DDP-eligible data items.  DDP-
   eligibility is specified in an Upper Layer Binding.  Direction of
   operation does not obviate the need for DDP-eligibility statements.

   Backward-only operation requires the client endpoint to establish a
   fresh connection.  The Upper Layer Binding can specify appropriate
   RPC binding parameters for such connections.

   Bi-directional operation occurs on an already-established connection.
   Specification of RPC binding parameters is usually not necessary in
   this case.

   For bi-directional operation, other considerations about sharing an
   RPC-over-RDMA transport with another ULP may apply.  Consult
   Section 7 of [I-D.ietf-nfsv4-rfc5666bis] for details about what else
   may be contained in an Upper Layer Binding.

Hopefully that addresses your concerns about conflating
backward direction and bidirection, and issues with the
use of the term 'Upper Layer Protocol.'

> On Thu, May 19, 2016 at 10:13 AM, Chuck Lever <[email-masked]> wrote:
> Hi Dave-
> 
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 20 May 2016 12:51:47 -0400

Works for me.

I think we can now (or at midnight) proceed to move these documents into WG
Consensus state.

On Fri, May 20, 2016 at 11:34 AM, Chuck Lever <[email-masked]>
wrote:

> > > that you are gong to do so.  You may think it is natural and obvious
> to treat these together but you can't assume the reader will
> > > Of course,, we now have to deal with the irony that arises from the
> fact that you originally began this work to enable the NFSv4.1 protocol to
> > > Instead, we have one ULP operating in the forward direction and
> another ULP, defined alongside it, operating in the backward direction.
> >
> > > > I think the clarification I am looking for is one that says where
> the ULB will be provided, rather than where it is not provided.
> > > > > >> I'm going to propose changes assuming that the necessary
> material will be added to this section, although a separate section is
> possible.
> > > >
> > > > A separate section for ULBs for bidirectional operation, as opposed
> to the existing section for backlward operation.
> > > >
> > > > RFC5661 does not say it describes two protocols.  It says it
> describes a single protocol, NFSv4.1, which operates in both directions.
> > > >
> > > > >  (and thus are
> > > > > different ULPs), IMO.
> > > >
> > > > Right, but my concern about section 7 was that it seemed that was
> the only case that was addressed there and, as I read it, bidirectional
> > > > >> By default, no data items in a ULP are DDP-eligible.  If there
> are no DDP-eligible data items to document, the specification of
> DDP-eligible items need not be part of the Upper Layer Binding for an Upper
> Layer Protocol that operates only in the backward direction.
> > > >

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 20 May 2016 17:29:34 +0000

Thanks for your opinion, Dave.  We will wait to ensure that no one else in the WG has comments.

Spencer

From: nfsv4 [mailto:[email-masked]] On Behalf Of David Noveck
Sent: Friday, May 20, 2016 9:52 AM
To: Chuck Lever <[email-masked]>
Cc: [email-masked]
Subject: Re: [nfsv4] review of draft-ietf-nfsv4-rpcrdma-bidirection-03

Works for me.

I think we can now (or at midnight) proceed to move these documents into WG Consensus state.

On Fri, May 20, 2016 at 11:34 AM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:

> On May 19, 2016, at 11:52 AM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
>

> On Thu, May 19, 2016 at 10:13 AM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:
> Hi Dave-
>
>
> > On May 18, 2016, at 10:21 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
> >
> > On Wed, May 18, 2016 at 5:25 PM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:
> >
> > > On May 18, 2016, at 5:13 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
> > >
> > > On Wed, May 18, 2016 at 3:44 PM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:
> > > Hi Dave-
> > >
> > >
> > > > On May 5, 2016, at 10:07 AM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:
> > > >
> > > >
> > > >> On May 4, 2016, at 1:10 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
> > > >>
> > > [email-masked]<mailto:[email-masked]>
> > > https://www.ietf.org/mailman/listinfo/nfsv4<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.ietf.org%2fmailman%2flistinfo%2fnfsv4&data=01%7c01%7csshepler%40microsoft.com%7cd0d8d162cc3746acfd4908d380cf139d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=%2fonEjqwWqYZEnE9eaRUZ1bCYi8fwuj7WURzT9fDDSxQ%3d>
> >
> > --
> > Chuck Lever
> >
> >
> >
> >
>
> --
> Chuck Lever
>
>
>
>
--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Fri, 20 May 2016 13:53:19 -0400

> On May 20, 2016, at 1:29 PM, Spencer Shepler <[email-masked]> wrote:
> 
>  

I plan to submit fresh revisions of rpcrdma-bidirection
and rfc5666bis (one minor change) next week which will
reflect comments and changes requested during WGLC.

> On Fri, May 20, 2016 at 11:34 AM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever