Subject: Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

From: David Noveck <[email-masked]>
Date: Fri, 3 Jun 2016 10:33:17 -0400

A week ago I wrote:
> The problem is that, in a multi-direction context, a given implementation
only knows whether it is a client or
> a server.  It only knows if it is a requester or a responder based on the
message it is processing at any given
> instant and if it can't figure that out, it is unsure whether it is a
requester or responder and may be neither at
> particular times.

It's now been at least two weeks since last call was supposed to end for
the Version One RDMA documents
and a week since Chuck posted any necessary updates.  I'm not sure why the
documents are still listed as in
WGLC, but I wanted to make it clear that my comments above should not be
holding these up.

Those comments refer to a problem that is inherent in Version One  and
cannot be fixed within the constraints
of a Version One no-XDR-change update.  Since the Working group has decided
to do a Version One no-XDR-change
update, the three existing RDMA documents should now go forward.  I don;t
believe there is any disagreement on
that point.

On Fri, May 27, 2016 at 1:33 PM, David Noveck <[email-masked]> wrote:

> > But true enough, there may be others.
>
> There are others but I'm coming to the conclusion that it would be too
> disruptive to try to address them in Version One :-(
>
> Fortunately, we don't have to since these are very unusual situations :-)
>
> For example, "RDMA_ERROR is responder-to-requester only" does not admit
> an ambiguity in a one-direction-only context.
>
> The problem is that, in a multi-direction context, a given implementation
> only knows whether it is a client or
> a server.  It only knows if it is a requester or a responder based on the
> message it is processing at any given
> instant and if it can't figure that out, it is unsure whether it is a
> requester or responder and may be neither at
> particular times.
>
> Sigh!
>
>
> On Fri, May 27, 2016 at 10:01 AM, Chuck Lever <[email-masked]>
> wrote:
>
>>
>> > On May 27, 2016, at 8:24 AM, David Noveck <[email-masked]> wrote:
>> >
>> > I wrote:
>> > > but I have a problem with writing the spec as if
>> > > such situations don't exist.
>> >
>> > I've been thinking more about this.  I think the problem lies in what I
>> mean by
>> > "the spec".  If that means "rfc566bis", then such situations cannot
>> exist.
>> > rfc566bis seems to only deal with forward direction operation.  In such
>> an environment,
>> > if you are a client and you receive a message, it has to be a reply.
>> Similarly, if you are
>> > server and you receive a message, it has to be a request.  So there is
>> no occasion for the
>> > kind of ambiguity I'm concerned about.
>> >
>> > On the other hand, if "the spec" means "the specification of
>> RPC-over-RDMA Version One",
>> > in which multiple directions of operation can be supported, the
>> ambiguity is unavoidable. If
>> > you (client or server) receive a message you do not know a priori,
>> whether it is a request or
>> > a reply and there are situations in which the receiver cannot
>> determine, given a malformed
>> > message, whether it has received a malformed request or reply.
>>
>> I left the existing language in rpcrdma-bidirection.
>> So Section 4.1 still has:
>>
>> >    When message direction is not fully determined by context or by an
>> >    accompanying RPC message with a call direction field, it is not
>> >    possible to tell whether the header credit value is a request or
>> >    grant, or whether the value applies to the forward direction or
>> >    backward direction.  In such cases, the receiver MUST NOT use the
>> >    header’s credit value.
>>
>>
>> Adding the "RDMA_ERROR is responder-to-requester only"
>> to rfc5666bis, plus this from Section 5.1:
>>
>> >    The rdma_vers field MUST contain the same value in backward and
>> >    forward direction Call messages on the same connection.
>>
>>
>> And this from Section 6:
>>
>> >    Therefore, an Upper Layer Protocol consumer MUST NOT perform backward
>> >    direction ONC RPC operations unless the peer consumer has indicated
>> >    it is prepared to handle them.
>>
>>
>> All serve to close the largest windows of ambiguity we
>> know about. But true enough, there may be others.
>>
>>
>> --
>> Chuck Lever
>>
>>
>>
>>
>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 3 Jun 2016 11:04:52 -0400

> On Jun 3, 2016, at 10:33 AM, David Noveck <[email-masked]> wrote:
> 
> A week ago I wrote:
> > The problem is that, in a multi-direction context, a given implementation only knows whether it is a client or 
> > a server.  It only knows if it is a requester or a responder based on the message it is processing at any given 
> > instant and if it can't figure that out, it is unsure whether it is a requester or responder and may be neither at
> > particular times.
> 
> It's now been at least two weeks since last call was supposed to end for the Version One RDMA documents
> and a week since Chuck posted any necessary updates.  I'm not sure why the documents are still listed as in 
> WGLC, but I wanted to make it clear that my comments above should not be holding these up.
> 
> Those comments refer to a problem that is inherent in Version One  and cannot be fixed within the constraints
> of a Version One no-XDR-change update.  Since the Working group has decided to do a Version One no-XDR-change
> update, the three existing RDMA documents should now go forward.  I don;t believe there is any disagreement on 
> that point.

I concur that rfc5666bis and rfc5666-implementation-experience
are ready to move forward.

However, I've been thinking about an earlier comment Dave made
in this thread:

> My only concern is with regard to a possible future extension in which multiple RPC messages are carried in a single SEND (i.e., the precise opposite of message continuation).  When the language for the new field is drafted, we should make sure that it doesn't assume all the messages in a SEND are all related to a single direction of operation. 

There seem to be two related issues, and one of them has
ramifications for rpcrdma-bidirection.

I. Ambiguity of the meaning of the credit field

In the rfc5666bis world, the credit field in all messages
flowing on a connection had an unambiguous meaning. If
the message was going from requester to responder, the
credit field was a credit request. In the other direction,
it was a credit grant.

rpcrdma-bidirection introduced the idea of two separate
credit flows, which depend on whether the message was
part of forward RPC operation or backward RPC operation
on that connection.

This is the source of the problem.

The meaning of the credit field depends on a field in
the upper layer, so it's a layering violation, to say
the least.

Also, the original concept of credit was to manage RDMA
receive buffers, but rpcrdma-bidirection interpretation
of the credit value is one-credit-per-RPC.

Now if we want to add support for partial RPC message
per credit (message continuation) or multiple RPC
messages per credit (as proposed above), or NO RPC
message per credit (for RDMA2_OPTIONAL control messages)
the credit field is unusable.

Would we add an independent pool of credits for each of
these transmission mechanisms? Probably not.

A logical course of action, then, would be to alter the
rpcrdma-bidirection I-D so that forward and backward
direction operation use the same pool of credits.

The question is whether it is feasible for the two
directions to share the credit pools without deadlocking.
Some prototyping and/or careful thought would be needed
to answer it.

(Spencer, I may be changing my mind about the readiness
of rpcrdma-bidirection).

II. Ambiguity of the meaning of the XID field

Today we have a one-to-one match between the XID value
in the RPC-over-RDMA header and the XID associated with
the RPC payload in that message.

With an extension for message continuation, there is
still a one-to-one match: the RPC-over-RDMA XID can
match the XID of the partial RPC payload.

But for multiple RPC payloads per Send, or no RPC
payloads per Send (control messages), we no longer have
a match. How should the RPC-over-RDMA header's XID field
be set in those cases?

RFC 5666 and rfc5666bis have similar language regarding
how the XID field in the RPC-over-RDMA header is to be
set. Here's rfc5666bis, Section 5.2.1:

>    The
>    receiver MAY perform its processing based solely on the XID in the
>    RPC-over-RDMA header, and thereby ignore the XID in the RPC message,
>    if it so chooses.

In other words, implementations are already allowed to
rely on the value of the RPC-over-RDMA header's XID
field, and ignore the XID field in the payload.

This issue has to be addressed for rpcrdma-version-two
to support extensions that enable multiple or no RPCs
per message.

I propose that here, an independent XID space for
RDMA2_OPTIONAL messages makes sense. This would allow
receivers to match RDMA2_OPTIONAL call and reply messages,
but would not tie the header's XID value to the upper
layer payload in any way.

--
Chuck Lever

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 3 Jun 2016 16:20:13 +0000

Thanks for the discussion.  I will wait on you, Chuck, to "clear" the comment you made below on readiness.

Just to be clear, the WGLC is complete and from my point of view we have consensus but want to ensure the author is comfortable with moving forward.

-----Original Message-----
From: Chuck Lever [mailto:[email-masked]] 
Sent: Friday, June 3, 2016 8:05 AM
To: David Noveck <[email-masked]>; Spencer Shepler <[email-masked]>
Cc: NFSv4 <[email-masked]>
Subject: Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

> > The problem is that, in a multi-direction context, a given 
> > implementation only knows whether it is a client or a server.  It 
> > only knows if it is a requester or a responder based on the message 
> > it is processing at any given instant and if it can't figure that out, it is unsure whether it is a requester or responder and may be neither at particular times.
> 
> It's now been at least two weeks since last call was supposed to end 
> for the Version One RDMA documents and a week since Chuck posted any 
> necessary updates.  I'm not sure why the documents are still listed as in WGLC, but I wanted to make it clear that my comments above should not be holding these up.
> 
> cannot be fixed within the constraints of a Version One no-XDR-change 
> update.  Since the Working group has decided to do a Version One 
> no-XDR-change update, the three existing RDMA documents should now go forward.  I don;t believe there is any disagreement on that point.

I concur that rfc5666bis and rfc5666-implementation-experience are ready to move forward.

> My only concern is with regard to a possible future extension in which multiple RPC messages are carried in a single SEND (i.e., the precise opposite of message continuation).  When the language fo…

There seem to be two related issues, and one of them has ramifications for rpcrdma-bidirection.

In the rfc5666bis world, the credit field in all messages flowing on a connection had an unambiguous meaning. If the message was going from requester to responder, the credit field was a credit request. In the other direction, it was a credit grant.

rpcrdma-bidirection introduced the idea of two separate credit flows, which depend on whether the message was part of forward RPC operation or backward RPC operation on that connection.

The meaning of the credit field depends on a field in the upper layer, so it's a layering violation, to say the least.

Also, the original concept of credit was to manage RDMA receive buffers, but rpcrdma-bidirection interpretation of the credit value is one-credit-per-RPC.

Now if we want to add support for partial RPC message per credit (message continuation) or multiple RPC messages per credit (as proposed above), or NO RPC message per credit (for RDMA2_OPTIONAL control messages) the credit field is unusable.

Would we add an independent pool of credits for each of these transmission mechanisms? Probably not.

A logical course of action, then, would be to alter the rpcrdma-bidirection I-D so that forward and backward direction operation use the same pool of credits.

The question is whether it is feasible for the two directions to share the credit pools without deadlocking.
Some prototyping and/or careful thought would be needed to answer it.

(Spencer, I may be changing my mind about the readiness of rpcrdma-bidirection).

Today we have a one-to-one match between the XID value in the RPC-over-RDMA header and the XID associated with the RPC payload in that message.

With an extension for message continuation, there is still a one-to-one match: the RPC-over-RDMA XID can match the XID of the partial RPC payload.

But for multiple RPC payloads per Send, or no RPC payloads per Send (control messages), we no longer have a match. How should the RPC-over-RDMA header's XID field be set in those cases?

>    RPC-over-RDMA header, and thereby ignore the XID in the RPC message,

In other words, implementations are already allowed to rely on the value of the RPC-over-RDMA header's XID field, and ignore the XID field in the payload.

This issue has to be addressed for rpcrdma-version-two to support extensions that enable multiple or no RPCs per message.

I propose that here, an independent XID space for RDMA2_OPTIONAL messages makes sense. This would allow receivers to match RDMA2_OPTIONAL call and reply messages, but would not tie the header's XID value to the upper layer payload in any way.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 3 Jun 2016 14:31:21 -0400

> are ready to move forward.

I guess the implication is that bidirection is not ready and I'm unclear
about the reason for that.

> > My only concern is with regard to a possible future extension in which
multiple RPC messages are carried in a single SEND (i.e., the precise
opposite of message continuation).
> > When the language for the new field is drafted, we should make sure
that it doesn't assume all the messages in a SEND are all related to a
single direction of operation.

> ramifications for rpcrdma-bidirection.

A few things I'd like to note here:

   - As far as I'm concerned, I only raised a single issue.
   - My issue is only with regard to how to deal with a potential Version
   Two extension issue.
   - When I said it was my "only concern" (emphasis added), I hoped to
   indicate that, as a potential future extension, this should not hold us up
   now.
   - Although it may relate to bidirection in general, I don't think it
   affects use of it in the Version one case.

> credit field was a credit request. In the other direction,

That remains the case, even with bidirection.  Your last
two sentences are still true.

The problem is that, when bidirection is in effect, it is not
always clear who is the requester and who is the responder:
The sendr always knows but the receiver might not know and,
in case where a message is not decipherable, the question
might not be answerable.

So,

   - With only forward direction operation, If the message was going from
   client to server, the credit field is a credit request. In the other
   direction, it is a credit grant.
   - With bidirectional operation, the corresponding statement with
   requester/responder is still true, but the code in the client and server
   whenthey receive a message do not know if thety are action as requester or
   responder :-(

> This is the source of the problem.

It solved one problem, but it created another.  The question we have is
what to do about it.

> the upper layer, so it's a layering violation, to say

I agree that it is layering violation and it may even be a layering
misdemeanor.
I don't think it is a layering felony, and I hope we can avoid any
Draconian penalty.

> of the credit value is one-credit-per-RPC.

If it switched to that, I think of it as a purely verbal mistake, which is
understandable
given the context, i.e. that we are in a one-message-per-RPC environment.
In any case,
I don't see how that shift is related to bidirectional operation.

> per credit (message continuation)

I think we do.  I discussed doing that in
draft-dnoveck-nfsv4-rprcrdma-rtissuues-00.

I will discuss the credit management issues in
draft-dnoveck-nfsv4-rocrdma-rtrext-00,
but I'm not sure if I'll be able to submit that before IETF96.

> or multiple RPC
> messages per credit (as proposed above),

I haven't actually proposed that but I agree that we don't want to
foreclose this option.

> or NO RPC

I actually have proposed messages that don't carry RPC in
draft-dnoveck-nfsv4-rpcrdma-xcharext-00.

I assumed that they required a credit because receiving the message
requires a buffer and there
is no way to avoid the fact that a buffer (and this a credit) is used up.
This is true whether the credit
*field *is a request or a grant or is ignored.

> the credit field is unusable.

My formulation would be "problematic for some classes of future extensions
to the next RPC-over-RDMA
version".  That's light-years (or even megaparsecs) away from "unusable".

> these transmission mechanisms? Probably not.

I agree that that is a REAL BAD IDEA.

> direction operation use the same pool of credits.

I think this is a possible course of action.  I don't think it is
logical because:

   - The potential ambiguity you are worried about is between credit and
   grant, and having a single pool wouldn't solve that.
   - There is no ambiguity about which credit pool you are dealing with, so
   this, not being broken, doesn't require a fix.
   - There are design/supplementation issues that you note below that may
   make this infeasible

> (Spencer, I may be changing my mind about the readiness

You're definitely changing your mind.  The only question is whether that is
done an odd or an even number of times.

> II. Ambiguity of the meaning of the XID field

I think this issue has no implications for rpcrdma-bidirection.

However, you have convinced me draft-cel-nfsv4-rpcrdma-version-two is not
ready for WGLC :-)

> the RPC payload in that message.

Right.

> match the XID of the partial RPC payload.

Yes.  We are OK for all currently-proposed and those likely to be
proposed in the next several months.

> a match. How should the RPC-over-RDMA header's XID field

In general, I would leave that up the specific extensions.

> field, and ignore the XID field in the payload.

Version One implementations may do that.  It make sense,
for Version Two implementations to have the same rules when they
are using the subset of Version Two that corresponds to Version

> to support extensions that enable multiple or no RPCs

I think it should be addressed.  I don't think it has to.

> RDMA2_OPTIONAL messages makes sense.

I'm OK with that as long as individual extensions have enough
flexibility in how they use this field.

> This would allow

It is not clear to me that we necessarily have to have
RDMA2_OPTIONAL call and reply messages.  We might have a
RDMA2_OPTIONAL Message that is used to transport a set of calls or
a set of replies or a mixture of the two.   An RDMA2_OPTIONAL message
that carries a set of calls is not itself necessarily a call itself and it
doesn't require
a reply, although each of the calls it sends may require replies (which may
or
may not come via RDMA2_OPTIONAL messages).

Note that if we do this, we may wind up with two xids used to send an
NFSv4.1
COMPOUND, where the basic function of xid, is, ironically enough, taken over
by slotid/sequence pair, which just happens to do the job better.

On Fri, Jun 3, 2016 at 11:04 AM, Chuck Lever <[email-masked]> wrote:

> implementation only knows whether it is a client or
> > > a server.  It only knows if it is a requester or a responder based on
> the message it is processing at any given
> > of a Version One no-XDR-change update.  Since the Working group has
> decided to do a Version One no-XDR-change
> > update, the three existing RDMA documents should now go forward.  I
> don;t believe there is any disagreement on
> > that point.
>
> > My only concern is with regard to a possible future extension in which
> multiple RPC messages are carried in a single SEND (i.e., the precise
> opposite of message continuation).  When the language for the new field is
> drafted, we should make sure that it doesn't assume all the messages in a
> SEND are all related to a single direction of operation.
>
>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 3 Jun 2016 15:34:53 -0400

> On Jun 3, 2016, at 2:31 PM, David Noveck <[email-masked]> wrote:
> 
> I guess the implication is that bidirection is not ready and I'm unclear about the reason for that.
> 
> > > My only concern is with regard to a possible future extension in which multiple RPC messages are carried in a single SEND (i.e., the precise opposite of message continuation).  
> > > When the language for the new field is drafted, we should make sure that it doesn't assume all the messages in a SEND are all related to a single direction of operation.
> 
> 	• As far as I'm concerned, I only raised a single issue.
> 	• My issue is only with regard to how to deal with a potential Version Two extension issue.
> 	• When I said it was my "only concern" (emphasis added), I hoped to indicate that, as a potential future extension, this should not hold us up now.
> 	• Although it may relate to bidirection in general, I don't think it affects use of it in the Version one case. 

I. relates to the Version One case because we may want
to rewrite parts of the rpcrdma-bidirection I-D, which
has a direct impact on NFSv4.1 on RPC-over-RDMA V1.

And recall that we stated earlier that rpcrdma-bidirection
was not meant to apply solely to V1. The current V2
specification requires support for RPC bidirection as
specified in rpcrdma-bidirection.

> 	• With only forward direction operation, If the message was going from client to server, the credit field is a credit request. In the other direction, it is a credit grant.
> 	• With bidirectional operation, the corresponding statement with requester/responder is still true, but the code in the client and server whenthey receive a message do not know if thety are action as requester or responder :-(

There may be problematic language in rfc5666bis as well.

RFC 5666 Section 3.3 attaches credit requests to RPC Call
messages and credit grants to RPC Reply messages.

The updated language in rfc5666bis has this: credit
request and grant is described in Sections 4.3.1 and 5.2.3
as being tied to requester and responder.

So, this is a problem not only for bidirectional operation
but also for any type of message where there are multiple
or no RPC messages associated with the RPC-over-RDMA
message.

My proposal is this:

The roles of client (active connector) and server (passive
accepter) are always the same for a given connection, and
are independent of direction of RPC operation.

Thus there is never any ambiguity about what the credit
field means: the active side always makes requests, and
the passive side always makes grants.

> It solved one problem, but it created another.  The question we have is

Agreed.

> I agree that it is layering violation and it may even be a layering misdemeanor.
> If it switched to that, I think of it as a purely verbal mistake, which is understandable 
> given the context, i.e. that we are in a one-message-per-RPC environment.  In any case,

Given that the one-credit-per-RPC language appears as
early as RFC 5666, perhaps you are correct.

> I think we do.  I discussed doing that in draft-dnoveck-nfsv4-rprcrdma-rtissuues-00.
> 
> I will discuss the credit management issues in draft-dnoveck-nfsv4-rocrdma-rtrext-00,
> I haven't actually proposed that but I agree that we don't want to foreclose this option.
> 
> > or NO RPC
> I actually have proposed messages that don't carry RPC in draft-dnoveck-nfsv4-rpcrdma-xcharext-00.

Sorry, the above "if" was a rhetorical "if". I do
presume that all three proposals are interesting to
pursue.

> I assumed that they required a credit because receiving the message requires a buffer and there
> is no way to avoid the fact that a buffer (and this a credit) is used up.  This is true whether the credit 
> field is a request or a grant or is ignored.
> 
> My formulation would be "problematic for some classes of future extensions to the next RPC-over-RDMA

Alright, then, "unreliable" might be closer to my
intended meaning.

> 	• The potential ambiguity you are worried about is between credit and grant, and having a single pool wouldn't solve that.
> 	• There is no ambiguity about which credit pool you are dealing with, so this, not being broken, doesn't require a fix.

I disagree with that. Having a single credit flow, with
fixed roles of which side sends a request, and which
sends a grant, solves the problem simply and fully, for
all message types.

> 	• There are design/supplementation issues that you note below that may make this infeasible
> 

It is also possible that this issue could be resolved
with language in rpcrdma-version-two, and allow the
one-credit-per-RPC concept to linger in V1 only.

I think that would mean rpcrdma-bidirection is turning
into a V1-only enhancement. Bidirection would have to
be partially or fully specified again in
rpcrdma-version-two.

Having multiple different ways to do backward direction
operation is onerous for implementations.

So I'd prefer to have this addressed in rfc5666bis and
rpcrdma-bidirection if we can muster it.

> I think this issue has no implications for rpcrdma-bidirection.

I agree. We can set this one aside for the moment.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sat, 4 Jun 2016 06:35:28 -0400

> Alright, then, "unreliable" might be closer to my

I don't see any unreliability either.  I think we have
something which is difficult to generalize, because,
as you point out, it has an unfortunate layering issue,
which we are unable to fix in Version One because it
is embedded in the Version One XDR.  We can fix it in subsequent versions
where we can change the XDR but we have to take extra care because a lot of
the layering confusion is located in the four words that we have defined as
not subject to change :-(

> sends a grant, solves the problem simply and fully, for

It solves the ambiguity problem but it creates a bunch of
new ones.  I image that the first is interoperability with existing
implementations, who I assume are using the model in rpcrdma-bidirection as
it is today.

I don't see how one can implement bidirectional operation with a single
credit pool.  The reason for credits is to ensure that there are sufficient
preposted receives to correspond to sends that the peer makes.  Since there
is a set of receives on each peer node then there needs to be a
corresponding credit pool for each direction.  I don't see how you can have
a situation in which both client and server may be requesters
and have them share a single credit pool.

> one-credit-per-RPC concept to linger in V1 only.

Or alternatively, it could apply to both V1 and those
parts of V2 that use the existing V1 message types.

> into a V1-only enhancement. Bidirection would have
> to be partially or fully specified again in rpcrdma-
> version-two.

I don't think you need to do that, if you structure/layer things properly
(see below for an approach).  Then any
further specification of bidirection for version two would  only address
how the new classes of features we want to enable in version two would
interact with the existing bidirectional operation.  In doing that we could
use our
freedom to define the needed new XDR to avoid propagating the layering
issue that you have identified.

> Having multiple different ways to do backward
> direction operation is onerous for implementations.

Agree.

> rpcrdma-bidirection if we can muster it.

So here's the proposed division of responsibility:

   - The responsibility for defining the basic structure of bidirectional
   operation belongs to rpcrdma-bidirection.  This includes the use of two
   separate credit pools and the fact each message may contain either a
   request or a grant applying to the credit pool owned by the message
   receiver.
   - The responsibility for defining how it is to be determined, whether
   any particular message has a grant or a request in the credit field is the
   responsibility of the specific RPC-over-RDMA version being used.
   - The responsibility of deciding whether bidirectional operation is
   OPTIONAL, REQUIRED, or not allowed  is the responsibility of the specific
   RPC-over-RDMA version being used.
   - The text that now defines how the credit field for specfic Version One
   messages is interpreted as illustrative of version one only.
   - draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the
   responsibility for defining this for Version Two.  We have some  issues to
   resolve which I'll address in a later message.

On Fri, Jun 3, 2016 at 3:34 PM, Chuck Lever <[email-masked]> wrote:

> > > > My only concern is with regard to a possible future extension in
> which multiple RPC messages are carried in a single SEND (i.e., the precise
> >       • When I said it was my "only concern" (emphasis added), I hoped
> to indicate that, as a potential future extension, this should not hold us
> up now.
> >       • Although it may relate to bidirection in general, I don't think
> it affects use of it in the Version one case.
>
> >       • With only forward direction operation, If the message was going
> from client to server, the credit field is a credit request. In the other
> >       • With bidirectional operation, the corresponding statement with
> >       • The potential ambiguity you are worried about is between credit
> and grant, and having a single pool wouldn't solve that.
> >       • There is no ambiguity about which credit pool you are dealing
> with, so this, not being broken, doesn't require a fix.
>

---

From: Chuck Lever <[email-masked]>
Date: Mon, 6 Jun 2016 11:17:35 -0400

> On Jun 4, 2016, at 6:35 AM, David Noveck <[email-masked]> wrote:
> 
> is embedded in the Version One XDR.  We can fix it in subsequent versions where we can change the XDR but we have to take extra care because a lot of the layering confusion is located in the four words that we have defined as not subject to change :-(
> 
> new ones.  I image that the first is interoperability with existing implementations, who I assume are using the model in rpcrdma-bidirection as it is today.
> 
> I don't see how one can implement bidirectional operation with a single credit pool.  The reason for credits is to ensure that there are sufficient preposted receives to correspond to sends that the peer makes.  Since there is a set of receives on each peer node then there needs to be a corresponding credit pool for each direction.  I don't see how you can have a situation in which both client and server may be requesters

Fair enough. One set of credits for each responder on
a connection seems to align with what's already in the
documents, and discourages the addition of credit
flows for new message types.

> I don't think you need to do that, if you structure/layer things properly (see below for an approach).  Then any 
> further specification of bidirection for version two would  only address how the new classes of features we want to enable in version two would interact with the existing bidirectional operation.  In doing that we could use our 
> freedom to define the needed new XDR to avoid propagating the layering issue that you have identified.

I think you're suggesting that we have a problem only
for RDMA2_OPTIONAL, and that can be dealt with in
rpcrdma-version-two?

> 	• The responsibility for defining the basic structure of bidirectional operation belongs to rpcrdma-bidirection.  This includes the use of two separate credit pools and the fact each message may contain either a request or a grant applying to the credit pool owned by the message receiver.

OK: the receiver has to decide whether each message
contains a credit request (the receiver is acting as
a responder) or a credit grant (the receiver is
acting as a requester).

I might refine the text of rpcrdma-bidirection in this
area, but otherwise I don't think there's a significant
change needed.

> 	• The responsibility for defining how it is to be determined, whether any particular message has a grant or a request in the credit field is the responsibility of the specific RPC-over-RDMA version being used.

I might add this suggestion to rpcrdma-bidrection.

> 	• The responsibility of deciding whether bidirectional operation is OPTIONAL, REQUIRED, or not allowed is the responsibility of the specific RPC-over-RDMA version being used.

I'm uncertain how that aligns with rfc5666bis, which
does not mention backwards direction operation at all.
This implies that it is allowed, and not REQUIRED. Is
that enough?

> 	• The text that now defines how the credit field for specfic Version One messages is interpreted as illustrative of version one only.

Is a text change needed for this, or are you simply
stating an operating assumption for spec writers?

> 	• draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the responsibility for defining this for Version Two.  We have some issues to resolve which I'll address in a later message.

Not sure what "this" is, and not sure we want version
two anything in rpcrdma-bidirection. Awaiting your
later message!

Thanks for helping me with this. Let's try to close on
what is needed for rpcrdma-bidirection and try to get
a fresh revision by the end of this week.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 6 Jun 2016 13:10:36 -0400

> for RDMA2_OPTIONAL, and that can be dealt with in

That was what I was suggesting, but I just read rpcdama-version-two-01
and so I think "can be dealt with in rpcrdma-version-two" can be replaced by
"has been dealt with in rpcrdma-version-two" :-)

>> The responsibility for defining the basic structure of bidirectional
operation belongs to rpcrdma-bidirection.
>> This includes the use of two separate credit pools and the fact each
message may contain either a request or a grant applying to the credit pool
owned by the message receiver.

> a responder) or a credit grant (the receiver is

Yes but I'd boil this down to the receiver deciding whether the credit
field is a request or a grant (and the
RPC-over-RDMA version telling him how to decide.

The place where this gets "interesting" is a subcase of the feature you
brought up.  If a transmission can
contain multiple rpc messages, it can be the case the sender would want to
send both requests and replies
and thus want/need to send both a request and a grant.

I think we want to be careful that, where we specify how the credit field
is to be used, we don't foreclose
passing credits or grants in other fields.    If an XMOPT_GROUP were
defined, that allowed a group of
RPC messages to be sent, the version two sender would make the direction
field reflect the majority, but
I would want it to have the option to have an other-credit field so it
could convey a request and a grant in
the same transmission.

> area, but otherwise I don't think there's a significant

there might be minor text tweaking required.

> >       • The responsibility for defining how it is to be determined,
whether any particular message has a grant or a request in the credit field
is the responsibility of the specific RPC-over-RDMA version being used.

> I might add this suggestion to rpcrdma-bidrection.

OK, bu so far I hven't seen any obvious place to put it.

> >       • The responsibility of deciding whether bidirectional operation
is OPTIONAL, REQUIRED, or not allowed is the responsibility of the specific
RPC-over-RDMA version being used.

> This implies that it is allowed, and not REQUIRED. Is

I think what is in 5666bis is fine.  Section 8.1 introduces it as a
potential conventional extension and references rpcrdma-bidirection.

>>       • The text that now defines how the credit field for specfic
Version One messages is interpreted as illustrative of version one only.

> stating an operating assumption for spec writers?

I was thinking of rewriting the last paragraph of section 4.1 of
rpcrdma-bidirection to read as follows:

When message direction is not fully determined by context or by an
accompanying RPC message with a call direction field, it may not be
clear whether the header credit value is a request or grant.  In such
cases the definition of the RPC-over-RDMA version used may provide
guidance as to how the credit value is to be interpreted. In such
cases in which such guidance is not provided or is adequate, the
receiver MUST NOT use the header's credit value.

I didn't see anything else that need to change.

>>       • draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the

> two anything in rpcrdma-bidirection. Awaiting your

The first point is that I meant to type
draft-ietf-nfsv4-rpcrdma-version-two. :-(

The issue I was referring to was dealing with cases in which deciding
whether the credit was a request or a grant was hard either:

   - The message in question is, strictly speaking, neither an RPC request
   or a reply.  I think that is now dealt with in -01

You now allow the sender to decide whether the credit is a request or a
grant and the extension is free to tell him

how to choose.

   - The transmission might contain both requests and replies as discussed
   above.

I think that can be resolved in the next iteration of
draft-cel-nfsv4-rpcrdma-version-two.

On Mon, Jun 6, 2016 at 11:17 AM, Chuck Lever <[email-masked]> wrote:

> > is embedded in the Version One XDR.  We can fix it in subsequent
> versions where we can change the XDR but we have to take extra care because
> a lot of the layering confusion is located in the four words that we have
> defined as not subject to change :-(
> >
> > I don't think you need to do that, if you structure/layer things
> properly (see below for an approach).  Then any
> >       • The responsibility for defining the basic structure of
> bidirectional operation belongs to rpcrdma-bidirection.  This includes the
> use of two separate credit pools and the fact each message may contain
> either a request or a grant applying to the credit pool owned by the
> message receiver.
>
> >       • The responsibility for defining how it is to be determined,
> whether any particular message has a grant or a request in the credit field
> is the responsibility of the specific RPC-over-RDMA version being used.
>
> >       • The responsibility of deciding whether bidirectional operation
> is OPTIONAL, REQUIRED, or not allowed is the responsibility of the specific
> >       • The text that now defines how the credit field for specfic
> Version One messages is interpreted as illustrative of version one only.
>
> >       • draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the

---

From: Chuck Lever <[email-masked]>
Date: Wed, 8 Jun 2016 16:05:13 -0400

> On Jun 6, 2016, at 1:10 PM, David Noveck <[email-masked]> wrote:
> 
> >> The responsibility for defining the basic structure of bidirectional operation belongs to rpcrdma-bidirection.  
> >> This includes the use of two separate credit pools and the fact each message may contain either a request or a grant applying to the credit pool owned by the message receiver.
> 
> Yes but I'd boil this down to the receiver deciding whether the credit field is a request or a grant (and the
> The place where this gets "interesting" is a subcase of the feature you brought up.  If a transmission can
> contain multiple rpc messages, it can be the case the sender would want to send both requests and replies
> I think we want to be careful that, where we specify how the credit field is to be used, we don't foreclose
> passing credits or grants in other fields.    If an XMOPT_GROUP were defined, that allowed a group of
> RPC messages to be sent, the version two sender would make the direction field reflect the majority, but
> I would want it to have the option to have an other-credit field so it could convey a request and a grant in
> the same transmission. 
> 
> I was thinking of rewriting the last paragraph of section 4.1 of rpcrdma-bidirection to read as follows:
> 
> When message direction is not fully determined by context or by an accompanying RPC message with a call direction field, it may not be clear whether the header credit value is a request or grant.  In such cases the definition of the RPC-over-RDMA version used may provide guidance as to how the credit value is to be interpreted. In such cases in which such guidance is not provided or is adequate, the receiver MUST NOT use the header's credit value.

OK, here's how rpcrdma-bidirection Section 4.1 reads now:

> 4.1.  Backward Credits
> 
>    RPC-over-RDMA credits work the same way in the backward direction as
>    they do in the forward direction.  However, forward direction credits
>    and backward direction credits on the same connection are accounted
>    separately.
> 
>    The forward direction credit value retains the same meaning whether
>    or not there are backward direction resources associated with an RPC-
>    over-RDMA transport connection.  This is the number of RPC requests
>    the forward direction responder (the RPC server) is prepared to
>    receive concurrently.
> 
>    The backward direction credit value is the number of RPC requests the
>    backward direction responder (the RPC client) is prepared to receive
>    concurrently.  Observe that the backward direction credit value MAY
>    be different than the forward direction credit value.
> 
>    During bi-directional operation, each receiver has to decide whether
>    an incoming message contains a credit request (the receiver is acting
>    as a responder) or a credit grant (the receiver is acting as a
>    requester) and apply the credit value accordingly.
> 
>    When message direction is not fully determined by context (e.g.,
>    suggested by the definition of the RPC-over-RDMA version that is in
>    use) or by an accompanying RPC message payload with a call direction
>    field, it is not possible for the receiver to tell with certainty
>    whether the header credit value is a request or grant.  In such
>    cases, the receiver MUST NOT use the header's credit value.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 8 Jun 2016 21:31:38 -0400

Looks fine to me.

On Wed, Jun 8, 2016 at 4:05 PM, Chuck Lever <[email-masked]> wrote:

> > > >       • The responsibility of deciding whether bidirectional
> operation is OPTIONAL, REQUIRED, or not allowed is the responsibility of
> the specific RPC-over-RDMA version being used.
> >
> accompanying RPC message with a call direction field, it may not be clear
> whether the header credit value is a request or grant.  In such cases the
> definition of the RPC-over-RDMA version used may provide guidance as to how
> the credit value is to be interpreted. In such cases in which such guidance
> is not provided or is adequate, the receiver MUST NOT use the header's
> credit value.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 9 Jun 2016 14:14:33 -0400

Hi Spencer-

Let's proceed with revision 05. I'm clearing my recent comment
about the readiness of rpcrdma-bidirection.

> On Jun 3, 2016, at 12:20 PM, Spencer Shepler <[email-masked]> wrote:
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 10 Jun 2016 01:02:45 +0000

Thanks, Chuck.  Will do.

-----Original Message-----
From: Chuck Lever [mailto:[email-masked]] 
Sent: Thursday, June 9, 2016 11:15 AM
To: Spencer Shepler <[email-masked]>
Cc: David Noveck <[email-masked]>; NFSv4 <[email-masked]>
Subject: Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

Hi Spencer-

Let's proceed with revision 05. I'm clearing my recent comment about the readiness of rpcrdma-bidirection.

>> Those comments refer to a problem that is inherent in Version One  
>> and cannot be fixed within the constraints of a Version One 
>> no-XDR-change update.  Since the Working group has decided to do a 
>> Version One no-XDR-change update, the three existing RDMA documents should now go forward.  I don;t believe there is any disagreement on that point.
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.i
> etf.org%2fmailman%2flistinfo%2fnfsv4&data=01%7c01%7csshepler%40microso
> ft.com%7cc7fb211893674020da6d08d39091ec90%7c72f988bf86f141af91ab2d7cd0
> 11db47%7c1&sdata=KiMVRSXc2Sj8%2blfR0vjWLJFQ4KAYKx95%2bMkReAuNQjA%3d

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 16 Jun 2016 16:13:34 -0400

Since Chuck has cleared his comment about bidirection and submitted -05, I
think it is
time to decide that WGLC has concluded (successfully) for
draft-ietf-nfsv4-rfc5666bis
draft-ietf-nfsv4-rfc5666-implementation, and
draft-ietf-nfsv4-rpcrdma-bidirection.

These should be able to move forward.

On Thu, Jun 9, 2016 at 9:02 PM, Spencer Shepler <[email-masked]>
wrote:

> > Just to be clear, the WGLC is complete and from my point of view we have
> consensus but want to ensure the author is comfortable with moving forward.
> >
> >>> it is processing at any given instant and if it can't figure that out,
> it is unsure whether it is a requester or responder and may be neither at
> particular times.
> >>
> >> necessary updates.  I'm not sure why the documents are still listed as
> in WGLC, but I wanted to make it clear that my comments above should not be
> holding these up.
> >>
> >> Version One no-XDR-change update, the three existing RDMA documents
> should now go forward.  I don;t believe there is any disagreement on that
> point.
> >
> > In the rfc5666bis world, the credit field in all messages flowing on a
> connection had an unambiguous meaning. If the message was going from
> requester to responder, the credit field was a credit request. In the other
> direction, it was a credit grant.
> >
> > rpcrdma-bidirection introduced the idea of two separate credit flows,
> which depend on whether the message was part of forward RPC operation or
> backward RPC operation on that connection.
> >
> > The meaning of the credit field depends on a field in the upper layer,
> so it's a layering violation, to say the least.
> >
> > Also, the original concept of credit was to manage RDMA receive buffers,
> but rpcrdma-bidirection interpretation of the credit value is
> one-credit-per-RPC.
> >
> > Now if we want to add support for partial RPC message per credit
> (message continuation) or multiple RPC messages per credit (as proposed
> above), or NO RPC message per credit (for RDMA2_OPTIONAL control messages)
> rpcrdma-bidirection I-D so that forward and backward direction operation
> use the same pool of credits.
> >
> > Today we have a one-to-one match between the XID value in the
> RPC-over-RDMA header and the XID associated with the RPC payload in that
> message.
> >
> > With an extension for message continuation, there is still a one-to-one
> match: the RPC-over-RDMA XID can match the XID of the partial RPC payload.
> >
> > But for multiple RPC payloads per Send, or no RPC payloads per Send
> (control messages), we no longer have a match. How should the RPC-over-RDMA
> header's XID field be set in those cases?
> >
> > RFC 5666 and rfc5666bis have similar language regarding how the XID
> field in the RPC-over-RDMA header is to be set. Here's rfc5666bis, Section
> 5.2.1:
> >
> >>   The
> > In other words, implementations are already allowed to rely on the value
> of the RPC-over-RDMA header's XID field, and ignore the XID field in the
> payload.
> >
> > This issue has to be addressed for rpcrdma-version-two to support
> extensions that enable multiple or no RPCs per message.
> >
> > I propose that here, an independent XID space for RDMA2_OPTIONAL
> messages makes sense. This would allow receivers to match RDMA2_OPTIONAL
> call and reply messages, but would not tie the header's XID value to the
> upper layer payload in any way.
> >
> >
> > --
> > Chuck Lever
> >
> >

---

From: David Noveck <[email-masked]>
Date: Wed, 29 Jun 2016 14:22:40 -0400

> Thanks, Chuck.  Will do.

Given the above, is there any reason that all of these documents are still
listed as "In WG Last Call"?.

I think we need to record the (successful) completion of WGLC before the
holiday weekend intervenes,
and then proceed to move these documents forward.   Chuck has already done
the hard part.  We need
to turn the administrative crank to get this work published. The whole
"Work in Progress" thing is getting
kind of old :-(

On Thu, Jun 16, 2016 at 4:13 PM, David Noveck <[email-masked]> wrote:

>> > Just to be clear, the WGLC is complete and from my point of view we
>> have consensus but want to ensure the author is comfortable with moving
>> forward.
>> >
>> >>> it is processing at any given instant and if it can't figure that
>> out, it is unsure whether it is a requester or responder and may be neither
>> at particular times.
>> >>
>> > Also, the original concept of credit was to manage RDMA receive
>> buffers, but rpcrdma-bidirection interpretation of the credit value is
>> one-credit-per-RPC.
>> >
>> > In other words, implementations are already allowed to rely on the
>> value of the RPC-over-RDMA header's XID field, and ignore the XID field in
>> the payload.
>> >