Subject: Re: [nfsv4] Credit management and one-way messages

From: Chuck Lever <[email-masked]>
Date: Tue, 1 Aug 2017 11:49:54 -0400

> On Jul 31, 2017, at 2:47 PM, David Noveck <[email-masked]> wrote:
> 
> One issue that has been mentioned as a possible impediment to advnacing RPC-over-RDMA to be a working group document, in its current form, concerns its use of one-way messages to convey transport properties.  These concerns seem to arise from the presentation of credit managemt as a "request-grant protocol".  If the credit logic in fact depended on that pairing, then there would be no roon for one-way meesages.
> 
> In fact, it doesn't. If you look at the actual operation of credit management, one sees a different picture from what is stated in the introductory paragraph of Section 3.1.1 of RFC 8166:
> 
> Flow control for RDMA Send operations directed to the Responder is implemented as a simple request/grant protocol in the RPC-over-RDMA header associated with each RPC message.
> 
> what follows focuses on the role of the grant. giving extensive information but how it is computed and nd why it is vsli for the requester to use it.
> 
> While the text says "Practically speaking, the critical value is the granted value", it is not clear what the exact role of the request is.  It aapears to be a hintwhich the receiver is under no obligation to take any notice of.  So it appears that the presentation of the creddit management approach as a request-grant protocol., is a reflection of the fact that, when it is used to transport RPC Requests and Replies, this pairing always exists.  When grants are sent with one-way messages, there is no problem that arises from the fact that there is no corresponding credit request.  The sender simply informs the receiver about his own receive resources and thus his ability to accept further sends.

The request-grant protocol sets an upper bound for the number of
messages that can be in flight at once. You are addressing the
narrow issue of how a receiver should interpret the value in the
rdma_credits field to set the negotiated credit limit.

The purpose of this limit is to prevent a sender from transmitting
more messages than the receiver has available receive buffers.

For example, a requester cannot RPC Call without first ensuring
there is a receive buffer available to catch the RPC Reply. A
responder cannot send an RPC Reply without first ensuring there
are an appropriate number of receive buffers ready for the granted
number of credits, minus the number of RPC transactions it is
currently processing. A requester computes the number of remote
receive buffers that are available, and thus how many more RPC
Calls it can send before waiting, by observing the number of
Replies it has received.

In other words, the credit limit is _enforced_ via the two-way
interchange. The underlying and more significant issue is therefore
that, when unidirectional messages are introduced, neither side can
properly compute the number of new messages that may be sent
relative to the negotiated upper bound.

Another header field could be introduced that contains the exact
number of receive buffers available on the sender when that message
was sent. Or such a field could replace rdma_credits. Either would
be another step away from general compatibility with RPC-over-RDMA
version 1.

> There is one potential issue connected with use of one-way messages to be addressed.  If the recever has N credits and then N one-way messages are sent without any traffic in the opposite direction, then it is possble for a deadlock to result, since there would be no way for the sender to find out about the receive resources.  For most one-way messages, this is not a problem, since many one-way messages naturally give rise to messages in the opposite direction even if the relationship is not formalized witin an RPC paradigm.  For example:
> 	• The RDMA2_CONNPROP sent by the client to the server is paired with an RDMA2_CONNPROP in the opposite direction.
> 	• An RDMA2_REQPROP results in RDMA2_RESPROP send in response.
> RDMA2_UPDPROP is an exception.  In the unlikely event that there are a a lage series of such messages sent in one irection while there are no RPC's being sent to the receiver it is possible for a ealock to arise.  This would result in a situation in which the receiver would have to send some message back in the other to provide a credit grant.   The most likely way to to that is for it to choose to send an RDMA2_UPDPROP with an empty property set in the reverse direction.
> 
> One other problem with one-way messages as they stand in rpcrdma-version-two concerns section 7.2.2 in which the following item in the bullet list is poblematic:
> 	• When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC message payload is present, a Requester MUST set the value of the rdma_optdir field to CALL, and a Responder MUST set the value of the rdma_optdir field to REPLY.  The Requester chooses a value for the rdma_xid field from the XID space that matches the message's direction.  Requesters and Responders set the rdma_credit field in a similar fashion: a value is set that is appropriate for the direction of the message.
> This cannot be acted on, because, in the context of a one-way message, it is not clear which party is the requester and which is the responder.  While the roles of client and server are fixed and clear, the roles of requester and responder vary from RPC to RPC.  If you are not in an RPC context, then any decision as to who is the requestor or responder is arbitrary.
> I think the best way to address these issues is for rpcrdma-version-two would be to:
> 	• Provide an explanation of credit management not so tied to the RPC paradigm.

That is appropriate to explore, but is much easier said than
done.

But first, we need to define what a unidirectional message is,
and even that's harder than it looks. Here are some examples.

 + An RPC Call that does not expect a Reply is unidirectional.

 + A one-way control message is unidirectional.

 + An RPC Call that is dropped or ignored by a responder is
   unidirectional.

 + An RPC retransmission without a connection break makes
   either the original or the retransmitted message
   unidirectional.

 + An RPC Call becomes unidirectional when the matching RPC
   Reply is lost for some reason.

 + RDMA_DONE would be unidirectional. (RDMA_DONE is proposed
   as a mechanism to manage Read chunks in RPC Replies in
   draft-cel-nfsv4-rpcrdma-reliable-reply-00).

 + An extension might introduce some new form of unidirectional
   message if the problematic text in section 7.2.2 is modified
   or removed.

These are some of the ways a requester or responder can lose
synchronization of outstanding credits.

> 	• Add a new value for the direction field for one-way messages

In some of the above cases, the sender does not know that a
unidirectional message is being sent, and thus cannot properly
set the direction field. 

> 	• Provide that one-way messages always contain a credit grant rather than a credit request

It doesn't make sense to me to send a message from a requester to
a responder with a "grant" credit value. What value would the
requester put in the rdma_credits field in the case where the
responder and requester have a different number of receive
buffers available?

It might be stronger to have a special value for either the
direction or the rdma_credits field which means "the value in the
rdma_credits field should be ignored".

But see above: sometimes a sender cannot know in advance whether
a message is unidirectional.

> 	• Explain how the potential deadlock with RDMA2_UPDPROP can be avoided.

In addition, to handle cases where a message becomes unidirectional
after it is sent, RPC-over-RDMA needs a reliable and non-destructive
mechanism for resynchronizing the number of outstanding credits at
the sender and receiver.

Today that mechanism is to drop and re-establish a connection (which
is 100% reliable but is not non-destructive).

We are introducing mechanisms that can build up state associated
with a connection (using unidirectional messages, possibly). That
state is lost and must be re-established if the connection is
dropped. Not to mention all the outstanding RPCs that have to be
retransmitted.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 1 Aug 2017 13:51:14 -0400

*Ouch!!*

If i was addressing an unduly narrow problem, it was because it was framed
around the difficulties that might be added by adding one-way messages such
as those added by Version 2.  If you consider, as I guess makes sense, RPCs
that are not replied to as effectively one-way messages, that puts an
entirely different spin (of possibly high angular momentum) on the problem.

Let me try to start addressing this larger problem.  I hope it doesn't make
anyone (else) dizzy.

One reason that this is hard to address is that we have different views
regarding the roles of the credit management logic regarding replies.  In
my view, there is no point in a requester trying to give a credit to the
responder allowing the responder to send the response to the requester.  I
thought that the right thing was for the requester to ensure that the
responders response could be read was for the requester to post a receive
(or ensure that one was present) before sending the request.  I though that
this happened outside the ambit of credit management because there is no
real need for it to be within the scope of credit  management.  The
requester can be responsible for providing receives corresponding with
in-flight requests without involving the responder in the decision, which
involves complexity, especially when multipeRPC directions are involved.  I
never found anythng in the spec that told me I was wrong in that, but i
just may be missing it.

I think we discussed this a while ago but no change in the spec was made to
either make it clear I was wrong, or indicate that I was right.  My
recollection is that you treated my observation as correct/valid
implementation advice and we never really addressed our different
interpretations of the spec.  It seems to me that the spec was ambiguous on
this point.   Of course, I might not be seeing something that is there and
you might be simply assuming that something is implied because it seems
natural, even if it is not explcitly there.  If the spec still is not clear
on this point, it might need to be fixed, if not to mandate my approach but
at least to indicate how the requester can safeguard itself against the
possibility that a response sent by the responder will cause a disconnect.

With this approach, RPC's not replied to would still be a problem, but the
problem cannot result in a disconnect.  For each not-replied-to request,
there would be a receive that was not included in the credits, which would
be devoted in this model to receives for receiving requests (and things
intended to be one-way messages).   Of course, with 1K or 4K buffers that
would not be such a big deal and you could defer any connection reset until
a time when it was easy to do.

On Tue, Aug 1, 2017 at 11:49 AM, Chuck Lever <[email-masked]> wrote:

> > One issue that has been mentioned as a possible impediment to advnacing
> RPC-over-RDMA to be a working group document, in its current form, concerns
> its use of one-way messages to convey transport properties.  These concerns
> seem to arise from the presentation of credit managemt as a "request-grant
> protocol".  If the credit logic in fact depended on that pairing, then
> there would be no roon for one-way meesages.
> >
> > In fact, it doesn't. If you look at the actual operation of credit
> management, one sees a different picture from what is stated in the
> introductory paragraph of Section 3.1.1 of RFC 8166:
> >
> > Flow control for RDMA Send operations directed to the Responder is
> implemented as a simple request/grant protocol in the RPC-over-RDMA header
> associated with each RPC message.
> >
> > what follows focuses on the role of the grant. giving extensive
> information but how it is computed and nd why it is vsli for the requester
> to use it.
> >
> > While the text says "Practically speaking, the critical value is the
> granted value", it is not clear what the exact role of the request is.  It
> aapears to be a hintwhich the receiver is under no obligation to take any
> notice of.  So it appears that the presentation of the creddit management
> approach as a request-grant protocol., is a reflection of the fact that,
> when it is used to transport RPC Requests and Replies, this pairing always
> exists.  When grants are sent with one-way messages, there is no problem
> that arises from the fact that there is no corresponding credit request.
> The sender simply informs the receiver about his own receive resources and
> thus his ability to accept further sends.
>
> > There is one potential issue connected with use of one-way messages to
> be addressed.  If the recever has N credits and then N one-way messages are
> sent without any traffic in the opposite direction, then it is possble for
> a deadlock to result, since there would be no way for the sender to find
> out about the receive resources.  For most one-way messages, this is not a
> problem, since many one-way messages naturally give rise to messages in the
> opposite direction even if the relationship is not formalized witin an RPC
> paradigm.  For example:
> >       • The RDMA2_CONNPROP sent by the client to the server is paired
> with an RDMA2_CONNPROP in the opposite direction.
> > RDMA2_UPDPROP is an exception.  In the unlikely event that there are a a
> lage series of such messages sent in one irection while there are no RPC's
> being sent to the receiver it is possible for a ealock to arise.  This
> would result in a situation in which the receiver would have to send some
> message back in the other to provide a credit grant.   The most likely way
> to to that is for it to choose to send an RDMA2_UPDPROP with an empty
> property set in the reverse direction.
> >
> > One other problem with one-way messages as they stand in
> rpcrdma-version-two concerns section 7.2.2 in which the following item in
> the bullet list is poblematic:
> >       • When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC
> message payload is present, a Requester MUST set the value of the
> rdma_optdir field to CALL, and a Responder MUST set the value of the
> rdma_optdir field to REPLY.  The Requester chooses a value for the rdma_xid
> field from the XID space that matches the message's direction.  Requesters
> and Responders set the rdma_credit field in a similar fashion: a value is
> set that is appropriate for the direction of the message.
> > This cannot be acted on, because, in the context of a one-way message,
> it is not clear which party is the requester and which is the responder.
> While the roles of client and server are fixed and clear, the roles of
> requester and responder vary from RPC to RPC.  If you are not in an RPC
> context, then any decision as to who is the requestor or responder is
> arbitrary.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 7 Aug 2017 17:46:20 -0400

> On Aug 1, 2017, at 1:51 PM, David Noveck <[email-masked]> wrote:
> 
> Ouch!!
> 
> If i was addressing an unduly narrow problem, it was because it was framed around the difficulties that might be added by adding one-way messages such as those added by Version 2.  If you consider, as I guess makes sense, RPCs that are not replied to as effectively one-way messages, that puts an entirely different spin (of possibly high angular momentum) on the problem.
> 
> Let me try to start addressing this larger problem.  I hope it doesn't make anyone (else) dizzy.
> 
> One reason that this is hard to address is that we have different views regarding the roles of the credit management logic regarding replies.  In my view, there is no point in a requester trying to give a credit to the responder allowing the responder to send the response to the requester.  I thought that the right thing was for the requester to ensure that the responders response could be read was for the requester to post a receive (or ensure that one was present) before sending the request.  I though that this happened outside the ambit of credit management because there is no real need for it to be within the scope of credit  management.  The requester can be responsible for providing receives corresponding with in-flight requests without involving the responder in the decision, which involves complexity, especially when multipeRPC directions are involved.  I never found anythng in the spec that told me I was wrong in that, but i just may be missing it.
> 
> I think we discussed this a while ago but no change in the spec was made to either make it clear I was wrong, or indicate that I was right.  My recollection is that you treated my observation as correct/valid implementation advice and we never really addressed our different interpretations of the spec.  It seems to me that the spec was ambiguous on this point.   Of course, I might not be seeing something that is there and you might be simply assuming that something is implied because it seems natural, even if it is not explcitly there.

Another interpretation is that the spec describes only protocol,
and that after some trial-and-error, implementers will conclude
that there are only one or two ways to implement that protocol.

> If the spec still is not clear on this point, it might need to be fixed, if not to mandate my approach but at least to indicate how the requester can safeguard itself against the possibility that a response sent by the responder will cause a disconnect.
> 
> With this approach, RPC's not replied to would still be a problem, but the problem cannot result in a disconnect.  For each not-replied-to request, there would be a receive that was not included in the credits, which would be devoted in this model to receives for receiving requests (and things intended to be one-way messages).   Of course, with 1K or 4K buffers that would not be such a big deal and you could defer any connection reset until a time when it was easy to do.  

A requester is responsible for posting enough receive buffers to
catch Replies for all outstanding RPC Calls. So either:

A. It can post a receive buffer as part of preparing to send a Call.
If a Reply is missed, that receive is still posted. The requester
has to accommodate for that.

B. It can batch-post receive buffers (say, as part of receive
completion handling) in order to keep more than enough available.

The problem, as I see it, occurs when a responder wants to send a
one-way message to a requester. In scenario A, that knocks down a
receive buffer that was intended to catch a Reply. The requester
is responsible for getting a replacement receive buffer posted,
but there's a possibility that it could not do so fast enough
to catch other incoming messages.

This is the fundamental issue with one-way messages. A sender
has no way to know when the receiver is prepared to receive
additional messages because there is no message acknowledgement
in RPC-over-RDMA version 1, other than RPC Call and Reply messages.

A responder is responsible for posting enough receive buffers to
catch "credit" RPC Calls. It relies on the fact that requesters
can't keep more than "credit" RPC Calls in flight, to know exactly
how many receive buffers it needs to keep posted. So either:

A. It posts "credit" receive buffers when accepting a connection,
and then it posts a receive buffer as part of preparing to send a
Reply. If sending fails, that receive is still posted, and the
responder has to accommodate for that.

B. It can batch-post receive buffers in order to keep more than
enough available.

The problem, as I see it, occurs when a requester sends enough
one-way messages that it prevents the transmission of more
RPC Calls. The responder cannot send Replies to those one-way
messages, so the requester has no way to know when the responder
is ready to receive another RPC Call.

I think that credit management will have to take a different
form if we believe one-way messages are a necessary part of the
future of RPC-over-RDMA.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 8 Aug 2017 07:30:56 -0400

> that there are only one or two ways to implement that protocol.

Generally, if you describe the protocol, you should not have all that
much trial and error.  Although you don't tell the implemrenter exactly
how to do things, if you expect a lot of trial and error, particularly
error,
there is a something in the spec that needs ro be clarified.

Also, if the two sides pick different implementations, evertthing should
work OK.

> catch Replies for all outstanding RPC Calls.

Agree with that.

> So either:

> has to accommodate for that.

Yes.

> completion handling) in order to keep more than enough available.

With that formulation, receives are still posted in the event of missed
reply but
they are not considred a problem.

What we seem to agree on and that A and B agree on, is that the responder
is free
to respond to requests without obtaining credits to validate his right to
do so.

> one-way message to a requester.

I don't think you can send a one-way message to a "requester" or
"responder".
A parrty is either a requester or resopnder only in the context of an RPC.
If you are not in the process of sending or processing an RPC, you are not
a
requester or a responder  although you are either a client or a server,

> In scenario A, that knocks down a
> receive buffer that was intended to catch a Reply.

It knocks down a receive buffer, clearly.

Specfic receives are not dedicated to specific purposes but I think that
one-way
messages, unlike replies should require a credit for the sender to send
them,
and should not deplete the pool of receves for which there are no
associated
credits (i.e. the ones posted to accommodate replies).

> The requester
> is responsible for getting a replacement receive buffer posted

I don't think the receiver is responsible for this, just as it is not
responsible for
replenishing receives taken up by requests.  It may choose to do so but it
is
not obliged  to do so.

> to catch other incoming messages.

Preventing that is what the credit logic is for, and you can use that
to prevent over-sending one-way messages, just as it prevnts oversending of
requests.  I accept that one-way messages add some additional issues
(see below) but the same basic logic is applicable.

> how many receive buffers it needs to keep posted.

I agree.

> So either:

I don't see the point of this typology.

> A. It posts "credit" receive buffers when accepting a connection,

This formulation is confusing.  I would say that it posts some
number of receives and then reports the number of such receives
as "credit".

> Reply.

It can post an aditional receive at any time it wants.  It is not tied
to sending a reply.

> If sending fails, that receive is still posted, and the

I don't see anything to be accommodated.  The responder is
required to report as "credit" the number of receives that have
been posted.  This situation is unlike the other "A" above.

> B. It can batch-post receive buffers in order to keep more than

There is no definition of "enough" or "more than enough".  The responder
chooses how may receives to post and should report that as the number of
credits.

> RPC Calls.

So any solution has to prevent that from happening.  I guess what I prposed
was:

   - Some items considered one-way messages are used in pairs, even though
   they are not part of RPC's that it is the job of the transport.
   - You can send otherwise "empty" one-way messages if you need to send
   credit information.

> The responder cannot send Replies to those one-way

He can send other one-way messages to get the credit information to the
peer.

> form if we believe one-way messages are a necessary part of the

I don't believe one-way messages per se are a necessary part of the futue
of RPC-over-RDMA.
I just seems to me simpler to add them then it does to you.

The thing that I think is necessary to the future is of RPC-over-RDMA, if
it has one,
is that there be some provision for the transfer of control messages used
by the
trannsport itself.  If there is a need to pair them, I'm OK with that but I
fell Version 1 is
funmentally handicapped by a strucure in which in which the only messages
to be sent are
either a request to be sent on behalf of a RPC Requester or a reply to be
sent on behalf of
an RPC Responder.  It may be that RPC-over-RDMA will not have a future and
that
relying on new pNFS mapping types is adequate, but I' prefer us to have
multiple paths forward.

This is probably mateial for a 4/1/2018 I-D, but I think we should think
about the fact that some
things do not now have appropriate names:

   - RPC-over-RDMA Version 1 woul be more appropiriately called
   RPC-over-RDMA Version Zero
   - NFSv4.1 would be better understood if it was called NFSv5.0.
   - NFSv4.2 would be better understood if it was called NFSv5.1.

On Mon, Aug 7, 2017 at 5:46 PM, Chuck Lever <[email-masked]> wrote:

> > If i was addressing an unduly narrow problem, it was because it was
> framed around the difficulties that might be added by adding one-way
> messages such as those added by Version 2.  If you consider, as I guess
> makes sense, RPCs that are not replied to as effectively one-way messages,
> that puts an entirely different spin (of possibly high angular momentum) on
> the problem.
> >
> > I think we discussed this a while ago but no change in the spec was made
> to either make it clear I was wrong, or indicate that I was right.  My
> natural, even if it is not explcitly there.
>
> > If the spec still is not clear on this point, it might need to be fixed,
> if not to mandate my approach but at least to indicate how the requester
> can safeguard itself against the possibility that a response sent by the
> responder will cause a disconnect.
> >
> > With this approach, RPC's not replied to would still be a problem, but
> the problem cannot result in a disconnect.  For each not-replied-to
> request, there would be a receive that was not included in the credits,
> which would be devoted in this model to receives for receiving requests
> (and things intended to be one-way messages).   Of course, with 1K or 4K
> buffers that would not be such a big deal and you could defer any
> connection reset until a time when it was easy to do.
>

---

From: Chuck Lever <[email-masked]>
Date: Tue, 8 Aug 2017 12:06:24 -0400

> On Aug 8, 2017, at 7:30 AM, David Noveck <[email-masked]> wrote:
> 
> With that formulation, receives are still posted in the event of missed reply but
> What we seem to agree on and that A and B agree on, is that the responder is free
> to respond to requests without obtaining credits to validate his right to do so.

An RPC-over-RDMA version 1 requester always gates RPC Calls based
on how many credits were granted. The responder depends on that
behavior to know how many Replies it may send. The responder
grants credits, so it is in control of any "rights" to send more
Replies. There is no purpose for sending more Replies than there
have been Calls.

> I don't think you can send a one-way message to a "requester" or "responder".

Yes, you can. This is exactly the problem with RPC-over-RDMA version 1.

We have a fundamentally asymmetric arrangement here. One side always
requests credits, and one side always grants them. One side always
sends RPC Calls, and one side always sends RPC Replies.

The requester requests credits, and the responder grants them.

> A parrty is either a requester or resopnder only in the context of an RPC.

The transport implementation responsibilities are different depending
on whether the receiver is the requester or the responder. This is
one reason bidirectional RPC-over-RDMA is so challenging.

> If you are not in the process of sending or processing an RPC, you are not a 

We're talking only about RPC-over-RDMA version 1 at this moment, in
just one direction. So the only thing that can happen on a requester
is that a receive buffer is posted to catch a Reply.

> Specfic receives are not dedicated to specific purposes but I think that one-way
> messages, unlike replies should require a credit for the sender to send them,
> and should not deplete the pool of receves for which there are no associated 

This is very confusing. First you say "specific receives are not
dedicated to specific purposes" (which is true) then you say that
"one-way messages should not deplete the pool of receives for which
there are no associated credits".

How can there be receives with no associated credits if receives
are not dedicated to a specific purpose?

So now we have three purposes for receive buffers:

 - Catching RPC Calls in either direction
 - Catching RPC Replies in either direction
 - Catching one-way messages

Another way to approach the problem of these "one-way" messages is
to treat them as a control plane and use a separate mechanism for
conveying them. For example, at connection setup, the two sides
can send R_key and offset for a small buffer that can be used
for these messages.

This doesn't help for things like message chaining.

> I don't think the receiver is responsible for this, just as it is not responsible for
> replenishing receives taken up by requests.  It may choose to do so but it is
> not obliged  to do so.

In order to prevent RNR and connection termination, the receiver is
responsible for keeping enough receive WRs in the receive queue. I'm
not sure what you are trying to say here.

> number of receives and then reports the number of such receives

Since there are other resources that must be in place before a
connection is accepted, a responder typically knows the maximum
number of credits it will support for this transport instance,
and then prepares those resources, including posting receive
buffers, before completing the accept.

Posting receives before the accept completes prevents a race
where the first Send from the requester might not find a
posted Receive waiting for it on the responder.

> It can post an aditional receive at any time it wants.  It is not tied

Here I'm describing one way of implementing the protocol.

To guarantee that enough receives are posted to handle the balance
of RPC Calls that might be sent, the responder ensures that one new
receive is posted _before_ it sends a Reply.

> I don't see anything to be accommodated.

The transport's Receive Queue and Receive Completion Queue are sized
based on the number of credits the responder supports for this
transport. The responder cannot post more Receives than there are
Receive WQEs and CQEs. In order to prevent a queue overrun, the
responder has to know when to stop posting Receives.

> The responder is

If, as you say, there is no relationship between sending a Reply
and posting a Receive, how does the responder know how many credits
it should grant in each Reply?

The number of granted credits typically does not change. It doesn't
go down after the requester has sent an RPC Call. It doesn't go back
up after the responder has sent an RPC Reply.

The number of credits granted is the total number of RPC Calls that
may be in flight at once.

Suppose a responder grants 8 credits. After the requester has sends
8 RPC Calls, the responder's credit grant is still 8. But the
requester cannot send another RPC Call until the responder sends one
Reply with 8 or more in its rdma_credits field.

There is more than just the value in the rdma_credit field. Both
sides count the number of RPCs in flight, and subtract those from
the credit limit to determine how many new RPC transactions can be
started.

> So any solution has to prevent that from happening.  I guess what I prposed was:
> 	• Some items considered one-way messages are used in pairs, even though they are not part of RPC's that it is the job of the transport.

The problem in that case is that both implementations with which I am
aware use RPC gating to prevent credit overflow. This is an
implementation problem, not a protocol problem, however.

> 	• You can send otherwise "empty" one-way messages if you need to send credit information.

This would be the same as an ACK, and essentially makes one-way
messages into two-way messages again. OK.

But it doesn't appear to help in cases where one half of a two-way
message is dropped.

> He can send other one-way messages to get the credit information to the peer.
> 
> I don't believe one-way messages per se are a necessary part of the futue of RPC-over-RDMA.

IMO you are simplifying by

- ignoring the issues of backwards compatibility with
   RPC-over-RDMA version 1
- ignoring the need to address lost messages or dropped RPCs

> The thing that I think is necessary to the future is of RPC-over-RDMA, if it has one,
> is that there be some provision for the transfer of control messages used by the
> trannsport itself.  If there is a need to pair them, I'm OK with that but I fell Version 1 is
> funmentally handicapped by a strucure in which in which the only messages to be sent are
> either a request to be sent on behalf of a RPC Requester or a reply to be sent on behalf of
> an RPC Responder.

I think that's what I've been saying all along.

> It may be that RPC-over-RDMA will not have a future and that 
> relying on new pNFS mapping types is adequate, but I' prefer us to have multiple paths forward.

--
Chuck Lever