Subject: [nfsv4] Comments/thoughts regarding credit accounting for RPC-over-RDMA Version Two

From: David Noveck <[email-masked]>
Date: Sun, 29 Jul 2018 09:04:47 -0400

These comments are prompted by your presentation at IETF 102 and I will be
following your presentation quite closely,  and offering my comments about
each of the issues you raise.   For those that have not seen the
presentation, it can be accessed at https://datatracker.ietf.org/
meeting/102/materials/slides-102-nfsv4-rpcrdma-credit-accounting-00.

First of all, thanks for the presentation.  While I had had a vague sense
of unease about version 2 credit accounting, I had been in denial about
addresssing this area.   By presenting the issues and possible solutions
you made a major step toward considering these issues seriously.   I hope
the working group will continue the discussion as work on version 2
progresses.

I'll start by looking at the "shortcomings" slides (slides 6-11)

Does not support cases where no RPC transaction is involved

As I recall, Tom Talpey asked whether this transport might ever handle
non-RPC traffic.   While I agree with your answer that that does not seem
likely, I think it is important to note that, even so, that this does not
mean that every transmission must be either an RPC request or reply.
Version One unfortunately makes that assumption and I feel that Version Two
needs to take a different path.

Control plane messages with no RPC XID

An important example are the transport-properties messages defined in the
current Version Two I-D.   At a minimum we need such messages to setup a
consistent view of transport characteristics between the two peers.   In
Version One, we only have the cm private data convention, but I think there
is general agreement that this is best considered a temporary
(i.e.one-version) expedient until the transport can handle this withn the
protocol.

Connection-level keep-alive

While v4 has the lease renewal requirement, that doesn't address the case
of multiple RDMA connections used as to support a single clientid, even
when multiple adapters are not used.   As we've heard, getting multiple
queue-pairs involved is often needed for best performance.  I think this is
best handled at the transport level, with a transport-level keep-alive.
THat could be an extension but I would argue it makes sense in the base
version two, even if most likely as an OPTIONAL feature.

Does not support unpaired messages

As a understand it,  the cases under this bullet are to deal with unpaired
messages within RPC while the previous slide dealt with uppaired message
ouside RPC.  I think some of the items here, while worth addressing, really
belong on the previous slide.

Retransmission of RPC Calls

Retransmission of RPC calls per se does no cause any difficulties.
However, what does cause difficulties are the situations that give rise to
retransmission:  A case where there is no reply (lack of assumed RPC
pairing), or a situation of great uncretainty about a reply will ever be
sent.

RPC Call with no Reply, like unicast or broadcast

True in theory but I don't see RPC-over-RDMA being used in these cases.

RDMA_DONE or similar

Although RDMA_DONE was terminated with prejudice and will not be revived,
there are some cases in which such one-way messages will be needed.  See
below.

Unsolicited Sends from responder to requester

I think unsolicited  sends are not really part of RPC and if a send is
unsolicited, it is not clear who is the responder and who is the
requester.   These terms only have a meaning within the context of an
ongoing RPC.

Does not support cases where ratio of transport Send to RPC message is not
1:1

The assumption that sends are bound 1:1 to RPC messages permeates v1.  It
might be that breaking this entirely is not possible within v2.  However,
given that v2 is extensible, we might be able to do so in an extension.
However, to enable that, we need to make sure, that the credit accounting
in base v2 is addequate to the task.

Multiple RPC messages in one transport Send

This would be an important optimization in the case of metadata-intensive
workloads, particularly with VersionTwo's larger inline threshold.   Even
though I don't see this as an extension that will be addressed soon, the
credit accounting mechanism needs to support it, or else Version Two's
extension mechanism will be undercut.

Multiple exchanges for a single RPC transaction

I don't understand the need for this.   While I could see a case in which
both the call and the reply require multiple sends, there would be only a
single call and a single reply and thus only a single interaction.

A single RPC message requiring multiple transport Sends

This is an important optimization that I want to see soon.   As this is not
part of the base Version Two, it needs to be done as an extension to
Version Two, and we need to make sure that the credit accounting scheme
does not foreclose this sort of extension.

Bi-directional RPC is problematic

It does have problems but I feel they can be addressed.

Two directions equals two responders, therefore there has to be one credit
grant per direction

Yes.

RPC/RDMA v1 re-uses the one "credits" field

Yes.   Unfortunately, it maintained the use of the credits field on
requests as a "credit request" even in the bidirectional case.

When v1 was conceived, only a single direction of transfer was considered.
In that context, there was no sunstabtial need for the "credits" field on
requests.   Since the space was reserved, the field has a "credit
request".  I'm not clear how a client might decide on a request value.  My
server pretty much ignores it.

In any case, the credit request is a hint and i don't see a need for every
request to contain such a hint.

In a single-sided message on a bi-directional connection, what does the
"credits" field mean?

RPC/RDMA sniffs RPC calldir field; if no RPC message, no calldir field to
snif

V1 sniffs the RPC message while V2 has a direction field.   Since this is a
32-bit field, there is room for a bit indicating how the credit field is to
be interpreted.

No mechanism to resynchronize if one side loses track of credits.

I think the isssue here is not one side losing track of credits, but rather
situation in which the rwo suides are not in agreement and need to
re-establish agreement.

Non-windowing credit accounting is inherently resilient to loss of credit
grant, but not to loss of a data packet

Only recourse is to break the connection to re-initialize credit accounting

There's a lot of situations in which ofed/ibta forces a connection break.
I think we want to avoid adding to that.

Provides no network Quality of Service guarantees

Never expected there would be.  Do you have a particular extension in mind?

No way to protect against noisy neighbors or DoS

Right now this is an implementation msatter.  The responder is free to keep
credit grants to a minimum and if the requester does not play along, he
loses his connection.

Lower bound of one on credit grants.

Yes.

No way to request a larger lower bound

This sound like a candidate for a transport poroperty.  What needs to be
specified is what requests are enttled to use credits within the reserved
lower bound.  This seems more likely to be deferred to an extension, rather
than being added to base v2.

Now I'll look at the "Goals" slide (slide 12).

One thing that is not on this slide but should be is first-class treatment
of reverse direction operation.

Incremental performance improvements

Clearly we want these but we have a time constraint.   We will need to
decide how much can be done in the base Version Two, and what needs to be
done using the extensibility model.

My feeling is that we probably won't have time to address these in base v2
but that we need to avoid excessive serialization.  In other words, we
don't want to wait until v2 is published as a Proposed Standard before
beginning work on optional extentions.   Once v2 is a working group
document, it makes sense to start consideration of potential extensions.
Otherwise, we would be considering the adeqacy of the extension facilities
and needed chnges in credit management in a vacuum.

Larger default inline threshold, remote invalidation

These are in the current I-D and do not raise credit accounting issues.

Extensibility as part of the base protocol

Yes!

Richer error reporting

This is in the current I-D.  As currently defined, it does not raise credit
accounting issues but there will be issues if it is extended, as it should
be, to cover reporting of errors in messages that are not requests.

Transport property negotiation

This is in the current I-D.  At a minimum, we need to address any credit
accounting issues raised by this.

Ability to send something other than a single RPC message per RDMA Send

This is presumably under the extension bullet because the vehicle by which
it will be delivered will be as an optional extension.  However, the
motivation for it is clearly performance.

These comments are about the "possible fixes" slide (slide 15):

Gate Sends rather than RPC transactions (no XDR change)

I think this is the right approch.  Since the sender of the grant only
knows how many receives he has posted, it is natural that this control the
number of sends the peer may do.   Given that v1 only allows one send per
RPC request, this is essentially a distinction without a difference.

I know there is some language in RFC8166 that suggest that a credit grant
gives you the right to perform a single request.  However, the fact of the
matter is that it gives you the ability to do a single send to match the
responder's single receive.  As a result, given the 1-1 send-request
binding of v1, gating on sends is indistinguishable from gating on rpc
requests, and the fact that v1 describes this as gating on rpc's doesn't
really matter, untill you break the 1-1 binding assumption.

With regard to sending of the reply, the credit grant has no real role.
 The reason the responder can safely send the response is not due to a
credit he receives from the requester.  In v1, the requester only sends a
credit request.  If v2 does send a credit grant with a equest, it should
only apply to reverse-direction requests (and one-way messages).  The
requester can make sure that the responder can do the needed sends is by
posting the needed receives.  These should not be included in the credit
grant, since, if you did so, the could be used up by reverse-direction
traffic.

Change from a non-windowing to a windowing scheme (no XDR change)

Even though there is no XDR change, this is a big change in implementation
terms.   Also, given that there will be implementations that support both
v1 and v2 and implement two different credit management schemes, it will be
hard to maintain interoperability with two different credit manageent
schemes while maintaining each one as specified.

Add a second credits field to the Transport Header.

I only think we should add words to the transport header unless there is a
real need and i don't see it in this case.

I think there was some discussion of diding the credit field into two
16-bit fields but RFC8166's language about the first four words of the
header seem to make that impossible.

... Each message would carry a credit request and a credit grant,

I see the point of the grant but since the credit requests are just hints,
they are not needed in every message.   It would be bettrer to make the
request grant level a transport property which the sender can update when
needed.

... and would apply to both directions concurrently

Each credit grant only applies to a single direction.  I don't see how
things could b done anyother way.

Add an RDMA_ACK proc that conveys current grants, to act as ACK of an
unpaired Send

Approve of an RDMA_GRANT to convey current grant.   I don't see the point
of trying to turn unpaired sends into RPCs by adding an ack facility to the
basic function of conveying the current grant status.

---

From: Chuck Lever <[email-masked]>
Date: Sun, 29 Jul 2018 14:03:29 -0400

> On Jul 29, 2018, at 9:04 AM, David Noveck <[email-masked]> wrote:
> 
> These comments are prompted by your presentation at IETF 102 and I will be following your presentation quite closely,  and offering my comments about each of the issues you raise.   For those that have not seen the presentation, it can be accessed at https://datatracker.ietf.org/meeting/102/materials/slides-102-nfsv4-rpcrdma-credit-accounting-00.
> 
> First of all, thanks for the presentation.  While I had had a vague sense of unease about version 2 credit accounting, I had been in denial about addresssing this area.   By presenting the issues and possible solutions  you made a major step toward considering these issues seriously.   I hope the working group will continue the discussion as work on version 2 progresses. 
> 
> As I recall, Tom Talpey asked whether this transport might ever handle non-RPC traffic.   While I agree with your answer that that does not seem likely, I think it is important to note that, even so, that this does not mean that every transmission must be either an RPC request or reply.  Version One unfortunately makes that assumption and I feel that Version Two needs to take a different path.  
> 
> An important example are the transport-properties messages defined in the current Version Two I-D.   At a minimum we need such messages to setup a consistent view of transport characteristics between the two peers.   In Version One, we only have the cm private data convention, but I think there is general agreement that this is best considered a temporary (i.e.one-version) expedient until the transport can handle this withn the protocol.

Tom Talpey pointed out:

Multiple Sends per RPC message: we probably want to rely on
transport ordering for the proper reassembly of multi-segment
RPC messages. Mixing Sends of different RPC messages, or sending
a control plane message while sending a multi-segment RPC message
will probably need to be forbidden in order to keep the protocol
and the transport implementations simple.

> While v4 has the lease renewal requirement, that doesn't address the case of multiple RDMA connections used as to support a single clientid, even when multiple adapters are not used.   As we've heard, getting multiple queue-pairs involved is often needed for best performance.  I think this is best handled at the transport level, with a transport-level keep-alive.  THat could be an extension but I would argue it makes sense in the base version two, even if most likely as an OPTIONAL feature.

Control plane messages would not have an RPC XID associated with
them. That is probably OK for single direction control messages,
but in cases where the control plane message is a request and
needs a response (eg keep-alive) there needs to be a way for the
peer sending the request to recognize and pair the response with
it.

Could be as simple as not allowing the peer to send more than one
control plane request at a time.

> As a understand it,  the cases under this bullet are to deal with unpaired messages within RPC while the previous slide dealt with uppaired message ouside RPC.  I think some of the items here, while worth addressing, really belong on the previous slide.
> 
> Retransmission of RPC calls per se does no cause any difficulties.  However, what does cause difficulties are the situations that give rise to retransmission:  A case where there is no reply (lack of assumed RPC pairing), or a situation of great uncretainty about a reply will ever be sent.
> 
> Although RDMA_DONE was terminated with prejudice and will not be revived, there are some cases in which such one-way messages will be needed.  See below.
> 
> I think unsolicited  sends are not really part of RPC and if a send is unsolicited, it is not clear who is the responder and who is the requester.   These terms only have a meaning within the context of an ongoing RPC.
> 
> Does not support cases where ratio of transport Send to RPC message is not 1:1
> 
> The assumption that sends are bound 1:1 to RPC messages permeates v1.  It might be that breaking this entirely is not possible within v2.  However, given that v2 is extensible, we might be able to do so in an extension.  However, to enable that, we need to make sure, that the credit accounting in base v2 is addequate to the task. 
> 
> This would be an important optimization in the case of metadata-intensive workloads, particularly with VersionTwo's larger inline threshold.   Even though I don't see this as an extension that will be addressed soon, the credit accounting mechanism needs to support it, or else Version Two's extension mechanism will be undercut.
> 
> I don't understand the need for this.   While I could see a case in which both the call and the reply require multiple sends, there would be only a single call and a single reply and thus only a single interaction.

draft-cel-nfsv4-rpcrdma-reliable-reply describes a possible
extension where an RPC transaction might require multiple
Send exchanges, and it does use RDMA_DONE for the final
part of the exchange.

> This is an important optimization that I want to see soon.   As this is not part of the base Version Two, it needs to be done as an extension to Version Two, and we need to make sure that the credit accounting scheme does not foreclose this sort of extension.
> 
> Two directions equals two responders, therefore there has to be one credit grant per direction
> 
> Yes.
> 
> Yes.   Unfortunately, it maintained the use of the credits field on requests as a "credit request" even in the bidirectional case.
> 
> When v1 was conceived, only a single direction of transfer was considered.  In that context, there was no sunstabtial need for the "credits" field on requests.   Since the space was reserved, the field has a "credit request".  I'm not clear how a client might decide on a request value.  My server pretty much ignores it.

It cannot ignore that value if there are to be multiple Sends
per RPC Reply. This is where the server needs to know the
client's Receive buffer count.

Or to put this another way: your server MAY ignore the credit
request today because RPC/RDMA v1 assumes a 1:1 ratio between
RPC Calls and granted credits. A v1 client can never send more
Calls than the server has granted credits, and will never send
more Calls than it has requested credits for.

That naturally constrains the number of Sends in both
directions to be the lesser of the credit request and the
credit grant. But that constraint no longer applies if we
allow multiple sends per RPC message.

> In any case, the credit request is a hint and i don't see a need for every request to contain such a hint.   
> 
> In a single-sided message on a bi-directional connection, what does the "credits" field mean? 
> 
> RPC/RDMA sniffs RPC calldir field; if no RPC message, no calldir field to snif
> 
> V1 sniffs the RPC message while V2 has a direction field.   Since this is a 32-bit field, there is room for a bit indicating how the credit field is to be interpreted.

The new direction field in the RPC/RDMA header was added to
disambiguate the meaning of the credits field. There is no
reason to add a second bit for this purpose.

> I think the isssue here is not one side losing track of credits, but rather situation in which the rwo suides are not in agreement and need to re-establish agreement.

> Non-windowing credit accounting is inherently resilient to loss of credit grant, but not to loss of a data packet
> 
> There's a lot of situations in which ofed/ibta forces a connection break.  I think we want to avoid adding to that.

Tom pointed out that losing synchronization should be rare in
a proper implementation of the protocol. Connection loss by
itself is not bad. Frequent or unpredictable connection loss
is what we need to avoid.

> Right now this is an implementation msatter.  The responder is free to keep credit grants to a minimum and if the requester does not play along, he loses his connection.

We want to ensure that a requester that _is_ playing by our
rules cannot overrun the responder or deny service to other
requesters.

> This sound like a candidate for a transport poroperty.  What needs to be specified is what requests are enttled to use credits within the reserved lower bound.  This seems more likely to be deferred to an extension, rather than being added to base v2.
> 
> One thing that is not on this slide but should be is first-class treatment of reverse direction operation.

> Clearly we want these but we have a time constraint.   We will need to decide how much can be done in the base Version Two, and what needs to be done using the extensibility model.
> 
> My feeling is that we probably won't have time to address these in base v2 but that we need to avoid excessive serialization.  In other words, we don't want to wait until v2 is published as a Proposed Standard before beginning work on optional extentions..   Once v2 is a working group document, it makes sense to start consideration of potential extensions.  Otherwise, we would be considering the adeqacy of the extension facilities and needed chnges in credit management in a vacuum.
> 
> This is in the current I-D.  As currently defined, it does not raise credit accounting issues but there will be issues if it is extended, as it should be, to cover reporting of errors in messages that are not requests.
>  
> This is in the current I-D.  At a minimum, we need to address any credit accounting issues raised by this..
> 
> This is presumably under the extension bullet because the vehicle by which it will be delivered will be as an optional extension.  However, the motivation for it is clearly performance.

But not _only_ performance.

The rpcrdma-reliable-reply draft describes a mechanism for
eliminating the need for the requester to predict the maximum RPC
Reply size. That is not necessarily a performance issue, but would
fall on under the rubric of "multiple exchanges per RPC transaction".

> I think this is the right approch.  Since the sender of the grant only knows how many receives he has posted, it is natural that this control the number of sends the peer may do.   Given that v1 only allows one send per RPC request, this is essentially a distinction without a difference.

I did not mean to imply that we must pick only one of these
solutions (nor that I had listed every possible solution).

> I know there is some language in RFC8166 that suggest that a credit grant gives you the right to perform a single request.  However, the fact of the matter is that it gives you the ability to do a single send to match the responder's single receive.  As a result, given the 1-1 send-request binding of v1, gating on sends is indistinguishable from gating on rpc requests, and the fact that v1 describes this as gating on rpc's doesn't really matter, untill you break the 1-1 binding assumption.
> 
> With regard to sending of the reply, the credit grant has no real role.   The reason the responder can safely send the response is not due to a credit he receives from the requester.  In v1, the requester only sends a credit request.  If v2 does send a credit grant with a equest, it should only apply to reverse-direction requests (and one-way messages).  The requester can make sure that the responder can do the needed sends is by posting the needed receives.  These should not be included in the credit grant, since, if you did so, the could be used up by reverse-direction traffic.

The credit request from the requester says "I can receive N messages".
Having both a credit request and a credit grant means there can be
asymmetric operation. For example:

A client could say "I have 128 receive buffers" and a server could
say "I have 32 receive buffers" . The server could then send multi-
segment replies to each of the 32 calls made by the client without
serialization. This asymmetry is sensible: a server often needs to
limit per-connection resources to scale the number of client
connections.

> Even though there is no XDR change, this is a big change in implementation terms.   Also, given that there will be implementations that support both v1 and v2 and implement two different credit management schemes, it will be hard to maintain interoperability with two different credit manageent schemes while maintaining each one as specified.

The first exchange on a connection determines the protocol version,
and we are limited to a single Send pair at that time. If the
responder does not support the presented protocol version, it MUST
ignore the fields after the version field (namely the rdma_credits
field).

Once the transport protocol version to be used on that connection
has been established, the receiver then inserts the correct set of
virtual credit management operations in the transport context. We
can additionally specify that the credit limit MUST remain one (1)
until the value in the credits field has been recognized; that is,
the protocol version is set and the transport header has been fully
parsed.

Where do you see difficulty with interoperation?

> I only think we should add words to the transport header unless there is a real need and i don't see it in this case.
> 
> I think there was some discussion of diding the credit field into two 16-bit fields but RFC8166's language about the first four words of the header seem to make that impossible.

> I see the point of the grant but since the credit requests are just hints, they are not needed in every message.   It would be bettrer to make the request grant level a transport property which the sender can update when needed.

In a non-windowed mode of operation, the credit grant rarely needs
to change. No server implementation today alters it. But no server
implements Shared Receive Queue, or adjusts resources based on
workload, neither of which are forbidden by RFC 8166. In these
cases, credit grants might change frequently. Thus I don't think we
can assume that the credit values are updated rarely in every case,
even with RPC/RDMA v1.

If we go with a windowed mode of operation, that makes the credits
field quite dynamic, thus it would be needed in every header. And,
note that we can't simply stop using the rdma_credits field, as you
have pointed out above.

> Each credit grant only applies to a single direction.  I don't see how things could b done anyother way.
> 
> Add an RDMA_ACK proc that conveys current grants, to act as ACK of an unpaired Send 
> 
> Approve of an RDMA_GRANT to convey current grant.   I don't see the point of trying to turn unpaired sends into RPCs by adding an ack facility to the basic function of conveying the current grant status.

An ACK would not turn a Send into an RPC transaction, but it would
turn an unpaired Send into a paired Send in order maintain proper
Receive buffer accounting. This is how the RDMA layer operates, for
example; and TCP does not allow unpaired Sends (there is always an
ACK for each segment, even one that is piggybacked or where several
segments are ACK'd at the same time).

The ACK is positive confirmation that a Receive buffer is now
available.

Tom suggested that if we want to switch to a windowing credit
management system, we should consider using sequence numbering
instead of counting free Receive buffers. That would make it easy
for a single ACK to acknowledge multiple Sends at once.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 31 Jul 2018 15:35:24 -0400

> The credit request from the requester says "I can receive N messages".

It could say that, but the normal way to say "I can receive N messages" is
by sending a credit grant of N.   That what the credit grant is for.

> say "I have 32 receive buffers" .

Client could say that by sending a credit grant of 128 while the server
says he has 32 receive buffers by sending a credit grant of 32.

With the request-grant narrative, the client would be asking the server
to accept 128 buffers and the server would essentially be rejecting
his reques and giving him only 32.  Actually, the client was only indicating
how many buffers he had, which is the function of a credit gramt.

> The server could then send multi-

Yes.

> Where do you see difficulty with interoperation?

I'm not sayng it is impossible.  You indicate above a clear path for
switching to the new protocol mode.

The difficulty I see is the effort involved in maintaining two siginicantly
different protocols.   As v2 was orignally conceived, the inteention was
to maintain v1 as it was and make a small number of changes to enable
extensibility and deal with some pressing issues.   There is a significant
level of effort in going beyond that.   The result would be that you would
wind up maintaing two protocols instead of one with a higher level of
effort for implemetations.   If we do make such a  shift
we need to be sure that the benefirs exceed the cosrs.

To get back to my choce of the word "difficulty", I was thinking of one
person
dealing with both versions, which might be realistic for many of the
impliementatiions.    In that case, I expect it to be hard to deal with two
protocols that are superficially so similar yet so different in credit
management.
Itis possible that people will find it difficult to keep two different
credit management
schenes straight and I expect bugs when they can't/don't.

On Sun, Jul 29, 2018 at 2:03 PM, Chuck Lever <[email-masked]> wrote:

> > These comments are prompted by your presentation at IETF 102 and I will
> be following your presentation quite closely,  and offering my comments
> about each of the issues you raise.   For those that have not seen the
> > First of all, thanks for the presentation.  While I had had a vague
> sense of unease about version 2 credit accounting, I had been in denial
> about addresssing this area.   By presenting the issues and possible
> solutions  you made a major step toward considering these issues
> seriously.   I hope the working group will continue the discussion as work
> on version 2 progresses.
> >
> > An important example are the transport-properties messages defined in
> the current Version Two I-D.   At a minimum we need such messages to setup
> a consistent view of transport characteristics between the two peers.   In
> > While v4 has the lease renewal requirement, that doesn't address the
> case of multiple RDMA connections used as to support a single clientid,
> even when multiple adapters are not used.   As we've heard, getting
> multiple queue-pairs involved is often needed for best performance.  I
> think this is best handled at the transport level, with a transport-level
> keep-alive.  THat could be an extension but I would argue it makes sense in
> the base version two, even if most likely as an OPTIONAL feature.
>
> > As a understand it,  the cases under this bullet are to deal with
> unpaired messages within RPC while the previous slide dealt with uppaired
> message ouside RPC.  I think some of the items here, while worth
> addressing, really belong on the previous slide.
> >
> > Although RDMA_DONE was terminated with prejudice and will not be
> revived, there are some cases in which such one-way messages will be
> needed.  See below.
> >
> > The assumption that sends are bound 1:1 to RPC messages permeates v1.
> It might be that breaking this entirely is not possible within v2.
> However, given that v2 is extensible, we might be able to do so in an
> extension.  However, to enable that, we need to make sure, that the credit
> accounting in base v2 is addequate to the task.
> >
> > This would be an important optimization in the case of
> metadata-intensive workloads, particularly with VersionTwo's larger inline
> threshold.   Even though I don't see this as an extension that will be
> addressed soon, the credit accounting mechanism needs to support it, or
> else Version Two's extension mechanism will be undercut.
> >
> > I don't understand the need for this.   While I could see a case in
> which both the call and the reply require multiple sends, there would be
> only a single call and a single reply and thus only a single interaction.
>
> > This is an important optimization that I want to see soon.   As this is
> not part of the base Version Two, it needs to be done as an extension to
> > When v1 was conceived, only a single direction of transfer was
> considered.  In that context, there was no sunstabtial need for the
> "credits" field on requests.   Since the space was reserved, the field has
> a "credit request".  I'm not clear how a client might decide on a request
> value.  My server pretty much ignores it.
>
> > V1 sniffs the RPC message while V2 has a direction field.   Since this
> is a 32-bit field, there is room for a bit indicating how the credit field
> is to be interpreted.
>
> > I think the isssue here is not one side losing track of credits, but
> rather situation in which the rwo suides are not in agreement and need to
> re-establish agreement.
>
> > Non-windowing credit accounting is inherently resilient to loss of
> credit grant, but not to loss of a data packet
> >
> > There's a lot of situations in which ofed/ibta forces a connection
> break.  I think we want to avoid adding to that.
>
> > Right now this is an implementation msatter.  The responder is free to
> keep credit grants to a minimum and if the requester does not play along,
> he loses his connection.
>
> > One thing that is not on this slide but should be is first-class
> treatment of reverse direction operation.
>
> > My feeling is that we probably won't have time to address these in base
> v2 but that we need to avoid excessive serialization.  In other words, we
> beginning work on optional extentions..   Once v2 is a working group
> > This is in the current I-D.  As currently defined, it does not raise
> credit accounting issues but there will be issues if it is extended, as it
> should be, to cover reporting of errors in messages that are not requests.
> >
> accounting issues raised by this..
> >
> > This is presumably under the extension bullet because the vehicle by
> which it will be delivered will be as an optional extension.  However, the
> > I know there is some language in RFC8166 that suggest that a credit
> grant gives you the right to perform a single request.  However, the fact
> of the matter is that it gives you the ability to do a single send to match
> the responder's single receive.  As a result, given the 1-1 send-request
> > Even though there is no XDR change, this is a big change in
> implementation terms.   Also, given that there will be implementations that
> support both v1 and v2 and implement two different credit management
> schemes, it will be hard to maintain interoperability with two different
> credit manageent schemes while maintaining each one as specified.
>
> > I only think we should add words to the transport header unless there is
> a real need and i don't see it in this case.
> >
> > I see the point of the grant but since the credit requests are just
> hints, they are not needed in every message.   It would be bettrer to make
> the request grant level a transport property which the sender can update
> when needed.
>
> > Approve of an RDMA_GRANT to convey current grant.   I don't see the
> point of trying to turn unpaired sends into RPCs by adding an ack facility
> to the basic function of conveying the current grant status.
>