Subject: [nfsv4] credit-related issues in draft-ietf-nfsv4-{rfc5666bis, rfc5666-implementation-experience, rpcrdma-bidirection}

From: David Noveck <[email-masked]>
Date: Mon, 18 Apr 2016 14:36:25 -0400

In looking at issues related to defining how message continuation might
work in an extension to RPC-over-RDMA Version Two, I've had occasion to
look at the documents now in WGLC and see what they say about the matter.
In general, I feel that they have clarified the issues, but there remain
some rough edges in that the field "credit" in the RPC-over-RDMA is an
overloaded field which has not really been recognized as such.  I think
this needs to be cleaned up eventuallly but I'll limit my comments to those
regarding things necessary to clean up Version One.

There will be some additional comments about
draft-cel-rpcrdmva-version-two.  I'll send those separately.

*Comments about draft-ietf-nfsv4-rfc5666bis-05*

*4.3.  Managing Receiver Resources*

It looks to me as if the second sentence of the first paragraph, is
wrong, or at least misleading.  As written, it suggests, that the RNIC
would scan pending receives to find one large enough to accommodate an
incoming message.  I don't think that is true, but if it is, an
extension might be able to take advantage of it by posting a small set
of large receives and a large set of more typically sized reads.

If my understanding is correct, I would revise this to be:

If any of the pre-posted receive buffers is not large enough to accept
an incoming RDMA Send, the RDMA Send operation may fail.

In the last sentence of the first paragraph, suggest the following changes:

   - Changing "a departure" to "different".
   - Changing "where"  to "in which"

*4.3.1.  Credit Limit*

I suggest that the first sentence of this paragraph be deleted, for
the following reasons:

   - It is impossible for this value to be accurate, since the number
of receive buffers can change due to receipt of a message.
   - It leads to the sort of confusion discussed in section 4.4 of
*draft-ietf-nfsv4-rfc5666-implem**entation-experience-02*
   - It is inconsistent with bidirectional operation.  While
bidirectional operation is described in
dra*ft-ietf-nfsv4-rpcdrma-bidirection*, I don't think we should say
things here that make it harder or more confusing to adapt to
bidirectional operation.

With regard to the first bulleted item, suggest the following changes:

   - Replacing "requested credit value for the responder" by "credit
value which it is asking the responder to grant".
   - Replacing "This is the maximum" by "This should be no greater
than the maximum".

*5.2.3.  Credit Value*

In the first two sentences suggest replacing "in an" by "with an".

Suggesting adding a new paragraph as follows:

When sent in a message header not used to transfer an RPC message
(e.g. RDMA_ERROR), the credit field is not used and should be ignored.

*Comments about dra**ft-ietf-nfsv4-rpcrdma-bidirection-02*

*2.1.  Flow Control*

The second paragraph as written is confusing.  While it seems to be
written saying it applies to all receive endpoints, it just discusses
handling of flow control on the responder.  Suggest the following
replacement paragraphs:

The RPC-over-RDMA protocol provides built-in send flow control to prevent
overrunning the number of pre-posted receive buffers on a responder
endpoint.  This is fully discussed in Section 4.3 of
[I-D.ietf-nfsv4-rfc5666bis].

With regard to preposted receives on the requester endpoint, it is the
responsibility of the requester to post sufficient receives to ensure
that the replies occasioned by the requests made of the responder can
be properly received.

*2.1.1.  Forward Credits*

This so much clearer than the corresponding section in
*draft-ietf-nfsv4-rfc5666bis-05*, that we should consider moving this
(minus the word "forward") to rfc5666bis.  Then the bidirection
document could limit itself to two basic points:

   - Credits work the same in each direction.
   - Credits for each direction are accounted for separately.

*Comments about dra**ft-ietf-nfsv4-rfc5666-implementation-experience-02:*

*4.4.1.  Recommendations*

In the first sentence suggest replacing "used to calculate" by "to be used
in calculating".

---

From: Chuck Lever <[email-masked]>
Date: Tue, 19 Apr 2016 12:22:25 -0700 (PDT)

Dave-

Thanks for you comments. Responses inline.

> On Apr 18, 2016, at 2:36 PM, David Noveck <[email-masked]> wrote:
> 
> In looking at issues related to defining how message continuation might work in an extension to RPC-over-RDMA Version Two, I've had occasion to look at the documents now in WGLC and see what they say about the matter.  In general, I feel that they have clarified the issues, but there remain some rough edges in that the field "credit" in the RPC-over-RDMA is an overloaded field which has not really been recognized as such.  I think this needs to be cleaned up eventuallly but I'll limit my comments to those regarding things necessary to clean up Version One.
> 
> There will be some additional comments about draft-cel-rpcrdmva-version-two.  I'll send those separately.
> 
> Comments about draft-ietf-nfsv4-rfc5666bis-05
> 
> 4.3.  Managing Receiver Resources
> It looks to me as if the second sentence of the first paragraph, is wrong, or at least misleading.  As written, it suggests, that the RNIC would scan pending receives to find one large enough to accommodate an incoming message.  I don't think that is true, but if it is, an extension might be able to take advantage of it by posting a small set of large receives and a large set of more typically sized reads.
> If any of the pre-posted receive buffers is not large enough to accept an incoming RDMA Send, the RDMA Send operation may fail.

I can replace the second sentence of the first paragraph
with the proposed sentence.

> 	• Changing "a departure" to "different".
> 	• Changing "where"  to "in which"

I can make those changes.

> 4.3.1.  Credit Limit
> I suggest that the first sentence of this paragraph be deleted, for the following reasons:
> 	• It is impossible for this value to be accurate, since the number of receive buffers can change due to receipt of a message.
> 	• It leads to the sort of confusion discussed in section 4.4 of draft-ietf-nfsv4-rfc5666-implementation-experience-02
> 	• It is inconsistent with bidirectional operation.  While bidirectional operation is described in draft-ietf-nfsv4-rpcdrma-bidirection, I don't think we should say things here that make it harder or more confusing to adapt to bidirectional operation.
> 	• Replacing "requested credit value for the responder" by "credit value which it is asking the responder to grant".
> 	• Replacing "This is the maximum" by "This should be no greater than the maximum".

See below, perhaps this entire section should be
replaced.

> 5.2.3.  Credit Value

I can make this change.

> When sent in a message header not used to transfer an RPC message (e.g. RDMA_ERROR), the credit field is not used and should be ignored.

I don't understand this suggestion. Why should the credit
field be ignored in RDMA_ERROR messages?

> Comments about draft-ietf-nfsv4-rpcrdma-bidirection-02
> 2.1.  Flow Control
> The second paragraph as written is confusing.  While it seems to be written saying it applies to all receive endpoints, it just discusses handling of flow control on the responder.

I don't see the word "responder" in Section 2.1.

> Suggest the following replacement paragraphs:
> endpoint.  This is fully discussed in Section 4.3 of [I-D.ietf-nfsv4-rfc5666bis].
> With regard to preposted receives on the requester endpoint, it is the responsibility of the requester to post sufficient receives to ensure that the replies occasioned by the requests made of the responder can be properly received.

I'm not certain what is being fixed here. The text in
question seems to apply to sending messages, not to
sending Calls or Replies in a particular direction.

> 2.1.1.  Forward Credits
> This so much clearer than the corresponding section in draft-ietf-nfsv4-rfc5666bis-05, that we should consider moving this (minus the word "forward") to rfc5666bis.  Then the bidirection document could limit itself to two basic points:
> 	• Credits work the same in each direction.
> 	• Credits for each direction are accounted for separately.

I assume you mean replacing some or all of rfc5666bis
Section 4.3.1 with Section 2.1.1 of rpcrdma-bidirection.
If there is no other objection to this, I will proceed
with this suggestion and then simplify Section 2.1.1.

> Comments about draft-ietf-nfsv4-rfc5666-implementation-experience-02:
> 
> 4.4.1.  Recommendations
> 
> In the first sentence suggest replacing "used to calculate" by "to be used in calculating". 

I find the existing sentence to be adequate:

"RFC 5666bis should clarify the method used to calculate
these values."

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 20 Apr 2016 06:25:48 -0400

> > When sent in a message header not used to transfer an RPC message (e.g.
RDMA_ERROR), the credit field is not used and should be ignored.

> field be ignored in RDMA_ERROR messages?

I don't know.  I think that if it is not ignored, you have to specify
whether it a request or a grant.  As I couldn't figure out how to do that,
I punted and said it should be ignored.  YMMV.

> > The second paragraph as written is confusing.  While it seems to be
written saying it applies to all receive
>> endpoints, it just discusses handling of flow control on the responder.

> I don't see the word "responder" in Section 2.1.

It isn't there, but I'm arguing (see below) that it should be.

> > With regard to preposted receives on the requester endpoint, it is the
responsibility of the requester to post
> > sufficient receives to ensure that the replies occasioned by the
requests made of the responder can be
> > properly received.

> I'm not certain what is being fixed here.

> The text in question seems to apply to sending messages,

It seems to but, if you examine section 4.3 of rfc5666bis, what you find is
an explanation of the credit mechanism, and my understanding of that
is that it is a mechanism to prevent the requester from sending too many
requests. If you  read section 4.3.1 carefully, there is no indication of
any connection to preposted receives on the requester and a lot of cases in
which the lack of attention to requester resource indicates tellingly that
only responder resources are addressed.  For example,

On occasion a requester or responder may need to adjust the amount o
resources available to a connection.  When this happens, the responder
needs to ensure that a credit increase is effected (i.e. RDMA Receives
are posted) before the next reply is sent.

Some things to note here:

   - Only the responder is told to post additional receives in the
event of a credit increase.  the requester is not told to do so.
   - It is said that either may need to adjust resources and can do so.
   - The conclusion I draw is that although a requester may adjust
resources and may post additional receives, that doesn't happen
through the credit mechanism

To illustrate the point , let us suppose that you have a responder,
which continues to grant ten credits for a while and then suddenly grants
one hundred.  Because of that change, in line with the material quoted
above, the responder has to post 90 additional receives before sending the
reply with the new credit grant.

The requester does not have to post additional receives at this point.  He
may have to but only if has more than ten outstanding requests at some
point in the future.  The credit mechanism affects the choices he makes but
he only has to post receives to match what he does.  that was the point of
ny suggested replacement.

> not to

I see that as the problem to be addressed.

On Tue, Apr 19, 2016 at 3:22 PM, Chuck Lever <[email-masked]> wrote:

> incoming message.  I don't think that is true, but if it is, an extension
> might be able to take advantage of it by posting a small set of large
> receives and a large set of more typically sized reads.
> >       • It is impossible for this value to be accurate, since the number
> >       • It leads to the sort of confusion discussed in section 4.4 of
> draft-ietf-nfsv4-rfc5666-implementation-experience-02
> >       • It is inconsistent with bidirectional operation.  While
> draft-ietf-nfsv4-rpcdrma-bidirection, I don't think we should say things
> here that make it harder or more confusing to adapt to bidirectional
> operation.
> >       • Replacing "requested credit value for the responder" by "credit
> > When sent in a message header not used to transfer an RPC message (e.g.
> RDMA_ERROR), the credit field is not used and should be ignored.
>
> handling of flow control on the responder.
>
> responsibility of the requester to post sufficient receives to ensure that
> the replies occasioned by the requests made of the responder can be
> properly received.
>
> draft-ietf-nfsv4-rfc5666bis-05, that we should consider moving this (minus
> the word "forward") to rfc5666bis.  Then the bidirection document could
> limit itself to two basic points:

---

From: Chuck Lever <[email-masked]>
Date: Wed, 20 Apr 2016 09:37:55 -0400

> On Apr 20, 2016, at 6:25 AM, David Noveck <[email-masked]> wrote:
> 
> I don't know.  I think that if it is not ignored, you have to specify whether it a request or a grant.  As I couldn't figure out how to do that, I punted and said it should be ignored.  YMMV.

I see.

rfc5666bis assumes only forward direction operation. In that
context it is clear whether the credit value is a request or
grant.

When forward and backward direction operation is enabled on
a connection, the context is no longer clear unless the call
direction field in the RPC message is consulted.

I can make the proposed change.

> > > The second paragraph as written is confusing.  While it seems to be written saying it applies to all receive 
> > > With regard to preposted receives on the requester endpoint, it is the responsibility of the requester to post 
> > > sufficient receives to ensure that the replies occasioned by the requests made of the responder can be 
> > > properly received.
> 
> It seems to but, if you examine section 4.3 of rfc5666bis, what you find is an explanation of the credit mechanism, and my understanding of that is that it is a mechanism to prevent the requester from sending too many requests. If you  read section 4.3.1 carefully, there is no indication of any connection to preposted receives on the requester and a lot of cases in which the lack of attention to requester resource indicates tellingly that only responder resources are addressed.  For example,
> 
> On occasion a requester or responder may need to adjust the amount o resources available to a connection.  When this happens, the responder needs to ensure that a credit increase is effected (i.e. RDMA Receives are posted) before the next reply is sent.
> 	• Only the responder is told to post additional receives in the event of a credit increase.  the requester is not told to do so. 
> 	• It is said that either may need to adjust resources and can do so.
> 	• The conclusion I draw is that although a requester may adjust resources and may post additional receives, that doesn't happen through the credit mechanism
> 
> To illustrate the point , let us suppose that you have a responder, which continues to grant ten credits for a while and then suddenly grants one hundred.  Because of that change, in line with the material quoted above, the responder has to post 90 additional receives before sending the reply with the new credit grant.
> 
> The requester does not have to post additional receives at this point.  He may have to but only if has more than ten outstanding requests at some point in the future.  The credit mechanism affects the choices he makes but he only has to post receives to match what he does.  that was the point of ny suggested replacement.
> 
> > not to

The endpoint handling requests does not know how many
requests to expect. So, a maximum is set, and both sides
agree to that. That endpoint posts enough Receives to
handle that maximum.

The endpoint handling replies always knows how many Receives
to post, since it knows how many requests are outstanding.
It has an opportunity to post the correct number of Receives
before sending another request.

IMO in RFC 5666 this is regarded as an implementation detail.
You reached the correct conclusion (above) without having
it written in the specification. That conclusion could be
mentioned as implementation guidance, however.

Conversely, does excluding such advice allow unwanted
behavior on the part of the requester?

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 20 Apr 2016 10:18:42 -0400

> IMO in RFC 5666 this is regarded as an implementation detail.

I think that's right.

> mentioned as implementation guidance, however.

It could.  It doesn't affect interoperability because the
responder is not affected.

> behavior on the part of the requester?

I would certainly consider getting the connection dropped
to be unwanted behavior.

On Wed, Apr 20, 2016 at 9:37 AM, Chuck Lever <[email-masked]> wrote:

> > > > With regard to preposted receives on the requester endpoint, it is
> the responsibility of the requester to post
> > It seems to but, if you examine section 4.3 of rfc5666bis, what you find
> is an explanation of the credit mechanism, and my understanding of that is
> that it is a mechanism to prevent the requester from sending too many
> needs to ensure that a credit increase is effected (i.e. RDMA Receives are
> posted) before the next reply is sent.
> >       • Only the responder is told to post additional receives in the
> >       • The conclusion I draw is that although a requester may adjust
> resources and may post additional receives, that doesn't happen through the
> credit mechanism
> >
> > The requester does not have to post additional receives at this point.
> He may have to but only if has more than ten outstanding requests at some