Subject: [nfsv4] Review of draft-cel-nfsv4-rpcrdma-version-two

From: Karen <[email-masked]>
Date: Wed, 8 Jun 2016 16:09:32 -0600

2.1

- "A connection's inline threadhold determines when full RDMA Read or 
Write operations..."

I understand what this is saying but the term "full RDMA Read or Write", 
threw me at first, wasnt originally sure what "full" really meant here.

- Typo, assume "operations are required because the RPC message to be 
send is larger"...send, should be sent.

- "If an RPC message does not contain DDP-eligible data items, a 
requester prepares a Long Call or Reply to convey the whole RPC message 
using an RDMA Read or Write"

Couldn't it still do this even if the RPC messaged did NOT contain 
DDP-eligible data times?  Maybe I'm confused what this is really saying, 
I take it to mean if the RPC message size exceeds the inline threshold, 
the message can be sent via a Long Call or Reply.

  - "the cost of registering and unregistering the memory where it 
resides",

where it resides seems too wordy, maybe simply "it", i.e "the cost of 
registering and unregistering it" (nit)

2.2

- "quadruples the default inline threshold size"

using the number 4096 is more clear than quadruples.

I believe that this means the default reply buffer of the requestor is 
also 4096 now also, correct? Not really clearly stated.

2.3

- "A mechanism for discovering a peer's preferred inline threshold value 
may be used to optimize RDMA Send operations further"

Where do I find out more info about this mechanism?  Is it a standard 
mechanism? Or a private mechanism? Or one of the new optional mechanisms?

- "The new default inline threshold size is no larger than the size of a 
hardware page on typical platforms.  This conserves the resources needed 
to Send and Receive base level RPC-over-RDMA Version Two messages, 
enabling RPC-over-RDMA Version Two to be used on a broad base of hardware"

If the inline threshold is larger than a page size, would that impact 
what hardware base it could be used on?

3.2
- "A requirement that an RPC Payload Stread MUST or MUST NOT follow the 
Transport Header Stream for each new opttype code added"
Are you meaning that it must define if it is RPC_MSG or RPC_NOMSG? If so 
using these terms here would be a bit more clear.

4.

"When the rdma_proc field has the value RDMA_MSG, an RPC payload Stream 
MUST follow the Transport Header Stream in the Send buffer"

Is this sentence needed?  Isn't this already a given based on version 1?

"When the rdma_proc field has any other value, and RPC Payload Stream 
MUST NOT follow the Transport Header Stream".

Is this new in v2?  Or is is just something that was not clarified in v1?

"Error handling works the same way in RPC-over-RDMA Version Two as it 
does in RPC-over-RDMA Version one, with one change described in Section 
3.1."

Adding something like "with the addition of the RDMA_ERR_INVAL_OPTION as 
described in Section 3.1" would be helpful.  Prevents me from having to 
go back and re-read all 4 paragraphs of section 3.1 to understand what 
you were referring to.

5.1

This section implies that the requestor should only send 1 request after 
the initial connection was created, until it determines the version and 
inline size.

The server will not know what size recv. buffers to post until at least 
the version is known.  It can start with 1k obviously to receive the 
first request.  Seems the server will not want to post RDMA_CREDIT 
number of recv buffers until it knows the version and size to expect.

However it if only posts 1, but the client sees the connection and sends 
more than one request, the connection can be closed if there are not 
enough posted recv. buffers.

Since the RDMA_CREDIT the server defines is not known to the client 
until the first request, the client doesnt really know how many initial 
requests it can send.  Seems like the inital rdma_credit should be 
assumed to be only 1 until a reply is received from the server.  Maybe 
this is already documented somewhere but I don't recall reading it.

5.2

"If the connection is lost immediately after the RDMA_ERROR reply is 
received, a requestor can avoid a possible version negotaion loop when 
re-establishing anohter connection by assuming htat particular responder 
does not support RPC-over-RDMA Version Two"

What if, the connection was lost because the server was being rebooted 
after an upgrade, which now does support version 2?

"To permit inline threshold values to change during negotiation of 
protocl version, RPC-over-RDMA Version Two implementations MUST allow 
inline threshold values to change wihtout triggering a connection loss"

If the connection allows more than 1 request before the initial 
negotiation, this might be difficult in the case where the server 
supports v2, the client supports v1, and now the server sees the client 
only supports v1, but had pre-posted rdma credit v2 size recv buffers, 
but wants to replace then with v1 size recv buffers.  How is this done 
cleanly if the client is sending more than 1 request during the initial 
negotiation?

Karen

---

From: Chuck Lever <[email-masked]>
Date: Thu, 9 Jun 2016 14:37:25 -0400

Hi Karen-

Thanks for the review comments. I'll queue the editorial
remarks for the next revision, and answer your questions
inline:

> On Jun 8, 2016, at 6:09 PM, Karen <[email-masked]> wrote:
> 
> 2.1
> 
> - "A connection's inline threadhold determines when full RDMA Read or Write operations..."
> 
> I understand what this is saying but the term "full RDMA Read or Write", threw me at first, wasnt originally sure what "full" really meant here.
> 
> - Typo, assume "operations are required because the RPC message to be send is larger"...send, should be sent.
> 
> - "If an RPC message does not contain DDP-eligible data items, a requester prepares a Long Call or Reply to convey the whole RPC message using an RDMA Read or Write"
> 
> Couldn't it still do this even if the RPC messaged did NOT contain DDP-eligible data times?  Maybe I'm confused what this is really saying, I take it to mean if the RPC message size exceeds the inline threshold, the message can be sent via a Long Call or Reply.

The text is describing implementation, in order to supplement the
definition of inline threshold. There's no MUST or SHOULD here.

> - "the cost of registering and unregistering the memory where it resides",
> 
> where it resides seems too wordy, maybe simply "it", i.e "the cost of registering and unregistering it" (nit)
> 
> 2.2
> 
> I believe that this means the default reply buffer of the requestor is also 4096 now also, correct? Not really clearly stated.
> 
> 2.3
> 
> - "A mechanism for discovering a peer's preferred inline threshold value may be used to optimize RDMA Send operations further"
> 
> Where do I find out more info about this mechanism?  Is it a standard mechanism? Or a private mechanism? Or one of the new optional mechanisms?

It hasn't been defined. This text leaves a space for future work.

> - "The new default inline threshold size is no larger than the size of a hardware page on typical platforms.  This conserves the resources needed to Send and Receive base level RPC-over-RDMA Version Two messages, enabling RPC-over-RDMA Version Two to be used on a broad base of hardware"
> 
> If the inline threshold is larger than a page size, would that impact what hardware base it could be used on?

Possibly. Some hardware has a very low max_sge.

Today our clients build an RPC message in two separate buffers
and use a gather Send. Now if the inline threshold is larger
than a page, more sges are needed per Send.

> 3.2
> - "A requirement that an RPC Payload Stread MUST or MUST NOT follow the Transport Header Stream for each new opttype code added"
> Are you meaning that it must define if it is RPC_MSG or RPC_NOMSG? If so using these terms here would be a bit more clear.

That bullet has been replaced in a more recent revision of
this document.

> 4.
> 
> "When the rdma_proc field has the value RDMA_MSG, an RPC payload Stream MUST follow the Transport Header Stream in the Send buffer"
> 

It is stated here for completeness, as V2 adds rdma_proc types
where this property must be stated explicitly.

> "When the rdma_proc field has any other value, and RPC Payload Stream MUST NOT follow the Transport Header Stream".
> 

Yes, this is new in V2. This text has also been updated in a
more recent revision of this document.

> "Error handling works the same way in RPC-over-RDMA Version Two as it does in RPC-over-RDMA Version one, with one change described in Section 3.1."
> 
> Adding something like "with the addition of the RDMA_ERR_INVAL_OPTION as described in Section 3.1" would be helpful.  Prevents me from having to go back and re-read all 4 paragraphs of section 3.1 to understand what you were referring to.
> 
> 5.1
> 
> This section implies that the requestor should only send 1 request after the initial connection was created, until it determines the version and inline size.
> 
> The server will not know what size recv. buffers to post until at least the version is known.  It can start with 1k obviously to receive the first request.  Seems the server will not want to post RDMA_CREDIT number of recv buffers until it knows the version and size to expect.
> 
> However it if only posts 1, but the client sees the connection and sends more than one request, the connection can be closed if there are not enough posted recv. buffers.
> 
> Since the RDMA_CREDIT the server defines is not known to the client until the first request, the client doesnt really know how many initial requests it can send.  Seems like the inital rdma_credit should be assumed to be only 1 until a reply is received from the server.  Maybe this is already documented somewhere but I don't recall reading it.

Yes, it is already documented. rfc5666bis Section 4.3.3 describes
the initial interaction between RPC-over-RDMA client and server.
Only 1 request is allowed until the client has received a credit
grant greater than 1.

> 5.2
> 
> "If the connection is lost immediately after the RDMA_ERROR reply is received, a requestor can avoid a possible version negotaion loop when re-establishing anohter connection by assuming htat particular responder does not support RPC-over-RDMA Version Two"
> 
> What if, the connection was lost because the server was being rebooted after an upgrade, which now does support version 2?

Then the client reconnects with V1. If the server supports V1,
then we're off to the races. If it doesn't, it reports
RDMA_ERROR, and the client renegotiates.

> "To permit inline threshold values to change during negotiation of protocl version, RPC-over-RDMA Version Two implementations MUST allow inline threshold values to change wihtout triggering a connection loss"
> 
> If the connection allows more than 1 request before the initial negotiation, this might be difficult in the case where the server supports v2, the client supports v1, and now the server sees the client only supports v1, but had pre-posted rdma credit v2 size recv buffers, but wants to replace then with v1 size recv buffers.  How is this done cleanly if the client is sending more than 1 request during the initial negotiation?

See above: operation is 1 request at a time until the client
has received a credit grant greater than 1.

--
Chuck Lever