Subject: [nfsv4] Marshaling and receiving RDMA_NOMSG calls

From: Chuck Lever <[email-masked]>
Date: Thu, 22 Jan 2015 13:11:52 -0500

I’m working on adding RDMA_NOMSG support to the Linux client and server
implementations. After some internal discussion, I’ve found that I’m
not quite clear on what exactly is allowed when sending an RPC call
marshaled as an RDMA_NOMSG type message.

Let’s have a look at a symlink creation request over NFSv4. The
compound looks like this:

 { PUTFH, CREATE(NF4LNK) }

When the symlink pathname is large, the transport marshals this
compound as an RDMA_NOMSG type request because it is larger than the
inline threshold. (Note that NFSv2 SYMLINK also has content trailing
the symlink pathname, and thus may also be marshaled as an RDMA_NOMSG
type request).

In my prototype, the read list contains three chunk elements: one for
what would have been the RPC header, one for the pathname, and one for
trailing parts of the compound. The client expects the server to pull
all three elements via separate RDMA READs. (Not terribly efficient,
but this eliminates the need for the sender to allocate a large buffer
and copy the whole compound into it).

The rc_position field in each element is set to zero. As I understand
it, that makes the s/g elements part of the same chunk/RPC argument.
The read list therefore contains one chunk with three s/g elements.

The Solaris server rejects this request. It appears to expect that
each s/g element should have an rc_position field that is the exact
offset of the s/g element in the XDR stream.

I’m not 100% sure of my understanding of the protocol, so I’m asking
here. The first bit of relevant language in RFC 5666, section 3.4:
 
> Position
>         For data that is to be encoded, the position in the XDR
>         stream where the chunk would normally reside.  Note that
>         the chunk therefore inserts its data into the XDR stream
>         at this position, but its transfer is no longer "inline”.
>         Also note therefore that all chunks belonging to a single
>         RPC argument or result will have the same position.  For
>         data that is to be decoded, no position is used.

“All chunks belonging to a single RPC argument or result will have
the same position” seems to be on point. A sender is allowed to
encode a read list containing multiple s/g elements with the same
value in the rc_position field.

Further, my understanding is that elements that share the same
position are inserted by the receiver into the XDR stream contiguously
and in order, and the final element at that position is allowed to not
include XDR padding.

RFC 5666 is not clear whether RDMA_NOMSG must send only a single chunk
(ie, one XDR position in one or more s/g elements), or whether it means
simply “no inline content”. Section 5.1 says:
 
>    If the receiver gets an RPC-over-RDMA header with a message type 
>    of RDMA_NOMSG and finds an initial read chunk list entry with a
>    zero XDR position, it allocates a registered buffer and issues an
>    RDMA Read of the long RPC message into it.  The receiver then
>    proceeds to XDR decode the RPC message as if it had received it
>    inline with the Send data.  Further decoding may issue additional
>    RDMA Reads to bring over additional chunks.

The most curious part of this paragraph is “and finds an initial read
chunk list entry with a zero XDR position.” Is there a case where an
RDMA_NOMSG type request is sent and the first read chunk list entry
position is not zero? I can’t think of a valid RPC header that would
start with all zeroes or uninitialized buffer space. Thus, perhaps this
implies that the first read list element in an RDMA_NOMSG call MUST
have position zero.

Anyway, “Further decoding may issue additional RDMA Reads to bring
over addition chunks” immediately suggests that more than one s/g
element may appear in an RDMA_NOMSG type request.

However, it does not specify whether those additional chunks must
have position zero, or whether a sender may specify multiple chunks
(positions) in an RDMA_NOMSG type message.

Potentially, without any inline content, a sender could send a sparse
XDR stream by sending s/g elements with different positions, and the
receiver would simply leave the gaps uninitialized.

Or a sender could specify s/g elements with unique positions simply to
send each s/g element with XDR padding.

It seems that the Solaris server expects that multiple s/g elements in
an RDMA_NOMSG request will each have unique positions. My prototype
Linux client sends a set of s/g elements with the same position. My
reading of the text of RFC 5666 is that both of these encapsulations
are valid, and servers should accept both. Is this reading correct?

(Until recently the Linux server did not even examine the position
field in read list s/g elements. It is clearly not going to accept
the former style of RDMA_NOMSG request, if that style is indeed valid,
without additional work).

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Tom Talpey <[email-masked]>
Date: Fri, 23 Jan 2015 16:14:35 +0000

> -----Original Message-----
> From: nfsv4 [mailto:[email-masked]] On Behalf Of Chuck Lever
> Sent: Thursday, January 22, 2015 10:12 AM
> To: nfsv4 list ([email-masked])
> Cc: Chunli Zhang; Shirley Ma; Dai Ngo; Karen
> Subject: [nfsv4] Marshaling and receiving RDMA_NOMSG calls
> 
> I'm working on adding RDMA_NOMSG support to the Linux client and server
> implementations. 

Great!

> After some internal discussion, I've found that I'm
> Let's have a look at a symlink creation request over NFSv4. The

Actually, any operation may be marshaled as an RDMA_NOMSG. Conversely,
even a SYMLINK can be sent as a normal RDMA_MSG, if it fits in the inline
size. But, we can set those cases aside for this discussion.

> and copy the whole compound into it).

Understood, but it's also possible to register the three segments as a single
registered region, which would allow the server to pull them with a single
RDMA Read. Of course, the decision to do that is dependent on whether
it's a good performance tradeoff. For a rare operation such as SYMLINK,
the answer is probably "no". Again, we can set this aside.

BUT - certainly, the server needs to be able to deal with a multi-segment list.

> The read list therefore contains one chunk with three s/g elements.

This is correct, and it's also critical that the rc_position be zero. If it were
not zero, then the resulting message would have "missing" bytes, from
zero through the rc_position value (since RDMA_NOMSG by definition
has no inline payload which could provide them). The requirement for
setting rc_position to zero for RDMA_NOMSG is called out in RFC5666
section 5.1, although it does not actually state it as a MUST:

--5.1.  Message as an RDMA Read Chunk
--   One relatively simple method is to have the client identify any RPC
--   message that exceeds the RPC server's posted buffer size and move it
--   separately as a chunk, i.e., reference it as the first entry in the
--   read chunk list with an XDR position of zero.

> offset of the s/g element in the XDR stream.

Bug, unless it is rejecting for some other reason. The position of a chunk
must be the same for all its elements. The entire chunk (all three elements
in this case) are transferred as one, and the segments are assembled by
the server into a single XDR segment beginning at that xdr position.

A simple workaround, if that's of interest, is for the client to register the
three segments and send a single-element chunk, as described above.

> 
> I'm not 100% sure of my understanding of the protocol, so I'm asking
> >         at this position, but its transfer is no longer "inline".
> "All chunks belonging to a single RPC argument or result will have
> the same position" seems to be on point. A sender is allowed to

In effect the quoted text is saying that the rc_position of any non-initial
chunk element is effectively ignored. I thought the RFC has a stronger
statement somewhere regarding required equality, but I can't find it.

> and in order, and the final element at that position is allowed to not

Absolutely they must be contiguous and in-order, exactly as they would have
appeared in the XDR stream. If they're not contiguous, there would be "holes"
in the stream, this would lead to an XDR decode failure. If they're not in-order,
then a whole lot of existing XDR code won't work.

> simply "no inline content". Section 5.1 says:
> 
> The most curious part of this paragraph is "and finds an initial read
> chunk list entry with a zero XDR position." Is there a case where an
> position is not zero? I can't think of a valid RPC header that would

See above, yes they "must", because if they didn't the XDR stream would have
a hole. Technically speaking, it's not an RPC-RDMA requirement, it's an XDR
one. But, that's a technicality.

> 
> Anyway, "Further decoding may issue additional RDMA Reads to bring
> over addition chunks" immediately suggests that more than one s/g

It's important to distinguish between "chunks" and "chunk elements". You're
referring to "chunk elements" here, these are just the segments of some
logical part of the RPC message. Again, here, the RPC message itself (the
SYMLINK).

It's also possible that the RPC message is large, but *also* has a large argument
which the upper layer expect to transfer via RDMA. Say, a large NFSv4 COMPOUND
containing a WRITE. The large COMPOUND would be marshaled as the first
chunk, but the WRITE payload would be marshaled as a second chunk, just
as it normally would. You'd have:

Message: <rpc header><NFSv4 COMPOUND <op1, op2, WRITE<data>, op3>

This would marshal as two chunks:
Chunk position 0: <rpc header><NFSv4 COMPOUND <op1, op2, WRITE, op3>
Chunk position N: <data>

Position N depends on where the write data lands in the actual XDR stream.

I hope that helps.

> receiver would simply leave the gaps uninitialized.

Which would fail to decode at the XDR layer.

> send each s/g element with XDR padding.

Not needed, since they all belong to the same chunk, which has a position.
And, if the following positions were not contiguous and ordered, they'd result
In invalid XDR streams, so it's a meaningless requirement to require them
to be significant.

> are valid, and servers should accept both. Is this reading correct?

Your prototype client is correct, and the server is incorrect, per the above.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 23 Jan 2015 13:03:38 -0500

Hi Tom-

Thanks for your response. More below.

>> Subject: [nfsv4] Marshaling and receiving RDMA_NOMSG calls

 [ . . . snip . . . ]

> the answer is probably "no". Again, we can set this aside.

The question for my prototype client is how easy is it to fit the
marshaling of RDMA_NOMSG type calls into the existing logic.
Marshaling the separate sections is the simplest path from A to B.

There is also the issue of what to do when local adapter capabilities
allow only page-size s/g elements. We have to deal with the
possibility of multiple chunk elements in an RDMA_NOMSG type call,
so that is explicitly a part of this prototype.

It’s been suggested that as part of the finished solution, I should
punt and go with RDMA_MSG with the pathname only in the read list,
since that is always going to be more efficient.

That is probably what will happen for SYMLINK. And I presume that
NFSv4 CREATE(NF4LNK) will be specifically grandfathered to allow the
use of RDMA_MSG with read list.

Still, other requests, such as SETACL, will need RDMA_NOMSG to work.
I should probably consider the single RDMA READ approach for these.

As part of an RFC 5667 refresh, I’d like to consider new operations
in NFSv4 which might benefit from RDMA_MSG with only the data payload
argument in the read list: SETACL and named attribute operations, for
example. These would benefit from efficiency in situations where
entire trees are copied (rsync, cpio, and so on).

> --   read chunk list with an XDR position of zero.

OK, so your intent is that discontiguous XDR streams are not allowed
for RDMA_NOMSG. I think it would be helpful if the RFC stated that
explicitly. The current RFC language doesn’t appear to exclude this
possibility.

One could read the remarks about implicit chunk padding and assume
that receivers would handle discontiguous XDR streams the same way.
Never mind that in most cases, that would be a nonsense RPC request.

>>>        data that is to be decoded, no position is used.

The above definition asserts a one-to-one mapping between a single
RPC argument and one Position.

For example, the data payload of an NFS WRITE operation conveyed via
an RDMA_MSG type call is currently sent in one or more chunk elements
at the same (non-zero) Position.

But an RDMA_NOMSG type call can be sent in a single chunk element.
In fact, that is the common case with Solaris, for instance. The
entire RPC message, including all RPC arguments, is contained in a
single chunk element. We agree this is a compliant use of RDMA_NOMSG.

Thus, there’s no mapping between RPC arguments and Positions for
RDMA_NOMSG. This is distinct from RDMA_MSG. I think it should be
called out in the RFC, since it seems inconsistent with the
existing definition of Position.

Or, perhaps the inconsistency is only for Position zero? See your
example below.

> three segments and send a single-element chunk, as described above.

You’ve confirmed that a server MUST handle the case where a client
sends more than one chunk element in Position zero.

> logical part of the RPC message. Again, here, the RPC message itself (the

Exactly right. Sorry for the confusion.

> Position N depends on where the write data lands in the actual XDR stream.

It makes my head hurt a little :-)

In other words, it is perfectly compliant to marshal the
CREATE(NF4LNK) operation as a chunk at position zero containing
the inline parts of the RPC message, and a chunk containing the
pathname. I hadn’t realized that was a possibility.

---

From: Tom Talpey <[email-masked]>
Date: Fri, 23 Jan 2015 19:16:00 +0000

> so that is explicitly a part of this prototype.

To be clear, and I think you agree with this - the memory registration choice by
the client is entirely a local matter not specified by the protocol, and in fact
the protocol allows a wide variety of approaches. The choice does have certain
implications on how the request is encoded, of course.

> Thus, there's no mapping between RPC arguments and Positions for

There is definitely a one-to-one relationship between RPC arguments and Positions.
There is also a one-to-one relationship between RDMA-eligible chunks and these.
The funny thing about RDMA_NOMSG is that it elevates the inline portion to use
the RDMA transfer model of a large RPC Argument. So there's an "extra" chunk.

Put another way, I believe you're pointing out that if the inline (NOMSG) portion
includes XDR data which has positions both before and after such an argument,
it's a little weird that they all simply state position zero. I guess I agree.

It also means the receiver needs to inspect the XDR stream a bit more deeply to
figure out "where the data goes". Off the top of my head, I see three possibilities:

1) Leaving the rules as they are and requiring the receiver to fetch and reassemble
all chunks before processing.
2) setting, and following, the rc_position fields in each chunk element of the initial
NOMSG read chunk
3) encoding additional read chunks with position != 0

So my example

Would be the following:

1)
(i.e, no change)

2)
Chunk element [ position 0: <rpc header><NFSv4 COMPOUND <op1, op2, WRITE> ]
            element [ position N+data: <, op3> ]
Chunk position N: <data>

Or...

3)
Chunk position 0: <rpc header><NFSv4 COMPOUND <op1, op2, WRITE >
Chunk position N: <data>
Chunk position N+data: <, op3>

The difference is a bit subtle. It's probably worth getting some implementor feedback.

The main issue I see with option #3 is that it is very unclear to the receiver up-front that
the second chunk belongs to an operation, while the third chunk doesn't. That would take
some (very) clever language in RFC5667. Even #2 is tricky, because it implies extra RDMA
Read operations, which may be unnecessary.

Tom.

> From: Chuck Lever [mailto:[email-masked]]
> Sent: Friday, January 23, 2015 10:04 AM
> To: Tom Talpey
> Cc: nfsv4 list ([email-masked]); Karen; Dai Ngo; Chunli Zhang; Shirley Ma
> Subject: Re: [nfsv4] Marshaling and receiving RDMA_NOMSG calls
> 
> Hi Tom-
> 
> It's been suggested that as part of the finished solution, I should
> As part of an RFC 5667 refresh, I'd like to consider new operations
> explicitly. The current RFC language doesn't appear to exclude this
> possibility.
> 
> Also, let's revisit the definition of Position for a moment:
> 
> >>> Position
> Thus, there's no mapping between RPC arguments and Positions for
> You've confirmed that a server MUST handle the case where a client
> pathname. I hadn't realized that was a possibility.
> 
> --
> Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Fri, 23 Jan 2015 15:58:18 -0500

> the RDMA transfer model of a large RPC Argument. So there's an "extra" chunk.

I think the one-to-one mapping rule explicitly does not apply to the
chunk in Position zero. Otherwise, sending a whole RPC in one chunk
would be invalid, since that chunk contains multiple RPC arguments.
That makes Position 0 exactly like the inline portion of a message,
which seems to be the intent of RDMA_NOMSG.

Even sending the first portion of the RPC message in Position zero
and later arguments in separate Positions means Position zero
contains more than one RPC argument.

> it's a little weird that they all simply state position zero.

Yes, that induced a small migraine.

> Chunk position 0: <rpc header><NFSv4 COMPOUND <op1, op2, WRITE, op3>

This seems consistent with the idea that Position 0 is special
(Position 0 contains all the inline content, and the one-to-one
mapping rule does not apply to it).

> Chunk position N+data: <, op3>

This is similar to how the Linux client (incorrectly) sends NFSv4
WRITE today:

Inline: <rpc header><NFSv4 COMPOUND <PUTFH, WRITE >
Chunk position N: <data>
Chunk position N: <, GETATTR>

Would this be acceptable if the last chunk was at Position N+data?

> The difference is a bit subtle. It's probably worth getting some implementor feedback.

My impression is that the current Solaris server expects 3).

The Linux server currently supports RDMA_NOMSG where all parts of
the message are conveyed in Position 0, contained in one or more
chunk elements.

I want to verify that the following is still valid:

Chunk element, position 0: <rpc header><NFSv4 COMPOUND <op1, CREATE>
Chunk element, position 0: <pathname data>
Chunk element, position 0: <rest of CREATE arguments>

and so is:

Chunk position 0: <rpc header><NFSv4 COMPOUND <op1, CREATE>

I like this solution best simply because the RPC/RDMA transport does
not need any special knowledge about the arguments to marshal the
request. This way is straightforward for NFS client implementations
that do not currently distinguish between RPC arguments and a large
data payload.

But allowing both 1) and “All chunk elements in Position zero” seems
workable.

---

From: Tom Talpey <[email-masked]>
Date: Fri, 23 Jan 2015 21:32:44 +0000

More agreement below. :-)

> Sent: Friday, January 23, 2015 12:58 PM
> To: Tom Talpey
> Cc: nfsv4 list ([email-masked])
> chunk in Position zero. 

Right, because position zero is not an RPC Argument! The RFC uses this term
rather formally to mean a specific parameter to a specific RPC method, e.g.
in the case of WRITE, the data to be written. The filehandle and count are not
eligible for RDMA. This "mapping" is the reason RFC5667 exists.

So when I say that RDMA_NOMSG is an exception, that's why it is necessary
to tease apart the meaning of Position. You hit on a good point in other words.

>Otherwise, sending a whole RPC in one chunk

Hmm, maybe. It would certainly be more correct, because position N is not.
However, it means the upper layer would see a chunk that it doesn't know
what to do with. That is, unless we work out some new rules for RFC5667.

> Chunk element, position 0: <rest of CREATE arguments>

You mean as the position-0 read chunk of an RDMA_NOMSG? Yes.

> that do not currently distinguish between RPC arguments and a large

That is exactly the goal of the RPC/RDMA protocol. It was intended to be
simply an encoding of a segmented, RDMA-able representation of an
abstract xdr stream. Nothing is tagged, or inserted, it is simply re-swizzled
as a chunk list, and that information is prepended for the receiver.

That's also why RFC5667 exists - for NFS to instruct RPC/RDMA where the
segment lines can be drawn. That is requires a bit more thinking for
RDMA_NOMSG and large NFSv4 COMPOUNDs, as we see.

Tom.

> 
> But allowing both 1) and "All chunk elements in Position zero" seems
> workable.
> 
> >>>> what would have been the RPC header, one for the pathname, and
> one
> >> for

---

From: Chuck Lever <[email-masked]>
Date: Tue, 27 Jan 2015 12:53:06 -0500

[ Snipping out some of last week’s conversation ]

>> mapping rule does not apply to it).

I can’t find anything wrong with 1), as long as each non-zero
chunk position conveys only one RPC argument.

However, IMO upper layer bindings such as RFC 5667 should
stipulate which operations are allowed to use this style of
RDMA_NOMSG, in the same way they stipulate which operations may
use the read list with RDMA_MSG.

Or we can simply make a blanket declaration that this style of
RDMA_NOMSG is allowed only when the upper layer binding allows
an operation to use the read list with RDMA_MSG.

RDMA_NOMSG type calls which use only position zero can be used
at any time, I assume.

>>>           element [ position N+data: <, op3> ]

We’re not clear what 2) is doing. I assume you are simply splitting
chunk position zero into two elements. Thus it doesn’t seem different
than 1).

Is there a requirement that each chunk element must contain only
a specific portion of the RPC message? If not, the elements that
make up Position zero MAY be divided arbitrarily.

They would have to be divided this way, for instance, if an upper
layer is attempting to send a series of large data arguments and
the transport is forced to split these at page boundaries to
accommodate adapter limitations.

Let me know if I’ve misunderstood.

>>> Chunk position N+data: <, op3>

3) seems to violate the rule that a non-zero chunk position must
carry no more than a single RPC argument. In this case, Position
N+data is conveying an entire NFSv4 operation with all of its
arguments.

---

From: Tom Talpey <[email-masked]>
Date: Tue, 27 Jan 2015 18:16:25 +0000

> Sent: Tuesday, January 27, 2015 12:53 PM
> To: Tom Talpey
> [ Snipping out some of last week's conversation ]
> 
> I can't find anything wrong with 1), as long as each non-zero

Correct, and it's always required that a single RPC argument map to a single RPC/RDMA chunk.

> use the read list with RDMA_MSG.

Agreed.

> RDMA_NOMSG type calls which use only position zero can be used

My preference is to not make exceptions on a per-upper-layer basis. The goal should
be to define a lower-layer encoding which can be implemented more generally.

It was certainly the goal of RFC5666 to allow RDMA_NOMSG to be used at any time.,

> >>> 2)
> >>> Chunk element [ position 0: <rpc header><NFSv4 COMPOUND <op1,
> op2,
> >> WRITE> ]
> We're not clear what 2) is doing. I assume you are simply splitting
> chunk position zero into two elements. Thus it doesn't seem different
> than 1).

It is different from 1) only in the sense that the Position is explicitly non-zero.

> make up Position zero MAY be divided arbitrarily.

In the RDMA_NOMSG case, the goal is that the XDR stream be fully encoded without
any holes, and without any ambiguity.

> Let me know if I've misunderstood.

Nope.

> N+data is conveying an entire NFSv4 operation with all of its

Yes, it does. This approach would require rewriting that section of RFC5666.

Tom.