Subject: Re: [nfsv4] RPC/RDMA read chunk round-up when inline content follows

From: David Noveck <[email-masked]>
Date: Thu, 1 Jan 2015 07:20:42 -0500

> It turns out that Linux is the only NFS client that supports RDMA and
> adds an operation (GETATTR) after WRITE in an NFSv4 compound.

Presumably it sends the GETATTR as a separate chunk.  I suppose that
could be a performance issue.

> Linux sends the part of the compound before the opaque file data
> via RDMA SEND. The opaque file data is put into the read list. Then
> the GETATTR operation is put in the read list after the file data.

> Existing NFS/RDMA servers are able to receive and process this
> request correctly.

The interesting question is whether they would work if one switched from
the chunks-only-at-the-end approach to the alternative chunks-only-for-WRITE
approach.

> First question: does the Linux client comply with RFCs 5666 and 5667
> when it sends an NFSv4 WRITE compound in this way, or should it be
> changed?

It clearly complies with RFC5666 which grants the sender a lot of freedom
for the sender as to how it chooses to sends individual data elements.

> The intention of RFC 5667 appears to be that the GETATTR belongs
> inline, following the front of the compound.

I think the basic idea is that RDMA is for large data elements, such as
data to be written, but I don't think it comes out and forbids you from
sending
other things.

The only basis on which you might base a case that this violates RFC5667
wiule be based on the following text from section2:

Large chunks of data,

such as the file data of an NFS WRITE request, MAY be referenced by

an RDMA Read list and be moved efficiently and directly placed by an

RDMA Read operation initiated by the server.

One could argue that this somehow implies you "MAY NOT" transfer other
sorts
of request data using RDMA read chunks.  I don't read it that way.

> Only the file data
> payload belongs in the read list, if NFSv4 Direct is to be consistent
> with NFSv2 and NFSv3.

I don't think NFSv4 can be consistent with previous NFS protocols because
it is different in having COMPOUND.  Your choice is either to be consistent
with NFSv3:

   - In having chunks only at the end of the request.
   - In having chunks only for WRITE data.

> Now suppose the Linux client is changed to place the GETATTR operation
> inline, following the front of the compound.

And assuming the existing servers are OK with that?

> When the opaque file data requires round-up because it’s length is not
> divisible by four, should there be a pad?

No.

> if so, where does it belong?

> To put it another way: Since inline data does not have a “position”
> field, should the receiver assume that inline content following the
> read list begins right at the end of the rounded-up chunk?

No.

> or should
> the receiver assume the inline content begins at the next XDR position
> after the end of the chunk?

Yes.

> If the former, that would require inserting a zero pad before the
> inline content. But that pad would cause the following inline content
> to be sent unaligned, since a zero pad is always shorter than 4 bytes
> long.

> By implication, then, the receiver MUST conclude that round-up is
> present in the case when inline data remains to be decoded (ie, the
> following inline content always begins at the next XDR position). The
> sender MUST NOT send a pad inline in this case. Is this correct?

Yes.

> I've read RFC 5666 section 3.7. Paragraph 4 seems most relevant, but
> it doesn't discuss the case where there is additional inline content
> following a chunk list.

I think the following is relevant:

Because this position will not match (else roundup would not

have occurred), the receiver decoding will fall back to inspecting

the remaining inline portion.

This may not be clear enough. Maybe a clarifying editorial errata would
be justified.

On Tue, Dec 30, 2014 at 12:53 PM, Chuck Lever <[email-masked]>
wrote:

> Hi-
>
> It turns out that Linux is the only NFS client that supports RDMA and
> adds an operation (GETATTR) after WRITE in an NFSv4 compound.
>
> Linux sends the part of the compound before the opaque file data
> via RDMA SEND. The opaque file data is put into the read list. Then
> the GETATTR operation is put in the read list after the file data.
>
> Existing NFS/RDMA servers are able to receive and process this
> request correctly.
>
> First question: does the Linux client comply with RFCs 5666 and 5667
> when it sends an NFSv4 WRITE compound in this way, or should it be
> changed?
>
> The intention of RFC 5667 appears to be that the GETATTR belongs
> inline, following the front of the compound. Only the file data
> payload belongs in the read list, if NFSv4 Direct is to be consistent
> with NFSv2 and NFSv3
>
> Now suppose the Linux client is changed to place the GETATTR operation
> inline, following the front of the compound.
>
> When the opaque file data requires round-up because it’s length is not
> divisible by four, should there be a pad? if so, where does it belong?
>
> To put it another way: Since inline data does not have a “position”
> field, should the receiver assume that inline content following the
> read list begins right at the end of the rounded-up chunk? or should
> the receiver assume the inline content begins at the next XDR position
> after the end of the chunk?
>
> If the former, that would require inserting a zero pad before the
> inline content. But that pad would cause the following inline content
> to be sent unaligned, since a zero pad is always shorter than 4 bytes
> long.
>
> By implication, then, the receiver MUST conclude that round-up is
> present in the case when inline data remains to be decoded (ie, the
> following inline content always begins at the next XDR position). The
> sender MUST NOT send a pad inline in this case. Is this correct?
>
> I’ve read RFC 5666 section 3.7. Paragraph 4 seems most relevant, but
> it doesn’t discuss the case where there is additional inline content
> following a chunk list.
>
> Thanks for reading!
>
> --
> Chuck Lever
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever <[email-masked]>
Date: Thu, 1 Jan 2015 13:51:24 -0500

Hi Dave-

Sorry for the length. There’s a lot to tease apart.

> could be a performance issue.

Or efficiency: extra work has to be done to register that 16 byte
piece of the compound RPC and send it in a separate RDMA READ. That’s
extra transport bytes on the wire, and an extra round trip (albeit a
very fast one).

> the chunks-only-at-the-end approach to the alternative chunks-only-for-WRITE

The current Linux NFS server and the Solaris 11 update 2 NFS server do
not handle a request with additional inline content. There is a minor
exception, but let’s put that aside for the moment.

Recall that the Linux NFS client sends { PUTFH, WRITE, GETATTR } as its
NFSv4 WRITE compound. I’ve constructed a Linux client that sends the
GETATTR as additional inline content rather than at the end of the read
list.

The Linux server accepts the PUTFH and WRITE, but then returns 10044
(OP_ILLEGAL) for the third op, because it finds nonsense when it looks
in the XDR stream for the arguments of the third op. (I do have a fix
for this).

I can make the converse point: sending an RPC request where only a
middle argument is placed in a chunk list is clearly allowed by RFC
5666, and is not forbidden by RFC 5667. Having “position” fields for
each chunk in the read list means a server MUST be prepared to accept
inline content following a read list

Therefore current NFS/RDMA server implementations are broken. If we
agree that sending the final GETATTR in the read list is allowed,
then servers should accept that in addition to NFSv4 compounds where
the final GETATTR is sent as additional inline content.

> data to be written, but I don't think it comes out and forbids you from sending
> other things.

This text from section 4 seems to disallow the use of a read list for
anything but opaque file data or symlink pathnames:

"Similarly, a single RDMA Read list entry MAY be posted by the client
 to supply the opaque file data for a WRITE request or the pathname
 for a SYMLINK request.  The server MUST ignore any Read list for
 other NFS procedures, as well as additional Read list entries beyond
 the first in the list."

“The server MUST ignore” is a clear statement that placing other
arguments in a read list will prevent interoperation, at least for
NFSv2 and NFSv3 Direct.

The “MAY” refers to an alternate mechanism of sending large RPC
requests: RDMA_NOMSG, where the entire RPC message is encapsulated in
a read list at position 0.

> One could argue that this somehow implies you "MAY NOT" transfer other sorts 

> it is different in having COMPOUND.

I don’t agree that NFSv4 cannot be consistent with legacy NFS. Perhaps
it is not stated clearly in RFC 5667 section 5 how to make NFSv4
equivalent to NFSv2/3 Direct, but observe that:

1. If you consider only WRITE operations, sending only the opaque file
   data payload is allowed in all three cases.

2. If you consider NFSv2 SYMLINK, there is an argument (sattr)
   following the link pathname argument. RFC 5667 says only the link
   pathname may be sent in a read list. This is a clear requirement
   that the middle argument is conveyed via a read list, and the
   remaining arguments inline. The case where both the pathname and
   the sattr argument are sent in a read list is explicitly not
   allowed.

The legacy-consistent way of sending an NFSv4 WRITE compound would be
to put only the opaque file data in a read list, no matter what else
appears in the NFSv4 COMPOUND request with the WRITE operation. (Small
WRITEs of course are always sent inline).

> Your choice is either to be consistent
> with NFSv3:
> 	• In having chunks only at the end of the request.
> 	• In having chunks only for WRITE data.

RFC 5667 section 5 has two sentences discussing the use of read lists
with NFSv4 COMPOUND requests:

"The situation is similar for RDMA Read lists sent by the client and
 applies to the NFSv4.0 WRITE and SYMLINK procedures as for v3.
 Additionally, inline segments too large to fit in posted buffers MAY
 be transferred in special "RDMA_NOMSG" messages."

Although it does suggest strongly that NFSv4 WRITE should be consistent
with NFSv3 WRITE, at first glance it doesn’t help us decide which of
your two bullets is the correct interpretation.

“The situation is similar” I believe references earlier text in section
5, not to the text in section 4 that discusses WRITE/SYMLINK. Earlier,
section 5 goes as far as to say:

"The Write list MUST be considered only for the COMPOUND procedure.
 This procedure returns results from a sequence of operations.  Only
 the opaque file data from an NFS READ operation and the pathname from
 a READLINK operation MUST utilize entries from the Write list."

The “Only . . . MUST” construction in the third sentence is slippery.

It can be read as allowing other operations and arguments to use a
write list, and as requiring READ and READLINK to use a write list in
all cases, including short payloads. That reading, while literal, seems
contradictory with the rest of the document.

Or it can be read as allowing a write list only for READ and READLINK
operations in NFSv4 compounds. That reading I believe is consistent
with the rest of the document and earlier versions of the NFS Direct
protocol, and would also apply to WRITE in the read list case.

Also worth mentioning is that NFSv4 does not have a separate SYMLINK
operation. Thus the explicit mention of an NFSv4 SYMLINK operation in
section 5 is incorrect. It could be replaced with CREATE(NF4LNK). Or
using a read list during symlink creation could simply be abandoned
for NFSv4, in favor of sending large CREATE requests via RDMA_NOMSG.

In sum, a very particular reading of section 5 suggests that only WRITE
payloads in NFSv4 COMPOUND requests can be sent in a read list, but
there seems to be some wiggle room. It would help me, as an implementer,
if RFC 5667 section 5 could be clarified.

> > sender MUST NOT send a pad inline in this case. Is this correct?

Thanks for confirming my interpretation, and for helping me to sharpen
my understanding of these RFCs.

> This may not be clear enough.  Maybe a clarifying editorial errata would

Let me know if you’d like me to file one.

> On Tue, Dec 30, 2014 at 12:53 PM, Chuck Lever <[email-masked]> wrote:
> Hi-
> 

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Tom Talpey <[email-masked]>
Date: Mon, 5 Jan 2015 18:50:44 +0000

I hope this additional reply doesn't make the thread too convoluted.

> -----Original Message-----
> From: nfsv4 [mailto:[email-masked]] On Behalf Of Chuck Lever
> Sent: Thursday, January 1, 2015 1:51 PM
> To: David Noveck
> Cc: Chunli Zhang; nfsv4 list ([email-masked]); Karen; Dai Ngo
> Subject: Re: [nfsv4] RPC/RDMA read chunk round-up when inline content
> follows
> 
> Hi Dave-
> 
> Sorry for the length. There's a lot to tease apart.
> 
> 
> > > It turns out that Linux is the only NFS client that supports RDMA
> > > and adds an operation (GETATTR) after WRITE in an NFSv4 compound.
> >
> Or efficiency: extra work has to be done to register that 16 byte piece of the
> compound RPC and send it in a separate RDMA READ. That's extra transport
> bytes on the wire, and an extra round trip (albeit a very fast one).

It's my opinion this is a bug, in both the client and the server here.

The intention of an RDMA Chunk is that it encodes all or part of an RPC
argument, RPC result, or an entire RPC message. The fact that the client
added a compounded GETATTR to the write data's Read Chunk is wrong.
The fact that the server parsed the additional data after the write payload
as an RPC request is also wrong. Keeping these segments separate is critical
to the RFC5667 specification of the NFS binding to RPC/RDMA.

That said, the text in RFC5667 may be unclear. I'd suggest that it needs to
be revisited anyway, to reflect implementation experience as well as the
new behaviors of NFSv4.1, v4.2 etc. The document is five years old, and it
reflects work done prior to that. In particular, while it was published at the
same time as RFC5661 (NFSv4.1), it actually doesn't make any specific
requirements for that minor version beyond mentioning the callback
channel. That, plus pNFS, layouts, CREATE, OPENATTR, ACLs, etc.

> 
> > > Linux sends the part of the compound before the opaque file data via
> > > RDMA SEND. The opaque file data is put into the read list. Then the
> > > GETATTR operation is put in the read list after the file data.
> >
> > The interesting question is whether they would work if one switched
> > from the chunks-only-at-the-end approach to the alternative
> > chunks-only-for-WRITE approach.
> 
> The current Linux NFS server and the Solaris 11 update 2 NFS server do not
> handle a request with additional inline content. There is a minor exception,
> but let's put that aside for the moment.
> 
> NFSv4 WRITE compound. I've constructed a Linux client that sends the
> GETATTR as additional inline content rather than at the end of the read list.
> 
> (OP_ILLEGAL) for the third op, because it finds nonsense when it looks in the
> XDR stream for the arguments of the third op. (I do have a fix for this).
> 
> I can make the converse point: sending an RPC request where only a middle
> argument is placed in a chunk list is clearly allowed by RFC 5666, and is not
> forbidden by RFC 5667. Having "position" fields for each chunk in the read list
> means a server MUST be prepared to accept inline content following a read
> list

I'm not sure I agree with "MUST", but I do agree that there is sufficient
(redundant) information in the encoding of the request for the server to
Perform such an action.

> agree that sending the final GETATTR in the read list is allowed, then servers
> should accept that in addition to NFSv4 compounds where the final GETATTR
> is sent as additional inline content.
> 
> > It clearly complies with RFC5666 which grants the sender a lot of
> > freedom for the sender as to how it chooses to sends individual data
> elements.

While freedom is the intention of RFC5666, it is (was) the intention of RFC5667
to narrowly define those freedoms for the NFS protocol family. There is a
requirement in RFC5666 that each upper layer provide such a binding, with
the appropriate rules to ensure interoperability.

> > I think the basic idea is that RDMA is for large data elements, such
> > as data to be written, but I don't think it comes out and forbids you
> > from sending other things.
> 
> This text from section 4 seems to disallow the use of a read list for anything
> but opaque file data or symlink pathnames:
> 
> "Similarly, a single RDMA Read list entry MAY be posted by the client  to
> supply the opaque file data for a WRITE request or the pathname  for a
> SYMLINK request.  The server MUST ignore any Read list for  other NFS
> procedures, as well as additional Read list entries beyond  the first in the list."

Note however that RFC5667 section 4 is explicitly about NFS versions 2 and 3,
and the "MUST ignore" statement is purely about simplifying the nfsdirect
layering to align with the (simple) requirements of NFS v2 and v3, transfer-wise.

Section 5 in turn is silent on the issue of SYMLINK, which of course is implemented
In NFSv4.x as the CREATE procedure. That's clearly an omission.

I agree with many of the points you make below, but in the absence of clear
normative statements in RFC5667, I think they simply reinforce the need
for updating the nfsdirect specification.

Tom.

> 
> "The server MUST ignore" is a clear statement that placing other arguments
> in a read list will prevent interoperation, at least for
> NFSv2 and NFSv3 Direct.
> 
> The "MAY" refers to an alternate mechanism of sending large RPC
> > The only basis on which you might base a case that this violates
> > RFC5667 wiule be based on the following text from section2:
> >
> > Large chunks of data,
> > such as the file data of an NFS WRITE request, MAY be referenced by an
> > RDMA Read list and be moved efficiently and directly placed by an RDMA
> > Read operation initiated by the server.
> >
> > sorts of request data using RDMA read chunks.  I don't read it that way.
> 
> > > Only the file data
> > > payload belongs in the read list, if NFSv4 Direct is to be
> > > consistent with NFSv2 and NFSv3.
> >
> > I don't think NFSv4 can be consistent with previous NFS protocols
> > because it is different in having COMPOUND.
> 
> I don't agree that NFSv4 cannot be consistent with legacy NFS. Perhaps it is
> not stated clearly in RFC 5667 section 5 how to make NFSv4 equivalent to
> NFSv2/3 Direct, but observe that:
> 
> to put only the opaque file data in a read list, no matter what else appears in
> the NFSv4 COMPOUND request with the WRITE operation. (Small WRITEs of
> course are always sent inline).
> 
> > Your choice is either to be consistent with NFSv3:
> > 	* In having chunks only at the end of the request.
> > 	* In having chunks only for WRITE data.
> 
> RFC 5667 section 5 has two sentences discussing the use of read lists with
> NFSv4 COMPOUND requests:
> 
> "The situation is similar for RDMA Read lists sent by the client and  applies to
> the NFSv4.0 WRITE and SYMLINK procedures as for v3.
>  Additionally, inline segments too large to fit in posted buffers MAY  be
> transferred in special "RDMA_NOMSG" messages."
> 
> with NFSv3 WRITE, at first glance it doesn't help us decide which of your two
> bullets is the correct interpretation.
> 
> "The situation is similar" I believe references earlier text in section 5, not to
> the text in section 4 that discusses WRITE/SYMLINK. Earlier, section 5 goes as
> far as to say:
> 
>  This procedure returns results from a sequence of operations.  Only  the
> opaque file data from an NFS READ operation and the pathname from  a
> READLINK operation MUST utilize entries from the Write list."
> 
> The "Only . . . MUST" construction in the third sentence is slippery.
> 
> It can be read as allowing other operations and arguments to use a write list,
> and as requiring READ and READLINK to use a write list in all cases, including
> short payloads. That reading, while literal, seems contradictory with the rest
> of the document.
> 
> operations in NFSv4 compounds. That reading I believe is consistent with the
> rest of the document and earlier versions of the NFS Direct protocol, and
> would also apply to WRITE in the read list case.
> 
> section 5 is incorrect. It could be replaced with CREATE(NF4LNK). Or using a
> read list during symlink creation could simply be abandoned for NFSv4, in
> favor of sending large CREATE requests via RDMA_NOMSG.
> 
> payloads in NFSv4 COMPOUND requests can be sent in a read list, but there
> seems to be some wiggle room. It would help me, as an implementer, if RFC
> 5667 section 5 could be clarified.
> 
> > > Now suppose the Linux client is changed to place the GETATTR
> > > operation inline, following the front of the compound.
> >
> > > When the opaque file data requires round-up because it's length is
> > > not divisible by four, should there be a pad?
> >
> > No.
> >
> > > To put it another way: Since inline data does not have a "position"
> > > the receiver assume the inline content begins at the next XDR
> > > position after the end of the chunk?
> >
> > Yes.
> >
> > > inline content. But that pad would cause the following inline
> > > content to be sent unaligned, since a zero pad is always shorter
> > > than 4 bytes long.
> >
> > > following inline content always begins at the next XDR position).
> > > The sender MUST NOT send a pad inline in this case. Is this correct?
> >
> > Yes.
> 
> Thanks for confirming my interpretation, and for helping me to sharpen my
> understanding of these RFCs.
> 
> > Because this position will not match (else roundup would not have
> > occurred), the receiver decoding will fall back to inspecting the
> > remaining inline portion.
> >
> > This may not be clear enough.  Maybe a clarifying editorial errata
> > would be justified.
> 
> Let me know if you'd like me to file one.
> 
> > Linux sends the part of the compound before the opaque file data via
> > RDMA SEND. The opaque file data is put into the read list. Then the
> > GETATTR operation is put in the read list after the file data.
> >
> > Existing NFS/RDMA servers are able to receive and process this request
> > correctly.
> >
> > When the opaque file data requires round-up because it's length is not
> > To put it another way: Since inline data does not have a "position"

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 5 Jan 2015 11:41:47 -0800

> On Jan 5, 2015, at 10:50, Tom Talpey <[email-masked]> wrote:
> 

So are you saying that RDMA is unusable for NFSv4? That would have to be the conclusion if it doesn’t allow clients to construct arbitrary COMPOUNDs. It would be completely unacceptable layering violation for the NFS client to have to worry about what kind of transport the RPC layer is using.

Trond

---

From: Tom Talpey <[email-masked]>
Date: Mon, 5 Jan 2015 20:50:29 +0000

> From: Trond Myklebust [mailto:[email-masked]]
> Sent: Monday, January 5, 2015 2:42 PM
> To: Tom Talpey
> Cc: Lever Charles Edward; Noveck Dave; Dai Ngo; Chunli Zhang;
> [email-masked]; Karen
> >> piece of the compound RPC and send it in a separate RDMA READ. That's
> >> extra transport bytes on the wire, and an extra round trip (albeit a very
> fast one).
> >
> > The intention of an RDMA Chunk is that it encodes all or part of an
> > RPC argument, RPC result, or an entire RPC message. The fact that the
> > client added a compounded GETATTR to the write data's Read Chunk is
> wrong.
> > The fact that the server parsed the additional data after the write
> > payload as an RPC request is also wrong. Keeping these segments
> > separate is critical to the RFC5667 specification of the NFS binding to
> RPC/RDMA.
> 
> 
> So are you saying that RDMA is unusable for NFSv4?

Certainly not, and there is no layering violation. The problem is that RFC5666
defines several transfer models, and because each has different semantics and
efficiency; it leaves the choice of which to use to a "binding" statement (e.g.
RFC5667). This is very similar to the XDR dot-x definitions of NFS over RPC, it
then goes one layer deeper, to map XDR to RDMA, if you will.

> That would have to be
> the conclusion if it doesn’t allow clients to construct arbitrary COMPOUNDs.
> It would be completely unacceptable layering violation for the NFS client to
> have to worry about what kind of transport the RPC layer is using.

The RFC5666 protocol allows arbitrary COMPOUNDs. In this case, the problem
comes from the client and server conflating the RPC arguments and results with
COMPOUND, since the client is tacking the GETATTR (a COMPOUND) onto the
write data (another COMPOUND argument's Read Chunk). So at a minimum that
ambiguity needs to be more carefully defined in RFC5667.

Some of these issues could be addressed with Errata. But, I think the v4.1 and
especially v4.2 text should be in the form of a new document.

Tom.

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 5 Jan 2015 21:17:47 +0000

You comment about new text was what I was looking for, Tom.

Chuck, is this something that you could pick up and work on for the WG?

Spencer

> From: nfsv4 [mailto:[email-masked]] On Behalf Of Tom Talpey
> Sent: Monday, January 5, 2015 12:50 PM
> To: Trond Myklebust; [email-masked]
> Cc: Chunli Zhang; Karen; Dai Ngo
> > >> Subject: Re: [nfsv4] RPC/RDMA read chunk round-up when inline
> > >> content follows
> > >>
> > >> Hi Dave-
> > >>
> > >>> Presumably it sends the GETATTR as a separate chunk.  I suppose
> > >>> that could be a performance issue.
> > >>
> > >> piece of the compound RPC and send it in a separate RDMA READ.
> > >> That's extra transport bytes on the wire, and an extra round trip
> > >> (albeit a very
> > fast one).
> > >
> > > RPC argument, RPC result, or an entire RPC message. The fact that
> > > the client added a compounded GETATTR to the write data's Read Chunk
> > > is
> > wrong.
> defines several transfer models, and because each has different semantics
> and efficiency; it leaves the choice of which to use to a "binding" statement
> (e.g.
> > It would be completely unacceptable layering violation for the NFS
> > client to have to worry about what kind of transport the RPC layer is using.
> 
> The RFC5666 protocol allows arbitrary COMPOUNDs. In this case, the
> problem comes from the client and server conflating the RPC arguments and
> results with COMPOUND, since the client is tacking the GETATTR (a
> COMPOUND) onto the write data (another COMPOUND argument's Read
> Chunk). So at a minimum that ambiguity needs to be more carefully defined
> in RFC5667.
> 
> Some of these issues could be addressed with Errata. But, I think the v4.1
> and especially v4.2 text should be in the form of a new document.
> 
> 
> Tom.
> 
> 
> 
> 
> >
> > Trond
>

---

From: Chuck Lever <[email-masked]>
Date: Mon, 5 Jan 2015 19:24:27 -0500

> Chuck, is this something that you could pick up and work on for the WG?

Hi Spencer, yes, I’d like to help revise the specification. I
have some cycles to prototype, write, or edit, as needed.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 6 Jan 2015 13:58:26 -0500

> to the RFC5667 specification of the NFS binding to RPC/RDMA.

So if I understand correctly: fixing the servers will make them fail all
writes from current clients, and fixing the clients will make their
writes fail against all current servers.

Isn't it a little late for that kind of change?  Or are there really so
few NFS/RDMA users that we can afford to completely break backwards
compatibility?

--b.

---

From: Tom Talpey <[email-masked]>
Date: Tue, 6 Jan 2015 21:00:48 +0000

> From: J. Bruce Fields [mailto:[email-masked]]
> Sent: Tuesday, January 6, 2015 1:58 PM
> To: Tom Talpey
> Cc: Chuck Lever; David Noveck; Dai Ngo; Chunli Zhang; nfsv4 list
> ([email-masked]); Karen
> > > > > It turns out that Linux is the only NFS client that supports
> > > > > RDMA and adds an operation (GETATTR) after WRITE in an NFSv4
> compound.
> > > >
> > > That's extra transport bytes on the wire, and an extra round trip (albeit a
> very fast one).
> >
> So if I understand correctly: fixing the servers will make them fail all writes
> from current clients, and fixing the clients will make their writes fail against all
> current servers.
> 
> Isn't it a little late for that kind of change?

You're asking a product question, and I don't think it's appropriate for me to
answer that. As a protocol matter, yes, I think there is a bug in both the client
and server, but given the possible ambiguity I guess one could argue the
behavior is compliant. I don't, however, think it is interoperable, and it will
certainly make it very difficult to properly implement COMPOUND when
the RPC arguments of each compounded operation are RDMA-eligible. So
it will constrain the implementation in future.

>  Or are there really so few
> NFS/RDMA users that we can afford to completely break backwards
> compatibility?

This is the IETF list, so I might suggest polling the Linux and Solaris
communities on this one. I would expect that any NFS/RDMA user
will want the implementation to work well for NFSv4.x, which might
factor into the decision.

Tom.

> 
> --b.
> 
> >
> > That said, the text in RFC5667 may be unclear. I'd suggest that it
> > needs to be revisited anyway, to reflect implementation experience as
> > well as the new behaviors of NFSv4.1, v4.2 etc. The document is five
> > years old, and it reflects work done prior to that. In particular,
> > while it was published at the same time as RFC5661 (NFSv4.1), it
> > actually doesn't make any specific requirements for that minor version
> > beyond mentioning the callback channel. That, plus pNFS, layouts, CREATE,
> OPENATTR, ACLs, etc.
> >
> > >
> > > > > via RDMA SEND. The opaque file data is put into the read list.
> > > > > Then the GETATTR operation is put in the read list after the file data.
> > > >
> > > > The interesting question is whether they would work if one
> > > > switched from the chunks-only-at-the-end approach to the
> > > > alternative chunks-only-for-WRITE approach.
> > >
> > > The current Linux NFS server and the Solaris 11 update 2 NFS server
> > > do not handle a request with additional inline content. There is a
> > > minor exception, but let's put that aside for the moment.
> > >
> > > (OP_ILLEGAL) for the third op, because it finds nonsense when it
> > > looks in the XDR stream for the arguments of the third op. (I do have a fix
> for this).
> > >
> > > 5666, and is not forbidden by RFC 5667. Having "position" fields for
> > > each chunk in the read list means a server MUST be prepared to
> > > accept inline content following a read list
> >
> > (redundant) information in the encoding of the request for the server
> > to Perform such an action.
> >
> > >
> > > > > First question: does the Linux client comply with RFCs 5666 and
> > > > > 5667 when it sends an NFSv4 WRITE compound in this way, or
> > > > > should it be changed?
> > > >
> > While freedom is the intention of RFC5666, it is (was) the intention
> > of RFC5667 to narrowly define those freedoms for the NFS protocol
> > family. There is a requirement in RFC5666 that each upper layer
> > provide such a binding, with the appropriate rules to ensure
> interoperability.
> >
> > > >
> > > > I think the basic idea is that RDMA is for large data elements,
> > > > such as data to be written, but I don't think it comes out and
> > > > forbids you from sending other things.
> > >
> > > This text from section 4 seems to disallow the use of a read list
> > > for anything but opaque file data or symlink pathnames:
> > >
> > > "Similarly, a single RDMA Read list entry MAY be posted by the
> > > client  to supply the opaque file data for a WRITE request or the
> > > pathname  for a SYMLINK request.  The server MUST ignore any Read
> > > list for  other NFS procedures, as well as additional Read list entries
> beyond  the first in the list."
> >
> > Note however that RFC5667 section 4 is explicitly about NFS versions 2
> > and 3, and the "MUST ignore" statement is purely about simplifying the
> > nfsdirect layering to align with the (simple) requirements of NFS v2 and v3,
> transfer-wise.
> >
> > Section 5 in turn is silent on the issue of SYMLINK, which of course
> > is implemented In NFSv4.x as the CREATE procedure. That's clearly an
> omission.
> >
> > I agree with many of the points you make below, but in the absence of
> > clear normative statements in RFC5667, I think they simply reinforce
> > the need for updating the nfsdirect specification.
> >
> > Tom.
> >
> >
> >
> > >
> > > "The server MUST ignore" is a clear statement that placing other
> > > requests: RDMA_NOMSG, where the entire RPC message is
> encapsulated
> > > in a read list at position 0.
> > >
> > > > such as the file data of an NFS WRITE request, MAY be referenced
> > > > by an RDMA Read list and be moved efficiently and directly placed
> > > > by an RDMA Read operation initiated by the server.
> > > >
> > > > One could argue that this somehow implies you "MAY NOT" transfer
> > > > other sorts of request data using RDMA read chunks.  I don't read it that
> way.
> > >
> > > > > Only the file data
> > > I don't agree that NFSv4 cannot be consistent with legacy NFS.
> > > Perhaps it is not stated clearly in RFC 5667 section 5 how to make
> > > NFSv4 equivalent to
> > > The legacy-consistent way of sending an NFSv4 WRITE compound would
> > > be to put only the opaque file data in a read list, no matter what
> > > else appears in the NFSv4 COMPOUND request with the WRITE
> operation.
> > > (Small WRITEs of course are always sent inline).
> > >
> > >  Additionally, inline segments too large to fit in posted buffers
> > > MAY  be transferred in special "RDMA_NOMSG" messages."
> > >
> > > Although it does suggest strongly that NFSv4 WRITE should be
> > > consistent with NFSv3 WRITE, at first glance it doesn't help us
> > > decide which of your two bullets is the correct interpretation.
> > >
> > > "The situation is similar" I believe references earlier text in
> > > section 5, not to the text in section 4 that discusses
> > > WRITE/SYMLINK. Earlier, section 5 goes as far as to say:
> > >
> > > the opaque file data from an NFS READ operation and the pathname
> > > from  a READLINK operation MUST utilize entries from the Write list."
> > >
> > > write list, and as requiring READ and READLINK to use a write list
> > > in all cases, including short payloads. That reading, while literal,
> > > seems contradictory with the rest of the document.
> > >
> > > Or it can be read as allowing a write list only for READ and
> > > READLINK operations in NFSv4 compounds. That reading I believe is
> > > consistent with the rest of the document and earlier versions of the
> > > NFS Direct protocol, and would also apply to WRITE in the read list case.
> > >
> > > operation. Thus the explicit mention of an NFSv4 SYMLINK operation
> > > in section 5 is incorrect. It could be replaced with CREATE(NF4LNK).
> > > Or using a read list during symlink creation could simply be
> > > abandoned for NFSv4, in favor of sending large CREATE requests via
> RDMA_NOMSG.
> > >
> > > In sum, a very particular reading of section 5 suggests that only
> > > WRITE payloads in NFSv4 COMPOUND requests can be sent in a read
> > > list, but there seems to be some wiggle room. It would help me, as
> > > an implementer, if RFC
> > > > > When the opaque file data requires round-up because it's length
> > > > > is not divisible by four, should there be a pad?
> > > >
> > > > No.
> > > >
> > > > > field, should the receiver assume that inline content following
> > > > > the read list begins right at the end of the rounded-up chunk?
> > > >
> > > > No.
> > > >
> > > > > or should
> > > > > If the former, that would require inserting a zero pad before
> > > > > the inline content. But that pad would cause the following
> > > > > inline content to be sent unaligned, since a zero pad is always
> > > > > shorter than 4 bytes long.
> > > >
> > > > > By implication, then, the receiver MUST conclude that round-up
> > > > > is present in the case when inline data remains to be decoded
> > > > > (ie, the following inline content always begins at the next XDR
> position).
> > > Thanks for confirming my interpretation, and for helping me to
> > > sharpen my understanding of these RFCs.
> > >
> > > > > I've read RFC 5666 section 3.7. Paragraph 4 seems most relevant,
> > > > > but it doesn't discuss the case where there is additional inline
> > > > > content following a chunk list.
> > > >
> > > > via RDMA SEND. The opaque file data is put into the read list.
> > > > Then the GETATTR operation is put in the read list after the file data.
> > > >
> > > > First question: does the Linux client comply with RFCs 5666 and
> > > > 5667 when it sends an NFSv4 WRITE compound in this way, or should
> > > > it be changed?
> > > >
> > > > consistent with NFSv2 and NFSv3
> > > >
> > > > not divisible by four, should there be a pad? if so, where does it
> belong?
> > > >
> > > > field, should the receiver assume that inline content following
> > > > the read list begins right at the end of the rounded-up chunk? or
> > > > should the receiver assume the inline content begins at the next
> > > > XDR position after the end of the chunk?
> > > >
> > > > present in the case when inline data remains to be decoded (ie,
> > > > the following inline content always begins at the next XDR
> > > > position). The sender MUST NOT send a pad inline in this case. Is this
> correct?
> > > >
> > > > I've read RFC 5666 section 3.7. Paragraph 4 seems most relevant,
> > > > but it doesn't discuss the case where there is additional inline
> > > > content following a chunk list.
> > > >
> > > > Thanks for reading!
> > > >
> > > > --
> > > > Chuck Lever
> > > >

---

From: David Noveck <[email-masked]>
Date: Wed, 7 Jan 2015 04:58:24 -0500

---------- Forwarded message ----------
From: David Noveck <[email-masked]>
Date: Tue, Jan 6, 2015 at 2:56 PM
Subject: Re: [nfsv4] RPC/RDMA read chunk round-up when inline content
follows
To: "J. Bruce Fields" <[email-masked]>

> writes from current clients,

If it hurts when you do that, don't do that.  it seems you can make servers
accept old and new client behavior, even if the old behavior is a BAD IDEA.

one way to map this into RFC2119 language is to say the servers SHOULD
reject
older-form requests and MUST accept newer-form requests.

> andf fixing the clients will make the writes fail against all current
servers.

not sure how to state this in rfc2119 language but I'd argue that one
should not fix the
clients until severs are fixed to accept newer-form request.

> Isn't it a little late for that kind of change?

I think it is.

> Or are there really so

i guess it depends on whether you are one of those users.

The approach described in draft-ietf-nfsv4-versioning is to make server
support for old-form
and new-form requests into separate features that the client can test for,
allowing clients
to be fixed to adapt to server behavior in parallel with making servers
which support both
behaviors.

Not only is that quite heavyweight for this problem, but this draft is a
ways away from
IESG approval and publishing.

On Tue, Jan 6, 2015 at 1:58 PM, J. Bruce Fields <[email-masked]>
wrote:

> piece of the
> payload
> via
> not
> in the
> middle
> 5667
> anything
> to
> and 3,
> clear
> arguments
> an
> other
> appears in
> applies to
> be
> section 5, not to
> the
> write list,
> with the
> using a
> there
> but
> request
> operation
> not

---

From: Chuck Lever <[email-masked]>
Date: Wed, 7 Jan 2015 10:57:38 -0500

Hi Dave-

On Jan 7, 2015, at 4:58 AM, David Noveck <[email-masked]> wrote:

> Subject: Re: [nfsv4] RPC/RDMA read chunk round-up when inline content follows

Whether or not existing implementations support the old way, new
servers should never have to. Arguably the spec already does not
allow it, thus this point is moot: existing implementations are
broken, according to RFC 5667, and always have been. The question
for the WG is whether the spec should be updated to explicitly
grandfather this behavior.

Implementers are free to do what is needed to march over the
incompatibility, including choosing temporarily to be non-
compliant with the specification (eg. to allow both ways during
a period of transition). No spec change is needed to do that.

A practical question is whether allowing servers to continue
accepting the old way is at all risky (that is, would affect an
implementation’s ability to interoperate in the future). I think
that would be valuable information for implementers who are faced
with the decision, and would also inform the WG about whether a
spec change is advisable.

Both the Solaris and Linux servers would need to be fixed to
accept the correct way. Then at a later time, they would need
to be updated to reject the old way. That seems feasible.

The only client implementation at issue is the Linux client. The
Solaris client sends only { PUTFH, WRITE }. Another way to put it
is a spec change is being requested to accommodate one broken
client implementation. That seems awkward.

> one way to map this into RFC2119 language is to say the servers SHOULD reject
> > andf fixing the clients will make the writes fail against all current servers.
> 
> not sure how to state this in rfc2119 language but I'd argue that one should not fix the

IMO it is not too late because practically speaking, this is not
the only interop issue that is facing these incomplete
implementations, whether or not they are deployed by customers.

Linux, for instance, does not support RDMA_NOMSG type RPC calls on
either the client or server. RDMA_NOMSG type calls are needed for
SYMLINK and large NFSv4 COMPOUNDs. Would the WG consider removing
RDMA_NOMSG from the spec, or do we rightly consider this a necessary
but missing feature in Linux?

At this stage I don’t think anyone has tried using a Solaris
client with a Linux server. I’m guessing there will be interop
problems there too.

> i guess it depends on whether you are one of those users.

This affects only NFSv4 on RDMA, and there is a work-around:
use NFS on IPoIB until the client and server interoperate again.

> The approach described in draft-ietf-nfsv4-versioning is to make server support for old-form
> and new-form requests into separate features that the client can test for, allowing clients
> to be fixed to adapt to server behavior in parallel with making servers which support both 
> behaviors.
> 
> Not only is that quite heavyweight for this problem, but this draft is a ways away from 
> On Tue, Jan 6, 2015 at 1:58 PM, J. Bruce Fields <[email-masked]> wrote:

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 7 Jan 2015 17:42:58 +0000

Thanks, Chuck.  Might I suggest a brief "design meeting" conference call where you and Tom (and other interested parties) can discuss the structure of Tom's suggestions for document updates. 

Workable?

> From: Chuck Lever [mailto:[email-masked]]
> Sent: Monday, January 5, 2015 4:24 PM
> To: Spencer Shepler
> Cc: Tom Talpey; Trond Myklebust; [email-masked]; Chunli Zhang; Dai Ngo;
> Karen
> Hi Spencer, yes, I'd like to help revise the specification. I have some cycles to
> prototype, write, or edit, as needed.
> 
> 
> 
> > Spencer
> >
> >>>> the client added a compounded GETATTR to the write data's Read
> >>>> Chunk is
> >>> wrong.
> >> Certainly not, and there is no layering violation. The problem is
> >> that RFC5666 defines several transfer models, and because each has
> >> different semantics and efficiency; it leaves the choice of which to
> >> use to a "binding" statement (e.g.
> >> RFC5667). This is very similar to the XDR dot-x definitions of NFS
> >> over RPC, it then goes one layer deeper, to map XDR to RDMA, if you will.
> >>
> >>> That would have to be
> >>> the conclusion if it doesn't allow clients to construct arbitrary
> COMPOUNDs.
> >> problem comes from the client and server conflating the RPC arguments
> >> and results with COMPOUND, since the client is tacking the GETATTR (a
> >> Chunk). So at a minimum that ambiguity needs to be more carefully
> >> defined in RFC5667.
> >>
> >> Some of these issues could be addressed with Errata. But, I think the
> >> v4.1 and especially v4.2 text should be in the form of a new document.
> >>
> >>
> >> Tom.
> >>
> >>
> >>
> >>
> >>>
> >>> Trond
> >>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 7 Jan 2015 13:02:37 -0500

> Thanks, Chuck.  Might I suggest a brief "design meeting" conference call where you and Tom (and other interested parties) can discuss the structure of Tom's suggestions for document updates. 

I’ll give Tom some time to collect his thoughts, then we can
schedule a call in a week or two. I’ll post call details here
when we are ready.