Subject: Re: [nfsv4] NFSv4.2 vs pNFS

From: Christoph Hellwig <[email-masked]>
Date: Wed, 16 Apr 2014 19:32:15 +0200

After this thread has been quiet for a while:

Does anyone disagree with adding some form of language like the one
I proposed below?  Otherwise I'll start to prepare a patch for the spec
taking the various comments into account.

On Tue, Nov 19, 2013 at 06:17:34PM +0100, Christoph Hellwig wrote:
> While reading through the current NFSv4.2 drafts I have few questions
> vs the interaction with pnfs.
> 
> For one rfc3530.txt generally seems to define the file layout as a
> "NFSv4.1" file layout, allowing access using a subset of the 4.1 protocol.
> 
> There is no language in the 4.2 defintining any sort of NFS v4.2 file
> layout, just a few implication that READ_PLUS should be fine on 4.2,
> without explicitly allowing it nor definining a protocol for DS access.
> 
> There also is no language saying if SEEK or WRITE_PLUS should be allowed
> against a DS, and how a DS should respond if it can't support the operation.
> E.g. a hole punch often is a metadata centric operation that I would not
> accept to be handled by a DS unless the cluster has a fully symmetric
> architecture.
> 
> I'd also love to know what the expected semantics for block or object
> layout to a file undergoing offloaded operations are.
---end quoted text---

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 16 Apr 2014 10:59:55 -0700

Dave Noveck and I have been chatting about adding text to the effect
that NFSv4.2 supports all the NFSv4.1 layout types. I (at least) had
the assumption that as almost everything in NFSv4.2 was an optional
add-on to NFSv4.1, this was understood. In rereading 5661, it does not
make this carry-on behavior the default.

My further assumption here is that in order to do pNFS over NFSv4.2,
the client would be communicating to both the MDS and the DSes via
NFSv4.2 and not NFSv4.1.

Let me clarify that, in order to utilize the new features, the client would
have to speak NFSv4.2 to the DSes. It could do a server-side copy
that chose to do the copy via NFSv4.1.

As for READ_PLUS, SEEK, WRITE_HOLE, and WRITE_SAME, they
can each return NFS4ERR_NOTSUPP from the DS to indicate that
they are not supported. (Note, I just fixed up READ_PLUS and SEEK
to allow them to communicate that fact.)

some more inline

>> without explicitly allowing it nor definining a protocol for DS access.

14.9.4.1.  Additional pNFS Implementation Information

   With pNFS, the semantics of using READ_PLUS remains the same.  Any
   data server MAY return a hole or ADH result for a READ_PLUS request
   that it receives.  When a data server chooses to return such a
   result, it has the option of returning information for the data
   stored on that data server (as defined by the data layout), but it
   MUST NOT return results for a byte range that includes data managed
   by another data server.

   A data server should do its best to return as much information about
   a ADH as is feasible without having to contact the metadata server.
   If communication with the metadata server is required, then every
   attempt should be taken to minimize the number of requests.

   If mandatory locking is enforced, then the data server must also
   ensure that to return only information that is within the owner's
   locked byte range.

>> layout to a file undergoing offloaded operations are.

What are the semantics of a file being copied?

I.e., the same answer should apply here.

> ---end quoted text---
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 16 Apr 2014 20:24:14 +0200

> make this carry-on behavior the default.

Yeah, that sounds good.

> that chose to do the copy via NFSv4.1.

Right, I came up with this language earlier in the spec to clarify that:

"X.Y.   NFSv4.2 as a Storage Protocol in pNFS: the File Layout Type

When using NFSv4.2 to speak to a pNFS MDS, the DSs are expected to use
NFSv4.2 semantics for the their operation if those different from NFSv4.1"

> to allow them to communicate that fact.)

Right.  Earlier in this thread I also came up with this section to
make sure these operations are treated as data ops.  This will need
a minor update to cover the WRITE_PLUS split at least, though.

"X.Y.1.  Operations Sent to NFSv4.2 Data Servers

In addition to the commands listed in [RFC5661], NFSv4.2 data servers may
accept COMPOUND containing the following additional operations:
READ_PLUS, WRITE_PLUS and SEEK, which will be treated like the subset
specified as "data-server I/O operations" in [RFC5661].

Additional details on the implementation of these operations in a pNFS
context are documented in the operation specific sections below."

Which I think we need in the spec in some form.

>    With pNFS, the semantics of using READ_PLUS remains the same.  Any

With the pNFS file layout?  None of this seems to apply to other layout types.