Subject: [nfsv4] [PATCH] Clarify use of pNFS and stateids

From: Thomas Haynes <[email-masked]>
Date: Thu, 24 Apr 2014 17:45:50 -0700

From: loghyr <[email-masked]>

This is item #3 in the issues file. I scoured the mailing list
to pick up the thread in November.

An issue has crept in with Rule 16 in the minor version rules,
which we'll need to fix. I'll await comments first!

loghyr (1):
  Clarify use of pNFS and stateids

 Makefile                 |  1 +
 issues.txt               |  2 ++
 nfsv42_middle_minorv.xml | 12 ++++++++++++
 nfsv42_middle_pnfs.xml   | 33 +++++++++++++++++++++++++++++++++
 4 files changed, 48 insertions(+)
 create mode 100644 nfsv42_middle_pnfs.xml

-- 
1.9.0

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 24 Apr 2014 17:45:51 -0700

Signed-off-by: loghyr <[email-masked]>
---
 Makefile                 |  1 +
 issues.txt               |  2 ++

diff --git a/Makefile b/Makefile
index 9ba2101..1139b93 100755
--- a/Makefile
+++ b/Makefile
@@ -428,6 +428,7 @@ IDXMLSRC_BASE = \
 	${DOC_PREFIX}_middle_start.xml \
 	${DOC_PREFIX}_middle_introduction.xml \
 	${DOC_PREFIX}_middle_minorv.xml \
+	${DOC_PREFIX}_middle_pnfs.xml \
 	${DOC_PREFIX}_middle_copy.xml \
 	${DOC_PREFIX}_middle_advise.xml \
 	${DOC_PREFIX}_middle_sparse.xml \
diff --git a/issues.txt b/issues.txt
index 715e9e4..d2e7b43 100644
--- a/issues.txt
+++ b/issues.txt
@@ -21,6 +21,8 @@
 
    See the recent WG ML chatter over all of this.
 
+   Done - 4/24 by [email-masked]
+
 4) Labeled NFS - FATTR4_CHANGE_SEC_LABEL - how big does it need to
    be?  Is seLinux going to support it?
 
diff --git a/nfsv42_middle_minorv.xml b/nfsv42_middle_minorv.xml
index 064ec47..2c114b7 100644
--- a/nfsv42_middle_minorv.xml
+++ b/nfsv42_middle_minorv.xml
@@ -216,4 +216,16 @@
       </t>
     </list>
   </t>
+
+  <section anchor="ss:minorv:objects" title="Server Object Sharing from NFSv4.1 to NFSv4.2">
+    <t>
+      While Rule 16 above is clearly states that stateids (which includes
+      layout stateids), filehandles, or other server objects MUST NOT
+      be used across minor versions, there are no changes occuring in
+      these server objects between NFSv4.1 and NFSv4.2. As such,
+      it is convientent to share them across releases!
+      <cref anchor='AI1 ' source='TH'> Purposedly worded this way
+      because Rule 16 and this can not co-exist! </cref>
+    </t>
+  </section>
 </section>
diff --git a/nfsv42_middle_pnfs.xml b/nfsv42_middle_pnfs.xml
new file mode 100644
index 0000000..916d3c4
--- /dev/null
+++ b/nfsv42_middle_pnfs.xml
@@ -0,0 +1,33 @@
+<!-- Copyright (C) The IETF Trust (2014) -->
+<!-- Copyright (C) The Internet Society (2014) -->
+
+<section anchor="ss:pnfs" title="NFSv4.2 as a Storage Protocol in pNFS: the File Layout Type">
+  <t>
+    A file layout provided by a NFSv4.2 server may refer either to a DS
+    that only implements NFSv4.1 as specified in <xref target='RFC5661' />,
+    or to a DS that implements additions from NFSv4.2, in which case the rules in
+    <xref target='ss:pnfs:ops' /> apply. As the File Layout Type does not
+    provide a means for informing the client as to which minor version a
+    particular DS is providing, it will have op negotiate this via the normal
+    RPC semantics of major and minor version discovery.
+  </t>
+
+  <section anchor="ss:pnfs:ops" title="Operations Sent to NFSv4.2 Data Servers">
+    <t>
+      In addition to the commands listed in <xref target='RFC5661' />,
+      NFSv4.2 data servers MAY accept a COMPOUND containing the following
+      additional operations: ALLOCATE (see <xref target='op:allocate' />),
+      DEALLOCATE (see <xref target='op:deallocate' />), READ_PLUS
+      (see <xref target='op:read_plus' />), WRITE_SAME (see
+      <xref target='op:write_same' />), and SEEK (see
+      <xref target='op:seek' />), which will be treated like the
+      subset specified as "Operations Sent to NFSv4.1 Data Servers"
+      in Section 13.6 of <xref target='RFC5661' />.
+    </t>
+
+    <t>
+      Additional details on the implementation of these operations in a pNFS
+      context are documented in the operation specific sections.
+    </t>
+  </section>
+</section>
-- 
1.9.0

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 24 Apr 2014 22:15:21 -0700

On Apr 24, 2014, at 9:54 PM, Christoph Hellwig <[email-masked]> wrote:

> On Thu, Apr 24, 2014 at 05:45:51PM -0700, Thomas Haynes wrote:
> In my mental model of the MDS/DS split ALLOCATE and DEALLOCATE would be
> something that only the MDS handles, similar to a SETATTR to a larger or
> smaller size.  Do you have an implementation in mind where a DS could
> handle ALLOCATE/DEALLOCATE properly?
> 

The space reservation I can see as being metadata.

But hole punching is not metadata, it is intimately tied to the DSes.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 25 Apr 2014 07:25:10 +0200

> But hole punching is not metadata, it is intimately tied to the DSes.

Hole punching is a generalization of truncate, aka SETATTR to a smaller
size:

 +-----------------------------------------------+
 |        file data                              |
 +-----------------------------------------------+

truncate:

 +----------------------------+------------------+
 |        file data           | truncated spapce |
 +----------------------------+------------------+

hole punch:

 +---------------------+-----------+------------+
 |        file data    |    hole   |  file data |
 +---------------------+-----------+------------+

Yes, it causes data to be deallocate, just like the SETATTR, but
more importanly if will also cause lots of metadata updates,
and most importantly recalls of layouts.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 25 Apr 2014 10:03:35 -0400

> Hole punching is a generalization of truncate, aka SETATTR to a smaller

In terms of pNFS, operations that are directed to DSes are considered
pNFS data operations. Operations directed to MDSes are considered pNFS
metadata operations.

Since HOLE PUNCH would be typically directed to a DS, it is considered
a data operation (even though, as you point out, it is basically a
special case of SETATTR).

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 25 Apr 2014 16:24:25 +0200

> pNFS data operations. Operations directed to MDSes are considered pNFS

I know, we are arguing here if it should be considered a pNFS data operation.

> special case of SETATTR).

Why would it be directed to a DS?  An actual hole punch involves minimal data
I/O (just zeroing out the partial blocks at the end of the hole if it's
unaligned), but involved lots of manipulation of the extent list, timestamps,
and layout recalls for the punched region.  To me that seems like an operation
that the MDS should handle, just like a SETATTR.

---

From: Marc Eshel <[email-masked]>
Date: Fri, 25 Apr 2014 08:00:04 -0700

"nfsv4" <[email-masked]> wrote on 04/25/2014 07:24:25 AM:

> From: Christoph Hellwig <[email-masked]>
> To: Chuck Lever <[email-masked]>, 
> Cc: Christoph Hellwig <[email-masked]>, Christoph Hellwig 
> <[email-masked]>, "nfsv4 list \([email-masked]\)" <[email-masked]>
> Date: 04/25/2014 07:24 AM
> Subject: Re: [nfsv4] [PATCH] Clarify use of pNFS and stateids
> Sent by: "nfsv4" <[email-masked]>
> 
> and layout recalls for the punched region.  To me that seems like 
anoperation
> that the MDS should handle, just like a SETATTR.

I will have to agree with Chuck, if we might do any data I/O, even if it 
is just zeroing, it must go to the DS
Marc.

> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 25 Apr 2014 17:05:33 +0200

> is just zeroing, it must go to the DS

So why doesn't this include SETATTR then?  Also for the blocklayout
and some possible file layouts a LAYOUTGET with unfortunate alignment
will have to zero partial blocks, so having to do some tiny amount of
data can't be a reason that a command needs to go to the DS.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 25 Apr 2014 11:24:15 -0400

On Apr 25, 2014, at 10:24 AM, Christoph Hellwig <[email-masked]> wrote:

> Why would it be directed to a DS?

Like SETATTR, a HOLE PUNCH passes no data to the server.

However, like WRITE, the data contents of a file are modified by the
operation.  A SETATTR that updates file size is a somewhat different
animal than one that simply modifies a file’s time stamp.

SETATTR also carries with it a state ID, just like WRITE, and is
serialized with other state ID-bearing operations so the client
can manage its data cache (unless I’ve misunderstood something).

> An actual hole punch involves minimal data

To me, those seem like server implementation details.  File time stamps
have to be updated on each WRITE anyway.

A layout recall wouldn't be REQUIRED for a HOLE PUNCH.  Certainly, an MDS
could do this as an optimization, for some layout types.  For others, the
MDS would simply be a delay server.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 25 Apr 2014 17:32:44 +0200

On Fri, Apr 25, 2014 at 11:24:15AM -0400, Chuck Lever wrote:
> operation.

a hole punch doesn't modify data in a file, it _discards_ data in a file.
The same as a SETATTR to a lower size.  If you think that's not enough
modification just include another SETATTR to the previous size, e.g.

DEALLOCATE(2g, 1g)

is exactly the same as a

SETATTR({FATTR4_WORD0_SIZE, 2g})
SETATTR({FATTR4_WORD0_SIZE, 3g})

> MDS would simply be a delay server.

At least for block layouts it is absolutely required, and the way I
understand the object layout it is required there as well.  For the
file layout it depends on the server implementation, but for one that
sits on top of a clustered posix-like filesystem it will also be required
for efficient operation.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 25 Apr 2014 11:57:17 -0400

On Fri, Apr 25, 2014 at 11:00 AM, Marc Eshel <[email-masked]> wrote:
> I will have to agree with Chuck, if we might do any data I/O, even if it is
> just zeroing, it must go to the DS
> Marc.

At most this I/O affects 2 blocks: one at the beginning of the hole
punch section and one at the end. Do you have any evidence of
scalability problems to put forward in order to justify "accelerating"
this by doing it on the DS?

  Trond

---

From: Marc Eshel <[email-masked]>
Date: Fri, 25 Apr 2014 09:14:03 -0700

Trond Myklebust <[email-masked]> wrote on 04/25/2014 08:57:17 AM:

> From: Trond Myklebust <[email-masked]>
> To: Marc Eshel/Almaden/IBM@IBMUS, 
> Cc: Christoph Hellwig <[email-masked]>, Christoph Hellwig 
> <[email-masked]>, "nfsv4 list ([email-masked])" <[email-masked]>,
> nfsv4 <[email-masked]>
> Date: 04/25/2014 08:57 AM
> >> Why would it be directed to a DS?  An actual hole punch involves 
minimal
> >> data

When a server gives a layout it distributes ranges of the file to 
different DSs and each DS should be the only one modifying its range, if 
you want the MDS to do it you will have to recall the layout. I don't 
think it is a big deal either way but I think we should have a clear model 
where data is modified only by the DSs unless there is an error and you 
have to fall back to the MDS.
Marc.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 25 Apr 2014 12:40:01 -0400

> sits on top of a clustered posix-like filesystem it will also be required

To me, this last argument makes a reasonable and specific case that HOLE
PUNCH should be a metadata operation. I don’t find the direct comparison
with SETATTR(truncate) as convincing.

---

From: Tom Haynes <[email-masked]>
Date: Fri, 25 Apr 2014 10:00:13 -0700

> sits on top of a clustered posix-like filesystem it will also be required

So it would not be required for Data ONTAP cDot, and that is a clustered
posix-like filesystem.

I’m willing to conceed that both ALLOCATE and DEALLOCATE are metadata
rich in a non-pNFS server.

I’m not willing to do so for all conceivable pNFS implementations. Consider
one in which the MDS and DSes are different machines. The metadata
on the MDS may have no relationship to the metadata for the object
on DSi.

The space available on DSi may not match the space available on DSj.

So during an ALLOCATE, the MDS will need to communicate with each
of the DSes to make sure that future WRITEs will not fail in the byte
ranges indicated.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 25 Apr 2014 13:28:33 -0400

> of the DSes to make sure that future WRITEs will not fail in the byte

Ultimately, it boils down to a question of what atomicity guarantees
we require for hole punching. If the operation needs to be atomic,
and/or it needs to offer guarantees that data is preserved on failure,
then it needs to be done by the MDS.

Trond

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 25 Apr 2014 20:53:04 +0200

> then it needs to be done by the MDS.

To use it to implement the hole punch case of fallocate for Linux
it defintively needs to be failure atomic, that is either the whole
hole is punched or nothing at all.

Also to make clear we have everyone on the same page here: this is for
the real, space deallocating hole punch as done by the DEALLOCATE operation
I submitted yesterday.  The possible asynchronous zeroing using WRITE_SAME
might be able to relax this a bit, but we'll defintively need to write
down the possible failure cases for that as well.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 25 Apr 2014 14:55:50 -0400

On Fri, 2014-04-25 at 09:14 -0700, Marc Eshel wrote:
> > >> I know, we are arguing here if it should be considered a pNFS
> data
> > >> operation.
> > >>
> different DSs and each DS should be the only one modifying its range,

That is just one very specific model for how to do pNFS. OnTAP, for
instance, does not do this.

>  if you want the MDS to do it you will have to recall the layout. I

Again, that is a model-specific statement. Recalling the layout is one
implementation.

>  don't think it is a big deal either way but I think we should have a
> clear model where data is modified only by the DSs unless there is an
> error and you have to fall back to the MDS. 

The blocks pNFS driver won't ever allow you to do this. In blocks, the
MDS MUST recall any outstanding layout segments that cover blocks that
will be deallocated.

If you are talking about the files layout, then maybe, however I
question the value of a hole punching mode, where the operation may
fail, and yet leave a partial hole. That doesn't fit the semantics of
the Linux fallocate(FALLOC_FL_PUNCH_HOLE) operation as far as I can
tell. It either succeeds, and leaves a hole, or it fails and leaves the
data. There is no warning in the manpage about it being able to corrupt
the file.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 25 Apr 2014 15:02:13 -0400

> hole is punched or nothing at all.

Right. I can't see how we can implement that if the client is
responsible for doing the hole punches one by one on each DS.

> down the possible failure cases for that as well.

Sure. However even DEALLOCATE can end up needing to allocate blocks if
there is a snapshot, for instance, pinning the blocks that need to be
partially zeroed out. A NFS client cannot therefore know a priori that
the operation will succeed or fail. Only the MDS has that information.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 25 Apr 2014 21:04:54 +0200

> data. There is no warning in the manpage about it being able to corrupt

The same is true for the F_UNRESVSP fcntl in IRIX.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 25 Apr 2014 21:06:02 +0200

> the operation will succeed or fail. Only the MDS has that information.

Right.  That's why I think DEALLOCATE must not be a operation that can be
sent to the DS.  WRITE_SAME when used to zero out a region on the other
hands sounds quite doable if we write down the rules for the error case
in detail.

---

From: Marc Eshel <[email-masked]>
Date: Fri, 25 Apr 2014 12:55:24 -0700

Trond Myklebust <[email-masked]> wrote on 04/25/2014 11:55:50 AM:

> <[email-masked]>, "nfsv4 list ([email-masked])" <[email-masked]>
> Date: 04/25/2014 11:55 AM

It is fanny that you use implementation specific to counter model 
specific. Like I said it easy do it all on the MDS for file, I was more 
concern about block and object. I was trying to have some simple rules 
that we can follow in our design but I guess it is too late with all the 
existing stuff that we have to deal with. I will have to go back and 
review the new document with Tom and Christoph changes and see if it all 
make sense. 
Marc.

---

From: Tom Haynes <[email-masked]>
Date: Fri, 25 Apr 2014 14:53:43 -0700

>> A NFS client cannot therefore know a priori that
> sent to the DS.

Then I think we need to call out that for both ALLOCATE and DEALLOCATE
that to paraphrase your earlier comment:

     they definitively need to be failure atomic, that is either the whole 
     hole is punched/space reserved or nothing at all.

This is regardless of whether these operations are to a singleton or a
pNFS server.