Subject: Re: [nfsv4] Copy Offload Questions

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 5 Aug 2013 21:25:08 +0000

Here is my proposal to handle the ca_destination issue raised in this thread.

Any comments?

loghyr:NFSv4.2 thomas$ git diff !$
git diff nfsv42_middle_op_copy.xml
diff --git a/nfsv42_middle_op_copy.xml b/nfsv42_middle_op_copy.xml
index 104a1d5..6221c88 100644
--- a/nfsv42_middle_op_copy.xml
+++ b/nfsv42_middle_op_copy.xml
@@ -1,5 +1,5 @@
-<!-- Copyright (C) The IETF Trust (2011-2012) -->
-<!-- Copyright (C) The Internet Society (2011-2012) -->
+<!-- Copyright (C) The IETF Trust (2011-2013) -->
+<!-- Copyright (C) The Internet Society (2011-2013) -->
 
 <section anchor="op:copy" title="Operation 59: COPY - Initiate a server-side copy">
   <section toc='exclude' title="ARGUMENT">
@@ -16,8 +16,7 @@
       copies. In both cases, the COPY is always sent from the client
       to the destination server of the file copy. The COPY operation
       requests that a file be copied from the location specified by the
-      SAVED_FH value to the location specified by the combination of
-      CURRENT_FH and ca_destination.
+      SAVED_FH value to the location specified by the CURRENT_FH.
     </t>
 
     <t>
@@ -69,26 +68,14 @@
     </t>
 
     <t>
-      The CURRENT_FH and ca_destination together specify the
-      destination of the copy operation. If ca_destination
-      is of 0 (zero) length, then CURRENT_FH specifies the target file.
-      In this case, CURRENT_FH MUST be a regular file and not a directory.
-      If ca_destination is not of 0 (zero) length, the ca_destination
-      argument specifies the file name to which the data will be copied
-      within the directory identified by CURRENT_FH. In this case,
-      CURRENT_FH MUST be a directory and not a regular file.
-    </t>
-
-    <t>
-      If the file named by ca_destination does not exist and the operation
-      completes successfully, the file will be visible in the file system
-      namespace. If the file does not exist and the operation fails, the
-      file MAY be visible in the file system namespace depending on when
-      the failure occurs and on the implementation of the NFS server
-      receiving the COPY operation. If the ca_destination name
-      cannot be created in the destination file system (due to
-      file name restrictions, such as case or length), the operation
-      MUST fail.
+      The CURRENT_FH specifies the destination of the copy operation.
+      The CURRENT_FH MUST be a regular file and not a directory. Note,
+      the file MUST exist before the COPY operation begins. It is the
+      responsibility of the client to create the file if necessary,
+      regardless of the actual copy engine used.
+      If the file cannot be created in the destination file system (due to
+      file name restrictions, such as case or length), the COPY
+      operation MUST not be called.
     </t>
 
     <t>

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 5 Aug 2013 18:10:45 -0400

> Here is my proposal to handle the ca_destination issue raised in this thread.

OK by me.

(Of course this'll also mean dropping some fields from the xdr, removing
language explaining how to handle unsupported attributes at the
destination, etc.)

> +      operation MUST not be called.

--b.

---

From: Bryan Schumaker <[email-masked]>
Date: Tue, 6 Aug 2013 09:02:19 -0400

> language explaining how to handle unsupported attributes at the

Looks good to me, too!

Bryan

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 8 Aug 2013 15:36:57 -0400

> > Here is my proposal to handle the ca_destination issue raised in this thread.

One other copy offload question: do we have any way for a server to
indicate that it supports copy on only some filesystems?

(For example supporting it only on filesystems that have some special
fast copy-on-write operation seems like an easy first implementation
that we'd like to allow.)

Maybe we can state that something like OFFLOAD_DENIED could handle that
case?

--b.

---

From: Bryan Schumaker <[email-masked]>
Date: Thu, 8 Aug 2013 15:44:29 -0400

My Linux implementation calls into the vfs copy function to do everything.  This allows the vfs to implement a generic copy() to use as a fallback if a specific fs doesn't have something yet.

- Bryan

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 8 Aug 2013 15:52:48 -0400

> My Linux implementation calls into the vfs copy function to do everything.  This allows the vfs to implement a generic copy() to use as a fallback if a specific fs doesn't have something yet.

Can't we just do the "fast" cases without any fallback and avoid having
to deal with the asynchronous case of COPY in the initial
implementation?

--b.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 9 Aug 2013 20:25:41 +0000

> language explaining how to handle unsupported attributes at the

Why does the attribute handling have to change?

>> +      operation MUST not be called.

If they are in the same compound, yes. What if they are in different compounds?

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 9 Aug 2013 16:54:54 -0400

> Why does the attribute handling have to change?

I assumed COPY only set attributes in the create case.  Looking at the
text...  I guess I was confused, there really isn't an explicit "create"
case, instead it copies attributes whenever COPY4_METADATA is set and
ca_count is zero.

Weird.  Why would COPY want to do anything with attributes in the case
the file already exists?

Anyway, if we drop file-creation then I think we should drop ca_flags
too, and everything to do with attributes.  The client can set
attributes copied from the source when it does the open, if it wants to.

> If they are in the same compound, yes. What if they are in different compounds?

That wouldn't really even make sense, since you wouldn't have the
destination filehandle in the case the create failed.

And in any case, there doesn't have to be an immediately preceding
create at all--we can copy to a preexisting file.

So I think the above "it is the responsibility of the client to create
file if necessary" already says what needs to be said better.

--b.

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 11 Aug 2013 20:28:28 +0000

On Aug 9, 2013, at 1:54 PM, J. Bruce Fields <[email-masked]<mailto:[email-masked]>> wrote:

On Aug 5, 2013, at 3:10 PM, J. Bruce Fields <[email-masked]<mailto:[email-masked]>> wrote:

OK by me.

So we are slowly moving to a model where the only supported copy protocol
will be NFSv4.x, where x >= 1.

By removing ca_destination, we've removed the ability of a server to
create the file and the metadata atomically. Further, any copy protocol
which operates by removing the destination file and creating a new
file object will cause the open and lock states to go away.

If we remove COPY4_METADATA (which has to go if ca_destination goes),
then we also at the very least violate Labeled NFS. I.e., regardless of the
copy protocol used, NFSv4.2 semantics ensures that a copy of a file
starts out with the same object label.

The COPY4_METADATA language is already broken - it states that if
the flag is set, the destination MUST fail with NFS4ERR_ATTRNOTSUPP
if it does not support the RECOMMENDED attributes. There is no
requirement however that the attribute MUST be copied.

I also believe that if space_reserved is available on the source but
not the destination that the COPY does not have to fail. In such a
scenario, the client should decide what it wants to have done.

If we look back at the XDR in draft 19, we see:

   struct COPY4args {
           /* SAVED_FH: source file */
           /* CURRENT_FH: destination file or */
           /*             directory           */
           stateid4        ca_src_stateid;
           stateid4        ca_dst_stateid;
           offset4         ca_src_offset;
           offset4         ca_dst_offset;
           length4         ca_count;
           uint32_t        ca_flags;
           component4      ca_destination;
           netloc4         ca_source_server<>;
   };

Bryan pointed out that ca_dst_stateid does not make sense if CURRENT_FH is
a directory object.

So we removed ca_destination.

It would be equally valid to provide a special stateid as ca_dst_stateid when
CURRENT_FH is a directory object. The problem then being that we would
need to add a stateid to the COPY4res. This points out that a COPY with
the CURRENT_FH as a directory object is in fact a variant of OPEN.

This means that no matter what is the copy protocol, then the client
MUST follow every COPY operation that has the CURRENT_FH as a directory
object with a GETFH in the same COMPOUND procedure. And at the end
of the entire server-side copy, it must use a CLOSE operation.

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 11 Aug 2013 22:25:40 +0000

On Aug 11, 2013, at 1:28 PM, "Haynes, Tom" <[email-masked]<mailto:[email-masked]>> wrote:

Sorry, this was mentioned earlier in the draft:

   If the metadata flag is set and the client is requesting a whole file
   copy (i.e., ca_count is 0 (zero)), a subset of the destination file's
   attributes MUST be the same as the source file's corresponding
   attributes and a subset of the destination file's attributes SHOULD
   be the same as the source file's corresponding attributes.  The
   attributes in the MUST and SHOULD copy subsets will be defined for
   each NFS version.

But given this text and the fact that any copy protocol could be used:

   In the case of an inter-server copy or an intra-server copy between
   file systems, the attributes supported for the source file and
   destination file could be different.  By definition,the REQUIRED
   attributes will be supported in all cases.  If the metadata flag is
   set and the source file has a RECOMMENDED attribute that is not
   supported for the destination file, the copy MUST fail with
   NFS4ERR_ATTRNOTSUPP.

I fail to see how the COPY operation can indeed fail with NFS4ERR_ATTRNOTSUPP
in this scenario. How does the destination know that the source file has any of the
attributes set?

The copy protocol may not even try to transfer those required attributes.

Also, say consider two different copy protocols:

CPA: Does minimal metadata, then does full data, then adjusts the meta-data.
CPB: Does majority of metadata, then does full data, then does final meta-data touch-ups.

If the file being copied is 30G, then CPA could fail asynchronously after the
bulk of the COPY is done. Whereas CPB could fail right away for all intents
and purposes.

One approach to solve this would be to add the attributes to the COPY4args and
let the operation always fail right away.

Besides this problem:

If we look at owner and owner_group, it isn't clear whether the destination
is supposed to pick these out of the COPY operation or they are supposed
to be copied over via the copy protocol. I.e., consider a proprietary copy
protocol that sent over a raw inode and data blocks between two servers
running the same OS. Is the owner the one recorded in the source inode
or that of the user doing the file copy?

On the whole, I agree with Bruce that we would be better served getting
rid of COPY4_METADATA and letting either the client or copy protocol
handle the attributes as needed. In the traditional copy, it is the client
implementation of copy which determines what attributes are set.

I do not think the protocol should mandate whether we do a 'cp':

loghyr:NFSv4.2 thomas$ ls -la nfsv42_rfc_start.xml
-rw-r--r--  1 thomas  staff  1646 Mar 13 22:43 nfsv42_rfc_start.xml
loghyr:NFSv4.2 thomas$ cp nfsv42_rfc_start.xml xxx
loghyr:NFSv4.2 thomas$ ls -la xxx
-rw-r--r--  1 thomas  staff  1646 Aug 11 15:22 xxx

Or a 'cp -p':

loghyr:NFSv4.2 thomas$ cp -p nfsv42_rfc_start.xml yyy
loghyr:NFSv4.2 thomas$ ls -la yyy
-rw-r--r--  1 thomas  staff  1646 Mar 13 22:43 yyy

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 12 Aug 2013 11:46:53 -0400

> will be NFSv4.x, where x >= 1.

We're pretty much just *removing* features from COPY.

If there's a server-to-server copy protocol we wish to support that does
not support the remaining features of COPY, then was it ever sufficient
to implement COPY?

If it's a requirement to allow such a protocol, we would probably need
changes.  I'm not convinced the cloud based object operations Alex
McDonald mentions are sufficient to implement the existing protocol
anyway.  I suspect they'd be better off with something completely
different from COPY.

> create the file and the metadata atomically.

The proposal is to make the client do the create.  OPEN can already set
attributes on new files.  As far as I can tell this only loosens
requirements on the server-to-server protocol by making creation and
attribute copying the client's responsibility.

In fact, I don't see any language in the current spec requiring that
COPY set attributes atomically on create, so if we want that we probably
want to make it explicit.

If we require COPY to copy attributes between servers, that makes it
more rather than less likely that the servers are going to be talking
NFSv4.x to each other....

> Further, any copy protocol

So I guess such a server would only be able to do whole-file copies, and
would only support COPY with ca_destination non-zero length and
ca_src_offset == ca_dst_offset = ca_count == 0?

Doesn't that need to be mentioned in the spec?  Maybe I missed it.

At a minimum we'd probably need an agreed-upon error for such a server
to return when the client attempts a copy it doesn't support.
(OFFLOAD_DENIED?)

> starts out with the same object label.

Again, this problem is already solved by OPEN.

> CURRENT_FH is a directory object.

I think the point of ca_dst_stateid was to allow the client to set up
some sort of locking on the destination file if it wanted to, for
example by taking an advisory lock on the destination range to make the
copy look atomic to other processes using locking.

If the server acquires this lock on behalf of the client, what sort of
lock should it be?  A byte-range lock, an open (what kind of open, with
what share bits?), or something else?

I'm not sure it makes sense.

> The problem then being that we would

I'd be happier if we could allow COPY to *just* move ranges of data
around and know nothing about file creation or attributes.

That's a much simpler model and I think we're much more likely to
specify it correctly.

Turning COPY into an OPEN sounds a lot more complicated.

If we really need to deal with a backend that can't do that, then I
think we need a lot more information about the use case.

--b.

---

From: "McDonald, Alex" <[email-masked]>
Date: Tue, 13 Aug 2013 17:47:19 +0000

[snip]

> anyway.  I suspect they'd be better off with something completely different
> from COPY.

True, since the model;

                            192.168.33.0/24
                 +-------------------------------------+
                 |                                     |
                 |                                     |
                 | 192.168.33.18                       | 192.168.33.56
         +-------+------+                       +------+------+
         |     Source   |                       | Destination |
         +-------+------+                       +------+------+
                 | 10.11.78.18                         | 10.11.78.56
                 |                                     |
                 |                                     |
                 |             10.11.78.0/24           |
                 +------------------+------------------+
                                    |
                                    |
                                    | 10.11.78.243
                              +-----+-----+
                              |   Client  |
                              +-----------+

may not be suitable. The model I would like to see has not only atomic operations, but encompasses copy-from and copy-to operations. That is, C ---> S ---> T and C --->T <--- S where the client has connectivity to only the source or the target -- but not both.

(BTW, don't think of this as "cloud" or "object" operations; think of them as file or sub-file movement operations that take place over geo distance with incomplete connectivity and high latency. And atomic ops are much better because they don't require the client to direct the cleanup, or get involved beyond the original request.)

I'll go back to sleep now :-)

[snip]

Alex McDonald
Standards & Industry Associations Group
CTO Office
NetApp
+44 7795 046686 Mobile Phone
[email-masked]
netapp.com
twitter: @alextangent

---

From: Bryan Schumaker <[email-masked]>
Date: Thu, 29 Aug 2013 16:55:02 -0400

Hi again,

Just a quick clarification, is it valid for the server to set wr_committed to UNSTABLE4 in the write_response4 for COPY?  I'm guessing this would cause the client to send a COMMIT after the callback returns?

- Bryan

On 07/25/2013 10:27 AM, Bryan Schumaker wrote:
> Hi,
> 
> I have a Linux prototype of the COPY operation, but I need a few clarifications about the spec.
> 
> 1) Why does CB_OFFLOAD need a stateid and a filehandle to represent the transaction?  Shouldn't a stateid alone be enough?
> 
> 2) The WRITE operation uses a count4 (uint32) to represent the amount of data written, but COPY uses length4 (uint64).  Is this to account for large files?
> 
> ~ Bryan
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>