Subject: [nfsv4] [RFC] Add-client-requirements-to-COPY

From: Thomas Haynes <[email-masked]>
Date: Sun, 4 May 2014 10:40:21 -0700

From: Tom Haynes <[email-masked]>

It made sense to roll the synchrnous and consecutive
requirements from the CLIENT into one set because
of the error handling.

I'm opening this up for review.

TODO:

1) Need to straighten out wr_callback_id vs cr_synchronous
for indicating that an asychronous copy is occuring.

2) Need to make stronger language about the decision
to be synchronous or asynchronous MUST occur before
any bytes are transferred.

3) Need to organize Section 15.2.3, it is too scrambled
and thus hard to follow. (Follow on patch)

4) Need to write more about error handling (for other
scenarios, see Christoph's email). (Follow on patch)

Tom Haynes (1):
  Add client requirements to COPY

 dotx.d/copy_args.x                  |  2 ++
 dotx.d/copy_res.x                   | 11 ++++----
 dotx.d/head.x                       |  3 ++-
 nfsv42_middle_copy.xml              | 53 ++++++++++++++++++++++++-------------
 nfsv42_middle_errors.xml            | 19 ++++++++++---
 nfsv42_middle_new_errors.xml        | 19 ++++++++++---
 nfsv42_middle_op_cb_offload.xml     |  2 +-
 nfsv42_middle_op_copy.xml           | 47 +++++++++++++++++++++++++++++---
 nfsv42_middle_op_offload_abort.xml  |  2 +-
 nfsv42_middle_op_offload_status.xml |  2 +-
 10 files changed, 120 insertions(+), 40 deletions(-)

-- 
1.9.0

---

From: Thomas Haynes <[email-masked]>
Date: Sun, 4 May 2014 10:40:22 -0700

Signed-off-by: Tom Haynes <[email-masked]>
---

diff --git a/dotx.d/copy_args.x b/dotx.d/copy_args.x
index d2929dd..66bf938 100644
--- a/dotx.d/copy_args.x
+++ b/dotx.d/copy_args.x
@@ -6,6 +6,8 @@ struct COPY4args {
 	offset4		ca_src_offset;
 	offset4		ca_dst_offset;
 	length4		ca_count;
+	bool		ca_consecutive;
+	bool		ca_synchronous;
 	netloc4		ca_source_server<>;
 };
 
diff --git a/dotx.d/copy_res.x b/dotx.d/copy_res.x
index d2e80d8..a0c810a 100644
--- a/dotx.d/copy_res.x
+++ b/dotx.d/copy_res.x
@@ -1,7 +1,8 @@
-union COPY4res switch (nfsstat4 cr_status) {
-case NFS4_OK:
-	write_response4	resok4;
-default:
-	length4		cr_bytes_copied;
+
+struct COPY4res {
+	nfsstat4	cr_status;
+	write_response4	cr_response;
+	bool		cr_consecutive;
+	bool		cr_synchronous;
 };
 
diff --git a/dotx.d/head.x b/dotx.d/head.x
index d530793..d5c2457 100644
--- a/dotx.d/head.x
+++ b/dotx.d/head.x
@@ -179,7 +179,8 @@ enum nfsstat4 {
  NFS4ERR_UNION_NOTSUPP  = 10090,/* Arm of union not supp   */
  NFS4ERR_OFFLOAD_DENIED = 10091,/* dest not allowing copy  */
  NFS4ERR_WRONG_LFS      = 10092,/* LFS not supported       */
- NFS4ERR_BADLABEL       = 10093 /* incorrect label         */
+ NFS4ERR_BADLABEL       = 10093,/* incorrect label         */
+ NFS4ERR_OFFLOAD_NO_REQS= 10094 /* dest not meeting reqs   */
 };
 
 /*
diff --git a/nfsv42_middle_copy.xml b/nfsv42_middle_copy.xml
index 4ea51a8..8e9552d 100644
--- a/nfsv42_middle_copy.xml
+++ b/nfsv42_middle_copy.xml
@@ -1,5 +1,5 @@
-<!-- Copyright (C) The IETF Trust (2011-2013) -->
-<!-- Copyright (C) The Internet Society (2011-2013) -->
+<!-- Copyright (C) The IETF Trust (2011-2014) -->
+<!-- Copyright (C) The Internet Society (2011-2014) -->
 
 <section anchor="sec:copy" title="Server Side Copy">
   <section title="Introduction">
@@ -10,7 +10,7 @@
       transmitted back and forth over the network through the NFS client.
       Without this feature, an NFS client copies data from one location
       to another by reading the data from the source server over the network, and
-      then writing the data back over the network to the destiniation server.
+      then writing the data back over the network to the destination server.
     </t>
 
     <t>
@@ -370,13 +370,28 @@
 
     <section title="Server-to-Server Copy Protocol" anchor="sec:SSC-proto">
       <t>
-        The source server and
-        destination server are not required to use a specific protocol to
-        transfer the file data. The choice of what protocol to use
-        is ultimately the destination server's decision.
+        The source server and destination server are not required to
+        use a specific protocol to transfer the file data. The choice
+        of what protocol to use is ultimately the destination server's
+        decision. However, the destination server has to be cognizant
+        that it is working on behalf of the client.
       </t>
 
-      <section title="Using NFSv4.x as a Server-to-Server Copy Protocol">
+      <section title="Considerations on Selecting a Copy Protocol">
+        <t>
+          The client may have requirements over both the size of
+          transactions and error recovery semantics. It may want to
+          split the copy up such that each chunk is synchronously
+          transferred. It may want the copy protocol to copy the bytes
+          in consecutive order such that upon an error, the client can
+          restart the copy at the last known good offset. If the
+          destination server cannot meet these requirements, the client
+          may prefer the traditional copy mechanism such that it
+          can meet those requirements.
+        </t>
+      </section>
+
+      <section title="Using NFSv4.x as the Copy Protocol">
         <t>
           The destination server MAY use standard NFSv4.x (where x >= 1)
           operations to read the data from the source server. If NFSv4.x is used
@@ -387,7 +402,7 @@
         </t>
       </section>
 
-      <section title="Using an alternative Server-to-Server Copy Protocol">
+      <section title="Using an Alternative Copy Protocol">
         <t>
           In a homogeneous environment, the source and destination servers
           might be able to perform the file copy extremely efficiently using
@@ -669,7 +684,7 @@
          compound authentication and RPC application defined structured
          privilege assertions. The combination of these features allows the
          destination server to authenticate to the source server as acting on
-         behalf of the user prinicpal, and to authorize the destination server
+         behalf of the user principal, and to authorize the destination server
          to perform READs of the file to be copied from the source on behalf
          of the user principal. Once the copy is complete, the client can
          destroy the RPCSEC_GSSv3 handles to end the source and destination
@@ -678,7 +693,7 @@
         <t>
          RPCSEC_GSSv3 introduces the notion of RPC application defined
          structured privileges.  We define three structured privileges
-         that work in tandum to authorize the copy:
+         that work in tandem to authorize the copy:
         </t>
 
         <t>
@@ -769,7 +784,7 @@
          and the source server to OPEN the file to be copied to provide the
          the necessary user principal identification to the source server
          from the destination server (acting as a client). This is accomplished
-         by sending the user principal indentification information: e.g
+         by sending the user principal identification information: e.g
          the rpc_gss3_gss_binding fields, in the
          copy_to_auth privilege established between the client and the
          destination server. This same information is then placed in the
@@ -835,7 +850,7 @@
              as explained above.
              A nounce is created, and GSS_MIC() is invoked on the
              nounce using the RPCSEC_GSSv1 (or v2) context shared between
-             user prinicpal and the source server. The nounce, nounce MIC,
+             user principal and the source server. The nounce, nounce MIC,
              context handle used to create the nounce MIC, and the context
              handle version are added to the copy_to_auth_priv instance which
              is placed in
@@ -921,7 +936,7 @@
              initiator principal, and nfs@&lt;source&gt; as the target
              principal.
              If NFS is to be used as the copy protocol, this means that the
-             destiniation server must mount the source server using RPCSEC_GSS.
+             destination server must mount the source server using RPCSEC_GSS.
            </t>
 
             <t>
@@ -943,7 +958,7 @@
             <t>
              The copy_confirm_auth RPCSEC_GSS3_CREATE call also includes
              a compound authentication component. The rpc_gss3_gss_binding
-             fields are filled in with information from the estalished
+             fields are filled in with information from the established
              "copy_to_auth" privilege
              (see <xref target="sec:SSC-sec-estab" />).
              The ctap_handle_vers, ctap_handle,
@@ -981,7 +996,7 @@
              All subsequent ONC RPC requests sent from the destination to
              copy data from the source to the destination will use the
              RPCSEC_GSSv3 handle returned by the source's RPCSEC_GSS3_CREATE
-             response. Note that as per the Compound Authenticaion section of
+             response. Note that as per the Compound Authentication section of
              <xref target="rpcsec_gssv3" />
              the resultant RPCSEC_GSSv3 context handle is bound to
              the user principal RPCSEC_GSS context and so it MUST be treated by
@@ -1010,7 +1025,7 @@
         </t>
 
       </section>
-      <section title="Finishing or Stoping a Secure Inter-Server Copy">
+      <section title="Finishing or Stopping a Secure Inter-Server Copy">
         <t>
          Under normal operation, the client MUST destroy the copy_from_auth
          and the copy_to_auth RPCSEC_GSSv3 handle once the COPY operation
@@ -1044,7 +1059,7 @@
          check on the status of an asynchronous copy, it uses the privileged
          "copy_to_auth" RPCSEC_GSSv3 handle and the osa_stateid in
          OFFLOAD_STATUS MUST be the same as the wr_callback_id specified
-         in the "copy_to_auth" privilege stored on the destiniation server.
+         in the "copy_to_auth" privilege stored on the destination server.
         </t>
 
         <t>
@@ -1052,7 +1067,7 @@
          cancel an asynchronous copy, it uses the privileged "copy_to_auth"
          RPCSEC_GSSv3 handle and the oaa_stateid in OFFLOAD_ABORT MUST be
          the same as the wr_callback_id specified in the "copy_to_auth"
-         privilege stored on the destiniation server. The destiniation server
+         privilege stored on the destination server. The destination server
          will then delete the &lt;"copy_to_auth", user id, source list,
          nounce, nounce MIC, context handle, handle version&gt; privilege and
          the associated "copy_confirm_auth" RPCSEC_GSSv3 handle.
diff --git a/nfsv42_middle_errors.xml b/nfsv42_middle_errors.xml
index 3da98c5..0bb25e3 100644
--- a/nfsv42_middle_errors.xml
+++ b/nfsv42_middle_errors.xml
@@ -87,6 +87,7 @@
       <c>NFS4ERR_NO_GRACE                  </c> <c> 10033  </c> <c> <xref target="ss:error_values:NC149" /> </c>
       <c>NFS4ERR_NXIO                      </c> <c> 6      </c> <c> <xref target="ss:error_values:NC51" /></c>
       <c>NFS4ERR_OFFLOAD_DENIED            </c> <c> 10091  </c> <c> <xref target="ss:error_values:NC191" /></c>
+      <c>NFS4ERR_OFFLOAD_NO_REQS           </c> <c> 10094  </c> <c> <xref target="ss:error_values:NC194" /></c>
       <c>NFS4ERR_OLD_STATEID               </c> <c> 10024  </c> <c> <xref target="ss:error_values:NC131" /> </c>
       <c>NFS4ERR_OPENMODE                  </c> <c> 10038  </c> <c> <xref target="ss:error_values:NC145" /> </c>
       <c>NFS4ERR_OP_ILLEGAL                </c> <c> 10044  </c> <c> <xref target="ss:error_values:NC113" /> </c>
@@ -1167,10 +1168,20 @@
 
       <section title="NFS4ERR_OFFLOAD_DENIED (Error Code 10091)" anchor="err_OFFLOAD_DENIED">
         <t>
-          The copy offload operation is supported by
-          both the source and the destination, but the destination is not
-          allowing it for this file.  If the client sees this error, it
-          should fall back to the normal copy semantics.
+          The copy offload operation is supported by both the source
+          and the destination, but the destination is not allowing it
+          for this file.  If the client sees this error, it should fall
+          back to the normal copy semantics.
+        </t>
+      </section>
+
+      <section title="NFS4ERR_OFFLOAD_NO_REQS (Error Code 10094)" anchor="err_OFFLOAD_NO_REQS">
+        <t>
+          The copy offload operation is supported by both the source and
+          the destination, but the destination can not meet the client
+          requirements for either consecutive byte copy or synchronous
+          copy. If the client sees this error, it should either relax the
+          requirements (if any) or fall back to the normal copy semantics.
         </t>
       </section>
 
diff --git a/nfsv42_middle_new_errors.xml b/nfsv42_middle_new_errors.xml
index 123fb44..325fc68 100644
--- a/nfsv42_middle_new_errors.xml
+++ b/nfsv42_middle_new_errors.xml
@@ -22,6 +22,7 @@
 
       <c>NFS4ERR_BADLABEL</c>         <c>10093</c> <c><xref target="err_BADLABEL" /></c>
       <c>NFS4ERR_OFFLOAD_DENIED</c>   <c>10091</c> <c><xref target="err_OFFLOAD_DENIED" /></c>
+      <c>NFS4ERR_OFFLOAD_NO_REQS</c>  <c>10094</c> <c><xref target="err_OFFLOAD_NO_REQS" /></c>
       <c>NFS4ERR_PARTNER_NO_AUTH</c>  <c>10089</c> <c><xref target="err_PARTNER_NO_AUTH" /></c>
       <c>NFS4ERR_PARTNER_NOTSUPP</c>  <c>10088</c> <c><xref target="err_PARTNER_NOTSUPP" /></c>
       <c>NFS4ERR_UNION_NOTSUPP</c>    <c>10090</c> <c><xref target="err_UNION_NOTSUPP" /></c>
@@ -50,10 +51,20 @@
 
       <section title="NFS4ERR_OFFLOAD_DENIED (Error Code 10091)" anchor="err_OFFLOAD_DENIED">
         <t>
-          The copy offload operation is supported by
-          both the source and the destination, but the destination is not
-          allowing it for this file.  If the client sees this error, it
-          should fall back to the normal copy semantics.
+          The copy offload operation is supported by both the source
+          and the destination, but the destination is not allowing it
+          for this file.  If the client sees this error, it should fall
+          back to the normal copy semantics.
+        </t>
+      </section>
+
+      <section title="NFS4ERR_OFFLOAD_NO_REQS (Error Code 10094)" anchor="err_OFFLOAD_NO_REQS">
+        <t>
+          The copy offload operation is supported by both the source and
+          the destination, but the destination can not meet the client
+          requirements for either consecutive byte copy or synchronous
+          copy. If the client sees this error, it should either relax the
+          requirements (if any) or fall back to the normal copy semantics.
         </t>
       </section>
 
diff --git a/nfsv42_middle_op_cb_offload.xml b/nfsv42_middle_op_cb_offload.xml
index d94ead0..1aaf487 100644
--- a/nfsv42_middle_op_cb_offload.xml
+++ b/nfsv42_middle_op_cb_offload.xml
@@ -28,7 +28,7 @@
 
       <list style="hanging">
         <t hangText='COPY:'>
-          for both intra- and inter-server asynchronous copies
+          for both intra-server and inter-server asynchronous copies
         </t>
 
         <t hangText='WRITE_SAME:'>
diff --git a/nfsv42_middle_op_copy.xml b/nfsv42_middle_op_copy.xml
index 597bd93..8cc8149 100644
--- a/nfsv42_middle_op_copy.xml
+++ b/nfsv42_middle_op_copy.xml
@@ -128,6 +128,44 @@
     </t>
 
     <t>
+      If ca_consecutive is set, then the client has specified that
+      the copy protocol selected MUST copy bytes in consecutive
+      order from ca_src_offset to ca_count. If the destination
+      server cannot meet this requirement, then it MUST return
+      an error of NFS4ERR_OFFLOAD_NO_REQS and set cr_consecutive
+      to be false. Likewise, if ca_synchronous is set, then the
+      client has required that the copy protocol selected MUST
+      perform a synchronous copy. If the destination server
+      cannot meet this requirement, then it MUST return an
+      error of NFS4ERR_OFFLOAD_NO_REQS and set cr_synchronous
+      to be false.
+    </t>
+
+    <t>
+      If both are set by the client, then the destination SHOULD
+      try to determine if it can respond to both requirements
+      at the same time. If it cannot make that determination,
+      it must set to false the one it can and set to true the
+      other. The client, upon getting an NFS4ERR_OFFLOAD_NO_REQS
+      error, has to examine both cr_consecutive and cr_synchronous
+      against the respective values of ca_consecutive and
+      ca_synchronous to determine the possible requirement
+      not met. It MUST be prepared for the destination server
+      not being able to determine both requirements at the same
+      time.
+    </t>
+
+    <t>
+      Upon receiving the NFS4ERR_OFFLOAD_NO_REQS error, the client has
+      to determine if it wants to either re-request the copy with a
+      relaxed set of requirements or if it wants to revert
+      to manually copying the data. If it decides to manually
+      copy the data and this is a remote copy, then the
+      client is responsible for informing the source that the
+      earlier COPY_NOTIFY by sending it an OFFLOAD_REVOKE.
+    </t>
+
+    <t>
       The copying of any and all attributes on the source file is
       the responsibility of both the client and the copy protocol.
       Any attribute which is both exposed via the NFS protocol on
@@ -161,10 +199,11 @@
     </t>
 
     <t>
-      If an immediate failure does occur, cr_bytes_copied will be
-      set to the number of bytes copied to the destination file before
-      the error occurred. The cr_bytes_copied value indicates the number
-      of bytes copied but not which specific bytes have been copied.
+      If an immediate failure does occur, wr_count will be set to the
+      number of bytes copied to the destination file before the error
+      occurred. If cr_consecutive is true, then the bytes were copied
+      in order. Otherwise, there is no assurance as to exactly which
+      bytes in the range were copied.
     </t>
 
     <t>
diff --git a/nfsv42_middle_op_offload_abort.xml b/nfsv42_middle_op_offload_abort.xml
index bf2fe2c..e9aa6aa 100644
--- a/nfsv42_middle_op_offload_abort.xml
+++ b/nfsv42_middle_op_offload_abort.xml
@@ -12,7 +12,7 @@
 
   <section toc='exclude' title="DESCRIPTION">
     <t>
-      OFFLOAD_ABORT is used for both intra- and inter-server
+      OFFLOAD_ABORT is used for both intra-server and inter-server
       asynchronous copies. The OFFLOAD_ABORT operation allows the client
       to cancel a server-side copy operation that it initiated. This operation
       is sent in a COMPOUND request from the client to the destination
diff --git a/nfsv42_middle_op_offload_status.xml b/nfsv42_middle_op_offload_status.xml
index acc0acf..1a6bd64 100644
--- a/nfsv42_middle_op_offload_status.xml
+++ b/nfsv42_middle_op_offload_status.xml
@@ -12,7 +12,7 @@
 
   <section toc='exclude' title="DESCRIPTION">
     <t>
-      OFFLOAD_STATUS is used for both intra- and inter-server asynchronous
+      OFFLOAD_STATUS is used for both intra-server and inter-server asynchronous
       copies. The OFFLOAD_STATUS operation allows the client to poll the
       destination server to determine the status of an asynchronous copy operation.
     </t>
-- 
1.9.0