Subject: Re: [nfsv4] COPY failure behavior

From: Christoph Hellwig <[email-masked]>
Date: Thu, 1 May 2014 18:19:55 +0200

On Wed, Apr 30, 2014 at 11:25:08AM -0700, Thomas Haynes wrote:
> Expanding the audience:

Thanks.

Btw, all the issues here apply to WRITE_SAME in the same manner,
except there isn't even the hint of a "consecutive" flag.

> > 1) a COPY that does not extend the file size
> > 
> >  What parts of the region COPYed to are clobbered after a failed operation?
> > 
> >  For a consecutive COPY I would expect the behavior to be the same as
> >  WRITE: N bytes are written, those have the new data, everything else is
> >  unmodified.
> > 
> >  But what happens for a non-consecutive?  Should all of the region
> >  be assumed to be clobbered and may contain random sub-regions of the
> >  old and new data?
> 
> 
> Unfortunately, yes.

Unless we assume COW tricks underneath that's the only think I could
come up with either, but it will need careful documentation.

> > 2) a COPY that does extend the file size:
> > 
> >  In addition to the above considerations what will the file size after
> >  a failed COPY be?
> > 
> >  Writes in posix file systems make sure that the visible file size is only
> >  extended to the offset that now contains data, usually by truncating
> >  the file on a failed extending write.  If we want to implement this
> >  for COPY how do we exclude other writes into the same region beyond
> >  the file size?
> 
> Doesn't the lock protect us here?

If all the involved applications choose to use locks, yes.  But very
few applications actually do.

> >  Should non-consecutive COPYs that span the file size extent the file
> >  size at all in the error case, and if yes to the full size of the COPY
> >  or to the last offset that contains copied data.  What will subregions
> >  that haven't been copied but are lower in the file than a copied subregion
> >  contain?
> 
> Were the sub-regions to be copied or not?
> 
> If not, then the normal semantics apply.

Think of a non-consecutive that happens in some sort of interleaved pattern:

+-----------------+------------------------------------
| old file size   | past old file size
+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| AAA | BBB | CCC | AAA | BBB | CCC | AAA | BBB | CCC |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+

Where each A, B and C are copy streams that happen in parallel.

Now and I/O error happens and each of this region has been partially
copied:

+-----------------+------------------------------------
| old file size   | past old file size
+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| AAA | BBB | CCC | AA_ | BBB | C__ | ___ | BB_ | ___ |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+

What size does the file have after the failed COPY, and if the
new file size is larger than the old one and contains blocks
that have not been copied to, what will a READ from those blocks
return?

---

From: David Noveck <[email-masked]>
Date: Thu, 1 May 2014 14:42:13 -0400

Unintentional.  Sorry, everyone.

On Thu, May 1, 2014 at 1:59 PM, Christoph Hellwig <[email-masked]> wrote:

> Was the reply to me only intentional?  I think this should go to the list,
> especially as I haven't been driving COPY, just trying to poke holes into
> it..
>
> On Thu, May 01, 2014 at 01:55:16PM -0400, David Noveck wrote:
> > instead of returning a byte count plus a flag you moight consider
> > returning  two offsetss:
> >
> >    - offset1 is an offset such that all bytes that precede it are
> correctly
> >    updated.
> >    - offset2 is an offset such that no bytes beyond it have been updated
> >
> > offset1 == offset2 corresponds to to the consecutive flag being true.
> >
> > When offset1 != offset2, the consecutive flag would be false but you can
> > restart from offset1.
> >
> >
> > On Thu, May 1, 2014 at 12:19 PM, Christoph Hellwig <[email-masked]> wrote:
> >
> > > > >  WRITE: N bytes are written, those have the new data, everything
> else
> > > is
> > > > >  unmodified.
> > > > >
> > > > >  Writes in posix file systems make sure that the visible file size
> is
> > > only
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/nfsv4
> > >
> ---end quoted text---
>

---

From: Tom Haynes <[email-masked]>
Date: Thu, 8 May 2014 14:33:33 -0500

> except there isn't even the hint of a "consecutive" flag.

WRITE_SAME doesn’t need a “consecutive flag”, we remove the
fluff about not mandating the way ADBs are stored and add a new
section:

15.13.3.2.  Error Handling of a Partially Complete WRITE_SAME

   WRITE_SAME will clone adb_block_count copies of the given ADB in
   consecutive order in the file starting at adb_offset.  An error can
   occur after writing the Nth ADB to the file.  WRITE_SAME MUST appear
   to populate the range of the file as if the client used WRITE to
   transfer the instantiated ADBs.  I.e., the contents of the range will
   be easy for the client to determine in case of a partially complete
   WRITE_SAME.

> Think of a non-consecutive that happens in some sort of interleaved pattern:

I’d say that if the blocks were not copied to, then the filesystem
semantics of unallocated blocks apply here, i.e., most sane
ones will result in sparse semantics.

As to the length, it is that of the farthest byte touched in the
file.

I’ll put this back, we can tweak it as needed:

diff --git a/nfsv42_middle_op_copy.xml b/nfsv42_middle_op_copy.xml
index 4237a10..2d8413a 100644
--- a/nfsv42_middle_op_copy.xml
+++ b/nfsv42_middle_op_copy.xml
@@ -203,26 +203,12 @@
     </t>
 
     <t>
-      If an immediate failure does occur, wr_count will be set to the
-      number of bytes copied to the destination file before the error
-      occurred. If cr_consecutive is true, then the bytes were copied
-      in order. Otherwise, there is no assurance as to exactly which
-      bytes in the range were copied.
-    </t>
-
-    <t>
-      A return of NFS4_OK indicates that either the operation is
-      complete or the operation was initiated and a callback will
-      be used to deliver the final status of the operation.
-    </t>
-
-    <t>
-      If the wr_callback_id is returned, this indicates that the operation
-      was initiated and a CB_OFFLOAD callback will deliver the final
-      results of the operation. The wr_callback_id stateid is termed a
-      copy stateid in this context. The server is given the option of
-      returning the results in a callback because the data may require
-      a relatively long period of time to copy.
+      If the wr_callback_id is returned, this indicates that the
+      operation was initiated and a CB_OFFLOAD callback will deliver
+      the final results of the operation. The wr_callback_id stateid
+      is termed a copy stateid in this context. The server is given
+      the option of returning the results in a callback because the
+      data may require a relatively long period of time to copy.
     </t>
 
     <t>
@@ -240,5 +226,29 @@
       might encrypt, compress, deduplicate, or otherwise represent the
       on disk data in the source and destination file differently.
     </t>
+
+    <t>
+      If a failure does occur for a synchronous copy, wr_count
+      will be set to the number of bytes copied to the destination
+      file before the error occurred. If cr_consecutive is true,
+      then the bytes were copied in order. If the failure occured
+      for an asynchronous copy, then the client will have gotten
+      the notification of the consecutive copy order when it got
+      the copy stateid. It will be able to determine the bytes
+      copied from the coa_bytes_copied in the CB_OFFLOAD argument.
+    </t>
+
+    <t>
+      In either case, if cr_consecutive was not true, there is no
+      assurance as to exactly which bytes in the range were copied.
+      The client MUST assume that there exists a mixture of the
+      original contents of the range and the new bytes. If the COPY
+      wrote past the end of the file on the destination, then the
+      last byte written to will determine the new file size. The
+      contents of any block not written to and past the original
+      size of the file will be as if a normal WRITE extended the
+      file.
+    </t>
+
   </section>
 </section>

---

From: Tom Haynes <[email-masked]>
Date: Thu, 8 May 2014 15:56:10 -0500

<t>

    <t>
@@ -240,5 +226,29 @@