Subject: [nfsv4] Patterns: INITIALIZE or WRITE_PLUS

From: Thomas Haynes <[email-masked]>
Date: Fri, 8 Apr 2011 02:41:55 -0500

I'm looking at the Enterprise Apps I-D, trying to decide whether we
want to have that INITIALIZE op or a WRITE_PLUS op.

But that can wait. In looking at the INITIALIZE op, the I-D states:

   The INITIALIZE operation is used to initialize an open file to an
   iterated pattern.  The pattern consists of a fixed string, and a
   block number.  The pattern is defined by the arguments.

   o  ia_offset: where to start the iterated pattern.  This value is
      specified in bytes.

   o  ia_blocksize: the size of each iteration of the pattern.  Each
      iteration is called a block.

We are missing a description of block count here...

   o  ia_reloff_pattern: the relative offset within a block where to
      write the specified pattern encoded in ia_pattern.

   o  ia_reloff_blocknum: the relative offset within a block where to
      write a 64 bit block number.  The block number is incremented once
      a block is written.  The block number is always written in little
      endian order.  If ia_reloff_blocknum is set to NFS4_UINT64_MAX,
      then this informs the server that no block number is to be
      written.

   o  ia_pattern: a fixed string written to every block.  If the length
      of ia_pattern is zero, then this informs the server that no string
      is to be written.

   The field ia_stateid is the stateid corresponding to the current
   filehandle's share reservation, delegation, or byte range lock.

   An example will illustrate how the client uses INITIALIZE.  Suppose
   the arguments (except for ia_stateid) are: { 0, 500, 1000, 8, 0,
   "DeadBeef" }.  Then starting with offset zero, the content of the
   file will have these contents.

   offset value (decimal or ASCII)
   0      0   0   0   0   0   0   0   0
   8      'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
   16-499 zeroes

   500    0   0   0   0   0   0   0   1
   508    'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
   516-999 zeroes

   ...

   499500  0   0   0   0   0   0   3   231
   499508  'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
   499516-499999 zeroes

2.5.  IMPLEMENTATION

   When an NFS server receives this operation, instead of writing the
   iterated pattern over each block, it should de-allocate the data of
   affected range of the file and record the the values of ia_offset,
   ia_blocksize, ia_blockcount, ia_reloff_pattern, ia_reloff_blocknum,
   and ia_pattern<> in the file's system metadata.  When a client sends
   a READ request, instead of returning zeroes, it should construct a
   response corresponding to the pattern specified in the arguments to
   INITIALIZE.

[1] I see two ways to interpret what gets sent back over the wire to READ_PLUS:

1) A single hole of size ia_blocksize * ia_blockcount with the pattern ia_pattern.

2) A data array of size a_blocksize * ia_blockcount with the contents as described
above.

If the application is looking for a block number inside the block, then I don't see
how we can either have a sparse file on the wire or send a hole back across the
wire.

[2] And if the application is expecting to see the contents described above, then I don't
see how it can detect badness inside one of the blocks. Say a client reads the above
1000 blocks of data. Then assume it inadvertently dirties the 565th block, but makes
no changes to the block.

The client flushes that back to the server, which will most likely cause a client sized
page (pick either 2k or 4k) around that byte to be made into actual content. The server splits
the metadata it has into two different holes described by the pattern. There is also
a client sized page sitting on disk.

Besides there being scenarios where a block is split (thus not recording the block
number on the disk), there is now no way for the app to tell whether or not any of the
data outside of the block number and recorded pattern has been corrupted. I.e.,
this is the same argument presented in the I-D as to why we want a non-zero pattern.

I don't see the use case for why the block number needs to be encoded as part of the
pattern and hence stored in the block. Further, I don't see the motivation for why we
want to mess with recording anything about blocks at all.

I.e., if the app wants to store data in these app blocks, then it can calculate which
block it should be in.

A counter-proposal would be a WRITE_PLUS op in the same spirit as the READ_PLUS op:

struct hole_pattern4 {
        offset4         hp_offset;
        length4         hp_length;
};

struct READ_PLUS4args {
        /* CURRENT_FH: file */
        stateid4        rpa_stateid;
        offset4         rpa_offset;
        count4          rpa_count;
}; 

union read_plus_res4 switch (bool is_hole) {
case TRUE:
        hole_pattern4   rpr_hole;
case FALSE:
        opaque          rpr_data<>;
}; 

union READ_PLUS4res switch (nfsstat4 status) {
case NFS4_OK:
        read_plus_res4  resok4;
default:   
        void;      
}; 

struct write_plus_data4 {
        offset4         wpd_offset;
        opaque          wpd_data<>;
};

union write_plus_arg4 switch (bool is_hole) {
case TRUE:
        hole_pattern4           wpa_hole;
case FALSE:
        write_plus_data4        wpa_data;
};

struct WRITE_PLUS4args {
        /* CURRENT_FH: file */
        stateid4        wpa_stateid;
        stable_how4     wpa_stable;
        write_plus_arg4 wpa_data;
};

struct WRITE_PLUS4resok {
        count4          wpr_count;
        stable_how4     wpr_committed;
        verifier4       wpr_writeverf;
};

union WRITE_PLUS4res switch (nfsstat4 status) {
case NFS4_OK:
        WRITE_PLUS4resok    resok4;
default:
        void;
};

Note that this approach is simpler than what we might finally adopt. E.g.,
we should consider the 64 bit array lengths for the data, we might want
to be able to return an array of holes and non-holes, etc.

But it does present a consistent approach to the new READ and WRITE ops.

The new WRITE_PLUS op incorporates both the INITIALIZE and HOLE_PUNCH
functionality from the other I-Ds.

To initialize a file, create it (via CREATE or OPEN) and in the same compound,
send a single WRITE operation over the range of the file. This has the effect
of only sending the hole_pattern4 and results in a sparse file on the server.

---

From: "Mike Eisler" <[email-masked]>
Date: Fri, 8 Apr 2011 06:25:19 -0700

inline ...

> We are missing a description of block count here...

It is the count of the number of consecutive instances of the pattern to
apply to the file.

>    offset value (decimal or ASCII)

So change READ_PLUS to indicate that a pattern is being returned.

> 1000 blocks of data. Then assume it inadvertently dirties the 565th block,

I don't understand how you can dirty something and not change it.

> a client sized page sitting on disk.

Yes. What's the point?

> this is the same argument presented in the I-D as to why we want a

I don't understand.

> want to mess with recording anything about blocks at all.

The authors have experience with data corruption caused by underyling file
systems mapping data that was at logical offset X to logical offset Y.
> of only sending the hole_pattern4 and results in a sparse file on the

I don't care what the operation is called. And if it results in fewer new
operations, that might be goodness, depending on switched union
complexity. But AFAIK, your proposal does not allow a non-zero pattern to
be expressed via write_plus.

>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 11 Apr 2011 18:40:38 -0500

> So change READ_PLUS to indicate that a pattern is being returned.

Yes. But first I wanted to see the requirements fleshed out a bit.

> I don't understand how you can dirty something and not change it.

Bug?

Yeah, I struggled when trying to write that up. I'll finish it up below.

>> a client sized page sitting on disk.

Consider that you dirty the N'th word with something valid. 

  500    0   0   0   0   0   0   0   1
  508    'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
  516-699 zeroes
  700    'f' '4' 'c' 'e' '7' '1' '0' '4'
  708- 999 zeroes

It gets written back to the server.

Sometime later, the app rereads it from the server. How does the app
know whether or not a bogus writ has occurred where all of the zeros
are present?

Another approach would be to state that the pattern is to be repeated
across the rest of this application logical block:

  500    0   0   0   0   0   0   0   1
  508    'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
  516-699  'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
  700    'f' '4' 'c' 'e' '7' '1' '0' '4'
  708- 999  'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'

And if the pattern did not evenly go into the block size, it would be
truncated on the last occurrence.

And as both READ_PLUS and WRITE_PLUS would understand
this common pattern, it would be the client's responsibility to expand
the pattern.

Or to put it another way:

1) Pattern aware server, client uses READ, then server expands

2) Pattern aware server, client uses READ_PLUS, then client expands.

> systems mapping data that was at logical offset X to logical offset Y.

Thank you for supplying the use case.

The following XDR uses the pattern data structure from the Enterprise Apps I-D 
with READ_PLUS and WRITE_PLUS. Note that it has the complexity required to
support multiple holes.

struct hole_pattern4 {
        offset4         hp_offset;
        length4         hp_blocksize;
        length4         hp_blockcount;
        length4         hp_reloff_pattern;
        length4         hp_reloff_blocknum;
        opaque          hp_pattern<>;
};

struct READ_PLUS4args {
        /* CURRENT_FH: file */
        stateid4        rpa_stateid;
        offset4         rpa_offset;
        count4          rpa_count;
}; 

        read_plus_res4  resok4<>;
default:   
        void;      
}; 

struct WRITE_PLUS4args {
        /* CURRENT_FH: file */
        stateid4        wpa_stateid;
        stable_how4     wpa_stable;
        write_plus_arg4 wpa_data<>;
};

---

From: Dean <[email-masked]>
Date: Tue, 12 Apr 2011 10:23:02 -0700

>          length4         hp_reloff_blocknum;

I don't think we should merge the zero and pattern responses, but create 
2 separate struct's.  With every field being 64bits, the standard sparse 
response just went from 128bits to 384bits.  A big waste for 99% of all 
use cases.
> union READ_PLUS4res switch (nfsstat4 status) {

So the current online draft seems to be corrupted and has COPY_NOTIFY in 
place of READPLUS.  So I can't tell what you have merged into the 
current NFSv4.2 draft.

But what happened to eof?
Dean

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 12 Apr 2011 13:02:32 -0500

On Apr 12, 2011, at 12:23 PM, Dean wrote:

> I don't think we should merge the zero and pattern responses, but create 2 separate struct's.  With every field being 64bits, the standard sparse response just went from 128bits to 384bits.  A big waste for 99% of all use cases.

Okay, I think this is a valid point.

It turns out a pattern is more than a repeated string...

> So the current online draft seems to be corrupted and has COPY_NOTIFY in place of READPLUS.  So I can't tell what you have merged into the current NFSv4.2 draft.

I don't want to keep on spamming the IETF with every change. The git repository already
has this fix.

> But what happened to eof?

Ack!

Let me rewhack the XDR to include your point above.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 12 Apr 2011 15:34:02 -0400

> union WRITE_PLUS4res switch (nfsstat4 status) {

How would any of this work in a pNFS-centric world (or have we replaced
the concept of "scale out" with "scale back")?

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 12 Apr 2011 14:40:14 -0500

> the concept of "scale out" with "scale back")?

Is the question the fact that the INITIALIZE operation would go to the MDS
and the WRITE_PLUS will go to the DS?

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 12 Apr 2011 16:54:11 -0400

> and the WRITE_PLUS will go to the DS?

(First an aside: Why do we want to replace WRITE with something that
does WRITE and holes/patterned data? It sort of makes sense to replace
READ with READ_PLUS (although probably not in a pNFS striped world)
since the client doesn't know whether to expect a hole or data, but if
it is writing data, it definitely knows what kind of information it is
sending.
IOW: I can see a need for a WRITE_PATTERNED, which does holes and
patterned fill, but not WRITE_PLUS, which does holes, patterned fill and
ordinary writes.)

That said, if you are writing patterned data, how is that supposed to
work in a pNFS world? Do you have to send a WRITE_(PATTERNED|PLUS)|
INITIALIZE to all the data servers? If so, how do you implement the same
functionality for non-files layout types?
If you send it to the MDS, then how does that interact with pNFS? Does
it cause layouts to be recalled? Presumably it has to in the case of
blocks and objects, but what about files?

BTW: the fact that layouts are described in separate documents rather
than being part of the main protocol is going to make adding new NFSv4.x
features "interesting". If I add 'feature X' to the main protocol, and
someone invents a new layout type that may be affected by said feature X
(because it may require a layout recall, or layout commit, or...) then
who is responsible for describing what happens, and in which document?

NetApp

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 12 Apr 2011 16:18:33 -0500

> patterned fill, but not WRITE_PLUS, which does holes, patterned fill and

The Enterprise Apps proposes 64 bit data lengths, which would cause a
new WRITE operation.

> functionality for non-files layout types?

Assuming we want the large data lengths and thinking off the top of my head:

WRITE_PLUS is for writing data, so it would go to the DSes.

WRITE_PATTERN is for writing patterns|holes, so it would go to the MDS.

And through the magic of the Control Protocol, the MDS would communicate
the pattern to the DSes.

> blocks and objects, but what about files?

Doesn't 5661 already describe what happens if a client has a layout and
tries to perform I/O through the MDS?

Based on the way this discussion is going, I would expect us to follow those
rules.

> who is responsible for describing what happens, and in which document?

The new layout type is responsible as far as I can tell. It is the one
deviating from the main protocol. Of course, with a new minor version
once per year, it might be easy to sync a change. <ducks />

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 12 Apr 2011 18:33:58 -0400

> The Enterprise Apps proposes 64 bit data lengths, which would cause a

Interesting... I'd be curious to see what the motivation is for that...
Are there any servers out there that actually support 4GB writes today?
What is the reason for wanting an atomic > 4GB write instead of just
splitting it up?

> tries to perform I/O through the MDS?

No. It punts even on basic information such as when it is appropriate or
not to recall a layout. If client developers want to to manage their I/O
efficiently, they do need to know this sort of thing.

> once per year, it might be easy to sync a change. <ducks />

Does that mean that each layout type may need to be revisited and
revised every time we publish a new minor version? That would certainly
be an efficient way to get rid of all the deadwood.

NetApp

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 18 Apr 2011 13:31:21 -0500

> I don't think we should merge the zero and pattern responses, but create 2 separate struct's.  With every field being 64bits, the standard sparse response just went from 128bits to 384bits.  A big…

Dean,

I have some observations on that statement.

If a hole or a pattern replaces a 32K chunk of data, does it matter if it is 128 bits or 384 bits?

And how many holes/patterns do we expect in relationship to data?

I.e., if holes/patterns are rare, then 99% is just as rare. Is optimizing for
~260 bits that important?

Also, if we allow for holes, patterns, and data, is it possible for someone
to implement holes and not patterns?

Thanks,
Tom

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 18 Apr 2011 13:39:33 -0500

> What is the reason for wanting an atomic > 4GB write instead of just

I'm pretty sure that there are not. The Enterprise Apps I-D points to 10G
ethernet and the future.

I wanted to pull this in simply to avoid having READ_PLUS_PLUS in
the next couple of years.

---

From: Dean <[email-masked]>
Date: Mon, 18 Apr 2011 23:35:43 -0700

> If a hole or a pattern replaces a 32K chunk of data, does it matter if it is 128 bits or 384 bits?
Its a good point, but it is still tripling the amount of data that could 
have been sent in the zero sparse case.
> And how many holes/patterns do we expect in relationship to data?
Well, I guess every sparse file is unique in its own way.  For VMs for 
example, it is dependent on how much data is in the image and the file 
system's allocation algorithm, but VMWare claims that customers regain 
up to 40% of their disk space when they switch to thin provisioned disk 
images.  So I guess a 2:3 ratio of hole to data is not unreasonable.
> ~260 bits that important?
Not sure if you are saying that sparse files are rare or if the number 
of holes in a sparse file are few?  If the latter, in one OS disk image 
we analyzed, it had over 10000 holes in its first 12GBs (can't remember 
the min size, at least 4K).
> to implement holes and not patterns?
I'll let the community decide this one.
Dean

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 19 Apr 2011 10:53:26 -0500

On Apr 19, 2011, at 1:35 AM, Dean wrote:

> Its a good point, but it is still tripling the amount of data that could have been sent in the zero sparse case.
> Well, I guess every sparse file is unique in its own way.  For VMs for example, it is dependent on how much data is in the image and the file system's allocation algorithm, but VMWare claims that customers regain up to 40% of their disk space when they switch to thin provisioned disk images.  So I guess a 2:3 ratio of hole to data is not unreasonable.
> Not sure if you are saying that sparse files are rare or if the number of holes in a sparse file are few?  

I would have been stating that holes per file are rare.

> If the latter, in one OS disk image we analyzed, it had over 10000 holes in its first 12GBs (can't remember the min size, at least 4K).

Thanks for the data.

What OS and how did they represent data?

> I'll let the community decide this one.

And that would be a non-compliant NFSv4.2 implementation.

---

From: Dean <[email-masked]>
Date: Tue, 19 Apr 2011 09:41:54 -0700

> I would have been stating that holes per file are rare.
Many holes per file, or sparse files in general?  I think we all agree 
that sparse files are not rare, which is why we are adding sparse 
support to the spec.  Regarding the number of holes, I only have limited 
data.
> What OS and how did they represent data?
This was just a rhel5 OS with ext3 and some iometer test files.
> And that would be a non-compliant NFSv4.2 implementation.
I don't understand this statement either.  Are you asking whether 
implementers can support holes and not patterns or are you just stating 
that you don't want this to happen?
Dean

---

From: "Mike Eisler" <[email-masked]>
Date: Tue, 19 Apr 2011 14:28:28 -0700

inline ...

>>>> to implement holes and not patterns?

This distinguishing of patterns and holes seems incorrect to me. A hole is
a deallocated space. In most file systems, its value is zero. We are
finding that that is not generally useful.

A suggestion: We should (re?)-define some terminology.

Pattern: fixed sized unit of data consisting of a region of zeroes
overlayed with an optional fixed string, and an optional variable 64 bit
unsigned integer that represents of the iteration of the pattern.

Trivial pattern: a pattern with no fixed string or pattern iteration, i.e.
all zeroes.

Hole: a region of data in a file that has contents that are equal to a
pattern. In the interest of space efficiency underlying file system
implementation can optimizing its allocation algorithm to store a pattern
just once and assigned to each hole a pattern identifier. However, this
does not preclude the use of the space_reserved attribute to ensure that
when/if a hole or part of a hole is over written with a a non-pattern that
that the over write is successful.

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 19 Apr 2011 17:00:58 -0500

> unsigned integer that represents of the iteration of the pattern.

Yes, I think this definition of pattern is key to understanding it. A pattern
is not taking a hole and stating the string is something other than
zeros.

I.e., you can't take a hole of '00 00 00 00' and say it is now a hole of
'de ad be ef'. That is not a pattern.

I would state a trivial pattern as being a pattern in which every byte
is the same.

---

From: <[email-masked]>
Date: Mon, 25 Apr 2011 16:13:04 -0400

Returning to the pNFS question, pattern support almost certainly varies by layout type.

The block layout is simple, because block device interfaces (e.g., SCSI) have no support for patterns.  The block range covered by the pattern is never granted in a layout, causing the pNFS client to read from the MDS, which in turn causes the proverbial "right thing" to happen (server returns space-optimized version of pattern that client expands into data to return to application).

For the file layout, the file data servers will have to be part of the pattern support for that to work as intended, but it should be ok if they have the support (modulo some block numbering interaction with striping).

For the object layout, I have no idea - Benny?

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Haynes
> Sent: Tuesday, April 19, 2011 6:01 PM
> To: Mike Eisler
> Cc: [email-masked]
> Subject: Re: [nfsv4] Patterns: INITIALIZE or WRITE_PLUS
> 
>

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 25 Apr 2011 15:55:32 -0500

> Returning to the pNFS question, pattern support almost certainly varies by layout type.

Ahh, the question was different:

If we allow both a pattern and a hole to be represented in the INITIALIZE and READ_PLUS
operations, is it conceivable that someone implements holes, but not patterns?

And that actually raises a far more interesting, and generic question, given the minor
versioning rules of NFSv4.x, is it possible to return NFS4ERR_NOTSUPP for one
variant of an operation, but not another?

Look at rule 3 of the Minor Versioning Rules:

   3.   Minor versions must not modify the structure of an existing
        operation's arguments or results.

        Again, the complexity of handling multiple structure definitions
        for a single operation is too burdensome.  New operations should
        be added instead of modifying existing structures for a minor
        version.

        This rule does not preclude the following adaptations in a minor
        version:

        *  adding bits to flag fields, such as new attributes to
           GETATTR's bitmap4 data type, and providing corresponding
           variants of opaque arrays, such as a notify4 used together
           with such bitmaps

        *  adding bits to existing attributes like ACLs that have flag
           words

        *  extending enumerated types (including NFS4ERR_*) with new
           values

        *  adding cases to a switched union

So if we define FOO thusly in NFSv4.2 and make it optional:

enum foo_enum {
  FOO_READ_NORMAL = 0
};

struct foo_reader {
       offset_t  fr_start;
};

union foo_content switch (foo_enum bar) {
case FOO_READ_NORMAL:
       foo_reader       fc_normal;
default:
        void;
};

struct FOO4args {
        foo_content     fa_content;
};

And if further, we redefine FOO in NFSv4.3 to include a new variant:

const NFS4_TWO_GB            = 0x80000000;

typedef opaque twoGB_byte_array4[NFS4_INT32_MAX];

struct fourGB_buffer4 {
     twoGB_byte_array4[2];
}

struct large_buffer4 {
        fourGB_buffer4 lb_big_buffers<>;
        opaque         lb_small_buffer<>;
}

struct foo_large_reader {
        large_buffer4        rwar_data;
};

enum foo_enum {
  FOO_READ_NORMAL = 0,
  FOO_READ_LARGE = 1
};

struct foo_reader {
       offset_t  fr_start;
};

union foo_content switch (foo_enum bar) {
case FOO_READ_NORMAL:
       foo_reader       fc_normal;
case FOO_READ_LARGE:
       foo_large_reader fc_large;
default:
        void;
};

struct FOO4args {
        foo_content     fa_content;
};

I.e., I've come up with a way to add large file support to new READ and WRITE operations without
having to form yet more operations.

An implementation may decide to ship NFSv4.3 without large file support, but with normal file
support for FOO.

I would argue that if it got:

PUTFH FOO {FOO_READ_LARGE} ...

it could return NFS4ERR_NOTSUPP for FOO, but that only means it is not supported for the given
value of the foo_enum.

I.e., if given

PUTFH FOO {FOO_READ_NORMAL}, then it would never return NFS4ERR_NOTSUPP. (Especially
if we made FOO mandatory in NFSv4.3.)