Subject: [nfsv4] [NFSv4.2] Meeting Notes 04/21/2011

From: Thomas Haynes <[email-masked]>
Date: Thu, 21 Apr 2011 13:58:54 -0500

Attendees:

Tom Haynes (NetApp)
Fred Isaman (NetApp)
Sorin Faibish (EMC)
Peter Staubach (EMC)
Dean Hildebrand (IBM)
Andy Adamson (NetApp)
Bruce Fields (Red Hat)

+ IETF Note Well Agreement

   http://www.ietf.org/NOTEWELL.html

+ Unstable file creation

Trond wants to get this into NFSv4.2.

Peter mentioned that he had a discussion with Trond on recovery race conditions.
He will dig that up.

Tom wondered about how NFS4_UNSTABLE_PNFS interacts with the
recent discussions of LAYOUT_COMMIT?

+ New I/O ops

++ Do we want to support large reads and writes?

We decided not to do so at this time.

++ Do we want to have an union which does holes/patterns/data?

This turned into a discussion of how are patterns meant to be used.

The common understanding was that the pattern was to be repeated
inside the block to detect corruption.

If it is not to be repeated and the information of concern is the
block number and guard pattern, then is it sufficient for the zero
blocks to be utilized? I.e., couldn't the application detect that
a hole was present and then fill in the required block number
(at the expected offset) and guard pattern?

[[-->
Some issues thought of post call:

1) Dumb applications like tar or dd might be used to copy the
file. In which case the disk blocks would be expanded. When the
file was written, dedup would be necessary to regain the
sparseness.

If dedup is present, having zero filled holes is optimal. I.e.,
if the patterns were applied during the expansion, then each
disk block would be unique because of the block number.
The dedup would thus have to be application aware to
shrink the actual storage.

2) Either the block size or guard pattern might be different
based on the expected use of the application data block.
In such a case, that information would need to be stored
with the file and not in the application.
<--]]

Bruce mentioned that the Enterprise Apps I-D did not
provide sufficient justification for why we would want to
consider doing patterns.

Andy wondered if Oracle could provide us with examples of
how corruption was detected.

Dean asked about how likely it was for Oracle to implement this
in the NFSv4.2 timeframe.

++ Do we want short reads or arrays of holes/patterns/data?

We decided to support both.

A server is always allowed to return a short read.

Using the array allows for fewer RPC calls.

++ WRITE_PATTERN will do pattern initialization and hole punching.

If the server does not supports sparse files, then it may return NFS4ERR_NOTSUPP
to the client upon such a request. However, to avoid sending a lot
of data across the wire, it may decide to modify the disk blocks
based on the given information.

Peter mentioned that Windows might have this issue. There was some issue
with 2G writes. Tom will try to track down people with implementation experience
with this wall.

Peter mentioned that the server might want to inform the client as to whether
a hole was created or blocks written.

Another point that came up, is the client willing to wait for the server to create
the disk blocks or to hole punch? I.e., if the client is not willing to wait, then
it will be responsible for driving the modifications.

The server may decide to not create a hole/pattern which is less than 1 disk
block. In this case, it will initialize the requested bytes.

++ For pNFS:

We tabled this item.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 21 Apr 2011 15:23:38 -0400

> Bruce Fields (Red Hat)

  [ ... snipped ... ]

> in the NFSv4.2 timeframe.

I don't see the Enterprise Apps I-D under http://datatracker.ietf.org/wg/nfsv4/ .  Doesn't that make it a personal draft?  Thus members of the working group would be better off directing comments specifically to the authors of the Enterprise Apps draft.  IIRC, the authors of the Enterprise Apps I-D are Mike Eisler (NetApp) and Margaret Susairaj (Oracle).

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 21 Apr 2011 14:42:14 -0500

On Apr 21, 2011, at 2:23 PM, Chuck Lever wrote:

> I don't see the Enterprise Apps I-D under http://datatracker.ietf.org/wg/nfsv4/ .  Doesn't that make it a personal draft?

It has expired.

>  Thus members of the working group would be better off directing comments specifically to the authors of the Enterprise Apps draft.  IIRC, the authors of the Enterprise Apps I-D are Mike Eisler (NetApp) and Margaret Susairaj (Oracle).

The disconnect here Chuck is that we are trying to pull that into the NFSv4.2 I-D. The
authors are both aware of this effort.

What was presented in the Meeting Notes are just that, notes of what went on. Under the
IETF Note Well Agreement, it is all open.

I guess I could have added AIs that I will take to contact the authors to ask for clarifications
of these comments, but I assumed those were apparent given my role as editor.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 21 Apr 2011 15:51:39 -0400

> authors are both aware of this effort.

I didn't want the working group's v4.2 effort to stall waiting for participation, or a response, from the wrong party.  Just saying "I wonder if Oracle..." doesn't seem actionable to me.

However, I'm satisfied as long as the working group is aware that "Oracle" is a very large company and isn't about to respond unless a specific request is made to the specific people responsible for that material.

Thanks for clarifying, Tom.

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 21 Apr 2011 14:00:50 -0700

Regretfully I was triple booked at this time. I do appreciate the
consideration being give to patterns.

Comments inline ...

> We decided not to do so at this time.

Fair enough.

> ++ Do we want to have an union which does holes/patterns/data?

Note that a hole is an unallocated region of a file. A pattern is
how the hole is represented.

A union of a pattern/data on the write side is unnecessarily complex. This
should be two operations.

A union a pattern/data on the read side is unavoidable.

> inside the block to detect corruption.

No.

http://tools.ietf.org/html/draft-eisler-nfsv4-enterprise-apps-01#section-2.4

"An example will illustrate how the client uses INITIALIZE.  Suppose
   the arguments (except for ia_stateid) are: { 0, 500, 1000, 8, 0,
   "DeadBeef" }.  Then starting with offset zero, the content of the
   file will have these contents.

   offset value (decimal or ASCII)
   0      0   0   0   0   0   0   0   0
   8      'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
   16-499 zeroes

   500    0   0   0   0   0   0   0   1
   508    'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
   516-999 zeroes

   ...

   499500  0   0   0   0   0   0   3   231
   499508  'D' 'e' 'a' 'd' 'B' 'e' 'e' 'f'
   499516-499999 zeroes

"

> (at the expected offset) and guard pattern?

No, because the presence of a zero could mean data corruption.

> file was written, dedup would be necessary to regain the

Yes. This is not the 1970s any more. Storage has advanced. New APIs
are needed.

> shrink the actual storage.

Storage is often application aware today. It does not strike me as
difficult for storage to detect the repeating "guard pattern" (ia_pattern)
and the increasing pattern iteration number to de-dupe. This could
conceivably be done  inline and or in the background. Since This would be
part of an official standard, namely NFSv4.2, storage vendors might be
motivated to add this to their de-dupe subsystems.

IOW, if you have made the investment in your storage implementation
to be sophisticated enough to do de-dupe, surely extending it to detect
patterns is a small incremental engineering effort.

> with the file and not in the application.

Or you detect it heuristically.

> provide sufficient justification for why we would want to

Because zeroes often indicate data corruption in addition to
uninitialized data. I suspect that one of the most frequently called
APIs in random O/S hardware is bzero() or memset(). Bzero() of a
block of memory that the caller thinks it owns but doesn't really is
pretty common bug. Or perhaps in 31 years of doing this stuff, it just
seems like I've fixed that bug a lot.

> how corruption was detected.

As I understand it, there are three states a block can be in:
- corrupt
- initialized
- allocated

Corrupt is anything not in the initialized and allocated states, and that
includes a block of all zeroes which is a common form of data corruption.

Initialized is as per the i-d from Oracle and NetApp.

Allocated means that there is real (i.e. valuable) data written on
the block. There will be a different ia_pattern value. There will also
be a checksum.

The model is that the app invokes initialize(fd, ...) on the file.

Then each time the database app wants to write to a block in the file, it
first reads the block to verify that it is in the expected initialized or
allocated state. If not, there is corruption, and recovery ensures.
Otherwise, the database app's block write includes:
- a new ia_pattern<> to indicate an allocated state
- the same block number
- the data
- a checksum of the entire block

A read of the database verifies the ia_pattern and block number, and
if applicable, verifies the checksum. If the verification fails,
the database is corrupt and recovery ensues.

As one can see, a block of zeroes will always be interpreted as
corruption. For that matter, so would a block of all ones. Indeed,
by crafting the ia_pattern<> string correctly (large prime numbers
look interesting to me), many common forms of
data corruption can be trivially filtered out.
> in the NFSv4.2 timeframe.

Are we now requiring implementation experience as condition to join the
train?

Regardless, understanding is that this is likely. Keep in mind that Oracle
does a lot of these mechanics today, except that instead of sending
a single INTIALIZE operation, Oracle has to send a lot of WRITE operations,
which impede provisioning of databases, and waste storage.

> ++ WRITE_PATTERN will do pattern initialization and hole punching.

Within Netapp we had a discussion about what if the NFS server's storage
does not support de-allocated regions within a file? The server might
still want to perform the writes local to its storage. Therefore leverging
the
COPY infrastructure for asynchronous and synchronous INITIALIZE might
be useful. As a NetApp employee I don't actually care since NetApp's
file system supports de-allocated regions.

> a hole was created or blocks written.

Maybe. What's the use case?

As long as the client creates the file with the space_reserved attribute,
it is golden. If it doesn't, then why does it care?

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Dean <[email-masked]>
Date: Thu, 21 Apr 2011 14:40:03 -0700

> A union a pattern/data on the read side is unavoidable.
One idea is to have something like a pattern type field.  If it is 1, 
then no pattern is needed and it is assumed to be zeroes, if it is some 
other value, then a pattern is needed.  This would reduce the data 
transferred for the more common zero case.
>     offset value (decimal or ASCII)
What is the expected block size?  If it closer to 500 or 4K or 1M, i.e., 
is it expected to be smaller than the file system block size or the NFS 
rsize?  The reason I ask is that if the sections of zeroes are large 
enough, then supporting just zeroes should be sufficient to reduce the 
data transferred.

Also, couldn't a file attr/xattr be used to implement store the 
pattern?  This would mean that future read requests would not need have 
a pattern included with every read operation.  Instead the server could 
just look at the pattern attribute and figure out patterns on the fly.
> Yes. This is not the 1970s any more. Storage has advanced. New APIs
I think this it the advantage of having the 'alloc' flag on the 
READ_PLUS response, which would allow smarter applications/clients to 
reproduce the file as close as possible to the original.
> which impede provisioning of databases, and waste storage.
At the meeting it was unclear whether Oracle supports the pattern 
concept presently, as it would seem silly to add a proposed product 
feature to the protocol.  If Oracle already supports this and is their 
defacto standard for detecting corruption, then my question is of less 
importance.

Dean

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 22 Apr 2011 11:00:35 -0500

On Apr 21, 2011, at 4:40 PM, Dean wrote:

> What is the expected block size?  If it closer to 500 or 4K or 1M, i.e., is it expected to be smaller than the file system block size or the NFS rsize?  The reason I ask is that if the sections of zeroes are large enough, then supporting just zeroes should be sufficient to reduce the data transferred.
> 
> Also, couldn't a file attr/xattr be used to implement store the pattern?  This would mean that future read requests would not need have a pattern included with every read operation.  Instead the server could just look at the pattern attribute and figure out patterns on the fly.

The intent all along has been for the server to store the pattern as metadata. Whether this
is in an extended inode or something else is implementation specific.

It is not sufficient for just the application to know the pattern. It has to be transferred to the
server. Any application (including dd, tar, cp, etc) that access the file over any protocol,
including NFSv3, NFSv4, NFSv4.1, NFSv4.2's READ operation, CIFS, etc, must get
the application data blocks with the ADBN and guard pattern set.

Furthermore, Mike's explanation of how an application might utilize these pieces of information
shows that the guard pattern is also a state indicator for an ADB.  I.e., initialized versus
allocated. It is easy enough to envision other states such as deallocated, which would
allow an application to detect in essence a dangling pointer.

As I've tried to indicate before, this approach being taken is very much like the memory
corruption detection used in OSes (see [1] and [2] for example).  One technique is to
write a guard pattern both before and after the payload data. By looking at the expected
state of the memory and the guard pattern, we can tell if there is a state mismatch or
if an over or under write has occurred.

We might want to consider adding a second guard to the ADBN - the cost is another
piece of information on the wire. The reason we don't have to do so with the scenario Mike presented
is that a checksum is present. If it is overwritten, corruption is detected. If the rest of
the block has a corruption, including where a second guard pattern would go, corruption
is detected. BTW - Oracle does currently use the checksum for detecting corruption [3].

Actually, I think a second guard can be handled by the application, just like the
checksum mentioned by Mike. In the definition of the initialized state, the only
things set are the ADBN and the guard pattern. If the application were to find
any other but set, it is corruption. In the transition to allocated, the application
now owns the contents and can set additional detection bytes, such as a checksum
or an end guard pattern.

Finally, as to whether Oracle is implementing this now or in the future seems like
a moot point. We are trying to drive up the adoption of NFSv4.x. The majority of
the new features targeted* to NFSv4.2 are geared towards that goal. Server-side copy
reduces network load, sparse files reduce both network load and disk consumption,
sparse files guarantee storage for an application, LNFS allows us entry to
a wide variety of use cases we are shut out of, deserialized file creation can be
a huge performance win.

Introducing this subset of the Enterprise App I-D will appeal to a variety of application
vendors. For example, I can easily see a VM application using this technique to manage
detection of corruption in their disk blocks.

Terms:

ADB - application data block
ADBN - ADB number - which block

Notes:

* I.e., ones we are seriously considering but which are not necessarily committed to being
on the train.

References:

[1] R. Hastings and B. Joyce. Purify: Fast detection of memory leaks and access errors. In Winter USENIX, 1992.

[2] http://dlc.sun.com/osol/docs/content/MODDEBUG/casestudy-26.html

[3] http://download.oracle.com/docs/cd/B28359_01/server.111/b28320/initparams046.htm

---

From: Trond Myklebust <[email-masked]>
Date: Sat, 23 Apr 2011 15:19:17 -0400

> Peter mentioned that he had a discussion with Trond on recovery race

I'm hoping that the discussion that was added in section 2.1.4 in draft
01 addresses Peter's concerns.

> recent discussions of LAYOUT_COMMIT?

You mean for the NFS files case, or in general?

For 'files', the only case that we need to care about is when
commit_to_mds is not set, and the DS is not setting the FILE_SYNC flag
on write. In that case, the MDS may want to set the 'NFS4_UNSTABLE_PNFS'
bit in 'stable_state' in order to indicate that it does not know if
there is any outstanding data that needs to be committed on the DSes.

In the case of an outstanding write layout in the 'objects' and 'blocks'
case too, the server may not know if the client has written any new data
to the DSes, and so again it uses the 'NFS4_UNSTABLE_PNFS' flag to
indicate that.

However, at all times it MUST know whether or not the ordinary metadata
(i.e. the basic inode metadata) has been committed to disk, since it is
responsible for doing so. So the lack of a NFS4_UNSTABLE_METADATA bit
should be interpreted as meaning that the filehandle is stable on disk,
the basic directory and inode information is stable on disk, and only
stuff that might be affected by the NFS4_UNSTABLE_METADATA and/or
NFS4_UNSTABLE_DATA (i.e. data + file size + change attribute +
ctime/mtime timestamps) bits being set are not guaranteed to be on disk
yet.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com