Subject: [nfsv4] [NFSv4.2] Meeting Agenda

From: Thomas Haynes <[email-masked]>
Date: Thu, 7 Apr 2011 09:24:58 -0500

Today's proposed agenda is below. Please let me know if
you would like to add additional topics.

   Time: Thursdays 12:30-1:30 PM CST/1:30-2:30 PM EST/10:30-11:30 AM PST
Dial-in: 1-888-765-3653 (us)
         303-218-2659 (international)
     Id: 2787108

Proposed Agenda
---------------

+ IETF Note Well Agreement

 This is a reminder that our discussions are governed by the 
 IETF Note Well Agreement. See:

   http://www.ietf.org/NOTEWELL.html

 We will start each week's meeting with this announcement.

+ I want to focus this week on the recent WG discussion on fadvise. We have
two proposals in place.

+ I also want us to start stripping the I/O pattern operations from the various
proposals in front of us.

The client wants to be able to create a file that has a certain pattern. I.e.,
it wants to be able to create sparse files with that pattern as the default.

The server wants to be able to return a pattern on reads. I.e., if there is
a sparse file, it does not want to clog the network with that pattern.

The client wants to be able to write a pattern to a file. I.e., it might want
to hole punch that pattern into a file. This deallocates any existing blocks
used by the file in the data range.

The key items here are that:

1) One copy of the pattern per create/write/read.

2) A pattern need not have blocks storing it on the server.

+ Space reservation

A client wants to be able to guarantee storage for a file.

+ Unstable file creation

Trond presented a new feature request in Prague that would deserialize
file creation (think untar).

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 7 Apr 2011 13:26:54 -0500

Attendees:

Tom Haynes (NetApp)
Fred Isaman (NetApp)
Jarrett Lu (Oracle)
James Lentini (NetApp)
Pranoop Erasani (NetApp)
Peter Staubach (EMC)
Andy Adamson (NetApp)

+ fadvise 

We identified that we need use cases for both proposals to be
discussed on the WG alias.

Pranoop also wanted to know how many systems use the Posix fadvise interface?

Finally, there needs to be some followup discussion about the use
cases for having the fadvise included with the read and write operations
as proposed in the Enterprise Apps I-D.

+ Patterns

We discussed rolling the various sparse file proposals into one common
thread for NFS4.2. In particular:

Create a file with a given pattern - sparse file
Read a pattern from a file 
Write a pattern to a file - hole punch

We discussed what happens if a client wanted to READ an 8k chunk of file
and the first 4k were data and the second 4k were a pattern hole. I.e., do
we want to allow operations to use a mix of holes and data?

The simplest approach would be for the server to return 4k and allow the
client to send a second compound to get the remaining data.

The current XDR does not allow a mixture of holes and data. We should consider
if we can allow for an arbitrary number of mixtures.

I also mentioned that resolving this area would be where I am spending
the chunk of my editorial time at first.

+ Space reservation

No real discussion.

+ Unstable file creation

Pushed off until the next meeting.

+ Overview of what is in NFS4.2

Gave a quick summary of the items in, close to being in, etc. This was based
off of the slides I presented in Prague.

---

From: Nico Williams <[email-masked]>
Date: Thu, 7 Apr 2011 13:41:26 -0500

> off of the slides I presented in Prague.

I could use more detail on this if anything was discussed beyond what
appears on those slides.

Nico
--

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 7 Apr 2011 14:19:28 -0500

> I could use more detail on this if anything was discussed beyond what

No, nothing more was discussed. I had just realized that not everyone had been
present for either the interim or IETF80 meetings.

The two slides of interest are:

Items onboard

* Bug fixes 
  - LAYOUTRETURN can provide errors to MDS 
  – Client can register CB for LEASE_MOVED
  – Change attribute semantic
* Functional Changes 
  – NFS Server-Side Copy
    * Need to pull in RPCSEC_GSS_v3
  – Simple and efficient read support for sparse files
  – NFS space reservation operations

Items at the ticket booth

* Server side caching of file creation (Myklebust)
* Enterprise Apps (Haynes)
  – Mainly want to leverage synergy with sparse files
* MAC Security Label Support (Myklebust/Faibish)
  - Need to pull in RPCSEC_GSS_v3

And the name in parenthesis was who I identified as the sponsor,
so not necessarily the author.

---

From: Dean <[email-masked]>
Date: Thu, 07 Apr 2011 15:12:02 -0700

I'm sorry I missed the meeting, but try to attend this in the future.

Just a couple questions/comments,
> Pranoop also wanted to know how many systems use the Posix fadvise 
Since it is a posix standard, I think in time more and more OSs will 
support fadvise (for files) and madvise (for mmaped files). I think a 
more interesting question is how many OSs could utilize such *hints* and 
do the posix fadvise hints support all of them.

Windows seems to support several flags on OPEN, including 
FILE_FLAG_NO_BUFFERING (direct i/o), FILE_FLAG_RANDOM_ACCESS, 
FILE_FLAG_SEQUENTIAL_SCAN.
See http://msdn.microsoft.com/en-us/library/aa363858%28VS.85%29.aspx

Solaris supports posix madvise but adds non-portable flags (which we 
could look into supporting) and also supports a directio() call.

Linux supports fadvise, madvise, and direct I/O.

AIX supports direct I/O.

> Write a pattern to a file - hole punch
Just to be clear, we should differentiate between a hole punch that 
deallocates blocks (which would be very useful) and a hole punch that 
writes a specified pattern.  Maybe we can support both?
> the chunk of my editorial time at first.

I'm currently updating my sparse draft to include comments from the 
interim meeting, but this has been delayed due to both project and paper 
deadlines as well as vacation.  My plan is to state that the return 
value is completely the prerogative of the server, who is the only one 
who can decide if it wants to classify a byte-range as a hole or not.  
In practice, I see servers using a sparse threshold, and any unallocated 
or all-zero/pattern byte range less than this threshold would NOT be 
considered a hole.

As for returning a mix of holes and data (which would boil down to 
returning an array of data chunks, with the rest considered holes, I 
think it is useful but may be a little complicated.

Dean
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 7 Apr 2011 18:52:41 -0500

On Apr 7, 2011, at 5:12 PM, Dean wrote:

> Just to be clear, we should differentiate between a hole punch that deallocates blocks (which would be very useful) and a hole punch that writes a specified pattern.  Maybe we can support both?

They are both the same thing.

If you write a pattern, you always want to deallocate blocks. Assuming of course
you write at least a block of data.

If you write less than a block of data, then it is up to the server if it wants to fill
the block with the pattern. And once that block becomes all of the pattern, then
the server could collapse it to a hole.

> I'm currently updating my sparse draft to include comments from the interim meeting,

Why are you updating your draft? This has now been pulled into the NFSv4.2 document.
All of the items pulled into the release are being tracked there. This message has been
consistently pushed on the WG alias and at the meetings.

By the way, you need to pull the XDR code down for the current READPLUS case. I had
to change it to get it to pass through rpcgen. I will not be accepting changes that do not
pass the 'make testx' target of the Makefile.

You can grab the source here: https://github.com/loghyr/NFSv4.2

To get the 'make testx' work on Linux, you might need to install:

 347  sudo yum install libgssglue-devel
 364  sudo yum install nfs-utils-lib-devel

> but this has been delayed due to both project and paper deadlines as well as vacation.  My plan is to state that the     return value is completely the prerogative of the server, who is the only one who can decide if it wants to classify a byte-range as a hole or not. 

Agreed.

> In practice, I see servers using a sparse threshold, and any unallocated or all-zero/pattern byte range less than this threshold would NOT be considered a hole.

That would be an implementation choice.

> 
> As for returning a mix of holes and data (which would boil down to returning an array of data chunks, with the rest considered holes, I think it is useful but may be a little complicated.

That is an item which we should be discussing.

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 8 Apr 2011 02:50:07 -0500

> As for returning a mix of holes and data (which would boil down to returning an array of data chunks, with the rest considered holes, I think it is useful but may be a little complicated.

A subtle difference caused by patterns is that nothing specifies that
all holes in a file are going to have the same pattern. We could
easily have:

                        +-------------+--------------------+
                        | Byte-Range  | Contents           |
                        +-------------+--------------------+
                        | 0-31999     | Data               |
                        | 32K-255999  | Hole of "deadbeef" |
                        | 256K-287999 | Data               |
                        | 288K-353999 | Hole of "514y5407" |
                        | 354K-417999 | Data               |
                        +-------------+--------------------+

An analogy would be how some OSes detect memory corruption with different
patterns used for guards around the data and for when the memory is released.

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 8 Apr 2011 03:26:55 -0500

> all holes in a file are going to have the same pattern. We could

An even more interesting observation is that you can easily have adjacent holes:

                        | 354K-417999 | Hole of "a1b2c3d4" |

---

From: <[email-masked]>
Date: Fri, 8 Apr 2011 08:55:46 -0400

This feels excessive.  I've been through the corresponding exercise in SCSI, and that came down to:
- Add support to the protocol that allows a server to announce that reads of holes always return zeroes.
- Allow servers to get more creative (e.g., in the fashion that Tom describes below).

Do we actually need the latter in NFS?  IMHO, it's simpler to leave it out because that functionality results in an "interesting" corner case - if the client specifies the pattern that reading the hole returns, but the server doesn't support returning that pattern for a hole, the result of the attempt to punch a hole is a bulk write instead of a hole punch (see the SCSI WRITE SAME command, which has hole punch functionality specified), which may not be what the client wanted ;-).

If we don't need the "creative" option, it's simpler to require that reading a hole always returns zeroes.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Haynes
Sent: Friday, April 08, 2011 3:50 AM
To: Dean
Cc: [email-masked]
Subject: Re: [nfsv4] [NFSv4.2] Meeting Notes 04/07/2011

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 8 Apr 2011 10:37:20 -0500

On Apr 8, 2011, at 7:55 AM, <[email-masked]> <[email-masked]> wrote:

> This feels excessive.  I’ve been through the corresponding exercise in SCSI, and that came down to:
> Do we actually need the latter in NFS?  IMHO, it’s simpler to leave it out because that functionality results in an “interesting” corner case – if the client specifies the pattern that reading the hole returns, but the server doesn’t support returning that pattern for a hole, the result of the attempt to punch a hole is a bulk write instead of a hole punch (see the SCSI WRITE SAME command, which has hole punch functionality specified), which may not be what the client wanted ;-).

David,

Let me try to address a point which you may be trying to make before I address
the real issue.

With the new code, the server either supports holes with patterns or no holes at all.

If the client does:

PUTFH WRITE_PLUS "f00dbarf"

the server if it does not support holes, will return NFS4ERR_NOTSUPP.

At that point, the client knows that it it wants to get a sparse file, it either has to
rely on something server-vendor specific or it is out of luck.

I.e., it could then proceed with

PUTFH WRITE "f00dbarff00dbarf....f00dbarf"

But, the client is very aware of what is going on here.

>  
> If we don’t need the “creative” option, it’s simpler to require that reading a hole always returns zeroes.

The real issue is that we are being asked by Oracle to provide a means to
be creative here. All zeros might be acceptable if the Oracle DB was able
to query the underlying client OS to determine whether the memory corresponded
to a hole. I.e., if it knows it is a hole, it SHOULD NOT care what pattern is there.

So is the creative approach more complex? Yes.

But just like wanting to bring in the 64 bit array lengths that Mike is proposing,
I'd like us to take this chance to only redo READ and WRITE once.

(Which means I think I can have the correct XDR in about 5 minutes. :-> )

Tom

---

From: Dean <[email-masked]>
Date: Fri, 08 Apr 2011 10:12:29 -0700

> you write at least a block of data.

This is very much not the case.  There is a big difference between have 
a block of all zeroes and have no block at all.  Performance favours the 
former, thin-provisioning favours the latter.  VMWare give the users the 
option of creating thick and thin disk images for just this reason.

So, I think we want to avoid the case where someone has intentionally 
created a *thick* file, and then we end up deallocating blocks within 
it, which will slow performance when those blocks are written again and 
could cause problems since they are no longer guaranteed to have space 
for those deallocated blocks.

>> interim meeting,
Well, there was a clear request by Spencer and others at the end of the 
interim meeting for me to clean it up to only include what was discussed 
and add in several comments and issues.

Now that it has been pulled into the NFSv4.2 document, how do I make 
these changes (as described below, but there was also several more 
discussed at the meeting)?
Dean
>> but this has been delayed due to both project and paper deadlines as 
>> well as vacation.  My plan is to state that the return value is 
>> completely the prerogative of the server, who is the only one who can 
>> decide if it wants to classify a byte-range as a hole or not.
>
>
> Agreed.
>
>
>> In practice, I see servers using a sparse threshold, and any 
>> unallocated or all-zero/pattern byte range less than this threshold 
>> would NOT be considered a hole.
>
>
>> [email-masked] <mailto:[email-masked]>

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 11 Apr 2011 18:22:57 -0500

On Apr 8, 2011, at 12:12 PM, Dean wrote:

> This is very much not the case.  There is a big difference between have a block of all zeroes and have no block at all.  Performance favours the former, thin-provisioning favours the latter.  VMWare give the users the option of creating thick and thin disk images for just this reason.
> 
> So, I think we want to avoid the case where someone has intentionally created a *thick* file, and then we end up deallocating blocks within it, which will slow performance when those blocks are written again and could cause problems since they are no longer guaranteed to have space for those deallocated blocks.

Wouldn't that be where space reservation would come into play?

If I want to reduce a thick file, I turn off space reservation and punch a hole.

If I want to keep a file thick, I keep space reservation and punch a hole.

That gives us both of the behaviors you want to see.

> Well, there was a clear request by Spencer and others at the end of the interim meeting for me to clean it up to only include what was discussed and add in several comments and issues.

Ahh, you had several weeks to do that. :->

We have an aggressive schedule and I'm trying to stay ahead of it. My plan of record is
to attack the controversial areas at first and then come back to work on the other areas.

> 
> Now that it has been pulled into the NFSv4.2 document, how do I make these changes (as described below, but there was also several more discussed at the meeting)?

I'm not suggesting you not do the work. I am suggesting that keeping the content in
two places is less than optimal. Also, as time goes on, the format used in the NFSv4.2
document will not look like that in your I-D.

I'm trying to settle with David Harrington and Spencer the final repository for the 4.2 I-D.
Right now it is on github.com and I've yet to have someone try to commit changes
back. Other projects do it there, so I'm sure we can figure it out.

At worst, you can mail me your changes to those files and I can pull them in.

---

From: Dean <[email-masked]>
Date: Tue, 12 Apr 2011 10:33:45 -0700

> An even more interesting observation is that you can easily have 
While interesting, is this realistic?  Isn't the whole point here to 
have a consistent pattern throughout the file to be able to 
differentiate corruption from holes?  Is there a usecase for supporting 
multiple patterns in a single file?

I'm also wondering how applications would set and write this pattern 
using the standard file system APIs available today?

Dean

---

From: Dean <[email-masked]>
Date: Tue, 12 Apr 2011 10:40:00 -0700

On 4/8/11 8:37 AM, Thomas Haynes wrote:
>
> On Apr 8, 2011, at 7:55 AM, <[email-masked] 
> <mailto:[email-masked]>> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>> Do we actually need the latter in NFS?  IMHO, it’s simpler to leave 
>> it out because that functionality results in an “interesting” corner 
>> case – if the client specifies the pattern that reading the hole 
>> returns, but the server doesn’t support returning that pattern for a 
>> hole, the result of the attempt to punch a hole is a bulk write 
>> instead of a hole punch (see the SCSI WRITE SAME command, which has 
>> hole punch functionality specified), which may not be what the client 
>> wanted ;-).
>
> David,
>

I don't mind getting creative, but to reiterate a previous post
1) The specialized corner case of using patterns should not add or slow 
down the common 'zero' case
2) Maybe we can get creative but keep it simple, like single pattern per 
file, set the pattern once per file, etc

Dean

---

From: Dean <[email-masked]>
Date: Tue, 12 Apr 2011 10:48:26 -0700

>>>> deallocates blocks (which would be very useful) and a hole punch 
>>>> that writes a specified pattern.  Maybe we can support both?
>>>
>>>
>> This is very much not the case.  There is a big difference between 
>> have a block of all zeroes and have no block at all.  Performance 
>> favours the former, thin-provisioning favours the latter.  VMWare 
>> give the users the option of creating thick and thin disk images for 
>> just this reason.
>>
>> it, which will slow performance when those blocks are written again 
>> and could cause problems since they are no longer guaranteed to have 
>> space for those deallocated blocks.
>
>
Ok, thanks.

I assume then that if space reservations are enabled, and the client 
punches a hole, the server has the following options:
a) Write a pattern to the byte range
b) De-allocate the blocks but do not free up the space, ensuring this 
file is always guaranteed its original space.

>> Well, there was a clear request by Spencer and others at the end of 
>> the interim meeting for me to clean it up to only include what was 
>> discussed and add in several comments and issues.
>
>
> Right now it is on github.com <http://github.com> and I've yet to have 
> someone try to commit changes

I've sent my changes/additions, please merge them when you have a moment.
Dean

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 12 Apr 2011 13:06:19 -0500

On Apr 12, 2011, at 12:40 PM, Dean wrote:

> 1) The specialized corner case of using patterns should not add or slow down the common 'zero' case

Agreed.

> 2) Maybe we can get creative but keep it simple, like single pattern per file, set the pattern once per file, etc
> 

I think that the majority of the uses will keep it simple. But with a low overhead
of getting the XDR right, we keep it open to allow the more complex cases.

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 12 Apr 2011 13:23:17 -0500

On Apr 12, 2011, at 12:33 PM, Dean wrote:

> While interesting, is this realistic?  Isn't the whole point here to have a consistent pattern throughout the file to be able to differentiate corruption from holes?  Is there a usecase for supporting multiple patterns in a single file?
> 

I can see the use case for multiple patterns:

0xdeadbeef means the data was initialized and never used.
0xfeed51a9 means the data was hole-punched.

I.e., think of the common ways that memory slab libraries
detect the various forms of corruption.

The flip side is do we want to mandate that the server
record the pattern and reject any WRITE_PLUS calls
which encode a different pattern?

I guess we could provide a SET_PATTERN operation
which set the only pattern allowed in a file. By default
it would be zeros. Then any INITIALIZE or HOLE_PUNCH
type of operation would not take a pattern, but when the
server responded to a READ, it would provide that set
pattern. Of course, then you have the issue if it is write
once or many.

Also, a server is not required to coalesce holes, is it?

So even with a set pattern, we could have:

                        +-------------+--------------------+
                        | Byte-Range  | Contents           |
                        +-------------+--------------------+
                        | 0-31999     | Data               |
                        | 32K-255999  | Hole of "deadbeef" |
                        | 256K-287999 | Data               |
                        | 288K-353999 | Hole of "deadbeef" |
                        | 354K-417999 | Hole of "deadbeef" |
                        +-------------+--------------------+

The READ_PLUS would have to handle that anyway, right?

> I'm also wondering how applications would set and write this pattern using the standard file system APIs available today?
> 

The client doing dedup of dirty pages?

I'm not sure, but as application writers see the functionality, they'll push
for new APIs to utilize it. You can even imagine a read() system call
returning the fact that it got a hole/pattern back.