Subject: [nfsv4] Meeting Agenda for 06/07/12

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 6 Jun 2012 12:02:41 -0700

Time: Thursdays 12:30-1:30 PM Central/1:30-2:30 PM Eastern/10:30-11:30 AM Pacific
Dial-in: 1-866-844-9418 (Toll Free)
       1-517-345-9208 (Caller paid)

     Id: 4082151519

Proposed Agenda
-------------------------

+ IETF Note Well Agreement

This is a reminder that our discussions are governed by the
IETF Note Well Agreement. See:

   http://www.ietf.org/NOTEWELL.html

We will start each week's meeting with this announcement.

+ We will be working from draft version 11

http://www.ietf.org/id/draft-ietf-nfsv4-minorversion2-11.txt
http://www.ietf.org/id/draft-ietf-nfsv4-minorversion2-dot-x-11.txt

+ Get a lined copy

1) Use cat:

$cat -n draft-ietf-nfsv4-minorversion2-11.txt > listing-11.txt

2) Peter suggested that you can use IE to see the numbers.

Load the draft page in IE.
Right click your mouse, that will get you a menu.
Select 'View Source'

That will show you a numbered copy of the draft.

+ We will start on line 3274

Target Date  Draft  Line  Text
05/03/12      09    4177  Section 13.9: LAYOUTRETURN
05/10/12      10    2417  Chapter 8: Sharing change attribute implementation details with NFSv4 clients
05/10/12      10    2602  Section 11.2.1: Attribute 79: change_attr_type
05/10/12      10    2551  Section 11.1: New RECOMMENDED Attributes - List and Definition References
05/10/12      10    3677  Section 13.6: EXCHANGE_ID
05/17/12      10    2007  Chapter 7: Labeled NFS
05/17/12      10    2537  Section 10.1.3: Labeled NFS Errors
05/17/12      10    2669  Section 11.2.2: Attribute 80: sec_label
05/17/12      10    4807  Section 14.1: CB_ATTR_CHANGED
05/24/12      11     379  Chapter 2: Server-side Copy
05/31/12      11    2959  Section 13.1: COPY
06/07/12      11    3368  Section 13.2: COPY_ABORT
06/07/12      11    3426  Section 13.3: COPY_NOTIFY
06/07/12      11    3550  Section 13.4: COPY_REVOKE
06/07/12      11    3608  Section 13.5: COPY_STATUS
06/07/12      11    4853  Section 14.2: CB_COPY
06/07/12      11    2502  Section 10.1.2: Server-side Copy Errors
                          Chapter 3: Support for Application IO Hints
                          Section 13.8: IO_ADVISE
                          Chapter 4: Sparse Files
                          Chapter 5: Space Reservation
                          Section 11.2.3: Attribute 77: space_reserved
                          Section 11.2.4: Attribute 78: space_freed
                          Chapter 6: Application Data Block Support
                          Section 13.7: INITIALIZE
                          Section 13.10: READ_PLUS
                          Section 13.11: SEEK
                          Chapter 12: Operations
                          Chapter 9: Security Considerations
                          Chapter 15: IANA Considerations
                          Chapter 16: References
                          Chapter 1: Introduction

-- 
[email-masked], ex-cfb

---

From: Dean <[email-masked]>
Date: Wed, 06 Jun 2012 13:23:26 -0700

Couple comments on IOADVISE in Draft 11

1)
   4148       If NFL42_UFLG_IO_ADVISE_THRU_MDS is set, then if the client
   4149       implements IO_ADVISE, then if it wants the DS to honor 
IO_ADVISE, the
   4150       client MUST send the operation to the MDS, and the server will
   4151       communicate the advice back each DS.
This is a little convoluted.  How about something like the following,

If NFL42_UFLG_IO_ADVISE_THRU_MDS is set, the client MUST send the 
IO_ADVISE operation to the MDS in order for it to be honored by the DS.  
Once the MDS receives the IO_ADVISE operation, it will communicate the 
advice to each DS.

2)
   4154       If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then this 
indicates to
   4155       client that if wants to inform the server via IO_ADVISE of the
   4156       client's intended use of the file, then the client SHOULD 
send an
   4157       IO_ADVISE to each DS....
This has a small mistake in it (missing the word 'it').  But maybe it 
can be re-written thus,

If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then the client SHOULD send 
an IO_ADVISE operation to the appropriate DS for the specified byte range.

3)
In section 13.8.6, I'm curious about the use of IO_ADVISE4_NOREUSE.  
Currently it is listed as an operation that applies to a specified byte 
range and is treated as a one time instruction.  I don't think this is 
correct.  For example, if we wish to use IO_ADVISE4_NOREUSE in the same 
compound as OPEN (if the file is opened with O_DIRECT), then it cannot 
be a 'one time' operation since the server will not remember that all 
forthcoming writes should not be cached.

A simple solution may be to move IO_ADVISE4_NOREUSE to the list of 
operations that apply to the entire file.  This way, the server can set 
a single flag when it receives IO_ADVISE4_NOREUSE that all forthcoming 
writes should not be cached.

Dean

---

From: Sebastian Moors <[email-masked]>
Date: Wed, 13 Jun 2012 16:22:50 +0200

Hi Dean,

Dean wrote:
> Currently it is listed as an operation that applies to a specified 
> byte range and is treated as a one time instruction.  I don't think 
> this is correct.  For example, if we wish to use IO_ADVISE4_NOREUSE in 
> the same compound as OPEN (if the file is opened with O_DIRECT), then 
> it cannot be a 'one time' operation since the server will not remember 
> that all forthcoming writes should not be cached.
Just a remark: I don't find it suiting to implement O_DIRECT by sending 
IO_ADVISE4_NOREUSE to the server. There is a small difference that 
occurs when doing a read operation: A server that sees an operation with 
O_DIRECT should make shure that the read operation bypasses the cache 
completely.  But the definition of POSIX_FADV_NOREUSE is:  "Specifies 
that the application expects to access the specified data once and then 
not reuse it thereafter.". Therefore, IO_ADVISE4_NOREUSE allows that 
read requests are getting satisfied from the cache (which stands in 
contrast to O_DIRECT). Imho, that's a good reason to handle both advices 
differently.
Why not introducing sth. like IO_ADVISE4_DIRECTIO to IO_ADVISE_type4 to 
avoid these conflicts?
> operations that apply to the entire file.  This way, the server can 
> set a single flag when it receives IO_ADVISE4_NOREUSE that all 
This would take the possibility to issue the advice on byte-ranges, 
which is especially interesting for HPC applications which work on large 
files.
If an application programmer uses this hint on the last half of a file, 
it would disable (write) caching for the complete file, which might be 
not his intention.

Best regards,
Sebastian

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 13 Jun 2012 15:37:02 +0000

On Wed, 2012-06-13 at 16:22 +0200, Sebastian Moors wrote:
> completely.

That wouldn't fit the concept of a "hint". That would be forcing the
server to manage the data caching in a certain way.

>   But the definition of POSIX_FADV_NOREUSE is:  "Specifies 

Why do you assume that we want to force the server to use O_DIRECT?

The hint exists in order to allow the NFS client to tell the server how
the _client_ intends to access that data. It is then up to the server to
figure out how to best manage its caching of that data.

> it would disable (write) caching for the complete file, which might be 

Right. Furthermore, it is trivial for the NFS client to add an IOADVISE
operation in the same COMPOUND as the WRITE (unless the server sets
NFL42_UFLG_IO_ADVISE_THRU_MDS - but why would it want to do that?).

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Dean <[email-masked]>
Date: Wed, 13 Jun 2012 09:49:23 -0700

Hi Sebastian,

> server to manage the data caching in a certain way.

Exactly.  Actually implementing O_DIRECT is not the goal, but rather to 
give the server a hint about the data access pattern.  The client cannot 
'rely' on the results of IO_ADVISE...

>> it would disable (write) caching for the complete file, which might be

The problem is that its unusable today for a byte-range since it is a 
one time operation.  For example, if the client wants to tell the server 
it doesn't need the byte-range any longer, it should use DONTNEED.  If 
the client wants to tell the server it won't (in the future) need the 
byte-range, using NOREUSE will have no affect because the server will 
have immediately forgotten the information.
> NFL42_UFLG_IO_ADVISE_THRU_MDS - but why would it want to do that?).
Same issue with the compound.  A one-time operation does not apply from 
the first operation in a compound to the last operation in the 
compound.  It applies at the point in time when the server executes it.  
So we are back to either using DONTNEED or having NOREUSE take no affect.

So I don't see any other choice than removing the one-time limitation 
and applying it to the entire file.

Dean

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 13 Jun 2012 13:44:49 -0400

On Wed, Jun 06, 2012 at 01:23:26PM -0700, Dean wrote:
> IO_ADVISE operation to the MDS in order for it to be honored by the
> DS.  Once the MDS receives the IO_ADVISE operation, it will
> communicate the advice to each DS.
> 
> 2)
>   4156       client's intended use of the file, then the client
> SHOULD send an
> This has a small mistake in it (missing the word 'it').  But maybe
> it can be re-written thus,
> 
> If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then the client SHOULD
> send an IO_ADVISE operation to the appropriate DS for the specified
> byte range.
> 
> 3)
> this is correct.  For example, if we wish to use IO_ADVISE4_NOREUSE
> in the same compound as OPEN (if the file is opened with O_DIRECT),
> then it cannot be a 'one time' operation since the server will not
> remember that all forthcoming writes should not be cached.

OK.  But why not send it with each write instead in that case?

--b.

---

From: Sebastian Moors <[email-masked]>
Date: Fri, 15 Jun 2012 00:50:10 +0200

Dean wrote:
> Exactly.  Actually implementing O_DIRECT is not the goal, but rather 
> to give the server a hint about the data access pattern.  The client 
> cannot 'rely' on the results of IO_ADVISE...
>
Thanks for the comments! Sorry, my comment was not precise... What i 
wanted to suggest was that a server might choose different optimizations 
for direct i/o and NOREUSE. I understand that an implementation of 
direct i/o via hints makes no sense.

> one time operation.  For example, if the client wants to tell the 
> server it doesn't need the byte-range any longer, it should use 
> DONTNEED.  If the client wants to tell the server it won't (in the 
> future) need the byte-range, using NOREUSE will have no affect because 
> the server will have immediately forgotten the information. 
I'm a little bit curious about the categorisation in 'one-time 
operations' and 'operations that affect the whole file'. Was this 
categorization done based on some standard or was it based on another 
criterion? At a first glance, it seems to mimic the semantics of the 
implementation of posix_fadvise in the linux kernel. I understand that 
the categorization in 13.8.6 eliminates the need to store information of 
dynamic size at the server (like a list of IO_ADVISE4_SEQUENTIAL advices 
for different byte ranges).

Best regards,
Sebastian

---

From: Dean <[email-masked]>
Date: Fri, 15 Jun 2012 10:25:11 -0700

Hi Sebastian,

> wanted to suggest was that a server might choose different 
> optimizations for direct i/o and NOREUSE. I understand that an 
> implementation of direct i/o via hints makes no sense.

I'd be fine with that, although as a hint, I'm not exactly sure what 
advantage it has.  Isn't it better to simply tell the server of your 
access pattern and let it decide on the best course of action?

>> future) need the byte-range, using NOREUSE will have no affect 
>> because the server will have immediately forgotten the information. 
> the categorization in 13.8.6 eliminates the need to store information 
> of dynamic size at the server (like a list of IO_ADVISE4_SEQUENTIAL 
> advices for different byte ranges).
This categorization of 'one-time operations' and 'operations that affect 
the whole file'  is based on the extra state that the NFS server would 
have to maintain in either case.

As I mentioned before, having the NFS server store state that contains 
byte-ranges and their associated hints would be too onerous.  So we 
decided that the amount of state that the server would be required to 
store is limited to an entire file (single byte range) and its 
associated hints.

With this amount of state, the server has limited actions that it can 
take w.r.t. incoming hints.  A hint must either apply to the entire 
file, which the server can then remember and apply this hint to future 
incoming requests, or, if the hint applies to only a byte-range, and 
since the server can only remember whole-file, then all action taken by 
the server must be completely confined to the execution of the IO_ADVISE 
operation.

Dean

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 15 Jun 2012 20:59:28 +0000

> store is limited to an entire file (single byte range) and its 

Is it really all that onerous to have the server remember it for the
duration of the enclosing COMPOUND?

Cheers
  Trond

NetApp

---

From: Dean <[email-masked]>
Date: Fri, 15 Jun 2012 15:01:30 -0700

> duration of the enclosing COMPOUND?
I'm ok with it if it will work.  Are you suggesting something like a 
PUT_IO_ADVISE?
Dean

---

From: Dean <[email-masked]>
Date: Wed, 27 Jun 2012 10:37:12 -0700

> duration of the enclosing COMPOUND?

I've been thinking about IO_ADVISE4_NOREUSE a little more.

One way to resolve the O_DIRECT discussion (and completely remove it 
from the draft), may be to go through an example of how an application 
would instead use posix_fadvise POSIX_FADV_NOREUSE to issue 
IO_ADVISE4_NOREUSE to the server.  This would allow applications to 
combine the use of O_DIRECT and posix_fadvise POSIX_FADV_NOREUSE to a) 
eliminate caching on the client and b) inform the server that this data 
is only being accessed once (and thus allowing the server to optimize 
this, which is the whole point of all of this).

When an application issues posix_fadvise POSIX_FADV_NOREUSE for a 
byte-range, if the NFS client is going to include it in the enclosing 
COMPOUND for every READ/READ_PLUS request, then the application must be 
able to specify the hint for every read issued to the server (including 
reads driven by readahead).  I don't see this as possible, since the 
application knows only the ranges for which it accesses, not the ranges 
read by the NFS client.

This could be resolved if the client remembers multiple ranges (one for 
each posix_fadvise call on the fd), but Mike E. and I have agreed that 
this would be too cumbersome for either the client or server.

The only way to really guarantee that every read issued to the server 
includes the hint is to only support whole file.  So I'm not sure if 
there is really any other choice but to support IO_ADVISE4_NOREUSE for 
the whole file to ensure that all reads for which it applies are 
covered.  From my point of view, this would handle O_DIRECT, and we 
could remove the O_DIRECT discussion from the spec.

Dean

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 27 Jun 2012 19:10:33 +0000

>this, which is the whole point of all of this).

Well, the example would still keep it in, no?

I don't see having requirements driven by O_DIRECT as making sense for
this document.

I do see a subsection of 13.8.4 that describes an example as above as
enhancing the document. What I'd also like to see is a discussion of both
what happens if an application uses O_DIRECT and does not use
posix_fadvise POSIX_FADV_NOREUSE and what happens if the server decides
not to use the hint.

I.e., we get the current behavior of NFSv4.1 - the server may or may not
cache the I/O.

>could remove the O_DIRECT discussion from the spec.

I see two issues here:

1) Is IO_ADVISE_NOREUSE really per file?

2) Are one time instructions per compound?

---

From: Chuck Lever <[email-masked]>
Date: Wed, 27 Jun 2012 15:41:47 -0400

>> this, which is the whole point of all of this).

I agree, an application should be entirely in control of whether a NOREUSE hint is issued when O_DIRECT is specified.  Having an NFS client automatically send a hint when O_DIRECT is requested could be problematic in some cases.

For example, four NFS clients running Oracle RAC will specify O_DIRECT to disable their local caches, but will benefit from having a server-side data cache by keeping active database pages in a memory working set shared by all the clients.

Thus O_DIRECT should stand by itself, and applications are free to augment it any way they like.  But the client shouldn't automatically do this for them.

> posix_fadvise POSIX_FADV_NOREUSE and what happens if the server decides

I prefer Dean's example of how both the client and server treat NOREUSE.  If one or the other decides to ignore that hint, you have basically the same discussion, but without bringing up O_DIRECT.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Dean <[email-masked]>
Date: Wed, 27 Jun 2012 15:42:54 -0700

> I prefer Dean's example of how both the client and server treat NOREUSE.  If one or the other decides to ignore that hint, you have basically the same discussion, but without bringing up O_DIRECT.
In the end, I want to make sure that O_DIRECT can be used in conjunction 
with posix_fadvise/POSIX_FADV_NOREUSE so that applications can disable 
caching in both the client and server.  So while we can remove the 
discussion of O_DIRECT, I want to ensure the spec has a discussion of 
how applications can disable server caching for forth coming read and 
write requests for an entire file via a single call to 
posix_fadvise/POSIX_FADV_NOREUSE.  I can provide text for this...

>1) Is IO_ADVISE_NOREUSE really per file?

Yes.  As I've tried to explain, I don't see any other way to do it without burdening the client or server with having to remember an unlimited number of byte-ranges for a file and their associated hints.  I would suggest we leave byte-range support to a future version, we could always invent a new hint name if required.

>2) Are one time instructions per compound?

This seems to be bantered about, but I don't see a mechanism in any implementation or in the NFSv4.1 spec to remember state per compound.  In addition, I don't see the usefulness since the client can't remember which byte-ranges (and hence which compounds) it would apply to anyways.

Dean

---

From: "Haynes, Tom" <[email-masked]>
Date: Thu, 28 Jun 2012 00:19:11 +0000

On 6/27/12 5:42 PM, "Dean" <[email-masked]> wrote:

>
>> I agree, an application should be entirely in control of whether a
>>NOREUSE hint is issued when O_DIRECT is specified.  Having an NFS client
>>automatically send a hint when O_DIRECT is requested could be
>>problematic in some cases.
>>
>> For example, four NFS clients running Oracle RAC will specify O_DIRECT
>>to disable their local caches, but will benefit from having a
>>server-side data cache by keeping active database pages in a memory
>>working set shared by all the clients.
>>
>> Thus O_DIRECT should stand by itself, and applications are free to
>>augment it any way they like.  But the client shouldn't automatically do
>>this for them.
>>
>> I prefer Dean's example of how both the client and server treat
>>NOREUSE.  If one or the other decides to ignore that hint, you have
>>basically the same discussion, but without bringing up O_DIRECT.
>>

Thanks

>Yes.  As I've tried to explain, I don't see any other way to do it
>without burdening the client or server with having to remember an
>unlimited number of byte-ranges for a file and their associated hints.  I
>would suggest we leave byte-range support to a future version, we could
>always invent a new hint name if required.

>This seems to be bantered about, but I don't see a mechanism in any
>implementation or in the NFSv4.1 spec to remember state per compound.  In
>addition, I don't see the usefulness since the client can't remember
>which byte-ranges (and hence which compounds) it would apply to anyways.

PUTFH/SAVEFH?

The text states:

    However, there are some hints that very ephemeral, and are essentially
amount
    to one time instructions to the NFS server, which will be forgotten
    momentarily after IO_ADVISE is executed.

So does this mean that IO_ADVISE applies to the next statement in the
compound or for the life of the compound?

Remembering per compound is not that onerous.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 28 Jun 2012 00:40:53 +0000

> Remembering per compound is not that onerous.

Agreed, and it adds a lot of value, since it basically allows the client
to tailor the byte-range hints to the particular READ or WRITE request
that is being sent. It basically allows sane semantics for
IO_ADVISE4_NOREUSE.

NetApp