Subject: [nfsv4] direct I/O examples in NFSv4.2 draft

From: Chuck Lever <[email-masked]>
Date: Thu, 21 Jun 2012 14:43:46 -0400

Hi-

After thinking about this a little more, it seems to me that direct I/O is a particularly bad use case for I/O hints.

Aside from ambiguity about whether an application's request for direct I/O applies only to client caching or to both the client and server, direct I/O really is not a hint at all.  Correct application operation REQUIRES that the client, at least, does not cache.  This is a directive, not a hint.

As Trond pointed out, the NFSv4.2 draft is supposed to be normative, thus on those grounds alone we should at least strike these examples from the IMPLEMENTATION discussion.

I'm in favor of removing all mention of direct I/O from chapter 3 and from 13.8.  Thoughts?

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: <[email-masked]>
Date: Thu, 21 Jun 2012 15:29:21 -0400

> Aside from ambiguity about whether an application's request for direct I/O applies 
> only to client caching or to both the client and server, 

I don't see that there is any ambiguity about the matter.

It is true that we are seeing uncertainty and what appears to me to 
be confusion.  Avoiding any such confusion is a good thing.

> direct I/O really is not a hint at all.  

It certainly never has been.

Existing direct IO implementations involve the client.

   Bypassing the cache on reads.

   Immediately writing through on writes.

   Doing synchronous writes so that any caching on the server is write-though.

If that has been unsatisfactory in any way, I haven't heard about it.

Is there some issue with this sort of implementation that we should be thinking about
in NFSV4.x for x > 2? 

> Correct application operation REQUIRES that the 
> client, at least, does not cache.  This is a directive,
> not a hint.

It always has been a directive to the client and if the client
needs semantic help from the server, any additional server feature
to help support it would also be a directive, and not a hint.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Chuck Lever
Sent: Thursday, June 21, 2012 2:44 PM
To: Tom Haynes; Trond Myklebust
Cc: nfsv4
Subject: [nfsv4] direct I/O examples in NFSv4.2 draft

Hi-

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Chuck Lever <[email-masked]>
Date: Thu, 21 Jun 2012 15:45:04 -0400

Hi-

The question is not whether direct I/O should be implemented as a hint, but whether IOADVISE is appropriate to use when an application requests direct I/O.

> be confusion.  Avoiding any such confusion is a good thing.

The NFSv4.2 draft suggests that IOADVISE4_NOREUSE is appropriate for a client to use when an application requests direct I/O.  The ambiguity arises because an application may request direct I/O because it doesn't intend to reuse the data, OR because it is providing its own locking and requires that the client not cache.    Applications currently have no mechanism by which to communicate this distinction to the client.

> Is there some issue with this sort of implementation that we should be thinking about

No dissatisfaction at all with direct I/O as it is implemented.

> to help support it would also be a directive, and not a hint.

IOADVISE is a hint only.  Thus it would not be appropriate to use it in the context of a direct I/O directive, and we should not suggest that it is by presenting it as a use case in section 3 or 13.8.  My thought is that the language in these sections conflates direct I/O with a caching hint that could be ignored.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 22 Jun 2012 16:17:11 +0000

On 6/21/12 2:45 PM, "Chuck Lever" <[email-masked]> wrote:

>Hi-
>
>The question is not whether direct I/O should be implemented as a hint,
>but whether IOADVISE is appropriate to use when an application requests
>direct I/O.
>

I've combined this with the other discussion yesterday, whether sections
3.2 and 3.3 were needed, and removed both those sections and all
references to direct I/O in the context of IO_ADVISE.

I'm going to go through Dean's earlier email exchange and probably push a
new version of the draft soon because of the churn here.

---

From: Dean <[email-masked]>
Date: Wed, 27 Jun 2012 09:59:25 -0700

> references to direct I/O in the context of IO_ADVISE.

I've been traveling, but I strongly disagree with this action.

>new version of the draft soon because of the churn here.

O_DIRECT was the original motivation behind IO_ADVISE, and so this text 
cannot be removed willy-nilly.  While discussion of IO_ADVISE and 
O_DIRECT in NFSv4.2 is simply in the context of possible use cases, and 
does not affect the protocol itself, I completely disagree that it is 
inappropriate.  It IS appropriate, and let me tell you why.

 >Chuck wrote:
 >IOADVISE is a hint only. Thus it would not be appropriate to use it in 
the context of a direct I/O directive, and we >should not suggest that 
it is by presenting it as a use case in section 3 or 13.8. My thought is 
that the language in >these sections conflates direct I/O with a caching 
hint that could be ignored.

As you've clearly stated, O_DIRECT is only applicable to the client, 
anything the server does is completely out of the definition of 
O_DIRECT.  Further, O_DIRECT is in fact a hint, and provides no 
guarantees whatsoever.  This is clear from the Linux OPEN man page:

"The  behaviour  of  O_DIRECT with NFS will differ from local file 
systems.  Older kernels, or kernels
  configured in certain ways, may not support this combination. The  
NFS  protocol  does  not  support
  passing  the  flag  to the server, so O_DIRECT I/O will only bypass 
the page cache on the client; the
  server may still cache the I/O.  The client asks the server to make 
the I/O synchronous  to  preserve
  the  synchronous  semantics of O_DIRECT.  Some servers will perform 
poorly under these circumstances,
  especially if the I/O size is small.  Some servers may also be 
configured to lie to clients about the
  I/O  having  reached  stable  storage;  this  will avoid the 
performance penalty at some risk to data
  integrity in the event of server power failure.  The Linux NFS client 
places  no  alignment  restric-
  tions on O_DIRECT I/O.

  In  summary,  O_DIRECT is a potentially powerful tool that should be 
used with caution.  It is recom-
  mended that applications treat use of O_DIRECT as a performance option 
which is disabled by  default."
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Based on this text, it is clear that the NFS client will bypass the 
client cache but what the server does is very implementation specific.  
Further, applications should not rely on O_DIRECT for correctness, and 
therefore it is in fact a HINT.

Finally, whether you agree with the Linux manual or not, the point is 
that providing a hint to the server based upon application behavior 
(either setting O_DIRECT or possibly like accessing a cache page a la 
IO_ADVISE4_INIT_PROXIMITY) both fall perfectly in line with IO_ADVISE.

Dean

---

From: Chuck Lever <[email-masked]>
Date: Wed, 27 Jun 2012 14:06:23 -0400

On Jun 27, 2012, at 12:59 PM, Dean wrote:

> Based on this text, it is clear that the NFS client will bypass the client cache but what the server does is very implementation specific.  Further, applications should not rely on O_DIRECT for correctness, and therefore it is in fact a HINT.

Here's yet another distinction between O_DIRECT and IO_ADVISE.  If the O_DIRECT flag is not supported, open(2) reports EINVAL.  Thus it is an error if the operating system cannot prevent data caching.  An application needs to know if it is not getting uncached data.  It will fail immediately (or fall back to some kind of secondary I/O plan) if the operating system does not support this feature.

Database applications now deployed in the field do rely on uncached I/O for correctness, and that makes it a de facto standard semantic that we can no longer will to be anything different.  My company sells such applications, which is why I'm standing up for this point.  I believe your employer does too.  We have gone to great lengths to make sure that when we specify O_DIRECT, the O/S is out of the way, guaranteed.

But none of that matters.  The important point is that the O_DIRECT use cases in the IO_ADVISE sections are not relevant to implementing posix_fadvise(3).  Removing these examples does not endanger the IO_ADVISE feature in any way.  Additionally, the place for discussing such use cases was in the requirements draft, not in the protocol specification.  What possible harm could come from removing them here?

The harm in leaving them in, however, is clear: it conflates a specific local caching directive with hints sent to the server which the server is free to ignore.  O_DIRECT has nothing to do with the server's I/O access pattern.  I continue to believe these use cases are confusing and reckless, and that we will regret leaving them in.

Finally, I'm not saying that we should actively prevent a client from sending some kind of hint if O_DIRECT is requested.  I'm saying that it's not appropriate to recommend in an IETF protocol spec that sending such a hint is a good idea if the application requested O_DIRECT.  O_DIRECT is orthogonal to server caching hints, making this suggestion one of implementation quality, and nothing more.

---

From: Margaret Susairaj <[email-masked]>
Date: Wed, 27 Jun 2012 14:22:46 -0500

I agree with Chuck on the importance of O_DIRECT not being a hint  but 
making sure it disallows any kind of  OS caching on the client node. 
Many applications depend on this property for correctness.

It is also important to have IO_ADVISE pass a hint  for a byte range 
instead of the whole file. Certain caching properties are specific to 
byte ranges inside a file and applications should be able to pas this 
hint along in the compound i/o operation so that server can do the 
appropriate optimization.

Thanks,
Margaret

---

From: Dean <[email-masked]>
Date: Wed, 27 Jun 2012 15:24:56 -0700

> Many applications depend on this property for correctness.

Don't confuse standards and implementations.  There is no standard for 
O_DIRECT, just manual pages and a variety of non-standard 
implementations.  It is completely legal for a file system to return OK 
on open when O_DIRECT is specified and then not actually bypass the 
cache.  This could happen for a variety of reasons, most notably 
including that the request was not-aligned with file system sector of 
block sizes.

In addition, no one is talking about changing how the NFS client 
supports O_DIRECT.  The only difference is whether more information is 
sent to the server.  Today, since O_DIRECT is not transferred to the NFS 
server, the file server always performs caching, which can cause lots of 
problems in cases where one-time data is cached at the expense of useful 
data that may actually be read multiple times (from other application).

> appropriate optimization.

Let's move this conversation to the other thread...
Dean

>>> client cache but what the server does is very implementation 
>>> specific.  Further, applications should not rely on O_DIRECT for 
>>> correctness, and therefore it is in fact a HINT.
>> Here's yet another distinction between O_DIRECT and IO_ADVISE. If the 
>> O_DIRECT flag is not supported, open(2) reports EINVAL. Thus it is an 
>> error if the operating system cannot prevent data caching.  An 
>> application needs to know if it is not getting uncached data.  It 
>> will fail immediately (or fall back to some kind of secondary I/O 
>> plan) if the operating system does not support this feature.
>>
>> Database applications now deployed in the field do rely on uncached 
>> I/O for correctness, and that makes it a de facto standard semantic 
>> that we can no longer will to be anything different.  My company 
>> sells such applications, which is why I'm standing up for this 
>> point.  I believe your employer does too. We have gone to great 
>> lengths to make sure that when we specify O_DIRECT, the O/S is out of 
>> the way, guaranteed.
>>
>>
>> But none of that matters.  The important point is that the O_DIRECT 
>> use cases in the IO_ADVISE sections are not relevant to implementing 
>> posix_fadvise(3).  Removing these examples does not endanger the 
>> IO_ADVISE feature in any way.  Additionally, the place for discussing 
>> such use cases was in the requirements draft, not in the protocol 
>> specification.  What possible harm could come from removing them here?
>>
>> The harm in leaving them in, however, is clear: it conflates a 
>> specific local caching directive with hints sent to the server which 
>> the server is free to ignore.  O_DIRECT has nothing to do with the 
>> server's I/O access pattern.  I continue to believe these use cases 
>> are confusing and reckless, and that we will regret leaving them in.
>>
>> Finally, I'm not saying that we should actively prevent a client from 
>> sending some kind of hint if O_DIRECT is requested.  I'm saying that 
>> it's not appropriate to recommend in an IETF protocol spec that 
>> sending such a hint is a good idea if the application requested 
>> O_DIRECT.  O_DIRECT is orthogonal to server caching hints, making 
>> this suggestion one of implementation quality, and nothing more.
>>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 27 Jun 2012 20:53:15 -0400

On Jun 27, 2012, at 6:24 PM, Dean wrote:

>> I agree with Chuck on the importance of O_DIRECT not being a hint but making sure it disallows any kind of  OS caching on the client node. Many applications depend on this property for correctness.
> 
> Don't confuse standards and implementations.  There is no standard for O_DIRECT, just manual pages and a variety of non-standard implementations.  It is completely legal for a file system to return OK on open when O_DIRECT is specified and then not actually bypass the cache.

If we're talking specifically about the Linux NFS client, NFS will always return EINVAL for open(O_DIRECT) if O_DIRECT is not supported.  For NFS, it is not a hint, it is a directive.  This is not equivalent to local file system behavior because NFS clients suffer from cache aliasing.

> This could happen for a variety of reasons, most notably including that the request was not-aligned with file system sector of block sizes.

NFS O_DIRECT is not the same as O_DIRECT on a local file system.  For NFS, it means specifically that the result of an I/O is never cached locally.  Local file systems can cache in this case because their cache is private and singular, unlike NFS, so it is not hazardous for a local file system to cache a direct I/O.  In the same way that it is not hazardous for the server to cache in this case.

Enabling uncached NFS I/O is exactly why we have NFS O_DIRECT on Linux and forcedirectio on Solaris (and so on).  There are significant applications that depend on the ability to disable a client's caching but still have good server performance.  Disabling the local cache is critical to the correctness of these applications.

I can't point you to a standard, but I can tell you that we purposely designed it to work this way a decade ago, and these are the reasons.  Oracle is interested in ensuring this remains the case.

> In addition, no one is talking about changing how the NFS client supports O_DIRECT.
> The only difference is whether more information is sent to the server.  Today, since O_DIRECT is not transferred to the NFS server, the file server always performs caching, which can cause lots of problems in cases where one-time data is cached at the expense of useful data that may actually be read multiple times (from other application).

And the converse is true as well: there are significant currently deployed applications that would perform WORSE if a client always sent a NOREUSE hint on O_DIRECT files.  NOREUSE should be up to applications, never sent automatically.  If an application wants to add the additional hint of NOREUSE after it opens a file with O_DIRECT, it should be free to do just that.  But a client should Never assume that this is always desirable, any more than always sending an ADV_RANDOM hint for O_DIRECT files would be.

>> It is also important to have IO_ADVISE pass a hint  for a byte range instead of the whole file. Certain caching properties are specific to byte ranges inside a file and applications should be able to pas this hint along in the compound i/o operation so that server can do the appropriate optimization.
>

---

From: "Welch, Brent" <[email-masked]>
Date: Thu, 28 Jun 2012 03:58:59 +0000

I think I can simplify this thread by stating that O_DIRECT is a client-side directive that bypasses the cache, and is orthogonal to any over-the-wire hints about what the server should do with its cache replacement policy.  Unwinding the thread, we should delete the paragraph in the spec that gives as an example a linkage between these two independent mechanisms.

--
Brent

---

From: <[email-masked]>
Date: Thu, 28 Jun 2012 08:24:10 -0400

I strongly agree with Brent.

Sent from my iPhone

---

From: Dean <[email-masked]>
Date: Thu, 28 Jun 2012 10:07:29 -0700

Which is fine, but let's not forget the motivation of communicating these hints to the server.  This universal call to just remove this section is not sufficient.  As I suggested in the other thread, let's replace this text with a discussion of how the NFS client can communicate such hints for disabling server caching to the NFS server.

Ok, let's kill this thread a move all discussion to the other thread.
Dean