Subject: RE: [nfsv4] re: byte range delegations

From: "Yoder, Alan" <[email-masked]>
Date: Thu, 14 Jul 2005 14:04:38 -0700

The most obvious application I can think of
for byte-range oplocks is distributed database 
applications in a grid environment.  I have put 
in some inquiries at Oracle to get opinions from
there.  Anyone know folks at SAP and/or IBM to 
get feedback/ideas from there as well?

Alan

===============================================================
Alan G. Yoder                                    [email-masked]
Technical Staff                           
Network Appliance, Inc.                            408-822-6919
=============================================================== 

> -----Original Message-----
> From: David Robinson [mailto:[email-masked]] 
> Sent: Thursday, July 14, 2005 1:22 PM
> To: Trond Myklebust
> Cc: [email-masked]
> Subject: Re: [nfsv4] re: byte range delegations
> 
> Trond Myklebust wrote:
> 
> > In the draft we list two commonly used operations that NFS currently
> > cannot support properly and that are directly linked to the 
> coherency
> > issue:
> > 
> >  - shared mmap()
> >  - append writes
> > 
> > There are quite a few applications out there that rely on 
> those working
> > correctly in a multi-threaded environment. If we want to 
> help port those
> > applications to a clustered environment without forcing people to
> > redesign their entire program around the I/O quirks of the 
> filesystem we
> > happen to have on offer, then we need to be able to present 
> them with
> > something that goes beyond close-to-open cache consistency.
> 
> While it is true that multi-threaded applications will rely on
> byte range consistency, I have yet to find any significant cases
> where there are multi-client applications that need this
> level of consistency. Generally speaking moving a single process
> application to a multi-process or multi-client application will
> need to have some sort of inter-node synchronization beyond the
> filesystem. I am very interested in the existence of any interesting
> number of such applications.
> 
> > We also make the point that byte range locks cause you to 
> take a huge
> > performance hit. They basically only work in an environment 
> where you
> > are using uncached I/O. If you are the sole user of the 
> file, then file
> > delegations help. If you are not, then irrespective of whether the
> > processes are behaving in a manner that causes caching 
> conflicts, you
> > will still be reduced to doing synchronous I/O.
> 
> There is no requirement that files that are locked must do uncached
> I/O. Most implementations are written that way because that is
> the easiest way to maintain correctness. For many years Solaris
> does cached I/O for files that contain byte range locks that
> encompass the entire file. It would not be difficult to increase
> the granularity to caching data for any lock ranges that were
> multiples of a page size. Just never had the demand to justify
> the work. I personally have designed sub-page sized cached I/O
> using some of the features of Solaris, but it is more complex
> and the demand isn't there and the performance gains are
> questionable.
> 
> While byte range delegations are interesting from a theoretical
> perspective, I tend to back Mike and strongly question if there is
> demand to justify the effort.
> 
> 	-David
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 14 Jul 2005 18:43:23 -0400

That was always the problem with an append-write.

Sessions provides a general replay protection facility,
which would avoid the issue, at least if you were using
sessions.  Handling it across a server-reboot is difficult
to implement but the protocol is there.

-----Original Message-----
From: Gordon Waidhofer [mailto:[email-masked]]
Sent: Thursday, July 14, 2005 6:06 PM
To: Noveck, Dave; Trond Myklebust; [email-masked]
Cc: [email-masked]
Subject: RE: [nfsv4] re: byte range delegations

I talked to somebody about an "append" flag
to write, or another write op, years ago.
I was told that the idempotency issues made
it pretty much impossible. Suppose the reply
gets lots and the server re-issues the append.
Does anything in NFSv4 overcome this problem?

That said, I still think it would be darn handy,
warts and all.

Regards,
	-gww

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Noveck, Dave
> Sent: Thursday, July 14, 2005 1:52 PM
> To: Trond Myklebust; [email-masked]
> Cc: [email-masked]
> Subject: RE: [nfsv4] re: byte range delegations
> 
> 
> >  - append writes
> 
> Although you can do append writes with this facility, it doesn't
> seem like the best way to do that, unless each append-writer is
> going to append more than can be done in a single write or 
> do many individual writes before it is likely that a different 
> client will do so.
> 
> In most cases, the best way to do an append write is simply to
> create an append option for WRITE and let the server put what 
> you give him at the end.  It would helpful to have the byte-range
> delegations to round out the support for enormous append writes but 
> most people would be much better off with a simple append-
> this-to-the-end-of-the-file-please facility.  A big issue with 
> this used to be sequencing/duplicates but sessions solves that.
> 
> From: Trond Myklebust [mailto:[email-masked]] 
> Sent: Thursday, July 14, 2005 1:51 PM
> To: [email-masked]
> Cc: [email-masked]
> to den 14.07.2005 Klokka 11:28 (-0400) skreiv
> [email-masked]:
> 
> > Are people clammering for better cache coherency and arguing that byte
> > range locks aren't sufficient for the cases they care about?
> 
> That's a very rhetorical question.
> 
>         The people that are happy to use NFSv2/v3 today? Probably not so
>         much.
> 
>         The people that chose to go with a clustered filesystem instead?
>         Yes, they probably feel quite strongly about the issue.
> 
> 8-)
> ---
> 
> More seriously:
> 
> cannot support properly and that are directly linked to the coherency
> issue:
> 
>  - shared mmap()
>  - append writes
> 
> There are quite a few applications out there that rely on those working
> correctly in a multi-threaded environment. If we want to help port those
> redesign their entire program around the I/O quirks of the filesystem we
> happen to have on offer, then we need to be able to present them with
> We also make the point that byte range locks cause you to take a huge
> performance hit. They basically only work in an environment where you
> are using uncached I/O. If you are the sole user of the file, then file
> processes are behaving in a manner that causes caching conflicts, you
> Finally, I assume that pNFS with non-files based back ends (e.g. objects
> and blocks) may want to be able to do atomic read-modify-write cycles on
> their data in order to support direct writes and such.
> 
> Cheers,
>   Trond
> 
>

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 11:45:12 -0700 (PDT)

> From: Trond Myklebust [mailto:[email-masked]] 
[...]
> something that goes beyond close-to-open cache consistency.

It is not clear if the draft does or doesn't guarantee that mmap and append writes 
always will work. It says:

  "A server MUST refuse to grant a read delegation on a range that would
   overlap with a write delegation held by another client. "

So if there is a conflicting delegation, we are back to NFSv3 (or NFSv4.0)
semantics, right?

Recall that when Dave Noveck introduced delegations he emphasized that
delegations were for caching, not coherency. Making NFSv4 a coherent
(read: clustered) filesystem might be beyond the scope of this
working group.

> are using uncached I/O. If you are the sole user of the file, 

For the case of making I/O combined with byte range locking more efficient,
I agree that byte range delegations are a win. The question is how many
applications need that.

You may want to have the byte range delegations have the option of granting
the client more than he asked for, and then allow the server to
recall ranges within the granted range.

> then file

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 14 Jul 2005 20:36:05 -0400

Agree.  Append-write only works for tarnsfers less than
the maximum transfer size.  Beyond that you have two 
choices that I can think of:

1) decide that solving 99% of the problem is better than 
   solving 0%.

2) byte-range delegations or something similar.

I have my doubts about byte-range delegations, but I'm not
dead set against them.  But I do feel that if we have them
they are not a good fit for the typical append-write workload.

I can see us deciding to do neither.

I can see us deciding to do both (although a better case needs
to be made).

I can see us deciding to do append-writes (although we need to
understand the limitations of doing that).

What I cannot see is doing byte-range delegations and not doing
append-writes.  For the typical use of append-write the performance
will be very poor.

-----Original Message-----
From: David Robinson [mailto:[email-masked]]
Sent: Thursday, July 14, 2005 7:09 PM
To: [email-masked]
Subject: Re: [nfsv4] re: byte range delegations

Noveck, Dave wrote:
> I must be missing something.  It seems to me that if
> you have an append-write op, it can decide where to 
> place the data and tell you what it decided.

As someone else mentioned, you need reliable replay
protection, which sessions will help out.  But if the
application performing the write() uses a length that
exceeds the server's transfer size you can't meet
the append write semantics of Posix.

	-David

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 18:10:21 -0700 (PDT)

> and the demand isn't there and the performance gains are

But upon release of a byte range lock that is on a subset of 
entire file, whether the range is page aligned or not,
you must still synchronously flush any modified data (and
invalidate any cache data) in the range before releasing the 
lock (and invalidate Similarlythe advisory lock is released.

Whereas, byte range delegations, even those used just for caching and not
coherency/consistency, allow the client to leisurely flush modified data,
and never invalidate cached data, until and unless there is a recall.

It might have been better to have startled the WG with byte
range caching before shocking it with byte range consistency.
Baby steps and all that.

---

From: "Yoder, Alan" <[email-masked]>
Date: Fri, 15 Jul 2005 12:22:21 -0700

Reply from an ASM architect at Oracle:

>From an Oracle database perspective, I do not see the value of
byte-range oplocks. Database files are owned exclusively by Oracle and
locking within a node, or even across nodes, in a cluster is managed by
Oracle using its locking protocol. Separate databases would never share
common data files unless the databases where clustered in a RAC
configuration and under control of Oracle's locking protocol. With that
said, I suppose non-database files used by Oracle applications could
conceivably benefit from byte-range oplocks, but even that feels like
even a stretch. 

Jim Williams
Oracle Corporation

Alan

> From: Yoder, Alan 
> Sent: Thursday, July 14, 2005 2:05 PM
> To: David Robinson; Trond Myklebust
> Cc: [email-masked]