Subject: RE: [nfsv4] re: byte range delegations

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 13:35:12 -0700 (PDT)

> From: Trond Myklebust [mailto:[email-masked]] 
[...]
> In the draft we list two commonly used operations that NFS currently
> cannot support properly and that are directly linked to the coherency
> issue:
> 
>  - shared mmap()
>  - append writes
> 
> There are quite a few applications out there that rely on 
> those working
> correctly in a multi-threaded environment. If we want to help 
> port those
> applications to a clustered environment without forcing people to
> redesign their entire program around the I/O quirks of the 
> filesystem we
> happen to have on offer, then we need to be able to present them with
> something that goes beyond close-to-open cache consistency.

It is not clear if the draft does or doesn't guarantee that mmap and append writes 
always will work. It says:

  "A server MUST refuse to grant a read delegation on a range that would
   overlap with a write delegation held by another client. "

So if there is a conflicting delegation, we are back to NFSv3 (or NFSv4.0)
semantics, right?

Recall that when Dave Noveck introduced delegations he emphasized that
delegations were for caching, not coherency. Making NFSv4 a coherent
(read: clustered) filesystem might be beyond the scope of this
working group.

> We also make the point that byte range locks cause you to take a huge
> performance hit. They basically only work in an environment where you
> are using uncached I/O. If you are the sole user of the file, 

For the case of making I/O combined with byte range locking more efficient,
I agree that byte range delegations are a win. The question is how many
applications need that.

You may want to have the byte range delegation operation
have the capability of granting the client more than he asked for, 
and then allow the server to recall ranges within the granted range.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 14 Jul 2005 17:42:32 -0400

> So if there is a conflicting delegation, we are back to NFSv3 (or NFSv4.0)

See the text immediately above:

   "The server MUST, however initiate the recall of any conflicting
   write delegation when the READW_LT variant is used whether or not the
   request is granted."

The sentence you quote is just a statement of the obvious requirement
that the server not give out simultaneously overlapping read and write
delegations to different clients.

> and then allow the server to recall ranges within the granted range.

That's all there.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 18:19:10 -0400

> So if there is a conflicting delegation, we are back to NFSv3 (or NFSv4.0)

No. Read two paragraphs further up. The draft allows the client to
request a recall of the write delegation. Once the existing write
delegation has been returned (or downgraded), the new request is allowed
to succeed.

> (read: clustered) filesystem might be beyond the scope of this

Are you arguing for a new IETF working group for cache consistent
networked filesystem protocols? 8-)

File delegations are indeed different as they do not attempt to
arbitrate between clients in case of a caching conflict. That does not
mean that we should not be able to offer such arbitration by other
means.

> I agree that byte range delegations are a win. The question is how many

My rapidly growing mailbox could use it. It suffers the occasional
growth spurt as postfix appends a new mail to it, but most of the
traffic consists of rereading data that did not actually change.

As Alan pointed out, databases are another case. Why bother with a
dedicated OCFS2 cluster if you can go with something like pNFS with a
lightweight locking+caching model?

Anything that uses file-backed shared mmap() for communication.

Streaming multimedia data on one client while it is being written to
disk by another?

....

> and then allow the server to recall ranges within the granted range.

This is already allowed for. The server may return a larger range than
the client requested in reply to the DELEG_RANGE request. It it not,
however, allowed to return a smaller range.

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 15:40:57 -0700 (PDT)

> delegations to different clients.

Mea culpa, though at least one part of the draft says:

"If, however locktype is either READW_LT or
   WRITEW_LT, the server should initiate a recall of all conflicting
                         ^^^^^^
                         MUST right?
   delegations prior to returning NFS4ERR_DENIED."

In any case, I'm not sure I like that the client that triggerred the
recall now has to retry an indefinite number of times in order to ensure
the mmap/append POSIX semantics, and suspect that some clients will
compromise. Thus rendering the feature moot. So to ensure POSIX semnatics
(thought not conceding [yet at least :-] that POSIX 
semantics are a good thing), the draft ought to synchronously grant
the conflicting delegation.

Regardless how conflicting range delegations are done, there is the
issue of how two conflicting clients make forward progress, which the
draft needs to address.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 18:52:03 -0400

to den 14.07.2005 Klokka 15:40 (-0700) skreiv Mike Eisler:

>    delegations prior to returning NFS4ERR_DENIED."

That is one that slipped through the net. Bruce helped me to reword the
READ case so that we didn't mandate that the server return
NFS4ERR_DENIED, and the intention was to do the same for writes.

> the conflicting delegation.

Agreed. It was our intention to allow for that.

> issue of how two conflicting clients make forward progress, which the

See the last paragraph in section 2.1.2. In the case where the
delegation cannot be granted immediately, and the client specifies READW
or WRITEW, the server will want to set up a queueing scheme like that
described for byte range locks in RFC3530.

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 16:47:50 -0700 (PDT)

> described for byte range locks in RFC3530.

"To avoid lock starvation for write delegations, the server is
   encouraged to implement the same queueing scheme that is described
   for byte range locks in Section 8.4 of [RFC3530]."

Which in turn says (excerpted): 

  "The server should maintain
   an ordered list of pending blocking locks.  When the conflicting lock
   is released, the server may wait the lease period for the first
   waiting client to re-request the lock.  After the lease period
   expires the next waiting client request is allowed the lock.  Clients
   are required to poll at an interval sufficiently small that it is
   likely to acquire the lock in a timely manner."

Given that I'm not clear as to whether the server is allowed to return
DENIED for a blocking conflicting delegation test. But regardless,
I don't believe the stuff (my stuff as I recall :-) in 8.4 of RFC3530 
applies, because advisory and mandatory
byte range locks of RFC3530 are non-recallable, whereas byte-range
delegations are recallable.

Let's say we have an app running on two clients, mmaping the same page of the
same file. The app loads a location, examimes the value, and then writes
into the location. Both apps run run in parallel. Referencing the page
triggers a page fault, thus triggering conflicting DELGE_RANGE requests
from both clients.

First one wins. Client A sends a READ to the filer. However, Client B, re-sends
his DELEG_RANGE, and the the server recalls A's delegation. Now, perhaps A is smart
enough to know that because a READ is outstanding for the conflicting range, it
should wait for the READ to complete before returning the delegation. And we'll
assume the server is smart enough to allow the READ to complete before sending
the recall. But client A's NFS file system most likely has now of knowing if the
load instruction that triggered the page fault has run. So it unmaps the page
as soon as the READ is done, and thus before the load instruction reads from memory.

B now has the delegation, but A's page fault causes B to lose the delegation.

Etc.

If you argue that A and B should get a byte range lock on the file first, I won't
disagree, except to point out that some apps mmap files into a mutex. So if A and B
are doing a mutex_enter on a mutex mapped from the same offset in the file,
the app is, by design, not going to use a byte range advisory lock.

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Jul 2005 17:41:22 -0700 (PDT)

> networked filesystem protocols? 8-)

As long as it doesn't use the term "NFS" in it name or RFC titles, absolutely,
as that will ensure it dies, like most other file access protocols not called
"NFS". :-)

Ethernet, IP, and NFS always win. :-)

> traffic consists of rereading data that did not actually change.

Well email matters. But let understand something. You are using imap/pop and
not NFS as your mailbox access protocol right? Are the imap/pop and postfix
processes running on different clients, and if so why?
 
> lightweight locking+caching model?

I don't know enough about OCFS2 works to concur or rebutt. But
for some reason the notion of combining parallel Oracle database
recovery with cache coherent NFSv4 makes me queasy.

> Anything that uses file-backed shared mmap() for communication.

That requires cache coherency to work properly though. My question was about
applications benefiting from the elimination of synchoronous I/O when byte
range locking is on. I.e the benefits of byte range caching, as opposed to
byte range cache coherency. But I think the next example is pretty

> Streaming multimedia data on one client while it is being written to

Or rather, streaming the AV stream from multiple clients while it is being
written by one or more clients, courtesy pNFS. That seems very interesting.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 15 Jul 2005 01:43:54 -0400

> B now has the delegation, but A's page fault causes B to lose the delegation.

I've read and this argument several times over, but I still don't see
your point. What you are describing appears to me to be an inefficient
implementation in the client.
It would appear to me that sort of design would be unable to support
mmap on any clustered filesystem. How is NFS w/ byte range delegations
special in this regard?

> the app is, by design, not going to use a byte range advisory lock.

A byte range delegation allows the client that does not hold the mutex
to monitor the state of that byte until it changes.

The problem here is again likely to be one of implementation: as long as
clients don't fault in/mark as dirty anything smaller than an entire
page, then they have no hints as to which range within the page to
monitor.
IOW you are saying that one is likely to see thrashing if the writer is
modifying other parts of the same page on which the mutex is located
since both clients will be trying to cache it at the same time?

I agree.

One strategy for dealing with this might be for the client that sees a
local application dirty the page to delay returning the delegation by a
second or so in order to allow further progress to be made. The
madvise() syscall might also be able to offer some hints about what to
cache here.

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Fri, 15 Jul 2005 08:37:03 -0700 (PDT)

> your point. What you are describing appears to me to be an inefficient

The point is that neither client can make progress, because each one is
triggering the recall of the 

> implementation in the client.

How so?

The virtual memory subsystem takes a page fault, the page is faulted in
via sequence of DELEG_RANGE and READ operations, and by the time the
process that triggered the fault is re-dispatched, the page is unmapped.

> mmap on any clustered filesystem. How is NFS w/ byte range delegations

Presumably clustered filesystems have strategies to cope with this, and
because they tend to not have independent clients and servers, there are
no cases of clients and servers failing to agree on the strategy.

> to monitor the state of that byte until it changes.

Sure, if the thread that is testing the value of that byte ever gets a chance 
to read it.

> page, then they have no hints as to which range within the page to

???

> since both clients will be trying to cache it at the same time?

I'm saying that thrashing is likely when two writers are modifying the same
page, and when a shared mutex mapped from the same offset of the same page 
is used to provide synchronization, it is possible that the thrashing 
is so bad that no client makes progress.

> second or so in order to allow further progress to be made. The

Possibly, but if the client's are CPU bound and loaded 
(which they might be if there's a lot of page faulting going on, 
a second might not be sufficient. Even so, a second is a long time 
though, and for pages that have conflicting range delegations, but 
far less contention than one second would warrant,
performance is going to be ugly. Apps that are used to POSIX semantics on
a local file system are going to experience latencies that will at best
disconcert the users of the apps, and at best

> madvise() syscall might also be able to offer some hints about what to

Well if you're going to change your app to add hints, why not change it to
not mmap, and use byte range locking instead. Byte range caching without
byte range consistency would still be a huge performance win, since cached data
need not be revalidated/flushed/invalidated upon lock acquisition/release.

At any rate, my point was that the proposal needs to address the issue.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 15 Jul 2005 19:53:13 -0400

> need not be revalidated/flushed/invalidated upon lock acquisition/release.

Agreed.

> At any rate, my point was that the proposal needs to address the issue.

I _strongly_ disagree. A proposal for cache consistent NFS should not
dictate to OSes how to implement mmap(), page faulting and scheduling of
processes.

All we should be showing is that the proposed protocol _allows_ you to
make progress in the case of thrashing under reasonable circumstances.
If you like, I can insert text to point out that the OS should make a
reasonable effort to delay throwing out the page and returning the
delegation until at the very least the op that triggered the page fault
has executed.
I don't see how a client can guarantee more than that. Locking too has
the same problem: a page can get ejected by the memory reclaimer for
instance before the process has accessed it. We may be widening the
window for thrashing, but we certainly didn't create it.

So, why do you believe that pNFS doesn't have the same potential issue
with its layouts?

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Sat, 16 Jul 2005 18:10:25 -0700 (PDT)

> dictate to OSes how to implement mmap(), page faulting and scheduling of

What's an existance proof of an open, cache coherent, posix-compliant file access
protocol with multiple independent implementations? That in itself is demonstrably
hard.

You want us to specify a protocol that has the explicit goal of supporting
correct mmap() and page faulting but don't want to us specify consistent rules for
how clients and servers achieve it. There is lots of stuff in RFC3530 from Dave Noveck 
that codified how clients MUST achieve close-to-open consistency. We (me) whined
for a while that this was out of bounds, but in the end we came to our senses.
If we want to do cache coherency, we have to codify how to do it.

> window for thrashing, but we certainly didn't create it.

If by "Locking" you mean byte range caching without cache coherency,
I don't see the problem. If a byte range delegation is recalled,
the client can send its byte range locks to the server, and after
that use synchronous I/O for the locked ranges. You lose performance, but
because you aren't attempting true POSIX semantics, you aren't introducing
new problems. This is the same contract NFSv4.0 whole file delegations 
provide. (But I've yet to be convinced that cache coherent NFS is NFS).
 
> So, why do you believe that pNFS doesn't have the same potential issue

It probably does have that issue (how to ensure forward progress)
with block and object-based layouts. I don't see that it has that 
issue with file-based layouts. If non-file-based
layouts prove to be impractical people will always have file-based layouts to use.
Regardless, a client always has the option of not using parallel I/O with pNFS,
so if it finds it is layout-thrashing on a file, it can back off to conventional NFS.
But once a client mounts with -t nfs4.posix, and starts up apps that depend on
POSIX semantics, backing off to non-posix is not an  option.

---

From: Trond Myklebust <[email-masked]>
Date: Sat, 16 Jul 2005 22:21:34 -0400

> If we want to do cache coherency, we have to codify how to do it.

I agree that the problem of thrashing is worth discussing. I do not
agree that considering specifying process scheduling on the client is
worth discussing.

> provide. (But I've yet to be convinced that cache coherent NFS is NFS).

You have the same problem. In the case of a low memory situation, the
kernel threads that handle recovering locks could end up not getting
scheduled until after the server forcibly revokes the delegation.

I agree this is unlikely, but you appear to be raising this issue as a
bogeyman in one case, but not the other. What is the central issue here
that I am missing?

Let's start simple with actual cases...

Two (or more) clients are contending for the same page. One obtains a
byte range delegation but with an immediate recall notice. It holds on
to it until the page has been accessed (or until one lease period has
elapsed) then returns the delegation on that page. The next client then
gets the delegation, etc... Where do you think there is a problem with
making progress in this scenario?

Agreed.

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Tue, 19 Jul 2005 13:11:36 -0700 (PDT)

> agree that considering specifying process scheduling on the client is
^^^^^^^^^^^^^^^^^^

It's worth discussing because it is an aspect of thrashing.
"Rough consensus and running code", the traditional "Implementation"
advice in NFS specs, etc.

> scheduled until after the server forcibly revokes the delegation.

Revocation is not recall. Nor is it a new problem.

> bogeyman in one case, but not the other. What is the central issue here

What you are missing is the difference between caching and cache consistency.
NFS has never taken on the latter; leaving it for other players, who have yet
to succeed in the market place. In NFS caching works as long as there isn't
contention. Once there's contention, clients stop caching.

> elapsed) then returns the delegation on that page. The next client then

Just because a lease period has elapsed doesn't mean the page has been
accessed. Just because the page has been accessed doesn't mean forward progress
is made. Although for an operating system with real time scheduling it
might be sufficient to assert that if a processed has been dispatched for
N units of time, that is a guarantee of forward progress. That may be
the solution.

Holding onto the page for the lease period in face of contention strikes
me as very undesirable.

> making progress in this scenario?

How does an NFS file system plugged into a file system switch know that
the mmapped file has been accessed? Only the processor, the memory, and 
all physical components in between the two know, and they know nothing
about NFSv4.x, though I can imagine extensions to hardware that efficiently
mark when an instruction has completed its memory access.