Subject: [nfsv4] Byte range delegations draft....

From: Trond Myklebust <[email-masked]>
Date: Mon, 11 Jul 2005 18:52:18 -0400

The draft was ready well before the deadline this morning, but I screwed
up w.r.t. the official submission process, so it probably won't be
accepted in time for the IETF meeting in Paris.

That said, I've put the document up for general perusal on our website.
The URL is

http://linux-nfs.org/rfc/draft-myklebust-nfsv4-byte-range-delegations-00.txt

Cheers,
  Trond

-------------

       Network File System (NFS) version 4 byte range delegations
            draft-myklebust-nfsv4-byte-range-delegations-00

Abstract

   This document describes a set of extensions to the NFS version 4
   protocol that enable the client to cache file data when caching
   conflicts prevent the server from handing out a file delegation.

   The proposed extensions enable the caching of only those specific
   byte ranges of data which the user application is reading or writing.

   As in the case of full delegations, a callback mechanism enables the
   server to request that the client flush cached data when a caching
   conflict occurs.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Aditya Pandit <[email-masked]>
Date: Wed, 13 Jul 2005 15:07:36 +0530

Hi Trond,

What are the events that would trigger the byte-range delegation? 
What are the events that would trigger the RPCs mentioned in the draft. e.g. 
when I open a file OPEN RPC gets called, do you recommend to pass the 
OPEN_DELEG at the same time ? similarly with others.

I see a bit of relation between this and layout delegation in pNFS.

Regards,
Aditya & Sujay

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 13 Jul 2005 07:41:42 -0400

> Hi Trond,

Hi,

> What are the events that would trigger the byte-range delegation? 

The client will want to use it if it does not hold a file delegation or
a byte range lock, but wants stronger cache consistency than that
offered by close-to-open.

> OPEN_DELEG at the same time ? similarly with others.

You cannot send the DELEG_OPEN together with the OPEN as it takes an
open_stateid and open_seqid argument. It must therefore be called after
OPEN, but before the first call to DELEG_RANGE or READ.

As for the others, if you want POSIX-like cache consistency, you
probably want to do something like:

  - for most read-only files, READ will do most of the job, as long as
you remember to precede it with a DELEG_PUT_STATEID.

  - for writeable files, you want to reserve the range to be modified as
soon as the user calls lockf() or write() by means of a call to
DELEG_RANGE. You probably want to downgrade to a read delegation as soon
as you call COMMIT in order to try to limit the number of recalls.

All this is very workload-dependent, though, so other schemes may make
more sense if the goal is not POSIX.

> I see a bit of relation between this and layout delegation in pNFS.

That is quite deliberate. 8-)

Note, though that they _are_ different since they refer to different
objects. You should never assume that a layout delegation gives you a
right to cache the actual data (see the discussion in section 11 of the
latest pNFS ops draft), hence the need for something like this.

Cheers,
  Trond

---

From: Benny Halevy <[email-masked]>
Date: Thu, 14 Jul 2005 08:44:43 +0300

> offered by close-to-open.

In reference to Aditya's question: these RPCs are not tied 
with any specific posix system call.  I can forsee them 
being used in relation to memory mapped files to achieve 
distributed cache consistency.  Basically, you want to 
protect your clean cached pages with a read delegation and 
any dirty page with write delegation (before the dirty daat 
is committed to the cache of course).

> OPEN, but before the first call to DELEG_RANGE or READ.

This reminds me of the discussion to add to the protocol a 
"current stateid", similar to the current fh, that could 
carry the stateid returned by one operation to further 
operations in the compound RPC...

I think that the only reasonable way to do it is defining a 
new special value for the stateid that will be interpreted 
as the "current stateid".

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 07:50:23 -0400

> operations in the compound RPC...

I've been one of the main supporters of that concept, so I'm well aware
of it. ;-)

I'm not sure that you want to do that here, though, since you would end
up bumping the same open sequence id twice in the same COMPOUND (once
for the OPEN call and once for the DELEG_OPEN). That again means that
the COMPOUND can never be replayed in case, say, where a network
partition causes the initial reply from the server to be lost.

Another case for the SESSIONS crowd to solve. 8-)

> as the "current stateid".

The proposal includes a concept of a current delegation stateid, which
is used in order to allow the client to pass both the ordinary open
stateid and the delegation stateid to operations like READ, WRITE and
friends.
Not exactly what you are proposing here, though.

Cheers,
  Trond

---

From: Benny Halevy <[email-masked]>
Date: Thu, 14 Jul 2005 16:42:19 +0300

> 4.4  Use of CB_GETATTR for querying the size attribute
 >
 >    If a client holds a write delegation that extends 
across the end of
 >    file, then it may cache SETATTR or WRITE operations 
that will cause
 >    the size attribute to change.
 > ...
 >    Note that the server MUST NOT issue a CB_GETATTR query 
for any
 >    attributes other than size.

What about the change attribute?
Should clients generate it when they have a write delegation?
How do clients sync back with the server when the write 
delegation is recalled?

Benny

---

From: Aditya Pandit <[email-masked]>
Date: Thu, 14 Jul 2005 19:22:12 +0530

Hi,

I agree with Benny. They could be very useful for mmaped files. 
I also support this concept.

-Aditya

On Thursday 14 Jul 2005 11:14 am, Benny Halevy wrote:
> >>What are the events that would trigger the RPCs mentioned in the draft.
> >> e.g. when I open a file OPEN RPC gets called, do you recommend to pass
> >> the OPEN_DELEG at the same time ? similarly with others.
> >
> >>>http://linux-nfs.org/rfc/draft-myklebust-nfsv4-byte-range-delegations-00
> >>>.tx t
> >>>
> >>>Cheers,
> >>>  Trond
> >>>
> >>>-------------
> >>>

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 10:09:24 -0400

> Should clients generate it when they have a write delegation?

I don't see how clients can know the immediate value of the change
attribute unless they have a file delegation.
It seems to me that the only sane behaviour is for the server to manage
it in much the same way we do today for the close-to-open case (i.e.
change attribute is bumped when the WRITE or COMMIT comes in).

> How do clients sync back with the server when the write 

See section 4.2. It is very much the same model as for file delegations
(client recovers locks, writes out dirty data, then returns or
downgrades the delegation - other clients wait until the process is
done).

Cheers,
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 10:25:43 -0400

> change attribute is bumped when the WRITE or COMMIT comes in).

Just to expand a bit on this. We can easily deal with the case of the
size attribute, because only 1 client can hold a write range delegation
that covers the end-of-file region, but for the case of the change
attribute, you would have to interrogate _all_ clients that hold write
delegations in order to figure out whether or not they are caching some
change to the file. That could never be made to scale adequately.

As the draft mentions, though, we are not trying to offer any new
guarantees on the metadata.

Cheers,
  Trond

---

From: Benny Halevy <[email-masked]>
Date: Thu, 14 Jul 2005 17:55:53 +0300

> attribute unless they have a file delegation.

If a client that has a write delegation caches a write and a 
getattr comes in locally, what should it return for "change"
(assuming the local API has it of course)?  My assumption is 
that the client wants to bump "change" locally (as well as 
update mtime) to indicate a modification to the file.

> downgrades the delegation - other clients wait until the process is

Rephrasing my question... I meant how does the client sync 
back with the server its local value of "change", in case it 
served the new value locally when it had the file write 
delegation.

---

From: David Robinson <[email-masked]>
Date: Thu, 14 Jul 2005 10:46:28 -0500

> I've been one of the main supporters of that concept, so I'm well aware

Good, I just wrote one.  It is in the I-D queue as it missed the -00
deadline. I'll send out a pointer once I get it on a public web site.

	-David

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 14 Jul 2005 11:53:26 -0400

> update mtime) to indicate a modification to the file.

I'm not sure that assumption is valid. We certainly do not do anything
like that now.

        NFS offers no guarantees that the client and server clocks are
        in sync, so client updates of mtime/ctime are generally
        considered a bad idea.

        propagating the change attribute to user applications would
        require non-portable extensions to fstat(), and there has so far
        been no demand for such extensions.

> served the new value locally when it had the file write 

It doesn't propagate an exact value. See the discussion in RFC3530,
section 9.4.3.

A client holding a file write delegation is encouraged to bump its
cached value of the change attribute by one unit if it has made at least
one change to the file. In subsequent CB_GETATTR callbacks, the client
MAY choose to reply with that same value, despite having made more
changes.

When the client returns the delegation after having modified the file,
the server is then supposed to bump the change attribute by at least one
more unit, and set the mtime and ctime to the current time.

---

In the case of a byte range delegation, it probably makes more sense for
the server just to assume that a client that requests a write delegation
is doing so because it wants to modify data. It can just preemptively
bump the change attribute rather than issuing a callback.

Upon return or downgrade of the delegation, it could then do the same
thing as described for file delegations.

Cheers,
  Trond