Subject: initial delegation experience

From: "Kendrick M. Smith" <[email-masked]>
Date: Thu, 31 Jan 2002 01:06:05 -0500 (EST)

Hi all,

I have recently finished implementing delegation for the Linux client and
server, and have a few implementation issues to report:

1.  Suppose that client A has a delegation on a file, and the server
receives a REMOVE, RENAME, or SETATTR request from client B on the same
file.  If A != B, the server must recall the delegation.  If A == B, it
would be nice if the server could handle the request without recalling
the delegation.  As the protocol stands, this is impossible: the server
has no way of knowing if the request really came from client A, or if
it came from a different client tunelling through the same NAT box as A.
One solution to this problem would be to add a clientid to REMOVE, RENAME,
and SETATTR (note that truncating SETATTR's contain a stateid, so the
clientid can be inferred, but non-truncating SETATTR's do not).

2.  The CLAIM_DELEGATE_CUR branch of OPEN calls for open-by-name rather
than open-by-filehandle.  In the Linux implementation, open-by-filehandle
would be considerably more convenient.  With open-by-name in the protocol,
we have to save the containing directory's filehandle, and the file's
name, side-by-side with the delegation.  The reason why we cannot save
them in the usual places in the VFS is a long, complicated story having to
do with race conditions against umount().  I suspect that other Unices may
have the same problem.  (I can describe the "long, complicated story" in
detail if anyone is interested in the gory details.)

Also, it seems to me that open-by-filehandle would make more sense than
open-by-name for the CLAIM_DELEGATE_CUR branch of OPEN.  For one thing,
servers have to support open-by-filehandle for the CLAIM_PREVIOUS branch
anyway, so the implementation burden for servers is not any higher.  For
another, the client is trying to open a specific file for which it already
holds a delegation (no other file will do; it has to be the file
corresponding to the delegation).  Why not just use the fh of that file,
rather than the fh of the containing directory along with the filename,
effectively asking the server to do a LOOKUP?  It seems simpler, less
error-prone, and more efficient.

Is there some reason that I'm missing for using open-by-name in the
CLAIM_DELEGATE_CUR branch of OPEN?

3.  This last one isn't really an implementation issue, more of a
question. One central requirement of the NFSv4 protocol design process has
been the requirement that a LOCK/LOCKU operation change the file's
stateid, so that replayed WRITE's are "locked out" and return
ERR_OLD_STATEID.  In this way, LOCK/LOCKU acts as a serialization point
for WRITE requests.  In fact, most of the complexity of the NFSv4 state
model can be attributed to this requirement, including the recent .x
changes.

The plot thickens, however, with sec. 9.4.2 of RFC 3010 in the picture.
According to this section, a client which holds a delegation is supposed
to perform its locking operations locally.  If such a client performs
interleaved locking and WRITE operations, the locking operations will
be performed locally but the WRITEs will eventually be flushed to the
server.  From the server's point of view, the client is only sending
WRITE's, with no LOCK/LOCKU's to update the stateid.  The serialization
properties of LOCK/LOCKU are lost when a delegation is held.

So, my question is, should the client really do as sec. 9.4.2 advocates,
and perform its locking operations locally?  In other words, which is
more valuable:
  (a) The potential performance advantages of performing locking
      operations locally whenever a delegation is held
  (b) The design requirement that LOCK/LOCKU act as a serialization point
      for WRITE requests?

Aside from these relatively minor issues, the Linux implementation of
delegation was quite successful.  In certain situations, delegations
drastically reduced the amount of traffic.  For example, one test that
I used during development is repeated execution of a shell script, after
it has already been read into the page cache.  In NFSv2/v3, this results
in a stream of GETATTR requests as the client revalidates the cache in
close-to-open fashion.  In NFSv4, with delegations enabled, there is no
traffic whatsoever! (aside from a one-time ACCESS call to warm up the
access cache)

Cheers,
 Kendrick

---

From: Jim Rees <[email-masked]>
Date: Thu, 31 Jan 2002 07:53:57 -0500

This is not a problem, as nfsv4 (or at least delegations) doesn't work thru
NAT.  Brent proposed adding this to the issues list in October but it
doesn't seem to have been added.  Has this been resolved?

---

From: David Robinson <[email-masked]>
Date: Thu, 31 Jan 2002 10:35:37 -0600 (CST)

> clientid can be inferred, but non-truncating SETATTR's do not).

I see this as the client getting what it asked for. The semantics
are clear on when the delegation is recalled by the server, if
the client decides to issue a request back through to the server
that will cause a recall then it gets what it asked for.  A smart
client that has the delegation will delay the request until either
the server recalls the delegation or the client is willing to
give it up. It means the client needs to maintain more state
about operations that need to be completed and of course makes
interesting recovery issues, but then all of delegation has that
property.

>   (b) The design requirement that LOCK/LOCKU act as a serialization point

But there is serialization, it is just now occuring at the granularity
of the delegation and not the LOCK/LOCKU. Serialization is important
for two reasons, creating consistent state between two clients
and protection against replays corrupting state. With a delegation
the former can't occur because any other client attempting an
operation will cause the delegation to be removed.  If that occurs
the client will flush the pending writes as well as the current
lock state.  From the server's perspective it just sees the consolidated
set of requests. Locally the client provides synchronization within
its buffers and the lock requests to replays are not an issue.
I have not checked the latest draft, but the only implication here
is that the stateid MUST change when a delegation is given or
recalled in order as the delegation is the course grained
synchronization. This is needed no only to handled the case
where the client has performed locking but to also keep a
replay from causing a recall or invalidating the clients cached
data.
	
	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 31 Jan 2002 09:23:56 -0800

David Robinson wrote:
[ > Kendrick Smith wrote:] 

> that will cause a recall then it gets what it asked for.  

True.  However, ...

> A smart
> give it up. 

In the case of RENAME, for instance, such a client would not be
so smart.  Suppose I have a delegation on file X and then one of my
processes renames it to Y.  If I were to hold back the rename
and only do it when the delegation were to be recalled, as you
suggest, my implementation would not provide the normally expected
semantics of rename.  First, other clients would not see the 
the name change in a READDIR for an unbounded time.  Also,
another client opening Y, would not cause the delegation to 
be recalled, as it should.  Instead, he would get ENOENT.
This state of affairs would continue until somebody tried
to open X, if that ever happened, or until the client got
tired of holding onto its delegation.

> It means the client needs to maintain more state

But RENAME is really a directory operation and we don't have
directory delegation.  (I know some people would like to have
it).  What Kendrick is really asking for is an extension which
would allow the name change to happen (since it is a directory
operation) and leave the delegation unchanged.  My inclination
is to draw a line under what we have and leave extensions (as
opposed to fixes and clarifications) for a future minor version.

It occurs to me that the information that Kendrick would like 
to have available to make this decision is present in DAFS, since
it is a connection-oriented protocol that has a virtually
identical delegation model.  It is probably OK in DAFS to do
what Kendrick suggests, although the 1.0 spec may not explicitly
address the issue.  This would be an issue for anyone else out
there who is implementing both protocols.  Because these two
protocols have so much in common, there can be considerable code
sharing between the two implementations.  However, a case like
this points out that you do have to be careful to note the 
differences.

---

From: David Robinson <[email-masked]>
Date: Thu, 31 Jan 2002 11:52:26 -0600 (CST)

> opposed to fixes and clarifications) for a future minor version.

Ah, I see my invalid assumption, I forgot that we do not have
directory delegations.  Thus a directory operation must
go through to the server.  I would not be in favor of an
effort to hack the protocol to make up for the lack of
directory delegations when it would be cleaner to just
provide directory delegations. Which in any case is probably
better to delay to a future minor version.

	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 31 Jan 2002 10:31:29 -0800

> server, 

It appears I'm going to be very busy for the next few weeks getting
ready to inter-operate.

> and have a few implementation issues to report:

Three is a lot fewer than I expected :-)

> [ 1. Deleted.]   

> detail if anyone is interested in the gory details.)

If nobody wants to hear it on the mail alias, you can explain it over
a drink at Connectathon.  I'll buy.  Everyone at Connectathon with a 
V4 implementation is invited.  

> CLAIM_DELEGATE_CUR branch of OPEN?

I think it was just a mistake.  When those cases were defined, I
think I just took the standard one and modified it only as much as I
had to.  I knew CLAIM_PREVIOUS had to be modified to be by-filehandle
but I didn't see that CLAIM_DELEGATE_CUR had to be, so I left it 
being by-name.  Sorry.

[ 3. Deleted]  

> traffic whatsoever! (aside from a one-time ACCESS call to warm up the

Cool!